i am in EXACTLY the same situation as the OP paulB and decided to go for the 3.2 V6
it has and extra 20 bhp or so on the 8L S3, around 50lbs more torque, quicker on the 0-60 and of course as mentioned in an earlier posts above, its a newer shape/chassi altogether which looks a WHOLE lot more better than the previous one, i reckon they'd drink about the same with my driving and both have quattro too so the 3.2 is a winner for me.
it just looks better in terms of exterior and interior
im sure the engine and car in general will be in good nick too as its a fairly new car (came out 2003, yours will be newer as you can afford a 2004-2005 model)
oh and i do like the roar that the V6 gives
of course as mentioned, the engine is not as tunable as the S3's and that i think is the ONLY thing the 8L S3 has on the 3.2, its turbo
you can do a little to the 3.2 (exaust system, performance filter, remap) but if you wanna get serious, just turbocharge/supercharge that bad boy, expensive but then it will own all