Hello mate, I'm a bit late to the party on this one but I thought I'd offer my bit anyway...
I've owned my S3 for over 2 years now, and my previous car was a mk4 Deep Pearl Blue VW Golf R32 - which I owned for 3 very happy years!
The R32 was a great car, and ticked a lot of boxes - it looked great, had strong perfromance, it wasn't a car you saw on the roads all that often, has a good reputation, sounds fantastic, has 4-wheel drive and handles well, and the interior is very decent and well built as well.
I loved my R32, but it was a noticeable step up in performance when I got the S3.
That was the biggest diffeerence for me. The R32 was a decent performer, but you don't get the same progressive surge in the R32 as you do in the S3. The mk5 would be closer performance-wise, but the mk4 would probably feel a little sluggish to be honest. I think you may also find the whole 'experience' of driving a mk4 R32 after an S3 a little disappointing as well - as it's a bigger, bulkier, and less agile feeling car.
As I mentioned earlier, the R32 ticked a lot of boxes and I loved mine, but I think you might struggle with the difference in performance, although it did sound absolutely BRUTAL when you put your foot down! God I miss that sound...
Another thing worth noting is that fact that unless you're looking at a pre March 06 R32, tax will cost you over £400 a year - and with regards to MPG, you're not going to save an awful lot because I pretty much got the same MPG from My R32 and I'm currently getting with my S3 - it may have actually even been a little bit worse...
Good luck mate, I'm sure you'll make the right choice...