I had this on my 2.8Q. It's most likely the blower motor. The brushes wear out over time which gives you the intermittant problem you're experiencing. Mine packed up altogether at one stage so I took it apart cleaned the brushes and it was fine again for about 2 weeks, then started playing up again (this was during a particularly hot 2 week summer).
I tried replacing them with ones from an alternator but this made the motor really noisy so in the end had to bite the bullet and buy a new motor.
Audi & VW, as always, wanted £365, but I eventually found the exact same Valeo unit from Audi4U for £190.
Nobody seems to do replacement brushes and it's a common problem (talking to various VAG specialists/breakers) and if you managed to find one at a breakers then there's no telling how long it will last.
Shortly afterwards there was a guy selling them on ebay for £120 brand new, but he seems to have disappeared.
Audi4U are on 01278-434141
The part no. for the climate control version is 8D2 820 021 (it's a different part for non CC)