Hi there, you said it got worse after heavy rain, this is a common fault on A4 & Passat and possibly some others. Its caused by water ingress into the passenger foot well because it runs over all the wires behind the panel on the left hand side. Its an easy fix just remove the pollen filter housing and put a generous bead of silicone around the foam seal that goes between it and the car and Bob's your mothers brother !!
Hope this helps