I don't think its as much about the amount of speakers but more about the quality they produce. Like I say I think it varies from person to person, and perhaps I have a distorted view of this based on my last speakers (which subject to opinion, but for me they were the best standard speakers I have ever heard in a car - 2010 Ford focus Titanium).
If I were buying again now, after hearing the B&O compared to the Audi sound system, I would consider the B&O to be a massive improvement. I know it is a big cost which is why I also didn't go for it, but for me the Audi Sound System is weak for the money you pay. The sub in the boot makes only a small difference (perhaps the car too well insulated...) and as I like my music loud, the clipping it does on the bass is too much. My Focus didn't distort at any volume - I was told the Audi speakers would improve in time but 3 months later I have still seen no improvement.
If your budget doesn't allow it, which mine didn't, I'd say save your £300 and spend an afternoon installing your own speakers. For £300 I am sure you can get some decent aftermarket ones which do a far better job.
But... apart from that..... I LOVE THIS CAR!!!!! It gets so much attention when driving about.