These are very strong, durable cars. The only real issue with the 3.0TDi is that they are prone to premature wear of the inlet manifold swirl flap assemblies, particularly on the offside bank, near to the turbo. This makes very little difference to the way the engine runs - a fraction less low-down torque perhaps - but it causes the engine management light to come on. Ask any Audi dealer, and they will insist the only solution is to replace the entire assembly, which costs in the region of £700 per bank. If you speak to Fontain in Iver, they have devised a modification to the linkage that will solve the problem in most cases. They charge £225 for it, and guarantee the work for twelve months.
That aside, the 3.0TDi is an outstandingly good car. It'll never feel sporty like an S4, but it goes like the wind, and is effortless in the process. Personally, I think they are best suited to an auto box, but others will differ on that. The answer is to try both, and decide for yourself.
The engine was initially rated at 201bhp (204PS) at launch, but in late 2005, was uprated to 229bhp (233PS) when the TDV model was introduced, with a diesel particulate filter. I believe they can be re-mapped up to about 275bhp. They have monstrous amounts of torque, and of course they are all Quattros.
Oh, whether you get an SE or an S-Line, make sure you fit the rear anti-roll bar from the RS4. Parts cost less than £100, and only need a hour or so's labour to fit. It will transform the handling without affecting the ride.