From my point of view (there might be someone here with more experience that will come along and answer for you but in the meantime...)
Basically, the law governs the standard of quality (workmanship and materials) under The Supply of Goods and Services Act 1982. As this is statute law, this will imply terms into your contract (even if not expressly written)...
4 Implied terms about quality or fitness.
(2A)For the purposes of this section and section 5 below, goods are of satisfactory quality if they meet the standard that a reasonable person would regard as satisfactory, taking account of any description of the goods, the price (if relevant) and all the other relevant circumstances.
13 Implied term about care and skill.
In a contract for the supply of a service where the supplier is acting in the course of a business, there is an implied term that the supplier will carry out the service with reasonable care and skill.
From what I can gather reading what you've said about the quality of his work, I'd say that he is in fact in breach of contract (taking into account the implied terms above). If the work is not to a standard that you'd expect an ordinary competent contractor to perform then he hasn't worked with reasonable skill and care.
I'd say that you're well within your right to withhold some money if you think the work is defective. Withholding payment all together would no doubt be breach of contract, but paying an acceptable amount (to cover the cost of what's been done so far) should be acceptable. What you will have to do though is formally write to him with a list of all the defects that need to be put right and give him a timescale in which to complete them - you might even wish to get a Chartered Surveyor to pass the works off once complete. If he refuses to do this then you should be within your right to seek another contractor to rectify the works (and in extreme circumstances you might even have a case to pursue damages if he's done such a bad job that it's made more work for you).
Make sure you're clear that if he doesn't finish the works to a satisfactory standard (with reasonable skill and care to that of which you'd expect an ordinary competent person to) then you intend to hire another contractor to complete the works on their behalf, deducting the cost from the remaining balance of their original contract sum.
The key for me is giving him the chance to rectify the works (and importantly making him aware of the poor quality of his work) before you rescind the contract, otherwise you may find yourself in breach of contract yourself and end up paying him the full amount regardless. Keep copies of all correspondence (letters, emails, quotes etc).
Sorry for the essay - it's easy to go off on one! I hope it's of some use to you.
Note: I've gone under the assumption that you have some form of contract in place (be it formal or an accepted quotation). Even if it was just a spoken agreement you will still have an oral contract (easier to prove the terms of if you have witnesses - a partner etc that was present at the time). I've also assumed that you don't have any form of retention agreement in place for defective works. Just one more thing to note - if your contract is essentially a signed quotation of his or something similar, just quickly read through his T's & C's on the back (assuming he has any) and make sure he's not slipped anything in that could trip you up. Even if he has, you might find yourself protected by the Unfair Contract Terms Act 1977. Lastly; good luck!