What I've done in the past is get two bankers drafts drawn up, one for the finance company based on the settlement letter I was shown (confirmed by me on the telephone too), and another for the balance to the seller.
On the day of pick up, I gave him his draft, and then in front of him I put the finance company draft in an envelope and we both went to the post office to send it Special Delivery.
If the seller won't agree to some similar arrangement that suits you both, then I agree, walk away.
The credit card payment over the phone might work too, but they'll probably want fees to do it that way, so make sure you knock it off what the seller is getting!