Drivers side window not going down

Hi all, had a quick search and most cases the window not working is the loom. In my case though everything else on the door , passenger window and wing mirrors all work and when I use the switch I can hear the motor clock in a relay clicking kind of way but only when I try upwards. There's no noise or action when I try and put it down.
I had a spare working motor kicking around but that hasn' changed anything.
Just wondered if anyone had any ideas of where to start looking? Cheers Russ

Thank you. I'll pull it apart and have a look. I doubt it's the loom though as everything else on the door works, the switch lights up with the lights on so should in theory be getting power. I would just buy another switch but I have a poverty spec one which has manual rear windows and every switch I see is a 4 button one for full electric windows