I think the contact is dirty on the switch. I think if the relay was at fault it wouldn't move at all and it would usually effect more than one window. Try removing the switch and disassemblying it, use a pencil eraser to clean off the contacts and then put it back together.
Troy
Troy