because you cannot go direct with a hand held GPS

Why not? I hardly ever file direct and when I do, ATC will amend the clearance.
However I often hear ATC tell me to go direct.
In my IFR flight plan I specify that I have a VFR GPS on board.
Hank