I'm having trouble seeing this in my mind...

Were you coming from up top (PSP) or from V64? 4000 I think is the MEA
between TRM and PSP, right (V137 TRM radial 304)? I am having trouble
"seeing" in my head how you can be "south of the VOR-C" and 4 miles
from the VOR and closer to the airport than the VOR when the VOR is
"generally" south of UDD... what was your heading at the time? Were
you pointed to the VOR or the field?

Even so, I think there must have been some sort of communication
breakdown, either between you and the instructor, you and the
controller, or the instructor and the controller. There could have
been erroneous assumptions made (perhaps the controller had just
cleared a bunch of aircraft VMC? perhaps the controller had told you to
expect vectors to visual and you forgot? If you were coming from PSP,
perhaps the controller erroneously thought that since you had pretty
much overflew the field at 4000, you could see the field (thinking it
was VMC), or the instructor radioed that s/he could see the field...

In any event...clarifications should have been in order...