Can the safety pilot be PIC even while the left-seat pilot is not under
the hood?

I'm guessing, no, based on:
(g) Logging instrument flight time. (1) A person may log instrument
time only for that flight time when the person operates the aircraft
solely by reference to instruments under actual or simulated instrument
flight conditions.

In simulated conditions, once the hood is removed, it would seem to me
that only one or the other could be PIC at that point???

So... I'm beginning to sway to the side of, logging XC as safety pilot
seems to be a stretch!

Another good argument for getting that instructor rating:
(3) An authorized instructor may log as pilot-in-command time all
flight time while acting as an authorized instructor.