Obviously it's OK to hand over PIC during the flight. It also seems that by definition if you hand off the controls to the other person, whether you're reading a map or taking a nap--you just handed over PIC responsibility and you can't log that time.

No, you did not hand over responsibility, but it's true you can't log the time.
Logging PIC and being PIC are two different things. Who is acting PIC is a matter
which should be decided before the flight.

On a slightly different topic, is it legal by FAA standards to let a passenger take the controls of an aircraft?

Yes. When this happens, you are still PIC, but you can't log the time as PIC.

George Patterson
If you want to know God's opinion of money, just look at the people
he gives it to.