Absolute lowest altitude you can fly (legally)

What regulations determine the absolute lowest altitude you can fly
above the ground in the U.S.? I understand that the area just above
the ground is usually Class G outside airports, and it only goes up to
700 or 1200 feet most of the time ... which implies that you can
actually fly at 500 feet AGL if you want. But is there some other
regulation that prohibits aircraft from flying this low, in general or
in certain conditions/areas?

--
Transpose mxsmanic and gmail to reach me by e-mail.