why high to low, look out below?

i'm studying for my private license and i am a bit confused about this
high to low, look out below. the way I see it:

imagine an altimeter sitting at sea level. lets say the atomspheric
pressure at the location of the altimeter is "10" (no units, just for
simplicity). now lets say you set the altimeter to "10". so now the
altimeter reads "0 feet"

now decrease the temperature of the air surrounding the altimeter, and
since the colder air gets the more dense it gets, the atmospheric
pressure goes up. the altimeter doesnt know this. all it sees is an
increase in pressure, which it thinks means an decrease in altitude, so
it indicates say "-10 feet". this is directly contrary to the "high to
low look out below" because the altimeter is telling you you are 10
feet lower than you are, which is not what that saying is implying.

SO WHATS THE DEAL?