I don't want to use anything that will hurt
the paint, aluminum, or rubber.

Mineral spirits.

mineral spirits will eat away at plastics and rubber. I can't
say it works but isopropyl alcohol works well at work where
materials and residue are quite sensitive (linear accelerators
with exotic materials and lots of radiation). If you don't
want a residue get high grade (90% plus) but you might
have to get that at a special store as it is considered
a HazMat.

Gerald