Does anyone know what compound is used to secure the nuts and threads on
aviation engines? I'm not talking about loctite, but the compound that is
placed on the outside of the nut and then onto the threads. Usually, it's a
very visible colour and laid in a straight line so that one can visually see
if a nut or bolt has backed off.

It is called Inspector's Anti Sabotage Lacquer or something similar depending
on the maker. It does not itself secure the nut or bolt but serves as a
witness mark to show whether or not the fastener has backed off. It is
required for use on the Cessna oil filter adapter. Most aviation supply
outlets carry it even if they don't list it in a catalog. I have on occasion
backed nuts off with the lacquer strip not breaking and it will sometimes break
and fall off without any lose in torque on the fastener.

I've seen it done with silicon,

I can't see any advantage to using silicon since it is so flexible the nut
could loose torque with silicon simply stetching slightly.