never really tried to fly with a slight layer of frost or ice, I
stayed; but I wonder if I was too cautious.

Never...you did the right thing

It would have been very convenient for me and my passengers to go.

It wouldn't have been so convenient to become a landmark in the side of a
mountain....

Here's my question: just
how much ice/frost is too much?

Any ice/frost is too much unless you want to be considered a test-pilot.
The challenge here is at what point do you call "uncle"... A little frost
probably would not be of great concern...but how about some frost and a
little ice....the ice is smooth, clear and only on the leading edges....etc
etc etc

In my opinion this could have been a recipe for disaster. You were in a
hurry to get home...at night...in the mountains...with airframe ice. Most
accidents are a series of events and/or poor decisions. Deep down you knew
the right answer, and heeded the caution your gut was telling you.... Your
passengers owe you a debt of gratitude.