One could legally be operating in VMC under IFR or IMC under VFR.

I think you are only half right. You can surely operate IFR in VMC --
and there are places where that's actually required, but if you are in
IMC, which is defined as in conditions below VFR minima, you had best
be flying IFR.