Class E extension

I am still confused on this particular issue and was hoping somebody in here
could point me in the right direction. I'm sure most of you are familiar
with the class E extension from C&D airports to help facilitate instrument
approaches. Well here's the question. If I can operate in VMC in the class
E extension for a D airport, am I required to talk with tower for that
airport and can they deny me entry to that airspace? The AIM says I have no
comms requirement so long as I maintain VMC. I can't find anything in the
FAR to back this up, nor can the FSDO, and thus far the tower is under the
impression that when they are IMC, they own the E extension as well.

A response to my e-mail as well as this newsgroup with any thoughts on this
matter would be very much appreciated.

Tks,