This is an interesting general design-pattern question, and not really
CLR-specific. The CLR does not have some magic way of doing this without
creating some sort of wrapper class. So you will need to create a public
wrapper class with all the APIs of the object, but it should defer to an
internal object that  is only created on demand. It means you have to write
a lot of scaffolding code depending on how many properties and methods
these classes have, and it will cost a little bit of performance, but it
sounds like it is worth it for you in this case for the working set. To
minimize that, the objects could be structured into sub-objects, with the
top-level object having just a few properties.

The CLR does offer you one magic trick for dealing with the memory here,
which is to use the System.WeakReference class. If your wrapper class uses
a WeakReference instead of a regular object reference to manipulate the
internal object, the GC itself will clean it up if memory is getting really
tight. This makes the implementation a lot simpler for you.

I hope this helps.

--------------------