Strategy for caches & GC

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

devdude - 20 Apr 2008 06:54 GMT

Hello,

I am currently using the Enterprise Library Caching Block on a server
that receives incoming calls and stores/retrieves random information.
I noticed while load testing, the system is slow to respond (a call
can take several seconds) during scavenging of old items when the
threshold is met. It is reclaiming approx 4gb of data during each
scavenge on approximately 14gb of cache. I realize this is a lot of
information but am looking for strategies to implement a performant
cache in .Net. I'm not particularly married to entlib so anything is
an option.

One thought I had was to perhaps rewrite the cache (or a new one from
scratch) in C++/CLI to handle all the memory storage/reclamation in
native code but am concerned with marshalling data back and forth
between native/managed code. That said, the GC multisecond swings are
worse then knowing that a call to the cache will take, for example,
5ms, instead of 1ms.

Thanks for any thoughts.

Reply to this Message

Barry Kelly - 21 Apr 2008 11:35 GMT

> I am currently using the Enterprise Library Caching Block on a server
> that receives incoming calls and stores/retrieves random information.
[quoted text clipped - 5 lines]
> cache in .Net. I'm not particularly married to entlib so anything is
> an option.

It sounds like you need a more aggressive cache expiration policy. As
Rico Mariani [Google] likes to say, a poorly implemented expiration
policy is just a memory leak by another name. What you say makes me
think that the cache is getting a too-low hit rate, so it needs throw
away most of it too often. Perhaps it shouldn't be caching much of this
stuff at all in the first place?

When dealing with large amounts of data, I find GC is often not the best
way to go. I end up writing manual allocators for byte arrays. This
makes most sense when the data you're caching is in fact a byte array,
or other value-typed array, of course. However, I still allocate those
byte arrays from the .NET GC. It's just that they're all over the large
block limit (80K or so), and I carefully recycle them through a pool.

> One thought I had was to perhaps rewrite the cache (or a new one from
> scratch) in C++/CLI to handle all the memory storage/reclamation in
> native code but am concerned with marshalling data back and forth
> between native/managed code. That said, the GC multisecond swings are
> worse then knowing that a call to the cache will take, for example,
> 5ms, instead of 1ms.

BTW, you can still do manual memory allocation in C#, using the Marshal
class or by writing the PInvoke signatures yourself. However, I'd take a
closer look at your expiration policy before I'd go off and try and
reimplement a cache.

-- Barry

Signature

http://barrkel.blogspot.com/

Reply to this Message

devdude - 21 Apr 2008 12:05 GMT

Barry, thanks for the response, see inline.

> It sounds like you need a more aggressive cache expiration policy. As
> Rico Mariani [Google] likes to say, a poorly implemented expiration
> policy is just a memory leak by another name. What you say makes me
> think that the cache is getting a too-low hit rate, so it needs throw
> away most of it too often. Perhaps it shouldn't be caching much of this
> stuff at all in the first place?

As mentioned, this was stress testing so I wanted to push the cache to
the size limit and see how it would respond while scavenging.

> When dealing with large amounts of data, I find GC is often not the best
> way to go. I end up writing manual allocators for byte arrays. This
> makes most sense when the data you're caching is in fact a byte array,
> or other value-typed array, of course. However, I still allocate those
> byte arrays from the .NET GC. It's just that they're all over the large
> block limit (80K or so), and I carefully recycle them through a pool.

Can you explain this further? Do you mean you alloc a huge block of
memory to insure it sits in the LOH and suballocate from there (ala
your own memory management on the managed heap?) If so, how does the
suballocations look? Does C# have a placement new like C++?

> BTW, you can still do manual memory allocation in C#, using the Marshal
> class or by writing the PInvoke signatures yourself. However, I'd take a
> closer look at your expiration policy before I'd go off and try and
> reimplement a cache.

How does one serialize/deserialize managed objects onto the native
heap if these managed objects can be derived from a common base (e.g.
CacheItem<-MyCacheItem, CacheItem<-MyOtherCacheItem)? I'm (also)
wondering if memory fragmentation will start occurring in the native
heap?

Thanks!

Reply to this Message

Barry Kelly - 21 Apr 2008 14:29 GMT

> Barry, thanks for the response, see inline.
>
> > It sounds like you need a more aggressive cache expiration policy. As

> As mentioned, this was stress testing so I wanted to push the cache to
> the size limit and see how it would respond while scavenging.

Sure. All I was saying is that if stress testing shows problems, then
perhaps it indicates a lax expiration policy.

> > When dealing with large amounts of data, I find GC is often not the best
> > way to go. I end up writing manual allocators for byte arrays. This
[quoted text clipped - 6 lines]
> memory to insure it sits in the LOH and suballocate from there (ala
> your own memory management on the managed heap?)

No, though I have done something like that too, and used a kind of
ArraySlice<T> handle to work with buffers. I worked strictly with byte
arrays, though the code was generic and would have made sense with any
value type. The primary use for the byte arrays in my case was as
buffers for I/O. However, if you implement your cache store such that it
can be processed in a streaming fashion, then you can rely on the LOH
and manual management for memory and GC generation 0 for efficient
cleanup of temporary structures.

> If so, how does the
> suballocations look? Does C# have a placement new like C++?

No. The CLR doesn't support directly mixing managed and unmanaged
memory. Managed->unmanaged needs to be an opaque pointer, usually
IntPtr, while unmanaged->managed needs to be a GC root, e.g. GCHandle,
or gcroot<> in C++.

> > BTW, you can still do manual memory allocation in C#, using the Marshal
> > class or by writing the PInvoke signatures yourself. However, I'd take a
[quoted text clipped - 4 lines]
> heap if these managed objects can be derived from a common base (e.g.
> CacheItem<-MyCacheItem, CacheItem<-MyOtherCacheItem)?

Some ideas: using Stream wrapped around one of the aforementioned LOH
buffers, use ordinary .NET serialization to iterate through structures,
a bit like IDataReader or XmlReader etc. It seems to me that the better
performance you want to get, the more specific your solution becomes to
your problem set.

> I'm (also)
> wondering if memory fragmentation will start occurring in the native
> heap?

If you allocate memory manually or semi-manually (e.g. GC LOH recycled
buffers), you can implement it such that you don't need to worry about
fragmentation, given some constraints. For example, emulate generations
by having a list of stack-oriented allocators which trivially deallocate
by changing the top location. Alternatively, use circular buffers, etc.

-- Barry

Signature

http://barrkel.blogspot.com/

Reply to this Message

Strategy for caches & GC

Free Magazines