I'm experiencing very weird problem.
Once in several days the server gets frozen for about 6 minutes!!.
I don't think this is deadlock and I'm almost confident that threads are
paused (by GC?),
because there is thread pool based timer which doesn't fire for
the same amount of time (Server uses about 4-5 threads so there are free
threads for timer to fire)
I'm running server on 32 bit .Net 2.0 SP1 on 4 processor computer with 4GB
ram and /3GB switch.
GC runs in server mode.
According to performance counters this freezing coincides with full garbage
collection.
I see no way to provide mininum code example because I have no clue
what the cause is and I have to wait many days to reproduce it.
For 30 seconds before the freezing I observe following values of performance
counters:
Available memory on server starts diminishing with rate of several
MB/s. (In total 100MB in 30 secs)
"Virtual bytes" for the server increases in total by 140 MB
Gen 2, LOH size are almost unchanged
Processors are loaded by 20%
"CLR LocksAndThreads\QueueLength" is around 1
During this 6 minutes
Available memory diminishes with rate of several MB/s ( 1 GB in total)
"Virtual bytes" remains constant
Gen 0, 1, 2 Heap size remains constant
All managed threads of the server consume 0% of processor resources
Garbage collector threads consume almost no processor resources
All 4 processors are loaded at less then 4%
"Memory pages/sec" is about ~20/sec (which is also at this rate when
everything works fine)
"CLR LocksAndThreads\QueueLength" is 0
"Bytes Allocated/sec" is 0
Right after this
Available memory increases by 700MB
Gen 2 collection counter increases by 1.
Gen 2 heap size is diminished by 800 Mb
LOH size remained the same
% Time in GC is almost 100% with later decreasing to 8%
All processors become loaded by ~50%
All managed threads start doing something
Garbage collector threads also start doing something
"CLR LocksAndThreads\QueueLength" becomes something around 1
All these makes me suspect that GC pauses threads and waits for something
for 6 minutes
(usually full garbage collection takes several seconds).
What could make GC to wait for so long?
Another question I'm trying to figure out know is what eats unmanaged
memory? Could it be GC?
If it's not GC, can this cause GC to freeze?
I'd appreciate very much if somebody knows what the problem is or has any
ideas how I could remove or identify it.
TIA, Sergey
> I'm experiencing very weird problem.
> Once in several days the server gets frozen for about 6 minutes!!.
[quoted text clipped - 4 lines]
> the same amount of time (Server uses about 4-5 threads so there are free
> threads for timer to fire)
How many threads in the threadpool? When it is maxed out, timers do not fire..
> I'm running server on 32 bit .Net 2.0 SP1 on 4 processor computer with 4GB
> ram and /3GB switch.
> GC runs in server mode.
Server mode GC - good!
> According to performance counters this freezing coincides with full garbage
> collection.
> I see no way to provide mininum code example because I have no clue
> what the cause is and I have to wait many days to reproduce it.
What are you GC'ing? ie what is in your Gen 2 heap?
File handles or anything with an underlying hard resource?
You need to run a profiler on the app and see what is accumulating.
Make sure you are using IDisposable on anything that has any significant
weight. Implement IDisposable on anything questionable.
Make your own performance counters for heavier objects, so you can
monitor them on your server.
> For 30 seconds before the freezing I observe following values of performance
> counters:
[quoted text clipped - 4 lines]
> Processors are loaded by 20%
> "CLR LocksAndThreads\QueueLength" is around 1
Since actual mem is increasing, yet virtual is flat, it sounds like you
may be doing some paging bringing virtual into real memory.
Watch your disk queue lengths, and keep them low.
> During this 6 minutes
> Available memory diminishes with rate of several MB/s ( 1 GB in total)
[quoted text clipped - 7 lines]
> "CLR LocksAndThreads\QueueLength" is 0
> "Bytes Allocated/sec" is 0
Low processor usage, yet slow. Sounds like the bottleneck is elsewhere.
Check the disks.. Maybe move the swap file to a different disk, or array.
> Right after this
> Available memory increases by 700MB
[quoted text clipped - 14 lines]
> memory? Could it be GC?
> If it's not GC, can this cause GC to freeze?
You may want to GC manually in your code, maybe after however many work items
complete in an hour. Personally, I would rather have a few seconds of GC every hour
than 30 secs - 6 minutes every few days.
What does the app do? What is the average object lifetime. How many transactions
per second?
Good luck.
Ben Voigt [C++ MVP] - 02 Apr 2008 21:02 GMT
> You may want to GC manually in your code, maybe after however many
> work items complete in an hour. Personally, I would rather have a
> few seconds of GC every hour than 30 secs - 6 minutes every few days.
Sadly, the .NET GC can't be amortized like this. Cost of a GC is
proportional to the number of live objects, dead objects don't matter.
You need to find out why transient objects are surviving into Gen2, Gen2
collections are very slow.