Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / Performance / April 2008

Tip: Looking for answers? Try searching our database.

Server freezes for minutes.

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Sergey - 18 Mar 2008 13:23 GMT
I'm experiencing very weird problem.
Once in several days the server gets frozen for about 6 minutes!!.

I don't think this is deadlock and I'm almost confident that threads are
paused (by GC?),
because there is thread pool based timer which doesn't fire for
the same amount of time (Server uses about 4-5 threads so there are free
threads for timer to fire)

I'm running server on 32 bit .Net 2.0 SP1 on 4 processor computer with 4GB
ram and /3GB switch.
GC runs in server mode.

According to performance counters this freezing coincides with full garbage
collection.
I see no way to provide mininum code example because I have no clue
what the cause is and I have to wait many days to reproduce it.

For 30 seconds before the freezing I observe following values of performance
counters:
    Available memory on server starts diminishing with rate of several
MB/s. (In total 100MB in 30 secs)
    "Virtual bytes" for the server increases in total by 140 MB
    Gen 2, LOH size are almost unchanged
    Processors are loaded by 20%
    "CLR LocksAndThreads\QueueLength" is around 1

During this 6 minutes
   Available memory diminishes with rate of several MB/s ( 1 GB in total)
   "Virtual bytes" remains constant
   Gen 0, 1, 2 Heap size remains constant
   All managed threads of the server consume 0% of processor resources
   Garbage collector threads consume almost no processor resources
   All 4 processors are loaded at less then 4%
   "Memory pages/sec" is about ~20/sec (which is also at this rate when
everything works fine)
   "CLR LocksAndThreads\QueueLength" is 0
   "Bytes Allocated/sec" is 0

Right after this
    Available memory increases by 700MB
    Gen 2 collection counter increases by 1.
    Gen 2 heap size is diminished by 800 Mb
    LOH size remained the same
    % Time in GC is almost 100% with later decreasing to 8%
    All processors become loaded by ~50%
    All managed threads start doing something
    Garbage collector threads also start doing something
    "CLR LocksAndThreads\QueueLength" becomes something around 1

All these makes me suspect that GC pauses threads and waits for something
for 6 minutes
(usually full garbage collection takes several seconds).

What could make GC to wait for so long?
Another question I'm trying to figure out know is what eats unmanaged
memory? Could it be GC?
If it's not GC, can this cause GC to freeze?

I'd appreciate very much if somebody knows what the problem is or has any
ideas how I could remove or identify it.

TIA, Sergey
Robert - 20 Mar 2008 16:27 GMT
> I'm experiencing very weird problem.
> Once in several days the server gets frozen for about 6 minutes!!.
[quoted text clipped - 4 lines]
> the same amount of time (Server uses about 4-5 threads so there are free
> threads for timer to fire)

How many threads in the threadpool?  When it is maxed out, timers do not fire..

> I'm running server on 32 bit .Net 2.0 SP1 on 4 processor computer with 4GB
> ram and /3GB switch.
> GC runs in server mode.

Server mode GC - good!

> According to performance counters this freezing coincides with full garbage
> collection.
> I see no way to provide mininum code example because I have no clue
> what the cause is and I have to wait many days to reproduce it.

What are you GC'ing?  ie what is in your Gen 2 heap?
File handles or anything with an underlying hard resource?
You need to run a profiler on the app and see what is accumulating.
Make sure you are using IDisposable on anything that has any significant
weight.  Implement IDisposable on anything questionable.

Make your own performance counters for heavier objects, so you can
monitor them on your server.

> For 30 seconds before the freezing I observe following values of performance
> counters:
[quoted text clipped - 4 lines]
>     Processors are loaded by 20%
>     "CLR LocksAndThreads\QueueLength" is around 1

Since actual mem is increasing, yet virtual is flat, it sounds like you
may be doing some paging bringing virtual into real memory.
Watch your disk queue lengths, and keep them low.

> During this 6 minutes
>    Available memory diminishes with rate of several MB/s ( 1 GB in total)
[quoted text clipped - 7 lines]
>    "CLR LocksAndThreads\QueueLength" is 0
>    "Bytes Allocated/sec" is 0

Low processor usage, yet slow.  Sounds like the bottleneck is elsewhere.
Check the disks..  Maybe move the swap file to a different disk, or array.

> Right after this
>     Available memory increases by 700MB
[quoted text clipped - 14 lines]
> memory? Could it be GC?
> If it's not GC, can this cause GC to freeze?

You may want to GC manually in your code, maybe after however many work items
complete in an hour.  Personally, I would rather have a few seconds of GC every hour
than 30 secs - 6 minutes every few days.

What does the app do?  What is the average object lifetime.  How many transactions
per second?

Good luck.
Ben Voigt [C++ MVP] - 02 Apr 2008 21:02 GMT
> You may want to GC manually in your code, maybe after however many
> work items complete in an hour.  Personally, I would rather have a
> few seconds of GC every hour than 30 secs - 6 minutes every few days.

Sadly, the .NET GC can't be amortized like this.  Cost of a GC is
proportional to the number of live objects, dead objects don't matter.

You need to find out why transient objects are surviving into Gen2, Gen2
collections are very slow.

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.