 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / CLR / August 2004Memory Leak Investigation
Okay guys, I've read 100 different threads on this subject, let's see if anyone has some good ideas for situation I'm in. Data: 1. My application fails with either an SEHException (during an unmanaged allocation) or an OutOfMemoryException (during a managed alloc) after 10-20 hours. 2. Monitoring with perfmon shows that the process "Private Bytes" increases throughout the life of the application while the "# of bytes in all heaps" remains low. 3. The modified, diagnostic version of my app is doing a System.GC.Collect() after each roughly 30-second operation. After each Collect(), I dump available memory info using the Win32 call GlobalMemoryStatus. This shows an average leak of 20-32k per operation, but the actual deltas are all over the map. The point-to-point graph shows alot of huge up and down changes in addition to some interesting downsloping stairstep behavior where it trends down, then recovers some memory, but doesnt recover enough to reverse the overall trend of losing memory. 4. SciTech's memory profiler (2.0), showed very little except a handful of objects which are still referenced after each 30s operation, and a bunch of temporaries. To the best of my knowledge, none of the managed objects it reported as "leaked" hold references to unmanaged objects. 5. Working off a hunch that the leak was unmanaged, I removed all allocations in our unmanaged code. I still call our unmanaged routines and iterate over the large data blocks being passed in, only to make sure those pages get pulled into my working set to simulate a real program execution. Under this situation, the memory behavior is essentially identical. 6. As measured by the Memory | Available MB counter, I'll lose 500+ MB over the course of roughly a 24 hour period. This outweighs the total size of the Bytes in all Heaps counter by a 100 to 1 ratio. 7. My app won't run with BoundsChecker enabled, still seeing if the guys at NuMega have any ideas. It stalls in random places without making any progress for hours; I don't believe its a consequence of crappy MT programming since I wrote most of that myself, and there's very few assumptions about speed, etc, and I've already accounted for those. Analysis: #1 indicates that a memory leak is occuring rather than the GC simply holding onto memory for future allocations. #2 indicates that the leak is occuring in unmanaged code rather than managed code (according to several other ng posts). #6 also indicates that it's an unmanaged leak. #5 confuses the hell out of me, because I am no longer making any unmanaged allocations. I am, however, interfacing with 3rd party OEM unmanaged DLLs, is that the best place to point at this time? How can I distinguish between memory leaked by my program vs. oem dlls. GRRRR, thanks, -ken [ Nasty Memory Leak] > Analysis: > #1 indicates that a memory leak is occuring rather than the GC simply > holding onto memory for future allocations. This is very plausable. I spent many days going through the Scitech profiler before I was able to track down some of my memory leaks. It took alot of time to really be able to tell what all the various data points in the profiler were. > #2 indicates that the leak is occuring in unmanaged code rather than > managed code (according to several other ng posts). Now this is interesting and may actually be two different things. Your unmanaged code could actually be leaking, or you may just be fragmenting the managed heap due to objects being pinned in memory. For an excellent description of the problem see: http://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx The solution that we ended up using was very similar to what Yun talks about in that blog. We create a large pool of objects (in our case, we allocate 10,000 instances of a 2K byte array) at application startup, and we then check these objects in and out of the pool over time. The pool grows in 10,000 object increments as needed. This eliminated the memory fragmentation that we were seeing by calling into unmanaged code. Something else to try - in terms of narrowing down if it's a manged or unmanged leak - when you look at the GC performance counters as your application runs, what's the breakdown by Generation? When managed code leaks, it shows itself by having the generation 2 heap growing without bounds over time. If you application should be more-or-less steady state in terms of memory consumption, this should help pin things down.  SignatureChris Mullins > > Analysis: > > #1 indicates that a memory leak is occuring rather than the GC simply [quoted text clipped - 4 lines] > time to really be able to tell what all the various data points in the > profiler were. Indeed. I'm still trying to understand what that tool is showing me. I was hoping for a leak in a 3rd party OEM dll, but rather than having a leak in blah.dll (OEM to remain nameless), I had an 80 MB difference in the OTHER category. > > #2 indicates that the leak is occuring in unmanaged code rather than > > managed code (according to several other ng posts). [quoted text clipped - 4 lines] > description of the problem see: > http://blogs.msdn.com/yunjin/archive/2004/01/27/63642.aspx Awesome, I love getting responses to my questions :) We're not doing any pinning, the data that is being passed from the managed world into the unmanaged world to be processed is essentially always unmanaged memory, it's in fact memory that even NT doesn't see. The managed world just holds a OEM-specific handle to memory that our imaging library understands, then we pass the raw pointer into UC++, this doesn't require a __pin. > The solution that we ended up using was very similar to what Yun talks about > in that blog. We create a large pool of objects (in our case, we allocate > 10,000 instances of a 2K byte array) at application startup, and we then > check these objects in and out of the pool over time. The pool grows in > 10,000 object increments as needed. This eliminated the memory fragmentation > that we were seeing by calling into unmanaged code. I thought it was UC++ fragmentation at first as well, but now that I'm not doing any allocations in UC++ I'm doubting that. > Something else to try - in terms of narrowing down if it's a manged or > unmanged leak - when you look at the GC performance counters as your [quoted text clipped - 4 lines] > steady state in terms of memory consumption, this should help pin things > down. I appear to have lost 40 MB in Gen #2 over a 12 hour period.. thats based on data from SciTech, but whats confusing is that perfmon shows # of bytes in all heaps remaining in the 5-10 MB range throughout. On a somewhat related note, Any links to GOOD explanations of all the perfmon counters? -ken "Ken Durden" <creepiecrawlies@hotmail.com> wrote in message > I appear to have lost 40 MB in Gen #2 over a 12 hour period.. thats > based on data from SciTech, but whats confusing is that perfmon shows > # of bytes in all heaps remaining in the 5-10 MB range throughout. In terms of finding usefull things with SciTech, it just took many hours of staring. I do remember the "age" column, in the instance section, was able to help me quite a bit. I found groups of related objects all of the same age, and that clued me in as to what was staying around. Unfortunatly there was no way to have the profiler sort by Age. Something that I thought about doing - the SciTech guys would probably accept some $$ to look at your heap dumps and tell you where the problems are. If not them, then DevelopMentor (with Richter, Robbins, et al) provides a debugging service that is fairly reasonably priced. > On a somewhat related note, > Any links to GOOD explanations of all the perfmon counters? Just the stuff in MSDN. The counter that I found most usefull was the "Gen 2 heap size" counter. It got to the point where that was the only counter I was even looking at. SignatureChris Mullins [Memory Leak stuff] Another good blog about how to track this stuff down is found here: http://blogs.msdn.com/mvstanton/archive/2004/04/05/108023.aspx SignatureChris Mullins
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.