 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / CLR / July 2006Diagnosing root cause of .NET application using nearly 500 MB memory
We have a .NET application which is obviously using nearly 500MB memory and we can't understand why. I created a dump for a different issue and the server ended up creating a process shutdown dump of this size. I have seen the DumpLargeObjectSegments command however I'm not sure what to look for next i.e. the output shows 0:000> !sos.dl Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll" Going to dump the Large Object Segments. Large object heap starts at 0x02071000 segment begin allocated size 02070000 02071000 02fbbf98 0x00f4af98(16,035,736) 0bd80000 0bd81000 0cd6f7c8 0x00fee7c8(16,705,480) 294b0000 294b1000 2a414908 0x00f63908(16,136,456) 1d010000 1d011000 1df65518 0x00f54518(16,074,008) 1e010000 1e011000 1ef99320 0x00f88320(16,286,496) 1f010000 1f011000 1ffd67a0 0x00fc57a0(16,537,504) 21010000 21011000 21fb4d48 0x00fa3d48(16,399,688) 23010000 23011000 23fe4ed0 0x00fd3ed0(16,596,688) 25010000 25011000 25fe7440 0x00fd6440(16,606,272) 274b0000 274b1000 284a4b18 0x00ff3b18(16,726,808) 2c4b0000 2c4b1000 2d4ac560 0x00ffb560(16,758,112) 2e4b0000 2e4b1000 2f47ddd0 0x00fccdd0(16,567,760) 2f4b0000 2f4b1000 304a3970 0x00ff2970(16,722,288) 304b0000 304b1000 314ad208 0x00ffc208(16,761,352) 08ce0000 08ce1000 09cd6d18 0x00ff5d18(16,735,512) 09ce0000 09ce1000 0accb370 0x00fea370(16,687,984) 0ace0000 0ace1000 0bc63c70 0x00f82c70(16,264,304) 0cd80000 0cd81000 0dd7f908 0x00ffe908(16,771,336) 0dd80000 0dd81000 0eba3a50 0x00e22a50(14,821,968) 0:000> !dumpheap -stat total 141,545 objects Statistics: which produced a ton of objects in order of total size but the bottom part is most useful I suspect i.e. 3379620 32,539 390,468 Tranmit.Sprinter.DataAccess.SQLDataAccess.SQLStaticDataStore 3379c3c 32,537 780,888 Tranmit.Sprinter.XMLBatchRecord 337bfcc 32,537 1,561,776 Tranmit.Sprinter.DataAccess.SQLDataAccess.SQLXMLBatchRecordData 95236c 127 7,808,984 System.Char[] 1501c8 801 76,190,256 Free 9526b0 438 116,192,444 System.Byte[] 79b94638 36,369 251,137,772 System.String Total 141,545 objects Hi, The next step to be is to find where all of those strings are rooted. Run a command like: !dumpheap -stat -mt <address of string method table> -min <whatever number. I usally use 85000 so I can find strings on the Large Object Heap> If the above command returns actual string objects than you can !gcroot their address to see where they are getting created. Also compare the size of all heaps from !eeheap -gc vs the size of private bytes perf counter(or the size of the dump) If there is a large discrepancy this would indicate that the majority of committed memory is from native code and not managed code. HTH, Dan This posting is provided "AS IS" with no warranties, and confers no rights. You assume all risk for your use. © 2002 Microsoft Corporation. All rights reserved. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm Please reply to the newsgroups only. Thanks however few questions How do I get the object addresses from the output below to call gcroot as MT is not an object address 0:000> !dumpheap -stat -mt 0x79b94638 -min 85000 total 385 objects Statistics: MT Count TotalSize Class Name 79b94638 385 112,012,752 System.String Total 385 objects The output of !eeheap -gc is shown below 0:000> !eeheap -gc Number of GC Heaps: 1 generation 0 starts at 0x07d02ae8 generation 1 starts at 0x07ce1000 generation 2 starts at 0x01071000 ephemeral segment allocation context: none segment begin allocated size 01070000 01071000 02021864 0x00fb0864(16,451,684) 2b4b0000 2b4b1000 2c315de0 0x00e64de0(15,093,216) 14010000 14011000 1500f768 0x00ffe768(16,770,920) 20010000 20011000 21003090 0x00ff2090(16,720,016) 22010000 22011000 22fef8fc 0x00fde8fc(16,640,252) 2a4b0000 2a4b1000 2b2e6a48 0x00e35a48(14,899,784) 2d4b0000 2d4b1000 2e4a30d4 0x00ff20d4(16,720,084) 059b0000 059b1000 06746dbc 0x00d95dbc(14,245,308) 06ce0000 06ce1000 07acc604 0x00deb604(14,595,588) 07ce0000 07ce1000 07d1eaf4 0003daf4(252,660) Large object heap starts at 0x02071000 segment begin allocated size 02070000 02071000 02fbbf98 0x00f4af98(16,035,736) 0bd80000 0bd81000 0cd6f7c8 0x00fee7c8(16,705,480) 294b0000 294b1000 2a414908 0x00f63908(16,136,456) 1d010000 1d011000 1df65518 0x00f54518(16,074,008) 1e010000 1e011000 1ef99320 0x00f88320(16,286,496) 1f010000 1f011000 1ffd67a0 0x00fc57a0(16,537,504) 21010000 21011000 21fb4d48 0x00fa3d48(16,399,688) 23010000 23011000 23fe4ed0 0x00fd3ed0(16,596,688) 25010000 25011000 25fe7440 0x00fd6440(16,606,272) 274b0000 274b1000 284a4b18 0x00ff3b18(16,726,808) 2c4b0000 2c4b1000 2d4ac560 0x00ffb560(16,758,112) 2e4b0000 2e4b1000 2f47ddd0 0x00fccdd0(16,567,760) 2f4b0000 2f4b1000 304a3970 0x00ff2970(16,722,288) 304b0000 304b1000 314ad208 0x00ffc208(16,761,352) 08ce0000 08ce1000 09cd6d18 0x00ff5d18(16,735,512) 09ce0000 09ce1000 0accb370 0x00fea370(16,687,984) 0ace0000 0ace1000 0bc63c70 0x00f82c70(16,264,304) 0cd80000 0cd81000 0dd7f908 0x00ffe908(16,771,336) 0dd80000 0dd81000 0eba3a50 0x00e22a50(14,821,968) Total Size 0x1b186bb0(454,585,264) ------------------------------ GC Heap Size 0x1b186bb0(454,585,264) > Hi, > [quoted text clipped - 22 lines] > http://www.microsoft.com/info/cpyright.htm > Please reply to the newsgroups only. > Thanks however few questions > [quoted text clipped - 15 lines] > generation 1 starts at 0x07ce1000 > generation 2 starts at 0x01071000 Not really knowing much about this, that gen 2 heap looks huge. So it's definitely managed memory being leaked. > ephemeral segment allocation context: none > segment begin allocated size [quoted text clipped - 61 lines] >> http://www.microsoft.com/info/cpyright.htm >> Please reply to the newsgroups only. Thanks Ben however does anyone know how I can pin point further where all this memory is being allocated and why it isn't being freed by the garbage collector 500MB is a lot of memory hence why it's so concerning for us. >> Thanks however few questions >> [quoted text clipped - 84 lines] >>> http://www.microsoft.com/info/cpyright.htm >>> Please reply to the newsgroups only. Hi, Sorry take the -stat out of the command. !dumpheap -mt 0x79b94638 -min 85000 This will filter out every string object that is not on the Large Object Heap. It will also print each string object that is on the LOH with the address of each string. You can then !gcroot <string address> to find out where it is rooted. If you find a lot of these strings rooted in the same area of code you can drill down on that code and see if you can find a way for the strings to go out of scope earlier. HTH, Dan This posting is provided "AS IS" with no warranties, and confers no rights. You assume all risk for your use. © 2002 Microsoft Corporation. All rights reserved. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm Please reply to the newsgroups only. Thanks however I'm issues with gcroot now i.e. small sample of output from dumpheap 0:000> !dumpheap -mt 0x79b94638 -min 85000 Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll" Address MT Size Gen 0x02262f10 0x79b94638 2,000,632 3 System.String <STE_Invoice><ProcessInstructi 0x02462550 0x79b94638 93,992 3 System.String <STE_Invoice><ProcessInstructi 0x024a4e08 0x79b94638 178,552 3 System.String <STE_Invoice><ProcessInstructi 0x024fba28 0x79b94638 176,776 3 System.String <STE_Invoice><ProcessInstructi 0:000> !gcroot 0x02262f10 Scan Thread 0 (0x15c4) 0:000> I thought gcroot would have output something here? > Hi, > [quoted text clipped - 20 lines] > http://www.microsoft.com/info/cpyright.htm > Please reply to the newsgroups only. > 0:000> !dumpheap -mt 0x79b94638 -min 85000 > Loaded Son of Strike data table version 5 from [quoted text clipped - 14 lines] > > I thought gcroot would have output something here? Check what the finalizer thread is doing (!threads, etc.). If it is e.g. blocked, some objects could not be collected even if they don't have roots. Regards, Oleg [VC++ MVP http://www.debuginfo.com/] Thanks Oleg however still can't see anything unusual. 0:000> !Threads ThreadCount: 4 UnstartedThread: 0 BackgroundThread: 2 PendingThread: 0 DeadThread: 1 PreEmptive GC Alloc Lock ID ThreadOBJ State GC Context Domain Count APT Exception 0 15c4 00150010 6020 Enabled 07d1bc40:07d1cae8 0014a260 1 STA (GC) XXX 131c 00158930 b220 Enabled 07d1cc98:07d1eae8 0014a260 0 Ukn (Finalizer) XXX 0 001da1d0 1800820 Enabled 00000000:00000000 0014a260 0 Ukn (Threadpool Worker) XXX 77c 0021cff0 1800220 Enabled 00000000:00000000 0014a260 0 Ukn (Threadpool Worker) 0:000> ~*e !clrstack Thread 0 ESP EIP 0012f660 7c82ed54 [FRAME: ECallMethodFrame] [DEFAULT] Void System.Environment.ExitNative(I4) 0012f670 79a15a86 [DEFAULT] Void System.Environment.Exit(I4) 0012f674 00cb02c3 [DEFAULT] Void BatchManager.BatchManager.Main(SZArray String) 0012f8b0 7923c069 [FRAME: GCFrame] 0012f9b0 7923c069 [FRAME: GCFrame] 0012fa94 7923c069 [FRAME: GCFrame] >> 0:000> !dumpheap -mt 0x79b94638 -min 85000 >> Loaded Son of Strike data table version 5 from [quoted text clipped - 22 lines] > Oleg > [VC++ MVP http://www.debuginfo.com/] > 0:000> ~*e !clrstack > Thread 0 [quoted text clipped - 5 lines] > 0012f9b0 7923c069 [FRAME: GCFrame] > 0012fa94 7923c069 [FRAME: GCFrame] It's too late to analyze threads and roots when the application is exiting. Can you repeat the same actions on a live application that exhibits the problem? Oleg Thanks Oleg, It's going to be difficult as this was on clients production server. Is there really nothing else we can get out of the dump. Why is it too late to analyse threads when the app is exiting? If the memory is still lying around, sure we must be able to find out where it's referenced from and the roots, it seems strange not to be able to get this information . >> 0:000> ~*e !clrstack >> Thread 0 [quoted text clipped - 14 lines] > > Oleg > It's going to be difficult as this was on clients production server. Is > there really nothing else we can get out of the dump. [quoted text clipped - 3 lines] > referenced from and the roots, it seems strange not to be able to get this > information Threads have already been terminated at that moment, memory could also be changed during shutdown/cleanup. I suspect that the finalizer thread could be blocked and thus prevent the objects from collection. But since the thread is already gone, I don't think it is possible to recover the state it was in while the application was running. ('dds' over the remains of its stack memory could be tried, but I am not sure that it could help much in this case) But I still would not say that it is impossible to recover any more information, may be I just don't know how to do it. What will be shown if you run 'dd 0x043900e0 L20' command? Oleg Hi, It could mean they are not rooted, however since objects on the LOH don't get collected until a full GC occurs it's possible that a full GC hasn't happened yet and that is why they are still around. The idea of using clrprofiler that another person had is also a good idea to find where things are rooted. Also I saw someone mention the finalize thread. To get that output you could run !finalizequeue BTW what type of application is this? Service, winform? How much RAM do you have and how many processors? Thanks, Dan This posting is provided "AS IS" with no warranties, and confers no rights. You assume all risk for your use. © 2002 Microsoft Corporation. All rights reserved. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm Please reply to the newsgroups only. I could use CLR profiler however this is a dump which has been taken on a clients production server whilst reviewing another issue. It would obviously take time to install the profiler on the server e.t.c. and may impact the performance of the running app e.t.c. I will certainly go down that route if I have to but I'd like to get as much out of this dump as possible 0:000> !finalizequeue Loaded Son of Strike data table version 5 from "C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\mscorwks.dll" SyncBlock to be cleaned up: 0 ---------------------------------- MTA interfaces to be released: 0 Total STA interfaces to be released: 0 ---------------------------------- generation 0 has 0 finalizable objects (0x043900e0->0x043900e0) generation 1 has 0 finalizable objects (0x043900e0->0x043900e0) generation 2 has 0 finalizable objects (0x043900e0->0x043900e0) Ready for finalization 0 objects (0x043900e0->0x043900e0) - Freachable queue All Finalizable Objects Statistics: MT Count TotalSize Class Name 0x79bba72c 1 60 System.Threading.Thread Total 1 objects, Total size: 60 I'm still no nearer to working out why all this memory is still being allocated, any other pointers? > Hi, > [quoted text clipped - 20 lines] > http://www.microsoft.com/info/cpyright.htm > Please reply to the newsgroups only. Hi, The finalizequeue looks good. Try running !traverseheap to output a log file that you can feed to clrprofiler. Also you never answered my question as to what type of application you are using and how much RAM and how many CPU's the machine has. Is your app using workstation or server version of the GC, and if it's workstation is it using concurrent or nonconcurrent GC? Thanks, Dan This posting is provided "AS IS" with no warranties, and confers no rights. You assume all risk for your use. © 2002 Microsoft Corporation. All rights reserved. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm Please reply to the newsgroups only. Hi Dan, I've input the log to clr profiler and can't find any easy way to see why objects are still on the heap and haven't been garbage collected. Are there any goods resources you know of on this? The application is a console app which imports XML files. The machine has about 2 GB ram, 1 CPU. It should be using the default version of the GC i.e. workstation I presume and I presume the concurrent GC is also the default. > Hi, > [quoted text clipped - 16 lines] > http://www.microsoft.com/info/cpyright.htm > Please reply to the newsgroups only. Imports XML files? Are you doing any string concatenation? That can be a big source of memory allocation going out of control. Also, make sure you're Disposing() of any objects that implement IDisposable once you're done with them. They might be hanging onto unmanaged memory that won't get released until they are finalized (tho another post of yours mentioned a lack of objects on the finalize and freachable queues) Also, just wondering... 500 megs is concerning, but is it actually a problem? Have you had the program fail due to a lack of memory? If you start manipulating large amounts of data in memory your heap size is going to expand, but it won't immediately contract when that memory is released. The CLR's memory management model is geared more for performance rather than small memory footprint. > Hi Dan, > [quoted text clipped - 26 lines] > > http://www.microsoft.com/info/cpyright.htm > > Please reply to the newsgroups only. Hi William No, there are no specific objects here implementing IDisposable. 500 megs hasn't caused an issue however I would not have expected so much memory to be left claimed through the processing of the app. Clearly it never needs anywhere this amount of memory any of the time. Cheers Matt > Imports XML files? Are you doing any string concatenation? That can be a > big source of memory allocation going out of control. Also, make sure [quoted text clipped - 49 lines] >> > http://www.microsoft.com/info/cpyright.htm >> > Please reply to the newsgroups only. > Hi William > [quoted text clipped - 3 lines] > memory to be left claimed through the processing of the app. Clearly it > never needs anywhere this amount of memory any of the time. ... Hi, recently I had similar problem with memory leak and I traced it to calling COM component from service that run under STA thread. objects were not rooted but still they were not garbage collected. Changing thread to MTA fixed the problem. May not apply to you, but you never know... Check: http://dotnetdebug.blogspot.com/2005/06/blocked-finalizer-thread.html Regards, Goran > Hi William > [quoted text clipped - 3 lines] > memory to be left claimed through the processing of the app. Clearly it > never needs anywhere this amount of memory any of the time. ... Ah, forgot to write in my previous post: I used memprofiler from scitech few times and it was great help. Check it, there is free 14 day trial. You can create heap snapshots at different times and compere difference. If you have a leak, you'll find it fasts (allocation stack is also shown). http://memprofiler.com/download.aspx Regards, Goran [Diagnosing a memory leak] > I used memprofiler from scitech few times and it was great help. Check it, > there is free 14 day trial. You can create heap snapshots at different > times and compere difference. If you have a leak, you'll find it fasts > (allocation stack is also shown). I've used the SciTech Memory profiler quite a few times to diagnose memory leaks. It's been one of the overall best .Net tools in my toolbox. The products that I work on day-to-day (the SoapBox Framework, SoapBox Server) would never have achieved the stability they have without using the SciTech profiler. Although it takes a few hours to learn how to read and follow the output, once you get the hang of this, tracking down memory leaks becomes a mechanical, repeatable process rather than the trial-and-error guesswork it's been in the past. I recommend their tool without reservation - and at $100 for a license, it's a real steal. -- Chris Mullins http://www.coversant.net/blogs/cmullins Try the CLRProfiler, but do not try to run it with 500MB -- use the same problematic program but stop much sooner, perhaps at less than 100MB. Otherwise you will wait forever. http://www.microsoft.com/downloads/details.aspx?familyid=a362781c-3870-43be-8926 -862b40aa0cd0 Regards, Frank Hileman check out VG.net: http://www.vgdotnet.com Animated vector graphics system Integrated Visual Studio .NET graphics editor > Thanks Ben however does anyone know how I can pin point further where all > this memory is being allocated and why it isn't being freed by the garbage > collector > > 500MB is a lot of memory hence why it's so concerning for us.
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.