 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / CLR / May 2006Major memory leak in sos.dll under .NET Framework 2.0
I have tried to use sos.dll together with the WinDbg debug engine to perform some automated retrieval of runtime data. Unfortunately, there seems to be a major memory leak in the version of sos.dll that is shipped with .NET Framework 2.0. Each time an sos command is executed, e.g !DumpDomain, !DumpModule or !DumpMT, the memory consumption of WinDbg increases by about 5 MB. This memory is never released, and since I want to execute a large amount of commands (> 500) the process quickly runs out of memory. This error only occurs in the v2.0 sos.dll. The sos.dll provided with the Debugging tools (for v1.x) seems to be working correctly. Is there any possibility that this problem will be fixed soon? E.g will an updated sos.dll be included with the debugging tools download? Hi ASuurkuusk, Thanks for your post. I have tried to use "Microsoft .NET Framework SDK v2.0" command prompt to lauch the windbg and load SOS.dll. However, in windbg, !DumpModule or !DumpMT does not cause any significant increase in "Task Manager" "VM size" value. Have you tried other VS2005 machine's SOS.dll? If this problem is reproducable on other machine, I suggest you provide the detailed steps to help us reproduce the problem. Thanks Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Hi, I have tried this on two machines, both with Windows XP SP2 as well as under a VMWare session. One machine was running a Swedish OS and VS 2005 Professional and one was running an English OS and VS 2005 Team Developer. I have also tried with all downloadable versions of WinDbg (v6.6.3.5, v6.5.3.8, v6.4.7.2, v6.3.17.0), except v6.2.13.1. The behaviour is the same. To reproduce this behaviour start WinDbg and then start or attach (invasive or non-invasive) a .NET executable (e.g an empty Windows Forms application). Then load sos.dll and issue some commands. After each command the memory usage of WinDbg increases by about 5MB. To show you this behaviour I recorded a movie from a VMWare session. This session runs a freshly installed Windows XP SP2 OS, with .NET Framework 2.0 and Debugging Tools for Windows v6.6.3.5 installed. The .NET application used is just an empty Windows Forms application created using VS2005. The Task manager in the movie shows the memory usage of WinDbg, as well as the total Commit charge. The movie can be downloaded from: http://www.scitech.se/temp/SosLeak2.avi Best regards, Andreas Suurkuusk SciTech Software AB > Hi ASuurkuusk, > [quoted text clipped - 17 lines] > ================================================== > This posting is provided "AS IS" with no warranties, and confers no rights. Hi Andreas and Willy, Thanks for your feedback. Yes, I can reproduce out this behavior, I will feedback this issue to the product team, I will reply any progress here. Thanks Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Hi Andreas , Sorry for letting you wait. Based on the discuss with the product team, they confirmed that the 2.0 SOS just makes calls into the framework to get the data and the framework gets loaded in windbg process. Also, some of the commands cache data so that subsequent lookups don't take as long. So this is not identified as a leak. Hope this helps! Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Hi, I did perform some debugging into sos.dll before reporting this issue, and I am aware that the runtime dll and mscordacwks.dll get loaded into the windbg process and that some data gets cached. However, this is not the problem. The problem is that the cache does not get reused between sos commands. If you looked at the movie I provided (http://www.scitech.se/temp/sosleak2.avi), you would have seen that I issued a DumpMT command for the same method table (7b473dfc) several times. Each time the memory usage of windbg.exe increased with about 6 MB. If the data about the provided method table was cached, then the memory usage should certainly not increase. In order to provide you with some more information about this problem, I performed some additional debugging of sos.dll and dbgeng.dll. I managed to pinpoint the problem to the LoadClrDebugDll function in sos.dll. The LoadClrDebugDll gets called each time an sos command is performed. It makes a call to dbeng.dll!ExtIoctl (using ExtensionsApis.lpIoctlRoutine) with IoctlType set to 0x26, which will create a new "CLRData" instance using CLRDataCreateInstance. This new instance is then assigned to g_clrData, overwriting the previous instance of "CLRData", which I believe causes the memory leak. I performed a test where I modified the LoadClrDebugDll function so that it only runs once (on subsequent calls it returns immediately). This avoids creating and leaking "CLRData" instances and it seems like sos.dll works correctly, without leaking any memory. At the end of this post I have provided my (partially and manually) decompiled code for LoadClrDebugDll and ExtIoctl. I have marked the sections where I believe there is a problem with "PROBLEM:". I don't know if it will cause any side effects, but adding "if( g_clrData != NULL ) return;" to LoadClrDebugDll, will probably avoid the memory leak. Another thing that I have noticed, is that this memory leak doesn't occur when sos.dll is loaded into VS 2005. I have not investigated this further, but since VS doesn't use dbgeng.dll, I assume that ExtIoctl (0x26) in the VS implementation will return the same "CLRData" instance each time it's requested. Best regards, Andreas Suurkuusk SciTech Software AB struct CLRDebugData { IID* iid; void** iface; } IID IID_IClrData = 5c552ab6_fc09_4cb3_8e36_22fa03c798b7 SOS.dll!LoadClrDebugDll() { CLRDebugData data; data.clsId = &IID_IClrData; if( ExtensionsApis.lpIoctlRoutine( 0x26, &data, sizeof( CLRDebugData ) ) != 0 ) { // PROBLEM: Will overwrite previous g_ClrTarget (memory leak ~ 5MB!) g_clrData = data.iface; } } ULONG dbgeng.dll!ExtIoctl(USHORT IoctlType, PVOID lpvData, ULONG cbSize) { switch( IoctlType ) { case 0x26: if( g_Process != NULL && cbSize == sizeof( CLRDebugData ) ) { CLRDebugData* data = (ClrDebugData*)lpvData; g_Process->LoadClrDebugDllForExt(); if( g_Process->pCLRDataCreateInstance != NULL ) { // PROBLEM: Will create a new IClrDataTargetInstance each time if( g_Process->pCLRDataCreateInstance( *data->iid, g_Process->target, data->iface ) == S_OK ) return (ULONG)TRUE; } } break; case ...: break; } ... } > Hi Andreas , > [quoted text clipped - 17 lines] > ================================================== > This posting is provided "AS IS" with no warranties, and confers no rights. Andreas, Great analysis, which I'm happy to confirm. Willy. | Hi, | [quoted text clipped - 104 lines] | > ================================================== | > This posting is provided "AS IS" with no warranties, and confers no rights. Hi Andreas, Very cool! I have forward your debugging information to the sos.dll owner, I will feedback any progress here. Personally, I am curious about some of your debugging details. 1. Do you get the source code of the functions in dbgeng.dll by disasssembling with symbols in windbg? 2. If you can do reverse engineering to the function, how do you "adding "if( g_clrData != NULL ) return;" to LoadClrDebugDll"? Do you modify the executing process in windbg in debugging-time? Thanks Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Hi, 1. Actually I prefer the VS 2005 debugger and only use windbg when I have to. Anyway, the source code was manually decompiled using disassembly, together with symbol files for dbgeng.dll and sos.dll. The symbol files provided me with some function and variables names. The IID_IClrData and CLRDebugData names I made up my self. 2.I have written my own wrapper around dbgeng.dll, and the modification I mentioned was performed by replacing the first instructions of LoadClrDebugDll with "xor eax, eax" and "ret". The modification was performed after having called an sos command once, so I didn't actually have to add "if( g_clrData != NULL )". In my first attempts I replaced the instructions manually within the debugger, but then I added the code below to my dbgeng wrapper: pDebugControl->Execute( 0, cmd, 0 ); if( cmd[0] == '!' && firstCommand ) { byte* pLoadClrDebugDll = (byte*)0x64280220; // Hard coded start address of LoadClrDebugDll DWORD oldProtect; VirtualProtect( pLoadClrDebugDll, 3, PAGE_EXECUTE_READWRITE, &oldProtect ); pLoadClrDebugDll[0] = 0x33; // xor eax, eax pLoadClrDebugDll[1] = 0xc0; // pLoadClrDebugDll[2] = 0xc3; // ret VirtualProtect( pLoadClrDebugDll, 3, oldProtect, NULL ); FlushInstructionCache( GetCurrentProcess(), pLoadClrDebugDll, 3 ); firstCommand = false; } Best regards, Andreas Suurkuusk SciTech Software AB > Hi Andreas, > [quoted text clipped - 21 lines] > ================================================== > This posting is provided "AS IS" with no warranties, and confers no rights. Hi Andreas, Thanks for your information. We have tried to contact the product team for this issue from a formal process, I will feedback any progress here. Thanks Note: it seems that I can not play the video from the link below, I hope you can give it a re-check http://www.scitech.se/temp/sosleak2.avi Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Hi Andreas, I tried the steps below, however, it seems that I can not reproduce out the memory leak behavior: 1. open .Net Framework2.0 SDK command prompt 2. paste the windbg exe path and lauch the windbg 3. open a .Net 2.0 exe winform application 4. use .load sos to load sos dll 5. type !dumpdomain, this will cause a significant VM Size increase in Task Manager 6. try typing !dumpdomain more times, the VM Size in Task Manager does not increase If !dumpdomain command leaks memory, further !dumpdomain command should cause VM Size increase. Current, I suggest you try to update your Windbg to the lastest version, and does this problem still exist? Thanks Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Did as you asked, the results are exactly the same, a ~4MB VM increase for each !dumpdomain command using following OS, WindDbg and SOS version: Windows XP Version 2600 (Service Pack 2) MP (2 procs) Free x86 compatible Product: WinNt, suite: SingleUserTS kernel32.dll version: 5.1.2600.2180 (xpsp_sp2_rtm.040803-2158) Debug session time: Mon May 1 10:44:04.496 2006 (GMT+2) System Uptime: 0 days 10:43:42.099 Process Uptime: 0 days 0:04:14.088 Kernel time: 0 days 0:00:00.203 User time: 0 days 0:00:00.062 Live user mode: <Local> command line: 'windbg' Debugger Process 0xFE4 dbgeng: image 6.6.0003.5, built Thu Jan 19 00:06:52 2006 [path: C:\Program Files\Debugging Tools for Windows\dbgeng.dll] dbghelp: image 6.6.0003.5, built Thu Jan 19 00:06:37 2006 [path: C:\Program Files\Debugging Tools for Windows\dbghelp.dll] DIA version: 51108 Extension DLL search Path: C:\Program Files\Debugging Tools for Windows\winext;C:\Program Files\Debugging Tools for Windows\winext\arcade;C:\Program Files\Debugging Tools for Windows\WINXP;C:\Program Files\Debugging Tools for Windows\pri;C:\Program Files\Debugging Tools for Windows;C:\Program Files\Debugging Tools for Windows\winext\arcade;c:\Program Files\Microsoft Visual Studio 8\Common7\IDE\;c:\Program Files\Microsoft Visual Studio 8\VC\BIN;c:\Program Files\Microsoft Visual Studio 8\Common7\Tools;c:\Program Files\Microsoft Visual Studio 8\Common7\Tools\bin;c:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools;c:\Program Files\Microsoft Visual Studio 8\VC\PlatformSDK\bin;c:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\bin;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;c:\Program Files\Microsoft Visual Studio 8\VC\VCPackages;C:\Program Files\Debugging Tools for Windows;c:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools;"C:\Program Files\Microsoft DirectX SDK (February 2006)\Utilities\Bin\x86";C:\Program Files\Windows Resource Kits\Tools\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Support Tools\;C:\Program Files\Microsoft SQL Server\80\Tools\Binn\;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files\Microsoft SQL Server\90\DTS\Binn\;C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files\Microsoft Driver Test Manager\Controller\;C:\Program Files\AMD\CodeAnalyst\bin;C:\Program Files\Windows PowerShell\v1.0\ Extension DLL chain: sos: image 2.0.50727.42, API 1.0.0, built Fri Sep 23 09:27:26 2005 [path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll] C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll: image 2.0.50727.42, API 1.0.0, built Fri Sep 23 09:27:26 2005 [path: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\sos.dll] dbghelp: image 6.6.0003.5, API 6.0.6, built Thu Jan 19 00:06:37 2006 [path: C:\Program Files\Debugging Tools for Windows\dbghelp.dll] ext: image 6.6.0003.5, API 1.0.0, built Thu Jan 19 00:06:08 2006 [path: C:\Program Files\Debugging Tools for Windows\winext\ext.dll] exts: image 6.6.0003.5, API 1.0.0, built Thu Jan 19 00:06:06 2006 [path: C:\Program Files\Debugging Tools for Windows\WINXP\exts.dll] uext: image 6.6.0003.5, API 1.0.0, built Thu Jan 19 00:06:13 2006 [path: C:\Program Files\Debugging Tools for Windows\winext\uext.dll] ntsdexts: image 6.0.5278.0, API 1.0.0, built Thu Jan 19 00:19:42 2006 [path: C:\Program Files\Debugging Tools for Windows\WINXP\ntsdexts.dll] Note that the same amount of VM leaks when using !dumpmt, worse, !dso leaks ~10MB per command. Willy. | Hi Andreas, | [quoted text clipped - 24 lines] | ================================================== | This posting is provided "AS IS" with no warranties, and confers no rights. Hi, I think this is very strange. As mentioned previously, I have tested this under several configurations (all Windows XP SP2) and this error is always reproducible. I have tested it with the latest version of WinDbg, as well as several older versions. I'm running the RTM version of .NET Framework (and sos.dll) v2.0.50727.42. Does it exist any hotfix that might affect this problem? I don't know why you cannot view the movie I provided, but I have uploaded a zipped version of it as well at: http://www.scitech.se/temp/sosleak2.zip. This movie is a recording from a freshly installed Windows XP SP2 VMWare session, with .NET Framework SDK 2.0 and Debugging Tools for Windows 6.6.35 installed. In the move I use the DumpMT command to show the memory leak, but I think that any sos command will cause a memory leak. I can't understand why you are not able to reproduce this problem, since it's completely reproducible by me. Have you tested it on a freshly installed computer or virtual machine? Best regards, Andreas Suurkuusk SciTech Software AB > Hi Andreas, > [quoted text clipped - 24 lines] > ================================================== > This posting is provided "AS IS" with no warranties, and confers no rights. Hi Andreas, Thanks for your feedback. Yes, I also feel it strange. I used to reproduce out in my working machine, however, currently, I have tried both my Win2003 Sp1 machine and WindowsXP SP2 machine both can not reproduce out the problem. Currently, for simplistic purpose, I only used !dumpdomain command to do the test. Also, your new zipped avi still can not be played on our side, it seems we do not have the decoder. Is it possible for you to attach a test .Net 2.0 assembly and list a detailed steps for me? Then we may follow your steps and reproduce it easier. Thanks Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Hi, The test application I use is an empty Windows Forms application, created as a new project under VS 2005 and then compiled directly. You can download the application from http://www.scitech.se/temp/WindowsApplication18.exe. The steps to reproduce: 1. Install .NET Framework 2.0 2. Install Debugging Tools for Windows (e.g. v6.6.3.5) 3. Start WinDbg (e.g. using the Start menu) 4. Select "Open Executable" and select the application to debug (e.g. WindowsApplication18.exe) 5. Run the application ("g") 6. When the application has started break into the debugger. 7. Type .load c:\windows\microsoft.net\framework\v2.0.50727\sos 8. Issue some sos commands, e.g. !DumpDomain, !DumpMT or !dso (as Willy noted, the !dso command will cause a bigger memory leak). 9. Watch the "Mem usage" and "VM Size" columns in the Task Manager. After each command the memory will increase by at least 5MB. I have tried this on two additional computers, and I see the same problem on them. I have tested with VS 2005 installed, with only the SDK installed and only the framework installed. I have the tested under VMWare, and with all available version of WinDbg. It doesn't matter whether I start the application from WinDbg or attach to it, it is always reproducible. Are we somehow using different versions of sos.dll? I checked the SosLeak2.avi movie, and it appears that you need to have the VMWare decoder installed to view it. If you want to view the movie, you can download the decoder from http://www.vmware.com/download/eula/moviedecoder_v55.html Best regards, Andreas Suurkuusk SciTech Software AB > Hi Andreas, > [quoted text clipped - 20 lines] > ================================================== > This posting is provided "AS IS" with no warranties, and confers no rights. Hi Andres, I was able to reproduce the problem with the virtual memory leak you are talking about. The repro was a little inconsistent, meaning I see the VM memory increase in Task Manager in 2.0 sos.dll when issuing the same SOS command more than once. First I thought the issue was a private memory leak which I did not see until reading the thread again :) The inconsistency I found was that after a few of the same commands are issued, I see a steady increase (not exactlt in the megabits), but it stopps after about 4-5 repititions. It then resumes, but stops again at a lower repitition. I was also able to reproduce it similarly in 1.1 SOS.DLL as well. The VM did not increase that much and it happened less often. This seems to be a problem in Windbg as we cannot reproduce the same behavior in VS. Support for Windbg bugs is not guarranteed. I would suggest you file a Product Feedback bug here with our products unit to let them be aware of the issue and they can address it . Product Feedback: http://lab.msdn.microsoft.com/productfeedback/ Thanks, Dat Bui This posting is provided "AS IS" with no warranties, and confers no rights. -------------------- >Thread-Topic: Major memory leak in sos.dll under .NET Framework 2.0 >thread-index: AcZuEoMgqwsUPguYQM6psaUvXVR2Ew== >X-WBNR-Posting-Host: 83.250.113.46 >From: =?Utf-8?B?QW5kcmVhcyBTdXVya3V1c2s=?= <ASuurkuusk@newsgroups.nospam> >References: <77FC2388-535C-4B36-835B-268293A5979D@microsoft.com> <YfNY$fsXGHA.4620@TK2MSFTNGXA01.phx.gbl> <1432653F-374A-48A5-B3E1-EE5B1DF04270@microsoft.com> <2s$9nz1YGHA.132@TK2MSFTNGXA01.phx.gbl> <1C1BEF0B-F9D7-4901-A404-283364F6E4F3@microsoft.com> <Rwc3v8QZGHA.932@TK2MSFTNGXA01.phx.gbl> <B99E77C3-D2C7-4130-BA97-BDFAC419A828@microsoft.com> <TmQFgbPbGHA.4600@TK2MSFTNGXA01.phx.gbl> <34478CAD-276D-4F48-8CC8-86FBCE223257@microsoft.com> <8uOGjOcbGHA.4672@TK2MSFTNGXA01.phx.gbl> >Subject: RE: Major memory leak in sos.dll under .NET Framework 2.0 >Date: Tue, 2 May 2006 11:02:01 -0700 [quoted text clipped - 77 lines] >> ================================================== >> This posting is provided "AS IS" with no warranties, and confers no rights. Hi ASuurkuusk, Sorry for letting you wait. I finally got the feedback from our CLR team SOS.dll owner, he has confirmed that your analysis is correct. Your fix however is not as simple as testing for NULL, since SOS needs to support the scenario where the debugger is simultaneously debugging multiple processes. Because I am using internal version of Windbg, which behaves different from external version, and internal version's internal work just effectively fixes the leak. This explains why I always can not reproduce the problem with the same steps. The SOS owner and I have contacted Windbg team owner for this issue, and they have filed a record in the database. Maybe this problem will be fixed in future release. Thanks for your reporting and hope my reply helps. Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Hi, Thank you. I'm glad that you have confirmed this problem. I suspected that my workaround didn't work in all cases, but it seems to be work well in my case. I will keep my workaround, and hopefully a future version of WinDbg (or sos.dll) will include a fix for this memory leak. Best regards, Andreas Suurkuusk SciTech Software AB > Hi ASuurkuusk, > [quoted text clipped - 24 lines] > ================================================== > This posting is provided "AS IS" with no warranties, and confers no rights. Yes, also thank you for sharing your analysis result and process with the community. :-) Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== When responding to posts, please "Reply to Group" via your newsreader so that others may learn and benefit from your issue. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. Jeffrey, Did you try !DumpDomain? This effectively leaks 5MB and more depending on the number of domains in the process. Willy. | Hi ASuurkuusk, | [quoted text clipped - 17 lines] | ================================================== | This posting is provided "AS IS" with no warranties, and confers no rights. Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.