 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / CLR / June 2005Managed Memory Consumption - Service vs. Command Line EXE
Hello All, I have written an application to process a file full of XML. For each "main element" in the xml file, the application parses it and writes the resulting object to an Oracle database. Here is the problem. In an XML file with 20000 objects, running as a service, the silly thing consums 500MB of memory. If I take the exact same code, and place it in a console app, it uses less than 30 MB of memory. That is, I simply took what was in the service OnStart() method, and placed it in Main() in the console app. I'm just starting up a couple of threads: One thread reads and parses the XML (serially, not using the DOM), creates a lightweight object and populates a bunch of properties, and sticks the object in an MSMQ queue. The other thread reads these lightweight objects out of the queue, and serializes them to an Oracle database. I have been over and over the code, and I can see no place where I am using unmanaged objects, or holding on to references. This is most perplexing. As a service, it runs like crap. As a console app, it runs just fine. When I examine managed memory with Perfmon while running the service, the Gen2 heap size grows enormously. No objects are pinned. When I examine the same thing running as a console app, Gen2 never gets very big at all (maybe 35MB max). Any idea what is going on here? Is there something special I need to do in a service to manage memory. Please respond via newsgroup. When I post my email address, the spambots have a field day! Thanks Robert Reineri Sr. Business Systems Engineer Credit One Bank > Hello All, > [quoted text clipped - 9 lines] > OnStart() method, and placed it in Main() in the console app. I'm just > starting up a couple of threads: My guess is it's Server vs Workstation mode of the CLR. I'm not sure how you can do it with .NET 1.1 but with .NET 2.0 you can specify in your .config file which version of the clr (server or workstation).  SignatureRegards Vincent Parrett VSoft Technologies http://www.finalbuilder.com Blogs : http://blogs.finalbuilder.com ---------------------------------------- Automate your Build Process with FinalBuilder Nope, it's not because your application is a "Service" that the CLR will switch to Server mode. Server and Workstation mode is selected through a Config parameter and requires multi-cpu boxes. Willy. "Vincent Parrett (VSoft Technologies)" <vincent@no-spam.finalbuilder.com> wrote in message news:uvN0WnVcFHA.3620@TK2MSFTNGP09.phx.gbl... >> Hello All, >> [quoted text clipped - 13 lines] > you can do it with .NET 1.1 but with .NET 2.0 you can specify in your > .config file which version of the clr (server or workstation). Sooo... is there something I should be setting in the .config file for the service??? Thanks Robert > Nope, it's not because your application is a "Service" that the CLR will > switch to Server mode. [quoted text clipped - 22 lines] >> you can do it with .NET 1.1 but with .NET 2.0 you can specify in your >> .config file which version of the clr (server or workstation). No, not at all, your issue does not relate to this. If you see a lot of objects ending in gen2, it's because you are: - holding live references to them, - or the GC cannot compact because there are pinned objects, or - in the case they are Finalizable objects, the finalizer cannot run for one or an other reason. It's hard to tell from the description you gave, it would be great if you could provide a sample that illustrates the issue. Willy. > Sooo... is there something I should be setting in the .config file for the > service??? [quoted text clipped - 29 lines] >>> you can do it with .NET 1.1 but with .NET 2.0 you can specify in your >>> .config file which version of the clr (server or workstation). Robert, the easiest way to figure out what objects are laying around for so long (and how big they are), is to use a profiler. MS has the free CLR Profiler which might help. There is also one called .Net Memory Profiler that i've had really good luck with (http://www.scitech.se/memprofiler/). bottom line, is that your just guessing unless you profile your app. That way you'll know for sure what the problem is > Hello All, > [quoted text clipped - 33 lines] > Sr. Business Systems Engineer > Credit One Bank I agree, and will definetly profile it. I have the CLR profiler. What has me stumped is why it works fine as a console app, and is a memory hog as a service. It implies there is nothing wrong with the code. What changes is the environment. I will try the profiler and see what happens. Thanks all Robert > Robert, > the easiest way to figure out what objects are laying around for so long [quoted text clipped - 53 lines] >> Sr. Business Systems Engineer >> Credit One Bank > Hello All, > [quoted text clipped - 4 lines] > In an XML file with 20000 objects, running as a service, the silly thing > consums 500MB of memory. Sounds like a garbage collection issue to me: I've learned not to depend on the framework to keep it clean. You have two options (I think): force garbage collection regularly, or call .Dispose() on everything that has such a method before it goes out of scope. That includes at times when something is made to point to another object of the same type, for example when you replace the image in a picturebox by another one: picturebox.Image.Dispose() picturebox.Image = NewImage Bad suggestion, which doesn't in no way explain why it runs from the console application, the GC has no knowledge about what kind of application it is - service or console , calling GC.Collect yourself only disturbs the normal functioning of the GC. If a Gen2 increases to 500MB the GC has run several thousand times already, just watch the GC performance counters. Willy. >> Hello All, >> [quoted text clipped - 23 lines] > picturebox.Image.Dispose() > picturebox.Image = NewImage > Bad suggestion, which doesn't in no way explain why it runs from the console > application, the GC has no knowledge about what kind of application it is - [quoted text clipped - 3 lines] > If a Gen2 increases to 500MB the GC has run several thousand times already, > just watch the GC performance counters. The more I use .Net, the more I get convinced that GC doesn't kick in even remotely soon enough. By relying on automatic garbage collection, I've seen a standard VB.Net app take up 600-700 MB on a 1GB RAM system when it didn't need more than a few tens of MB at the time. That application had loaded a lot of small images, most of which had gone out of scope. They remained in memory, and garbage collection only kicked in after the app had eaten up over 600 MB. Two solutions solved it: either force garbage collection regularly (not the best way, I agree - prevention is better than fixing symptoms), or call Dispose() on each image before it went out of scope. I implemented the latter. Sometimes it's even worse than just keeping memory occupied. The CLR also keeps files open unnecessarily, as a simple example illustrates: put this code in the load event of a form with a picturebox (both jpeg files must exist, of course). PictureBox1.Image = Image.FromFile("img1.jpg") PictureBox1.Image = Image.FromFile("img2.jpg") Start the app, then try to delete file "img1.jpg". It won't work before the app is stopped, because the CLR still has the file open. To solve it either include a line "PictureBox1.Image.Dispose()" before loading the second image, or force garbage collection after loading it. That's not what the GC is used for, the GC manages the "managed heap", that is memory used to store the "managed objects", that's all. The GC doesn't know anything about unmanaged memory nor about unmanaged resources like handles and files, so it's up to you to clean up unmanaged stuff by disposing objects that implement IDisposable. The GC cannot compact the heap if there are a lot of pinned objects, PInvoke interop and COM interop are the manifest causes of GC collection prevention, so watch out for dangling COM object references and long running PInvoke calls. If ever you have notices memory increases to 500MB it's not because the GC doesn't run, again do yourself a favor and look at the GC memory counters using perfmon, you will see that the GC runs more often than you think, but the GC cannot free objects that are still referenced, forcing a GC Collect wont help anyway. Another point to consider is that the GC doesn't compact memory that holds large objects (>85Kb), it's possible that this "Large Object Heap" gets so badly fragmented that no memory can be returned to the CLR nor the OS, so watch out for large objects. Willy. >> Bad suggestion, which doesn't in no way explain why it runs from the >> console [quoted text clipped - 38 lines] > To solve it either include a line "PictureBox1.Image.Dispose()" before > loading the second image, or force garbage collection after loading it. > If ever you have notices memory increases to 500MB it's not because the GC > doesn't run, Then why did the problem go away by forcing GC? That was the first solution I tried, I started calling Dispose() on images that were about to go out of scope only later. I have looked at the GC counters yesterday while writing that reply. VS.Net had been open constantly for over 2 hours, a second instance had been open with another application for about half an hour, and a program was running in the debugger for a couple of minutes. There were 4 instances of the GC counters at that moment. At least two were at all zeroes, the _Global_ instance was at 1 for each level (I don't remember checking the 4th instance). >> If ever you have notices memory increases to 500MB it's not because the >> GC [quoted text clipped - 13 lines] > were at all zeroes, the _Global_ instance was at 1 for each level (I > don't remember checking the 4th instance). Run your code in release mode out of the debugger and watch the perf counters for gen0, 1, 2 (size of and number of collection runs). It's not possible they are at zero, when the gen0 size exceeds a certain threshold the GC kicks in, this threshold is lower than 512Kb in general on v1.1 (it's somewhat higher on v2.0 where the GC is less aggressive), so it should kick in each time the threshold is exceeded possibly move the remaining live objects to gen1. The same thing is done for gen1 (the threshold being larger ), surviving objects are move to gen2 where they stay as long as they live. The story for large objects (>85kb) is different though, they are only collected when the GC does a full collect (gen2 collect) or a forced GC.Collect(). Note that this heap, as I told you before, is never compacted!, this leads to memory overflows when the LOH gets badly fragmented. Willy. I was about to throw in a comment about release/debug. The GC is very different (almost non-existent) when you run in debug mode. Release builds are essential. SignaturePhil Wilson [MVP Windows Installer] ---- > >>> If ever you have notices memory increases to 500MB it's not because the [quoted text clipped - 31 lines] > > Willy. > I was about to throw in a comment about release/debug. The GC is very > different (almost non-existent) when you run in debug mode. Release builds > are essential. That would explain all or most of the symptoms that lead me to say GC doesn't kick in nearly often enough. On my own system, it's virtually always debug builds.
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.