 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / CLR / December 2005Memory allocation strategy
I am considering different options of memory allocation for a workstation application, and would LOVE some feedback and thought.. The application goes through the following life-cycle: 1) Started. Low memory usage. 2) Loading data. Massive memory increase. 3) Working with loaded data. Loads of data processed, no increase. 4) Disposing data. Massive memory decrease. 5) Application idle. Low memory usage. 6) Loading second set of data. Goto 2). * The data is in the order of 100 to 1000 Mb, each allocation piece being typically 0.5 - 1 Mb. * When working with the loaded data there are loads of memory lookups, so performance is important. * Step 3) goes on for say 10 minutes, ie not hours or days. With the current strategy I run into 2 separate problems: a) Data loading fails after a few tasks, due to OutOfMemoryException. Since there is little memory allocated at that time I presume this is related to fragmentation of the LOH. b) I cannot load as much data as could be expected, even on a process that was just started. On a machine with 2GB RAM I would expect to be able to load 1000-1500 Mb of data, leaving say 500-1000 Mb for application overhead and Windows stuff. This is not the case, loading more than 700 Mb seems very difficult. So, considering the scenario above, what allocation strategy should I use? Perhaps split my smallest allocation unit, so that they fit the small object heap? This would probably avoid the fragmentation issue, but would it bring worse performance due to array/index conversion operations? And would allocating HUGE amounts of long-lived objects mess up GC performance and other internal processes? Or should I try and use a smaller allocation unit, but still keep it large enough to be put on the LOH? Again, hoping that this would help fragmentation... And what about my problems with loading larger amounts of data? Have you investigated the memory by taking a dump of the memory when the application crashes. Have you been noticing heap fragmentation in dumps? Where are the objects are they in the Large Object Heap. I would investigate thoroughly before I decide on the strategy. I had a feeling this reply was going to show up... Perhaps I wrote the post hoping for some the-one-and-only-true-way kind of answer :) Having done some initial probing I have come to the following conclusions: * In a simple test app with no overhead allocating 1500 Mb in 0.5Mb chunks works fine. * In some spooky way my application seems to be able to load more data when the debugger is attached. Could it be that I allocate faster than the GC could free my disposed data? * The following OOM situation was analyzed: - 715 Mb of 0.5 Mb chunks and 34 Mb of 125k chunks were loaded OK. - Total managed heap size was 830 Mb, of which 750 Mb was in LOH. - Total physical memory was 1015 Mb, total comitted was 1230 Mb. - Largest free region was 41 Mb How should I move on with this? What confuses me at this point is why a fresh process (I had not loaded any data earlier) with the numbers above fails to find a free region? In a perfect world there should be at least 750 Mb of free memory for me left to use... What further investigations should be done? I am not that experienced in memory profiling and tools, so please help me! Or is this simply the way thinks work - one minute I can load 1000Mb of data, and the next only 700? > Have you investigated the memory by taking a dump of the memory when > the application crashes. Have you been noticing heap fragmentation in > dumps? Where are the objects are they in the Large Object Heap. I would > investigate thoroughly before I decide on the strategy. The link below should help you using tools to analyze the memory problem. blogs.msdn.com/mvstanton/ Thanks! I've been playing around some with SOS but haven't quite found what I'm looking for yet. There seems to be tons of ways to examine my managed heap, both with debugger tools and my memory profiler. But what about remaining VM? If my managed heap is about 1 GB and the largest remaining block is less than 64 Mb, I would be very interested in finding out what has happened with the remaining hundreds of Mb of VM. Any ideas on commands/switches that would provide information about the state of my non-managed VM? > If my managed heap is about 1 GB and the largest remaining block is less > than 64 Mb, I would be very interested in finding out what has happened [quoted text clipped - 3 lines] > Any ideas on commands/switches that would provide information about the > state of my non-managed VM? '!address -summary' in windbg can be very useful for initial troubleshooting of unmanaged VM usage problems.  SignatureThis posting is provided "AS IS" with no warranties, and confers no rights. > Thanks! > [quoted text clipped - 12 lines] > Any ideas on commands/switches that would provide information about the > state of my non-managed VM? 1. Windbg + SOS loaded. !vmstat 2. !address -summary 3. vadump.exe (my prefered tools for quick VM lookup) Willy. inline Willy. >I am considering different options of memory allocation for a workstation > application, and would LOVE some feedback and thought.. [quoted text clipped - 20 lines] > there is little memory allocated at that time I presume this is related to > fragmentation of the LOH. This is not possible when all the objects in the LOH have been collected, when this happens the excess LHO heap segments should have returned to the OS (assuming point 6 above is true), a new run should recreate the LOH segments. > b) I cannot load as much data as could be expected, even on a process that > was just started. On a machine with 2GB RAM I would expect to be able to > load > 1000-1500 Mb of data, leaving say 500-1000 Mb for application overhead and > Windows stuff. This is not the case, loading more than 700 Mb seems very > difficult. You can use vadump.exe (found in "resource kit" and "Windows debugger tools") and check your virtual address state at each stage in the process. Another option is to use a decent memory profiler. > So, considering the scenario above, what allocation strategy should I use? No stategy sofar, first you have to measure, now you are only guessing. IMO you aren't releasing all large objects (note: this is also a guess). > Perhaps split my smallest allocation unit, so that they fit the small > object [quoted text clipped - 9 lines] > > And what about my problems with loading larger amounts of data? >I am considering different options of memory allocation for a workstation > application, and would LOVE some feedback and thought.. > > The application goes through the following life-cycle: ... > And what about my problems with loading larger amounts of data? Well, I can confirm the problems with loading large amount of data from my own experience. I noticed the problem on machine with 1 Gb of RAM, where loading 200 - 300 Mb of data caused extensive paging and total performance degradation. Strange... Goran >>I am considering different options of memory allocation for a workstation >> application, and would LOVE some feedback and thought.. [quoted text clipped - 11 lines] > > Goran That doesn't means anything, how much of this 1GB is there left free when your application starts? That's what is important. Also how do you know that you are loading 200-300Mb of data, what kind of data is it, does it contain string data, remember strings in .NET are UNICODE (16 bit per char), so when loading ASCII string data the size of the strings will double! Willy. ... >> Well, I can confirm the problems with loading large amount of data from >> my own experience. I noticed the problem on machine with 1 Gb of RAM, [quoted text clipped - 9 lines] > char), so when loading ASCII string data the size of the strings will > double! ... It's been while since I did the test, but genarally about 700 mb was left free. Application uses 100mb when under load. All data is pictures of size 1.5 mb (byte[]) + small overhead. That leaves 600 mb. However, when I hit 300 mb in buffer (about 200 pics), it just stalls. Application does very intensive work, 100% CPU, allocates betwen 20 - 40 mb/sec and reads/writes additional 10-20 mb/sec peak to disc. Damn thing GC collects every 2-3 seconds :)). That's where that 100 mb mem usage comes when buffer is empty. In idle state it is about 20 mb (and it is almost never idle in production, 24/7 app). I check mem usage with .NET memory profiler. I moved buffer to disc (traded IO throughput for memory) and it stabilazed, so I skipped further investigation. Generally, .NET performs really nice, but I suspect w2k starts swapping agressivly when you hit about 80% mem usage (there is also need for disk cache which could generally account for additional 100mb). Because of allocation rates, application is very sensitive to swaping. Since w2k is general purpose OS, I see no problem with it (I have additional room for optimizing, since design is sub-optimal). Anyway, is there a tool that would distinguish between hard and soft page faults? Task manager shows total page faults and to distinguish between hard page fault and soft page faults I have to resort to unorthodox methods... Regards, Goran
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.