 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / CLR / February 2004Performance, Scaling and GC
All I am in the process of load testing an application and I am not happy with the scalabity of the preliminary system. Under a moderate load of 300-400 users per test machine (Dual P4-700,3 Gig of ram), I am noticing some very slow page loads. To simulate the load, I have been using ACT (although the results haven't been consistant - is there a better tool?) Some observations/questions 1) We are object heavy. For each table in our application, we have a corresponding Class with properties equal to that of the columns, as well as a Collection class that is typed to the new DB Item class. Some of our DB Item classes have collections of other DB Items. The objects are nice and handy to code with and include many helper methods for sorting, cloning, etc. 3) Almost ALL of the problems seems to be related to web performance (90+ CPU%). The DB server hums along at 12-14% CPU usage 4) The '% Time in GC' performance counter hovers around 50-60% when monitoring per second. From the articles I've read, this shouldn't be above 30% under *max* load a) The total memory is 3 gig, but under the load test, the aspnet_wp never grows above around 200 meg b) Under this load, there is a Gen 2 Collections called about every second. My understanding is that the Gen 2 Collections are the equivalent of calling GC.Collect() and a complete GC dump. Which is bad c) The Gen 2 heap size consumes 80% of all the bytes in the Heap. Is this normal 5) We use InProc, cookieless sessions for basic state management 6) We store a lot of objects in Web.Cache to save DB hits and to access those cached objects we use Context.Items The Questions A) For almost all of our objects, we never implemented .Dispose(). It was my understanding that with managed code it wasn't needed. But I have read some recent articles about implementing your own .Dispose() to SupressFinalize and "clean up" your large properties/child objects. i) Is .Dispose() worth adding to a 100% Managed code app ii) Will setting the fields to null (large strings, arrays, etc) and calling child object .Dispose() get my objects to clean up faster? Perhaps Gen 0 or 1 collection instead of so much Gen 2 B) With so much RAM avaiable, why wouldn't the aspnet_wp use the extra ram available? Am I setting objects to Nothing too much and marking it for cleanup? I don't have a full understanding of what triggers a collection and how to better leverage the RAM versus CPU C) Is there a tool to analize what objects or types reside in was GC heaps D) I read that I should cache everything I can within RAM limitations. It it possible I am not storing enough in Cache? Storing too much? I dont' want to sound vague, I am just trying to figure out why the app performs so slow under load Any advice on any of the points would be welcome, as well as any references to other resources I could pose these questions Thanks for your tim Tyson, I may not be the best help you can get on this subject, but I can offer what I can and hopefully others can fill in the gaps. <you> > A) For almost all of our objects, we never implemented .Dispose(). It was my understanding that with managed code it wasn't needed. But I >have read some recent articles about implementing your own .Dispose() to SupressFinalize and "clean up" your large properties/child objects. <me> What articles? That's not meant to tell you you're wrong; I'm just interested. Since SuppressFinalize only makes sense when you have a Finalizer, and since you're 100% .Net and therefore probably don't have even one in your custom code, I would say you're fine not worrying about Dispose(). Even if you "clean up" your large objects from within Dispose(), you have no guarantee as to when they're cleaned since timing is all up to the GC. <you> > 6) We store a lot of objects in Web.Cache to save DB hits and to access those cached objects we use Context.Items. <me> Are you caching DataSets? This can be a bad thing. <you> > C) Is there a tool to analize what objects or types reside in was GC heaps? <me> Yes, there are. CLRProfiler is nice. I still use windbg for a lot of heap inspection too (but I don't necessarily have the luxury of 100% .Net). Again, there are some very smart people on this group who may be able to answer you better than I, but hopefully I can help you get a head start. > All, > > I am in the process of load testing an application and I am not happy with the scalabity of the preliminary system. Under a moderate load of 300-400 users per test machine (Dual P4-700,3 Gig of ram), I am noticing some very slow page loads. To simulate the load, I have been using ACT (although the results haven't been consistant - is there a better tool?). > Some observations/questions: > 1) We are object heavy. For each table in our application, we have a corresponding Class with properties equal to that of the columns, as well as a Collection class that is typed to the new DB Item class. Some of our DB Item classes have collections of other DB Items. The objects are nice and handy to code with and include many helper methods for sorting, cloning, etc. > 3) Almost ALL of the problems seems to be related to web performance (90+ CPU%). The DB server hums along at 12-14% CPU usage. > > 4) The '% Time in GC' performance counter hovers around 50-60% when monitoring per second. From the articles I've read, this shouldn't be above 30% under *max* load. > a) The total memory is 3 gig, but under the load test, the aspnet_wp never grows above around 200 meg. > b) Under this load, there is a Gen 2 Collections called about every second. My understanding is that the Gen 2 Collections are the equivalent of calling GC.Collect() and a complete GC dump. Which is bad. > c) The Gen 2 heap size consumes 80% of all the bytes in the Heap. Is this normal? > [quoted text clipped - 4 lines] > The Questions! > A) For almost all of our objects, we never implemented .Dispose(). It was my understanding that with managed code it wasn't needed. But I have read some recent articles about implementing your own .Dispose() to SupressFinalize and "clean up" your large properties/child objects. > i) Is .Dispose() worth adding to a 100% Managed code app? > ii) Will setting the fields to null (large strings, arrays, etc) and calling child object .Dispose() get my objects to clean up faster? Perhaps Gen 0 or 1 collection instead of so much Gen 2? > B) With so much RAM avaiable, why wouldn't the aspnet_wp use the extra ram available? Am I setting objects to Nothing too much and marking it for cleanup? I don't have a full understanding of what triggers a collection and how to better leverage the RAM versus CPU. > C) Is there a tool to analize what objects or types reside in was GC heaps? > > D) I read that I should cache everything I can within RAM limitations. It it possible I am not storing enough in Cache? Storing too much? I dont' want to sound vague, I am just trying to figure out why the app performs so slow under load. > Any advice on any of the points would be welcome, as well as any references to other resources I could pose these questions. > > Thanks for your time Ben Thanks so much for writing <you What articles? That's not meant to tell you you're wrong; I'm jus interested. Since SuppressFinalize only makes sense when you have Finalizer, and since you're 100% .Net and therefore probably don't have eve one in your custom code, I would say you're fine not worrying abou Dispose(). Even if you "clean up" your large objects from within Dispose() you have no guarantee as to when they're cleaned since timing is all up t the GC Here are a couple articles I was referencing. I could have read them wrong and they only covered Managed code http://msdn.microsoft.com/library/en-us/dndotnet/html/dotnetperftechs.asp?frame= true#dotnetperftechs_topic I also found this article which seems to also point to the problem http://weblogs.asp.net/ricom/archive/2003/12/04/41281.asp <me Are you caching DataSets? This can be a bad thing We have one dataset that stores the table schema for a user. Then on page_load, we clone that dataset stored in cache and populate the information for that user. I've actually done some testing and I actually found that that doing that Dataset.Clone() has a HUGE impact. I can simlulate 300 users and do nothing on a page but call that .clone() and it will cause Gen 2 collection every second or so. Definately a huge issue I am going to move forward knowing that is a performance issue, but I worry that it isn't the only one.. Tyson Tyson, Take a look at the section "Object Cleanup" in the article on the http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/d otnetperftechs.asp page. It mentions special things like sockets, file pointers, and the like requiring Dispose and Finalize(), but nothing more. The reason is that the GC doesn't know what to do with these special things, and so leaves it up to you to implement specialized cleanup (cleanup that requires more than simply reclaiming managed memory) in Dispose and your Finalizer. Can you give me an example of one of your Dispose or Finalizer functions? Can you show me what you're disposing of, exactly, in them? And, yes, the .Clone method does have a huge impact. We found the same thing and fixed it. But we are implementing a huge eCommerce system with thousands of transactions a day on a ~90% pure .Net system and aren't seeing any penalties, in fact, we've actually reduced the load on our servers a lot (and the stability is amazing compared to traditional ASP). Ben > Ben, > [quoted text clipped - 10 lines] > > Here are a couple articles I was referencing. I could have read them wrong and they only covered Managed code. http://msdn.microsoft.com/library/en-us/dndotnet/html/dotnetperftechs.asp?frame= true#dotnetperftechs_topic2 > I also found this article which seems to also point to the problem: > http://weblogs.asp.net/ricom/archive/2003/12/04/41281.aspx [quoted text clipped - 3 lines] > > We have one dataset that stores the table schema for a user. Then on page_load, we clone that dataset stored in cache and populate the information for that user. I've actually done some testing and I actually found that that doing that Dataset.Clone() has a HUGE impact. > I can simlulate 300 users and do nothing on a page but call that .clone() and it will cause Gen 2 collection every second or so. Definately a huge issue. > I am going to move forward knowing that is a performance issue, but I worry that it isn't the only one... > > Tyson Ben I haven't written any .Dispose() functions yet, but what they would do is set any large strings to null as well as call the .Dispose method for any child classes of the object As far as the dataset.clone. I was surprised by how much overhead there appeared to be under load to do this clone. Let me explain the dataset. It's a strongly typed dataset with just one table with 280 or so columns. Does this overhead sound about right? How did you guys circumvent the need for a clone? Create a more streamlined class Another question - I am not sure of the architecture of your app, but do you guys use application caching? I am wondering if output or API caching everything I can will help with the CPU issues. Thanks again Tyson > I haven't written any .Dispose() functions yet, but what they would do is set any large strings to null as well That's unnecesary. Setting object references to null in a dispose isn't going to help, since if your main object is null, and there's no other path to the child object, then it has no roots and will be collected. If the child object is referenced somewhere else, then setting one of your references to it to null isn't going to do anything. > as call the .Dispose method for any child classes of the object. So long as the child classes really need a Dispose method, that works (even if they don't have unmanaged code, since they might reference something that eventually does). This is the only real case where you need a Dispose (but not a finalizer) on a pure managed class. -mike MVP Hi Tyson >A) For almost all of our objects, we never implemented .Dispose(). It was my understanding that with managed code it wasn't needed. But I have read some recent articles >about implementing your own .Dispose() to SupressFinalize and "clean up" your large properties/child objects. > i) Is .Dispose() worth adding to a 100% Managed code app? > ii) Will setting the fields to null (large strings, arrays, etc) and calling child object .Dispose() get my objects to clean up faster? Perhaps Gen 0 or 1 collection instead of >so much Gen 2? i) The thing you should remember about finalizers is that they're expensive to run, the order they're run in is nondeterministic, and there's no guarantee they will even be run at all. Even with a 100% Managed application, you may want to consider using Dispose. For example, closing file handles, sockets, database connections, etc. should be done in Dispose, and called when you're done with the objects. ii) You shouldn't be calling GC.Collect() yourself. The Garbage Collector will make decisions based on your memory usage and tune itself accordingly. Setting your references to null when you're done with them *may* coerce the GC to collect early, but no guarantees. Hope that helps. -Chris -------------------- >Thread-Topic: Performance, Scaling and GC >thread-index: AcPq4MjK+AF22zB4TZynCTjHOzi7/A== [quoted text clipped - 20 lines] > >All, I am in the process of load testing an application and I am not happy with the scalabity of the preliminary system. Under a moderate load of 300-400 users per test machine (Dual P4-700,3 Gig of ram), I am noticing some very slow page loads. To simulate the load, I have been using ACT (although the results haven't been consistant - is there a better tool?). Some observations/questions: 1) We are object heavy. For each table in our application, we have a corresponding Class with properties equal to that of the columns, as well as a Collection class that is typed to the new DB Item class. Some of our DB Item classes have collections of other DB Items. The objects are nice and handy to code with and include many helper methods for sorting, cloning, etc. 3) Almost ALL of the problems seems to be related to web performance (90+ CPU%). The DB server hums along at 12-14% CPU usage. 4) The '% Time in GC' performance counter hovers around 50-60% when monitoring per second. From the articles I've read, this shouldn't be above 30% under *max* load. a) The total memory is 3 gig, but under the load test, the aspnet_wp never grows above around 200 meg. b) Under this load, there is a Gen 2 Collections called about every second. My understanding is that the Gen 2 Collections are the equivalent of calling GC.Collect() and a complete GC dump. Which is bad. c) The Gen 2 heap size consumes 80% of all the bytes in the Heap. Is this normal? 5) We use InProc, cookieless sessions for basic state management. 6) We store a lot of objects in Web.Cache to save DB hits and to access those cached objects we use Context.Items. The Questions! A) For almost all of our objects, we never implemented .Dispose(). It was my understanding that with managed code it wasn't needed. But I have read some recent articles about implementing your own .Dispose() to SupressFinalize and "clean up" your large properties/child objects. i) Is .Dispose() worth adding to a 100% Managed code app? ii) Will setting the fields to null (large strings, arrays, etc) and calling child object .Dispose() get my objects to clean up faster? Perhaps Gen 0 or 1 collection instead of so much Gen 2? B) With so much RAM avaiable, why wouldn't the aspnet_wp use the extra ram available? Am I setting objects to Nothing too much and marking it for cleanup? I don't have a full understanding of what triggers a collection and how to better leverage the RAM versus CPU. C) Is there a tool to analize what objects or types reside in was GC heaps? D) I read that I should cache everything I can within RAM limitations. It it possible I am not storing enough in Cache? Storing too much? I dont' want to sound vague, I am just trying to figure out why the app performs so slow under load. Any advice on any of the points would be welcome, as well as any references to other resources I could pose these questions. Thanks for your time  SignatureThis posting is provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm Note: For the benefit of the community-at-large, all responses to this message are best directed to the newsgroup/thread from which they originated. Chris, Thanks so much for responding. I am adding .Dispose() to our code generator and am modeling it off on a MSDN article here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cp conimplementingdisposemethod.asp Is this a sound design? Anything to watch out for? I have two distinct scenarios that occur that I want to make sure adding Dispose() would help out with: 1) Our classes often have several fields for child objects. From simple ArrayLists to fulled typed Collections. I should be adding code to call those objects' .Close, .Dispose, etc. and nulling them out, right? 2) Should I null out the internal fields that are known to be large? Like large strings? From what I can tell, the goal is to do everything I can to tell GC that these things are available to clean up ASAP so that they don't last until Gen 2 Collection and cause CPU bottlenecks. Tyson Adding Dispose() and finalizers isn't something you do because you're worried about performance, it's something you do because you have to. The GC handles managed memory and handles it well. What it doesn't handle are unmanaged resources, such as connections, file handles, and unmanaged memory. You'll find that framework classes that are wrappers around unmanaged resources implement IDisposable (or sometimes, have a Close() method instead). When you use these objects, you should use the 'using(...) {...}' construct. If you hold a reference to one of these objects in your own classes, that's when you need to implement a finalizer and Dispose(). [It's often useful to put a debug assertion in the finalizer to check that people are correctly using using]. Setting references to null (in the right places!) can help the GC, but it won't force a GC. Adding a finalizer and/or Dispose method that simply clears member references is (a) unnecessary and (b) a good way to slow your application down. The rules of thumb are: (1) If your child needs disposing, you need a dispose method, and (2) if you use an object with a dispose method, call it when you're finished, using 'using'. That removes the object from the finalization queue (which is expensive, and keeps objects alive for at least one extra GC pass). > Chris, > > Thanks so much for responding. I am adding .Dispose() to our code generator and am modeling it off on a MSDN article here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cp conimplementingdisposemethod.asp > Is this a sound design? Anything to watch out for? > > I have two distinct scenarios that occur that I want to make sure adding Dispose() would help out with: > 1) Our classes often have several fields for child objects. From simple ArrayLists to fulled typed Collections. I should be adding code to call those objects' .Close, .Dispose, etc. and nulling them out, right? > 2) Should I null out the internal fields that are known to be large? Like large strings? > > From what I can tell, the goal is to do everything I can to tell GC that these things are available to clean up ASAP so that they don't last until Gen 2 Collection and cause CPU bottlenecks. > Tyson Hi Tyson 1) Yes, that's a good pattern to use. Using the GC.SuppressFinalize() API is also good in case you need to use finalizers as well as Dispose (for example, there are code paths that bypass your call to Dispose, like exceptions, or rude aborts). Like you said, calling child objects' Dispose in your Dispose ensures proper cleanup. 2) Like I said before, nulling the fields won't necessarily make a difference, but it won't hurt. You'll probably notice that in debug mode, the GC is more conservative about collecting (makes debugging easier), so nulling won't make such a difference in release. Large objects are put in their own heap (the aptly named Large Object Heap), which is considered generation 2, so don't get worried if large objects immediately get put in there. It's more expensive for the GC to collect large objects, but it will if memory is getting low. Cheers, -Chris -------------------- >Thread-Topic: Performance, Scaling and GC >thread-index: AcPrbpfA4gn15wILRTe7E33rPtxPgQ== [quoted text clipped - 21 lines] > >Chris Thanks so much for responding. I am adding .Dispose() to our code generator and am modeling it off on a MSDN article here http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cp conimplementingdisposemethod.as Is this a sound design? Anything to watch out for I have two distinct scenarios that occur that I want to make sure adding Dispose() would help out with 1) Our classes often have several fields for child objects. From simple ArrayLists to fulled typed Collections. I should be adding code to call those objects' .Close, .Dispose, etc. and nulling them out, right 2) Should I null out the internal fields that are known to be large? Like large strings From what I can tell, the goal is to do everything I can to tell GC that these things are available to clean up ASAP so that they don't last until Gen 2 Collection and cause CPU bottlenecks Tyson SignatureThis posting is provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm Note: For the benefit of the community-at-large, all responses to this message are best directed to the newsgroup/thread from which they originated. Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.