Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / CLR / October 2003

Tip: Looking for answers? Try searching our database.

Memory problem

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Sergey Ivasenko - 21 Oct 2003 11:04 GMT
I have a Win form application, that receives large (more than 100 000
records) datasets
using ADO.NET
I open a form and execute data select. The form obtains about 140 000
records pretty quickly.
I reopen the query for this form several times. Each time I reopen the
query, it takes more and more
time to retreive all data. The application occupies more and more memory and
it slowes down.
I'm really frustrated about this fact.
How could I EFFECTIVELY controll memory usage? GC.Collect() doesn't help as
it works longer
and longer as I perform data retreival.
Seems like GC technology needs to be improved (or am I doing something
wrong)?

Thank you in advance,
Sergey.
Cowboy \(Gregory A. Beamer\) - 21 Oct 2003 16:46 GMT
The GC collects when memory reaches a critical mass. While it is not optimal
for all situations, it is a good balance of performance for cleanup versus
performance of running apps. You can attempt to force a Collect(), but it is
marginal, at best, unless you want to take over complete control of the CLR.
I would not advise that tack.

If the underlying data is not changing, you should consider caching the data
rather than retrieving every time the form is brought up. Also, since a user
cannot go through 140,000 records in one sitting, much less one page view,
consider filtering before placing on the form. Reporting is the only thing I
can see that might break this, and aggregating on the data server is a
better option here, in most cases, rather than churning records in your .NET
app.

Signature

Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

**********************************************************************
Think Outside the Box!
**********************************************************************

> I have a Win form application, that receives large (more than 100 000
> records) datasets
[quoted text clipped - 14 lines]
> Thank you in advance,
> Sergey.
Subramaniyan Neelagandan - 21 Oct 2003 19:23 GMT
One speculation on this topic:

Based on what you have said and with the following assumption, this makes
sense.

Create a dataset containing 140,000 records.

Hold a reference to the dataset either directly or indirectly.
 Directly:    Keep it in a variable or in one of you class instance.
Indirectly:  A reference is kept in hashtable or arraylist or WinForm
control and etc.

Now when you retrive the data for the second time, due to lack of space in
the heap, GC might get kicked in.
But GC can not reclaim the space consumed by the previous dataset because
someone is holding an reference to it.
So then why it takes long time? It takes long time because everytime the GC
kicks in, it has to traverse the entire objects refered directly or
indirectly by your program. Since the dataset has 140000 records, it needs
to visit these records. On top of it, if these records has reference to
string or byte[] fields then it needs to visit those objects as well. At the
end it would have found comparatively only few objects for which the space
can be reclaimed.

From now on the GC will get kick in more often till the heap space use
comes down below the critical limit.

So, do a code-walk-through of your application to make sure that the
dataset is not referenced by anyone.

You can validate whether you have the above said problem or not by doing the
following few steps.

1. Before you retrieve and store a dataset in a variable, clear the variable
to null. This will ensure that always you clear the reference to the
existing dataset before you create a new dataset.

2. If you still experiencing the problem, remove all records from the
existing dataset before you clear the variable to null. This will ensure
that even if someone is holding a reference to your existing dataset, the
dataset will not have any records thereby letting GC to traverse few objects
and reclaim almost all the spaces.

After the above change, if the program performs as good as the first one,
the you know that someone is holding a reference to the existing dataset
which is not cleared at all. Now, just findout the culprit and clean it up.

-
S N

> I have a Win form application, that receives large (more than 100 000
> records) datasets
[quoted text clipped - 14 lines]
> Thank you in advance,
> Sergey.
Peter Koen - 21 Oct 2003 23:46 GMT
[...]

I want to add a little bit:

Maybe it would be usefull to hold the reference to the Dataset not directly
from the ApplicationRootReferences, but through a WeakReference. That would
free up the memory in case that there would be urgent need to do so.

Signature

best regards

Peter Koen
-----------------------------------
MCAD, CAI/R, CAI/S, CASE/RS, CAT/RS
http://www.kema.at

Sergey Ivasenko - 22 Oct 2003 08:45 GMT
Thaks for your help!

>  So, do a code-walk-through of your application to make sure that the
> dataset is not referenced by anyone.

It is not.

> 2. If you still experiencing the problem, remove all records from the
> existing dataset

It didn't help. Memory allocation still took a long time.

>... before you clear the variable to null.

And this works pretty fine. I just Dispose DataSet object and that set it to
null.
Furthermore, even call to Dispose is enough. I think, that when I deleted
tables
from the DataSet - DataSet.Tables.Clear() - and even rows later on, GC
could not reclaim memory that fast. But when I disposed DataSet, which is
the root object for all data, then it helped GC to "understand" that the
whole
rows collection is not used any more.
Well, I could be mistaken, but I think that in order to provide better
perfomance and
fast memory allocation you should always try to dispose "root" objects. Like
DataSet
in this case.

Best regards,
Sergey.

> > I have a Win form application, that receives large (more than 100 000
> > records) datasets
[quoted text clipped - 16 lines]
> > Thank you in advance,
> > Sergey.

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.