 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / ADO.NET / April 2006Large Datasets
Has anyone seen any documentation or white papers with suggestions on how to handle large datasets. My task is to read in 10K records and then step through each one processing it's data. The processing involved fetching 3 other records that match the current, manipulate the data and then write out 5 records to a different database. DataReader is out of the question, because this whole precessing time could take 10 or 20 minutes, so it looks like a DataSet. My concern is what happens if a DataSet exceeds memory. Is it cached on disk in the swapfile or does .Net handle it's temp storage. I'd love to find some kind of guideline on how large can a DataSet get without causing calaterail damage in the environment. Thanks for any guidance. Bernie Hi Bernie, Is server processing an option for you? (stored procedure or something) What databaes are we talking?  SignatureMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ > Has anyone seen any documentation or white papers with suggestions on how > to handle large datasets. My task is to read in 10K records and then step [quoted text clipped - 13 lines] > > Bernie No, There are two applications where this applies. First the originating database is an ODBC connection to a Faircom database. The vast majority of the customers using this will be running XP Pro with no server. The second application is for data conversion where the originating database could be just about anything, Pervasive, Advantage and dBase are the most common. Bernie > Is server processing an option for you? (stored procedure or something) > What databaes are we talking? Bernie, A dataset can be as large as the program leaves enough memory for it. Where the program can be AFAIK not larger than 800Mb. I hope this helps Cor > Has anyone seen any documentation or white papers with suggestions on how > to handle large datasets. My task is to read in 10K records and then step [quoted text clipped - 13 lines] > > Bernie Hi Cor, > Bernie, > > A dataset can be as large as the program leaves enough memory for it. > Where the program can be AFAIK not larger than 800Mb. Where did you get this number from? Voodoo? SignatureMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ > Miha Markic [MVP C#] > Where did you get this number from? Voodoo? What do you mean? Cor I mean where do you get the 800Mb limit from. SignatureMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ >> Miha Markic [MVP C#] >> Where did you get this number from? Voodoo? >> > What do you mean? > > Cor >I mean where do you get the 800Mb limit from. From Herfried K. Wagner who gave this link http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx Cor I could discuss here, but there are 64 bit CPUs out there and there is .net 2 that manages memory better. Ah, and if I recall correctly, there is a switch that lets you use more than 2GB memory for user appsin 32bit windows, too. And 800Mb is not accurate - it is just an assumption. SignatureMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ > >I mean where do you get the 800Mb limit from. > [quoted text clipped - 3 lines] > > Cor >I could discuss here, but there are 64 bit CPUs out there and there is .net >2 that manages memory better. >Ah, and if I recall correctly, there is a switch that lets you use more >than >2GB memory for user appsin 32bit windows, too. >And 800Mb is not accurate - it is just an assumption. Did I write something else? Although the by you mentioned switch (the name is 3Gb switch) seems not affect this behaviour it stays aprox 800Mb per process. Cor >>I could discuss here, but there are 64 bit CPUs out there and there is >>.net [quoted text clipped - 5 lines] > > Did I write something else? " Where the program can be AFAIK not larger than 800Mb." Note: 64bit CPUs can address terabytes of memory. SignatureMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ > " Where the program can be AFAIK not larger than 800Mb." > > Note: 64bit CPUs can address terabytes of memory. Yes I wrote As Far As I Know. Can you than give it for a 64bit CPU. Because this is not a memory behaviour but an OS behaviour and than even as it is written with the layer Net 1.x. However as you can enlighten me with some more precise figurs for a 64bit XP OS with Net 2.0 and/or Vista than I will be delighted. Cor Cor: I did some extensive testing on this a couple of years back with the 1.1 .Net Framework and console/web services. The /3GB switch *might* affect the amount of memory a .Net application can use. It really depends on the flavor of Operating System. For example, a test app on 32-bit Windows XP or Windows 2003 Standard Server (using /3GB) will only allocate an additional 900MB - 1200 MB of memory. But on 32-bit Windows 2003 Enterprise Server, using the /3GB switch will provide an application with the additional memory. Note, the /3GB switch is a double edged sword as it will limit any OS processes max memory size to use 1GB. Some useful info from the original thread (watch word wrap) - http://groups.google.com/group/microsoft.public.dotnet.framework.aspnet.w ebservices/tree/browse_frm/thread/cf6567aa5b6ee5d8/265abc31aa209df0?rnum= 1&hl=en&q=%2F3gb+clausius&_done=%2Fgroup% 2Fmicrosoft.public.dotnet.framework.aspnet.webservices%2Fbrowse_frm% 2Fthread%2Fcf6567aa5b6ee5d8%2F265abc31aa209df0%3Ftvc%3D1%26q%3D% 2F3gb+clausius%26hl%3Den%26#doc_265abc31aa209df0 Jeff Clausius SourceGear > Although the by you mentioned switch (the name is 3Gb switch) seems > not affect this behaviour it stays aprox 800Mb per process. > > Cor > Has anyone seen any documentation or white papers with suggestions on how > to handle large datasets. My task is to read in 10K records and then step [quoted text clipped - 9 lines] > guideline on how large can a DataSet get without causing calaterail damage > in the environment. 10,000 rows in a DataSet really isn't all that much. Just try it. David Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.