.NET Forum / .NET Framework / New Users / October 2006
Large text file - in memory ( > 60mb)
|
|
Thread rating:  |
Jurjen de Groot - 10 Oct 2006 17:24 GMT My program needs to search a large textfile (>60MB).
At this time I'm using a streamreader to read the file into a string-variable (objString = sr.ReadToEnd). Before reading the file the proces running my programm uses about 10mb, after reading the text-file into the string, it uses over 200mb. I would expect the program to use between 70 and 100mb.
Is there a more efficient way of storing this data in-memory and still be able to search through it ... ?
TIA, Jurjen.
Carl Daniel [VC++ MVP] - 10 Oct 2006 18:06 GMT > My program needs to search a large textfile (>60MB). > [quoted text clipped - 6 lines] > Is there a more efficient way of storing this data in-memory and still be > able to search through it ... ? Keep in mind that .NET strings are UTF-16, so reading an ANSI text file will typically double the size in bytes.
If you read the file into a byte array, you'll use less memory, but you won't be able to use the .NET string searching facilities (e.g. System.String member functions, regular expressions, etc).
Depending on your searching requirements, you might be able to use a simpler search facility, such as the one in this article:
http://www.codeproject.com/cs/algorithms/BoyerMooreSearch.asp
(but you'd have to modify the code to search a byte array instead of a char array). If your file is MBCS or UTF-8, you're likely best off just sticking with the .NET string classes.
-cd
Greg Young - 10 Oct 2006 19:41 GMT As CD has said this is expected as strings are UTF-16 .. my question would be how you are searching this file.
Are you just doing keyword searches? Depending on the type of search you might be much better off doing something like building an index of the file and loading the index into memory.
Cheers,
Greg
> My program needs to search a large textfile (>60MB). > [quoted text clipped - 9 lines] > TIA, > Jurjen. Jurjen de Groot - 11 Oct 2006 07:58 GMT Greg,
The text-file consists of records of 80 characters seperated by a NewLine. These records all have a record type 1 thru 9, a set of records start with record 1 and end with record 9 at wich point the next set will start with record type 1.
I search the contents of the file for the search criteria as entered by the user f.i. 2742281, when I find this sequence I have to make sure it's found in exactly the right position within the record to make sure I have compared it to the right field. Then I have to show this record found (wich should be record type 1) and show all records until I find recordtype 9 (of EOF).
I could create an index but that would complicate the app, I also thought of maybe creating a datatable to ease the search but I'm pretty sure memory consumption would be even worse...
I was just wondering why the current app is consuming so much memory wich is now clear to me. I guess my client will have to make the decision, cheap app wich will use much memory, little more expensive app using less memory.
Regards, Jurjen.
> As CD has said this is expected as strings are UTF-16 .. my question would > be how you are searching this file. [quoted text clipped - 20 lines] >> TIA, >> Jurjen. Morten Wennevik - 11 Oct 2006 08:39 GMT Hi Jurjen,
Sounds to me like you could just use a ReadLine() and do a search per record. You should use the encoding used in the file. If you don't specify an encoding, UTF-8 is used. You would need some logic added to keep track of an entire recordset, which can be a string[] of length 9
StreamReader sr = new StreamReader("", Encoding.Default); string s = null; string[] recordset = new string[9]; int index = 0;
while ((s = sr.ReadLine()) != null) { int i = GetRecordNumber(s);
if (i >= (index + 1)) ;// missing record
recordset[index] = s;
index++;
if (i == 9) // complete record { if (SearchRecordSet(recordset)) return true;
Array.Clear(recordset, 0, 9); index = 0; } }
PS! Your system clock is a bit too fast
> Greg, > [quoted text clipped - 55 lines] >>> TIA, >>> Jurjen.
 Signature Happy Coding! Morten Wennevik [C# MVP]
Jurjen de Groot - 11 Oct 2006 13:31 GMT Morten,
Thanks for your reply, I understand what you're doing in the code, but isn't reading line by line slow ? The file is over 64mb in size, reading it line by line to do a search seems like a lot of overhead, especially when the user does many searches while running the app, it would mean reading/searching the >64mb file many times, that's why I opted to keep the file in memory which might not be the best idea. I'm currently trying to get some more time from my client to try and optimize by creating an index of the file (which doesn't change that often) and searching through that and retrieving part of the text-file corresponding to the index...
Jurjen.
Hi Jurjen,
Sounds to me like you could just use a ReadLine() and do a search per record. You should use the encoding used in the file. If you don't specify an encoding, UTF-8 is used. You would need some logic added to keep track of an entire recordset, which can be a string[] of length 9
StreamReader sr = new StreamReader("", Encoding.Default); string s = null; string[] recordset = new string[9]; int index = 0;
while ((s = sr.ReadLine()) != null) { int i = GetRecordNumber(s);
if (i >= (index + 1)) ;// missing record
recordset[index] = s;
index++;
if (i == 9) // complete record { if (SearchRecordSet(recordset)) return true;
Array.Clear(recordset, 0, 9); index = 0; } }
PS! Your system clock is a bit too fast
On Wed, 11 Oct 2006 09:01:00 +0200, Jurjen de Groot <info@gits-online.nl> wrote:
> Greg, > [quoted text clipped - 55 lines] >>> TIA, >>> Jurjen.
 Signature Happy Coding! Morten Wennevik [C# MVP]
Morten Wennevik - 11 Oct 2006 13:38 GMT I haven't done speed tests, but you may well find the speed taken to locate a recordset by reading the file line by line is not much considering the processing power of todays computers.
 Signature Happy Coding! Morten Wennevik [C# MVP]
Morten Wennevik - 11 Oct 2006 14:14 GMT I created a ~80mb text file consisting of 80 characters per line and a known word on the second last line
Opening and searching line by line took ~2.4 seconds each time. Opening and reading the entire file before a search took ~6.4 seconds, with ~0.4 seconds for each search.
Added complexity to the processing code will have less impact on the first method in percentages, considering you then already have a complete recordset where in the second method you would need additional searchs.
 Signature Happy Coding! Morten Wennevik [C# MVP]
Morten Wennevik - 11 Oct 2006 14:33 GMT Given time, the fastest option is probably using a FileStream and a search algorithm like Boyer-Moore, but the complexity of the code would also increase accordingly.
 Signature Happy Coding! Morten Wennevik [C# MVP]
Carl Daniel [VC++ MVP] - 11 Oct 2006 16:14 GMT > Given time, the fastest option is probably using a FileStream and a search > algorithm like Boyer-Moore, but the complexity of the code would also > increase accordingly. See my earlier reply to the original post. That's exactly what I do in a program that reads similar fixed-format text files: Read the entire file into a byte array via a single call to FileStream.Read and then use a Boyer-Moore search on that. It's about 5x faster than reading the entire file into a string and using string.IndexOf. There's a link to the Boyer-Moore implementation in my earlier post - the version in the article works on strings, but it's straightforward to convert it to work on byte arrays.
I found the byte array search using BM to be about 2x faster than the "Find" function in Visual Studio (which is quite fast), but about 2x slower than the "Find" function in Notepad - which is also Boyer-Moore (well, QuickSearch, actually), but is written in C instead of C#. The array bounds checking on access to the supplemental arrays used by BM really hurt the performance - but it's still quite speedy.
-cd
Morten Wennevik - 12 Oct 2006 07:54 GMT Ah, sorry, didn't catch the last part of your first post.
 Signature Happy Coding! Morten Wennevik [C# MVP]
Nick Malik [Microsoft] - 11 Oct 2006 18:54 GMT Hi Jurjen,
my napkin calculations show that you have a 750,000 record table that doesn't change often and that you need to search many times by each user.
Dude... is SQL Server really so bad of an option? c'mon! Why write this capability into your app when it is available to you for free?
 Signature --- Nick Malik [Microsoft] MCSD, CFPS, Certified Scrummaster http://blogs.msdn.com/nickmalik
Disclaimer: Opinions expressed in this forum are my own, and not representative of my employer. I do not answer questions on behalf of my employer. I'm just a programmer helping programmers. --
> Morten, > [quoted text clipped - 109 lines] >>>> TIA, >>>> Jurjen. Jurjen de Groot - 12 Oct 2006 16:09 GMT Nick,
SQL isn't an option as this is supposed to be a very simple (+/- 2 hour) solution to a problem. The programm will most probably only be used for a week or so. Doing SQL would be overkill in this specific situation.
I would like to thank everyone for their replies.
Jurjen.
> Hi Jurjen, > [quoted text clipped - 121 lines] >>>>> TIA, >>>>> Jurjen.
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|