Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / New Users / October 2006

Tip: Looking for answers? Try searching our database.

Large text file - in memory ( > 60mb)

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Jurjen de Groot - 10 Oct 2006 17:24 GMT
My program needs to search a large textfile (>60MB).

At this time I'm using a streamreader to read the file into a
string-variable (objString = sr.ReadToEnd). Before reading the file the
proces running my programm uses about 10mb, after reading the text-file into
the string, it uses over 200mb. I would expect the program to use between 70
and 100mb.

Is there a more efficient way of storing this data in-memory and still be
able to search through it ... ?

TIA,
Jurjen.
Carl Daniel [VC++ MVP] - 10 Oct 2006 18:06 GMT
> My program needs to search a large textfile (>60MB).
>
[quoted text clipped - 6 lines]
> Is there a more efficient way of storing this data in-memory and still be
> able to search through it ... ?

Keep in mind that .NET strings are UTF-16, so reading an ANSI text file will
typically double the size in bytes.

If you read the file into a byte array, you'll use less memory, but you
won't be able to use the .NET string searching facilities (e.g.
System.String member functions, regular expressions, etc).

Depending on your searching requirements, you might be able to use a simpler
search facility, such as the one in this article:

http://www.codeproject.com/cs/algorithms/BoyerMooreSearch.asp

(but you'd have to modify the code to search a byte array instead of a char
array).  If your file is MBCS or UTF-8, you're likely best off just sticking
with the .NET string classes.

-cd
Greg Young - 10 Oct 2006 19:41 GMT
As CD has said this is expected as strings are UTF-16 .. my question would
be how you are searching this file.

Are you just doing keyword searches? Depending on the type of search you
might be much better off doing something like building an index of the file
and loading the index into memory.

Cheers,

Greg

> My program needs to search a large textfile (>60MB).
>
[quoted text clipped - 9 lines]
> TIA,
> Jurjen.
Jurjen de Groot - 11 Oct 2006 07:58 GMT
Greg,

The text-file consists of records of 80 characters seperated by a NewLine.
These records all have a record type 1 thru 9,  a set of records start with
record 1 and end with record 9 at wich point the next set will start with
record type 1.

I search the contents of the file for the search criteria as entered by the
user f.i. 2742281, when I find this sequence I have to make sure it's found
in exactly the right position within the record to make sure I have compared
it to the right field. Then I have to show this record found (wich should be
record type 1) and show all records until I find recordtype 9 (of EOF).

I could create an index but that would complicate the app, I also thought of
maybe creating a datatable to ease the search but I'm pretty sure memory
consumption would be even worse...

I was just wondering why the current app is consuming so much memory wich is
now clear to me. I guess my client will have to make the decision, cheap app
wich will use much memory, little more expensive app using less memory.

Regards,
Jurjen.

> As CD has said this is expected as strings are UTF-16 .. my question would
> be how you are searching this file.
[quoted text clipped - 20 lines]
>> TIA,
>> Jurjen.
Morten Wennevik - 11 Oct 2006 08:39 GMT
Hi Jurjen,

Sounds to me like you could just use a ReadLine() and do a search per  
record.  You should use the encoding used in the file.  If you don't  
specify an encoding, UTF-8 is used.  You would need some logic added to  
keep track of an entire recordset, which can be a string[] of length 9

            StreamReader sr = new StreamReader("", Encoding.Default);
            string s = null;
            string[] recordset = new string[9];
            int index = 0;

            while ((s = sr.ReadLine()) != null)
            {
                int i = GetRecordNumber(s);

                if (i >= (index + 1))
                    ;// missing record

                recordset[index] = s;

                index++;

                if (i == 9) // complete record
                {
                    if (SearchRecordSet(recordset))
                        return true;

                    Array.Clear(recordset, 0, 9);
                    index = 0;
                }
            }

PS! Your system clock is a bit too fast

> Greg,
>
[quoted text clipped - 55 lines]
>>> TIA,
>>> Jurjen.

Signature

Happy Coding!
Morten Wennevik [C# MVP]

Jurjen de Groot - 11 Oct 2006 13:31 GMT
Morten,

Thanks for your reply, I understand what you're doing in the code, but isn't
reading line by line slow ?
The file is over 64mb in size, reading it line by line to do a search seems
like a lot of overhead, especially when the user does many searches while
running the app, it would mean reading/searching the >64mb file many times,
that's why I opted to keep the file in memory which might not be the best
idea.
I'm currently trying to get some more time from my client to try and
optimize by creating an index of the file (which doesn't change that often)
and searching through that and retrieving part of the text-file
corresponding to the index...

Jurjen.

Hi Jurjen,

Sounds to me like you could just use a ReadLine() and do a search per
record.  You should use the encoding used in the file.  If you don't
specify an encoding, UTF-8 is used.  You would need some logic added to
keep track of an entire recordset, which can be a string[] of length 9

            StreamReader sr = new StreamReader("", Encoding.Default);
            string s = null;
            string[] recordset = new string[9];
            int index = 0;

            while ((s = sr.ReadLine()) != null)
            {
                int i = GetRecordNumber(s);

                if (i >= (index + 1))
                    ;// missing record

                recordset[index] = s;

                index++;

                if (i == 9) // complete record
                {
                    if (SearchRecordSet(recordset))
                        return true;

                    Array.Clear(recordset, 0, 9);
                    index = 0;
                }
            }

PS! Your system clock is a bit too fast

On Wed, 11 Oct 2006 09:01:00 +0200, Jurjen de Groot <info@gits-online.nl>
wrote:

> Greg,
>
[quoted text clipped - 55 lines]
>>> TIA,
>>> Jurjen.

Signature

Happy Coding!
Morten Wennevik [C# MVP]

Morten Wennevik - 11 Oct 2006 13:38 GMT
I haven't done speed tests, but you may well find the speed taken to  
locate a recordset by reading the file line by line is not much  
considering the processing power of todays computers.

Signature

Happy Coding!
Morten Wennevik [C# MVP]

Morten Wennevik - 11 Oct 2006 14:14 GMT
I created a ~80mb text file consisting of 80 characters per line and a  
known word on the second last line

Opening and searching line by line took ~2.4 seconds each time.
Opening and reading the entire file before a search took ~6.4 seconds,  
with ~0.4 seconds for each search.

Added complexity to the processing code will have less impact on the first  
method in percentages, considering you then already have a complete  
recordset where in the second method you would need additional searchs.

Signature

Happy Coding!
Morten Wennevik [C# MVP]

Morten Wennevik - 11 Oct 2006 14:33 GMT
Given time, the fastest option is probably using a FileStream and a search  
algorithm like Boyer-Moore, but the complexity of the code would also  
increase accordingly.

Signature

Happy Coding!
Morten Wennevik [C# MVP]

Carl Daniel [VC++ MVP] - 11 Oct 2006 16:14 GMT
> Given time, the fastest option is probably using a FileStream and a search
> algorithm like Boyer-Moore, but the complexity of the code would also
> increase accordingly.

See my earlier reply to the original post.  That's exactly what I do in a
program that reads similar fixed-format text files:  Read the entire file
into a byte array via a single call to FileStream.Read and then use a
Boyer-Moore search on that.  It's about 5x faster than reading the entire
file into a string and using string.IndexOf.  There's a link to the
Boyer-Moore implementation in my earlier post - the version in the article
works on strings, but it's straightforward to convert it to work on byte
arrays.

I found the byte array search using BM to be about 2x faster than the "Find"
function in Visual Studio (which is quite fast), but about 2x slower than
the "Find" function in Notepad - which is also Boyer-Moore (well,
QuickSearch, actually), but is written in C instead of C#.  The array bounds
checking on access to the supplemental arrays used by BM really hurt the
performance - but it's still quite speedy.

-cd
Morten Wennevik - 12 Oct 2006 07:54 GMT
Ah, sorry, didn't catch the last part of your first post.

Signature

Happy Coding!
Morten Wennevik [C# MVP]

Nick Malik [Microsoft] - 11 Oct 2006 18:54 GMT
Hi Jurjen,

my napkin calculations show that you have a 750,000 record table that
doesn't change often and that you need to search many times by each user.

Dude... is SQL Server really so bad of an option?  c'mon!  Why write this
capability into your app when it is available to you for free?

Signature

--- Nick Malik [Microsoft]
   MCSD, CFPS, Certified Scrummaster
   http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
  I do not answer questions on behalf of my employer.  I'm just a
programmer helping programmers.
--

> Morten,
>
[quoted text clipped - 109 lines]
>>>> TIA,
>>>> Jurjen.
Jurjen de Groot - 12 Oct 2006 16:09 GMT
Nick,

SQL isn't an option as this is supposed to be a very simple (+/- 2 hour)
solution to a problem. The programm will most probably only be used for a
week or so. Doing SQL would be overkill in this specific situation.

I would like to thank everyone for their replies.

Jurjen.

> Hi Jurjen,
>
[quoted text clipped - 121 lines]
>>>>> TIA,
>>>>> Jurjen.

Rate this thread:







Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.