Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / New Users / November 2006

Tip: Looking for answers? Try searching our database.

How to speed up reading of large disk files

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Valerie Hough - 20 Nov 2006 17:16 GMT
Sorry if this is the wrong news group for this post.

I have some large disk files (15-20 MB) that are made up of many records of
different types and sizes. I use a BinaryReader to process all the data
sequentially (i.e. with many reads operations) and load it into my .NET app.

I will have several .NET Apps running on the same computer each of which
will need to read these files but never change them.

The performance is not awful, but I would like it to be faster. I have
looked at Memory File mapping, RAM disks, etc. but I would like to know if
someone has a better idea especially if it stays within normal .NET I/O
funcationality.

Thanks in advance for your help.
Valerie Hough
Peter Duniho - 20 Nov 2006 18:19 GMT
> [...]
> The performance is not awful, but I would like it to be faster. I have
> looked at Memory File mapping, RAM disks, etc. but I would like to know if
> someone has a better idea especially if it stays within normal .NET I/O
> funcationality.

The easiest thing you can do to speed up disk i/o is to read larger portions
of the file at a time.  With .NET, you either do this explicitly with large
reads, or you can explicitly set the FileStream to use buffered i/o when you
open the file, and specify a large buffer size.

I know it's not a very sophisticated technique, but I've had good success
with it.  I try to use buffers that are multiples of the default page size
(4096 bytes), and have found through experimentation that buffers larger
than two or three pages reach a point of diminishing returns.  At that
point, to get better performance would probably require a lot more work on
my part.  :)

The other thing you can do is try to make sure that you can operate on the
data directly from the buffers used to read it.  If you stick another copy
operation of the data in your code, that will slow you down, especially if
you're dealing with a lot of data.

My apologies if the above suggestions seem obvious and not useful.  :)

Pete

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.