Loading large files

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Gregor Wind - 16 Jun 2004 12:40 GMT

Hello all,

I'm trying to make a 3D transformation with image data. For the 3rd 1 D
transformation I have to jump in a 12Gbyte file, to get the data for the
transformation. The jumpimg in the file cost a lot of time. I am doing the
filehandling with help of the filestream and binaryreader class in C#. Is
there an other (faster) way existing, to read the data out of the file?

Thanks in advance,

regards

Gregor

Reply to this Message

Jon Skeet [C# MVP] - 16 Jun 2004 12:49 GMT

> I'm trying to make a 3D transformation with image data. For the 3rd 1 D
> transformation I have to jump in a 12Gbyte file, to get the data for the
> transformation. The jumpimg in the file cost a lot of time. I am doing the
> filehandling with help of the filestream and binaryreader class in C#. Is
> there an other (faster) way existing, to read the data out of the file?

Do you have memory requirements which prevent you from loading the
whole file into memory?

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Reply to this Message

Jerry Pisk - 16 Jun 2004 15:49 GMT

I would say the 4GB addressable space limit on a 32-bit system might be a
problem, especially when coupled with Windows' limit of 2GB for each
application and the OS (or 3/1 split).

Jerry

>> I'm trying to make a 3D transformation with image data. For the 3rd 1 D
>> transformation I have to jump in a 12Gbyte file, to get the data for the
[quoted text clipped - 5 lines]
> Do you have memory requirements which prevent you from loading the
> whole file into memory?

Reply to this Message

Jon Skeet [C# MVP] - 16 Jun 2004 16:53 GMT

> I would say the 4GB addressable space limit on a 32-bit system might be a
> problem, especially when coupled with Windows' limit of 2GB for each
> application and the OS (or 3/1 split).

Yup, I'd misread 12Gbyte as 12Mbyte. Oops :(

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Reply to this Message

Yuancai \(Charlie\) Ye - 16 Jun 2004 17:52 GMT

Hi, Gregor:
try window file service inside SocketPro at www.udaparts.com. Note that
both client and server source codes are available to you. It is not written
from C# but from C++. Also there is a sample project MoveFile inside. If
your machine hard disk is not slow and connected with a 100 Mbps LAN, it
will need roughly 1000 seconds to transfer 12 Gbytes data.

Signature

Yuancai (Charlie) Ye

Fast and securely accessing all of remote data sources anywhere with
SocketPro using batch/queue, asynchrony and parallel computation with online
compressing

See 30 well-tested and real OLEDB examples

www.udaparts.com

> Hello all,
>
[quoted text clipped - 9 lines]
>
> Gregor

Reply to this Message

Jon Skeet [C# MVP] - 16 Jun 2004 17:55 GMT

Yuancai (Charlie) Ye <someone@yahoo.com> wrote:

> try window file service inside SocketPro at www.udaparts.com. Note that
> both client and server source codes are available to you. It is not written
> from C# but from C++. Also there is a sample project MoveFile inside. If
> your machine hard disk is not slow and connected with a 100 Mbps LAN, it
> will need roughly 1000 seconds to transfer 12 Gbytes data.

Who said anything about transferring a file? He's just reading the
file, as far as I can see. No need for SocketPro at all, to my mind.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Reply to this Message

Fred Aston - 20 Jun 2004 15:03 GMT

Have you considered using memory-mapped files?

There's no support in the framework for this, but writing
a wrapper class for your special need is probably not
that difficult.

See CreateFileMapping() in MSDN.

Reply to this Message

Sriram Krishnan - 17 Jul 2004 15:11 GMT

Actually, I remember seeing a managed wrapper for this somewhere in the blog
world. Try Googling for it

Signature

Sriram Krishnan
Microsoft Student Ambassador
http://www.dotnetjunkies.com/weblog/sriram

> Have you considered using memory-mapped files?
>
[quoted text clipped - 3 lines]
>
> See CreateFileMapping() in MSDN.

Reply to this Message

Loading large files

Free Magazines