I'm writing a C# app which downloads large files via http from a server. These files are often in excess of a gigabyte.

I am trying to figure out the best way to download these kinds of large files in .NET. While I've successefully downloaded test
files using WebClient, I don't think that's going to be a viable final solution because I need to be able process the data stream
before writing it to the local disk (basically, I need to strip off certain bytes). I haven't found a WebClient method which allows
me to "intercept" the downloaded data stream to process it.

I've played around with WebRequest and WebResponse, but am running into a situation where no more than the first 256K bytes of data
is readable. In other words, if I do something like this (pseudo code):

WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();

while( not enough bytes read yet )
{
    stream.Read(buffer...)
    localFileStream.Write(buffer...)
}

only the first 256K bytes are read from the stream. After that, the stream.Read() has a return value of 0.

So...what advice do folks have on how I should move forward?

- Mark