Hi,
I am trying to take some latency measurement of HTTP download times for a
number of zip files varying in size from 100K to 500M. I wrote some code
shown below but it produces out of memory errors when the files get up to
about 200M. I am not sure if the code I have produced is what I want because
I used a streamreader that requires an encoding type to be specified, as its
a binary file type I don't think I should need to worry about encoding only
bits and bytes. Can you make any suggestions on what would be the best way
acheive this please.
'setup local file to write to
Dim fs As New FileStream(LocalFilePath, FileMode.CreateNew,
FileAccess.Write, FileShare.None)
' Creates an HttpWebRequest for the specified URL.
Dim myHttpWebRequest As HttpWebRequest = CType(WebRequest.Create
(WebPageURL), HttpWebRequest)
' Sends the request and waits for a response.
Dim myHttpWebResponse As HttpWebResponse = CType
(myHttpWebRequest.GetResponse(), HttpWebResponse)
' Calls the method GetResponseStream to return the stream associated with
the response.
Dim receiveStream As Stream = myHttpWebResponse.GetResponseStream()
' Pipes the response stream to a higher level stream reader with the
required encoding format.
Dim readStream As New StreamReader(receiveStream, encode)
pfileWriter.Write(readStream.Read())
Thanks
Joerg Jooss - 03 Nov 2006 21:27 GMT
Thus wrote smarty,
> Hi,
>
[quoted text clipped - 5 lines]
> type to be specified, as its a binary file type I don't think I should
> need to worry about encoding only bits and bytes.
[...]
You *must not* use a StreamReader at all. It will only destroy your binary
payload, test or not ;-)
Worse, all of the broken binary content will end up as a string object, which
becomes garbage immediately.
You should instead consume the response stream using a small read buffer
(4-16 kB), and simply drop all bytes read.
// request is your HttpWebRequest
using(HttpWebResponse response = (HttpWebResponse) request.GetResponse())
using(Stream responseStream = response.GetResponseStream()) {
byte[] buffer = new byte[0x1000];
int bytes;
while((bytes = responseStream.Read(buffer, 0, buffer.Length)) > 0) {
// Do nothing ;-)
}
}
Cheers,

Signature
Joerg Jooss
news-reply@joergjooss.de
Laura T - 05 Nov 2006 17:57 GMT
Any reason to not use WebClient.DownloadData/DownloadDataAsync,
or even better, WebClient.DownloadFile/DownloadFileAsync?
You'd get much better performance using those. And no memory problems.
> Hi,
>
[quoted text clipped - 34 lines]
>
> Thanks
Joerg Jooss - 05 Nov 2006 18:16 GMT
Thus wrote Laura,
> Any reason to not use WebClient.DownloadData/DownloadDataAsync, or
> even better, WebClient.DownloadFile/DownloadFileAsync?
>
> You'd get much better performance using those. And no memory problems.
Better performance where? It won't improve the HTTP throughput.
And why no memory problems? That has nothing to do with async I/O.

Signature
Joerg Jooss
news-reply@joergjooss.de
smarty - 05 Nov 2006 22:15 GMT
Hi,
Thanks to you both for your response. I did use the WebClient class in the
end because I assumed this would have the most most efficient buffering
worked out automatically rather guessing what buffer size would be efficient.
Thanks for showing me how to setup the buffer code though I will find this
useful in the future and may still use it yet.
> Thus wrote Laura,
>
[quoted text clipped - 6 lines]
>
> And why no memory problems? That has nothing to do with async I/O.