HttpWebRequest and Multi Threaded apps Question

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Mark Fletcher - 10 Aug 2005 22:25 GMT

Hi,

Im in the process of writing a program that crawls a website. Im using the
HttpWebRequest and HttpWebResponse classed to get content. To make my
application more scalable, my application is multithreaded, with each thread
making a different request.

One problem Ive ran into is that when writing the response stream to the
filesystem, from the HttpWebResponse.GetResponseStream() method, if the
response is of a sufficient size, it blocks all other other executing
threads. If this exceeds the timeout, then all the other threads that are in
the process of making a Web Request time out.

Ive taken every effort to ensure that my application is not locking, I was
wondering if given the scenario that Ive listed, would using Asynchronous Web
Requests be more efficient? Ive not worked with them before so Id be grateful
of your opinions.

Cheers,

Mark

Reply to this Message

rviray - 10 Aug 2005 23:35 GMT

if the application is multithreaded, how would one thread stop or bloc
execution of another thread?

As for your original question about Asynchronous calls, it might hel
you on process time, if you batch up your httprequest - say 10 URL
that you crawl and begin/end invoke the request asynchronously.
Basically you spawn multiple threads and on those threads yo
asynchronoulsy crawl x amount of sites.

But, I would concentrate on your threading methods first because i
your application is truely multi-threaded (spawn worker threads) URL
that need to be crawled are stopped in there tracks, then you
application is not threading?

I may not understand exactly what you are doing and how your cod
works, so if you include some snippets, I would be glad to hel
further.

Thanks,

Ralp

--
rvira

Reply to this Message

Joerg Jooss - 12 Aug 2005 11:48 GMT

> Hi,
>
[quoted text clipped - 13 lines]
> Asynchronous Web Requests be more efficient? Ive not worked with them
> before so Id be grateful of your opinions.

I agree with Ralph. Without being able to look at your code, this seems
pretty bizarre.

One thing you might have run into is the connection limit of persistent
or non-persistent connections a ServicePoint allows to a single host.
This will block your threads due to the limited number of available
physical TCP connections.

Cheers,

Signature

http://www.joergjooss.de
mailto:news-reply@joergjooss.de

Reply to this Message

Mark Fletcher - 17 Aug 2005 16:39 GMT

Hi Joerg,

Thanks for the response. Im wondering if indeed my problem is ServicePoint
related. Do you know of any good resources where I can go and learn about
this? In my scenario each active thread makes one Http Request and gets the
response.

Ive looked into my problem further and I do find that I have occasional
problems, when reading the response stream. I find that when calling
Stream.Read() this sometimes times out with the following exception:

Message: The operation has timed out
Stack Trace: at System.ConnectStream.Read(...)

Now it doesnt matter if I set HttpRequest.Timeout = 10000 or 100000, I still
occasionally get these timeouts. When I debug the application its like the
thread is in the process of reading bytes part of the way through and then
hangs.

I can provide code if requested, but I was wondering if anyone else has seen
symptoms like this, and if they could suggest a remedy.

Thanks,

Mark

> > Hi,
> >
[quoted text clipped - 23 lines]
>
> Cheers,

Reply to this Message

Joerg Jooss - 17 Aug 2005 20:00 GMT

> Hi Joerg,
>
> Thanks for the response. Im wondering if indeed my problem is
> ServicePoint related. Do you know of any good resources where I can
> go and learn about this? In my scenario each active thread makes one
> Http Request and gets the response.

Mark, ServicePoints are unfortunately one of the more arcane classes in
the BCL. I'm not even aware of any MSDN article discussing them.

You'll find some information in "Network Programming for the .NET
Framework" (MS Press), but other than that the class documentation is
your best friend.

Note: The connection limit imposed by ServicePoint(Manager) has a
reason -- it is specified in RFC 2616 (HTTP 1.1).

> Ive looked into my problem further and I do find that I have
> occasional problems, when reading the response stream. I find that
[quoted text clipped - 8 lines]
> its like the thread is in the process of reading bytes part of the
> way through and then hangs.

Note that the Timeout property is only relevant for pending responses,
not responses that are already being received. Faulty HTTP
implementations can cause a lot of problems here (e.g. wrong
Content-Length, or broken chunking).

If you need a timeout for "in-flight" responses, use async I/O with
ThreadPool.WaitForSingleObject() to register a timeout for the async
operation.

Cheers,

Signature

http://www.joergjooss.de
mailto:news-reply@joergjooss.de

Reply to this Message

shriop - 22 Aug 2005 04:32 GMT

My guess is that you're making a request to an HTTPS url. There's a
default limit of 2 simultaneous https requests to the same domain built
into the framework. Just put this line of code into your code after
you've created the HttpWebRequest object,

request.ServicePoint.ConnectionLimit = 50;

Bruce Dunwiddie
http://www.csvreader.com

> > Hi Joerg,
> >
[quoted text clipped - 36 lines]
>
> Cheers,

Reply to this Message

Joerg Jooss - 22 Aug 2005 20:01 GMT

> My guess is that you're making a request to an HTTPS url. There's a
> default limit of 2 simultaneous https requests to the same domain
> built into the framework. Just put this line of code into your code
> after you've created the HttpWebRequest object,
>
> request.ServicePoint.ConnectionLimit = 50;

Ouch... http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.4

Cheers,

Signature

http://www.joergjooss.de
mailto:news-reply@joergjooss.de

Reply to this Message

HttpWebRequest and Multi Threaded apps Question

Free Magazines