Thread pool and HTTPRequest

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Brendan - 05 Jul 2004 14:25 GMT

Hi,

I'm trying to use the .Net httpWebRequest objects to make asynch calls. I'm creating a crawler and I'm hoping to increase performance. Initially I was running into thread pooling issues but was able to find a way to increase the default 25 threads. This has partially solved my problem.

Now, my problem is that I get better performance going through a proxy server than a direct connection. The behaviour I'm seeing is that the proxy allows the requests to be queued much quicker than if I'm running locally. I'd like to be able to not use the proxy server but haven't been able to get performance near that of going through a proxy. Any help in suggesting a way of increasing request/response time outside of a proxy would be greatly appreciated! :)

Thanks,
Brendan

Reply to this Message

Feroze [msft] - 16 Jul 2004 20:09 GMT

What do you mean by "performance" ? Is it that the pages are being
downloaded faster than if you hit the server directly ?

IS the server you are accessing on the same network as the client ?

Are you downloading dynamic contnet (as opposed to static content) ?

It might be that the proxy server is serving the pages from it's cache, that
is why it is faster.

feroze
=====================
this posting is provided as is. It offers no warranties and confers no
rights.
=====================

> Hi,
>
> I'm trying to use the .Net httpWebRequest objects to make asynch calls. I'm creating a crawler and I'm hoping to increase performance. Initially I

was running into thread pooling issues but was able to find a way to
increase the default 25 threads. This has partially solved my problem.

> Now, my problem is that I get better performance going through a proxy server than a direct connection. The behaviour I'm seeing is that the proxy

allows the requests to be queued much quicker than if I'm running locally.
I'd like to be able to not use the proxy server but haven't been able to get
performance near that of going through a proxy. Any help in suggesting a
way of increasing request/response time outside of a proxy would be greatly
appreciated! :)

> Thanks,
> Brendan

Reply to this Message

Justin Rogers - 16 Jul 2004 20:31 GMT

To add, have you looked at ServicePoint's and the possibility that you are
making a maximum of two connections to each server at a time without
raising the limit? The proxy probably shuns the 2 connection limit while
the ServicePoint classes used by the HttpWebRequest does not. Try raising
your limit a bit and see if that helps. I ran into a similar problem making a
multi-part download application.

Signature

Justin Rogers
DigiTec Web Consultants, LLC.
Blog: http://weblogs.asp.net/justin_rogers

> What do you mean by "performance" ? Is it that the pages are being
> downloaded faster than if you hit the server directly ?
[quoted text clipped - 29 lines]
> > Thanks,
> > Brendan

Reply to this Message

Thread pool and HTTPRequest

Free Magazines