Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / New Users / July 2005

Tip: Looking for answers? Try searching our database.

Successive processing of URLs with HttpWebRequest

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Yavor Stoev - 22 Jul 2005 09:35 GMT
I have a windows service which downloads .html pages by generating URLs and
passing them to HttpWebRequest variable and after that calling the
GetResponse() method. The requests are processed in a for cycle.
Assuming the time for write operations on my Harddisk is negligible am I
using the whole capacity of my internet connection, or is there a way to
improve the speed of successive download ? So far, on my computer the service
seems to use the whole capacity(around 24 KB/s), but when I start it on other
machines (with 2MB/s connection) the results are much better when there are
two(strangely not three or more) threads or services generating the requests.

Thanks in advance

Yavor Stoev
Nick Malik [Microsoft] - 24 Jul 2005 20:57 GMT
The HTTP protocol is not that efficient.  It sits on top of TCP/IP which has
a good bit of effort involved with acknowledging the receipt of packets.  In
addition, HTTP itself has some work to do at the protocol level, before your
app even takes over.  Therefore, assuming your app is extremely efficient,
the retrieval of a page will not be.  You are also gated by the efficiency
of the system serving the pages.  This is why IE will spawn as many as 20
threads to retrieve all of the resources indicated in a web page, and will
begin before it even completes the download of the current page.

If you are running out at two threads, make sure that you aren't in a
situation where two threads are competing for the same resources.

Signature

--- Nick Malik [Microsoft]
   MCSD, CFPS, Certified Scrummaster
   http://blogs.msdn.com/nickmalik

Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
  I do not answer questions on behalf of my employer.  I'm just a
programmer helping programmers.
--

>I have a windows service which downloads .html pages by generating URLs and
> passing them to HttpWebRequest variable and after that calling the
[quoted text clipped - 13 lines]
>
> Yavor Stoev

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.