Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / Performance / January 2005

Tip: Looking for answers? Try searching our database.

WebResponse StreamReader memory and handles high consume

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Hugo - 20 Dec 2004 09:46 GMT
Hello, I'm running the following code that i found on msdn documentation of
GetResponseStream method. If i try to transfer a big file using this code
and during that operation take a look to task manager i see that process
memory usage and handles keep increasing without stoping. The line of code
behind this
is count = readStream.Read(read, 0, 256);.
Does any body can help me?
Thanks in advance.

// Create a 'WebRequest' object with the specified url.
WebRequest myWebRequest = WebRequest.Create("http://www.constoso.com");

// Send the 'WebRequest' and wait for response.
WebResponse myWebResponse = myWebRequest.GetResponse();

// Obtain a 'Stream' object associated with the response object.
Stream ReceiveStream = myWebResponse.GetResponseStream();
               
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");

// Pipe the stream to a higher level stream reader with the required
encoding format.
StreamReader readStream = new StreamReader( ReceiveStream, encode );
Console.WriteLine("\nResponse stream received");
Char[] read = new Char[256];

// Read 256 charcters at a time.    
int count = readStream.Read( read, 0, 256 );
Console.WriteLine("HTML...\r\n");

   while (count > 0)
   {
       // BEGIN : THE IMPORTANT ZONE  
       

    /* Coment to avoid any other consume inside "while" loop */
    //String str = new String(read, 0, count);
       //Console.Write(str);
       
   
    /* BEGIN : CONSUME ZONE */
    count = readStream.Read(read, 0, 256);
    /* END : CONSUME ZONE */

    // END : THE IMPORTANT ZONE  
       

   }

      Console.WriteLine("");
    // Release the resources of stream object.
    readStream.Close();

    // Release the resources of response object.
Joerg Jooss - 21 Dec 2004 08:44 GMT
> Hello, I'm running the following code that i found on msdn
> documentation of GetResponseStream method. If i try to transfer a big
[quoted text clipped - 46 lines]
>     // Release the resources of stream object.
>     readStream.Close();

I wouldn't use this decode while download scheme -- just read raw bytes from
the response stream and decode the entire byte array once the download is
finished.

Cheers,

Signature

Joerg Jooss
www.joergjooss.de
news@joergjooss.de

Jon Skeet [C# MVP] - 21 Dec 2004 08:51 GMT
> I wouldn't use this decode while download scheme -- just read raw bytes from
> the response stream and decode the entire byte array once the download is
> finished.

Why, out of interest? It seems pretty wasteful to me - why have the
whole thing in memory twice, when you can decode as you go and only end
up having one buffer + the whole thing in memory at the worst point?

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Joerg Jooss - 21 Dec 2004 10:38 GMT
>> I wouldn't use this decode while download scheme -- just read raw
>> bytes from the response stream and decode the entire byte array once
[quoted text clipped - 4 lines]
> end up having one buffer + the whole thing in memory at the worst
> point?

Of course you're right -- all this hardly matters if you need to extract
some text from a web site you perfectly know.

But if not?

1. Raw byte streaming always works. There are just too many posts where devs
happily trash downloaded binaries by using TextReaders ;-)

2. If downloading also means writing the response to a file, the whole
decoding process becomes moot. Actually, the OP didn't really state this
clearly and probably just wants a text-based dump.

3. Decoding on-the-fly isn't always possible simply because you don't know
the character encoding to use.

Here's what Google does:

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html
Set-Cookie:
PREF=ID=1b94198c0aae78a7:LD=de:TM=1103624086:LM=1103624086:S=sIQVyI26QS2mQmjx;
expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.de
Server: GWS/2.1
Transfer-Encoding: chunked
Date: Tue, 21 Dec 2004 10:14:46 GMT
<html><head><meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1"><title>Google</title><style><!--

Ouch! I'll have to parse the HTML to get the character encoding...

Cheers,

Signature

Joerg Jooss
www.joergjooss.de
news@joergjooss.de

Jon Skeet [C# MVP] - 21 Dec 2004 11:42 GMT
> >> I wouldn't use this decode while download scheme -- just read raw
> >> bytes from the response stream and decode the entire byte array once
[quoted text clipped - 12 lines]
> 1. Raw byte streaming always works. There are just too many posts where devs
> happily trash downloaded binaries by using TextReaders ;-)

That's certainly true if you definitely want to want to treat the file
as binary. If you're going to treat it as text anyway, however, it
doesn't matter when you do that step.

> 2. If downloading also means writing the response to a file, the whole
> decoding process becomes moot. Actually, the OP didn't really state this
> clearly and probably just wants a text-based dump.

Yes, we don't know the use here.

> 3. Decoding on-the-fly isn't always possible simply because you don't know
> the character encoding to use.

If you don't know the encoding to use, you often don't know it
afterwards either. Yes, in some cases you can parse the HTML (which
involves knowing which encoding to read the HTML as, of course) but I
suspect this isn't a problem in this case.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Hugo - 21 Dec 2004 11:56 GMT
I don't understand this. The only thing i can say is that on unmanaged c++
using WinHTTP lib this works fine. I never had a memory problem. On .Net
problems raise every time!! The problem is that i'm building a Smart Client
and looks like umanaged c dll are not allowed.

> > >> I wouldn't use this decode while download scheme -- just read raw
> > >> bytes from the response stream and decode the entire byte array once
[quoted text clipped - 30 lines]
> involves knowing which encoding to read the HTML as, of course) but I
> suspect this isn't a problem in this case.
Joerg Jooss - 21 Dec 2004 15:05 GMT
> I don't understand this. The only thing i can say is that on
> unmanaged c++ using WinHTTP lib this works fine. I never had a memory
> problem. On .Net problems raise every time!! The problem is that i'm
> building a Smart Client and looks like umanaged c dll are not allowed.

As long as the garbage collector hasn't run, memory consumption will
steadily increase. Once the GC has run, dead obejcts will be removed from
memory. That's how automatic memory management works.

There's also no problem using unmanaged C DLLs -- you can access them using
P/Invoke.

Cheers,

Signature

Joerg Jooss
www.joergjooss.de
news@joergjooss.de

Hugo - 21 Dec 2004 15:26 GMT
Hello again.
I now that gc is not running all the time. But I let my program running
until ~30.000 handles. The application that i'm building is to be distributed
as a Smart Client.
The user types a url on ie witch download's the application into cache and
when such download comes to the end the application starts.
I guess this only works with managed assemblies i'm isn't that so?
If i use p/Invoke the unmanaged dll wont be download rigth?
Thanks again.

> > I don't understand this. The only thing i can say is that on
> > unmanaged c++ using WinHTTP lib this works fine. I never had a memory
[quoted text clipped - 9 lines]
>
> Cheers,
Jon Skeet [C# MVP] - 21 Dec 2004 16:24 GMT
> Hello again.
> I now that gc is not running all the time. But I let my program running
[quoted text clipped - 4 lines]
> I guess this only works with managed assemblies i'm isn't that so?
> If i use p/Invoke the unmanaged dll wont be download rigth?

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Joerg Jooss - 22 Dec 2004 11:36 GMT
> Hello again.
> I now that gc is not running all the time. But I let my program
> running until ~30.000 handles. The application that i'm building is
> to be distributed as a Smart Client.

As Jon said, can you provide a demo or at least a web site where the
problems shows up?

> The user types a url on ie witch download's the application into
> cache and when such download comes to the end the application starts.
> I guess this only works with managed assemblies i'm isn't that so?
> If i use p/Invoke the unmanaged dll wont be download rigth?

Sorry, I'm not a SmartClient guy, so I cannor answer this one -- but my
guess would be "managed only".

Cheers,

Signature

Joerg Jooss
www.joergjooss.de
news@joergjooss.de

Hugo - 03 Jan 2005 11:39 GMT
Yes. It's true unmanaged dll's won't be download. Unless (I guess) if you use
mixed code on managed C++ dll.

> Hello again.
> I now that gc is not running all the time. But I let my program running
[quoted text clipped - 19 lines]
> >
> > Cheers,
Hugo - 21 Dec 2004 09:14 GMT
Thanks for the reply. I can't expect the download to finish. In fact i don't
no when is going to finish. The server response implements HTTP Chunk
Transfer Encoding. I also need to present the lines that i read from the
stream during download time. I don't want to show all the lines when the
download is finished.
Thanks

> > Hello, I'm running the following code that i found on msdn
> > documentation of GetResponseStream method. If i try to transfer a big
[quoted text clipped - 52 lines]
>
> Cheers,
anil mane - 31 Dec 2004 13:41 GMT
Hello Gurus

Could you help me for converting utf8 encoded text to normal html page
code

i am facing problem for displaying the data like

<td width=\"41%\" valign=\"top\" bgcolor=\"#FFFFFF\"> <span
class=\"normaltext\">\r\n\t<strong>B.Tech/B.E.</strong>\r\n\t
\r\n\t\t\t(Automobile) \r\n\t&nbsp;</td>

how can i get the normal code to get the html page
Joerg Jooss - 31 Dec 2004 15:47 GMT
> Hello Gurus
>
[quoted text clipped - 6 lines]
> class=\"normaltext\">\r\n\t<strong>B.Tech/B.E.</strong>\r\n\t
> \r\n\t\t\t(Automobile) \r\n\t&nbsp;</td>

I don't get the question. What exactly do you want to convert? A normal HTML
page may be UTF-8 encoded text, or may be encoded differently. But
regardless of its encoding, it will always be an HTML page.

If you want to know how to get text (like HTML) from a UTF-8 byte sequence,
use System.Text.Encoding.UTF8.GetString().

Cheers,

Signature

Joerg Jooss
www.joergjooss.de
news@joergjooss.de

anil mane - 03 Jan 2005 11:02 GMT
here what i wrote

    HttpWebResponse
myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse();

            // Gets the stream associated with the response.
            Stream receiveStream = myHttpWebResponse.GetResponseStream();
            Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
            // Pipes the stream to a higher level stream reader with the required
encoding format.
            StreamReader readStream = new StreamReader( receiveStream, encode );
            //Console.WriteLine("\r\nResponse stream received.");

            Char[] read = new Char[256];
            // Reads 256 characters at a time.    
            int count = readStream.Read( read, 0, 256 );
        //    Console.WriteLine("HTML...\r\n");
            string tot_str="";
            while (count > 0)
            {
                // Dumps the 256 characters on a string and displays the string to
the console.
                String str = new String(read, 0, count);
                Console.Write(str);
                tot_str=tot_str+str;
                Console.WriteLine();
                String curr_line = str;
             

                count = readStream.Read(read, 0, 256);
            }
        //    Console.WriteLine("");
            // Releases the resources of the response.
            myHttpWebResponse.Close();

System.Text.Encoding sourceEncoding = System.Text.Encoding.Default;
 byte[] data = sourceEncoding.GetBytes(tot_str);
            int i=data.Length;
string s = System.Text.Encoding.Default.GetString(data);
string s1 =System.Text.Encoding.UTF8.GetString(tot_str);

// here within s and s1 string i dont get the desired data

what i need is instead of /r,/t,/n there should be respective tags or
can able to remove from there like in normal html pages we have

thanks for your reply and waiting for again .......... :)
anil mane - 03 Jan 2005 11:03 GMT
here what i wrote

    HttpWebResponse
myHttpWebResponse=(HttpWebResponse)myHttpWebRequest.GetResponse();

            // Gets the stream associated with the response.
            Stream receiveStream = myHttpWebResponse.GetResponseStream();
            Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
            // Pipes the stream to a higher level stream reader with the required
encoding format.
            StreamReader readStream = new StreamReader( receiveStream, encode );
            //Console.WriteLine("\r\nResponse stream received.");

            Char[] read = new Char[256];
            // Reads 256 characters at a time.    
            int count = readStream.Read( read, 0, 256 );
        //    Console.WriteLine("HTML...\r\n");
            string tot_str="";
            while (count > 0)
            {
                // Dumps the 256 characters on a string and displays the string to
the console.
                String str = new String(read, 0, count);
                Console.Write(str);
                tot_str=tot_str+str;
                Console.WriteLine();
                String curr_line = str;
             

                count = readStream.Read(read, 0, 256);
            }
        //    Console.WriteLine("");
            // Releases the resources of the response.
            myHttpWebResponse.Close();

System.Text.Encoding sourceEncoding = System.Text.Encoding.Default;
 byte[] data = sourceEncoding.GetBytes(tot_str);
            int i=data.Length;
string s = System.Text.Encoding.Default.GetString(data);
string s1 =System.Text.Encoding.UTF8.GetString(tot_str);

// here within s and s1 string i dont get the desired data

what i need is instead of /r,/t,/n there should be respective tags or
can able to remove from there like in normal html pages we have

thanks for your reply and waiting for again .......... :)
Joerg Jooss - 03 Jan 2005 13:11 GMT
> here what i wrote
>
[quoted text clipped - 7 lines]
> encoding format.
> StreamReader readStream = new StreamReader( receiveStream, encode );

Note that you can use StreamReader's default constructor, because it uses
UTF-8.

> //Console.WriteLine("\r\nResponse stream received.");
>
[quoted text clipped - 9 lines]
> String str = new String(read, 0, count);
> Console.Write(str);

You can dump the character array right to the console. There's no reason to
create yet another string within each iteration.

> tot_str=tot_str+str;

You should use a StringBuilder here.

> Console.WriteLine();
> String curr_line = str;

curr_line is never being used.

> count = readStream.Read(read, 0, 256);
> }
[quoted text clipped - 7 lines]
>  string s = System.Text.Encoding.Default.GetString(data);
> string s1 =System.Text.Encoding.UTF8.GetString(tot_str);

I'm not sure what that code is supposed to be good for. There is no method
"string System.Text.Encoding.GetString(string s)"
because it makes no sense at all...

> // here within s and s1 string i dont get the desired data

Save for the non-existing method mentioned already, this is a null
operation. You encode a string and decode it back again.

> what i need is instead of /r,/t,/n there should be respective tags or
> can able to remove from there like in normal html pages we have

There are no HTML tags for CR, TAB or LF. There is <br />, but this has
nothing to do with any end-of-line character. The former creates a line
break in the rendered HTML, the latter a line break in the HTML source. What
are you actually trying to achieve?

Cheers,

Signature

Joerg Jooss
www.joergjooss.de
news@joergjooss.de

anil mane - 03 Jan 2005 18:41 GMT
thanks for quick response ...

what i need is
create an html page and dumb down the output value
now if i dumb as it is like text containing  of /r,/t,/n
i wont be able to properly in html page
let try to copy bellow text and save as test.htm

<span class=\"normaltext\">\r\n\t<strong>b.tech/b.e.</strong>\r\n\t
\r\n\t\t\t(Automobile) \r\n\t&nbsp;</span>

now we wont be able to get the desire output if we have a look at
test.htm

i hope it would be clear

As i am newbie to this technology i will be thankful to your  help

.................
Joerg Jooss - 04 Jan 2005 09:39 GMT
> thanks for quick response ...
>
[quoted text clipped - 11 lines]
>
> i hope it would be clear

I'm afraid not really...

Signature

Joerg Jooss
www.joergjooss.de
news@joergjooss.de


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.