Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / CLR / December 2005

Tip: Looking for answers? Try searching our database.

Handle and memory leaks

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Lucvdv - 08 Dec 2005 16:52 GMT
A .Net 1.1-based service application I wrote seems to be slowly leaking
memory and handles.  Not enough to notice over a short timespan (normal
fluctuations as it's doing it work are larger than the amount of rise), but
memory use and handle count keep slowly going up.

Has anyone had a similar experience?

It does use a few COM components, but those (the same versions) have been
used in an older version of the project for years, and are definitely not
responsible.

All the rest is 100% managed code.  A few objects that have a 'dispose'
method are explicitly destroyed after use, others just go out of scope, so
the garbage collector should take care of them.

There are no visible UI components, but the COM objects are OCXes placed on
a hidden window (there normally is no reason why a service app shouldn't
have a window and a message pump, I've heard that discussion before ;)

I don't think it's an interop problem: communication between the managed
code and the OCXes is rather heavy, it would cause a faster grow if the
reason was there.

The executable is a release build, so it's not caused by running a debug
build (in which garbage collection almost never takes place) either.

Also, the size of the memory leak seems to depend on what OS the service is
running on (more on NT4 than on XP).

Shortly after startup, the process's memory usage (in task manager) is
about 18000-19000 kB, with the handle count fluctuating between 250 and
somewhere slightly above 300.

After running for several days on two test systems, one XP Embedded and the
other NT4, and both started simultaneously, these are the memory use and
handle counts:

NT4: 156264 kB, 97846 handles (384 MB phys, total load close to 300 MB).
XP: 78996 kB, 109801 handles (1GB physical RAM, total commit about 250 MB).

Stopping the service takes nearly a minute, during which the handle count
gradually decreases.

In the previous (longer) test run, the NT4 system completely locked up at a
certain point.  Most functions were still accessible over the network, the
leaking service was even still running, but trying to start any application
locally resulted in a message that user32.dll failed to load.  I stopped
the service remotely from another machine, but that didn't help either: I
had to reboot.

The service's own load (the work it has to do) is approximately the same on
both systems.

I intend to start a new test run tomorrow, where garbage collection is
forced once every hour or so, to see if it makes a difference.  It may be a
few days though before I have results that are somewhat conclusive.
Lucvdv - 08 Dec 2005 17:01 GMT
> NT4: 156264 kB, 97846 handles (384 MB phys, total load close to 300 MB).
> XP: 78996 kB, 109801 handles (1GB physical RAM, total commit about 250 MB).

Oops - minor mistake: the XP Embedded system is currently booted to
Win2000, so the second line should read Win2000 instead of XP.
Lucvdv - 09 Dec 2005 10:14 GMT
> I intend to start a new test run tomorrow, where garbage collection is
> forced once every hour or so, to see if it makes a difference.  It may be a
> few days though before I have results that are somewhat conclusive.

No point in trying: the CLR profiler shows GC being done all the time.

When I started a new run under the profiler, pool use after startup shrunk
by 20 K over the first hour.

Yet the process working set went up by 2 MB in the same period (which seems
to indicate that growth is much faster during the first day or so, because
it had 'only' grown to 78996 KB after two weeks on the same machine in the
previous run).

I still can't believe my OCXes are responsible, because they've been used
for years in a non-.Net container, sometimes for more than a month at a
stretch, without any sign of a leak.
Willy Denoyette [MVP] - 09 Dec 2005 10:49 GMT
>> I intend to start a new test run tomorrow, where garbage collection is
>> forced once every hour or so, to see if it makes a difference.  It may be
[quoted text clipped - 15 lines]
> for years in a non-.Net container, sometimes for more than a month at a
> stretch, without any sign of a leak.

If your OCX's aren't running in a thread that pumps messages, you will leak
OS handles because the finalizer cannot clean-up the RCW. The result of this
is: Handle leaks and a blocked finalizer thread resulting in more unmanaged
resource leaks etc...

Willy.
Lucvdv - 09 Dec 2005 11:59 GMT
> If your OCX's aren't running in a thread that pumps messages, you will leak
> OS handles because the finalizer cannot clean-up the RCW. The result of this
> is: Handle leaks and a blocked finalizer thread resulting in more unmanaged
> resource leaks etc...

Would their events ever be fired without a working message pump?
I thought not.

Anyhow, I did call 'Application.Run' in the thread that creates the hidden
window to start it - but when I just rechecked it, I noticed that I forgot
to pass it the window (I called the parameterless overload which starts a
message pump without a window).

Assuming that must be it, I just recompiled and I'm now about to start a
new test where it passes the correct form.
Lucvdv - 09 Dec 2005 16:35 GMT
> Anyhow, I did call 'Application.Run' in the thread that creates the hidden
> window to start it - but when I just rechecked it, I noticed that I forgot
[quoted text clipped - 3 lines]
> Assuming that must be it, I just recompiled and I'm now about to start a
> new test where it passes the correct form.

No luck :(

It turns out to be something entirely different, 100% managed code: failed
connection attempts on a Net.Sockets.TcpClient.

- Each time a TcpClient is created and attempts to connect, 8 handles
- If the attempt suceeds, all 8 are closed when the TcpClient is closed.
- If the attempt fails (time-out because the IP doesn't exist), 4 handles
 and some memory are leaked when the TcpClient is closed.

I'll try if I can create a minimalistic demo app that reproduces the
problem by monday, but here is the relevant (VB) code in my app.

It's part of a class that handles the connection; the .Close member is
always called, for succeeded as well as failed connection attempts (called
directly, not through Finalize, but I put it in Finalize as well just in
case I forgot):

   Private m_Connection As Net.Sockets.TcpClient
   Private m_Connected As Boolean = False
   Private m_Error As Boolean = False
   Private m_Stream As Net.Sockets.NetworkStream

   Public Sub New(ByVal IP As String)
       Try
           m_Connection = New Net.Sockets.TcpClient
           m_Connection.Connect(IPAddress.Parse(IP), 2101)
           m_Connected = True  ' Skipped if connect fails
           m_Stream = m_Connection.GetStream
       Catch ex As Exception
           m_Error = True
       End Try
   End Sub

   Public Sub Close()
       On Error Resume Next    ' in case stream not open
       If Not m_Stream Is Nothing Then m_Stream.Close()
       If Not m_Connection Is Nothing Then m_Connection.Close()
       m_Connected = False
       m_Stream = Nothing
       m_Connection = Nothing
   End Sub

   Protected Overrides Sub Finalize()
       If Not m_Connection Is Nothing Then Close()
       MyBase.Finalize()
   End Sub

I left out the rest, which is only one method that sends a command through
m_stream and returns the response.
Willy Denoyette [MVP] - 09 Dec 2005 20:49 GMT
>> Anyhow, I did call 'Application.Run' in the thread that creates the
>> hidden
[quoted text clipped - 15 lines]
> - If the attempt fails (time-out because the IP doesn't exist), 4 handles
>  and some memory are leaked when the TcpClient is closed.

TYhey should not leak if disposed corectly.

> I'll try if I can create a minimalistic demo app that reproduces the
> problem by monday, but here is the relevant (VB) code in my app.
[quoted text clipped - 36 lines]
> I left out the rest, which is only one method that sends a command through
> m_stream and returns the response.

When Connect fails, m_Connection  will not be destroyed (you won't call
Close() do you?), but the object has a live reference right? So you rely on
Finalize to be called (and this will call Close()).
Now, if the finalizer doesn't run because it's blocked (see my previous
reply. the sockect handles will leak together with some unmanaged memory
allocated by WinSock.
So what you absolutely haved to do is make SURE the finalizer isn't blocked,
you can do this by attaching a debugger, or by tracing the Finalizer.

Willy.
Lucvdv - 10 Dec 2005 09:23 GMT
> When Connect fails, m_Connection  will not be destroyed (you won't call
> Close() do you?), but the object has a live reference right? So you rely on
> Finalize to be called (and this will call Close()).

No, I said:

> > [...] the .Close member is
> > always called, for succeeded as well as failed connection attempts (called
> > directly, not through Finalize, but I put it in Finalize as well just in
> > case I forgot) [...]

Also, this part of the application is pure .Net code, unrelated to the
OCX handling, even running in another thread (each thread does a
distinct job, none of them ever touches any object or variable that
was created by the other, with a String that is initialized to a SQL
connection string as the only exception).

Or do you mean that cleanup of the unmanaged part of the winsock class
(between the CLR and win32) would _also_ be blocked if the OCXes are
not handled correctly?

I didn't mention this before, but commenting out the line that starts
the thread that does the winsock thing, completely stopped the leak.
The OCX part was still running as before, and didn't leak anything.
Lucvdv - 10 Dec 2005 11:24 GMT
I said I'd see if I could create a small demo app that reproduces the
problem.  Here it is.

Just begin a new VB.Net windows app in VS 2003, and replace the
auto-generated code with the code below.  I tried to break up lines
with continuation characters where newsreaders would wrap them.

It doesn't use any COM objects so I'm sorry for leading the that
direction first (I assumed that's where the problem would be, but
assuming is always a dangerous activity ;)

As for the TcpClient, it uses it in a way as close as possible to how
the real app does is (esp. threading: what is done in separate
threads).

When I run this program on my home machine, I get the exact same
symptoms as with the other app at work, so it's not limited to one
machine.  [Both machines CLR 1.1 SP1]

The numbers below were obtained with a release build, started outside
of the VS debugger.  I never created two connections at once (i.e.
clicked the "test" button only after the previous attempt had
completely finshed, even though it will attempt as many as you click).

I watched the connections in SysInternals' TCPView to see when each
attempt started and ended, while monitoring the program's handle count
in task manager.

Handle count:
at startup: 66
after first click: 84
after connection failed: 78
after second click: 86
after connection failed: 82
after third click: 90
after connection failed: 86

The series just continues like that: I followed it as far as 94 - 90 -
98 - 94 - 102 - 98.

In my service app at work it ran into the tens of thousands, because
most of the addresses in the test setup were fictitious.
In a real life environment it would be the same, because the machines
it talks to are often switched off for the night while the service
keeps running.

Code:

Imports System.Net
Imports System.Net.Sockets
Imports System.Threading

Public Class Form1
   Inherits System.Windows.Forms.Form

#Region " Windows Form Designer generated code "

   Public Sub New()
       MyBase.New()

       'This call is required by the Windows Form Designer.
       InitializeComponent()

       'Add any initialization after the InitializeComponent() call

   End Sub

   'Form overrides dispose to clean up the component list.
   Protected Overloads Overrides Sub Dispose _
       (ByVal disposing As Boolean)

       If disposing Then
           If Not (components Is Nothing) Then
               components.Dispose()
           End If
       End If
       MyBase.Dispose(disposing)
   End Sub

   'Required by the Windows Form Designer
   Private components As System.ComponentModel.IContainer

   'NOTE: The following procedure is required
   ' by the Windows Form Designer
   'It can be modified using the Windows Form Designer.  
   'Do not modify it using the code editor.
   Friend WithEvents Label1 As System.Windows.Forms.Label
   Friend WithEvents txtIP As System.Windows.Forms.TextBox
   Friend WithEvents cmdTest As System.Windows.Forms.Button

   <System.Diagnostics.DebuggerStepThrough()> _
   Private Sub InitializeComponent()
       Me.cmdTest = New System.Windows.Forms.Button
       Me.Label1 = New System.Windows.Forms.Label
       Me.txtIP = New System.Windows.Forms.TextBox
       Me.SuspendLayout()
       '
       'cmdTest
       '
       Me.cmdTest.Location = New System.Drawing.Point(40, 48)
       Me.cmdTest.Name = "cmdTest"
       Me.cmdTest.TabIndex = 0
       Me.cmdTest.Text = "Test"
       '
       'Label1
       '
       Me.Label1.Location = New System.Drawing.Point(16, 16)
       Me.Label1.Name = "Label1"
       Me.Label1.Size = New System.Drawing.Size(16, 16)
       Me.Label1.TabIndex = 1
       Me.Label1.Text = "IP"
       '
       'txtIP
       '
       Me.txtIP.Location = New System.Drawing.Point(40, 16)
       Me.txtIP.Name = "txtIP"
       Me.txtIP.Size = New System.Drawing.Size(104, 20)
       Me.txtIP.TabIndex = 2
       Me.txtIP.Text = "10.123.234.56"
       '
       'Form1
       '
       Me.AutoScaleBaseSize = New System.Drawing.Size(5, 13)
       Me.ClientSize = New System.Drawing.Size(160, 77)
       Me.Controls.Add(Me.txtIP)
       Me.Controls.Add(Me.Label1)
       Me.Controls.Add(Me.cmdTest)
       Me.FormBorderStyle = _
           System.Windows.Forms.FormBorderStyle.FixedDialog
       Me.MaximizeBox = False
       Me.MinimizeBox = False
       Me.Name = "Form1"
       Me.Text = "IP Leak test"
       Me.ResumeLayout(False)

   End Sub

#End Region

   Private Sub cmdTest_Click(ByVal sender As System.Object, _
       ByVal e As System.EventArgs) Handles cmdTest.Click

       Dim IP As IPAddress = IPAddress.Parse(txtIP.Text)

       ' Generate new address for next attempt
       Dim NewAddr() As Byte = IP.GetAddressBytes
       For i As Integer = 3 To 0 Step -1
           NewAddr(i) += 1
           If NewAddr(i) = 255 Then
               NewAddr(i) = 1
           Else
               Exit For
           End If
       Next
       ' IPAddress constructor accepts byte array,
       ' but won't accept result of .GetAddressBytes ???
       ' txtIP.Text = New IPAddress(NewAddr).ToString
       txtIP.Text = NewAddr(0) & "." & NewAddr(1) _
           & "." & NewAddr(2) & "." & NewAddr(3)

       Dim Tester As New TestConnect(IP)
   End Sub

   Private Class TestConnect
       Private m_socket As TcpClient
       Private m_IP As IPAddress
       Private m_thr As Thread

       Public Sub New(ByVal IP As IPAddress)
           m_IP = IP
           m_thr = New Thread(AddressOf DoTest)
           m_thr.Start()
       End Sub

       Private Sub DoTest()
           Try
               m_socket = New TcpClient
               Debug.WriteLine(m_IP.ToString & ": Connecting")
               m_socket.Connect(m_IP, 1025)
           Catch ex As Exception
               Debug.WriteLine(m_IP.ToString & ": " & ex.Message)
           Finally
               Debug.WriteLine(m_IP.ToString & ": Closing")
               If Not m_socket Is Nothing Then m_socket.Close()
               m_socket = Nothing
           End Try
       End Sub
   End Class

End Class
Lucvdv - 10 Dec 2005 12:16 GMT
> I said I'd see if I could create a small demo app that reproduces the
> problem.  Here it is.

And by further reducing it, I now see that it's not the TcpClient
that's responsible either.

Even if the thread does nothing at all, the program still gets 4 more
handles each time the button is clicked, so it looks like it's the way
I use the thread.

But after running up to nearly 500 handles, GC suddenly cleaned them
up, and for some reason that doesn't happen in the full program - so
back to your original explanation that something is blocking
finalizers?

I fixed that, so maybe the problem is solved now, but I just didn't
let it run long enough to see the effect?  I did let it run for more
than an hour though.
TT (Tom Tempelaere) - 12 Dec 2005 15:33 GMT
Hi Lucvdv,

> > Anyhow, I did call 'Application.Run' in the thread that creates the hidden
> > window to start it - but when I just rechecked it, I noticed that I forgot
[quoted text clipped - 54 lines]
> I left out the rest, which is only one method that sends a command through
> m_stream and returns the response.

I don't like your constructor. It swallows all exceptions, and just sets an
error member to true. Better is to let the exception percolate through, so
you don't end up with an object that is in a bad state and whose sole purpose
is to be closed/disposed or gb-collected.

I would write something like (in C#, dunno VB sorry):

[code]
public class YourClass
{
 private TcpClient m_Connection = null;
 private NetworkStream m_Stream = null;

 public YourClass( ) // constructor (New in VB.NET)
 {
   try
   {
     m_Connection = new TcpClient( );
     m_Connection.Connect( ... );
     m_Stream = m_Connection.GetStream( );
     // ...
   }
   catch
   {
     if( m_Stream != null )
       m_Stream.Close( );
     if( m_Connection != null )
       m_Connection.Close( );
     throw;
   }
 }
 // ...
}
[/code]

I think ActiveX's may well be the worst invention ever. Nothing but trouble
with those beasts. Well probably the fault lies with those that chose ActiveX
as technology rather than ActiveX being the problem. Even more probable, is
that VB programmers use ActiveX technology for about everything, including
things that don't need a UI (and ActiveX was definitely meant as a UI COM
component).

Anyway, I don't really know why you would be leaking. I think Willy D. might
be correct.

Kind regards,
PS: I'm not too happy about how VB sees things (the old or new VB).
Signature

Tom Tempelaere.

Lucvdv - 12 Dec 2005 16:48 GMT
> I think ActiveX's may well be the worst invention ever. Nothing but trouble
> with those beasts. Well probably the fault lies with those that chose ActiveX
> as technology rather than ActiveX being the problem. Even more probable, is
> that VB programmers use ActiveX technology for about everything, including
> things that don't need a UI (and ActiveX was definitely meant as a UI COM
> component).

My plea is "guilty" for using ActiveX because ActiveX was there when I
created those OCXes for use in VB6.  But I blame MS: they made it far too
easy to create an OCX in VC++ 6, compared to other (windowless) COM
projects.

MS engineers must have noticed it too, because VB6 came with OCX components
that didn't have anything to do with a UI (winsock, comm port controls).
I just modeled my code after that.

In VS6 I've always used sort of a layered approach: UI in VB6 for the ease
of design and debugging, and the real work in VC++ for the flexibility and
multithreading capability.  I switched to plain DLL's (not even COM)
instead of OCXes later, but a few of those old tarts still live and are a
bit too large to consider a rewrite :(

> Anyway, I don't really know why you would be leaking. I think Willy D. might
> be correct.

I wish :(

The service was starting two threads to perform two distinct functions. The
only thing they had in common is that they're using the same database.

So now I split it up to two separate services.
The part that uses the OCX runs perfectly, no trace of anything leaking.

The other one is now 100% managed code, and feels like tring to carry water
in a sieve.

The CLR profiler shows that all thread objects (per connection as in the
example) just keep existing forever.  I've been trying to find "how come"
for hours today: some reference must still exist or GC would take care of
them, but I found none.

In the code sample I posted, GC does take care of them after a while.
Maybe not as soon as I'd like, but it does.

In the service it doesn't clean them up at all, and I can't image where
there would be a remaining reference that doesn't exist in the sample.
Forcing GC didn't solve it either (not that I expected it to, but it does
shorten the time each new test has to run before I'm sure the problem still
exists).

In a desperate attempt, I'm now half way through translating it from VB to
C#, to see if looking at it in another language sheds some light.
TT (Tom Tempelaere) - 13 Dec 2005 10:39 GMT
Hi Luc,

[...]
> > Anyway, I don't really know why you would be leaking. I think Willy D. might
> > be correct.
[quoted text clipped - 26 lines]
> In a desperate attempt, I'm now half way through translating it from VB to
> C#, to see if looking at it in another language sheds some light.

That indeed is desperate. Although I prefer C# there should be no difference
in run time since they both compile to IL and defer functionality to the
framework. The run time environment has the GC, so I don't think that another
language will solve the problem.

Without a complete repro program I can't really tell what the problem is.
The only advice I can give you is to Dispose/Close as early as possible and
do it for every disposable/closable object. Use the using statement in C# for
disposable objects: it guarantees disposal in the face of exceptions. Anyway
what I'm trying to say is to keep your finalizers empty, except for unmanaged
resource cleanup (and then only as a safe-guard when you would forget to
Dispose). I'm not a big fan of finalizers.

Don't forget that at the end of your Dipose method, you should tell the GC
not to finalize the instance anymore. Like this (C#):
 GC.SuppressFinalize( this );

This means that your instance will not be put on the finalizer queue.

What I do find weird is that your program does not leak when run in a
window, but when run in a service that you experience leaks. A complete repro
program would be nice.

Kind regards,
--
Tom Tempelaere.
Lucvdv - 13 Dec 2005 12:56 GMT
> > In a desperate attempt, I'm now half way through translating it from VB to
> > C#, to see if looking at it in another language sheds some light.
[quoted text clipped - 3 lines]
> framework. The run time environment has the GC, so I don't think that another
> language will solve the problem.

Problem solved, without finishing the rewrite.
As I expected, it was something so small you could look at it and not
notice it: a single letter.

The original service created a window to hold the OCXes, so I started it as
STAThread.  The second half didn't exist yet at that time.

Then later I added the second thread, and started it from the main thread
(it never touched the window, so there shouldn't be any harm), without
looking at or specifying the threading model.

When I split the service in two, I created a copy and removed half of the
functionality, but I forgot that it was still starting as STAThread.

Changed it to MTAThread --> no more leak.

Thanks to you and Willy for the help.
Lucvdv - 13 Dec 2005 14:22 GMT
> Changed it to MTAThread --> no more leak.

But strange, when I create a new one that does nothing but start threads
that sleep for a second and exit, it can be either MTA or STA, no leak.
Willy Denoyette [MVP] - 09 Dec 2005 20:39 GMT
>> If your OCX's aren't running in a thread that pumps messages, you will
>> leak
[quoted text clipped - 6 lines]
> Would their events ever be fired without a working message pump?
> I thought not.

Sure they will, OLE32 (COM) will pump message on the hidden window if you
have this OCX in a thread that doesn't pump it's message queue. But this is
at the unmanaged side, the problem is at the managed side, where you have
your RCW (created by the CLR remember), if the RCW is running in an STA
thread (the same as your COM object), you should pump messages. Note that
you shouldn't create a Window and a message pump, the Framework provides a
number of method that intrinsically pump the message queue (for STA threads
only). One of these methods is WaitForPendingFinalizers(), another one is
WaitHandle.WaitOne(). So what you need to do is call
WaitForPendingFinalizers().
The reason for this all is that the finalizer thread runs in an MTA, when he
needs to run the "finalize" method on the RCW, the call has to be marshaled
and this only works if the STA thread (runing your RCW) pumps messages,
failing to pump will block the fnalizer thread.

> Anyhow, I did call 'Application.Run' in the thread that creates the hidden
> window to start it - but when I just rechecked it, I noticed that I forgot
[quoted text clipped - 3 lines]
> Assuming that must be it, I just recompiled and I'm now about to start a
> new test where it passes the correct form.

Rate this thread:







Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.