.NET Forum / .NET Framework / CLR / December 2005
Handle and memory leaks
|
|
Thread rating:  |
Lucvdv - 08 Dec 2005 16:52 GMT A .Net 1.1-based service application I wrote seems to be slowly leaking memory and handles. Not enough to notice over a short timespan (normal fluctuations as it's doing it work are larger than the amount of rise), but memory use and handle count keep slowly going up.
Has anyone had a similar experience?
It does use a few COM components, but those (the same versions) have been used in an older version of the project for years, and are definitely not responsible.
All the rest is 100% managed code. A few objects that have a 'dispose' method are explicitly destroyed after use, others just go out of scope, so the garbage collector should take care of them.
There are no visible UI components, but the COM objects are OCXes placed on a hidden window (there normally is no reason why a service app shouldn't have a window and a message pump, I've heard that discussion before ;)
I don't think it's an interop problem: communication between the managed code and the OCXes is rather heavy, it would cause a faster grow if the reason was there.
The executable is a release build, so it's not caused by running a debug build (in which garbage collection almost never takes place) either.
Also, the size of the memory leak seems to depend on what OS the service is running on (more on NT4 than on XP).
Shortly after startup, the process's memory usage (in task manager) is about 18000-19000 kB, with the handle count fluctuating between 250 and somewhere slightly above 300.
After running for several days on two test systems, one XP Embedded and the other NT4, and both started simultaneously, these are the memory use and handle counts:
NT4: 156264 kB, 97846 handles (384 MB phys, total load close to 300 MB). XP: 78996 kB, 109801 handles (1GB physical RAM, total commit about 250 MB).
Stopping the service takes nearly a minute, during which the handle count gradually decreases.
In the previous (longer) test run, the NT4 system completely locked up at a certain point. Most functions were still accessible over the network, the leaking service was even still running, but trying to start any application locally resulted in a message that user32.dll failed to load. I stopped the service remotely from another machine, but that didn't help either: I had to reboot.
The service's own load (the work it has to do) is approximately the same on both systems.
I intend to start a new test run tomorrow, where garbage collection is forced once every hour or so, to see if it makes a difference. It may be a few days though before I have results that are somewhat conclusive.
Lucvdv - 08 Dec 2005 17:01 GMT > NT4: 156264 kB, 97846 handles (384 MB phys, total load close to 300 MB). > XP: 78996 kB, 109801 handles (1GB physical RAM, total commit about 250 MB). Oops - minor mistake: the XP Embedded system is currently booted to Win2000, so the second line should read Win2000 instead of XP.
Lucvdv - 09 Dec 2005 10:14 GMT > I intend to start a new test run tomorrow, where garbage collection is > forced once every hour or so, to see if it makes a difference. It may be a > few days though before I have results that are somewhat conclusive. No point in trying: the CLR profiler shows GC being done all the time.
When I started a new run under the profiler, pool use after startup shrunk by 20 K over the first hour.
Yet the process working set went up by 2 MB in the same period (which seems to indicate that growth is much faster during the first day or so, because it had 'only' grown to 78996 KB after two weeks on the same machine in the previous run).
I still can't believe my OCXes are responsible, because they've been used for years in a non-.Net container, sometimes for more than a month at a stretch, without any sign of a leak.
Willy Denoyette [MVP] - 09 Dec 2005 10:49 GMT >> I intend to start a new test run tomorrow, where garbage collection is >> forced once every hour or so, to see if it makes a difference. It may be [quoted text clipped - 15 lines] > for years in a non-.Net container, sometimes for more than a month at a > stretch, without any sign of a leak. If your OCX's aren't running in a thread that pumps messages, you will leak OS handles because the finalizer cannot clean-up the RCW. The result of this is: Handle leaks and a blocked finalizer thread resulting in more unmanaged resource leaks etc...
Willy.
Lucvdv - 09 Dec 2005 11:59 GMT > If your OCX's aren't running in a thread that pumps messages, you will leak > OS handles because the finalizer cannot clean-up the RCW. The result of this > is: Handle leaks and a blocked finalizer thread resulting in more unmanaged > resource leaks etc... Would their events ever be fired without a working message pump? I thought not.
Anyhow, I did call 'Application.Run' in the thread that creates the hidden window to start it - but when I just rechecked it, I noticed that I forgot to pass it the window (I called the parameterless overload which starts a message pump without a window).
Assuming that must be it, I just recompiled and I'm now about to start a new test where it passes the correct form.
Lucvdv - 09 Dec 2005 16:35 GMT > Anyhow, I did call 'Application.Run' in the thread that creates the hidden > window to start it - but when I just rechecked it, I noticed that I forgot [quoted text clipped - 3 lines] > Assuming that must be it, I just recompiled and I'm now about to start a > new test where it passes the correct form. No luck :(
It turns out to be something entirely different, 100% managed code: failed connection attempts on a Net.Sockets.TcpClient.
- Each time a TcpClient is created and attempts to connect, 8 handles - If the attempt suceeds, all 8 are closed when the TcpClient is closed. - If the attempt fails (time-out because the IP doesn't exist), 4 handles and some memory are leaked when the TcpClient is closed.
I'll try if I can create a minimalistic demo app that reproduces the problem by monday, but here is the relevant (VB) code in my app.
It's part of a class that handles the connection; the .Close member is always called, for succeeded as well as failed connection attempts (called directly, not through Finalize, but I put it in Finalize as well just in case I forgot):
Private m_Connection As Net.Sockets.TcpClient Private m_Connected As Boolean = False Private m_Error As Boolean = False Private m_Stream As Net.Sockets.NetworkStream
Public Sub New(ByVal IP As String) Try m_Connection = New Net.Sockets.TcpClient m_Connection.Connect(IPAddress.Parse(IP), 2101) m_Connected = True ' Skipped if connect fails m_Stream = m_Connection.GetStream Catch ex As Exception m_Error = True End Try End Sub
Public Sub Close() On Error Resume Next ' in case stream not open If Not m_Stream Is Nothing Then m_Stream.Close() If Not m_Connection Is Nothing Then m_Connection.Close() m_Connected = False m_Stream = Nothing m_Connection = Nothing End Sub
Protected Overrides Sub Finalize() If Not m_Connection Is Nothing Then Close() MyBase.Finalize() End Sub
I left out the rest, which is only one method that sends a command through m_stream and returns the response.
Willy Denoyette [MVP] - 09 Dec 2005 20:49 GMT >> Anyhow, I did call 'Application.Run' in the thread that creates the >> hidden [quoted text clipped - 15 lines] > - If the attempt fails (time-out because the IP doesn't exist), 4 handles > and some memory are leaked when the TcpClient is closed. TYhey should not leak if disposed corectly.
> I'll try if I can create a minimalistic demo app that reproduces the > problem by monday, but here is the relevant (VB) code in my app. [quoted text clipped - 36 lines] > I left out the rest, which is only one method that sends a command through > m_stream and returns the response. When Connect fails, m_Connection will not be destroyed (you won't call Close() do you?), but the object has a live reference right? So you rely on Finalize to be called (and this will call Close()). Now, if the finalizer doesn't run because it's blocked (see my previous reply. the sockect handles will leak together with some unmanaged memory allocated by WinSock. So what you absolutely haved to do is make SURE the finalizer isn't blocked, you can do this by attaching a debugger, or by tracing the Finalizer.
Willy.
Lucvdv - 10 Dec 2005 09:23 GMT > When Connect fails, m_Connection will not be destroyed (you won't call > Close() do you?), but the object has a live reference right? So you rely on > Finalize to be called (and this will call Close()). No, I said:
> > [...] the .Close member is > > always called, for succeeded as well as failed connection attempts (called > > directly, not through Finalize, but I put it in Finalize as well just in > > case I forgot) [...] Also, this part of the application is pure .Net code, unrelated to the OCX handling, even running in another thread (each thread does a distinct job, none of them ever touches any object or variable that was created by the other, with a String that is initialized to a SQL connection string as the only exception).
Or do you mean that cleanup of the unmanaged part of the winsock class (between the CLR and win32) would _also_ be blocked if the OCXes are not handled correctly?
I didn't mention this before, but commenting out the line that starts the thread that does the winsock thing, completely stopped the leak. The OCX part was still running as before, and didn't leak anything.
Lucvdv - 10 Dec 2005 11:24 GMT I said I'd see if I could create a small demo app that reproduces the problem. Here it is.
Just begin a new VB.Net windows app in VS 2003, and replace the auto-generated code with the code below. I tried to break up lines with continuation characters where newsreaders would wrap them.
It doesn't use any COM objects so I'm sorry for leading the that direction first (I assumed that's where the problem would be, but assuming is always a dangerous activity ;)
As for the TcpClient, it uses it in a way as close as possible to how the real app does is (esp. threading: what is done in separate threads).
When I run this program on my home machine, I get the exact same symptoms as with the other app at work, so it's not limited to one machine. [Both machines CLR 1.1 SP1]
The numbers below were obtained with a release build, started outside of the VS debugger. I never created two connections at once (i.e. clicked the "test" button only after the previous attempt had completely finshed, even though it will attempt as many as you click).
I watched the connections in SysInternals' TCPView to see when each attempt started and ended, while monitoring the program's handle count in task manager.
Handle count: at startup: 66 after first click: 84 after connection failed: 78 after second click: 86 after connection failed: 82 after third click: 90 after connection failed: 86
The series just continues like that: I followed it as far as 94 - 90 - 98 - 94 - 102 - 98.
In my service app at work it ran into the tens of thousands, because most of the addresses in the test setup were fictitious. In a real life environment it would be the same, because the machines it talks to are often switched off for the night while the service keeps running.
Code:
Imports System.Net Imports System.Net.Sockets Imports System.Threading
Public Class Form1 Inherits System.Windows.Forms.Form
#Region " Windows Form Designer generated code "
Public Sub New() MyBase.New()
'This call is required by the Windows Form Designer. InitializeComponent()
'Add any initialization after the InitializeComponent() call
End Sub
'Form overrides dispose to clean up the component list. Protected Overloads Overrides Sub Dispose _ (ByVal disposing As Boolean)
If disposing Then If Not (components Is Nothing) Then components.Dispose() End If End If MyBase.Dispose(disposing) End Sub
'Required by the Windows Form Designer Private components As System.ComponentModel.IContainer
'NOTE: The following procedure is required ' by the Windows Form Designer 'It can be modified using the Windows Form Designer. 'Do not modify it using the code editor. Friend WithEvents Label1 As System.Windows.Forms.Label Friend WithEvents txtIP As System.Windows.Forms.TextBox Friend WithEvents cmdTest As System.Windows.Forms.Button
<System.Diagnostics.DebuggerStepThrough()> _ Private Sub InitializeComponent() Me.cmdTest = New System.Windows.Forms.Button Me.Label1 = New System.Windows.Forms.Label Me.txtIP = New System.Windows.Forms.TextBox Me.SuspendLayout() ' 'cmdTest ' Me.cmdTest.Location = New System.Drawing.Point(40, 48) Me.cmdTest.Name = "cmdTest" Me.cmdTest.TabIndex = 0 Me.cmdTest.Text = "Test" ' 'Label1 ' Me.Label1.Location = New System.Drawing.Point(16, 16) Me.Label1.Name = "Label1" Me.Label1.Size = New System.Drawing.Size(16, 16) Me.Label1.TabIndex = 1 Me.Label1.Text = "IP" ' 'txtIP ' Me.txtIP.Location = New System.Drawing.Point(40, 16) Me.txtIP.Name = "txtIP" Me.txtIP.Size = New System.Drawing.Size(104, 20) Me.txtIP.TabIndex = 2 Me.txtIP.Text = "10.123.234.56" ' 'Form1 ' Me.AutoScaleBaseSize = New System.Drawing.Size(5, 13) Me.ClientSize = New System.Drawing.Size(160, 77) Me.Controls.Add(Me.txtIP) Me.Controls.Add(Me.Label1) Me.Controls.Add(Me.cmdTest) Me.FormBorderStyle = _ System.Windows.Forms.FormBorderStyle.FixedDialog Me.MaximizeBox = False Me.MinimizeBox = False Me.Name = "Form1" Me.Text = "IP Leak test" Me.ResumeLayout(False)
End Sub
#End Region
Private Sub cmdTest_Click(ByVal sender As System.Object, _ ByVal e As System.EventArgs) Handles cmdTest.Click
Dim IP As IPAddress = IPAddress.Parse(txtIP.Text)
' Generate new address for next attempt Dim NewAddr() As Byte = IP.GetAddressBytes For i As Integer = 3 To 0 Step -1 NewAddr(i) += 1 If NewAddr(i) = 255 Then NewAddr(i) = 1 Else Exit For End If Next ' IPAddress constructor accepts byte array, ' but won't accept result of .GetAddressBytes ??? ' txtIP.Text = New IPAddress(NewAddr).ToString txtIP.Text = NewAddr(0) & "." & NewAddr(1) _ & "." & NewAddr(2) & "." & NewAddr(3)
Dim Tester As New TestConnect(IP) End Sub
Private Class TestConnect Private m_socket As TcpClient Private m_IP As IPAddress Private m_thr As Thread
Public Sub New(ByVal IP As IPAddress) m_IP = IP m_thr = New Thread(AddressOf DoTest) m_thr.Start() End Sub
Private Sub DoTest() Try m_socket = New TcpClient Debug.WriteLine(m_IP.ToString & ": Connecting") m_socket.Connect(m_IP, 1025) Catch ex As Exception Debug.WriteLine(m_IP.ToString & ": " & ex.Message) Finally Debug.WriteLine(m_IP.ToString & ": Closing") If Not m_socket Is Nothing Then m_socket.Close() m_socket = Nothing End Try End Sub End Class
End Class
Lucvdv - 10 Dec 2005 12:16 GMT > I said I'd see if I could create a small demo app that reproduces the > problem. Here it is. And by further reducing it, I now see that it's not the TcpClient that's responsible either.
Even if the thread does nothing at all, the program still gets 4 more handles each time the button is clicked, so it looks like it's the way I use the thread.
But after running up to nearly 500 handles, GC suddenly cleaned them up, and for some reason that doesn't happen in the full program - so back to your original explanation that something is blocking finalizers?
I fixed that, so maybe the problem is solved now, but I just didn't let it run long enough to see the effect? I did let it run for more than an hour though.
TT (Tom Tempelaere) - 12 Dec 2005 15:33 GMT Hi Lucvdv,
> > Anyhow, I did call 'Application.Run' in the thread that creates the hidden > > window to start it - but when I just rechecked it, I noticed that I forgot [quoted text clipped - 54 lines] > I left out the rest, which is only one method that sends a command through > m_stream and returns the response. I don't like your constructor. It swallows all exceptions, and just sets an error member to true. Better is to let the exception percolate through, so you don't end up with an object that is in a bad state and whose sole purpose is to be closed/disposed or gb-collected.
I would write something like (in C#, dunno VB sorry):
[code] public class YourClass { private TcpClient m_Connection = null; private NetworkStream m_Stream = null;
public YourClass( ) // constructor (New in VB.NET) { try { m_Connection = new TcpClient( ); m_Connection.Connect( ... ); m_Stream = m_Connection.GetStream( ); // ... } catch { if( m_Stream != null ) m_Stream.Close( ); if( m_Connection != null ) m_Connection.Close( ); throw; } } // ... } [/code]
I think ActiveX's may well be the worst invention ever. Nothing but trouble with those beasts. Well probably the fault lies with those that chose ActiveX as technology rather than ActiveX being the problem. Even more probable, is that VB programmers use ActiveX technology for about everything, including things that don't need a UI (and ActiveX was definitely meant as a UI COM component).
Anyway, I don't really know why you would be leaking. I think Willy D. might be correct.
Kind regards, PS: I'm not too happy about how VB sees things (the old or new VB).
 Signature Tom Tempelaere.
Lucvdv - 12 Dec 2005 16:48 GMT > I think ActiveX's may well be the worst invention ever. Nothing but trouble > with those beasts. Well probably the fault lies with those that chose ActiveX > as technology rather than ActiveX being the problem. Even more probable, is > that VB programmers use ActiveX technology for about everything, including > things that don't need a UI (and ActiveX was definitely meant as a UI COM > component). My plea is "guilty" for using ActiveX because ActiveX was there when I created those OCXes for use in VB6. But I blame MS: they made it far too easy to create an OCX in VC++ 6, compared to other (windowless) COM projects.
MS engineers must have noticed it too, because VB6 came with OCX components that didn't have anything to do with a UI (winsock, comm port controls). I just modeled my code after that.
In VS6 I've always used sort of a layered approach: UI in VB6 for the ease of design and debugging, and the real work in VC++ for the flexibility and multithreading capability. I switched to plain DLL's (not even COM) instead of OCXes later, but a few of those old tarts still live and are a bit too large to consider a rewrite :(
> Anyway, I don't really know why you would be leaking. I think Willy D. might > be correct. I wish :(
The service was starting two threads to perform two distinct functions. The only thing they had in common is that they're using the same database.
So now I split it up to two separate services. The part that uses the OCX runs perfectly, no trace of anything leaking.
The other one is now 100% managed code, and feels like tring to carry water in a sieve.
The CLR profiler shows that all thread objects (per connection as in the example) just keep existing forever. I've been trying to find "how come" for hours today: some reference must still exist or GC would take care of them, but I found none.
In the code sample I posted, GC does take care of them after a while. Maybe not as soon as I'd like, but it does.
In the service it doesn't clean them up at all, and I can't image where there would be a remaining reference that doesn't exist in the sample. Forcing GC didn't solve it either (not that I expected it to, but it does shorten the time each new test has to run before I'm sure the problem still exists).
In a desperate attempt, I'm now half way through translating it from VB to C#, to see if looking at it in another language sheds some light.
TT (Tom Tempelaere) - 13 Dec 2005 10:39 GMT Hi Luc,
[...]
> > Anyway, I don't really know why you would be leaking. I think Willy D. might > > be correct. [quoted text clipped - 26 lines] > In a desperate attempt, I'm now half way through translating it from VB to > C#, to see if looking at it in another language sheds some light. That indeed is desperate. Although I prefer C# there should be no difference in run time since they both compile to IL and defer functionality to the framework. The run time environment has the GC, so I don't think that another language will solve the problem.
Without a complete repro program I can't really tell what the problem is. The only advice I can give you is to Dispose/Close as early as possible and do it for every disposable/closable object. Use the using statement in C# for disposable objects: it guarantees disposal in the face of exceptions. Anyway what I'm trying to say is to keep your finalizers empty, except for unmanaged resource cleanup (and then only as a safe-guard when you would forget to Dispose). I'm not a big fan of finalizers.
Don't forget that at the end of your Dipose method, you should tell the GC not to finalize the instance anymore. Like this (C#): GC.SuppressFinalize( this );
This means that your instance will not be put on the finalizer queue.
What I do find weird is that your program does not leak when run in a window, but when run in a service that you experience leaks. A complete repro program would be nice.
Kind regards, -- Tom Tempelaere.
Lucvdv - 13 Dec 2005 12:56 GMT > > In a desperate attempt, I'm now half way through translating it from VB to > > C#, to see if looking at it in another language sheds some light. [quoted text clipped - 3 lines] > framework. The run time environment has the GC, so I don't think that another > language will solve the problem. Problem solved, without finishing the rewrite. As I expected, it was something so small you could look at it and not notice it: a single letter.
The original service created a window to hold the OCXes, so I started it as STAThread. The second half didn't exist yet at that time.
Then later I added the second thread, and started it from the main thread (it never touched the window, so there shouldn't be any harm), without looking at or specifying the threading model.
When I split the service in two, I created a copy and removed half of the functionality, but I forgot that it was still starting as STAThread.
Changed it to MTAThread --> no more leak.
Thanks to you and Willy for the help.
Lucvdv - 13 Dec 2005 14:22 GMT > Changed it to MTAThread --> no more leak. But strange, when I create a new one that does nothing but start threads that sleep for a second and exit, it can be either MTA or STA, no leak.
Willy Denoyette [MVP] - 09 Dec 2005 20:39 GMT >> If your OCX's aren't running in a thread that pumps messages, you will >> leak [quoted text clipped - 6 lines] > Would their events ever be fired without a working message pump? > I thought not. Sure they will, OLE32 (COM) will pump message on the hidden window if you have this OCX in a thread that doesn't pump it's message queue. But this is at the unmanaged side, the problem is at the managed side, where you have your RCW (created by the CLR remember), if the RCW is running in an STA thread (the same as your COM object), you should pump messages. Note that you shouldn't create a Window and a message pump, the Framework provides a number of method that intrinsically pump the message queue (for STA threads only). One of these methods is WaitForPendingFinalizers(), another one is WaitHandle.WaitOne(). So what you need to do is call WaitForPendingFinalizers(). The reason for this all is that the finalizer thread runs in an MTA, when he needs to run the "finalize" method on the RCW, the call has to be marshaled and this only works if the STA thread (runing your RCW) pumps messages, failing to pump will block the fnalizer thread.
> Anyhow, I did call 'Application.Run' in the thread that creates the hidden > window to start it - but when I just rechecked it, I noticed that I forgot [quoted text clipped - 3 lines] > Assuming that must be it, I just recompiled and I'm now about to start a > new test where it passes the correct form.
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|