 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / CLR / June 2005CLR Deadlock?
Hi, I am diagnosing a strange problem in .NET service (written in C#, framework 1.1 with sp1 installed - 1.1.4322.2032). Service is multhithreaded (heavily) and takes huge load (in CPU time, memory consumption and IO load). Service uses remoting to expose some interfaces and makes a lot of calls to unmanaged C dlls. After some time, approx 10 days of real havy load (100% cpu usage almost constant), service is totally locked. All threads hang. Memory consumption is OK, service process is there, yet there is no response. 0% processor load, stops listening to remoting requests (telneting to remoting port yields "connection refused" - service is not listening on the port anymore or cannot accept connection). I also log any unhandled exceptions (installed UnhandledExceptionEventHandler), no exception thrown. I cannot rule out deadlock in my code, but as far as I checked, it is unlikely that all threads can deadlock (there are some threads that periodicly log system status that should not aquire locks). Any info is appreciated - has anybody noticed such strange things under really heavy load?. Possibly GC blocked or similar (I would at least like to rule out .NET bug). Number of allocation/deallocation is really heavy in the system. I have checked MSDN and see some problems with deadlock, all claimed to be solved with 1.1 sp1. Remoting is my best suspect for now, as there is no timeout on remoting call (TCP binary channel), which could potentially create such problems. Regards, Goran YOu will have to attach a debugger to the process to figure out what is going wrong. It could be a whole host of things, and debugging the process is the first step to getting a resolution on this. Also, since you are not able to connect to the process using telnet, it could mean that the machine is out of ports. Can you do a "netstat -n -P TCP" to see what is the connection state on that machine?  Signatureferoze ----------------- This posting is provided as-is. It offers no warranties and assigns no rights. See http://weblogs.asp.net/feroze_daud for System.Net related posts. ---------------- > Hi, > I am diagnosing a strange problem in .NET service (written in C#, [quoted text clipped - 34 lines] > Regards, > Goran > YOu will have to attach a debugger to the process to figure out what is > going wrong. It could be a whole host of things, and debugging the process > is the first step to getting a resolution on this. I know, but the problem is that this is remote installation which I have limited access to and also this is not easily reproducible in lab. I am also aware of the fact that there are lot of things which could lead to such behaviour and I'm not claiming there is bug in .NET framework, rather I just wanted to check whether somebody has noticed such deadlock (obviously, track this bug will consume lot of time and effort). > Also, since you are not able to connect to the process using telnet, it > could mean that the machine is out of ports. Can you do a "netstat -n -P > TCP" to see what is the connection state on that machine? I did netstat, I connected with VNC and networking was functioning OK (I mapped drive also). Only thing I noticed it that there are multiple idle connections on remoting port (checked with TCPView also, CONNECTED state). There were no suspicious connections (eg. LAST_ACK, CLOSE_WAIT state etc.). It seems that when you disable/enable network connection, remoting opens new connection and old one is never released, I have yet to check this. We had lot of problems with remoting as there is no timeout on TCP channel. Timeout on remoting call through TCP channel would be nice - we had to move some calls to new thread and abort the thread on timeout which is bad to do. Anyway, I have added one thread that will just touch one file every few minutes. That thread does not acquire any locks, so it should not deadlock. If it continues to work after rest is locked, I'll guess classical deadlock in the app. I plan to remove all calls to lock(...) with my own lock mechanisam that will throw on timeout, which should at least prevent deadlock. Pitty that lock keyword does not support this... Regards, Goran > > Hi, > > I am diagnosing a strange problem in .NET service (written in C#, [quoted text clipped - 34 lines] > > Regards, > > Goran Hi, Reproduced it in the debugger (started through DEV studio), but unfortunatelly, IDE got totally locked also. Could not break execution (or do anything else in the ide). All threads locked, even keepalive thread that does not hold any locks. Very strange... There were few sockets in CLOSE_WAIT state, few connected though other end was shutdown. Regards, Goran > YOu will have to attach a debugger to the process to figure out what is > going wrong. It could be a whole host of things, and debugging the process [quoted text clipped - 3 lines] > could mean that the machine is out of ports. Can you do a "netstat -n -P > TCP" to see what is the connection state on that machine? You may try to use adplus (install with windbg) and try to take a hangdump to figure it out why your service hangs. How to use ADPlus to troubleshoot "hangs" and "crashes" http://support.microsoft.com/default.aspx?scid=kb;en-us;286350 Cris > Hi, > Reproduced it in the debugger (started through DEV studio), but [quoted text clipped - 17 lines] >> could mean that the machine is out of ports. Can you do a "netstat -n -P >> TCP" to see what is the connection state on that machine? Thnx, I have istalled new version of debugger but still I cannot attach to process. It complains about other thread holding NT loader lock. I'll try with adplus next hang, as unfortunatelly the process exited while I tried to attach (obviously clicked wrong button). Though search on "NT loader lock" returned some interesting articles which I have to check. Regards, Goran > You may try to use adplus (install with windbg) and try to take a hangdump > to figure it out why your service hangs. ... Hi, After few days of debugging and searching, it looks like http://support.microsoft.com/?kbid=839343 could be my problem. It seems it is not included in SP1 (not listed on http://support.microsoft.com/?kbid=867460). Regards, Goran > YOu will have to attach a debugger to the process to figure out what is > going wrong. It could be a whole host of things, and debugging the process [quoted text clipped - 3 lines] > could mean that the machine is out of ports. Can you do a "netstat -n -P > TCP" to see what is the connection state on that machine? ... Inline Willy. > Hi, > After few days of debugging and searching, it looks like [quoted text clipped - 3 lines] > could be my problem. It seems it is not included in SP1 (not listed on > http://support.microsoft.com/?kbid=867460). The v1.1 SP1 build version is 2032, the fix has build number 1031, so these fixes should be included. Don't trust the list of SP1 fixes, not all fixes are included in the KB lists. > Regards, > Goran [quoted text clipped - 8 lines] >> TCP" to see what is the connection state on that machine? > ...
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.