 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / .NET Framework / New Users / June 2007System.Threading.Timer is intermittently late
I have an application which uses many instances of System.Threading.Timer. This application is housed inside an HTTPAsyncHandler. When I receive certain types of messages, I need to take an action 90 seconds later. For each session I have a System.Threading.Timer created with infinite values for the two intervals.. When one of these messages is received, I Change() the timer so that the first interval is 90 seconds - the second is set to Infinite. For the most part, the timers work just fine. However, I notice that rather frequently (maybe 1 in 100), the timer takes much longer than 90 seconds to fire - sometimes 100 sec, sometimes up to 150 sec. It looks like the timer does fire - it's just late. This late firing is causing all sorts of other problems with the app. There does not seem to be a correlation with load - sometimes this happens when there is only one active session, and nothing else is happening in the meantime. Of course it only happens in production, not in development or test. The production machine at the moment is a virtual server running under win2003 web edition. SQL Server Express is also running on that server. I've seen the KB article about Timers not firing in some circumstances under load, but that appears to apply only to .NET 1.1. (I'm running 2.0). Any suggestions as to what might be going on or how I can fix this?  Signature...Mike > [...] > For the most part, the timers work just fine. However, I notice that [quoted text clipped - 4 lines] > > Any suggestions as to what might be going on or how I can fix this? Nothing specific. Two things though. First, I don't see any need to create a timer with an infinite value. Just start and stop a given timer as needed. I would be surprised if this is related to your problem, but you never know. Second, if you expect a useful response, you need to post a concise-but-complete code sample that reliably reproduces the problem. Often, in the process of distilling your code down to the bare minimum required to reproduce the issue, you will discover what it is you're doing in your own code that is causing the problem. At a minimum, doing so and posting the results to the newsgroup provides context to anyone who might be able to help. Pete Peter, thanks for your reply. I suspect you're confusing this timer with one of the other variants of timer in the CLR. With a System.Threading.Timer, there are no start and stop methods. You effectively turn it on or off by using the Change method. I'd be happy to post a simple example, but there's no way - as I say, this problem doesn't manifest itself in development or test environments, but only in production. It's a pretty complicated app, and it only happens intermittently. I've been trying to reduce it, but with no luck. That's why I'm asking if others have seen similar behaviors. Signature...Mike > > [...] > > For the most part, the timers work just fine. However, I notice that [quoted text clipped - 18 lines] > > Pete > I suspect you're confusing this timer with one of the other variants of > timer in the CLR. [...] Quite so. So I was at least correct in assuming that had nothing to do with your problem. That said, the rest of my reply is accurate, as difficult as it may be for you to accomplish. There are a variety of reasons why a timer may appear to be late, but I doubt that any of them actually involve a problem in the Timer class itself. The most likely reason that comes to mind is that you are using the same thread pool used by the Timer class for other things, and that at the moment the timer should be signaled, there are not any idle threads available to handle the timer. In this case, the timer delegate will be queued for execution until the thread pool has an idle thread available for the timer. Note that the timer may not be the first one in line to get the next idle thread. But if you don't have a reproducible scenario, then even if someone has seen similar behavior it may or may not be due to similar causes. The best thing would be to identify what in your code is causing the behavior. This can often be accomplished by gradually removing or disabling areas of the code, until the problem goes away...even in complicated code there is usually a way to accomplish this, resulting in performance that wouldn't be acceptable in the real world but at least helps diagnose the issue. Pete Hi Mike, Actually, this type of production timer late issue is tough to troubleshoot in newsgroup. Looking at the nature of this issue, it would require intensive troubleshooting which would be done quickly and effectively with direct assistance from a Microsoft Support Professional through Microsoft Product Support Services. You can contact Microsoft Product Support directly to discuss additional support options you may have available, by contacting us at 1-(800)936-5800 or by choosing one of the options listed at http://www.microsoft.com/services/microsoftservices/srv_support.mspx. Personally, I agree with Peter's reply that this late may be caused by no free threads in thread pool for invoking the timer callback. However, without definite diagnostic and debugging, it is hard to see this is the real root cause. So Microsoft PSS should be a better support channel for you. Thanks for your understanding. Best regards, Jeffrey Tan Microsoft Online Community Support ================================================== Get notification to my posts through email? Please refer to http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif ications. Note: The MSDN Managed Newsgroup support offering is for non-urgent issues where an initial response from the community or a Microsoft Support Engineer within 1 business day is acceptable. Please note that each follow up response may take approximately 2 business days as the support professional working with you may need further investigation to reach the most efficient resolution. The offering is not appropriate for situations that require urgent, real-time or phone-based interactions or complex project analysis and dump analysis issues. Issues of this nature are best handled working with a dedicated Microsoft Support Engineer by contacting Microsoft Customer Support Services (CSS) at http://msdn.microsoft.com/subscriptions/support/default.aspx. ================================================== This posting is provided "AS IS" with no warranties, and confers no rights. What's the chance that you timer isn't firing because all 25 thread pool threads are busy doing something? You mentioned that you "Many Instances", so it sounds as if that could be it. If all 25 threadpool threads are busy, there's no way for the timer (which calls you back on a threadpool thread) can do anything. SignatureChris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP http://www.coversant.com/blogs/cmullins >I have an application which uses many instances of System.Threading.Timer. > This application is housed inside an HTTPAsyncHandler. When I receive [quoted text clipped - 27 lines] > > Any suggestions as to what might be going on or how I can fix this? I thought that might be the case, even though I was able to catch it once when virtually nothing was happening in the application. I added some logging that reports ThreadPool.GetAvailableThreads. I seem to always get 99 worker threads and 100 completion port threads available. Is that the appropriate thing to test? and why are the values 99, rather than 24? Could the fact that this is running under virtual server be relevant? Signature...Mike > What's the chance that you timer isn't firing because all 25 thread pool > threads are busy doing something? [quoted text clipped - 34 lines] > > > > Any suggestions as to what might be going on or how I can fix this? Well, the Completion Port threads are something else entirely. They're realted to Async I/O, and they're not relevant. As for the value of 25 vs 99, that's not really much of a clue either. The '25' value no longer applies to the Thread Pool, as documented by Duffy at: http://www.bluebytesoftware.com/blog/PermaLink,guid,ca22a5a8-a3c9-4ee8-9b41-667d bd7d2108.aspx ... but if you're having minor stuttering problems, the CLR Thread Pool may be saying, "I've got 5 threads, and I think that's enough". Then realizing it needs another one, and creating it. I'm not sure what that delay is. I don't think this is the case, but it's a possability. SignatureChris Mullins, MCSD.NET, MCPD:Enterprise, Microsoft C# MVP http://www.coversant.com/blogs/cmullins >I thought that might be the case, even though I was able to catch it once > when virtually nothing was happening in the application. [quoted text clipped - 54 lines] >> > >> > Any suggestions as to what might be going on or how I can fix this? > [...] > ... but if you're having minor stuttering problems, the CLR Thread Pool > may > be saying, "I've got 5 threads, and I think that's enough". Then > realizing > it needs another one, and creating it. I'm not sure what that delay is. From reading the docs, my understanding is that the thread pool maintains a small margin of idle threads, up to a specified maximum number of total threads. It only checks the count of idle threads every half second, and so I suppose if you had code that quickly started consuming threads in the thread pool, then you could get delays even without reaching the thread pool's maximum number, since you'd incur the cost of creating the thread needed to service the timer event handler before the thread could be run. How that would lead to a timer being delayed by tens of seconds, I can't say. I wouldn't call the original problem description a "minor stuttering problem". :) But you could easily get smaller delays, maybe even as long as a second depending on what else is going on, just from not having a thread idle and waiting to run the timer event. (It should go without saying that on a heavily loaded system, where the timer event handler code has been swapped out of physical RAM, all manner of delay could occur, easily tens of seconds or longer depending on what other i/o might be going on. I'm assuming that such a trivial explanation has already been ruled out by the OP). Pete Not sure, but I don't like changing the interval on an existing timer object, I have seen some weird issues with that and just prefer to side step any issue all together. Just create a new timer as needed like so: // Need to start some work in 2 seconds now. System.Threading.Timer t = null; t = new System.Threading.Timer( delegate(object state) { t.Dispose(); Console.WriteLine("My stuff here after 2 sec."); }, null, 2000, -1); If this helps, then we found the issue. If not, then most likely hitting TP max as others have said. That would be easy to diag a Max issue with a simple interlocked counter var. Also, might you be making blocking calls in the callback? If possible, you should make no blocking calls in the callback or make sure they are very short. If you need to block, then fire that work on another thread or a seperate queued worker thread pool. SignatureWilliam Stacey [C# MVP] |I have an application which uses many instances of System.Threading.Timer. | This application is housed inside an HTTPAsyncHandler. When I receive certain [quoted text clipped - 19 lines] | | Any suggestions as to what might be going on or how I can fix this? I'll try destroying and re-creating a new timer - thanks for the idea. I was reusing the timer, since I was a little worried about extra work for the GC et al. But I also tried another simpler experiment: I fired off a new thread (not a pool thread) and just wrote a simple loop with a sleep(2000), then logging a message. This loop seems to follow the same pattern as the timer - every once in a while, ther's a gap of 10-15 seconds! That seems to exonerate the thread pool. blocking in the callback is another idea - in all of these cases, I am logging to the database, so it is conceivable that I am blocking there. But I've been logging the durations of the database calls as well, and nothing takes longer than a few ms. I would find it hard to believe that a DB call was taking 10 seconds. but I'll try doing the logging via an async call, see if that makes a difference. thanks Signature...Mike > Not sure, but I don't like changing the interval on an existing timer > object, I have seen some weird issues with that and just prefer to side step [quoted text clipped - 49 lines] > | > | Any suggestions as to what might be going on or how I can fix this? Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.