.NET Forum / .NET Framework / New Users / October 2006
doing some background work occasionally
|
|
Thread rating:  |
bonk - 26 Oct 2006 07:55 GMT I have an application that needs to perform some background work, i.e. Logging, wich must not block the main thread. How would I basically design such a scenario? It is obvious that I should do that on an extra thread but I think it is a bad idea to spawn a new thread everytime a log-message is written and let it die once the message was written. But how do I keep a thread alive and trigger the log-messages and pass the string to that thread? Also, should I give that thread a lower priority? Maybe there is a best practice for this kind scenario?
Peter Duniho - 26 Oct 2006 08:01 GMT > [...] But > how do I keep a thread alive and trigger the log-messages and pass the > string to that thread? Also, should I give that thread a lower > priority? Maybe there is a best practice for this kind scenario? I believe that the .NET worker thread pool will address what you want.
(non-existent newsgroup removed from Newsgroups: field)
bonk - 26 Oct 2006 08:41 GMT Thank you for that tip. One questions though:
1. If an exception occurs inside a thread of a queued workitem how can I rethrow that exception on the main thread?
> > [...] But > > how do I keep a thread alive and trigger the log-messages and pass the [quoted text clipped - 4 lines] > > (non-existent newsgroup removed from Newsgroups: field) Marc Gravell - 26 Oct 2006 08:54 GMT That sounds more like an async callback... in which case, create a delegate to a method, and call BeginInvoke on the delagate, also providing the callback delegate (to a method e.g. on your form). In the callback, you can then use the Form's BeginInvoke etc to get back to the UI thread.
Marc
Peter Duniho - 26 Oct 2006 08:59 GMT > Thank you for that tip. One questions though: > > 1. If an exception occurs inside a thread of a queued workitem how can > I rethrow that exception on the main thread? I'm not really sure what that would mean, exactly. Since the exception doesn't occur on the main thread in that case, and the main thread would be happily going along executing its own code, at what point would you have the main thread get interrupted with the exception? How should the main thread be expected to recover from the exception, given that the exception would have nothing to do with what the main thread actually was doing?
IMHO, a much better approach would be to come up with some signaling mechanism that your threads can use to notify the main thread of an error, and have the main thread check that now and then. Or maybe even better, have the worker thread, upon catching a thrown exception, use Invoke or BeginInvoke to run a delegate on the main thread, causing the main thread to process the error in whatever way you feel is appropriate.
Pete
bonk - 26 Oct 2006 09:34 GMT The scenario is this: as soon as any excpetion occurs in one of the threads the thread will be finished (there is one big try{}catch{} inside the thread's callback. In case an exception occured the main thread must be resonsible of handling the excpetion. This is a requirement by design of the main application.
> Or maybe even better, > have the worker thread, upon catching a thrown exception, use Invoke or > BeginInvoke to run a delegate on the main thread, causing the main thread to > process the error in whatever way you feel is appropriate. This sounds like a good idea for my scenario. I now need to find out how to run the delegate on the main thread an pass the exception correctly.
> > Thank you for that tip. One questions though: > > [quoted text clipped - 16 lines] > > Pete Jon Skeet [C# MVP] - 26 Oct 2006 10:20 GMT > > Or maybe even better, > > have the worker thread, upon catching a thrown exception, use Invoke or [quoted text clipped - 4 lines] > how to run the delegate on the main thread an pass the exception > correctly. Use Control.Invoke or Control.BeginInvoke.
See http://www.pobox.com/~skeet/csharp/threads/winforms.shtml for more details.
Jon
Robert Ludig - 26 Oct 2006 10:31 GMT Sorry, I forgot to mention that it a console application. How would I do it in this case? Invoke a delegate on a main thread from another thread?
> > > Or maybe even better, > > > have the worker thread, upon catching a thrown exception, use Invoke or [quoted text clipped - 11 lines] > > Jon Peter Duniho - 26 Oct 2006 20:03 GMT > The scenario is this: as soon as any excpetion occurs in one of the > threads the thread will be finished (there is one big try{}catch{} > inside the thread's callback. In case an exception occured the main > thread must be resonsible of handling the excpetion. But what do you mean by "handle the exception"?
Typically, that phrase is used to describe the use of try/catch/finally to provide a point at which the code can recover from the exception. But that only will occur on the thread on which the exception occurred.
I suppose you could devise some way to duplicate the specifics of the exception thrown, and then provide a bottleneck on the main thread where you throw the duplicate. But I don't see the point in doing that, since all the main thread will (presumably) do at that point is run some error-handling code and continue normally. In that case, it makes more sense for the main thread to simply run the error-handling code and be done with it, rather than trying to emulate in the main thread an exception that happened on a different thread.
> [...] >> Or maybe even better, [quoted text clipped - 6 lines] > how to run the delegate on the main thread an pass the exception > correctly. IMHO, what you really want to know is how to extract the information that you need from the exception, and pass *that* to the main thread. That said, I suppose you could pass the exception itself. I'm just not sure what the point of that would be, and it could tempt you into trying to rethrow the exception on the main thread, which I think would be a mistake.
As Jon said, you can use Invoke or BeginInvoke to execute the delegate on the main thread. You write that you have a console application, but that doesn't preclude having a message pump. So one option is to go ahead and run a message pump in your console application.
If that's not a suitable solution for you, then it seems to me you could easily emulate the BeginInvoke behavior by creating a queue into which you place your own "events" to be handled, and from which the main thread retrieves for processing. Of course, by doing that you'd basically be reinventing the wheel, and I'd think you'd be better off just putting a message pump in your console application. But you could do it that way.
Pete
ssamuel - 26 Oct 2006 20:22 GMT > But what do you mean by "handle the exception"? > [quoted text clipped - 10 lines] > than trying to emulate in the main thread an exception that happened on a > different thread. To elaborate and add some friendly amendments, exceptions come in different flavors:
If your exception is, "some resource that I expected isn't available," the main thread may be able to fix that by restarting the resource and re-spawning the background process. This requires that the main thread know how to and be capable of restarting failed processes.
If the exception is, "the user gave me some data to process that has errors," the main thread -- a UI thread, probably -- would be the only one who could fix it, generally by informing the user that their data doesn't compute and asking them to retry. This requires saving user input and letting the user modify it and resend.
If the exception is, "something I didn't expect happened," chances are, no one's going to be able to fix that. It should bubble up, therefore being caught at the thread boundary and possibly rethrown (you're right: it's dangerous to just rethrow, but also sometimes appropriate) or repackaged and thrown upwards. A good example of this is running out of memory or disk space. Rarely can you do more than tell the user what happened.
This is part of the reason one should avoid catching Exception. Instead, it's better to catch specific exceptions and know what each one means. On a background thread boundary, however, it's often useful to catch Exception after everything else, just in case something slips through, so it's not lost in ether.
On the other hand, if you're not going to do anything other than kill the background process if there's an exception, you might as well not catch it. Don't catch anything you're not going to use, and never do the following:
try { ... } catch { throw; }
That's nothing more than a waste of the overhead of a try/catch block. If nothing else, put some diagnostic information somewhere so that someone who knows what they're doing can inspect it and deduce what happened.
Stephan
bonk - 27 Oct 2006 07:54 GMT Thank you for the detailed replies. Just to make things more clear, all I actually wanted to do is, in case in one of the backround threads occurs an excpetion I want to terminate that thread and the main thread (it has to be the main thread that does that) needs to write details of the exception to a logstream. What would be a good approach to implement this. Is introducing the afore mentioned messagepump the best/only option I have?
> > The scenario is this: as soon as any excpetion occurs in one of the > > threads the thread will be finished (there is one big try{}catch{} [quoted text clipped - 46 lines] > > Pete Peter Duniho - 27 Oct 2006 08:41 GMT > Thank you for the detailed replies. Just to make things more clear, all > I actually wanted to do is, in case in one of the backround threads [quoted text clipped - 3 lines] > implement this. Is introducing the afore mentioned messagepump the > best/only option I have? Using the worker thread pool, you don't really terminate the thread so much as you simply stop doing work on it. That part is easy...just make sure you have an exception handler in the worker method itself, where you can catch the exception and exit the method if one occurs.
As far as notifying the main thread of that, you can use any of the methods suggested here already. Since what you *really* want is simply to have some data written to a logfile, you could just create a data queue to which the worker threads write if an exception occurs (before exiting their worker method, of course), and from which the main thread reads for the purpose of generating your log file. It's probably not necessary to get so complicated as creating delegates and invoking them on the main thread.
Pete
Brian Gideon - 27 Oct 2006 16:15 GMT > > [...] But > > how do I keep a thread alive and trigger the log-messages and pass the [quoted text clipped - 4 lines] > > (non-existent newsgroup removed from Newsgroups: field) Peter,
In this case I don't think the ThreadPool will work because it will process the log messages concurrently. There's no guarentee that they will be recorded to the storage media in the order they were queued. That's typically a fundamental requirement for any logging component.
Brian
Peter Duniho - 27 Oct 2006 18:43 GMT > In this case I don't think the ThreadPool will work because it will > process the log messages concurrently. There's no guarentee that they > will be recorded to the storage media in the order they were queued. > That's typically a fundamental requirement for any logging component. Please read the whole thread. My response was in answer to the question of "how do I keep a thread alive". The original poster was clearly mostly concerned with how to avoid creating a new thread over and over again, and that was the point of suggesting the thread pool.
As far as ordering the logs, as I've pointed out elsewhere, my suggestion is not to actually output the logged data from the worker thread, but rather to queue the data in a shared location so that the main thread can write the logged data out.
Nevertheless, the fact is that as long as the work is being done amongst multiple threads, it is impossible to ensure that data logging happens sequentially, whether the actual output is done from each thread, or delegated to some other single thread. The only practical way to ensure that the logged data is in order is to time-stamp the data so that whatever component writes the data out can make sure it is written in the correct order, and even using that method, there exists a theoretical possibility that unless ALL of the logged data is sorted based on the timestamp once ALL of the processing is done, that the logged data will still be out of order.
Otherwise, no matter the logging mechanism, there is always the chance that just before one thread tries to log some data, another thread will be given its timeslice, generate an event (error), and log that event before the first thread gets a chance to run again.
The question of how to ensure that the logged data is in order is completely independent of the question of what thread mechanism is used. As long as *any* thread mechanism is used, out-of-order data logging is a problem. Using the thread pool doesn't make this better of course, but nor does it make it worse, as compared to any other way of running tasks on a thread.
Pete
Marc Gravell - 26 Oct 2006 08:48 GMT I would look at a queue, accessed (for instance) via a static/singleton pattern. Access to the queue would be sync'd. callers would * lock * enqueue string * check count; if ==1 then Monitor.Pulse as have just restarted the Q * unlock The (single) logging thread (started in the static/singleton ctor) would * lock * check count * if == 0 * Monitor.Wait * unlock * back to the top * else * dequeue * unlock * log * back to the top
This way, you have a single thread that stays alive, but doesn't hammer the CPU while the Q is empty; callers only block to enqueue, not to log. You might also add something to allow graceful teardown of the thread (without Thread.Abort), and I'd probably make the logging thread a background thread anyway...
Marc
Chris Fulstow - 26 Oct 2006 14:18 GMT Check out the ThreadPool class: http://msdn2.microsoft.com/en-us/library/system.threading.threadpool.aspx
> I have an application that needs to perform some background work, i.e. > Logging, wich must not block the main thread. How would I basically [quoted text clipped - 4 lines] > string to that thread? Also, should I give that thread a lower > priority? Maybe there is a best practice for this kind scenario? Brian Gideon - 26 Oct 2006 22:34 GMT bonk,
The best way to do this is by using a special kind of queue. Marc already touched on this, but let me explain some more. This queue's Dequeue method will block when it's empty. This is known as a blocking queue. It is notoriously difficult to implement. I've seen more failed attempts at this than I can count. But, once you have it everything else is pretty easy.
Basically, you'll create a dedicated logger thread. That thread will spin around in an infinite loop calling the queue's Dequeue method. Remember, the Dequeue method will block when empty so most of the time the thread will be idle. When an item appears the Dequeue method will return and you can process the returned object accordingly and then call Dequeue again.
All of the other threads in your application will be enqueueing the log message contents into the queue.
If you choose to use this approach then I highly recommend you model your queue off of Jon's producer-consumer example. Producing is the same as enqueueing and consuming is the same as dequeueing so it should be pretty easy if you follow his example exactly.
(Look for the ProducerConsumer class) http://www.yoda.arachsys.com/csharp/threads/deadlocks.shtml
Brian
> I have an application that needs to perform some background work, i.e. > Logging, wich must not block the main thread. How would I basically [quoted text clipped - 4 lines] > string to that thread? Also, should I give that thread a lower > priority? Maybe there is a best practice for this kind scenario? jcreasy - 27 Oct 2006 14:25 GMT Brian:
A thread queue would fit the specific needs here, but it would take more work and I wonder if it gives any advantages to the ThreadPool approach. A dedicated thread for background processing I have found to only be efficient enough for use when there are enough tasks to keep the thread busy.
If instead you have a handful of short and quick tasks to accomplish that are all relatively the same thing, it seems to me that the ThreadPool makes more sense (not to mention most of the code is already in place). This after all is why the ThreadPool was put into .NET.
Please let me know if I am missing something or making an incorrect assumption about your approach.
> bonk, > [quoted text clipped - 33 lines] > > string to that thread? Also, should I give that thread a lower > > priority? Maybe there is a best practice for this kind scenario? Brian Gideon - 27 Oct 2006 16:10 GMT > Brian: > [quoted text clipped - 11 lines] > Please let me know if I am missing something or making an incorrect > assumption about your approach. Typically I would I agree with you. The problem I see with the ThreadPool is that the order the log messages are processed would be nondeterministric because it's using multiple threads. It's usually a requirement to process log messages serially instead of concurrently so that they get written to a file (or whatever) in time order.
Brian
Peter Duniho - 27 Oct 2006 18:56 GMT > Typically I would I agree with you. The problem I see with the > ThreadPool is that the order the log messages are processed would be > nondeterministric because it's using multiple threads. Using a shared queue doesn't fix that. Logging to the queue can still be nondeterministic, as long as the work is still being done using multiple threads.
> It's usually a > requirement to process log messages serially instead of concurrently so > that they get written to a file (or whatever) in time order. I'm not sure about "usually". I'll agree with "sometimes". :) In any case, we haven't been given the requirement or lack thereof one way or the other.
Pete
jcreasy - 27 Oct 2006 21:04 GMT Very true. I was going under the assumption that these log messages rarely needed to be entered. You could also get around that problem by including a DateTime object into the entry, but a seperate dedicated thread would help ensure the timing of your log entries. Another possible problem with the ThreadPool is there are only 25 threads I believe available to use, but that restriction I also assumed would not be a problem in this setting. Thanks for your thoughts Brian.
> > Brian: > > [quoted text clipped - 19 lines] > > Brian jcreasy - 27 Oct 2006 22:41 GMT Very true. I was going under the assumption that these log messages rarely needed to be entered. You could also get around that problem by including a DateTime object into the entry, but a seperate dedicated thread would help ensure the timing of your log entries. Another possible problem with the ThreadPool is there are only 25 threads I believe available to use, but that restriction I also assumed would not be a problem in this setting. Thanks for your thoughts Brian.
> > Brian: > > [quoted text clipped - 19 lines] > > Brian Peter Duniho - 27 Oct 2006 18:53 GMT > [...] > Basically, you'll create a dedicated logger thread. That thread will [quoted text clipped - 6 lines] > All of the other threads in your application will be enqueueing the log > message contents into the queue. You appear to be suggesting to simply use a queue for the logging itself. However, you haven't suggested any mechanism by which the queue itself will be guaranteed to be in order.
Now, it happens that the original poster never suggested that the logged messages need to be in exactly the same order in which they occurred. It seems likely that if he's dealing with multiple concurrent tasks that are not otherwise synchronized with each other, then it doesn't really matter whether the logged output from those tasks is ordered as well.
*But*...if ordering the logged messages *is* important, as you seem to be assuming, then simply implementing a queue for the logging doesn't resolve the out-of-order issue. It just moves it from the i/o part of the code to the queueing part of the code.
Queueing the *work* itself in a single thread would resolve the ordering issue with the logged data (as would timestamping the data, as long as it's sorted once all processing is done), but a) the original poster hasn't suggested that the logged data needs to be in order, and b) the original poster hasn't suggested that it's suitable for each work item to be done serially (he may prefer that a relatively shorter work item started after a longer one be allowed to run concurrently, so it can complete before the longer one does).
Pete
Brian Gideon - 28 Oct 2006 04:21 GMT > You appear to be suggesting to simply use a queue for the logging itself. > However, you haven't suggested any mechanism by which the queue itself will > be guaranteed to be in order. I was suggesting the use of a FIFO queue. It's impossible for such a queue to store items in anything but temporal order.
> Now, it happens that the original poster never suggested that the logged > messages need to be in exactly the same order in which they occurred. It [quoted text clipped - 6 lines] > the out-of-order issue. It just moves it from the i/o part of the code to > the queueing part of the code. You're right. The OP never said that was a requirement. I made the leap myself because it's a reasonable assumption to make. It would be weird for the application to do something in a particular order and then report that it happened in another.
> Queueing the *work* itself in a single thread would resolve the ordering > issue with the logged data (as would timestamping the data, as long as it's [quoted text clipped - 4 lines] > longer one be allowed to run concurrently, so it can complete before the > longer one does). I'm not understanding where sorting comes into play or why the OP would not want log messages to appear in the log in the order they occurred. Regardless, if the OP doesn't care about the order then certainly the ThreadPool would be the easiest solution.
Brian Gideon - 28 Oct 2006 04:48 GMT > I was suggesting the use of a FIFO queue. It's impossible for such a > queue to store items in anything but temporal order. I just realized what point you were making. You're right, if there are multiple threads producing log messages then it would be *very* difficult to guarentee ordering across all threads. It would not be too difficult to make the guarentee within a thread though. It would certainly require more than a trivial queue implementation though. But, I didn't get the impression from the OP that more than one thread would be producing log messages. In fact, with all of the talk about a "main" thread and using the ISynchronizeInvoke methods it sounded like only 1 was in play. It could very well just be me though :)
Peter Duniho - 28 Oct 2006 06:00 GMT > I just realized what point you were making. You're right, if there are > multiple threads producing log messages then it would be *very* > difficult to guarentee ordering across all threads. I'm glad you now understand what I was saying. :)
> It would not be > too difficult to make the guarentee within a thread though. But it's not difficult to make the guarantee within a thread when writing directly to a log file either. The question of whether errors are logged directly to a file or put into an in-memory queue first is orthogonal to the question of how to ensure that the logged entries are in the correct order. As far as ordering goes, any issues that exist with respect to logging to a file also exist with respect to logging to queue (and vice a versa).
> It would > certainly require more than a trivial queue implementation though. Well, I consider timestamping the log entries to be a pretty trivial solution. Other than that, it's pretty much *impossible* to ensure the queue entries are ordered. Because there's no way to cause an error and the logging of that error to be an atomic operation, there is *always* the possibility that one thread will be interrupted between an error occurring and the error being logged.
Note that even the timestamping solution doesn't really completely guarantee the logged events are in the correct order, since a thread could even finish its timeslice just before retrieving the time for the logged error.
On the bright side, as I mentioned before, when one has multiple threads operating concurrently doing work independent of each other, it would be *highly* unusual for anyone to care that errors (or other events) logged by each individual thread are put into the log in precisely the time order in which they occurred. Only when the threads are somehow working together is it likely someone would care about the exact order of logged events, and in that case, the threads can also work together to ensure that order (and yes, you're right, in that case the implementation of the logging queue would be non-trivial).
> But, I didn't get the impression from the OP that more than one thread > would be producing log messages. If you say so. :) Personally, I think it a bit hard to see why you would say that, at the same time that you talk about the problems of keeping a log in order. The latter problem occurs only when there is more than one thread logging. But I'll take your word for it that somehow you had both mutually exclusive circumstances in mind. :)
Pete
Brian Gideon - 29 Oct 2006 11:44 GMT > > It would not be > > too difficult to make the guarentee within a thread though. [quoted text clipped - 5 lines] > As far as ordering goes, any issues that exist with respect to logging to a > file also exist with respect to logging to queue (and vice a versa). I did some more thinking on this. I misspoke. My solution already already guarentees relative ordering for every thread. That's because it's impossible for a thread to enqueue messages out of order relative to itself. And since there's only one thread dequeueing it's impossible for them to be dequeued out of order.
> On the bright side, as I mentioned before, when one has multiple threads > operating concurrently doing work independent of each other, it would be [quoted text clipped - 5 lines] > you're right, in that case the implementation of the logging queue would be > non-trivial). Hmm...I sort of agree. I do disagree on one important point. No one cares that thread A races with thread B when logging. It doesn't matter that if A does something first and then B does its thing next that B's log message appears first in the log. What people do care about is that A's log messages are written to the log in the order that they occurred. For example, if A performs tasks 1 and 2 then the log message for 1 should be written before the log message for 2. A solution using the ThreadPool won't guarentee that because the persisting of log messages can be dispatched to different threads. Contrast that with my solution where log messages are dispatched to a single thread.
A little code might help.
public class Logger { private Thread thread; private BlockingQueue queue;
public Logger() { queue = new BlockingQueue(); thread = new Thread(this.ThreadMethod); thread.Start(); }
public void Log(string message) { queue.Enqueue(message); }
private void ThreadMethod() { while (true) { string message = queue.Dequeue(); // Persist the message to a file, database, etc. } } }
Notice that since there is only one thread removing from the queue the order that messages are received by the Logger is the order they are persisted. And it's impossible for any specific thread to queue its messages out of order.
Brian
Peter Duniho - 29 Oct 2006 23:32 GMT > I did some more thinking on this. I misspoke. My solution already > already guarentees relative ordering for every thread. That's because > it's impossible for a thread to enqueue messages out of order relative > to itself. No message logging implementation should ever have a problem with message *from a single thread* being out of order. Someone would have to go to *extra* work to make that a possibility.
So I don't really see what you're trying to say here. You might as well say that a thread never has to worry about the program statements executed by that thread ever executing out of order (ignoring for a moment CPU implementations that do just that). It's trivially true, but not all that interesting or useful to know.
In any case, whatever guarantees you can make using a "queue message, write to log file later" implementation, you can just as easily make using a "write to log file immediately" implementation. The underlying implementation doesn't affect the question of what order log entries occur in.
> [...] > Hmm...I sort of agree. I do disagree on one important point. No one > cares that thread A races with thread B when logging. It doesn't > matter that if A does something first and then B does its thing next > that B's log message appears first in the log. That's exactly what I said. How are you disagreeing with me?
> What people do care > about is that A's log messages are written to the log in the order that > they occurred. Yes, they do care about that. However, that happens naturally in any typical message logging implementation. Since the statements within a given thread always execute in order, it is trivial to ensure that logged messages from a given thread are always in order.
> For example, if A performs tasks 1 and 2 then the log > message for 1 should be written before the log message for 2. A > solution using the ThreadPool won't guarentee that because the > persisting of log messages can be dispatched to different threads. I have no idea why you think that the "persisting of log messages" would be "dispatched to different threads". Certainly no one here has suggested anything like that. You'd have to go to extra work to do that. The threads aren't present for the purpose of logging messages...they are present for the purpose of doing work. Any messages they log will necessarily occur in the correct order, relative to each thread's own work.
> Contrast that with my solution where log messages are dispatched to a > single thread. Why dispatch a log message to a thread at all? Messages should be logged to a data structure, if not written directly to a file, shared (and synchronized, of course) by all threads using it.
> A little code might help. > [...] > Notice that since there is only one thread removing from the queue the > order that messages are received by the Logger is the order they are > persisted. And it's impossible for any specific thread to queue its > messages out of order. You could just as easily replace your queue and thread with a single "Log" method that does the "Persist the message to a file, database, etc" work you have delegated to a whole new thread. It would work just as well, from a message ordering standpoint.
Pete
Brian Gideon - 30 Oct 2006 04:08 GMT > No message logging implementation should ever have a problem with message > *from a single thread* being out of order. Someone would have to go to > *extra* work to make that a possibility. But, that's the exact problem you get by using a ThreadPool implementation for an asynchronous logger.
> So I don't really see what you're trying to say here. You might as well say > that a thread never has to worry about the program statements executed by > that thread ever executing out of order (ignoring for a moment CPU > implementations that do just that). It's trivially true, but not all that > interesting or useful to know. I don't think it's that trivial. But anyway, I wanted to mention it to eliminate any confusion.
> In any case, whatever guarantees you can make using a "queue message, write > to log file later" implementation, you can just as easily make using a > "write to log file immediately" implementation. The underlying > implementation doesn't affect the question of what order log entries occur > in. I don't think so. ThreadPool.QueueUserWorkItem will queue the message and write later, but do so in an unpredictable order. A "write to log file immediately" approach would always result in a correctly ordered file.
> > [...] > > Hmm...I sort of agree. I do disagree on one important point. No one [quoted text clipped - 3 lines] > > That's exactly what I said. How are you disagreeing with me? I apologize. I misunderstood what you said then. In fact, I went back and reread your post and can see that we agree on this.
> > What people do care > > about is that A's log messages are written to the log in the order that [quoted text clipped - 4 lines] > thread always execute in order, it is trivial to ensure that logged messages > from a given thread are always in order. We're not discussing a typical logger implementation though.
> I have no idea why you think that the "persisting of log messages" would be > "dispatched to different threads". Certainly no one here has suggested > anything like that. You'd have to go to extra work to do that. The threads > aren't present for the purpose of logging messages...they are present for > the purpose of doing work. Any messages they log will necessarily occur in > the correct order, relative to each thread's own work. The log messages have to be dispatched to some asynchronous mechanism for them to be written asynchronously without blocking the application thread. The ThreadPool is an excellent mechanism for asynchronous processing which dispatches work items to several threads. However, it doesn't guarentee a completion order for those work items. That's the issue I have with it.
> Why dispatch a log message to a thread at all? Messages should be logged to > a data structure, if not written directly to a file, shared (and > synchronized, of course) by all threads using it. Because the OP wants an asynchronous logger. There's no way that I know of to prevent one thread from blocking without the use of another thread, thread pool, fiber, IO completion port, or some other type of asynchronous mechanism.
> You could just as easily replace your queue and thread with a single "Log" > method that does the "Persist the message to a file, database, etc" work you > have delegated to a whole new thread. It would work just as well, from a > message ordering standpoint. That would block the logging thread which is what the OP wanted to prevent. It would satisfy the ordering requirement (which I claim is necessary), but wouldn't satisfy the asynchronous requirement.
Peter Duniho - 30 Oct 2006 10:23 GMT > Because the OP wants an asynchronous logger. I have no idea where you got that idea. The original poster was asking about how best to do "some background work occasionally" (see the subject line, if not his original post). There has been no mention whatsoever about the *logging* itself occurring asyncronously. Only that the worker threads should be able to log errors. In fact, so far the *only* event that the OP has stated a need to log is an error that would end the worker thread's work; obviously this need not be asynchronous.
Frankly, I'm a bit dubious that one could do any better creating new threads (or using a thread pool) to log data than to simply use some synchronized object (queue, file, whatever). Especially an in-memory queue, but even a file, is not going to remain blocked for very long. The time it would take to start up a new thread, or even to unblock an existing one and get it to start running some code is easily in the same ballpark as, if not worse than, the time a thread would be expected to take to add a new bit of data to a queue or write it to disk (remember...file i/o is normally cached...even if you're dealing with a slow i/o device, it is not likely that the thread doing the writing is actually going to have to wait for that as long as the data is reasonably small, which it would have to be if the programmer expects to get decent performance out of ANY logging mechanism).
But, even if you make the assumption that running the *logging* itself on a different thread is desirable in some situations, there's nothing about this thread that suggests that's the design goal here.
> [...] > That would block the logging thread which is what the OP wanted to > prevent. Again, I've seen nothing in this thread to suggest that the OP wants to avoid blocking the logging thread.
Pete
Marc Gravell - 30 Oct 2006 10:27 GMT > There has been no mention whatsoever about the *logging* itself occurring > asyncronously see the first post:
OP> i.e. Logging, wich must not block the main thread.
Logging was only an example
Marc
Marc Gravell - 30 Oct 2006 10:32 GMT And for the grammar police: OK - fair enough "i.e." does not mean this is an example; I inferred (from context) that this meant "e.g."...
Marc
Brian Gideon - 30 Oct 2006 15:05 GMT > > Because the OP wants an asynchronous logger. > [quoted text clipped - 5 lines] > has stated a need to log is an error that would end the worker thread's > work; obviously this need not be asynchronous. The OP did specifically mention that logging should not block the main thread.
> Frankly, I'm a bit dubious that one could do any better creating new threads > (or using a thread pool) to log data than to simply use some synchronized [quoted text clipped - 8 lines] > as long as the data is reasonably small, which it would have to be if the > programmer expects to get decent performance out of ANY logging mechanism). Yes, that I can definitely agree with. I'm also unclear as to why asynchronous logging is a requirement. Most people write logs to the console, file, or event log which are both reliable and fast. But, I have seen some who like to write logs to a file on a network share, database, or some other remote resource. In that case an asynchronous logger could be considered imperative.
> But, even if you make the assumption that running the *logging* itself on a > different thread is desirable in some situations, there's nothing about this [quoted text clipped - 6 lines] > Again, I've seen nothing in this thread to suggest that the OP wants to > avoid blocking the logging thread. Strange. I thought the theme of an asynchronous logger pervaded the OP's entire post.
Peter Duniho - 30 Oct 2006 23:08 GMT > [...] > Strange. I thought the theme of an asynchronous logger pervaded the > OP's entire post. Well, I didn't. I do see how the word "logging" got included as something the worker thread would do, and I admit that I was not focusing on that with respect to my replies. But I think "pervade" is overstating things, and the fact that I didn't pick up on that didn't appear to concern the original poster in my replied.
That said, I agree that if the original poster intends to use the worker threads only for logging, *and* if he intends to assign a new thread each time he wants to log something, he's heading for trouble, and in exactly the way you suggest.
Thanks, Pete
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|