Hi...

I've got a console app that fetches, reads, and processes text files and
I've been trying to profile it to improve performance.  Based on its
configuration file, it supports multithreading with a ThreadPool.

I've tried profiling with Compuware's Devpartner Studio, but I find the
debugger gets in the way too much; it induces about 100x overhead and only
accounts for about 70x of it, so you find the results skewed pretty badly by
frequently executed lines.  Plus, with multiprocessors sometimes it just
flakes out.

I've been demoing ANT and it's been a great help.  I've also been using some
of the perfmon tips and tricks.

The nub of what I've found out is that the performance characteristics
between running single threaded and multi-threaded are pretty radically
different, and the major part of the difference comes in GC handling.  Single
threaded, it runs at about 4-7% of the time in GC.  Running with 5 threads,
it's between 45-90% of the time in GC.

My guess is that so much of the GC action is from the strings created by
stream reading.  I've been trying to cut down on all of the extra string
creation from String.Split() and redundant Substring calls and I've gotten a
lot of them out but it's a little trickier with the strings from I/O.

I've created a subroutine to effectively find the lines in a Byte [] (to
replace .ReadLine()), but getting from a subsection in a byte array to make
string of the parts I want seems a bit trickier.  HttpUtility.UrlDecode()
supports subsection from bytes to string, but StringBuilder doesn't.  Do I
have to write my own utility routines to cast up each Byte to Char or
StringBuilder.Append each byte individually?

Any suggestions from others would be appreciated...

Thanks
_mark