I found a solution, but I have not found the reason for the problem.
I uninstalled .NET framework 2.0 on the client PC (my test PC) and let my
installer re-install it, this has cured the current issue on my test
machine. Why did this happen and how did this happen I will probably never
know -- the error message provided by the Install (MSIEXEC) is pathetic at
best. I'm going to do more testing because I'd like to duplicate this
problem since I've already raised a bug on M$ feedback. I know M$ don't
give a rats a.s, but I hope that this might help others that have struggled
with this turd of a development platform called VS 2005.
Rob.
For some reason that I am unaware of, Visual Studio decided to install
services in a rather convoluted way. There's a call from the msiexec process
to InstallUtilLib.dll passing a bunch of parameters, and then calls on .NET
Installer classes that locate your assembly, find its Installer classes,
instantiate them, call methods that install the service etc. As you can
probably imagine, this is rather complicated and sometimes fragile. However
Windows Installer has bult-in support for installing services (the
ServiceInstall and ServiceControl tables) that people used for years before
.NET came along, so why Visual Studio invented installer classes to install
services is a mystery to me.
Many installation developers who have been building setups for a while
consider managed custom actions a bad thing:
http://www.installsite.org/pages/en/msifaq/a/1044.htm
I wonder if this is the first time you've built an MSI-based setup. There
seem to be a lot of Visual Studio developers doing this since VS
setup&deployment projects came along and, although the lack of
InstallUtilLib diagnostics is a pain, it's not the only (and probably not
even the best) tool for building MSI-based setups.
http://www.installsite.org/pages/en/msi/authoring.htm

Signature
Phil Wilson [MVP Windows Installer]
----
>I found a solution, but I have not found the reason for the problem.
>
[quoted text clipped - 38 lines]
>> 2. Don't "release" dev tools that are clearly BETA
>> 3. Firing VS 2005 team members doesn't help me here and now
Rob R. Ainscough - 05 Jul 2006 21:40 GMT
Hi Phil,
The clue that lead me to think there was more wrong with .NET framework
installation was when I tried to uninstall another .NET 2.0 project that had
installed just fine a few days ago -- my attempt to remove via Control Panel
resulted in the same InstallUtilLib.dll error. The only activity that
happened between these states was the uninstall and re-install of .NET 1.1.
This seems to have created some problems for .NET 2.0 installer. Fragile is
very true. However, I've looked into other tools such as InstallShield and
have found that many folks are experiencing the same or similiar problems
with InstallUtilLib.dll for their .NET 2.0 projects being distributed via
the InstallShield projects.
I'm not a dedicated "Installation Developer" -- I'm really the ONLY
developer that works on ALL our .NET projects (web, click once, windows
services, windows forms, etc.).
I've built many MSI setups before.
> For some reason that I am unaware of, Visual Studio decided to install
> services in a rather convoluted way. There's a call from the msiexec
[quoted text clipped - 61 lines]
>>> 2. Don't "release" dev tools that are clearly BETA
>>> 3. Firing VS 2005 team members doesn't help me here and now
Phil Wilson - 06 Jul 2006 16:34 GMT
It won't matter what tool you buiild an MSI file with if something has
broken the InstallUtilLib internals. However you don't need InstallUtilLib
to install services, and tools from InstallShield, Wise, Wix etc let you
install services using the standard MSI Service table instead of using
InstallUtilLib. All I can do is point out that managed custom actions can be
fragile, and if you're willing to use another tool to do things like install
services, and write custom actions that call a C++ Dll you can completely
avoid the fragility of calling managed custom actions.

Signature
Phil Wilson
[Microsoft MVP-Windows Installer]
Definitive Guide to Windows Installer
http://apress.com/book/bookDisplay.html?bID=280
> Hi Phil,
>
[quoted text clipped - 80 lines]
>>>> 2. Don't "release" dev tools that are clearly BETA
>>>> 3. Firing VS 2005 team members doesn't help me here and now
Rob R. Ainscough - 06 Jul 2006 18:31 GMT
Understand Phil, but I'm trying to stay within the context of VS 2005 and
get things done the VS 2005 way (and I'm staying 100% managed). The concept
is good, just the implementation is bad, in some cases really bad. That
aside, poor error messages that provide no insight is a significant problem
with VS 2005, MSI, or just about anything under the M$ umbrella. NULL NULL
NULL -- at the very least they could provide the parameter list or what was
expected or some clues, NULL can be a clue, but by itself in this context it
is completely useless.
I've coded several Windows Services now, and my best diagnostics for when
problems occur is my extra effort I've put into trapping errors --
especially with multi-threaded processing, sometimes the only way to
identify a problem is proper error handling. M$ apparently teach this
concept put do not implement it which should concern anyone developing on an
M$ platform -- "just enough to meet a deadline" isn't a professional
approach and certainly not what I would expect from M$.
> It won't matter what tool you buiild an MSI file with if something has
> broken the InstallUtilLib internals. However you don't need InstallUtilLib
[quoted text clipped - 88 lines]
>>>>> 2. Don't "release" dev tools that are clearly BETA
>>>>> 3. Firing VS 2005 team members doesn't help me here and now