What really stumps me is that (remember) this melting
pot of varying technologies in my window service """"works""""
in Debug mode from the IDE. I compile, it runs - external
exe and all. But, when I use installutil and make it an proper
windows service, the external exe breaks. This feels like a
security problem but that is just a gut hunch.

Thanks again.

Never got this email.

Anyway, what you should do is, ask yourself whether this must be driven by
a service, if the answer is YES then you should get rid of all
features/technologies that really aren't designed to be used from a service.
Basically you have three possible options, but be warned, none of them are
trivial.
1)
The most important thing is,  that you should get rid of the VB exe, it
doesn't solve anything, if you can successfully access the COM interfaces
(implemented in the C/C++ DLL's) from VB6, then they should be directly
accessible from a WS as well.
Keep in mind, that the COM components (the DLL's that call into DirectShow)
must be accessed from an STA thread, and that STA threads *must* pump a
message queue, that is, they should not block. This is especially true
running managed code, blocking STA's may block the finalizer thread, and
this is a big NO NO in managed code.
So, your service thread you should fire off a thread that gets initialized
to enter a STA, this thread should run a message queue, that is, you should
create an empty form, create an instance of the COM object(s) and call
Application.Run().
The service thread could (should) use Windows messages to pass command to
the STA/pumping thread. That means that you must define and register private
Windows message ID's for each command that corresponds to a COM method call
(you could also define one single message and pass the command as message
parameter), you must handle these messages in your WndProc override.
To communicate back with the service thread, you can use a
SynchronizationContext, the service thread must register a delegate to get
called when an event is fired by the pumping STA thread.
The STA pumping thread must implement the callbacks required by the COM
object(s).

2) Get rid of the VB6 exe and of the C/C++ DLL's. This requires a rewrite of
the code in the C dll's in C#. The CS code must declare/implement the COM
interfaces and call into DirectShow just like was done in the C/C++ code.
Note that you can also call into the Media Format API's, but it everything
depends on what exactly you are doing in the C/C++ COM dll's.

3) Write a C++/CLI wrapper that wraps the existing COM dll's, note that this
wrapper must run his own message queue exactly like explained in 1).

Willy.

Ok, so far so good ;-)
What makes you think the exec is not started, is the process not visible in
the process list? If not, how does the program signals success/failure to
the parent process (the service in this case)?
How do you start the exec, show us the code that start the exec.
Give also some details about the exec, type of program, what security and
user context does it require to run in, does it expect a user profile to be
loaded?

Willy.