I think in custom action you cannot call external file which is not part
of your deployment project, such as
[WindowsDIR]\Microsoft.Net\Framwork\v1.1.4322\InstallUtil.
If such file is part of your setup project you can call it as I
described before:
1. In Solution Explorer, select your setup project (click on its name).
Right click, context menu appears.
2. Click the View submenu and select Custom Actions.
3. Right click Install in the tree and Add custom action. You can select
a file to execute and its arguments. You can do the same in Uninstall
section.

But for you there is still an easy way. VS .NET has predefined custom
actions including registration of services. I don't know much about it
but you can check the following help topic:
ms-help://MS.VSCC/MS.MSDNQTR.2003FEB.1033/vsintro7/html/vbconpredefinedcustomactions.htm
(type the whole line including ms-help: prefix in the URL bar in VS .NET
or just search for "Adding Predefined Custom Actions in the Custom
Actions Editor" in VS .NET help)
Sorry I cannot help you more on this topic.