Is there a reason why you can't use .Net data objects and execute the query
directly from VB.Net code?  It sounds like ExecuteFunction does exactly the
same as the ExecuteScalar method of a .Net SqlCommand object.

Imports System.Data.SqlClient
[...]
Dim cn As New SqlConnection(ConnectionString)
cn.Open
Dim cmd As New SqlCommand("Your query", cn)
Dim x As Object = cmd.ExecuteScalar()

I could be wrong on some or all points, but I see more than one problem
here.

First, I think you'll have to pass the query differently: Char is a single
character, not a pointer to a string as the function expects.
I've never tried, but I think it will pass the ASCII value of the first
character of your query instead of a pointer to the string.

Second, but I'm really not sure here, I think "auto" assumes that the
function exists in ExecuteFunctionA and/or ExecuteFunctionW variations in
the DLL (as most API functions that take strings do), use either 'ansi' or
'unicode' depending on what the DLL expects.  Again, I've never tried, but
I think "auto" may append either an A or a W to the function name.

Third, I don't know if AsAny works on return values, but try this:

Private Declare Ansi Function ExecuteFunction Lib "database.dll" ( _
   ByVal sSQL As String) As <MarshalAs(UnmanagedType.AsAny)> Object

I think it will tell you that you can't use AsAny on a return value though.
Even if you can, the error message makes me expect you'll still get the
same error because returning variants is really not supported.