Hi Dan

An error has crept into my last post. The HelloWorld web service implementation should be:
        [WebMethod]
        public string HelloWorld()
        {
            string msgStr = "This is a quote \" or this is a lesser than symbol < or these are german Umlaute ?, ?, ?";
            throw new SoapException( msgStr,new XmlQualifiedName("Server.Problem"));
        }

Regards
Dany

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """, "<", ">", "?","?", etc.

The way I understand, we have no influence as to how the SOAP response is created here. So I don't understand what we could do differently. Please elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP response and falsly converts the characters in the fault string to XML twice. The fact that a pure .net consumer can't even handle converting the fault string back to a correct string seems to prove, that it can't be correct.

We have done a workaround (=hack) now, assuming that MS won't be able to fix the bug quick enough. We've created a SOAP Extension that takes the SOAP response and undoes the wrongly converted characters. But this is by no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
        [WebMethod]
        public string HelloWorld()
        {
                                               string msgStr = "This is a quote: ""
            throw new SoapException( "Now I've got a problem \"here\". ><",new XmlQualifiedName("Server.Problem"));
        }
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it TestWebService)
5. Add a button and add the following code for it's click handler
        private void button1_Click(object sender, System.EventArgs e)
        {
            try
            {
                TestWebService.Service1 appserv = new TestWebService.Service1();
                string result = appserv.HelloWorld();
                MessageBox.Show( result );
            }
            catch (Exception exc)
            {
                                                               MessageBox.Show( exc.Message );
            }
        }
6. Now run the client and after pressing the button1 you will see a message box with the incorrectly converted error message. If you use a proxy to analyse the SOAP request and response you can see that the fault string in the SOAPFault is: This is a quote &amp;quot; or this is a lesser than symbol &amp;lt; or these are german Umlaute &amp;#246;, &amp;#228;, &amp;#252;
This is of course the wrong XML representation and should actually be: This is a quote &quot; or this is a lesser than symbol &lt; or these are german Umlaute &#246;, &#228;, &#252;

Regards
Dany

Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire.  What it will do is take an XML node and insert
it into the proper place in a fault.  It sounds like there may be the case
where the code you are using to set up the XML is double converting.  This
could happen if you try and treat a string with embedded angle brackets in
it like XML.  Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
    <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
    <#1p7txJ3EHA.4068@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
    correcty
Message-ID: <et$A51P3EHA.1524@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1        
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP09
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (?, ?, ?) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum f?r das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&amp;#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the &amp; is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
f&#252;r das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ? character is converted to a valid character reference "&#252;".  
But then this is replaced by character references once again which results
in "&amp;#252;" which is of course not a valid XML representation of the
"?" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
.net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML.  Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire?  UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy.  For the most part, 1.1
fixed bugs.  It is possible that it introduced some as well.  I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
    <hzdWUQ92EHA.3200@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
    correcty
Message-ID: <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1        
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP15
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
            try
            {
                localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
                string result = appserv.HelloWorld();
                MessageBox.Show( result );
            }
            catch (Exception exc)
            {
                                                             
MessageBox.Show( "Error has occured: " + exc.Message );
            }

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method &quot;HelloWorld&quot; is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany

Hi,

What you are describing is the use of encoded strings.  These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data?  Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1        
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP14
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (?, ?, ?) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

 string errMsg = "Here is a \"quote\".";
 throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <soap:Body>
   <soap:Fault>
     <faultcode>Server.Problem</faultcode>
     <faultstring>Here is a &quot;quote&quot;.</faultstring>
     <detail />
   </soap:Fault>
 </soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
     <faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergn?gen". It is converted incorrectly to
<faultstring>Fahrvergn&amp;#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ?,?, ?, ? in them and these are
not correctly processed.

Thanks for any help!
Dany

Hi Dany,

You can manuallyl supply the XML for the fault node.  The issue is
converting a unicode string to XML.  When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.  
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that).  And populating the fault yourself.  I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
    <et$A51P3EHA.1524@TK2MSFTNGP09.phx.gbl>
    <034VKAV4EHA.768@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
    correcty
Message-ID: <#f4Owvd4EHA.3644@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1        
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!tk2msftngp13
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "?","?", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
        [WebMethod]
        public string HelloWorld()
        {
                                               string msgStr = "This is a
quote: ""
            throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
        }
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
        private void button1_Click(object sender, System.EventArgs e)
        {
            try
            {
                TestWebService.Service1 appserv = new TestWebService.Service1();
                string result = appserv.HelloWorld();
                MessageBox.Show( result );
            }
            catch (Exception exc)
            {
                                                               
MessageBox.Show( exc.Message );
            }
        }
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &amp;quot; or this is a lesser than
symbol &amp;lt; or these are german Umlaute &amp;#246;, &amp;#228;,
&amp;#252;
This is of course the wrong XML representation and should actually be: This
is a quote &quot; or this is a lesser than symbol &lt; or these are german
Umlaute &#246;, &#228;, &#252;

Regards
Dany

Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire.  What it will do is take an XML node and insert
it into the proper place in a fault.  It sounds like there may be the case
where the code you are using to set up the XML is double converting.  This
could happen if you try and treat a string with embedded angle brackets in
it like XML.  Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
    <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
    <#1p7txJ3EHA.4068@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
    correcty
Message-ID: <et$A51P3EHA.1524@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1        
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP09
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (?, ?, ?) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum f?r das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&amp;#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the &amp; is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
f&#252;r das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ? character is converted to a valid character reference "&#252;".  
But then this is replaced by character references once again which results
in "&amp;#252;" which is of course not a valid XML representation of the
"?" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
.net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML.  Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire?  UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy.  For the most part, 1.1
fixed bugs.  It is possible that it introduced some as well.  I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
    <hzdWUQ92EHA.3200@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
    correcty
Message-ID: <eDppBzF3EHA.2316@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1        
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP15
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
            try
            {
                localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
                string result = appserv.HelloWorld();
                MessageBox.Show( result );
            }
            catch (Exception exc)
            {
                                                             
MessageBox.Show( "Error has occured: " + exc.Message );
            }

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method &quot;HelloWorld&quot; is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany

Hi,

What you are describing is the use of encoded strings.  These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data?  Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Dany@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uCfPi7f2EHA.3132@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1        
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP14
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (?, ?, ?) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

 string errMsg = "Here is a \"quote\".";
 throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <soap:Body>
   <soap:Fault>
     <faultcode>Server.Problem</faultcode>
     <faultstring>Here is a &quot;quote&quot;.</faultstring>
     <detail />
   </soap:Fault>
 </soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
     <faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergn?gen". It is converted incorrectly to
<faultstring>Fahrvergn&amp;#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ?,?, ?, ? in them and these are
not correctly processed.

Thanks for any help!
Dany