 |
 | Home | Contact Us | FAQ | Search & Site Map | Link to Us |
 | Sign In | Join | Other 45 Sites in Network |
.NET Forum / ASP.NET / Web Services / December 2006MTOM and WSE3
Hi, I am trying to return a 500Mb file from IIS to a winform client via a web service. I would like to use MTOM and have installed WSE3 on the client and server. I can see there is a chunking setting in the WSE3 settings set to 64K but I am not sure how I should get the chunking to work correctly. I have used some example code for the server setup as shown below but this seems to involve reading the full 500M into a byte array before returning it so WSE can take care of the chunking for me. I can see all the server memory being used up when I run the code and the client times out. Can you provide some example code and advise to efficiently transfer the file from the server to the client using chunking without using too much memory? <WebMethod(Description:="This is the MTOM test")> _ Public Function Test16() As Byte() Dim response As Byte() Dim filepath As String = AppDomain.CurrentDomain.BaseDirectory + "download\" + filename response = File.ReadAllBytes(filepath) Return response End Function Thanks hi smarty i wrote an article on codeproject which may be of use to you. http://www.codeproject.com/soap/MTOMWebServices.asp my approach is more concerned with providing detailed progress to a winforms user interface. to assist with this, the chunking process is done by code, sending each chunk using MTOM but without the built-in MTOM chunking. the solution is memory efficient and may be helpful for your scenario. hope it helps tim > Hi, > [quoted text clipped - 26 lines] > > Thanks Hi, Thanks for that. I have looked at your codeproject article. For downloading from the server it seems to do it by making succesive web service calls. I am after evaluating built in chunking that is specified by the MTOM protocol. I am just confused because the WSE3.0 seems to provide settings for changing chunking but I can't see how to get these to work. The only examples I have seen populate a byte array with the whole file (storing it in memory) and then return the byte array. Any ideas? > hi smarty > i wrote an article on codeproject which may be of use to you. [quoted text clipped - 37 lines] > > > > Thanks hi smarty, that is correct, each chunk is sent as a separate web service call. this makes for easy feedback to the user interface. it also allows for very robust 'resume' functionality, should one of the chunks fail. i remember experimenting with the MTOM built-in chunking and i was not very impressed with it. i can't remember all the reasons why, but i found it better to 'roll my own', especially from the perspective of handling very large files. my code has scaled well for files of several gigabytes. hope this helps tim > Hi, > [quoted text clipped - 62 lines] >> > >> > Thanks Hi Tim, Thanks again for the reply I did like you code and could find it useful for the future but I am trying to evaluate chunking provided with MTOM in .NET and if I get time possilby compare to a Java implementation too. If you have any examples of what you did with the built in MTOM I would be grateful. > hi smarty, > that is correct, each chunk is sent as a separate web service call. this [quoted text clipped - 74 lines] > >> > > >> > Thanks hi smarty, sorry i don't have any actual code available. it was mostly reading from the WSE documentation and looking at the sample applications. good luck tim > Hi Tim, > [quoted text clipped - 94 lines] >> >> > >> >> > Thanks Thanks for Tim's input. Hi Smarty, As for the "chunking provided with MTOM" you mentioned, would you provide some further explanation on this? What's the configuration you've seen on this and where did you find the reference about the built-in chunk support of WSE MTOM? Based on my research, the WSE MTOM just use some special encoding rules to compact binary data, however, it will still need to load the complete data into memory and send them through a complete message. What I can find about chunk setting in WSE is internally and not exposed as public setting or interface. I'll discuss with some other WSE engineers to see whether there is a better means to do chunked data transfering, I'll update you as soon as I get any information. Sincerely, Steven Cheng Microsoft MSDN Online Support Lead This posting is provided "AS IS" with no warranties, and confers no rights. Hi Steven, I was mistaken by the setting for multipart mime thinking this set a chunk value for the streaming. With respect to chunking I was looking at the RRSHB and thought this allowed the message to be chunked, similar to DIME. Looking at the documentation for WSE, there are two examples, one where the whole message is read into a byte array before returning and the other examples utilising the IXMLSerializable interface. The documentation says the streaming support with IXMLSerializable reduces memory and the need to read the information into a byte array before returning. How does this work, does it rely on the client receiving x bytes before the server reads anymore into memory and sends? Thanks > Thanks for Tim's input. > [quoted text clipped - 24 lines] > > This posting is provided "AS IS" with no warranties, and confers no rights Hi Smarty, Thanks for your reply. Yes, we does support streaming webservice message through wrapper class that implement IXMLSerializable interface, this is not specific to MTOM data but any request/response object return or pass to webservice webmethods. And I've discussed with some other product team engineers and they've given me some sample code. If you need I can send you through email. You can reach me through the email in my signature(remove "online"). Sincerely, Steven Cheng Microsoft MSDN Online Support Lead This posting is provided "AS IS" with no warranties, and confers no rights. Hi, I have a couple of questions 1. When trying to use the streaming with large files the client returns a timeout error. I have changed both the file size and timout to -1 in web.config but the error still occurs. After the error has occured I get another error saying the type is invalid. I think this is becuase the previous request is still attempting to stream to the client on the next request. If I leave it for some time the memory on the server is released and I can continue to send smaller files successfully. Any idea how to resolve this? 2. I have seen a note saying WS-Security cannot be applied at an application level when streaming, what is the reason for this, is it due to the DOM parsing? and is this resolved in WCF? Thanks > Hi Smarty, > [quoted text clipped - 16 lines] > > This posting is provided "AS IS" with no warranties, and confers no rights Hello Smarty, The existing issue I've got from another engineer seems related to ASP.NET webservice that will report a System.Net.WebException after few thousand of calls. This does not crash the service and when caught and handled on the client side the client can start sending out messages again. We have confidence that this issue is outside WSE. Therefore, I think it maybe a bit different from your scenario. BTW, if what you encounter is a timeout issue, you can consider adjust the both the timeout at serverside clientside: **for server-side, in addition to WSE messaging's timeout setting, you can also enlarge ASP.NET <httpruntime> executionTimeout setting. http://msdn2.microsoft.com/en-us/library/e1f13641.aspx ** for client-side, the webservice client proxy class also has a "timeout" property we can configure. If you're still curious about the hotfix or still suffering the timeout issue, I suggest you contact CSS about for further troubleshooting. As far as I know, this kind of message streaming (through the IXmlSerializable wrapper class) is supported. Sincerely, Steven Cheng Microsoft MSDN Online Support Lead This posting is provided "AS IS" with no warranties, and confers no rights. I have found Microsoft's MTOM implementation included in WSE 3.0 to be useless for large data transfer. The underlying problem is that no matter how you implement your IXMLSerializable, the infrastructure STILL outputs the data into a MemoryStream awaiting transfers to the client (which, of course, may never come). I'm sure that they do this because it was the simplest way to free up the thread filling the MemoryStream buffer. However, the ramification is that you get 'out of memory' exceptions from the server. I was able to reduce the memory footprint somewhat by disabling encryption and signature of the response body. And, of course, you have to completely turn off diagnostic tracing in both the server and the client. Even so, I would regularly run out of memory on a response as small as 300 MB. Sheesh!!! If you want details about the policy that does server-client X.509 authentication without encrypting the response, just ask here or email me. I hate inventing my own wheel when there is a perfectly good specification available (MTOM), but I had to fail back to my own 5 MB chunking algorithm with repeated web service calls in order to avoid Microsft's MTOM implementation in WSE 3.0. Hi, I have been using the MTOM implementation for streaming and can see an improvment in performance. When using a simple byte array transfering a 100Mb file uses up almost all memory available on my server (800Mb available on perf mon) and I cannot send much larger files. When using the streaming I can see an improvement and have sent files upto 500Mb in size. However the larger the files being sent the larger the memory footprint used. When throughput testing I will receive timeouts if I open more than 1 web service call for a 500Mb file at a time. Is this correct, shouldn't streaming give a similar memory footprint whatever the file size? I am trying to determine if I have implemented this correctly, maybe it isn't streaming correctly? Some example memory footprints for a single web service call with a 100Mb file are... The server memory is 825Mb with no activity. When returning a normal byte array this drops to 320mb and then down to 200mb after a period. When using my streaming implementation (wse3 sample code) the memory drops from an initial 825mb to 650mb. Thanks I will appreciate any help you can give on this. > Hello Smarty, > [quoted text clipped - 31 lines] > > This posting is provided "AS IS" with no warranties, and confers no rights Hi Smarty, I think this should be due to the WSE MTOM's natural limitation, it still need to load block of data into memory when transfering large data through streaming, it is not unlimited. Have you ever contact the product support team on this issue and find any result on this? Sincerely, Steven Cheng Microsoft MSDN Online Support Lead This posting is provided "AS IS" with no warranties, and confers no rights. hi tim, i played with your sample code in http://www.codeproject.com/soap/MTOMWebServices.asp with MTOM. one question: wehre can i see that the file is really transported via MTOM? when i activate the trace function in wse i can't see anything - it's only base64 coded: <inputMessage utc="26.11.2006 14:22:53" messageId="urn:uuid:bf171798-1536-4f00-b2e5-5929177536a7"> <processingStep description="Unprocessed message"> <soap:Envelope xmlns:xop="http://www.w3.org/2004/08/xop/include" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"> <soap:Header> <wsa:Action>http://tim.mackey.ie/CodeProject/MTOM/AppendChunk</wsa:Action> <wsa:MessageID>urn:uuid:bf171798-1536-4f00-b2e5-5929177536a7</wsa:MessageID> <wsa:ReplyTo> <wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:Address> </wsa:ReplyTo> <wsa:To>http://localhost:1423/WebService/MTOM.asmx</wsa:To> <wsse:Security> <wsu:Timestamp wsu:Id="Timestamp-669ca490-d417-4c0a-a4bd-1ae7f71ec5a2"> <wsu:Created>2006-11-26T14:22:53Z</wsu:Created> <wsu:Expires>2006-11-26T14:27:53Z</wsu:Expires> </wsu:Timestamp> </wsse:Security> </soap:Header> <soap:Body> <AppendChunk xmlns="http://tim.mackey.ie/CodeProject/MTOM"> <FileName>test.kml</FileName> <buffer>PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVR</buffer> <Offset>0</Offset> <BytesRead>16384</BytesRead> </AppendChunk> </soap:Body> </soap:Envelope> </processingStep> or can't you see the MTOM transport in the trace file? greetings from germany stefan > hi smarty, sorry i don't have any actual code available. it was mostly > reading from the WSE documentation and looking at the sample applications. [quoted text clipped - 105 lines] >>> >> > >>> >> > Thanks hi stefan, sorry i haven't been watching the newsgroups for a while, i just saw your post. my understanding of MTOM is that when it is enabled for a web service on client and server, any web methods that send byte[] as parameters or return types, automatically get handled specially by MTOM, specifically that they are not xml-serialized inside the XML message, as happens on a standard web service, which is obviously very costly for both processing and bandwidth. from my point of view, this embedding of the raw binary data within the message is the most useful feature of MTOM for sending binary data over web services. it is indeed very basic but perhaps you know something i don't? did i miss something? i did a good few tests to compare the performance advantages of an MTOM enabled web service, simply by switching on or off MTOM, and i seem to recall about a 10% performance improvement over DIME and i think about 30% again over plain web services. interested to hear your response. cheers tim > hi tim, > [quoted text clipped - 185 lines] >>>> >> > >>>> >> > Thanks Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
Reply to this Message
|
 Sign In
 Join
 My Latest Posts My Monitored Threads My Blog My Photo Gallery My Profile My Homepage Start New Thread Enable EMail Alerts Rate this Thread
|
©2008 Advenet LLC Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.