Large file upload

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Bradley Plett - 23 Mar 2005 01:35 GMT

I need to upload large files (100MB+) via a web service. The files
are typically XML or CSV files. (Yes, they compress well, and I'll
probably want to add compression later, though it's not immediately
urgent since, for now, this is all internal on high-speed links.)
I've been researching this for a while now, and I've come to the
conclusion that the best way to do this would be to implement
IXmlSerializable and stream the files. Unfortunately, I haven't
managed to figure out how to do this for file uploads. Despite my
searching, I have not been able to find a good, full example either.
I'm trying to use WSE2. Can anyone help me either by showing me some
code or pointing me to a good example? I found one reasonably good
example of the server sending a large file to the client, but the
other direction seems to be just different enough that I haven't
gotten it to work in reverse.

Any help would be appreciated!

Thanks!
Brad.

Reply to this Message

[MSFT] - 23 Mar 2005 10:11 GMT

Brad,

I wouldn't suggest to upload a file more than 100M with web service. This
may fail with connection broken or other unexpected issues. I suggest you
may consider divide the file to 10M ~ 20M and send them in batch mode, or
you may consider a FTP instead.

Luke

Reply to this Message

Bradley Plett - 23 Mar 2005 17:34 GMT

I understand that such large file transfers may be better suited to
other methods, but I would still like to learn how to do this.
Initially these transfers will all be done on a small LAN, and be
quick despite their size. Also, as I said, I hope to add compression
as soon as I can. If/when I compress them, it will be much less of an
issue. For example, I have a 90MB file that compresses to under 3MB!
However, I still need to figure out how to upload a file to the
server, and since streaming has the advantage of allowing for progress
reports, that's how I would prefer to do it.

So... any more hints?

Brad.

>Brad,
>
[quoted text clipped - 4 lines]
>
>Luke

Reply to this Message

Bradley Plett - 23 Mar 2005 17:58 GMT

P.S. Take a look at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/scal
enetchapt10.asp
under "Implementing IXmlSerializable". This is essentially what I
want to do, but there isn't enough detail.
http://blogs.msdn.com/yassers/archive/2004/11/10/255212.aspx provides
more detail, but is for downloads instead of uploads. I know I should
be able to get that one working for uploads, but I've been having
trouble with it. Here's another example of downloading:
http://www.google.ca/groups?hl=en&lr=&selm=3E422F9F.B5060053%40nospam.com.
Here's a posting from someone who appears to be from MSFT promising
some sample code to do what I want, but I can't find any follow-up to
it or the code that he mentions:
http://groups-beta.google.com/group/microsoft.public.dotnet.framework.webservice
s.enhancements/msg/ec1db686ab61d119.

So... I know I'm not the first to run into this. :-S

Brad.

>I understand that such large file transfers may be better suited to
>other methods, but I would still like to learn how to do this.
[quoted text clipped - 18 lines]
>>
>>Luke

Reply to this Message

Pril - 23 Mar 2005 10:51 GMT

You have to use FTP!

lp, pl

> I need to upload large files (100MB+) via a web service. The files
> are typically XML or CSV files. (Yes, they compress well, and I'll
[quoted text clipped - 15 lines]
> Thanks!
> Brad.

Reply to this Message

Bradley Plett - 23 Mar 2005 17:36 GMT

Whereas FTP might be a better choice, it's simply not true that I HAVE
to use FTP.

Thanks anyway!
Brad.

>You have to use FTP!
>
[quoted text clipped - 19 lines]
>> Thanks!
>> Brad.

Reply to this Message

Pril - 24 Mar 2005 15:24 GMT

Sorry I miss the point ...

Can you explain why ftp is not ok? I don't understand the nature of the
problem yet...

lp, pl

> Whereas FTP might be a better choice, it's simply not true that I HAVE
> to use FTP.
[quoted text clipped - 25 lines]
> >> Thanks!
> >> Brad.

Reply to this Message

Bradley Plett - 24 Mar 2005 18:20 GMT

The point of my last post was that I don't HAVE to use FTP. There are
a variety of ways of accomplishing this. I could also do an HTTP
upload. I could do WebDAV. For that matter, in my case I could even
do an operating system file copy. The fact remains that, due to the
nature of my application, I would prefer to be able to accept the file
via a web service. Furthermore, I would like the web service to be
hosted by my application using WSE2, as opposed to using IIS. If I
can't get that to work, I'll explore other options. However, the
comments and code samples that I've found lead me to believe that this
is possible. In fact, the code samples are tantalizingly close to
what I need already. For example, if you look at
http://blogs.msdn.com/yassers/archive/2004/11/10/255212.aspx, it is
very close to what I need. Unfortunately, it's not quite there. For
one thing it doesn't go quite far enough into describing how to
consume the service. Of course, it's also based on .NET 2.0, and it's
for downloads rather than uploads, but those could probably be worked
around.

Thanks!
Brad.

>Sorry I miss the point ...
>
[quoted text clipped - 32 lines]
>> >> Thanks!
>> >> Brad.

Reply to this Message

Jonathan Eggert - 23 Mar 2005 18:20 GMT

Brad,

I feel your pain. Way too many folks out there that want to inject their
opinion instead of an actual answer.

You CAN send large files usigng WSE. This post might be helpful.

http://devjunkies.com/?name=viewart&type=1&tid=91

It was helpful to me, but I'm not sending files quite as large as yours. I
believe that WSE also supports chunking which can reduce the problems of
large files.

http://www.codeguru.com/Cpp/I-N/internet/webservices/article.php/c8691/

Reply to this Message

Bradley Plett - 23 Mar 2005 20:49 GMT

Well, it is always easier to throw in a quick opinion rather than
taking the time to think about something. :-)

Thanks a lot for your response! That got me a lot closer. I now have
a web service running in the context of IIS that accepts large files.

I have two issues left.

1) I need to try to get the same thing working in a server application
using WSE (no IIS).

2) The method described in those posts uses attachments. This has
three downsides: a) there is still a limit to the size of the file
(though it looks to be high enough for my files right now), b) it
seems the entire attachment must be loaded into memory on both ends,
consuming significant amounts of memory, and c) no status updates. I
still feel that streaming would be a better solution, since it would
eliminate these issues, but...

The second article mentions chunking, which could also solve issue #2,
but it provides insufficient detail on implementing it. The code
sample doesn't address it at all.

Thanks again!
Brad.

>Brad,
>
[quoted text clipped - 10 lines]
>
>http://www.codeguru.com/Cpp/I-N/internet/webservices/article.php/c8691/

Reply to this Message

[MSFT] - 24 Mar 2005 04:06 GMT

Hello Brad,

Even with WSE and IXmlSerializable, we still don't recommend uploading such
a large file in one request. As we know, ASP .NET has setting for max
request length it can accept. Even we can adjust this value, but we still
recommend a vaue less than 10M~20M. As you have seen in this sample:

http://blogs.msdn.com/yassers/archive/2004/11/10/255212.aspx

It is about how download a song. I guess the size of the file will be
5M~6M. So I think FTP should be proper solution for the question and you
even can monitor the progress with it.

Luke

Reply to this Message

Bradley Plett - 24 Mar 2005 09:06 GMT

You know, I REALLY don't want to be rude, but do you read and
understand the postings before you reply? I have explained the file
size issues, and I posted that URL to this thread well before I
received this response. I also explained the issue I had with that
URL.

Frankly, I would prefer no response to this. Sorry - as I said, I
don't want to be rude. However, I would appreciate just a little
respect and the courtesy to understand the question and the issues
prior to responding to me.

Thanks!
Brad.

>Hello Brad,
>
[quoted text clipped - 10 lines]
>
>Luke

Reply to this Message

[MSFT] - 28 Mar 2005 03:57 GMT

Hi Brad,

I apologize if I misunderstood you on the question. I have studied the urls
your provided and got what you mean. But I haven't seen you mentioned the
file size issues. For downloading, we may consider chunk, but for
uploading, it cannot help here.

Luke

Reply to this Message

Rate this thread:

Large file upload

Free Magazines