Large Size XML files for data transfer

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

RAJ - 08 Jul 2004 14:28 GMT

Hi,
Need some advice with regards to XML files. Here is what I am trying to
achieve:

I have an SQL Server 7.0 database and a Web service running on a Windows
2003 enterprise machine (for development). The web service (developed using
VB.NET) creates an XML file which contains most of the database table. The
creation of this file takes less then couple of minutes. This XML is about
57 MB in size and is used to build a local database. The file is then zipped
to about 2.5 MB and sent to the calling application.

On the client the file is unzipped to 57 MB file, before the application
starts reading it. I am using the load method of the DOM object model to
check whether it is a well formed XML. The problem starts as soon as I try
to load this XML file. The client machine becomes unresponsive for a long
time and the whole process from there on is very slow as I assume that the
XML is loaded into memory while it is parsed. I have to read the whole XML
file and do not need to search for any nodes/tags inside the XML.

I would like to know whether it is right to use XML in this scenario? Am I
better off by generating a simple delimited file instead of an XML file? The
size of the XML file is at least 3 folds as compared to any other delimited
file.

Any advice in this regard will be helpful.

Thanks and regards,

RAJ

Reply to this Message

Kondratyev Denis - 08 Jul 2004 15:59 GMT

> Hi,
> Need some advice with regards to XML files. Here is what I am trying to
[quoted text clipped - 14 lines]
> XML is loaded into memory while it is parsed. I have to read the whole XML
> file and do not need to search for any nodes/tags inside the XML.

Use asynhrone calls of web service methods for avoid unresponsive
application.

> I would like to know whether it is right to use XML in this scenario? Am I
> better off by generating a simple delimited file instead of an XML file? The
> size of the XML file is at least 3 folds as compared to any other delimited
> file.

If u store some table with different structure imho xml file is better
solution.

Reply to this Message

Iain - 08 Jul 2004 17:00 GMT

Personally, if you are sending a single table, I would send a flat file of
some kind - CSV is fine.

However, if you want to use XML then I would use an XMLReader rather than
an XMLDocument. This should be dramatically faster (I'm presuming that the
ultimate database target is NOT an XMLDocument).

I would be tempted to including the unzipping as part of the reading
processing by using zlib or something. However, a quick glance at the docs
suggest this will not be straightforward.

Iain

> Hi,
> Need some advice with regards to XML files. Here is what I am trying to
[quoted text clipped - 25 lines]
>
> RAJ

Reply to this Message

RAJ - 08 Jul 2004 18:05 GMT

Thanks for your reply.
I am sending a part of the database with a few tables. The advantage of
using XML is that it is readable. However, I can change it to use either
tab-separated or CSV format.

The webservice is called asynchronously with the implementation of Zipping
and unzipping using third party tool (AbaleZip) at both the webservice and
client side respectively.

The problem occurs when I use the load command :
dim objXML as XMLDocument
objxml.load(<XML file name>)

The application hangs when reading the file on the client machine.
During development however, the load command is successful but then windows
is out of virtual memory and the reading of the XML file from there on is
very slow!

Never used XMLReader before. But it looks like that I have to use it as it
is a forward only reader which is what I am looking for as I need to read
whole data sequentially. Will I be able to use it without loading the XML?

I have already developed both client and web service part and now in the
process of testing on deployment machine.

If there are any examples of using XMLReader in VB..NET, it would be helpful

Thanks for your help.

Regards,

RAJ

> Personally, if you are sending a single table, I would send a flat file of
> some kind - CSV is fine.
[quoted text clipped - 42 lines]
> >
> > RAJ

Reply to this Message

Kondratyev Denis - 08 Jul 2004 21:10 GMT

Imho Iain mean XmlTextReader class, becauce XmlReader is abstract class and
u cant use it for work with XML (rtfm). U can find examples in .NET
Framework SDK,
http://msdn.microsoft.com/vcsharp/downloads/samples/23samples/default.aspx ,
http://support.microsoft.com/default.aspx?scid=kb;en-us;301225 or in Google.

> Thanks for your reply.
> I am sending a part of the database with a few tables. The advantage of
[quoted text clipped - 28 lines]
>
> RAJ

Reply to this Message

RAJ - 09 Jul 2004 08:29 GMT

Thanks a lot. You help is much appreciated.

regards,
RAJ

> Imho Iain mean XmlTextReader class, becauce XmlReader is abstract class and
> u cant use it for work with XML (rtfm). U can find examples in .NET
[quoted text clipped - 37 lines]
> >
> > RAJ

Reply to this Message

Iain - 09 Jul 2004 09:28 GMT

> Imho Iain mean XmlTextReader class, becauce XmlReader is abstract class and
> u cant use it for work with XML (rtfm). U can find examples in .NET
> Framework SDK,
> http://msdn.microsoft.com/vcsharp/downloads/samples/23samples/default.aspx ,
> http://support.microsoft.com/default.aspx?scid=kb;en-us;301225 or in Google.

Sorry. Yes. - my earlier point was it should be possible to write an
XmlZippedTextReader class (and it's counterpart) which would be useful..

However, I'm not volounteering.

Iain

Reply to this Message

Large Size XML files for data transfer

Free Magazines