Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / XML / November 2003

Tip: Looking for answers? Try searching our database.

xml document

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
androger - 19 Nov 2003 18:28 GMT
hi,

I have a huge xmldocument that represent data from a
table. How can I break this down into multiple
xmldocuments, say of a certain number of records each?
Christoph Schittko [MVP] - 20 Nov 2003 04:08 GMT
The least memory intensive solution would be to read over the source
document with an XmlTextReader and switch to a different output ( presumably
a file ) every so often.

<pseudoCode>

XmlTextReader reader = new XmlTextReader( ... );
XmlWriter writer = new XmlTextWriter( GetNextFilename() )
int pageSize = 100; // chunk every 100 rows
int currentRow = 0;

// Init the writer
writer.WriteStartElement( "myRoot" ); // because you can't write fragments
with the XmlTextWriter

reader.Read() // skip over the top level element

while( reader.Read() )
{
   if( 0 == ((++currentRow)%pageSize) )
   {
       writer.WriteEndElement();
       writer = new XmlTextWriter( GetNextFileName() );
   }
   writer.WriteNode( reader, false );
}

writer.WriteEndElement();
</pseudoCode>

Using the Reader / Writer combination you never have to read the entire XML
document into memory, which was probably the concern in the first place.

Alternatively, you might shred the XML document into SQL Server tables using
SQLXML bulkload and then use queries to retrieve subsets of the data.

Is there any way you can just reduce the size of the XML document in the
first place or are you just receiving those huge documents.

Signature

HTH
Christoph Schittko [MVP, XmlInsider]
Software Architect, .NET Mentor

> hi,
>
> I have a huge xmldocument that represent data from a
> table. How can I break this down into multiple
> xmldocuments, say of a certain number of records each?
androger - 20 Nov 2003 14:23 GMT
Thanks for replying!

The main problem is that I have a webservice method that
takes a xmldocument as a parameter. it works well, but
when that document is huge like 5 MB, I get this error:

The underlying connection was closed: An unexpected error
occurred on a receive.

I tried to change the size of the request in
machine.config, but no change! So my solution was to break
that xml file into something the web service can handle
and make multiples calls.

>-----Original Message-----
>The least memory intensive solution would be to read over the source
[quoted text clipped - 43 lines]
>
>.
Vadim Chekan - 21 Nov 2003 03:35 GMT
> Thanks for replying!
>
[quoted text clipped - 9 lines]
> that xml file into something the web service can handle
> and make multiples calls.

And you'll have another problem: you'll have to store chunks somwhere to
give them out on request.
Than for some reasons client can make request, pick up the first portion
of data, and don't request other ones, for fail reson, quit application,
connection problem, etc. So your temporary storage will grow with
garbage information.
Then you'll have to match somehow user session id with chunks set, and
you'll have to figure out somehow which chunk is going to be next.
Than you'll have to join chunks on client.
Aha, and how client will know either there are more chunks to retrieve?
Looks ugly.

You'd better spend some time on tracing and optimizing application.
5Mb shouldn't be so performance killing.

Vadim Chekan.
Christoph Schittko [MVP] - 28 Nov 2003 22:00 GMT
What exactly did you change in machine.config. Did you make all the changes
outlined in [0]?

Signature

HTH
Christoph Schittko [MVP]
Software Architect, .NET Mentor

[0]  http://www.dotnet247.com/247reference/msgs/7/36455.asp

> Thanks for replying!
>
[quoted text clipped - 66 lines]
> >
> >.

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.