Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / .NET Framework / Compact Framework / March 2004

Tip: Looking for answers? Try searching our database.

Data Insertion

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Dave - 24 Mar 2004 10:56 GMT
Hi All,

I am working with C#, SQL CE and ADO.net using the .net compact framework on a PDA.

I need to populate quite large tables from XML.
Initially I just used Dataset.ReadXML but this quickly ran into memory problems with large amounts of data.
I am now using XmlTextReader to read the data in piecemeal fashion.

It has been suggested to me that I read a single record from the XML and then insert each record one by one and work my way through the XML.
This seems far too clumsy.
I would rather extract a certain amount of data to a dataset and then update the table regularly.
Is this possible or is there another simpler more efficient way of adding the data?

Thanks,
Dave
"Ilya Tumanov [MS]" - 24 Mar 2004 20:01 GMT
Dave,

First of all, how large are your data tables and how big is the XML file?

Next, are you using schema to load your XML? If you do not have schema,
inference is used.
This involves loading entire XML file into XmlDocument, which takes huge
amount of memory and a lot of time.

If you do have a schema (loaded from schema file, embedded into XML or
created programmatically), DataSet.ReadXml() will use XmlTextReader to load
records one by one and add them to the DataSet, so you won't have to.
However, data will be stored in memory anyway, so if you have a lot of
records, it might be just too much.

Using XmlTextReader and adding records directly to SQL CE is the way to go
in this case.

Best regards,

Ilya

This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
> Thread-Topic: Data Insertion
> thread-index: AcQRhjr1NH4yZoruSRyv/2H+WPk1pw==
[quoted text clipped - 16 lines]
> Path: cpmsftngxa06.phx.gbl
> Xref: cpmsftngxa06.phx.gbl
microsoft.public.dotnet.framework.compactframework:49318
> NNTP-Posting-Host: tk2msftcmty1.phx.gbl 10.40.1.180
> X-Tomcat-NG: microsoft.public.dotnet.framework.compactframework
>
> Hi All,

I am working with C#, SQL CE and ADO.net using the .net compact framework
on a PDA.

I need to populate quite large tables from XML.
Initially I just used Dataset.ReadXML but this quickly ran into memory
problems with large amounts of data.
I am now using XmlTextReader to read the data in piecemeal fashion.

It has been suggested to me that I read a single record from the XML and
then insert each record one by one and work my way through the XML.
This seems far too clumsy.
I would rather extract a certain amount of data to a dataset and then
update the table regularly.
Is this possible or is there another simpler more efficient way of adding
the data?

Thanks,
Dave
Dave - 25 Mar 2004 10:41 GMT
The XML File is 4.6 MB
There is no Schema

You recomend using the xmltextreader an adding the records direct to SQL CE
Do you mean individually
This seems to take a long time

Would adding them in batches to a dateset prove quicker
Is this possible

Thanks
Dave
"Ilya Tumanov [MS]" - 25 Mar 2004 21:11 GMT
4.6 MB is pretty bug, but device should be able to handle it.
It'll take few minutes to load, but should be just fine.
You _have_ to use schema to accomplish that, though.
Inference is OK for prototyping and, may be, loading 1-2K XML files; it's
not good for anything else and should be avoided.

Here are your options:
1. Change your XML file to contain embedded schema. This might be
impossible if you have no control over XML.
2. Create schema using inference (or schema designer) on a desktop and load
it from separate file before loading XML.
   You can also reformat you XML to improve loading performance and
minimize XML size. You can use utility below to do that.
3. Create schema programmatically (i.e. add tables, columns, relations,
etc.) before loading XML. This is not very flexible, though, but might be
OK if your schema is fixed.
4. Create typed DataSet using desktop's designer, remove unsupported stuff
manually and use it in device project.
   This is pretty much the same as #3, but instead of coding you would
have to comment out unsupported code.

Now to the XmlTextReader solution...

I do not recommend using it unless you really desperate. It's inflexible
and would require significant coding on your part (not to mention testing).
It is, however, the fastest solution and it uses the least possible amount
of memory.
It would work even for 50 MB XML, but probably not worth is with 4.6 MB.
It won't be slower. DataAdapter.Fill() will add records one by one anyway
as SQL CE does not support batching.
Thus, loading small batches into the DataSet would not improve performance.
I see no good way to accomplish this, anyway. You might be able to use
Fragment mode, but it's not exactly what you need.

Best regards,

Ilya

PS Utility source code...

using System;
using System.Data;
using System.IO;
using System.Text;
using System.Xml;

namespace FixSchema
{
    class DoFix
    {
        const int OK     = 0;
        const int ERROR = 1;

       static void DumpRelation ( DataRelation dr) {
            string parentColumns = "";
            string childColumns  = "";
           
            foreach (DataColumn dc in dr.ParentColumns)
                parentColumns += String.Format("'{0}' ", dc.ColumnName);

            foreach (DataColumn dc in dr.ChildColumns)
                childColumns += String.Format("'{0}' ", dc.ColumnName);

           Console.WriteLine ("Relation '{0}': '{1}' ({4})->'{2}' ({4}),
{3}", dr.RelationName, dr.ChildTable, dr.ParentTable, dr.Nested ? "nested"
: "related",
                                            parentColumns, childColumns );
       }

        static int Main(string[] args)
        {
            if (args.Length == 0) {

                Console.WriteLine ("This utility loads XML file, changes relations from
nexted to related,\nmaps all columns as attributes and saves resulted
schema and data.");
                Console.WriteLine ("Usage: {0} xml_file",
Path.GetFileName(System.Reflection.Assembly.GetExecutingAssembly().GetName()
.CodeBase));

                return ERROR;
            }

            try {
                string fileName = args[0];

                DataSet ds = new DataSet();

                Console.WriteLine ("Loading XML file '{0}'", fileName);

                XmlReadMode mode = ds.ReadXml (fileName);

                if (XmlReadMode.InferSchema == mode ) {
                    Console.WriteLine ("WARNING! WARNING! WARNING! WARNING! WARNING!
WARNING!\nXML has been loaded in inference mode, all type information is
now lost.\nConsider changing schema to restore type information.");
                }

                Console.WriteLine ("Done loading file, changing column mappings to
attributes...");

                foreach (DataTable dt in ds.Tables) {
                    foreach (DataColumn dc in dt.Columns){
                        if (dc.ColumnMapping != MappingType.Attribute) {
                            Console.WriteLine ("Changing mapping '{0}' of column '{1}' from
table '{2}' to 'Attribute'",
                                    dc.ColumnMapping, dc.ColumnName, dt.TableName );               

                            dc.ColumnMapping = MappingType.Attribute;
                        }
                    }
                }

                Console.WriteLine ("Now changing relations from nested to related...");

                foreach (DataRelation r in ds.Relations ) {

                    DumpRelation (r);
               
                    if (r.Nested && (r.ParentColumns.Length == 1) &&
(r.ChildColumns.Length == 1)) {

                        Console.WriteLine ("Changing relation '{0}' from nested to
related...", r.RelationName);

                        r.Nested = false;
                    }               
                }

                Console.WriteLine ("Done processing relations, saving schema...");

                ds.WriteXmlSchema (fileName + ".xsd");

                Console.WriteLine ("Schema saved to file '{0}'", fileName + ".xsd");

                Console.WriteLine ("Now saving data with schema...");

                ds.WriteXml (fileName + ".xml", XmlWriteMode.WriteSchema);

                Console.WriteLine ("Data saved to file '{0}'", fileName + ".xml");
            }
            catch (Exception e) {

                Console.WriteLine ("Error while processing: {0}", e);

                return ERROR;
            }           

            return OK;                   
        }
    }
}

This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
> Thread-Topic: Data Insertion
> thread-index: AcQSTUsR44/NKI2IQW+5cd6h8lFY3w==
> X-Tomcat-NG: microsoft.public.dotnet.framework.compactframework
> From: "=?Utf-8?B?RGF2ZQ==?=" <anonymous@discussions.microsoft.com>
> References:  <8F6152A2-6CE3-4887-876A-85349EE4AE69@microsoft.com>
<lVUadJdEEHA.3924@cpmsftngxa06.phx.gbl>
> Subject: RE: Data Insertion
> Date: Thu, 25 Mar 2004 01:41:05 -0800
[quoted text clipped - 12 lines]
> Path: cpmsftngxa06.phx.gbl
> Xref: cpmsftngxa06.phx.gbl
microsoft.public.dotnet.framework.compactframework:49423
> NNTP-Posting-Host: tk2msftcmty1.phx.gbl 10.40.1.180
> X-Tomcat-NG: microsoft.public.dotnet.framework.compactframework
>
> The XML File is 4.6 MB.
There is no Schema.

You recomend using the xmltextreader an adding the records direct to SQL CE

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.