Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / Languages / C# / December 2005

Tip: Looking for answers? Try searching our database.

Sorting data in fixed file format

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
booksnore - 16 Dec 2005 18:17 GMT
I'm looking for a way to sort text files consisting of fixed file
format. The files are big, typically over 10 million records and they
consist of about 100 fields with the record being over 600 bytes in
length. I need to
sort on a combination of 8 of the fields. Has anyobe attempted a to sort
data in this way using C#? Performance and scalability are the main
factors - the input data I
have is likely to grow probably up to 40 million records. I want to
avoid loading data into a database just to sort it.

Joe
Ignacio Machin ( .NET/ C# MVP ) - 16 Dec 2005 18:25 GMT
Hi,

IMO your best (if not unique) solution is to put everything in a database,
sort it and then export it. With this amount of data there is no a better
solution.

Importing/Exporting is very easy using DTS packages. let me know if you
need code for it

Signature

Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

> I'm looking for a way to sort text files consisting of fixed file
> format. The files are big, typically over 10 million records and they
[quoted text clipped - 9 lines]
>
> *** Sent via Developersdex http://www.developersdex.com ***
Joe - 16 Dec 2005 18:37 GMT
Thanks Ignacio, what sort of code is it that you have? I would be
interested in taking a look.
regards
Joe

Joe

--
Sent via .NET Newsgroups
http://www.dotnetnewsgroups.com
Ignacio Machin ( .NET/ C# MVP ) - 16 Dec 2005 18:55 GMT
Hi,

this is what I use, I create a DTS from enterprise manager , where the data
is comnig from and where to put it, then select "create a file" or something
similar in the DTS wizard, it does create a .dts file this is the one you
will use later

Here is the code, note that I change the datasource from the code, you can
do a similar thing with the destination:
Also you need to add a reference to DTS COM library

using DTS;
using System.Data;
using System.Data.SqlClient;

 void RunPackage( string packSource, string packName, string dataSource)
 {
  try
  {
   Package2Class package = new Package2Class();
   object pVarPersistStgOfHost = null;

   // if you need to load from file
   package.LoadFromStorageFile(
    packSource,
    null,
    null,
    null,
    packName,
    ref pVarPersistStgOfHost);

   /*
   package.LoadFromSQLServer(
    "",
    null,
    null,
    DTSSQLServerStorageFlags.DTSSQLStgFlag_UseTrustedConnection,
    null,
    null,
    null,
    "Test Import Package",
    ref pVarPersistStgOfHost);
   */

   package._Package_Connections.Item(1).DataSource = dataSource;
   package.Execute();
   package.UnInitialize();

   // force Release() on COM object
   //
   System.Runtime.InteropServices.Marshal.ReleaseComObject(package);
   package = null;
  }
  catch(System.Runtime.InteropServices.COMException e)
  {
   Console.WriteLine("COMException {0}", e.ErrorCode.ToString() );
   Console.WriteLine("{0}", e.Message);
   Console.WriteLine("{0}", e.Source);
   Console.WriteLine("Stack dump\n{0}\n", e.StackTrace);
   Console.ReadLine();
  }
  catch(System.Exception e)
  {
   Console.WriteLine("Exception");
   Console.WriteLine("{0}", e.Message);
   Console.WriteLine("{0}", e.Source);
   Console.WriteLine("Stack dump\n{0}\n", e.StackTrace);

   Console.ReadLine();
  }
 }

Signature

Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

> Thanks Ignacio, what sort of code is it that you have? I would be
> interested in taking a look.
[quoted text clipped - 6 lines]
> Sent via .NET Newsgroups
> http://www.dotnetnewsgroups.com 
Kevin Spencer - 16 Dec 2005 18:30 GMT
You have over 10 million records in a fixed file format, and you don't want
to put it into a database? Dude, that's what databases are *for*. Fixed file
formats are fine for storing static data. But filtering, selecting,
ordering, that's all stuff for databases.

Signature

HTH,

Kevin Spencer
Microsoft MVP
.Net Developer
You can lead a fish to a bicycle,
but it takes a very long time,
and the bicycle has to *want* to change.

> I'm looking for a way to sort text files consisting of fixed file
> format. The files are big, typically over 10 million records and they
[quoted text clipped - 9 lines]
>
> *** Sent via Developersdex http://www.developersdex.com ***
booksnore - 17 Dec 2005 17:18 GMT
I want to sort the data in the file - that's it nothing else. Unix has a
sort command, I wondered if there was an equivalent in Windows.  

Joe

Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.