Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / Languages / C# / March 2008

Tip: Looking for answers? Try searching our database.

Need Help On Best Querying  ( is LINQ work with Huge amount of data     ..)

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
ranganadh - 16 Mar 2008 12:18 GMT
Dear Group members,

 I am new to LINQ, pls help on the deeling with huge amount of data
with the C# stand Alone application.

I have two file, which contains more then 2 lacs lines in every file

suppose file1 like ...

999361894 422021100257001
899119011 422021100344217
899373022 422021100262179
892044443 422021100426945
899292491 422021000154860
----
----

and file2 looks like

92000000,422021100420675
92000002,422021100420403
92000003,422021100420614
92000004,422021100425785
92000005,422021100422232
---
----

values under each file divided by different set of delimiter, and this
delimiter can be changed... ie some files came with some other
delimiter with more fields also.

 My motive is to import different kind of formated files to my
application, and generate some auto-summarized report like ... 99999
is missed in  file 2, and 99999 no is missed in file 1, and for this
id 999999, values are not matching ..

for this thing, i succeeded agenest the import various typed  files to
my data grid.

Remain-thing, major thing is compare the files and generate summary
report, kindly remember  i don't have Database support for my
application, and all data came with different formates and data will
be came like text files.

 So, that i am starting with small comparison with LINQ,

like
    DataTable dt1;//hold's the first file data
    DataTable dt2; // hold's the second file data.

When i try to find out the records which are having dt1 and not in
dt2. with LINQ

var test=   from k in dt1.AsEnumerable() where !(from t in
dt2.AsEnumerable() select
Field<string>(0)).Contains(k.Field<string>(0)) select k;

myResult_DataGrid.Datasource = test.AsDataView();

These steps running from server from mints, still application is look
like executing same statment.

i don't know what is the cause of hangout the application,  it's look
like a problem with the huge amount of data, ofcource it nearly
having  2lac's X 2 lacs

is there any other solutions, to do this type of thigs? is there any
problem in my LINQ statement?

pls let me know , if anybody having the solutions on this type  of
programs.

Thanks and Regards
ranganadh Kodali
Jon Skeet [C# MVP] - 16 Mar 2008 16:24 GMT
<snip>

>   So, that i am starting with small comparison with LINQ,
>
>  like
>      DataTable dt1;//hold's the first file data
>      DataTable dt2; // hold's the second file data.

Any reason for using DataTables to start with? Why not just load the
data into strongly typed objects yourself?

>  When i try to find out the records which are having dt1 and not in
> dt2. with LINQ
>
> var test=   from k in dt1.AsEnumerable() where !(from t in
> dt2.AsEnumerable() select
> Field<string>(0)).Contains(k.Field<string>(0)) select k;

That will indeed take a long time. If you want to find elements which
are in the first sequence but not not in the second, use the Except()
method.

That will quite possibly be easier if you use your own objects rather
than DataTables.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk


Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.