Large CSV Files

[tomisurf]

I have a set of extremely large CSV files, 4million plus lines each. Essentially they contain info on a vast number of sites, items that have been ordered and quantities etc. Each file represents a months worth of data.

I have been manipulating the data using a text editor, Ultraedit, which has been fine however I would like to query the data more effectively.

I have managed to import one CSV file into a Base DB by starting a new DB and connecting to an existing DB, a spreadsheet. I am not sure however if it is picking up all the data as if i run a query that should pick out a site id near to the bottom of the file i get nothing returned.

Im not sure if I am doing this correctly, copying and pasting data doesnt work, there is just too much of it. Am i just better off using the text editor or should i be able to import all the data in all the files into one database to query really effectively.

Thanks

Tom

[Villeroy]

Use a true database such as MySQL or HSQLDB. Let it index the important fields. Then you may connect Base to that database if you really need those data in your office suite.

You may even try the built in HSQLDB as described in http://forum.openoffice.org/en/forum/vi ... 83&t=23260

Code: Select all

CREATE TEXT TABLE "Import"(
  column definition 1,
  column definition 2,
  column definition 3,
 and so on
);
SET "Import" SOURCE "blah.csv;with csv options"