I have a set of extremely large CSV files, 4million plus lines each. Essentially they contain info on a vast number of sites, items that have been ordered and quantities etc. Each file represents a months worth of data.
I have been manipulating the data using a text editor, Ultraedit, which has been fine however I would like to query the data more effectively.
I have managed to import one CSV file into a Base DB by starting a new DB and connecting to an existing DB, a spreadsheet. I am not sure however if it is picking up all the data as if i run a query that should pick out a site id near to the bottom of the file i get nothing returned.
Im not sure if I am doing this correctly, copying and pasting data doesnt work, there is just too much of it. Am i just better off using the text editor or should i be able to import all the data in all the files into one database to query really effectively.
Thanks
Tom
Large CSV Files
Large CSV Files
OpenOffice 3.4.1 on Windows 7
Re: Large CSV Files
Use a true database such as MySQL or HSQLDB. Let it index the important fields. Then you may connect Base to that database if you really need those data in your office suite.
You may even try the built in HSQLDB as described in http://forum.openoffice.org/en/forum/vi ... 83&t=23260
You may even try the built in HSQLDB as described in http://forum.openoffice.org/en/forum/vi ... 83&t=23260
Code: Select all
CREATE TEXT TABLE "Import"(
column definition 1,
column definition 2,
column definition 3,
and so on
);
SET "Import" SOURCE "blah.csv;with csv options"
Please, edit this topic's initial post and add "[Solved]" to the subject line if your problem has been solved.
Ubuntu 18.04 with LibreOffice 6.0, latest OpenOffice and LibreOffice
Ubuntu 18.04 with LibreOffice 6.0, latest OpenOffice and LibreOffice