André Figueira

Systems engineer - I write apps, I make websites, opinions are my own...

How to split large files

I recently had to run a big ingestion at work, and as it was generally a test run we didn't want to do too many at a time, so we split them into chunks of 1 million each, see here how to do it with unix.

Splitting the file

Easy just use the split command, you can also provide a suffix

 split --lines=1000000 huge-file.csv --additional-suffix=.csv

Adding the header to each file

Again a nice easy one acomplished with sed, need some more info as this one isn't as easy to read as the split, we're giving sed the pattern and the header line we can to place it in, then looking for all files starting with x that we're going to apply this to, use whatever file pattern to match your files.

sed -i '1 i your header line here' ./x*