Investigate support for large(r) files

[herrfugbaum]

Streams should be helpful to handle larger files.
In Papa Parse, the csv parser used in this project, there is already support for streams built in.

Files could be handled row by row instead of "all or nothing".

[herrfugbaum]

First test with the results of the 2018 StackOverflow survey (~186MB, 98856 rows) took 383.422 seconds
😅

const Papa = require('papaparse')

const fs = require('fs')

const file = fs.createReadStream('./huge.csv')

const start = Date.now()

Papa.parse(file, {
  header: true,
  skipEmptyLines: true,
  step: function (row) {
    console.log("Row:", row.data)
  },
  complete: function () {
    const duration = (Date.now() - start) / 1000
    console.log('Reading the file took ' + duration + ' seconds')
  }
})

[herrfugbaum]

Using chunk instead of step took 360.858 seconds.

const Papa = require('papaparse')

const streamFile = require('./streamFile')

const start = Date.now()

const file = streamFile('./huge.csv')

Papa.parse(file, {
  header: true,
  skipEmptyLines: true,
  /* step: function (row) {
    console.log("Row:", row.data)
  }, */
  chunk: function (chunk) {
    console.log(chunk)
  },
  complete: function () {
    const duration = (Date.now() - start) / 1000
    console.log('Reading the file took ' + duration + ' seconds')
  }
})

[ankush981]

Ah, I just created #23. Maybe remove that one and copy over the ideas here?

[ankush981]

Files could be handled row by row instead of "all or nothing".

Maybe we can try, say, 200 lines at a time . . . row by row is still going to kill, espcially on non-SSd drisks. 😓