Error: CSV File to Large
For branch excel
When the dataset contains CSV files larger than 2GB the following error occurs
RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2472454648) is greater than 2 GB at tryCreateBuffer (fs.js:369:13) at Object.readFileSync (fs.js:405:14) at main (C:\Users\qolomanyb\url-twitter-search\app.js:43:41)(Use `node --trace-warnings ...` to show where the warning was created)
(node:1276) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1276) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
This is because the fs readFile only has a buffer of 2GB. The solution is to use a file stream. Working on the solution now.