Produce a MVP multi-(threaded|process) solution which can process logs efficiently. Minimum requirements:
- bound on number of open file descriptions
- bound of memory usage
- save progress on a days worth of logs as compressed checkpoints
Other features which would be nice to have:
- logging to diagnose errors
- sending out an email on errors so that the processing can be set up with a nightly cron job without worry about missing information
- alternatively, posting a webhook to a discord channel might be preferable?
Speed requirements are not super important, should be able to process a day's worth of logs onn Smogon faster than smogon/usage-stats:
Stretch feature: compress files after each run so that at the end of the month archiving is just a matter of the final tar cvf $OUTPUT * step
Produce a MVP multi-(threaded|process) solution which can process logs efficiently. Minimum requirements:
Other features which would be nice to have:
Speed requirements are not super important, should be able to process a day's worth of logs onn Smogon faster than smogon/usage-stats:
Stretch feature: compress files after each run so that at the end of the month archiving is just a matter of the final
tar cvf $OUTPUT *step