very nice to see this!
Couple of quick thoughts for dealing with large buffers and perhaps saving on the memory footprint (I'm not too familiar with R internals tho).
-
Maybe consider shaving off the double memory allocation? (first, second). Since the read is sequential, would it be feasible to move the payload reading to the C side, allocate the vector and then fread the chunks directly into the destination buffer? (header parse can stay in R and pass the path + offset, or move to C too?).
-
Longer term, it would be ~natural to think about having the equivalent of the mmap_mode=r of numpy, for lazy/partial reads. Sounds like this can be done using the ALTREP API?
If any of this is useful for this package and its community I could look into drafting a PoC, it's a problem that was in my roadmap for convergent reasons. Thanks for reading!
very nice to see this!
Couple of quick thoughts for dealing with large buffers and perhaps saving on the memory footprint (I'm not too familiar with R internals tho).
Maybe consider shaving off the double memory allocation? (first, second). Since the read is sequential, would it be feasible to move the payload reading to the C side, allocate the vector and then
freadthe chunks directly into the destination buffer? (header parse can stay in R and pass the path + offset, or move to C too?).Longer term, it would be ~natural to think about having the equivalent of the
mmap_mode=rof numpy, for lazy/partial reads. Sounds like this can be done using theALTREPAPI?If any of this is useful for this package and its community I could look into drafting a PoC, it's a problem that was in my roadmap for convergent reasons. Thanks for reading!