Crashing since v2.23 with large CSV file

[jwoodwardtfx]

Processing a large CSV file (containing tens of millions of lat/lon points) causes the following crash:

For layer 0, using name "data"
pread(index): Success24%

Up to and including v2.22 works fine.

Here is some noddy Python3 code to generate a test file with 100 million entries (the file will be a couple of gigs in size):

import os

if os.path.exists("data.csv"):
	os.remove("data.csv")

file1 = open("data.csv", "a")

file1.write("lat,lon\r");

currentLat = 0;
currentLon = 0;

for x in range(1000):
	currentLat += 0.0000001;
	if currentLat >= 90:
		currentLat -= 180;
	for y in range (100000):
		currentLon += 0.0000001;
		if currentLon >= 180:
			currentLon -= 360;
		file1.write("{:.8f}".format(currentLat));
		file1.write(',');
		file1.write("{:.8f}".format(currentLon));
		file1.write('\n');

file1.close();

And here is the command used:

tippecanoe --output "output.sqlite" "data.csv" --simplification=1 --full-detail=16 --minimum-zoom=6 --maximum-zoom=16 --no-tile-size-limit --cluster-distance=1 -r1

I'm running in an 24.04.2 LTS Ubuntu VM (from a Windows 10 host), but have plenty of RAM (20GBytes) available (it never seems to top out due to memory if I watch it using "top" while tippecanoe is running).

Any ideas?

[e-n-f]

Thanks for the report! I'll try your script and see if I can figure out what is going wrong.

[e-n-f]

I can't reproduce this on my laptop but will try on Ubuntu.

[e-n-f]

I can't reproduce it on Ubuntu 24.04.2 LTS either. (On AWS EC2, not Windows VM.)

[e-n-f]

It has to be this line:

        if (pread(a->indexfd, (void *) s.c_str(), end - start, start) != end - start) {

I suppose pread isn't actually guaranteed to do the whole read in a single call, so I can change it to support partial reads, but it would give me a lot more confidence in the fix if I could reproduce it.

[e-n-f]

Please give #343 a try and see if this fixes it for you.

[jwoodwardtfx]

Thanks for looking at this. I'll give that fix a try.

I put a quick bit of printf debugging in and it seems to fail as it's trying to read a massive amount of data into memory (2147483648 bytes / 2 GBytes), but only gets 2147479552 bytes returned from the pread() call (i..e, 4KBytes short). So, your looping fix may well sort this out.

I guess this may be due to my host O/S (Win 10) or the version of Virtual Box I'm using to run Ubuntu.

I did try two different Ubuntu builds (an older one from a few years ago and the latest LTS using a brand new VM). Both behaved the same.

I'll report back on how I get on with that fix.

[jwoodwardtfx]

It looks like the pread side is now fixed by that pull, but (probably not unsurprisingly) the pwrite side fails in the same way. Using the same fix on that side, seems to cure it :)

		qsort((void *) s.c_str(), (end - start) / a->bytes, a->bytes, indexcmp);

		dest = s.c_str();
		off = start;
		count = end - start;
		
		while (count > 0) {
			ssize_t n = pwrite(a->indexfd, (void *) dest, count, off);
			
			if (n < 0) {
				fprintf(stderr, "pwrite(index): %s\n", strerror(errno));
				exit(EXIT_WRITE);
			}

			dest += n;
			off += n;
			count -= n;
		}

It'll take a few more hours to actually generate the tiles, but, it's got passed the reordering phase now.

Thanks for the help, much appreciated.

[jwoodwardtfx]

I can confirm that with both the pread and pwrite looping, this issue is resolved and I can successfully build the map tiles.

Thanks.

[jwoodwardtfx]

@e-n-f, sorry to chase, but would you be able to get that fix merged (#343) and do a matching fix on the pwrite side? Thanks.