Compressed Plots

[Jsewill]

Using the cuda-compression-debug branch, I was able to plot a compressed plot using my CPU. At C6, this k32 plot weighed in at 79.63 GiB, but took almost 150% of my usual average plotting time. I'm pleased that the plotting worked without a problem. Now, I'd like to test the resulting plot. Is there a way to do that yet?

I understand that the farmer for these compressed plots is still in (planned) development, and I don't see any changes to chiapos that look like it knows how to test these plots--but that could be because I don't know what to look for.

[harold-b]

You can tests compressed plots below level 8 (so 1-7) at the moment with bladebit itself. Although you can build levels 8 and 9 plots, those are for testing purposes only and are not queryable yet.

To query a compressed plot for a challenge run:
bladebit_cuda -v -t <thread_count> validate --f7 <integer> <path_to_plot_file>

Set the thread_count value to the thread count you want to use when decompressing. The default is 8.

The <integer> value must be a number between 0 and 2^32-1 (inclusive). This represents a the portion of the challenge used by a k32 plot. A plot will have most of those numbers, but some will be missing. Some of those will have several proofs for a given challenge.

[Jsewill]

While looping that command, it spat out the following:

Assertion Failed @ /redacted/bladebit/src/harvesting/GreenReaper.cpp:268 grFetchProofForChallenge().
Trace/breakpoint trap (core dumped)

This occurred for --f7 values, 15, 36, 53, 59, 62, and 72, out of 75. It did fetch a bunch of proofs, though, so that's encouraging.

[harold-b]

Thank you for the report. These can be considered missing proofs. It's just unfinished to handle these cases. A bit strange that there are that many together, though. Is this with the same plot id from the log below?

Also is this the same instance of bladebit_cuda that you plotted with? If so, it seems to be in DEBUG mode as the assert should not trigger otherwise.

[Jsewill]

Yes, that is the same plot. As I stated earlier, this is all done with bladebit (not bladebit_cuda) built from the cuda_compression_debug branch, using the CPU (not GPU).

[harold-b]

bladebit and bladebit_cuda are the same, except one has a cuda plotter. I understand you used ramplot. But the latest branch is cuda_compression not cuda_compression_debug that is a personal branch from a colleague who is bug huntin' :)

Anyway, that could explain your plotting performance penalty.
Try building instead from the cuda_compression branch.

[Jsewill]

That makes sense. I checked out the correct branch, and built it using the build script in .github/actions/. This time it finished within a relatively normal time (24 minutes). I usually see plot times between 22 minutes and 26 minutes in v2.0.1, so that is right in line.

[harold-b]

As for it taking more. This should not be the case. Can you share a log and info about the system you used? Is it a NUMA system?

[Jsewill]

It is a NUMA system. An old Dell r720 running Ubuntu server 22.10 with two E5-2690 v2 CPUs. I used ramplot, using a brd ramdisk for the output, and only used 20 of 40 threads so as to keep the operation to one of the two CPUs, which has typically resulted in faster plot times. I just so happened to run it with the output piped to a log file, which I've attached.
bladebit-k32-c6.log

I'll try the challenge run shortly.