Zlib decompression on the DPU

[Penjiboy]

We are currently able to compile the dpu program, however we fail to load the binary on the DPU. Requesting a merge so that Joel can take a look at the issue

[jnider]

I compiled & ran this to compress a file from the host - works!
When running the decompressor from the DPU, it fails with
host_simple_example.c:161(DPU_decompress): DPU Error (error during elf process)
Inspecting that line shows it is during the dpu_load() function. I added a variable to print the return code, and saw 21 (DPU_ERR_ELF_FILE) which means "DPU binary cannot be parsed". Almost certainly something section in the ELF is too big to be loaded.

Show the section headers in the ELF with: readelf -S decompress.dpu
We can see that the .text section (code) is 0x7a10 = 31248 bytes. That looks pretty big. The manual says the IRAM can hold 2^12 x 48-bit instructions (=4K x 6 bytes = 24KB) so it looks like this program is exceeding the maximum.

I modified the Makefile to produce intermediate .o files when compiling the DPU program by removing the 'flto' and 'w' flags from DPU_CFLAGS, and adding a new list of .o targets, and a new rule to build .o files from each .c file. That produced the following files:

objs=$(find . -name '*.o')
echo $objs
./dpu_decompress.o ./common/zutil.o ./decompress/inflate.o ./zlib/inftrees.o ./zlib/crc32.o ./zlib/inffast.o ./zlib/adler32.o
for obj in $objs; do echo "$obj" >> sections; readelf -S $obj >> sections; done
grep 'AX' sections > text_sections

By reading the ELF section headers on all files, and searching the results for 'AX' (loadable and executable) concatenating them to a single file, we can see that .text.inflate is the largest section (0x3920 bytes), and it belongs to decompress/inflate.o

readelf -s decompress/inflate.o | grep FUNC
By checking the symbol table of decompress/inflate.o, we can see that 'inflate' is the largest function (14624 = 0x3920 bytes) and actually seems to be the only function in that section (since the sizes are identical). A quick inspection show that this is a BIG function, and there is no easy solution. We will have to think about this more carefully.

[craiig]

By checking the symbol table of decompress/inflate.o, we can see that 'inflate' is the largest function (14624 = 0x3920 bytes) and actually seems to be the only function in that section (since the sizes are identical). A quick inspection show that this is a BIG function, and there is no easy solution. We will have to think about this more carefully.

Is this that large without LTO?, but with O2/O3? I would try optimizing for size (-Os) to see how well it does.

FWIW I feel like other embedded systems have probably wanted to use inflate, so I wonder what approaches they have taken, here. You could always try Os on just the inflate object, too.

[Penjiboy]

Is this that large without LTO?, but with O2/O3? I would try optimizing for size (-Os) to see how well it does.

FWIW I feel like other embedded systems have probably wanted to use inflate, so I wonder what approaches they have taken, here. You could always try Os on just the inflate object, too.

I've just tried optimizing for size with (-Os) and it brings down the size of the .text section to around 29K, which is still too much. I'll look into finding out how others have approached this for embedded systems

[craiig]

Ok - it might also be worth looking at what other things are making the file large. inflate is required right? are there other things being linked in that are not needed?

[Penjiboy]

Ok - it might also be worth looking at what other things are making the file large. inflate is required right? are there other things being linked in that are not needed?

I'll probably try looking in this direction as well. Inflate is required, but I'm sure we don't need everything that we're getting from inflate. Out of intuition, for example, we probably don't need both crc32 and adler32 at the same time so we can remove one. I'll look through the inflate code itself to see what parts we can trim and see how much of an impact that has on the size.

[Penjiboy]

After trimming down parts of inflate that aren't needed, the next issue I'm struggling with is to do with an Illegal WRAM read that is happening somewhere. I'm also struggling to attach to the DPU while the host program is running...