Skip to content

Python

Jump to bottom
Allex edited this page Apr 30, 2026 · 10 revisions

perf performance measurements

Includes the C calls. ref

  • Make sure Python is installed with frame-pointers, the JIT way is too slow.
    CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" pyenv install <version>
    If you set LD_LIBRARY_PATH (for example due to pyo3), make sure to unset it during install.
  • Check if your python is installed with frame-pointers.
    python -m sysconfig | grep 'no-omit-frame-pointer'
  • Run your program with perf.
    perf record -F max -g python -X perf <python command>
  • Show the profile data.
    perf report

firefox profiler

The output of perf record can also be loaded into firefox profiler. First convert it perf script -F +pid > /tmp/test.perf. Then open that file using https://profiler.firefox.com

PyO3

When compiling PyO3, it looks for which libpython to link to. When building your package using maturin, linking is disabled, breaking binaries, tests, and examples. ref

To inspect the location where PyO3 is looking, run: PYO3_PRINT_CONFIG=1 cargo b 2>&1 | grep lib_dir.

To inspect the lib_dir of a specific python executable:
python -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))'

If these don't match, something is wrong. This can happen for some reason (https://github.com/PyO3/pyo3/issues/1576) in PyEnv.

By using a venv during the build, the linking step is solved.
This however does not solve running rust binaries (like cargo test and cargo run), which requires LD_LIBRARY_PATH to be set correctly. When using the system python version, there is no issue.
When using an alternative location, which is not searched by ldd, you'll have to provide the LD_LIBRARY_PATH.

So to tie everything together, when using poetry:
LD_LIBRARY_PATH=$(poetry run python -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))') poetry run cargo test

Profiling python + rust

Using perf we can profile both python and rust code together.

First make sure the Python you are using was build with frame-pointers enabled, see previous section.

To enable frame-pointers in rust, set RUSTFLAGS="-C force-frame-pointers=yes -C symbol-mangling-version=v0" during the build.

Ensure you are building the release profile. When your pyproject.toml is setup properly, you should be able to run RUSTFLAGS="-C force-frame-pointers=yes -C symbol-mangling-version=v0" pip install . --force-reinstall in your virtualenv to get a build with frame-pointers.

ref

Clone this wiki locally