Add Python-based emulator for jbpf codelets and hooks#123
Open
doctorlai-msrc wants to merge 4 commits into
Open
Add Python-based emulator for jbpf codelets and hooks#123doctorlai-msrc wants to merge 4 commits into
doctorlai-msrc wants to merge 4 commits into
Conversation
This reverts commit 47333e5.
doctorlai-msrc
changed the title
There was a problem hiding this comment.
Pull request overview
This pull request adds a new Python-driven JBPF emulator executable and supporting Python utilities/tests to enable running codelets and hooks from Python test scripts, plus CI/Docker plumbing to build and run those emulator tests. It also introduces a new test codelet (simple_output2) and updates build tooling/docs to support an additional “both” static+shared build mode.
Changes:
- Add
src/emulator/(C++ emulator embedding CPython + Python utilities) and a suite of Python-based emulator tests. - Extend build/CI (Dockerfiles, workflows, build scripts) to install Python deps and optionally run emulator tests.
- Add
simple_output2test codelet and update verifier allowlist; expose helper-function tables via new accessors in core.
Reviewed changes
Copilot reviewed 37 out of 40 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/logger/jbpf_logging.c | Minor formatting/line-number-only adjustment in logger wrapper. |
| src/emulator/test/test.yaml | Add YAML codeletset descriptor used by emulator tests. |
| src/emulator/test/test_time_event.yaml | Add YAML descriptor for time-event emulator test. |
| src/emulator/test/test_time_event.py | Add Python test for emulated time events and output validation. |
| src/emulator/test/test_report_stats.yaml | Add YAML descriptor for report_stats emulator test. |
| src/emulator/test/test_report_stats.py | Add Python test for report_stats hook path and output validation. |
| src/emulator/test/test_periodic_call.yaml | Add YAML descriptor for periodic_call emulator test. |
| src/emulator/test/test_periodic_call.py | Add Python test for periodic_call hook output validation. |
| src/emulator/test/test_2.py | Add Python test for input-message send + hook invocation + output handling. |
| src/emulator/test/test_1.py | Add Python test for basic hook invocation and output handling (plus stats_report codelet). |
| src/emulator/test/test_0.py | Add basic Python smoke test for random stream-id helper. |
| src/emulator/patch.py | Add post-processor to force ctypes structs packing in generated bindings. |
| src/emulator/hooks.h | Add emulator hook include aggregation (auto-generated wrappers + Python headers). |
| src/emulator/hook_wrapper.h | Add macro helpers for defining hook wrappers callable from Python. |
| src/emulator/helper_functions.hpp | Add sample custom helper-function registration and a helper Python method. |
| src/emulator/emulator.cpp | Add the C++ emulator runtime embedding CPython and exposing JBPF APIs to Python. |
| src/emulator/emulator_utils.py | Add Python utilities for codeletset load/unload, IO handling, YAML parsing, stream-id helpers. |
| src/emulator/CMakeLists.txt | Add build rules for emulator binary, wrapper generation, ctypes bindings generation, and file staging to out/. |
| src/emulator/CMakeLists_emulator.txt | Add “custom emulator” standalone build template staged into out/emulator/. |
| src/emulator/autogen-hook-wrappers.py | Add generator that emits Python-callable wrappers for declared hooks from headers. |
| src/core/jbpf_helper_impl.h | Expose new helper-function accessor declarations (for emulator/introspection). |
| src/core/jbpf_helper_impl.c | Implement new helper-function accessor functions. |
| lsan_agent.supp | Add LSAN suppression for libpython leaks in sanitizer runs. |
| jbpf_tests/verifier/jbpf_basic_verifier_test.cpp | Allowlist new simple_output2.o codelet in verifier tests. |
| jbpf_tests/test_files/codelets/simple_output2/simple_output2.c | Add new simple output codelet used by emulator tests. |
| jbpf_tests/test_files/codelets/simple_output2/Makefile | Add build rules for new codelet. |
| helper_build_files/test_build_utils.sh | Extend build-utils test coverage for JBPF_STATIC=2 / “both” mode. |
| helper_build_files/build_utils.sh | Add JBPF_STATIC=2 mapping to -DJBPF_STATIC=both. |
| helper_build_files/build_jbpf_lib.sh | Add option to run emulator tests inside the build container. |
| docs/overview.md | Link emulator documentation from the overview. |
| docs/integrate_lib.md | Document JBPF_STATIC “both” mode in integration instructions. |
| docs/emulator.md | Add full emulator documentation and usage guide. |
| deploy/ubuntu24.04.Dockerfile | Install Python dev tooling and pip deps for emulator build/tests. |
| deploy/ubuntu22.04.Dockerfile | Install Python dev tooling and pip deps for emulator build/tests. |
| deploy/mariner.Dockerfile | Install Python dev tooling and pip deps for emulator build/tests. |
| CMakeLists.txt | Add emulator subdirectory to the top-level build. |
| .github/workflows/docker-build-and-test-arm.yaml | Add emulator test step (static builds only) + extra system info logging. |
| .github/workflows/docker_build_and_test.yaml | Add emulator test step (static builds only) + extra system info logging. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+166
to
+181
| static void | ||
| io_channel_check_output(jbpf_io_stream_id_t* stream_id, void** bufs, int num_bufs, void* ctx) | ||
| { | ||
| auto message = std::unique_ptr<message_t, MessageDeleter>(new message_t); | ||
| message->stream_id = static_cast<jbpf_io_stream_id_t*>(malloc(sizeof(jbpf_io_stream_id_t))); | ||
| if (!message->stream_id) { | ||
| jbpf_logger(JBPF_ERROR, "Memory allocation failed for stream_id\n"); | ||
| return; // Handle memory allocation failure | ||
| } | ||
| memcpy(message->stream_id, stream_id, sizeof(jbpf_io_stream_id_t)); | ||
| message->data = static_cast<void**>(malloc(num_bufs * sizeof(void*))); | ||
| if (!message->data) { | ||
| jbpf_logger(JBPF_ERROR, "Memory allocation failed for data buffers\n"); | ||
| return; | ||
| } | ||
| memcpy(message->data, bufs, num_bufs * sizeof(void*)); |
Comment on lines
+270
to
+289
| int count = 0; | ||
| while (num_of_messages > 0) { | ||
| if (timeout_ns > 0) { // unit is nano second | ||
| struct timespec now; | ||
| clock_gettime(CLOCK_REALTIME, &now); | ||
| if (now.tv_sec - start.tv_sec >= timeout_ns / 1000000000) { | ||
| break; | ||
| } | ||
| } | ||
|
|
||
| // Pop out all the messages from the queue and re-construct stream_id, buf, nbuf, and ctx | ||
| while (!message_queue.empty()) { | ||
| std::unique_ptr<message_t, MessageDeleter> message; | ||
| { | ||
| std::lock_guard<std::mutex> lock(message_mutex); | ||
| if (message_queue.empty()) | ||
| break; | ||
| message = std::move(message_queue.front()); | ||
| message_queue.pop(); | ||
| } |
Comment on lines
+457
to
+459
| static PyModuleDef jbpfHelperModule = { | ||
| PyModuleDef_HEAD_INIT, "helper_functions", NULL, -1, jbpfHelperMethods, NULL, NULL, NULL, NULL}; | ||
|
|
Comment on lines
+371
to
+388
| // Parse the stream_id as bytes, data as bytes, and size as an int | ||
| if (!PyArg_ParseTuple(args, "y#y#i", &stream_id_data, &stream_id_size, &data, &data_size, &size)) { | ||
| PyErr_SetString(PyExc_TypeError, "Expected arguments: stream_id (bytes), data (bytes), size (int)"); | ||
| return NULL; | ||
| } | ||
|
|
||
| // Ensure stream_id is the expected size for jbpf_io_stream_id_t (e.g., 16 bytes) | ||
| if (stream_id_size != sizeof(jbpf_io_stream_id_t)) { | ||
| PyErr_SetString(PyExc_ValueError, "Invalid stream_id size"); | ||
| return NULL; | ||
| } | ||
|
|
||
| // Cast the stream_id_data to a jbpf_io_stream_id_t pointer | ||
| jbpf_io_stream_id_t* stream_id = reinterpret_cast<jbpf_io_stream_id_t*>(stream_id_data); | ||
|
|
||
| // Call the original C function | ||
| int result = jbpf_send_input_msg(stream_id, static_cast<void*>(data), static_cast<size_t>(size)); | ||
|
|
Comment on lines
+239
to
+249
| struct jbpf_codeletset_unload_req* codeletset_unload_req_c1; | ||
| jbpf_codeletset_load_error_s err; | ||
| size_t size; | ||
| if (!PyArg_ParseTuple(args, "y#", &codeletset_unload_req_c1, &size)) { | ||
| PyErr_SetString(PyExc_TypeError, "wrong argument type"); | ||
| return Py_BuildValue("i", 1); | ||
| } | ||
|
|
||
| // Unload the codeletset | ||
| int res = jbpf_codeletset_unload(codeletset_unload_req_c1, &err); | ||
| if (res != 0) { |
Comment on lines
24
to
+35
| if [[ "$JBPF_STATIC" == "1" ]]; then | ||
| OUTPUT="$OUTPUT Building jbpf as a static library\n" | ||
| FLAGS="$FLAGS -DJBPF_STATIC=on" | ||
| else | ||
| fi | ||
| if [[ "$JBPF_STATIC" == "0" || "$JBPF_STATIC" == "" ]]; then | ||
| OUTPUT="$OUTPUT Building jbpf as a dynamic library\n" | ||
| FLAGS="$FLAGS -DJBPF_STATIC=off" | ||
| fi | ||
| if [[ "$JBPF_STATIC" == "2" ]]; then | ||
| OUTPUT="$OUTPUT Building jbpf as a both shared and static libraries.\n" | ||
| FLAGS="$FLAGS -DJBPF_STATIC=both" | ||
| fi |
Comment on lines
+85
to
+88
| ## handle output bufs | ||
| stream_id_c2 = emulator_utils.create_random_stream_id() | ||
| print(f"stream_id_c1: {list(stream_id_c1.id)}") | ||
| print(f"stream_id_c2: {list(stream_id_c2.id)}") |
Comment on lines
+487
to
+492
| char* jbpf_path = getenv("JBPF_PATH"); | ||
| jbpf_logger(JBPF_INFO, "JBPF_PATH is %s\n", jbpf_path); | ||
|
|
||
| char tempPath[255]; | ||
| sprintf(tempPath, "%s/out/", jbpf_path); // Make sure to format first | ||
| pyTempPath = PyUnicode_FromString(tempPath); |
Comment on lines
+715
to
+717
| // check if the path exists | ||
| sprintf(tempPath, "%s/%s.py", path, python_module); | ||
| if (!checkFileExists(tempPath)) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Originall this was in #4 however, it got merged in by mistakes and reverted in #119
Re-opening this PR.
Overview
This PR introduces a Python-based emulator (
src/emulator/) that lets developers loadjbpf codelets, invoke hooks, call helper functions, and inspect output channels directly from
Python test scripts. It also hardens the LCM CLI / reverse-proxy parsers, refactors codeletset
validation, and adds a
JBPF_STATIC=bothbuild mode.What's included
src/emulator/emulator.cpp): embeds CPython, exposes jbpf hooks andhelper functions to Python, manages an output-channel message queue, time-event handling, and
report_stats/periodic_callhooks. Statically links againstlibjbpf.a.autogen-hook-wrappers.pygeneratesauto_generated_hooks_wrapper.hpp;ctypesgenconverts C headers (jbpf_lcm_api.h,jbpf_test_def.h,jbpf_perf_ext.h) intoPython bindings;
patch.pypost-processes them.src/emulator/test/):test_0,test_1,test_2,test_report_stats,test_periodic_call,test_time_event. Wired into CI viaRUN_EMULATOR_TESTS.docs/emulator.md.Supporting changes
JBPF_STATIC=bothbuilds shared + static libraries; covered bytest_build_utils.sh.const&, bounds checks vsJBPF_MAX_*,explicit required-field checks, fixed double
linked_mapslookup.validate_codeletsetrefactored to a singlegoto errexit with consistent logging.__get_custom/default_helper_functionsaccessors for the emulator.simple_output2test codelet + verifier coverage;lsan_agent.suppfor libpython leaks.Testing
Build with
RUN_EMULATOR_TESTS=1; emulator tests run viajbpf_emulator out/emulator/test/ <test>. Requiresctypesgenand Python dev headers.