Skip to content
BranchesTags

Repository files navigation

erpl-rev — query and replicate SAP through DuckDB

CI

Run DuckDB SQL on SAP data, and bulk-replicate SAP into DuckDB — with nothing to install inside SAP.

erpl-rev is a small C++ server that registers at the SAP gateway as an RFC destination. Your ABAP calls it with CALL FUNCTION '…' DESTINATION 'ERPL_REV' and DuckDB runs behind it. Two things, in equal measure: query SAP tables, CDS views and BW calculation views with full SQL (and join them to cloud data), and replicate them into DuckDB, parquet, DuckLake, Iceberg or any DuckDB-attachable warehouse. No DuckDB extension in SAP, no DDIC structures, no SLT/SDI/BODS — payloads travel as plain JSON over RFC.

It's the inverse of erpl — where erpl makes DuckDB call into SAP, erpl-rev has SAP call out into DuckDB.

Status: research prototype, but both paths — query (SQL, federation, console, live serving) and replication (table / CDS / BW → DuckDB, parquet

  • attached catalogs) — are verified end-to-end against a live SAP ABAP (A4H) system in scripts/e2e.sh.

Query — SQL on SAP data, from ABAP and beyond

  • Run DuckDB SQL from ABAP. Send a query, get typed rows back — or an interactive ALV grid via the in-GUI SQL console (Z_ERPL_REV_SQL, SE38).
  • Do what Open SQL can't. Joins across tables, window functions, aggregations and parquet scans over live SAP data, without staging anything.
  • Federate in one statement. Join a SAP slice against a cloud parquet / Iceberg / DuckLake dataset or an attached Postgres / BigQuery table, and hand the result back to ABAP — cross-system queries SAP can't do alone.
  • Query live, over the network. With --quack, remote DuckDB clients (notebooks, BI tools) query the same in-process data directly — no export step.
  • Big results, fixed memory. A streamed OPEN/FETCH/CLOSE cursor decodes results page-by-page, so result size isn't bounded by ABAP memory.

Replicate — bulk-load SAP into DuckDB & the lakehouse

  • Any SAP source. Replicate tables, CDS views (incl. WITH PARAMETERS) and BW / HANA calculation views into typed DuckDB — full or filtered, with source-side WHERE, column selection and idempotent UPSERT. Built for >100M-row loads.
  • Delta (incremental) too. Keep a target in sync loading only what changed — watermark, change-document (CDHDR/CDPOS), and snapshot-diff (deletes) methods, all merging server-side, idempotent and re-runnable. Customer-owned Open SQL only (no ODP / SAPI / RFC_READ_TABLE). See docs/delta.md.
  • Land in the open lakehouse. parquet / partitioned datasets, DuckLake or Iceberg, on local disk or cloud object storage (S3 / GCS / Azure).
  • Publish into a warehouse. ATTACH Postgres / MySQL / BigQuery / MotherDuck and push a SAP slice in with one SQL statement — see Push to MotherDuck below.
  • Fast & parallel — a live 10M-row run (50-of-400-column BSEG-shaped table, BELNR-partitioned, on the A4H trial / loopback):

Parallel replication throughput: aggregate rises to ~167k rows/s at 5 workers (10M rows in 60s) while per-worker throughput tapers from 47k to 33k rows/s

Workers Wall time Aggregate Per worker
2 106 s ~94,000 rows/s ~47,000 rows/s
4 65 s ~154,000 rows/s ~38,000 rows/s
5 60 s ~167,000 rows/s ~33,000 rows/s

Peak ~167k rows/s — 10M rows in a minute. Each worker bulk-loads a disjoint key range with a DuckDB Appender (~230× a naive per-row path); memory is bounded by batch size, and loads are restartable and idempotent (test/bench_ingest.cpp).

Push to MotherDuck (DuckDB's cloud)

MotherDuck is just another DuckDB-attachable catalog, so the embedded engine reaches it exactly like Postgres / BigQuery / Iceberg — only the ATTACH and credentials differ. Point the server's boot init at MotherDuck once, then replicate or publish SAP slices straight into the cloud.

1. Boot the server attached to MotherDuck. Supply the token via the motherduck_token env var (or a CREATE SECRET in an --init-file) — never commit it:

export motherduck_token='<your-md-token>'
ERPL_REV_GWHOST=<gw> ERPL_REV_GWSERV=sapgw00 \
  ./build/erpl_rev_server --db erpl-rev.duckdb \
  --init-sql "INSTALL motherduck; LOAD motherduck; ATTACH 'md:';"

Your MotherDuck databases now appear as catalogs (e.g. my_db.main.<table>).

2. Push a SAP slice from ABAP — stage locally, then publish to the cloud:

zcl_erpl_rev_util=>replicate( iv_tab = 'MARA' iv_target = 'mara' ).
zcl_erpl_rev_util=>publish(                       " FULL = overwrite, APPEND = insert
  iv_source = 'mara' iv_kind = 'TABLE'
  iv_dest   = 'my_db.main.mara' iv_mode = 'FULL' ).

The publish field of Z_ERPL_REV_REPLICATE does the same from the GUI.

3. Query MotherDuck from the SQL console. Z_ERPL_REV_SQL ships an example dropdown with ready-to-run queries: the classic NYC-taxi public-Parquet aggregate, MotherDuck's shared sample_data (taxi + Hacker News), a SUMMARIZE, and a push-a-table round-trip — pick one and hit Execute.

The released bundle ships DuckDB with parquet / json built in; motherduck (and httpfs) auto-install from extensions.duckdb.org on first use, so the host needs outbound HTTPS — or pre-stage the extension for air-gapped systems.

Why it fits the SAP data stack

  • No SLT, no SDI, no Data Services, no add-on — a transport (package ZERPL) plus a registered RFC server. No core modification, no HANA license, no BTP. Runs against any NetWeaver ABAP stack (ECC, S/4HANA, BW/4HANA).
  • Reads what you model — tables, CDS views (keys auto-detected), BW/HANA calc views ("_SYS_BIC"."pkg/CV") — semantics intact, not raw dumps.
  • SLT semantics you know (LTRS) — field selection, source-side filter, key UPSERT — without standing up SLT.
  • DDIC-typed & provably faithful — NUMC / DATS / CURR / DECIMAL / RAW map to real DuckDB types; a built-in diff harness checks the target against the source cell-by-cell (incl. a 400-column BSEG-shaped table).

Security & authorizations. Only Z* objects (no core mod), reached via a type-T destination locked down with a gateway reginfo allow-list and (off-box) SNC. FM calls run as a comm user scoped to S_RFC for function group ZERPL_REV only; the reports run under the end user's auth (CDS DCL is enforced, raw-table SELECT is not implicitly S_TABU-checked — gate program execution; the native/ADBC BW path reads cross-client). Full guide: docs/security.md.

Install & setup

Just want to run it? Use uvx — nothing to download, nothing to unpack:

uvx erpl-rev --help          # runs the server; no install, no clone
uvx erpl-rev --smoke         # prove the RFC backend and DuckDB both load

or pip install erpl-rev if you would rather have it on PATH. Wheels are published for Linux x86-64, macOS arm64 and Windows x64.

Then operate it from the shell

Everything the SAP GUI reports do is also a command, so a headless server is a first-class place to work from:

uvx erpl-rev sql "SELECT count(*) FROM mara"      # the Z_ERPL_REV_SQL console
uvx erpl-rev replicate --table MARA --target mara # the Z_ERPL_REV_REPLICATE report
uvx erpl-rev sync ls                              # registered delta jobs
uvx erpl-rev sync run mara                        # one delta cycle
uvx erpl-rev sync schedule --every 5              # the periodic background job

sql and sync ls|show read the live DuckDB directly — over the server's quack listener when one is running, or the file when it is not, and they always print which. Anything that needs SAP to read source data submits the same report the GUI submits as a background job, so a load that takes hours is not bounded by an HTTP timeout.

Add --print-abap to any of them to see the ABAP instead of running it, and --dry-run to see the plan. Nothing writes to SAP or DuckDB without a terminal confirmation or an explicit --yes.

Parameters reach SAP as data: the CLI writes the command into a DuckDB table and the pre-deployed ZCL_ERPL_REV_CLIDRV executes it, so these commands need no SAP authorisation and create nothing. --queue-only does not contact SAP at all — the periodic ERPL_REV_DELTA job picks the command up.

Where the driver is not deployed they fall back to generating a temporary class, which does need S_DEVELOP. erpl-rev doctor reports which applies.

Then set up the SAP side

Getting the binary was always easy; getting SAP ready used to be an afternoon across four documents — a type-T destination, a function group, eight Z_DUCKDB_* modules, fourteen ABAP objects, a reginfo line. Two commands now do what a client can do, and hand over what it cannot:

uvx erpl-rev doctor          # read-only: what is missing, and the fix for each
uvx erpl-rev setup --dry-run # the exact change set, nothing written
uvx erpl-rev setup           # deploy, then prove a round trip before claiming success

setup deploys the ABAP over ADT, creates the destination and the function modules, and only reports success once ABAP has actually called back out through the registered server — "the objects exist" is not the same claim. Re-running it changes nothing. Every write needs a terminal confirmation or an explicit --yes.

Two things genuinely cannot be done from a client: the gateway reginfo allow-list and the gw/acl_mode profile parameters. For those setup writes erpl-rev-basis-handout.md, filled in for your system, with nothing left to compose. See docs/INSTALL.md for the manual path.

Three ways to get the same server, pick whichever suits:

how notes
PyPI uvx erpl-rev / pip install erpl-rev fastest; any Python 3
Release binary download from releases one self-extracting file, no Python
Docker docker pull ghcr.io/datazoode/erpl-rev:latest bakes the same bundle — see Run with Docker

All three carry DuckDB and nothing else: since v2026.08.30 the RFC protocol is erpl-proto, our pure-Rust implementation, linked statically — so there is no SAP NW RFC SDK, no ICU, and no LD_LIBRARY_PATH to set.

Whichever you pick you still do the one-time SAP-side wiring (step 3) and then run it (step 4).

The numbered steps below build from source — only needed to develop erpl-rev or to produce the bundle yourself (make bundle → dist/erpl-rev).

Prerequisites

  • Linux host with CMake ≥ 3.16, a C++17 compiler, Ninja, and vcpkg (supplies Catch2 for the tests).
  • The proprietary SAP NW RFC SDK (not redistributed — see below).
  • A reachable SAP gateway (any NetWeaver ABAP; a local A4H docker trial works).

1. Provide the SDK + DuckDB

The NW RFC SDK lives in a repo-local, gitignored nwrfcsdk/linux/ (same convention as erpl). Download it from the SAP Software Center, or copy it from an erpl checkout. DuckDB is fetched as an official prebuilt:

cp -a /path/to/nwrfcsdk ./nwrfcsdk     # provides nwrfcsdk/linux/{include,lib}
make duckdb-dist                       # fetch prebuilt libduckdb 1.5.4 into vendor/

2. Build & test

make build      # -> build/erpl_rev_server + build/erpl_rev_tests
make test       # the Catch2 suite against real DuckDB (no mocks)

make build also initialises the third_party/posthog-telemetry submodule, so a fresh clone needs no extra git submodule step.

3. Wire up the SAP side (one-time)

Production = import the ABAP transport and run the setup classrun — full guide in docs/INSTALL.md. You need three things in the SAP system:

  • a type-T ERPL_REV destination in registration mode (method='R') — created by ZCL_ERPL_REV_SETUP;
  • the ZERPL_REV function group + FMs (Z_DUCKDB_*) — created by ZCL_ERPL_REV_MKFM;
  • gateway registration allowed for the server's host — docs/enable-rfc-registration.md.

4. Run the server

Running the downloaded release binary (or the Docker image) needs no setup — just ./erpl-rev-linux-amd64 with the ERPL_REV_* env below; the bundle self-extracts and sets its own loader path. The LD_LIBRARY_PATH line is only for the from-source build/erpl_rev_server, whose libs live elsewhere in the tree:

export LD_LIBRARY_PATH=$PWD/nwrfcsdk/linux/lib:$PWD/vendor/duckdb-1.5.4
ERPL_REV_GWHOST=<gateway-host> ERPL_REV_GWSERV=sapgw00 \
ERPL_REV_DB_PATH=erpl-rev.duckdb \
  ./build/erpl_rev_server            # add --quack for the network server
# convenience: `make run` (quack on), `make run-mem` (in-memory), or `make run-no-quack`

Easiest is scripts/run-rfc-server.sh: it sets LD_LIBRARY_PATH, registers as ERPL_REV, and — opt-in via the environment — attaches MotherDuck (motherduck_token) and/or BigQuery (ERPL_REV_BQ_PROJECT). Pass -r to restart.

To publish to external / cloud catalogs (parquet, postgres, ducklake, bigquery, motherduck), give DuckDB boot SQL that runs INSTALL/LOAD/ATTACH (and CREATE SECRET) once on a global connection — via --init-sql "<sql>", --init-file <path>, or the ERPL_REV_DUCKDB_INIT env var.

For production, run it as a systemd service (deploy/erpl-rev.service) or via Docker (image below).

Run with Docker

Prebuilt linux/amd64 images are published to GitHub Container Registry:

docker run -d --name erpl-rev \
  -e ERPL_REV_GWHOST=<gateway-host> -e ERPL_REV_GWSERV=sapgw00 \
  -e ERPL_REV_PROGRAM_ID=ERPL_REV \
  -v erpl-data:/data \
  ghcr.io/datazoode/erpl-rev:latest
# add `--quack` (and `-p 9494:9494`) for the DuckDB network server

Config is entirely via ERPL_REV_* env vars; the DuckDB file lives on the /data volume. RFC registration is outbound to the gateway, so no inbound port is needed — the gateway's reginfo ACL must allow ERPL_REV_PROGRAM_ID from the container's host. Add --quack and publish -p 9494:9494 for the network server; docker run --rm ghcr.io/datazoode/erpl-rev:latest --smoke checks a pulled image loads with no gateway. See docs/docker.md.

5. Smoke test

  • ./build/erpl_rev_server --smoke (or the bundled binary) — loads the SAP NW RFC SDK + DuckDB and prints their versions; needs no gateway.
  • Z_ERPL_REV_SQL (SE38) → run SELECT 42 to confirm the ABAP → server → DuckDB round-trip (server must be running and registered).
  • Run Z_ERPL_REV_REPLICATE (SE38) on a small table and check row parity.

Quick start

Start the server — from PyPI, with nothing installed beforehand:

uvx erpl-rev                              # file-backed in ./erpl-rev.duckdb
uvx erpl-rev --db :memory:                # throwaway, nothing on disk
uvx erpl-rev --quack                      # + expose DuckDB to external clients

It registers its PROGRAM_ID at the SAP gateway and waits for ABAP to call out to it. The SAP side (destination + the Z_DUCKDB_* function modules) is the one-time setup in step 3 above.

Query SAP data with SQL — Z_ERPL_REV_SQL (SE38) opens a DuckDB SQL console in the SAP GUI: type any query (over replicated SAP data, cloud parquet, or attached catalogs) and get an ALV grid back; or call the query FM from ABAP and receive typed rows. From an external DuckDB client (with --quack):

-- from any DuckDB client
INSTALL quack; LOAD quack;
SELECT * FROM quack_query('quack:host:9494',
                          'SELECT * FROM <table>',
                          token => '<token>');   -- the live in-process data

Replicate a table — Z_ERPL_REV_REPLICATE (SE38): pick the source (F4 to search the DDIC), optionally pick columns (F4) and a WHERE, choose a target; keys are auto-kept so re-runs dedup. For >100M rows, tick parallel and run in background. Mirrors SLT's LTRS knobs — details below.

Replicating SAP tables — the SLT-style detail

Z_ERPL_REV_REPLICATE maps to the three per-table controls of SAP SLT (LTRS):

SLT concept Parameter Behaviour
Table selection p_tab source SAP table / CDS view (F4 search).
Field selection p_cols columns to replicate (blank = all); keys always kept.
Filter (at source) p_where OpenSQL WHERE, applied in the SAP SELECT so non-matching rows never transfer.
target / init / mode p_target p_init p_mode p_maxrow p_verify DuckDB table name; pre-SQL; UPSERT/INSERT; row cap; count-parity check.

Reads are package-wise (keyset pagination, 50k/batch) so memory is bounded; full-load-replace makes a crashed run safely re-runnable. The data-identity test (zcl_erpl_rev_difftest) compares target vs source cell-by-cell (SFLIGHT, ZWIDE_BSEG, REPOSRC + a negative control), byte for byte including trailing zeros — RAW columns replicate faithfully.

How it works

ABAP ──CALL FUNCTION 'Z_DUCKDB_QUERY'/'Z_DUCKDB_INGEST' DESTINATION 'ERPL_REV'──►
   SAP gateway (registered-server routing, RFCOPTIONS H=RFCSERVER)
      └──► erpl_rev_server (C++) ──► DuckDbBridge ──► DuckDB (parquet / lakehouse)

A registered RFC server (RfcCreateServer/RfcLaunchServer) hosts a handful of function modules whose payloads are JSON / binary-sXML over scalar STRING params — schema-generic, so no custom DDIC structures. It links the official prebuilt DuckDB 1.5.4 (libduckdb.so, parquet+json+quack built in); our code plus libstdc++/libgcc are static, leaving only libduckdb.so and the SAP .so trio dynamic.

Configuration (env vars & flags)

12-factor: config from the environment, logs to stderr, graceful SIGINT/SIGTERM. CLI flags override env (flag > env > default); --help prints the full surface.

Concern Flag Env var Default
Gateway PROGRAM_ID — ERPL_REV_PROGRAM_ID ERPL_REV
Gateway host / service — ERPL_REV_GWHOST / ERPL_REV_GWSERV localhost / 3300
Parallel registrations — ERPL_REV_REG_COUNT 5
Enable quack --quack[=<listen>] ERPL_REV_QUACK off
Quack bind / token --quack-listen / --quack-token ERPL_REV_QUACK_LISTEN / ERPL_REV_QUACK_TOKEN quack:localhost (port 9494) / random
DuckDB file --db <path> ERPL_REV_DB_PATH erpl-rev.duckdb (:memory: for in-mem)
Boot init SQL --init-sql / --init-file ERPL_REV_DUCKDB_INIT — (ATTACH/secrets for external/cloud targets)
Telemetry opt-out --no-telemetry ERPL_REV_NO_TELEMETRY / DATAZOO_DISABLE_TELEMETRY on by default (docs)
Self-check & exit --smoke — —
Logging — ERPL_REV_LOG_{LEVEL,FORMAT,COLOR} info / console / auto

A file-backed --db makes ingested (and quack-served) data durable across restarts. The quack token is a bearer credential — pin a high-entropy value via --quack-token (it's redacted from the log) and keep the listener on loopback unless you intend remote access.

Build internals & troubleshooting
  • The build resolves the SDK from nwrfcsdk/linux (override -DSAPNWRFC_HOME=… / make build NWRFC_HOME=…); Catch2 via vcpkg manifest mode (VCPKG_ROOT).
  • CI builds the server + runs tests on every push; it pulls the SDK from S3 via the same GitHub-OIDC→AWS role as erpl (scripts/download_and_extract_nwrfc.sh).
  • Registered destination must be method='R' (H=RFCSERVER) — "start" mode makes the gateway try to launch an executable and the call never reaches us.
  • The FM interface must exist in the backend or ABAP marshalling returns SYSTEM_FAILURE — ZCL_ERPL_REV_MKFM creates them.
  • Run with LD_LIBRARY_PATH=$NWRFC_HOME/lib — libsapnwrfc.so dlopens ICU by name, so rpath alone is insufficient.

Feedback

If erpl-rev misbehaves, please open an issue. This is a registered RFC server talking to gateways and SAP releases we cannot reproduce here, so a report with your setup is the fastest path to a fix.

If it saved you time, a star on the repo helps others find it.

Because this process usually runs under a service manager rather than on a terminal, the feedback pointer is emitted as an INFO log line at startup — where operators actually read it. On an interactive start you also get a small banner, at most once a day; silence both with DATAZOO_NO_BANNER=1.

Telemetry

The server sends anonymous usage telemetry (application_start / application_stop with app/version/platform/DuckDB-version only — no SAP data, query text, or table/field names) to help us understand adoption. It is on by default and disabled by any one of --no-telemetry, ERPL_REV_NO_TELEMETRY, or DATAZOO_DISABLE_TELEMETRY. Air-gapped SAP hosts drop the request silently with zero impact. Details: docs/telemetry.md.

Docs

License

Business Source License 1.1 (BSL), Licensor DataZoo GmbH, Change License MPL 2.0 — same terms as erpl.

About

Standalone C++ SAP RFC server bridging ABAP RFC into DuckDB (the inverse of erpl)

Topics

abapcppdata-integrationduckdblakehousereplicationsapsap-rfc

Resources

Readme
License

Security policy

Security policy
Activity
Custom properties

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages