Skip to content

docs: how to parallelise a narrow sap_read_table extract, and what it is worth #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jrosskopf wants to merge 1 commit into from
Closed

docs: how to parallelise a narrow sap_read_table extract, and what it is worth #132

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions API_REFERENCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1442,6 +1442,48 @@ running a large parallel extract against a production system.
`erpl_rfc_fetch_size`. Both remain supported and write the same value — they are two
names for one knob, not two knobs.

#### Narrow tables: `threads` will not help

`sap_read_table`'s parallelism is **per column**: it issues one concurrent
`RFC_READ_TABLE` call per projected column. That is a good fit for a wide extract and
no help at all for a narrow one — a single-column scan runs one call, and `threads`
has nothing to spread.

If you are extracting few columns from a large table, split it by a key range instead
and let SQL run the pieces:

```sql
SELECT * FROM sap_read_table('DD02L') WHERE TABNAME < 'D'
UNION ALL
SELECT * FROM sap_read_table('DD02L') WHERE TABNAME >= 'D' AND TABNAME < 'R'
UNION ALL
SELECT * FROM sap_read_table('DD02L') WHERE TABNAME >= 'R' AND TABNAME < 'V'
UNION ALL
SELECT * FROM sap_read_table('DD02L') WHERE TABNAME >= 'V';
```

The range predicates are pushed to SAP (see [Filter Pushdown](#filter-pushdown)), so
each branch reads only its own slice rather than filtering locally.

**What to expect.** Measured on a 164,664-row single-column extract, four branches ran
in 6.6s against 8.3s for the equivalent single scan — about **1.2x**, not 4x. Each
branch pays its own connection and metadata round-trip, and the SAP system applies each
range filter separately, so the gain is far smaller than the branch count suggests.
Splitting into many small branches makes it worse, not better.

Two things matter more than the branch count:

- **Pick boundaries that split the data evenly.** A range that covers most of the table
leaves you with one long branch and three short ones, and the long one sets the wall
time.
- **Split on an indexed key.** If SAP has to scan the whole table to evaluate each
range, you have multiplied the server's work rather than divided it.

For a genuinely large extract, running several *processes* against SAP scales
considerably better than either approach — concurrency limits on the SAP side apply per
client program, so separate processes are not aggregated the way threads within one are.
Check with your Basis team before pointing many parallel readers at a production system.

### SSH Tunnel + SAP Connection

Complete workflow for connecting through an SSH jump host:
Expand Down
Loading