feat(tools): add -j/--jobs option to idf.py (IDFGH-18013)

[nebkat]

Description

Add a global -j/--jobs option to idf.py that sets the number of parallel build jobs passed to the underlying build tool (Ninja or Make).

Previously, build parallelism could only be configured via the IDF_PY_BUILD_JOBS environment variable, and that value was only honored for the Ninja generator — the Make generator always used a hardcoded -j <cpu_count + 2>.

Usage:

idf.py -j 6 build              # or --jobs 6
IDF_PY_BUILD_JOBS=6 idf.py build   # still works; used as the default for -j

Being a root-level option (like -G and --ccache), it must appear before the subcommand (idf.py -j 6 build), at most once.

Testing

Behavior was verified by exercising the real modules directly:

• CLI parsing — -j 4, --jobs 8, and IDF_PY_BUILD_JOBS=6 all resolve to the correct integer; -j 0, negative, and non-integer values (including a bad IDF_PY_BUILD_JOBS) are rejected at parse time by click.IntRange(min=1).
• With IDF_PY_BUILD_JOBS set, -j no longer conflicts with the env var (the option is now root-level only, not duplicated onto every subcommand).
• Command construction (run_target):
  • Ninja: ninja when unset, ninja -j 4 when -j 4 is given.
  • Make: make -j 16 by default (cpu + 2), make -j 4 when overridden — a single -j, not a duplicated flag.

Checklist

Before submitting a Pull Request, please ensure the following:

• 🚨 This PR does not introduce breaking changes.
• All CI checks (GH Actions) pass.
• Documentation is updated as needed.
• Tests are updated or added as necessary.
• Code is well-commented, especially in complex areas.
• Git history is clean — commits are squashed to the minimum necessary.

---

Note

Low Risk
Documentation and idf.py build-wrapper changes only; behavior is additive and preserves existing env-based configuration for Ninja.

Overview
Adds a global idf.py -j / --jobs option (with IDF_PY_BUILD_JOBS as the default when the flag is omitted) to set how many parallel jobs are passed to Ninja or Make.

run_target() now resolves job count in one place: CLI/env value first, otherwise the generator’s default_jobs (Make keeps cpu_count + 2; Ninja stays unpinned when unset). Make no longer bakes -j into its static generator command, avoiding duplicate -j flags when overridden.

English and Chinese build-system docs describe idf.py -j 6 build and the env var.

^{Reviewed by Cursor Bugbot for commit db6b310. Bugbot is set up for automated code reviews on this repo. Configure here.}

[dobairoland]

Hi @nebkat. Thank you for the contribution. LGTM in general. One request about where the option is declared.

global_options (tools/idf_py_actions/global_options.py) is the list for -D/--define-cache-entry, a CMake cache entry, and it gets spliced into the option list of the subcommands that run the configure step. -j is a different kind of knob: it is read only in run_target() and only affects the ninja/make invocation. Reusing that list has some odd effects. idf.py reconfigure -j 8 is accepted and ignored, because reconfigure only calls ensure_build_directory(). flash and erase-otadata take -j, while app-flash and bootloader-flash do not, since the list was never curated for this.

It also breaks the environment variable case. The option ends up registered both at root level and on each action, so click resolves IDF_PY_BUILD_JOBS separately for the two copies and idf.py reports them as conflicting:

$ IDF_PY_BUILD_JOBS=6 idf.py -j 4 build
ERROR: Option "jobs" provided for "all" is already defined to a different value.
This option cannot be set in command line if the IDF_PY_BUILD_JOBS environment
variable is set to a different value.

CI sets IDF_PY_BUILD_JOBS: "6" for every job in .gitlab/ci/common.yml, so -j would be unusable there with any other value, and the same happens for anyone who exports the variable in their shell.

Could you move the option to root_options['global_options'] in core_ext.py, next to -G, -v and --ccache? Those are the other options that control how the build tool is invoked, and they are root level only. Declaring the type as click.IntRange(min=1) also moves validation to parse time, so -j 0 and a non-integer environment variable are rejected before CMake runs and the manual check in run_target() can go away.

{
    'names': ['-j', '--jobs'],
    'help': 'Number of parallel build jobs passed to the build tool (Ninja or Make).',
    'envvar': 'IDF_PY_BUILD_JOBS',
    'type': click.IntRange(min=1),
    'default': None,
},

The visible difference for users is that -j then has to come before the subcommand, idf.py -j 4 build instead of idf.py build -j 4, which is how -G and --ccache already work.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit db6b310. Configure here.}

[cursor]

Post-subcommand jobs option rejected

Medium Severity

-j/--jobs is declared only in root options with default scope, so Click accepts it before a subcommand but not after. The change is described as a global option like -D that works as idf.py build -j 6, but that form fails with an unknown-option error because the flag is never attached to subcommands or given scope: 'global'/'shared'.

 

^{Reviewed by Cursor Bugbot for commit db6b310. Configure here.}

[fhrbata]

LGTM, thank you for the contribution!

[fhrbata]

Hi @nebkat, thanks again, and for turning around the earlier review so quickly. The change itself looks good to me.

There's just one small CI thing to sort out before we merge. The pre_commit_check job is failing on the ruff-format hook, and it's not actually your feature. The only reason it trips is that the PR touches tools/idf_py_actions/global_options.py (just the copyright-year line). That file has some pre-existing formatting our current ruff-format wants to rewrite (the -D / --define-cache-entry list literal), and pre-commit only reformats files that a change actually touches. So the moment your PR modifies that file, ruff reformats the unrelated -D block and CI fails because pre-commit changed a file. It is a latent quirk on our side, nothing you did wrong.

The clean fix: the -j / --jobs option lives entirely in core_ext.py, so global_options.py doesn't need to be in this PR at all. Its only change is the copyright bump, so reverting that one line (back to what's on master) drops the file out of the diff completely, ruff won't touch it, and the job goes green:

# from your feat/idfpy-jobs branch, restore the file to upstream esp-idf master
git checkout upstream/master -- tools/idf_py_actions/global_options.py
git commit --amend --no-edit
git push --force-with-lease

(Adjust upstream/master to whatever remote you use for espressif/esp-idf.)

Would you be able to do that? The reason I'm asking rather than just patching it on our side: we merge contributions through our internal repo and then sync back to GitHub. If we merge your commit unchanged, GitHub automatically marks this PR as Merged and it's recorded as your contribution. If we have to modify your commit ourselves, even a one-line revert, that automatic link breaks and we'd have to close this PR manually instead, so it would not show as Merged or count toward your contributions. I'd rather you keep the credit, hence the small ask.

If you'd prefer not to, no problem at all, just let me know and I'll fix it on our side during the merge. In that case the PR would end up Closed rather than Merged, but the change will still land.

Thanks!

[hfudev]

LGTM.