Skip to content

Stable code identifier pickling - #40711

Open
kristynsmith wants to merge 24 commits into
apache:masterfrom
kristynsmith:stable-code-identifier-pickling
Open

kristynsmith wants to merge 24 commits into
apache:masterfrom
kristynsmith:stable-code-identifier-pickling

Conversation

@kristynsmith

Copy link
Copy Markdown
Contributor

Fixes several edge cases in code_object_pickler when generating and resolving stable code object identifiers:

  • Addressed the case where multiple nested def functions have the same name (e.g. defined across if/else branches) by applying the same signature and bytecode-hash disambiguation in _search_locals (formerly _search_lambda) to any duplicate code object name in co_consts, and updating _OBJECT_WITH_ARGS_PATTERN and _OBJECT_WITH_HASH_PATTERN to match named functions in addition to <lambda>.
  • Addressed the case where one of multiple lambdas/nested functions takes no arguments by updating _OBJECT_WITH_ARGS_PATTERN to match empty argument tuples ().
  • Addressed the case where two lambdas or nested functions have identical co_code bytecode instructions (e.g. referencing different closure variables or constants) by including co_consts, co_names, co_varnames, co_freevars, and co_cellvars in _create_bytecode_hash.
  • Addressed the case where a parent package re-exports a symbol with the same name as a submodule (e.g. from pkg.submod import submod in pkg/__init__.py) by resolving the longest matching module prefix in get_code_from_identifier before traversing attributes.
  • Addressed the case where an outer and inner function share the same decorator wrapper by returning None (falling back to default pickling) instead of raising ValueError('Qual name parts too long') when a branch does not match the full __qualname__.

Replace config assignment with dataclasses.replace for stability.
Refactor code object handling and improve normalization for hashing.
Added new test cases for lambda functions and decorators in code_object_pickler.
Refactor code object search logic to handle local code objects and update lambda handling. Replace lambda-specific functions with more general object handling functions.
Added tests for handling same-named nested functions with different and same arguments in code object pickler.
@github-actions github-actions Bot added the size/m M: 50-249 lines changed label Oct 27, 2026
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.82%. Comparing base (a8d6afd) to head (f78c349).
⚠️ Report is 135 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #40711      +/-   ##
==========================================
- Coverage   80.04%   79.82%   -0.23%     
==========================================
  Files         490      494       +4     
  Lines       82916    83594     +678     
==========================================
+ Hits        66373    66731     +358     
- Misses      16543    16863     +320     
Flag Coverage Δ
python 79.82% <100.00%> (-0.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Added tests for handling complex nested functions and edge cases in code object pickling.
Refactor function names for clarity and consistency in tests.
@kristynsmith kristynsmith reopened this Oct 29, 2026
@kristynsmith
kristynsmith marked this pull request as ready for review October 29, 2026 17:19

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves several edge cases in code_object_pickler around duplicate nested function names, zero-argument signatures, identical bytecode instructions with differing constants/closures, module prefix resolution, and shared decorator wrappers. The logic changes are clean, accurate, and well-covered by new unit tests.

🔍 General Feedback

  • The generalization of _search_locals and the regex patterns (_OBJECT_WITH_ARGS_PATTERN and _OBJECT_WITH_HASH_PATTERN) cleanly handles both <lambda> and duplicate named local functions.
  • Including normalized co_consts, co_names, co_varnames, co_freevars, and co_cellvars in _create_bytecode_hash properly prevents hash collisions when co_code alone is identical.
  • One minor issue in _normalize_const: frozenset iteration order is non-deterministic across processes, so its elements should be sorted when normalizing for hashing.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Comment thread sdks/python/apache_beam/internal/code_object_pickler.py Outdated
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @tvalentyn for label python.
Backup reviewer: claudevdm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant