Conversation
The committed src/parser.c at HEAD was generated against ABI 15, but package.json pinned tree-sitter-cli ^0.24.3 (which emits ABI 14). That mismatch made `npm test` fail on a fresh checkout with "Incompatible language version 15. Expected minimum 13, maximum 14". Bumping the dev pin to ^0.25.10 aligns the tooling with the parser already in the tree. No grammar or behavioral change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython's C tuple types ``(T1, T2, ...)`` were rejected anywhere a c_type
appears: as cdef/cpdef return types and parameter types, in ctypedef
declarations, in cast expressions, in sizeof, and in cdef extern from
blocks. The fix adds a new c_tuple_type rule and threads it through the
positions that need it.
c_tuple_type is *not* folded into c_type itself, because that puts it on
the path of typed_parameter (Cython's override sitting in Python's
parameter choice), where (a, b) collides with tuple_pattern in cases
like ``lambda (a, b): ...`` and ``def f((a, b)=v):``. Instead it's added
as an explicit alternative to:
* maybe_typed_name — cdef return type, parameter type, variable decl
* cvar_decl — ctypedef and cdef extern from declarations
* cast_expression — ``<(T1, T2)>x``
* sizeof_expression — ``sizeof((T1, T2))``
This matches the positions where real Cython code uses tuple types
(verified against h3-py and the cext-review-toolkit calibration corpus).
Real-world impact: closes the parse cascade in h3-py's _cy/{vertex,
util,edges,latlng,cells}.pyx, which collectively went from 47 ERROR
nodes to 0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython 3 supports Python-style ``x: T = default`` parameters in def functions, but Cython's override of typed_default_parameter only defined the C-style ``T x = default`` form, dropping the Python annotation form from the parent grammar. Mirrors the structure of typed_parameter, which already has both forms as a choice. Either form is accepted; the Python-style branch produces the same AST shape as the original tree-sitter-python grammar. Real-world impact: closes lxml/etree.pyx:1382-style declarations like ``def index(self, child: _Element, start: int = None, stop: int = None):``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``ctypedef public T (*name)(...)`` failed to parse because c_function_pointer required a c_type at its head, with no slot for the public/api/inline storageclass that Cython accepts on ctypedefs. Other ctypedef forms accepted post-ctypedef storageclasses through cvar_decl, but the function-pointer branch was missing it. Adding ``repeat($.storageclass)`` mirrors the existing prefix on cvar_decl and on struct/enum. Real-world impact: closes lxml/etree.pyx:481-style declarations like ``ctypedef public xmlNode* (*_node_to_node_function)(xmlNode*)``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython tolerates a C-style trailing semicolon on declarations, but the
grammar required none. Added ``optional(';')`` before ``$._newline`` in
the variable-declaration branches of cvar_def and cvar_decl, and on the
declaration-only branch of c_function_definition.
This handles three real patterns:
* ``cdef int64_t x;`` inside a def body (cvar_def)
* ``int func(int a);`` inside cdef extern from (c_function_definition)
* ``cdef int x; cdef int y;`` two cdef on one line (works as a
side effect — the optional ';' eats the separator)
Real-world impact: closes blosc2/blosc2_ext.pyx-style declarations with
trailing semicolons (excerpts that contain a complete extern block now
parse cleanly).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``const char* const* X "alias"`` failed because the parser greedily took the second ``*`` as a c_name type_modifier prefix instead of extending the c_type's modifier list. After committing to c_name, no required identifier remained and the declaration errored out. The c_name rule's ``optional($.type_modifier)`` prefix was unused — no corpus test exercised it, and any modifier can equivalently appear as a trailing modifier on c_type. Removing it (c_name is now just $.identifier) makes c_type's ``repeat($.type_modifier)`` consume the entire modifier chain, leaving the identifier slot for the actual declared name. Real-world impact: closes lxml/etree.pyx:349-style declarations like ``const char* const* _LXML_LIB_FEATURES "_lxml_lib_features"``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adding ``type`` to keyword_identifier alongside match/async/await/api makes it usable as a regular identifier when the surrounding context isn't a type-alias statement (Python 3.12's ``type X = T`` form). Without this, statements like ``type, value, traceback = self._exc_info`` (common in Python-2-compatible exception handling) and plain ``type = something`` failed: the grammar's filter dropped the [type_alias_statement, primary_expression] conflict from the parent Python grammar without compensating with a soft-keyword fallback, so the parser deterministically committed to type_alias_statement and errored when the syntax didn't fit. Both ``type`` as identifier and ``type X = int`` as type-alias parse correctly with this change. Real-world impact: closes lxml/etree.pyx:472-style tuple unpacks like ``type, value, traceback = self._exc_info``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython function-pointer types can carry the same trailing clauses as function definitions: ``except``/``except?``/``noexcept`` and ``nogil``/``with gil``, in either order. The grammar's c_function_pointer rule was missing these slots, so common patterns like ctypedef int (*execute_fn)(...) except? -1 failed to parse. Adding the optional clauses creates an LR(1) ambiguity (the parser can't tell if a trailing ``with`` extends c_function_pointer or starts the next statement), so c_function_pointer now also requires an explicit ``$._newline`` terminator. This is consistent with how cvar_def and cvar_decl already terminate their declaration branches, and matches how struct_suite separates fields anyway. Real-world impact: closes the last remaining error in the cext-review calibration corpus (msgpack-python/_unpacker.pyx:49). Aggregate corpus now parses with 0 ERROR/missing nodes (was 60 at HEAD 575484a). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython's cppclass templates allow default types like ``cdef cppclass Map[K, V=int]:`` (the V parameter defaults to int when not explicitly specified). The existing template_default rule only accepted ``=*`` (Cython's "infer default" sentinel), rejecting any concrete type as a default. Extending the choice to include c_type covers the common case while preserving the ``=*`` sentinel form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Several patterns common in Cython .pxd headers wrapping C++ libraries
were rejected by the grammar. This single commit groups the related
fixes because they all enable parsing of C++ extern blocks (the
characteristic content of cppclass-heavy .pxd files):
* **enum aliases**: ``cdef enum Type "ns::Type":`` — the C-name alias
syntax that struct/union/cppclass already supported was missing on
enum.
* **cppclass inheritance**: ``cdef cppclass Derived(Base):`` and
multiple/template inheritance — the rule had no parent-class slot.
* **destructor names**: ``~ClassName()`` inside cppclass bodies —
added a destructor_name rule (~+identifier) and threaded it into
maybe_typed_name's name slot alongside identifier and operator_name.
* **method aliases in cppclass bodies**: ``int method "renamed"(args)``
— cvar_def now accepts an optional alias string between the typed
name and the function definition, mirroring cvar_decl.
* **decorators on cppclass members**: ``@staticmethod`` before a
method declaration — the cppclass body now accepts a sequence of
decorators followed by cvar_def.
Real-world impact: pyarrow's libarrow.pxd dropped from 1627 to 0
ERROR/missing nodes; the broader pyarrow .pxd suite from 2068 to 3
errors. The remaining 3 errors are unrelated patterns to investigate
separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython supports the ``except +*`` shorthand on C++ extern functions — combining ``except +`` (translate C++ exceptions to Python) with the ``except *`` semantics (probe for any exception state on every return). The grammar's exception_value rule allowed ``except +`` followed by an optional identifier (for ``except + CustomException``), but not the ``*`` form. Extending the choice to ``optional(choice($.identifier, "*"))`` covers the documented patterns without affecting existing ones. Real-world impact: closes the last 3 errors in pyarrow's libparquet_encryption.pxd (and other arrow .pxd files using the same pattern). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related extensions for code generated by modern Cython projects: 1. **Hybrid C-type + Python annotation on parameters**: forms like ``def fn(Comm comm: Comm | None = None)`` combine Cython's type-first declaration with a Python-style annotation. typed_parameter and typed_default_parameter each gained an optional ``": " <annotation>`` slot between the C-typed name and any default. 2. **Soft-keyword type names**: ``cdef type cls = Comm`` and similar uses of soft keywords (type, match, etc.) as type names. The maybe_typed_name rule's "type" and "name" slots now accept keyword_identifier alongside identifier — matching how the parent Python grammar treats these positions. 3. **Constant expressions as array sizes**: ``cdef int arr[N+1]`` inside extern blocks and cdef bodies. type_index previously accepted only ``$.integer`` as the bracket contents; expanded to ``$.expression`` (a strict superset). Two new conflicts were declared ([typed_default_parameter, typed_parameter] and [typed_parameter]) to resolve the optional-annotation ambiguity at the LR boundary between the new annotation slot and what follows it. Real-world impact: mpi4py/Comm.pyx dropped from 446 to 0 errors; similar improvements in mpi4py's other modules (File, Win, Datatype, Request) and in Cython's own MemoryView utility. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related extensions for code in pyhmmer (244 → 0 errors after fix): 1. **``not None`` / ``or None`` constraint with default value**: forms like ``Alphabet alpha not None = Alphabet.dna()`` combine the non-None constraint (already supported on typed_parameter) with a default value (only on typed_default_parameter). The constraint was only honored when no default was present. 2. **``new`` as soft identifier**: Cython uses ``new`` as a keyword for the C++ ``new`` expression (``new vector[int]()``), but real-world code also uses ``new`` as a variable name (e.g., ``cdef Foo new = Foo(); new.method()``). Adding ``new`` to keyword_identifier alongside ``type``/``match``/``api`` lets it serve both roles — the new_expression rule still wins in its specific context, and the soft-keyword fallback handles plain identifier use. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython lets a cast expression spell out a function-pointer type with its calling-convention clauses, e.g. (<float(*)(float, float) noexcept nogil>fn_ptr)(x, y) The c_function_pointer_type rule only had ``c_type (* ) c_parameters`` and rejected anything after — so noexcept/nogil at the tail produced ERROR nodes inside the cast. Adding ``optional($.exception_value)`` and ``optional($.gil_spec)`` mirrors the structure of c_function_pointer (recently extended for ctypedef) without the trailing _newline that the standalone form requires (here the cast's ``>`` delimits the type). Real-world impact: scipy/special/cython_special.pyx dropped from 67 to 0 errors; similar patterns clear in scipy's _ufuncs.pyx. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython lets cppclass bodies declare nested types directly (without
prefixing each with ``ctypedef``):
cdef cppclass Outer:
struct Inner: # nested struct
int x
union U: # nested union
int i
enum E: # nested enum
VAL_A
The _cppclass_suite rule already accepted ctypedef_statement,
cvar_def, cppclass, and decorated cvar_def — but not bare struct or
enum (which the standalone struct/enum rules already define). Adding
``$.struct`` and ``$.enum`` to the body's choice closes the gap.
Real-world impact: closes Cython's own cpp_nested_classes.pyx test;
fixes similar nested-type patterns in any cppclass-heavy .pxd.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython supports the 1-element ``(T,)`` form (analogous to Python's
1-tuple syntax) and trailing commas in multi-element tuple types.
The original c_tuple_type rule required ``c_type "," commaSep1(c_type)``,
which forced at least 2 elements (no way to get a 1-tuple) and didn't
naturally permit a trailing comma after the last element.
Restructured to ``c_type (',' c_type?)+`` — at least one comma, with
the c_type after each comma being optional. This admits:
(T,) — 1-tuple
(T1, T2) — 2-tuple
(T1, T2,) — 2-tuple with trailing comma
(T1, T2, T3, ...)
while still excluding ``(T)`` (which is paren-wrap, not a tuple).
Real-world impact: closes the MISSING-node errors in Cython's own
ctuple.pyx test (line 143-144 ``cdef (Union,) a = ...``).
Note: tuple types in Python-style ``def`` parameter lists (e.g.
``def f((int, double) xy)``) remain unsupported — adding them here
would conflict with tuple_pattern in ``lambda (a, b): ...`` and
``def f((a, b)=v)`` cases that pass corpus tests today. The cdef/
cpdef route already supports tuple-typed parameters via maybe_typed_name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
C++ libraries declare free-function operator overloads in extern
blocks like:
cdef extern from "lib.h":
bint operator==(const string_view& lhs, const string_view& rhs)
The cvar_decl rule expected a plain identifier or function name in the
c_name slot. Extending c_name from ``$.identifier`` to
``choice($.identifier, $.operator_name)`` lets ``operator==`` (and the
full set of operator names already defined for member functions) serve
as the function name for top-level extern declarations.
Real-world impact: closes resiliparse_inc/string_view.pxd (20 → 0
errors), and similar patterns in any .pxd wrapping C++ free-function
operator overloads.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ``[type T, object O]`` external_definition specifier (used on
``ctypedef api class`` declarations to bind C type / object type names)
permitted no trailing comma. Real Cython code (mpi4py, vtk-mpi4py)
formats this list across lines with a trailing comma, e.g.
ctypedef api class Datatype [
type PyMPIDatatype_Type,
object PyMPIDatatypeObject,
]:
cdef MPI_Datatype ob_mpi
Adding ``optional(",")`` after the commaSep1 mirrors how Cython itself
permits trailing commas in argument lists, tuple types, and parameter
lists.
Real-world impact: mpi4py/MPI.pxd 20 → 0 errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython's .pxd declaration files use ``= *`` and ``= ?`` as "default-value-elided" sentinels — the actual default lives at the .pyx definition. The grammar accepted ``= *`` already but rejected ``= ?``, even though both are documented Cython. Real-world impact: pyhmmer/plan7.pxd (17 → 0 errors), and any other .pxd that elides default values with ``=?``. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython's external type spec ``[object T, type T2]`` also accepts ``check_size <value>`` (where value is one of ``min``, ``max``, ``ignore``, ``error``, ``warn``) — Cython 3 uses this to control struct size compatibility checking against the wrapped C type. The grammar only allowed the ``object`` and ``type`` directives. Real-world impact: hunter/_event.pxd 16 → 0 errors, plus other .pxd / .pxi that wrap numpy/cpython internals using check_size. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vizonex
approved these changes
Jun 26, 2026
Vizonex
left a comment
Vizonex
left a comment
There was a problem hiding this comment.
This looks really well thought out. I thought I help with the reviewing process incase needed. Seems to be untouched for some strange reason. I would say write a test with a pytest workflow of some kind so that the reviewer is more likely to accept this but all in all this change looks fantastic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
We're building a tool that needs to parse Cython into ASTs and found your parser. Along the way we added support for some syntax patterns we ran into in real-world Cython code and would like to share it back.
What's in this PR
19 fix commits + 1 chore, each atomic (one grammar change per commit, with a corpus test). Grouped by area:
Type-system declarations
bc1578cC tuple types(T1, T2, ...)in cdef return types, parameters, ctypedef, casts, sizeof887a676doubled-const declarationsconst char* const* Xin extern blocks8779f6c1-tuple form(T,)and trailing commas in tuple types114827dcppclass template defaults[K, V=int]7b469a1(part of) array sizes with constant expressionsint arr[N+1]; soft-keyword type namescdef type clsParameter forms
7321edcPython-style annotated parameter with defaultdef f(x: int = 1)7b469a1(part of) hybrid C type + Python annotationComm comm: Comm | None = None1924b7fnot None/or Noneconstraint combined with default value2322629=?default sentinel for.pxdfilesFunction-pointer & exception clauses
746e80bgil_specandexception_valueon function-pointer ctypedef (e.g.except? -1)4e18172gil_specandexception_valueon cast-to-function-pointer (e.g.<T(*)(args) noexcept nogil>)4331559except +*shorthand for C++ exception conversionC++ extern expansion (
.pxdfiles)e76d7ffenum aliasescdef enum Type "ns::Type":, cppclass inheritance(Base), destructor names~ClassName(), method aliasesint method "renamed"(args), decorators on cppclass members14f2e47nestedstruct/union/enuminside cppclass bodies3a672dafree-functionoperator==etc. in extern blocksb2f6fd0check_size <ident>directive in[object T, ...]external_definition104d94ctrailing comma in external_definitionLexical tweaks
b474449typeas a soft keyword (usable as identifier)1924b7f(part of)newas a soft keywordb420027storageclass on ctypedef function-pointer (ctypedef public int (*F)(int))232d3a8optional trailing;on declarationsRepo tooling
ece06d5bumptree-sitter-clipin to0.25.10. The committedsrc/parser.cwas at LANGUAGE_VERSION 15 butpackage.jsonpinned^0.24.3(which emits LANGUAGE_VERSION 14). On a fresh checkoutnpm testfails withIncompatible language version 15. Expected minimum 13, maximum 14. Aligning the dev pin with the parser already in the tree fixes the build.Validation
Known not fixed
A few patterns we tried but couldn't land cleanly without breaking existing corpus tests:
defparameters (def f((int, double) xy)): collides withtuple_patternin yourlambdasandDefault Tuple Argumentscorpus tests. Thecdef/cpdefroute already handles tuple-typed parameters via a separate path.externinsidecdef extern fromblocks (extern int func()): addingexternas astorageclassproduces an LR conflict with the outercdef externform.*(void kernel(args, ...)): structurally indistinguishable from typed parameters; broke 5+ corpus tests in our attempts.function[double(double, int) noexcept](function-signature as a template argument): would needtype_indexto accept function-like type expressions; we couldn't disambiguate cleanly.Happy to revise
If you'd prefer this split into smaller PRs, we have natural groupings (the section headers above are good cut-lines) and can re-organize. Equally happy to revise individual commits, drop any you'd reject, or rework the approach in any other way. Just let us know what works for your review style.
Thanks for the great parser!