Repository navigation
[pull] main from apache:main - #282
Merged
Merged
Conversation
…ntations (#22913) ## Which issue does this PR close? - Closes #22912. ## Rationale for this change This change makes testing null_aware behavior easier, and also makes the performance of various joins clearer - null_aware joins do extra work. This was originally part #21585, but it seems like there is a bunch of activity around null-aware joins, so I figured its worth splitting out. ## What changes are included in this PR? Add a `null_aware` indication to relevant Display implementations when appropriate. ## Are these changes tested? SLT tests ## Are there any user-facing changes? Display only --------- Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
Bumps [pyjwt](https://github.com/jpadilla/pyjwt) from 2.12.0 to 2.13.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/jpadilla/pyjwt/releases">pyjwt's releases</a>.</em></p> <blockquote> <h2>2.13.0</h2> <h1>PyJWT 2.13.0 — Security Release</h1> <p>This release bundles five security fixes plus three additional hardening / spec-compliance changes. We recommend all users upgrade.</p> <h2>Security</h2> <ul> <li> <p><strong><a href="https://github.com/jpadilla/pyjwt/security/advisories/GHSA-xgmm-8j9v-c9wx"><code>GHSA-xgmm-8j9v-c9wx</code></a> — JWK JSON accepted as HMAC secret (algorithm confusion).</strong> <code>HMACAlgorithm.prepare_key</code> previously rejected PEM- and SSH-formatted asymmetric keys but did not catch a JWK passed as a raw JSON string. In a verifier configured with both symmetric and asymmetric algorithms in <code>algorithms=[…]</code> and a raw-JSON JWK as the key, an attacker could forge HS256 tokens using the JWK text as the HMAC secret. The guard has been extended to reject any JWK-shaped JSON. <em>Reported by <a href="https://github.com/aradona91"><code>@aradona91</code></a>.</em></p> </li> <li> <p><strong><a href="https://github.com/jpadilla/pyjwt/security/advisories/GHSA-jq35-7prp-9v3f"><code>GHSA-jq35-7prp-9v3f</code></a> — Algorithm allow-list bypass with <code>PyJWK</code> / <code>PyJWKClient</code>.</strong> When verifying with a <code>PyJWK</code>, the caller's <code>algorithms=[…]</code> allow-list was checked against the token header <code>alg</code> as a string only; actual verification used the algorithm bound to the <code>PyJWK</code>. An attacker who controlled a registered JWKS key could sign with one algorithm and advertise another on the header. PyJWT now requires the token header <code>alg</code> to match the <code>PyJWK</code>'s algorithm before verification. <em>Reported by <a href="https://github.com/sushi-gif"><code>@sushi-gif</code></a>.</em></p> </li> <li> <p><strong><a href="https://github.com/jpadilla/pyjwt/security/advisories/GHSA-w7vc-732c-9m39"><code>GHSA-w7vc-732c-9m39</code></a> — DoS via base64 decode of unused payload segment when <code>b64=false</code>.</strong> For detached-payload JWS (<code>b64=false</code>), the compact-form payload segment was base64-decoded before being discarded in favor of the caller-supplied <code>detached_payload</code>. An attacker could inflate the unused segment to force CPU + memory cost without holding a valid signature. The segment is now required to be empty per RFC 7515 Appendix F, and is no longer decoded. <em>Reported by <a href="https://github.com/thesmartshadow"><code>@thesmartshadow</code></a>.</em></p> </li> <li> <p><strong><a href="https://github.com/jpadilla/pyjwt/security/advisories/GHSA-993g-76c3-p5m4"><code>GHSA-993g-76c3-p5m4</code></a> — <code>PyJWKClient</code> accepts non-HTTP(S) URIs.</strong> <code>PyJWKClient.fetch_data</code> passed its URI to <code>urllib.request.urlopen</code>, which by default also handles <code>file://</code>, <code>ftp://</code>, and <code>data:</code> schemes. An application that fed an attacker-influenced URI into <code>PyJWKClient</code> could be coerced into reading local files or reaching other unintended schemes. <code>PyJWKClient</code> now rejects any URI whose scheme isn't <code>http</code> or <code>https</code>. <em>Reported by <a href="https://github.com/KEIJOT"><code>@KEIJOT</code></a>.</em></p> </li> <li> <p><strong><a href="https://github.com/jpadilla/pyjwt/security/advisories/GHSA-fhv5-28vv-h8m8"><code>GHSA-fhv5-28vv-h8m8</code></a> — <code>PyJWKClient</code> cache wiped on fetch error.</strong> A <code>finally</code>-block <code>put(jwk_set=None)</code> cleared the JWK Set cache whenever a fetch raised, turning a transient JWKS-endpoint outage into application-wide auth failure. The cache write was moved into the success path; transient errors no longer evict valid cached keys. <em>Reported by <a href="https://github.com/eddieran"><code>@eddieran</code></a>.</em></p> </li> </ul> <h2>Fixed</h2> <ul> <li>Reject empty HMAC keys outright in <code>HMACAlgorithm.prepare_key</code> with <code>InvalidKeyError</code> instead of accepting them with only a warning. Defends against the <code>os.getenv("JWT_SECRET", "")</code> footgun. <em>Thanks to <a href="https://github.com/SnailSploit"><code>@SnailSploit</code></a> and <a href="https://github.com/spartan8806"><code>@spartan8806</code></a> for the reports.</em></li> <li>Forward per-call <code>options</code> (including <code>enforce_minimum_key_length</code>) from <code>PyJWT.decode</code> through to <code>PyJWS._verify_signature</code>. The option was previously silently dropped between the two layers, so it only took effect when set on the <code>PyJWT</code> instance. <em>Thanks to <a href="https://github.com/WLUB"><code>@WLUB</code></a> for the report.</em></li> <li><strong>RFC 7797 §3 compliance for <code>b64=false</code>:</strong> the encoder now auto-adds <code>"b64"</code> to <code>crit</code>, and the decoder rejects tokens that set <code>b64=false</code> without listing it in <code>crit</code>. <em>Thanks to <a href="https://github.com/MachineLearning-Nerd"><code>@MachineLearning-Nerd</code></a> for the report.</em></li> </ul> <h2>Changed</h2> <ul> <li>Migrate the <code>dev</code>, <code>docs</code>, and <code>tests</code> package extras to dependency groups, by <a href="https://github.com/kurtmckee"><code>@kurtmckee</code></a> in <a href="https://redirect.github.com/jpadilla/pyjwt/pull/1152">#1152</a>.</li> </ul> <h2>Upgrade notes</h2> <p>Most fixes are invisible to correctly-configured callers. A few behavioral changes you may encounter:</p> <ul> <li><strong>Empty HMAC keys now raise.</strong> If your app passed <code>""</code> or <code>b""</code> as a secret (often via a missing env var, e.g. <code>os.getenv("JWT_SECRET", "")</code>), <code>encode</code>/<code>decode</code> will now raise <code>InvalidKeyError</code>. This is the intended behavior — fix the configuration.</li> <li><strong><code>PyJWK</code> decoding now requires the token's <code>alg</code> to match the JWK's algorithm.</strong> Previously a mismatch was silently honored if the header <code>alg</code> appeared in the allow-list. Tokens that relied on this mismatch will now fail with <code>InvalidAlgorithmError</code>.</li> <li><strong><code>PyJWKClient</code> now rejects non-HTTP(S) URIs at construction time.</strong> Tests or dev environments that fetched JWKS from <code>file://</code> URIs need to switch to a local HTTP server or load the JWKS by other means (e.g. construct <code>PyJWKSet.from_dict(...)</code> directly).</li> <li><strong><code>b64=false</code> tokens are now strictly RFC 7515 / 7797 compliant.</strong> Tokens with a non-empty compact-form payload segment, or that omit <code>"b64"</code> from <code>crit</code>, will be rejected. PyJWT-produced tokens always satisfy both invariants, so round-trips through PyJWT are unaffected.</li> <li><strong><code>enforce_minimum_key_length</code> set per-call now takes effect.</strong> Callers who passed <code>options={"enforce_minimum_key_length": True}</code> to <code>jwt.decode()</code> previously got no enforcement; they will now get <code>InvalidKeyError</code> on undersized keys, as documented.</li> </ul> <p><strong>Full changelog:</strong> <a href="https://github.com/jpadilla/pyjwt/compare/2.12.1...2.13.0">https://github.com/jpadilla/pyjwt/compare/2.12.1...2.13.0</a></p> <h2>2.12.1</h2> <h2>What's Changed</h2> <ul> <li>Add typing_extensions dependency for Python < 3.11 by <a href="https://github.com/jpadilla"><code>@jpadilla</code></a> in <a href="https://redirect.github.com/jpadilla/pyjwt/pull/1151">jpadilla/pyjwt#1151</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/jpadilla/pyjwt/compare/2.12.0...2.12.1">https://github.com/jpadilla/pyjwt/compare/2.12.0...2.12.1</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/jpadilla/pyjwt/blob/master/CHANGELOG.rst">pyjwt's changelog</a>.</em></p> <blockquote> <h2><code>v2.13.0 <https://github.com/jpadilla/pyjwt/compare/2.12.1...2.13.0></code>__</h2> <p>Security</p> <pre><code> - Reject JWK JSON documents passed as raw HMAC secrets in ``HMACAlgorithm.prepare_key`` to close an algorithm-confusion gap that the existing PEM/SSH guard did not cover. Reported by @aradona91 in `GHSA-xgmm-8j9v-c9wx <https://github.com/jpadilla/pyjwt/security/advisories/GHSA-xgmm-8j9v-c9wx>`__. - Bind the JWT header ``alg`` to ``PyJWK.algorithm_name`` during verification so the caller's ``algorithms=[...]`` allow-list cannot be bypassed when decoding with a ``PyJWK`` / ``PyJWKClient`` key. Reported by @sushi-gif in `GHSA-jq35-7prp-9v3f <https://github.com/jpadilla/pyjwt/security/advisories/GHSA-jq35-7prp-9v3f>`__. - Reject non-``http(s)`` URI schemes in ``PyJWKClient`` so attacker- influenced URIs cannot read local files or reach unintended schemes via urllib's default ``file://`` / ``ftp://`` / ``data:`` handlers. Reported by @KEIJOT in `GHSA-993g-76c3-p5m4 <https://github.com/jpadilla/pyjwt/security/advisories/GHSA-993g-76c3-p5m4>`__. - Preserve the cached JWK Set on fetch errors in ``PyJWKClient.fetch_data``. The previous ``finally``-block ``put(None)`` pattern cleared the cache on any transient outage, turning one bad JWKS request into application- wide auth failure. Reported by @eddieran in `GHSA-fhv5-28vv-h8m8 <https://github.com/jpadilla/pyjwt/security/advisories/GHSA-fhv5-28vv-h8m8>`__. - Skip the unconditional base64 decode of the compact-form payload segment when ``b64=false`` is set in the protected header, and require that segment to be empty (RFC 7515 Appendix F detached form). Closes an unauthenticated DoS amplifier. Reported by @thesmartshadow in `GHSA-w7vc-732c-9m39 <https://github.com/jpadilla/pyjwt/security/advisories/GHSA-w7vc-732c-9m39>`__. <p>Fixed</p> <pre><code> - Reject empty HMAC keys outright in ``HMACAlgorithm.prepare_key`` with ``InvalidKeyError`` instead of accepting them with only a warning. Thanks to @SnailSploit and @spartan8806 for independently flagging the footgun. - Forward per-call ``options`` (including ``enforce_minimum_key_length``) from ``PyJWT.decode`` through to ``PyJWS._verify_signature`` so the option actually takes effect when set at the call site rather than only on the ``PyJWT`` instance. Thanks to @WLUB for the report. - RFC 7797 §3 compliance for ``b64=false``: the encoder now auto-adds ``&quot;b64&quot;`` to the ``crit`` header parameter, and the decoder rejects tokens that set ``b64=false`` without listing it in ``crit``. Thanks to @MachineLearning-Nerd for the report. Changed </code></pre> <ul> <li>Migrate the <code>dev</code>, <code>docs</code>, and <code>tests</code> package extras to dependency groups by <a href="https://github.com/kurtmckee"><code>@kurtmckee</code></a> in <code>[#1152](jpadilla/pyjwt#1152) &lt;https://github.com/jpadilla/pyjwt/pull/1152&gt;</code>__</li> </ul> <p><code>v2.12.1 &lt;https://github.com/jpadilla/pyjwt/compare/2.12.0...2.12.1&gt;</code>__ </tr></table> </code></pre></p> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/jpadilla/pyjwt/commit/7144e4534c34810f4525dc4578a32addd8212cff"><code>7144e45</code></a> Apply ruff format</li> <li><a href="https://github.com/jpadilla/pyjwt/commit/d2f4bec4963897c0ef96ef64a875894f2c8542ab"><code>d2f4bec</code></a> Restore <code>cast()</code> calls with cross-version <code>type: ignore</code> for <code>prepare_key</code></li> <li><a href="https://github.com/jpadilla/pyjwt/commit/22f478cebddd8294259c30f037ecb92b0b348774"><code>22f478c</code></a> Remove redundant casts in <code>RSAAlgorithm.prepare_key</code> and `ECAlgorithm.prepare...</li> <li><a href="https://github.com/jpadilla/pyjwt/commit/95791b1759b8aa4f2203575d344d5c78564cdc81"><code>95791b1</code></a> Bundle security fixes and hardening into 2.13.0</li> <li><a href="https://github.com/jpadilla/pyjwt/commit/dcc27a9d3182a2349c30b160758785c6ce7a6508"><code>dcc27a9</code></a> [pre-commit.ci] pre-commit autoupdate (<a href="https://redirect.github.com/jpadilla/pyjwt/issues/1155">#1155</a>)</li> <li><a href="https://github.com/jpadilla/pyjwt/commit/9d08a9a1896845ed8eaf88e6f6ac61e5800c3e7a"><code>9d08a9a</code></a> [pre-commit.ci] pre-commit autoupdate (<a href="https://redirect.github.com/jpadilla/pyjwt/issues/1146">#1146</a>)</li> <li><a href="https://github.com/jpadilla/pyjwt/commit/b87c10014d4109f0214fea188d00faaaf8a80e64"><code>b87c100</code></a> Bump codecov/codecov-action from 5 to 6 (<a href="https://redirect.github.com/jpadilla/pyjwt/issues/1154">#1154</a>)</li> <li><a href="https://github.com/jpadilla/pyjwt/commit/40e3147eb5f790d8d041772e5fc00728a176c812"><code>40e3147</code></a> Migrate development extras to dependency groups (<a href="https://redirect.github.com/jpadilla/pyjwt/issues/1152">#1152</a>)</li> <li><a href="https://github.com/jpadilla/pyjwt/commit/a4e1a3d1218b01c5806420b8f16d9308ac4adc30"><code>a4e1a3d</code></a> Add typing_extensions dependency for Python < 3.11 (<a href="https://redirect.github.com/jpadilla/pyjwt/issues/1151">#1151</a>)</li> <li>See full diff in <a href="https://github.com/jpadilla/pyjwt/compare/2.12.0...2.13.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? - Closes #22932 ## Rationale for this change `depcheck` stopped working because an underlying dependency released a non-semver compatible change (`time 0.3.48`, not going to link to the issue there because people are already spamming it). Because there was no lockfile, every run pulled the most recent versions for all dependencies, which don't currently compile. ## What changes are included in this PR? 1. Add lockfile to `depcheck` 2. Make sure to run `depcheck` with `--lock` in CI ## Are these changes tested? Tested locally ## Are there any user-facing changes? None Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
## Which issue does this PR close? - Closes #19057. ## Rationale for this change A binary gcd and lcm UDF in the datafusion-functions crate supports only Int64, but not Decimals. Adding missing support for decimals. ## What changes are included in this PR? 1. Updated gcd and lcm functions to add decimal support. The integer path is more performant and stays intact. For decimals, the Euclidean algorithm is used for GCD 2. Added coercion rules: casting to decimals if any argument is decimal; otherwise, stay with ints as before 3. Common functionality extracted to `common.rs` to avoid inter-UDF dependency 4. In order to use `calculate_binary_math` for Decimals, updated it to accept a target type instead of raw `Decimal128Type::DATA_TYPE` - it causes scaling issues for these UDFs, see #19621 A bit more on (4). The driving force is this failing example: ```sql query R select gcd(2::decimal(38, 0), 3::decimal(38, 0)); ---- 1 ``` Previously in #19874, I suggested a more complicated solution to extend `calculate_binary_math`. However, it only affected gcd/lcm and could be considered overkill. This PR extends these functions with an extra parameter `cast_target` for `calculate_binary_decimal_math` to perform a proper cast to the actual type used, rather than to the default `Decimal128Type::DATA_TYPE` - it is much lighter. ## Are these changes tested? - Added unit test for UDFs with decimals for array and scalar paths - Added unit tests for the gcd/lcm math itself - Added new SLT tests for decimals ## Are there any user-facing changes? No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )