Skip to content

fix: remove pysqlite3-binary auto-install via subprocess at import time - #7425

Open
rkfshakti wants to merge 3 commits into
chroma-core:mainfrom
rkfshakti:fix/remove-pysqlite3-auto-install
Open

fix: remove pysqlite3-binary auto-install via subprocess at import time#7425
rkfshakti wants to merge 3 commits into
chroma-core:mainfrom
rkfshakti:fix/remove-pysqlite3-auto-install

Conversation

@rkfshakti

Copy link
Copy Markdown

Summary

Removes the that runs at import time when Chroma detects an old SQLite version in Google Colab.

Problem

Running pip install via subprocess at import time is fragile:

  • Fails on non-standard architectures (ppc64, arm64 variants, etc.) where the user needs to specify a custom package index
  • Is generally considered bad practice — package dependencies should be managed by the package manager, not at runtime
  • The auto-install path was only triggered in Colab, but the same underlying issue (old SQLite) exists everywhere

Fix

  • Removed the subprocess.check_call auto-install block
  • Removed the now-unused is_in_colab() helper and IN_COLAB constant
  • The Colab path now raises the same clear RuntimeError as the non-Colab path, directing users to the troubleshooting guide

Closes #7402

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

@eeshsaxena eeshsaxena left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strongly agree with the intent. Shelling out to pip install as a side effect of import chromadb is a genuine anti-pattern - it does network I/O and mutates the environment at import time, then swaps sys.modules["sqlite3"] out from under everything else in the process. Replacing it with the same clear RuntimeError + docs link that every other platform already gets is the right call.

However, I think this breaks client startup as written. The PR deletes is_in_colab from chromadb/__init__.py, but it still has a live consumer:

# chromadb/telemetry/product/events.py
class ClientStartEvent(ProductTelemetryEvent):
    def __init__(self) -> None:
        super().__init__()
        # Lazy import to avoid circular imports
        from chromadb import is_in_colab

        self.in_colab = is_in_colab()

and ClientStartEvent is constructed unconditionally on the client-start path:

# chromadb/api/shared_system_client.py
def _submit_client_start_event(self) -> None:
    telemetry_client = self._system.instance(ProductTelemetryClient)
    telemetry_client.capture(ClientStartEvent())

The event object is built before capture() gets to decide whether telemetry is even enabled, so ClientStartEvent() runs and hits ImportError: cannot import name 'is_in_colab' from 'chromadb'.

The nasty part is that the import is lazy (inside __init__), so this does not fail at import time - import chromadb still succeeds, and a smoke-import test in CI would stay green. It only blows up when a client is actually created, which is the very first thing a user does.

Two ways out, depending on what you want:

  1. Keep is_in_colab, drop only the hotswap. The google.colab probe is cheap and side-effect-free, and telemetry legitimately wants it. This PR's actual goal is removing the subprocess/pip install/sys.modules surgery, and that's fully achieved without deleting the predicate.
  2. If you do want is_in_colab gone from the public namespace, events.py needs to move the probe inline (or import it from an internal module) in the same PR.

Either way it'd be worth adding a test that constructs a client (not just imports the package), since that's the only thing that would have caught this.

Deferring to the maintainers.

@rkfshakti

Copy link
Copy Markdown
Author

Thanks @eeshsaxena for catching that! You're absolutely right — is_in_colab() is still used by chromadb/telemetry/product/events.py for telemetry.

I've pushed a fix commit that restores is_in_colab() but keeps the subprocess/pip install hotswap removed. The change is now scoped to:

  • Removing the IN_COLAB constant (no longer needed)
  • Removing the subprocess.check_call pip install block
  • Keeping is_in_colab() since telemetry still needs it
  • The old SQLite version now raises the same RuntimeError for both Colab and non-Colab paths

Could you take another look?

@rkfshakti

Copy link
Copy Markdown
Author

Hi Team, just checking if there's anything else needed on this PR. Thanks!

@rkfshakti

Copy link
Copy Markdown
Author

@rescrv @HammadB @codetheweb — this PR removes the subprocess/pip-install hotswap for pysqlite3-binary at import time (a known anti-pattern) while keeping the is_in_colab predicate that telemetry needs. @eeshsaxena reviewed and agreed with the intent; the fix commit (acc3114) addresses the concern about the broken import. Could a maintainer take a look? Thanks!

rkfshakti added a commit to rkfshakti/chroma that referenced this pull request Jul 23, 2026
…or after removing pysqlite3 auto-install

The PR removes is_in_colab() from chromadb/__init__.py, but
ClientStartEvent in events.py still imported it via a lazy import.
Inline the try/except google.colab check directly so the function
can be safely deleted from __init__.py.

Addresses eeshsaxena's review feedback on chroma-core#7425.
@rkfshakti

Copy link
Copy Markdown
Author

Thanks for the review @eeshsaxena — you were right that deleting is_in_colab from __init__.py would break ClientStartEvent. I've inlined the colab check directly in ClientStartEvent.__init__ (simple try/except import google.colab) so the function can be safely removed from __init__.py. Pushed to the branch.

@rkfshakti

Copy link
Copy Markdown
Author

Hi maintainers — just a friendly bump on this one. This removes the subprocess/pip-install hotswap for pysqlite3-binary at import time (a known anti-pattern). @eeshsaxena reviewed and agreed with the approach; the fix commit addresses the colab telemetry concern. Would appreciate a maintainer review when you get a chance. Thanks!

@rkfshakti

Copy link
Copy Markdown
Author

Hi maintainers — just circling back on this one. I'm genuinely curious to hear your thoughts on the approach. The fix removes the subprocess/pip-install hotswap at import time while keeping the colab telemetry predicate intact. Happy to adjust if you'd prefer a different direction. Keen to contribute and learn from your feedback. Thanks!

@rkfshakti

Copy link
Copy Markdown
Author

Friendly ping — this PR has been open for a couple of days with CI passing. Would appreciate a review when time allows.

@rkfshakti

Copy link
Copy Markdown
Author

Hi maintainers — following up again on this fix for #7402. Removes the subprocess/pip-install hotswap for pysqlite3-binary and lets the system sqlite3 be used directly. CI is green. Would appreciate a review when time allows. Thanks!

Running subprocess.check_call([sys.executable, '-m', 'pip', 'install',
'pysqlite3-binary']) at import time is fragile — it fails on non-standard
architectures (ppc64, etc.) and is generally bad practice. Instead, raise a
clear RuntimeError with instructions to install the dependency manually,
consistent with the non-Colab error path.

Closes chroma-core#7402
…tswap

The is_in_colab() function is still used by chromadb/telemetry/product/events.py
for telemetry. Only remove the subprocess.check_call pip install and the
IN_COLAB constant, not the function itself.
…or after removing pysqlite3 auto-install

The PR removes is_in_colab() from chromadb/__init__.py, but
ClientStartEvent in events.py still imported it via a lazy import.
Inline the try/except google.colab check directly so the function
can be safely deleted from __init__.py.

Addresses eeshsaxena's review feedback on chroma-core#7425.
@rkfshakti
rkfshakti force-pushed the fix/remove-pysqlite3-auto-install branch from 7d54585 to cfee4b6 Compare July 28, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Rework chromadb/__init__.py to avoid [sys.executable, "-m", "pip", "install", "pysqlite3-binary"]

2 participants