Skip to content

fix: SQLite 大批次查询与写入回滚 / SQLite chunked queries and rollback - #1217

Open
mfurkanakinci wants to merge 7 commits into
sansan0:masterfrom
mfurkanakinci:fix/sqlite-reliability
Open

mfurkanakinci wants to merge 7 commits into
sansan0:masterfrom
mfurkanakinci:fix/sqlite-reliability

Conversation

@mfurkanakinci

Copy link
Copy Markdown

中文

大批次写入时,SQLite 的 IN 查询会超过 999 个变量的上限并导致整批失败。写入失败时也没有回滚,错误被宽泛地吞掉。

  • 新增 trendradar/utils/sqlite.py,把 IN ({placeholders}) 按块执行(每块 500 个值)
  • 热榜、RSS 和 MCP 解析里的大 IN 查询改走这条路径
  • 写入失败时回滚连接,并以 StorageError 抛出不可忽略的存储错误
  • 不引入 PostgreSQL 依赖

tests/test_sqlite_chunked_in.py 覆盖超过旧上限的 IN 查询,tests/test_sqlite_mixin.py 覆盖写入、合并、排名历史和 AI 标签生命周期。

English

Large batches can exceed SQLite's 999-variable limit on IN queries and crash the write. Failed writes were also left unrolled back, and the error was swallowed.

  • Add trendradar/utils/sqlite.py and run IN ({placeholders}) in chunks of 500 values
  • Use that path for the large IN queries in hotlist storage, RSS storage, and the MCP parser
  • Roll back the connection when a write fails, and raise StorageError for storage failures that must not be ignored
  • No PostgreSQL dependency

tests/test_sqlite_chunked_in.py covers an IN query past the old limit. tests/test_sqlite_mixin.py covers writes, merges, rank history, and the AI-tag lifecycle.

- IN 查询按块执行,避免超过 SQLite 999 变量上限
- Chunk IN queries so batches stay under SQLite's 999-variable limit
- 写入失败时回滚,并以 StorageError 抛出不可忽略的存储错误
- Roll back failed writes and raise StorageError for storage failures that must not be ignored
- IN 查询按块执行,避免超过 SQLite 999 变量上限
- Chunk IN queries so batches stay under SQLite's 999-variable limit
- 写入失败时回滚,并以 StorageError 抛出不可忽略的存储错误
- Roll back failed writes and raise StorageError for storage failures that must not be ignored
- IN 查询按块执行,避免超过 SQLite 999 变量上限
- Chunk IN queries so batches stay under SQLite's 999-variable limit
- 写入失败时回滚,并以 StorageError 抛出不可忽略的存储错误
- Roll back failed writes and raise StorageError for storage failures that must not be ignored
- IN 查询按块执行,避免超过 SQLite 999 变量上限
- Chunk IN queries so batches stay under SQLite's 999-variable limit
- 写入失败时回滚,并以 StorageError 抛出不可忽略的存储错误
- Roll back failed writes and raise StorageError for storage failures that must not be ignored
- IN 查询按块执行,避免超过 SQLite 999 变量上限
- Chunk IN queries so batches stay under SQLite's 999-variable limit
- 写入失败时回滚,并以 StorageError 抛出不可忽略的存储错误
- Roll back failed writes and raise StorageError for storage failures that must not be ignored
- IN 查询按块执行,避免超过 SQLite 999 变量上限
- Chunk IN queries so batches stay under SQLite's 999-variable limit
- 写入失败时回滚,并以 StorageError 抛出不可忽略的存储错误
- Roll back failed writes and raise StorageError for storage failures that must not be ignored
- IN 查询按块执行,避免超过 SQLite 999 变量上限
- Chunk IN queries so batches stay under SQLite's 999-variable limit
- 写入失败时回滚,并以 StorageError 抛出不可忽略的存储错误
- Roll back failed writes and raise StorageError for storage failures that must not be ignored

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant