Sieve

A bounded, source-pluggable promotion tracker with natural-language preferences, BM25 filtering, and poisoning-resistant Gemini presentation.

How it works • Quick start • Configuration • CLI • Rollout

---

Warning

Sieve is beta software and releases may break existing deployments. Pin a known-working release, and do not update a healthy bot without reviewing CHANGELOG.md. Back up SQLite state before every update. Downgrades require restoring the pre-migration backup. Sieve never requires or stores phone numbers.

Overview

Sieve watches enabled promotion sources, evaluates each promotion independently against every active member's private preference profile, and forwards the survivors to that member's private Telegram chat.

To track something, just send the bot a message. You can describe a product by name, set a maximum price, require a model or attribute, add aliases, or exclude entire categories—no preference syntax or configuration-file edit is required:

Track Sony WH-1000XM5 headphones under $300
I want OLED monitors with at least 144 Hz
Treat “graphics card” and “GPU” as the same thing
Stop showing me perfume deals

The point is cost and noise control. Most promotions never reach Gemini: fixed spam checks and prioritized, declarative hard rules reject unwanted categories; deduplication kills reposts; a rolling Okapi BM25 score kills weak lexical matches; and an alias-aware structured-interest gate rejects broad candidates. Only concrete interest matches are sent to the configured evaluator. Accepted promotions are rendered deterministically from normalized source data.

It's designed to sit on a Raspberry Pi 4B (2 GB) indefinitely: one process, a bounded queue, a 256 MB container cap, a 220 MB in-app tripwire that restarts before the OOM killer does, and a SQLite WAL database that prunes itself.

Important

Promotion delivery is always live and audible. BM25 keeps its own independent off/shadow/live auto-forward calibration control; shadow there records candidates but does not create test deliveries.

Sieve uses immutable UUIDv4 member identities and never uses Telegram names or phone numbers as identifiers. The administrator invites members with /invite; members redeem the single-use token in a private /start <token>, see only their own state, and can retrieve their UUID with /account.

---

How it works

flowchart TD
    TG[Telegram groups<br/><i>Telethon user session</i>] --> Q
    PL[Pelando /recentes<br/><i>JSON-LD polling</i>] --> Q
    Q[Bounded asyncio queue<br/>capacity 256]

    Q --> PS[Atomic preference snapshot]
    PS --> HF{Spam / exclusion / hard rule}
    HF -->|fixed spam, explicit exclusion,<br/>prioritized deny rule| X1[discard]
    HF --> DD{Seen before?}
    DD -->|content hash / native id| X2[discard]

    DD --> PC{Constraint violation?}
    PC -->|reliable price / attribute mismatch| X5[discard]
    PC --> EX{Exceptional?}
    EX -->|proven price error, historical low,<br/>&gt;50% discount or Pelando temp &ge; 300| DEL[Deliver]
    EX -->|required attributes unknown| LLM
    EX -->|no| READY{BM25 ready?}
    READY -->|cold corpus or alias rebuild| LLM
    READY -->|yes| BM{Score band}
    BM -->|score &lt; 2| AUDIT{5% audit sample?}
    AUDIT -->|no| X3[discard]
    AUDIT -->|yes| ALLM[Gemini audit]
    ALLM -->|record label, never deliver| X3
    BM -->|2 &le; score &lt; 7| INT{Concrete interest match?}
    BM -->|score &ge; 7| GATE{Deterministic gates?}
    GATE -->|no| INT
    GATE -->|yes + BM25 shadow| CAND[Mark candidate]
    CAND --> INT
    GATE -->|yes + validated live mode| DEL
    INT -->|no| X4[discard: interest candidate miss]
    INT -->|yes| LLM[Gemini structured decision]

    LLM -->|forward| DEL
    LLM -->|discard| X4
    LLM -->|transient error| RQ[Retry queue<br/>100 items / 1h TTL]
    RQ --> LLM

    DEL --> OBOX[Persistent per-user outbox]
    OBOX --> CARD[Deterministic normalized card<br/>plain text + UTF-16 entities]
    OBOX --> MEDIA[Media resolver<br/><i>never sent to Gemini</i>]
    MEDIA --> CARD
    CARD --> OUT[sendPhoto or sendMessage<br/><i>audible, localized delivery</i>]

Loading

Stage notes

Stage	Behaviour
Spam / exclusions / hard rules	Fixed spam checks run first, followed by live explicit exclusions. Ordered, token-aware allow/deny rules use the first matching priority, so narrow exceptions can precede broader category denials.
Deduplication	Global native replay protection plus per-user, cross-source URL/product fingerprints for 10 minutes. Equal or higher-priced reposts are suppressed; a lower price becomes the new minimum to beat.
Constraints	Reliably matched interest price violations and excluded attributes are discarded before any exceptional bypass. Missing required attributes remain undecided.
Exceptional bypass	Normally skips BM25 and the LLM. If a promotion may match an interest but its required attributes cannot be proven, it skips BM25 and goes to Gemini instead.
BM25	Weighted Okapi BM25 (k1=1.2, b=0.75) with lower 2.0 and experimental upper 7.0 routing thresholds. Importance 0–100 maps to 0.5×–1.5×. BM25 fails open to Gemini during cold start and alias rebuilds.
Interest admission	Gemini is called only after an alias-aware structured-interest match. At most five matching interests, applicable context, and aliases are sent in a 2,500-character context; unrelated profile content and enforced exclusions/rules are omitted.
Gemini evaluation	Optional structured decision before acceptance, minimal thinking, serialized requests, persisted Pacific-day/stage budgets, and at most two actual attempts for transport/5xx failures. A 429 is never retried immediately.
Presentation	Deterministic by default from normalized promotion data, including enriched Pelando price/comment data. Telegram entities keep untrusted markup plain; the original safe fallback remains available.
Delivery	Per-user persistent outbox with restart recovery, still-image cards, bounded retries, visible permanent failures, and safe original-content fallback. Delivery is at least once, so an ambiguous timeout may rarely duplicate a notification rather than lose it.

Gemini promotion evaluation is optional and presentation is deterministic by default. When pipeline.gemini_evaluation_enabled is false, natural-language preference messages can still use Gemini, but promotion decisions remain deterministic. Only proven exceptional deals and candidates above the configured BM25 auto-forward threshold that pass every gate in live mode can be delivered. Intermediate or uncertain candidates, cold corpus items, alias-rebuild items, audits, and pending evaluation retries are discarded without a Gemini request.

Design properties

• Source-neutral core. A Promotion dataclass plus PromotionSource, PipelineStage, LLMEvaluator, PromotionSink and StateStore protocols. Every implementation is resolved from a module:factory string in YAML — a new source drops in without touching the pipeline.
• Bounded everywhere. Input queue 256, preference queue/outbox 20, retry queue 100 items / 1 hour TTL, corpus 10,000 docs, preference state 500 entries / 128 KB, incremental pruning.
• Live revisioned preferences. YAML seeds revision zero once. SQLite is authoritative after that, every mutation is audited, and each promotion keeps one immutable snapshot for its full run.
• Source-agnostic still images. Telegram photo/image-document identifiers and Pelando image URLs are carried as serializable references, resolved only after acceptance, validated and stored under /state/media, and deleted after every referencing delivery succeeds or permanently fails.
• Locked-down container. Non-root user, read-only root filesystem, all capabilities dropped, no-new-privileges, 64 PIDs, 16 MB tmpfs, 10 MB × 3 log rotation.
• Observable. JSON logs to stdout; Telegram alerts for transport and other source failures, database errors, LLM outages and memory pressure. Pelando schema drift stays visible in structured logs and the persisted health snapshot without sending Telegram alerts.

Deterministic card pipeline

Accepted promotions are rendered without Gemini. Sieve uses the normalized title and price, retains source text such as Pelando's publication comment, and creates buttons only from captured HTTP(S) URLs. Untrusted angle-bracket markup stays literal because Telegram formatting is expressed through explicit UTF-16 entities rather than a parsing mode.

Still images are resolved only after acceptance, validated, bounded to 10 MB, reference-counted under /state/media, and removed after every delivery finishes. If media resolution or a definite Telegram photo send fails, the same card is delivered as safe text. Oversized captions become a minimal caption plus bounded plain-text follow-ups.

Optional Gemini presentation remains available behind gemini.presentation_enabled, but is off in the production configuration. Each enabled stage performs one validated logical request and falls back immediately on invalid output.

The math

For query term t and promotion document d, Sieve uses weighted Okapi BM25:

score(q, d) = Σ IDF(t) · [ f(t,d) · (k₁ + 1) ]
                         ───────────────────────── · w(t)
                         f(t,d) + k₁ · (1 - b + b · |d| / avgdl)

with inverse document frequency:

IDF(t) = ln(1 + (N - df(t) + 0.5) / (df(t) + 0.5))

f(t,d) is term frequency, |d| is promotion length, avgdl is average corpus document length, and df(t) is the number of corpus documents containing the term. The defaults are k₁=1.2 and b=0.75. Rare, specific terms consequently carry more evidence than common deal vocabulary.

Interest importance I is mapped linearly from 0–100 to a term multiplier:

w(I) = 0.5 + I / 100,     0 ≤ I ≤ 100

Thus 0 → 0.5×, 50 → 1.0×, and 100 → 1.5×. Aliases expand indexed terms without changing the stored source text. Context informs Gemini but does not independently add BM25 relevance.

Why the thresholds are 2 and 7

BM25 is neither a percentage nor a probability: its scale changes with N, df(t), document length, aliases, and weights. For an average-length promotion containing a term once, the saturation fraction is exactly 1:

f = 1 and |d| = avgdl  →  f·(k₁+1) / (f+k₁) = 1
term contribution ≈ IDF(t) · w(t)

At the first stable corpus size, N = 500, representative values are:

Corpus frequency	IDF(t)	Contribution at weight 1.0
df = 1	5.81	≈ 5.81
df = 10	3.87	≈ 3.87
df = 50	2.29	≈ 2.29

The 2.0 cutoff removes weak lexical matches; 7.0 creates a strong-score observation band. It is not proof of relevance: one maximally weighted rare term can contribute 5.81 × 1.5 ≈ 8.72 by itself. A candidate must therefore also match a literal term from a structured interest, prove all required constraints, and not look like an accessory when the interest targets the main product. Aliases affect BM25 but cannot satisfy that literal gate alone.

score < 2.0       → discard; audit 5% with Gemini without ever delivering
2.0 ≤ score < 7.0 → Gemini decides
score ≥ 7.0       → apply gates; shadow records a candidate and still lets Gemini decide
BM25 unavailable → Gemini decides

7.0 is an experimental starting point, not a universal mathematical optimum. Live mode should only be considered after at least 300 eligible shadow candidates with no confirmed false forward. With zero failures in n observations, the rule of three puts the approximate 95% upper risk bound at 3/n, or about 1% for n = 300. A cold corpus or alias rebuild returns all decisions to Gemini.

Price and attribute constraints use three-valued logic: satisfied, violated, or unknown. A proven violation is discarded before exceptional handling; a proven match keeps the usual bypass; an unknown required attribute on a potentially relevant exceptional deal goes to Gemini. This avoids treating missing information as evidence either way.

---

Quick start

Requirements

• Docker and Docker Compose (or Python 3.12+ for local runs)
• For Telegram group ingestion, a Telegram user account with API credentials
• A separate Telegram bot that already has an open private conversation with your account — this is what delivers
• A Gemini API key for evaluation and natural-language preference commands

1. Configure secrets

git clone https://github.com/koobzaar/Sieve.git
cd Sieve
cp .env.example .env

Fill in GEMINI_API_KEY, TELEGRAM_BOT_TOKEN and TELEGRAM_ADMIN_USER_ID in .env. The example enables Pelando and the private preference bot. Telegram group ingestion is optional and additionally needs TELEGRAM_API_ID, TELEGRAM_API_HASH and the one-time login below:

TELEGRAM_API_ID=
TELEGRAM_API_HASH=
TELEGRAM_BOT_TOKEN=
TELEGRAM_ADMIN_USER_ID=
GEMINI_API_KEY=

Warning

Keep .env, Telegram session files, and SQLite state private. The tracked config/config.yaml is complete and contains no secrets; secrets are resolved from environment variable names.

Common settings are in .env: SIEVE_GEMINI_MODEL, SIEVE_GEMINI_EVALUATION_ENABLED, SIEVE_BM25_THRESHOLD, SIEVE_BM25_AUTO_FORWARD_THRESHOLD, SIEVE_BM25_AUTO_FORWARD_MODE, SIEVE_PELANDO_ENABLED, SIEVE_TELEGRAM_ENABLED, SIEVE_PREFERENCES_ENABLED and SIEVE_LOG_LEVEL. Advanced limits, initial aliases/rules and optional initial Telegram chat_ids stay in config/config.yaml. Use the bot to add your product interests after startup. Without environment overrides, the tracked YAML keeps sources and preferences disabled.

2. Authenticate the Telegram user session

Skip this step for Pelando-only setups. Otherwise, build the image and run the interactive login once. Telethon writes the session file into the persistent /state volume.

docker compose build sieve
docker compose run --rm sieve \
  --config /app/config/config.yaml auth-telegram --source telegram-principal

Scan the terminal QR code from Telegram's Settings → Devices → Link Desktop Device. Enter only your 2FA password if Telegram requests it. Then set SIEVE_TELEGRAM_ENABLED=true in .env. Its chat_ids are only a one-time bootstrap for a fresh SQLite database; they may be empty when groups will be selected from the administrator's Groups menu.

3. Validate and run

docker compose config --quiet
docker compose build sieve
docker compose run --rm sieve validate-config --runtime
docker compose up -d --wait
docker compose logs -f sieve

Health is checked automatically every 60s via the health subcommand — it verifies PRAGMA quick_check and a runtime heartbeat newer than 180 seconds. JSON logs report component, exception type, provider status/code, consecutive failures, retry timing, and alert decisions without logging message bodies, contact data, Telegram tokens, usernames, or display names.

Send /start to the bot, then describe the products you want to track. After changing .env or YAML, run docker compose up -d --force-recreate --wait; no image rebuild is needed for settings. Preferences already saved in SQLite continue to be managed through the bot.

For debugging, use docker compose ps, docker compose exec sieve health and docker compose logs --since 10m sieve. Set SIEVE_LOG_LEVEL=DEBUG in .env and recreate the container for more detail. validate-config checks YAML without requiring credentials; validate-config --runtime also checks startup requirements without making API requests. Shutdown allows 30 seconds to drain promotions before cancellation; durable outbox/retry entries remain available after restart. A shutdown_queue_timeout log identifies an incomplete drain.

Running without Docker

python -m venv .venv
.venv/bin/pip install -e .
.venv/bin/sieve --config config/config.yaml run

Export the variables from .env into your shell first: Python does not load dotenv files. Point state.path, state.media_path and session_path at writable local directories instead of /state. On Windows use .venv\Scripts\. SIEVE_CONFIG and SIEVE_LOG_LEVEL set CLI defaults; explicit --config and --log-level flags override them.

---

Supported promotion sources

All sources are disabled in the tracked configuration. Enable only the adapters you have configured.

Source	Coverage	Default	Required settings
Telegram groups/channels	Any groups or channels accessible to your Telegram user account	Disabled	Telethon API credentials and a persisted user session; optional bootstrap chat_ids
Pelando /recentes	Brazil-focused deal website	Disabled	No account; optional polling interval, timeout, and user agent

Enable or disable each source independently in config/config.yaml:

sources:
  - name: telegram-principal
    enabled: true
    settings:
      chat_ids: [-1001234567890]

  - name: pelando
    enabled: false

At least one source must be enabled before run. To add another website, implement the PromotionSource protocol and a factory that accepts the configured settings, name, shared HTTP client, and health reporter; then reference it with a module:factory path under sources. The pipeline does not need source-specific changes.

For each Telegram source, YAML chat_ids are imported as enabled groups only the first time that source is seen in a fresh database. SQLite is authoritative afterward: disabling a group in the administrator-only Groups menu survives restarts and later YAML changes. The Telethon account refreshes its visible groups/channels into that menu; newly discovered dialogs remain disabled until selected. Zero enabled groups is valid—the account stays connected for discovery but emits no promotions.

---

Configuration

The complete advanced settings live in config/config.yaml. Configuration inheritance is not supported. To use an untracked complete local copy, copy it to config/config.local.yaml and set SIEVE_CONFIG_FILE=./config/config.local.yaml in .env. Compose mounts that file read-only at /app/config/config.yaml.

Whole YAML values support ${VARIABLE:-default}. Defaults retain their type: numeric and boolean defaults accept YAML values of those types; string defaults remain strings (including off). ${VARIABLE} without a default requires a nonempty environment variable. Compose injects .env using env_file; interpolation inside the application happens after YAML parsing. Secrets remain environment variable names in *_env settings and are never stored in YAML.

Upgrading from the layered configuration

Back up your database and preserve the existing configuration before replacing layered settings:

mv config/config.local.yaml config/config.yaml
# Or copy it if you want to retain the old file:
# cp config/config.local.yaml config/config.yaml

Old overrides may contain an extends line and omit values that came from a base file. Remove extends and compare the migrated file with config/config.yaml. Removed runtime/source modes, owner/chat keys, or sink chat destinations cause a clear validation error and must be deleted.

Key sections

Block	Notable keys
runtime	queue_capacity, memory_limit_mb, failure_alert_threshold, llm_outage_alert_seconds, shutdown_timeout_seconds
state	path, media_path (/state/media), retention/corpus limits, and retry bounds
pipeline	gemini_evaluation_enabled, BM25 thresholds/mode/audit parameters, profile, aliases and rules
evaluator	Decision-evaluator factory and its decision-specific max_output_tokens override
gemini	Shared key/model/provider settings; presentation toggle; Pacific daily, evaluation, preference and RPM caps; ledger retention; optional presentation stage bounds
preferences	enablement, admin Telegram user-ID env var, max_users, polling/queue limits, confirmation TTL, state caps, parser overrides
sink	factory, token env var, API URL, timeout
sources	list of {name, factory, enabled, settings}

The profile is imported as a baseline note and contributes BM25 terms. Gemini evaluation is admitted only by live structured interests and receives only the matching subset, not the complete profile.

Aliases are bidirectional — placa_video: ["gpu", "graphics card", "geforce", "radeon"] means a listing mentioning any of those terms matches the profile's mention of any other.

Hard rules are evaluated in ascending numeric priority after the fixed spam checks. Each rule has a stable id, an allow or deny action, and either any phrases or all groups. Phrase matching is accent-insensitive and token-aware. The first matching rule wins, which makes narrow allows predictable and leaves generic products to broader denials.

Deterministic promotion filtering

To keep Gemini for natural-language preference management but stop using it to judge promotions:

pipeline:
  gemini_evaluation_enabled: false
  bm25_threshold: 2.0
  bm25_auto_forward_threshold: 7.0  # Tune this for your sources and profile
  bm25_auto_forward_mode: live      # Use shadow while validating the threshold

Disabling evaluation changes the fallback from Gemini to discard. BM25 has no universal score scale, so collect replay and shadow evidence and tune bm25_auto_forward_threshold manually before using live. Accepted promotions still use deterministic presentation.

Gemini attempts are serialized and persisted before each HTTP request. The defaults allow 400 attempts per Pacific quota day, including at most 350 evaluations and 25 preference calls, with a five-request rolling minute limit. Daily exhaustion fails locally until the next Pacific midnight; it does not grow the retry backlog.

Live preference commands

Exceptional offers are disabled by default for every user, including existing users without a saved choice. In the bot's Offer settings menu, each user can enable or disable them; explicitly saved choices survive upgrades and restarts. When disabled, hot deals, large discounts and price-error offers must pass normal interest filtering. A user without saved interests receives no promotions unless they explicitly enable exceptional offers.

On the first startup for a database, Sieve imports a nonblank YAML profile losslessly as a baseline note and imports each alias and hard rule as its own revision-zero entry. It never imports YAML again for that database; deleting the preference database is the automatic reseed path. During an upgrade, an untouched revision-zero placeholder shipped by the old templates is removed only when structured interests already exist, in one audited system revision. Adding the first interest removes that placeholder atomically in the same audited revision. User-authored, revision-touched, and no-interest baselines are preserved.

Each interest keeps a search_terms list whose values are alternative match expressions, not words that must all appear together across the list. Keep each alternative short and discriminative (for example, a model, product, or product-plus-brand identity). Matching ignores case, accents, connector words, word order, and letter/number spacing while still requiring every significant token in the selected alternative.

The configured private owner can send natural-language instructions to the delivery bot. The app checks both private chat and sender IDs before Gemini sees a message. On the first message it uses the Telegram profile language as a hint; the selected language is then persisted in SQLite and can be changed with /language. User-facing UI and generated reasons are currently available in English and Brazilian Portuguese. All Telegram copy lives in validated YAML catalogs under promo_bot/locales; startup rejects missing/extra keys, mismatched placeholders, duplicate YAML keys, translated HTML, and inconsistent plural forms. Unknown locale hints fall back to English.

The interface uses safe HTML templates and a hybrid navigation model. Typed requests, onboarding, promotions, and operational alerts create messages; inline menu navigation edits one live panel. Callbacks are acknowledged before state access, confirmation cards replace themselves with their final state, language choices show a check mark, and preference pagination uses compact ‹ / 2/4 / › controls. Home is role-aware and exposes Preferences, History, Language, Account, Help, and administrator membership management. Deterministic commands are:

• /start and /help
• /preferences and /history
• /preview <instruction> (validates without changing state)
• /undo
• /confirm <id> and /cancel <id>
• /language
• /account

The presentation follows Telegram's official Bot API HTML and inline keyboard features and its guidance for discoverable /start, /help, and command-menu experiences in Telegram Bot Features.

Greetings, help requests, obvious preference-display requests and unknown slash commands are also handled locally. Gemini classifies more flexible natural-language queries, but application code always renders the authoritative SQLite preference snapshot in the reply.

Risky changes receive inline Confirm/Cancel buttons and expire after ten minutes. Bare “yes” is not accepted. Hard-rule edits, bulk/category deletion, changes affecting more than five entries, dated reverts, and multi-entry undo always require an ID-bound confirmation. Applied revisions are kept indefinitely and rollback always creates a new revision.

Telegram commands use Bot API long polling (getUpdates, 30 seconds, 20 updates) and refuse to start if the bot has an active webhook. The durable SQLite reply outbox persists both send and edit operations with target message IDs. Telegram's “message is not modified” result completes an edit; an unavailable edit target is converted once into a fresh send so restarts cannot create an edit/fallback loop. The processed-update offset makes update handling restart-safe. Gemini-backed mutations are limited persistently to five per minute and twenty per hour; previews consume the limit, while queries and confirmations do not.

Promotion notifications default to a still-image sendPhoto card and use sendMessage when no image is available. Sieve assembles the title, normalized Brazilian price, source description or publication comment, localized match reason, and captured offer URLs in application code. It uses explicit UTF-16 bold and blockquote entities instead of a Telegram parse mode, so hostile source markup stays literal and Gemini never writes notification text or links. Captions stay within Telegram's 1,024-character photo-caption limit, with a safe plain-text fallback when media or entity delivery fails.

For example, this source promotion:

Tênis Nike Court Lite 4 (38 a 44)
De R$ 589 por R$ 308
Use o Cupom: PRAMODA
Loja Oficial Nike no ML
menor preço histórico

becomes a deterministic card for a user whose selected language is Brazilian Portuguese:

🏷️ Tênis Nike Court Lite 4 (38 a 44)
R$ 308,00

De R$ 589 por R$ 308
Use o Cupom: PRAMODA
Loja Oficial Nike no ML
menor preço histórico

▎Apareceu porque está no menor preço histórico.
[Ver oferta]

Cards never expose source names, raw/internal match reasons, stages, poisoning verdicts, or Gemini metadata. Brands, model identifiers, coupon codes, seller names, and source offer URLs are preserved.

---

CLI reference

The sieve entrypoint takes a global --config and --log-level, then a subcommand.

Command	Purpose
run	Start the service.
auth-telegram [--source NAME]	Create or refresh the persisted Telethon source session with QR login.
smoke-telegram-preferences [--source NAME] [--session-path PATH] [--timeout SECONDS]	Run the non-mutating live preference-bot gate with a dedicated user session.
replay FIXTURE [--no-fail]	Score pre-LLM filtering against a labeled JSONL file.
health	Print JSON health status; exit 1 if unhealthy. Used by the Docker healthcheck.
validate-config [--runtime]	Validate settings; optionally check required environment variables and enabled factories without API calls.

---

Replay calibration

Replay measures how well the pre-LLM stages perform against labeled data — the whole point being to verify you're not paying Gemini for obvious junk, and not silently dropping deals you wanted.

Each JSONL line is a Promotion.to_dict() payload plus a relevant boolean, either flat or nested under a promotion key. See fixtures/labeled.example.jsonl for the shape.

sieve --config config/config.yaml replay fixtures/labeled.example.jsonl

The command exits non-zero unless pre-LLM filtering rejects ≥90% of labeled irrelevant deals while retaining ≥95% of relevant ones. Use --no-fail to print metrics without the gate.

Note

Replay runs against a fully warmed offline corpus. The live pipeline still fails open to Gemini for its first 500 accepted corpus documents, so early live behaviour will be more permissive than replay suggests. The bundled two-line fixture documents the format only — it is not a dataset.

---

Tests

python -m venv .venv
.venv/bin/pip install -e ".[test,dev]"
.venv/bin/python -m ruff check promo_bot tests
.venv/bin/python -m pytest

On Windows PowerShell, use .venv\Scripts\pip and .venv\Scripts\python.

Docker installs the hash-verified runtime versions in requirements.lock. When changing dependencies, regenerate it with uv pip compile pyproject.toml --python-version 3.12 --universal --generate-hashes --no-annotate --no-header --output-file requirements.lock, then run the Docker gate. Developer conventions and required checks are in AGENTS.md.

The default suite never touches live services; the contract and Docker system gates are skipped unless explicitly enabled. Tests run against saved HTML/JSON-LD fixtures, synthetic events, mocked HTTP transports, deterministic clocks and temporary SQLite files. The suite covers BM25, normalization, filters and exceptional detection, revisioned preference CRUD, Gemini parsing and poisoning controls, bilingual card/cache isolation, UTF-16 entities, media validation and cleanup, Telegram authorization and outbox recovery, restart-safe alias generations, pipeline integration, replay, and soak-marked promotion, 500-entry, 10,000-document rebuild and command-flood runs.

.venv/bin/python -m pytest -m "not soak"   # skip the long one
SIEVE_RUN_GEMINI_CONTRACT=1 GEMINI_API_KEY=... .venv/bin/python -m pytest -m contract
SIEVE_RUN_SYSTEM=1 .venv/bin/python -m pytest -m system
RUN_SOAK=1 .venv/bin/python -m pytest -m soak

Set SIEVE_GEMINI_MODEL to override the contract test's default gemini-3.5-flash-lite model. The Docker gate requires a running Docker daemon. It builds the production image, starts only the synthetic compose.system.yaml stack with dummy credentials, and removes its isolated SQLite volume unconditionally.

---

Rollout

auto-deploy.sh uses its own repository directory and the current Docker context. A scheduler can override these with SIEVE_REPO_DIR and SIEVE_DOCKER_CONTEXT; set SIEVE_OTHER_DOCKER_CONTEXT to check another context for a duplicate running bot. These are shell/scheduler variables, not container settings. Deployment validates configuration before recreating the service and refuses tracked working-tree changes.

Promotion delivery has no shadow mode. Calibrate the independent BM25 auto-forward path before changing pipeline.bm25_auto_forward_mode to live:

• Run the full suite — fixture, contract, integration, recovery and soak tests
• Run the deterministic Docker black-box gate: SIEVE_RUN_SYSTEM=1 python -m pytest -m system
• Keep BM25 auto-forward in shadow until its candidate evidence is acceptable
• Review replay metrics; tune the initial profile, aliases, rules, and BM25 threshold
• With the deployed preference bot running, run the non-mutating live gate: docker compose run --rm sieve --config /app/config/config.yaml smoke-telegram-preferences --source telegram-principal
• Switch only BM25 auto-forward to live; promotion sources themselves have no mode
• Enable remaining Telegram sources, then Pelando

Do not advance unless all of the following hold: no duplicate live deliveries, no sustained memory growth, clean recovery across restarts, and the 90% rejection / 95% retention targets met.

The live gate resolves the bot username from its configured token, verifies that the dedicated Telethon user is the configured preference owner, sends /preferences, performs a nonce-bearing /preview, and proves the authoritative response is unchanged. Its default session is /state/telegram-smoke-user; never point --session-path at a configured source session. First-time authorization displays a QR code and may request a 2FA password; no phone number is accepted. The command prints only a concise JSON report and exits non-zero on timeouts, identity/webhook conflicts, ambiguous previews, or changed state. --timeout defaults to 90 seconds and accepts a bounded 10–300 seconds.

---

Project layout

promo_bot/
├── cli.py            # argparse entrypoint, subcommands
├── runtime.py        # service orchestration, queues, alerts, memory tripwire
├── pipeline.py       # snapshot → filters → constraints → exceptional → BM25 → LLM
├── models.py         # Promotion, Decision, Evaluation, PipelineResult, RetryJob
├── preferences.py    # immutable preference domain, validation, constraints, weights
├── preference_store.py # SQLite entries, revisions, confirmations, rate state, outbox
├── preference_interpreter.py # Gemini natural-language operation parser
├── preference_bot.py # authorized Telegram Bot API long polling and commands
├── telegram_smoke.py # dedicated-session, non-mutating live Telegram gate
├── gemini.py         # persisted request broker + structured-output REST client
├── presentation.py   # deterministic/optional presentation + media/cards
├── protocols.py      # Source / Stage / Evaluator / Sink / Store interfaces
├── config.py         # YAML loading, factory resolution, env secrets
├── filters.py        # fixed spam checks and prioritized token-aware hard rules
├── exceptional.py    # price-error / historical-low / discount bypasses
├── bm25.py           # Okapi BM25
├── normalization.py  # tokenization, alias expansion, content hashing
├── store.py          # SQLite WAL: corpus generations, dedup, retries, audit, health
├── evaluator.py      # Gemini structured-decision client
├── sink.py           # Telegram Bot API sendPhoto/sendMessage delivery + alerts
├── replay.py         # offline calibration metrics
├── logging.py        # JSON stdout logging
└── sources/
    ├── telegram.py   # Telethon user-session ingestion
    └── pelando.py    # conditional /recentes polling, JSON-LD parsing

---

Credits

Telethon lifecycle handling follows the stable Telethon documentation. Pelando structured-feed parsing follows the precedent set by ffirenn/pelando_bot.

License

Sieve is available under the MIT License.