release: wishful 0.4.0 — release readiness + post-release quality, reviewed - #10
Merged
Merged
Conversation
Capture the 2026-06-11 whole-codebase review (74 validated findings) plus the two-plan remediation: plan 001 (P0/P1 + quick wins, release-gated) and plan 002 (post-0.3.0 quality). Add STRATEGY.md anchor and the real-model smoke proof bundle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… nest-asyncio - litellm >=1.83.14 (lock 1.88.1): day-0 gpt-5.5 support + completion->responses bridge; excludes the quarantined 1.82.7/1.82.8 builds - remove pyglove (declared but imported nowhere) - add nest-asyncio (was imported at runtime in explore but undeclared) - uv_build cap lifted to <1.0.0; rich >=14, ruff >=0.15; locks refreshed - mypy stays 1.x; 2.x adoption deferred to the companion plan
- add Settings.request_timeout (WISHFUL_REQUEST_TIMEOUT, default 300s) passed to litellm.completion/acompletion so an import can no longer hang on litellm's multi-minute default - empty-content responses retry once, then raise GenerationError naming the model with a reasoning-model max_tokens hint (gpt-5.5 returns empty when the budget is spent on hidden reasoning tokens) - _FAKE_MODE becomes _is_fake_mode(), read per call so tests toggle without module reimport; drop pragma:no-cover on the now-tested error paths
The split-based stripper left 'python' as the first source line of a standard ```python response, which parsed as a bare name and crashed at exec (then got cached). Replace with a regex that extracts only fenced content, joins multiple blocks, drops the info string, and passes fence-free responses through.
- _generate_and_cache validates generated source BEFORE writing it, so a failed generation never survives as a cache entry (was: write-then-exec, which permanently poisoned the cache on any post-write failure) - malformed generations retry once then raise GenerationError; policy violations (SecurityError) raise immediately without caching - validator re-raises SyntaxError as SyntaxError (not ImportError), reviving the loader's regenerate-once path under default (safety-on) settings - cache writes are atomic (temp file + os.replace); empty/torn cache files read as a miss; cache-hit source is still re-validated on load - add safety-ON integration tests (the composition bug the conftest masked)
module_path() previously stripped both the static and dynamic prefixes, so wishful.static.foo and wishful.dynamic.foo mapped to the same .wishful/foo.py. A regenerate(), review-rejection, or explore-winner write against one namespace then deleted or overwrote the other's cache. Static now stays at <cache>/<name>.py (documented, user-editable); dynamic moves under <cache>/_dynamic/. All call sites route through module_path(), so the fix is namespace-wide. Pre-0.3.0 dynamic caches sat at the static path; they regenerate under _dynamic/ on next import (changelog notes the relocation). No destructive migration: the README documents an edit-and-commit cache workflow, so quarantining existing files would be hostile; the only residue is an orphaned same-named static file.
…ache - underscore-prefixed names (dunders, _private, IPython _repr_html_ probes) raise AttributeError immediately in both the static __getattr__ and the dynamic proxy — a hasattr()/getattr() probe no longer costs a paid LLM call - a regeneration triggered by a public attribute miss is committed only when the new source actually defines the requested symbol (checked via AST); otherwise the existing module and cache are left untouched - split _generate_validated (validate, no write) from _generate_and_cache so the getattr path can validate then conditionally commit
Close the demonstrated bypasses: __import__/compile as calls, importlib/builtins/ ctypes imports, __builtins__ subscript and getattr gadgets, globals()/vars() subscripts, non-literal open() modes, and attribute calls on unbound os/subprocess/sys/importlib/ctypes. Local bindings are tracked so a shadowed 'os = platform.system()' is not a false positive. Aliased/computed access still slips through — AST scanning cannot catch it. That residual is now an xfail corpus, and the README/AGENTS safety sections say plainly this is defense-in-depth, not a sandbox: generated code runs in-process. Validator coverage 97% (negative + positive + binding-form corpus).
…adless - review now runs inside _exec_source between validate and exec, so approval guards real execution on the initial import, the syntax-retry, and all four regeneration paths (it used to run AFTER exec — the code had already run) - review=True with non-interactive stdin raises an actionable ImportError before any LLM call; EOF at the prompt is treated as rejection - notebooks (ipykernel/IPython) count as promptable so review stays usable for the core audience; only CI/pytest/piped stdin fail closed - compile before validate so malformed generations retry uniformly regardless of the safety flag
Remove the global allow_unsafe=True from conftest so every pipeline test exercises the validator that ships — the masked composition bugs (dead retry, cache poisoning) lived exactly where the suite had safety off. Add an opt-in unsafe_settings fixture for the rare test that needs the bypass, a posture guard against regressing the default, and an end-to-end clean-generation import under validation. Update the syntax-retry logging assertion to the safety-on path.
- track wishful's own loguru sink ids and remove only those on reconfigure; never bare logger.remove(), which deleted sinks the host app installed - log_to_file now defaults off (WISHFUL_LOG_TO_FILE=1 to opt in) so a bare 'import wishful' creates no files or directories in the user's CWD - wrap log dir/file creation: a read-only or full filesystem degrades to console-only logging with one warning instead of crashing mid-import
- compare dataclass fields against dataclasses.MISSING; the old self-comparison (field.default_factory is not field.default_factory, always False) made the factory branch dead and emitted <_MISSING_TYPE> repr into the LLM prompt - resolve generics via get_origin/get_args before the bare __name__ shortcut, which on 3.10+ dropped type args (list[str] -> list, Optional[int] -> Optional) - use inspect.isclass / a captured _NONE_TYPE since this module shadows builtin type with its decorator; resolve string annotations via get_type_hints - rewrite the F811 duplicate-class test into a real last-registration-wins check
An adversarial verification pass over the U4-U9 changes confirmed five P0
bypasses and two correctness regressions:
- locals() subscript reached __builtins__ (only globals()/vars() were blocked) —
add locals() to the blocked-subscript set, killing the locals()['__builtins__']
.eval/.exec/.compile/__import__ gadget chains
- aliased and default-arg dangerous builtins (f = open; f(p,'w'); def g(x=open))
bypassed the call-site checks — block bare references to open/getattr/eval/exec/
compile/__import__/globals/vars/locals/__builtins__ that are not a call target.
This also closes the previously-documented aliased-getattr residual.
- _exec_source caught only SyntaxError, so a SecurityError on a rejected static
cache left the poisoned file in place; now delete it and re-raise
- explore() winner is re-validated under current settings before caching
- _source_defines now recognizes tuple/list/starred unpacking targets, so a
regeneration that binds the requested symbol via unpacking is no longer
wrongly rejected (a U6 regression)
Only two genuinely-computed residuals remain (globals().get('__builtins__'),
runtime-built attribute names), documented as xfail. Validator coverage 97%.
… version - __version__ derives from importlib.metadata (was hardcoded '0.1.0' vs 0.2.4) - [project.scripts] wishful = wishful.__main__:main so the documented 'wishful' command works, not just 'python -m wishful' - argparse CLI: --json on inspect/clear/regen, --version, exit codes (0 ok, 1 error, 2 usage); regen validates the module name against an allowlist so a crafted name can't map to an arbitrary file (the path-traversal guard ships with the CLI even though the broad guard is deferred) - .env.template documents all ~13 environment variables
…iformly - swap model precedence so the wishful-specific WISHFUL_MODEL wins over the generic DEFAULT_MODEL (which stays as the fallback); warn once when both are set and disagree, since the swap otherwise changes the active model silently - move every env-derived Settings field to default_factory so a fresh Settings() (as reset_defaults does) re-reads the current environment uniformly — previously only cache_dir re-read, the rest froze at import
…riginal fitness - timeout_per_variant now bounds the mutation call and the user-supplied test/fitness callables via a per-variant ThreadPoolExecutor; a timeout or any BaseException (incl. SystemExit) is recorded as a failed variant and the loop continues instead of hanging or killing the host - remove the verbose parameter, which was accepted but never used (passing it now raises TypeError); re-add when there is real progress output to gate - scoring the original function is wrapped so a fitness() that raises on the original is recorded as failed instead of crashing the whole run
…shful.cache - add WishfulError base; GenerationError/SecurityError extend (WishfulError, ImportError) so except ImportError still works, ExplorationError/EvolutionError extend WishfulError. Callers can catch the whole family with one except. - drop 'type' from __all__ so 'from wishful import *' no longer shadows the builtin; wishful.type still works as an attribute - rename the cache alias (manager as _cache) so importing wishful no longer overwrites the wishful.cache subpackage attribute; import wishful.cache.manager works again - annotate reimport() -> ModuleType; verify subpackage __all__ names resolve Full mypy clean is deferred to the companion plan with mypy 2.x.
- delete explore/strategies.py (selection logic duplicated inline in explorer.py, 0% coverage, no importers) - delete ExploreProgressPrinter (~100 lines, never instantiated) - ruff --fix sweep + hand-fixes: reorder loader/__init__ imports to the top, drop an unused test var, replace == True/False comparisons in example 13; ignore E402 in examples/ where mid-file wishful imports are the demonstration
…gate - tests/smoke/: pytest -m smoke, skipped unless WISHFUL_SMOKE=1 and provider credentials are present. Four cases (static import, dynamic call, explore, evolve) exercise the real-model paths and record a proof entry; the bundle is written to docs/proofs/<version>/summary.json at session end - proof content is metadata only (model, timing, pass/fail, attempts, short result summary) — never prompts or generated source, so it can't leak - always-on harness tests verify the gate and the proof schema without API calls - CI runs the suite under a coverage gate (fail_under=80) plus ruff; register the smoke marker and omit generated cache from coverage - RELEASE_CHECKLIST.md and docs/proofs/README.md document the flake policy, secret scan, and proof convention
- badges: 154->263 tests, 78%->86% coverage - README/AGENTS: project tree adds evolve/, exceptions.py, logging.py; drop the deleted strategies.py; full 15-example list; public API + Settings fields updated (WishfulError, request_timeout, log_to_file default off, console script + --json CLI, model precedence); uv run python everywhere - docs-site: 0.3.0 changelog entry, Explore in the sidebar, log_to_file/ request_timeout/precedence corrections - gitignore + git rm the stale committed coverage.json
U1a falsification finding: the litellm bump was necessary but not sufficient. gpt-5.5 is a reasoning model that spends part of its token budget on hidden reasoning, so the old max_tokens=4096 left too little for output — producing empty content (example 09) or truncated/invalid syntax (example 00). Raising the default to 16384 fixes both classes (example 09 now passes end to end). Also harden two examples for the headless smoke sweep: - example 10: rich prompts fall back to defaults without a TTY and the section loop is bounded, so it runs headless instead of raising EOFError - example 00: the count_headers hint steers the model to open()+str methods and away from os, which the safety validator (correctly) blocks
Bump to 0.3.0 with the kept real-model proof. The smoke harness passes 4/4 against gpt-5.5 (docs/proofs/0.3.0/summary.json): static import, dynamic call, explore, and evolve all generate validated output end to end. U1a falsification (docs/proofs/0.3.0/examples-sweep.md): the litellm 1.88 bump alone did not fix the gpt-5.5 empty-content failures — raising max_tokens to 16384 (reasoning models spend budget on hidden tokens) is the real fix. The previously-broken core examples (00, 09) now pass; the heavy dynamic-creative examples (08, 10) are slow pending the plan-002 dynamic-economics fix, and the LLM-as-judge example (13) is excluded per plan 001 U1a. Closes plan 001 (P0/P1 remediation + quick wins).
Follow-up to the 0.3.0 branch review: the hardened validator still let the classic Python sandbox escape through — ().__class__.__bases__[0].__subclasses__()[N].__init__.__globals__['__builtins__']['eval'] and nested ns['__builtins__']['eval'] subscripts. These reach eval/exec/RCE and were undocumented, contradicting the 'blocks dangerous patterns' claim. - block escape-dunder attribute access (__subclasses__/__bases__/__base__/__mro__/ __globals__/__code__/__closure__/__builtins__/__subclasshook__/__getattribute__) - block subscripts with a forbidden string key (__builtins__/eval/exec/...) regardless of the base object - common introspection (obj.__class__.__name__) still passes; only the escape primitives are blocked - README narrows the residual claim to genuinely computed access; validator coverage 98%
Follow-up to the 0.3.0 branch review: - evolve()'s per-variant timeout used a ThreadPoolExecutor whose worker threads are non-daemon, so a timed-out (uncancellable) callable would block interpreter exit and accumulate across runs. Use a daemon thread instead: a runaway candidate is abandoned cleanly without leaking into shutdown. - the CLI regen allowlist accepted 'wishful' and 'wishful.text', letting 'wishful regen wishful' target framework packages. A wishful.* name must now be a fully-qualified static/dynamic module; bare names map to the static namespace.
…ction The cache entry described the old colliding behavior; it now documents the namespace isolation fix. Add an explicit Breaking changes section (max_tokens default raise, evolve verbose removal, log_to_file default, model precedence) and note the validator's introspection-gadget blocking.
…arning Document the 0.3.0 finding for future LLM-integration work: reasoning models (gpt-5.x) need a generous max_tokens because hidden reasoning tokens consume the budget (empty content = fully spent, truncated = nearly spent), and pre-register load-bearing dependency bets so a dedicated step can falsify them early rather than at the release gate. Surface docs/solutions/ in AGENTS.md.
Second re-verification found RCE through getattr with a non-literal attribute
name: getattr(x, a) where a='__class__', getattr(o, 'sy'+'stem'), and
getattr(globals().get('__builtins__'), 'open') all bypassed the escape-attr and
constant checks (which only saw .attr syntax / literal strings).
- getattr/setattr/delattr/hasattr now require a literal attribute name; a
variable/concatenated/computed name is blocked outright
- the literal name is checked against escape dunders + dangerous builtins + open
- reaching builtins as the target (getattr(__builtins__/globals()/vars()/locals(),
...)) is blocked
- the documented residual is narrowed honestly to library-level reflection
(operator.attrgetter, value-level mapping indirection) that AST cannot model
…klist docs Final security gate found two more vectors: - type.__dict__['__subclasses__'] reached the escape dunders via subscript (I blocked the attribute form but not the subscript form) — add the escape dunders to the forbidden subscript keys so the two forms mirror - pathlib.Path.write_text + runpy.run_path achieved RCE — block the runpy/pickle/ marshal/shutil/code/codeop/socket/multiprocessing import family and the write_text/write_bytes/run_path/exec_module method names (regardless of base); reads (Path.read_text, json.loads) stay allowed These two are fixed, but they expose the fundamental truth: a blocklist over Python is incomplete by nature. The docstring and README now say so plainly — the validator is a seatbelt, and the real boundary for untrusted code is the review gate plus an environment where code execution is acceptable (or an out-of-process sandbox). Further stdlib/reflection vectors are that documented, fundamental residual.
…ntity Plan 002 U1. DynamicProxyModule.__getattribute__ eagerly called _regenerate_for_proxy on every attribute access, then _call_with_runtime generated again with the real args — every dynamic call paid for TWO LLM generations, and hasattr/dir probes paid for one. This is the economics issue behind the slow examples 08/10 in the 0.3.0 release. - __getattribute__ is now lazy: a public name returns a callable that regenerates exactly once, with runtime args, when invoked. Attribute access and hasattr/dir cost zero generations. - _exec_source preserves __name__/__spec__/__loader__/__package__/__file__/ __builtins__ across the clear_first re-exec, so a dynamic module keeps its identity after regeneration. - delete the now-dead _regenerate_for_proxy. - tests assert the new per-call count, zero-cost probes, and identity survival.
Plan 002 U3 (depth): - cache.module_path/dynamic_snapshot_path now validate every name component is a plain identifier and resolve the final path inside the cache dir (symlink-safe), so a crafted name passed to the public regenerate()/reimport() API can't escape the cache directory. The CLI already pre-validated; this closes the API path. - prompt/context bodies (which can carry the caller's source and secrets) are redacted from logs unless WISHFUL_LOG_PROMPTS=1, even at DEBUG. Metadata (module, model, lengths, function names) still logs. Remaining U3 sub-item (untrusted-context prompt delimiting) tracked as follow-up.
Plan 002 U2 (partial). explore() wrote winner.__wishful_source__ over the whole module file, so exploring one function wiped every other symbol already cached there. Now the winner is merged: the explored function is replaced and all sibling defs/classes are preserved (re-validated before write). Also make the defaults headless-aware: - verbose defaults to stdout.isatty() (quiet in CI/scripts) - save_results defaults to WISHFUL_EXPLORE_SAVE_RESULTS (on unless '0') Remaining U2 (asyncio runtime refactor off nest_asyncio, bounded candidate executor, full VariantResult.error_message) tracked as follow-up.
The ce-code-review loop gate confirmed 3 P1 + several P2 issues introduced by
this branch's own work. Fixes:
P1 — validator over-correction (security): blocking ALL non-literal getattr
names broke the extremely common getattr(obj, field) / {f: getattr(o,f)} idiom
by default. Revert to blocking only forbidden *literal* names + builtins targets;
computed/variable getattr is now a documented residual (consistent with the
'best-effort blocklist, not a sandbox' stance). Real gadgets stay blocked.
P1 — explore() merge broke on 'from __future__': appending a winner whose source
starts with __future__ after a sibling put it mid-file (compiles to SyntaxError,
silently invalidating the cache). _merge_into_module now hoists/dedupes __future__
to the top and compile()-checks the result, falling back to the winner alone.
P1 — duplicate 'import pytest' in test_cache.py failed the new CI ruff gate on
its own diff. Removed.
P2 — explore() raised SecurityError and discarded a valid winner when a
pre-existing dangerous sibling was in the module; now falls back to caching the
winner alone and still returns it (contract honored). Tested.
P2 — subscript-key blocklist over-blocked benign data access (config['system'],
row['eval']); keep only the dunder escape keys, plain words dropped. The real
builtins gadgets are still caught by the base/dunder checks.
P2 — README max_tokens default corrected 4096 -> 16384.
304 tests pass, ruff clean.
After bypass-hunting closed real RCE gadgets in the validator, the tightened blocklist over-blocked common legitimate code (getattr(obj, field), config['system']). The bypass-hunting passes never caught it — only a paired false-positive/usability review did. Document the lesson: tightening a blocklist against an exploit needs a before/after benign-corpus diff, and ambiguous forms belong in the documented residual.
The static review-gating paths were tested but the dynamic per-call regeneration path (_call_with_runtime -> _exec_source -> _maybe_review) was not. Add a test asserting a runtime dynamic call routes through the review prompt and a rejection raises before the freshly generated source executes.
Plan 002 U6 (type debt). The bulk (24) were the deliberate 'type' shadow in types/registry.py — the public @wishful.type decorator shadows the builtin, so every 'x: type' annotation resolved to the function. Capture _Class = type before the shadow for plain annotations and use typing.Type[T] for parameterized ones; runtime is unchanged (annotations are deferred). The rest: - discovery._matches_import_from: narrow Optional[str] with an explicit guard so mypy sees the non-None path (also clearer than relying on bool() short-circuit) - progress: annotate the mixed-renderable list as list[Any] - justified ignores for two genuinely-dynamic attributes (builtins._wishful_settings stash, fn.__wishful_source__ marker) - [tool.mypy] override so the stub-less nest_asyncio import is quiet mypy src/wishful: Success, no issues. ruff clean, 305 tests pass.
A relative cache_dir (the default '.wishful') silently relocated if the process chdir'd after configuring. Pin it to an absolute path when set via configure().
…platform.system() Add posix/nt/_posixsubprocess to the import deny-list — `from posix import system` reached os.system with safety ON. Drop the blanket .system/.popen/.spawn method block (and the system/popen getattr-literal entries) that rejected the legitimate platform.system(); the dangerous os forms stay caught by the import ban plus the unbound-base check. Remove the now-dead _FORBIDDEN_ATTR_STRINGS set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
configure_logging() left loguru's bootstrap sink 0 in place, so every record printed twice (its stderr default plus wishful's Rich console sink) and wishful's DEBUG internals leaked to the host stderr even at WARNING. Remove sink 0 once on first configure; host-added sinks (ids >= 1) are never touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…untime error Thread from_cache through _exec_source: rejecting the review prompt for a cache-hit (or hand-edited) module no longer deletes the user's file, and a freshly generated module that raises at exec is removed from the static cache so the import can recover instead of breaking permanently. Drop the pre-delete in _regenerate_with — a transient generation failure no longer wipes the working cache (atomic write overwrites on success). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hermetic tests The mutation call inherited the global 300s request timeout, so a timed-out variant's abandoned daemon thread kept its HTTP call alive far past timeout_per_variant. Thread an optional per-call timeout through generate_module_code -> mutate_with_llm and cap it at the per-variant budget. Stub the LLM in the three TestEvolveContract tests so they stop making real paid OpenAI calls (conftest never forced fake mode). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The winner is appended after the kept siblings, so a winner helper sharing a sibling's top-level name silently rebinds it. Detect the collision and log a warning instead of wiping the sibling silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…S.md Capture the symmetric deny-list lesson from the validator review (under-block via the os/subprocess C-level aliases vs over-block via blanket name bans) as a best-practice doc, cross-linked to the false-positive-review sibling. Seed CONCEPTS.md with the safety-area core nouns: Validator, Review gate, Residual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…claim Correct max_tokens 4096 -> 16384 (AGENTS, configuration.mdx, .env.template), test counts 261/263 -> 326 (README badge + tree, AGENTS), the stale 'tests run allow_unsafe=True' line (U9 flipped the suite to safety-on), and cli.mdx's 'static and dynamic share the same cache file' claim. Document WISHFUL_CONTEXT_RADIUS and WISHFUL_EXPLORE_SAVE_RESULTS; add a CONCEPTS.md pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dynamic proxy semantics are now documented and pinned by tests: attribute access is lazy and free (hasattr always True, wrappers for every public name), generation happens only at call time, and dynamic modules expose functions only. _call_with_runtime gets the same commit-guard _dynamic_getattr already had: a generation that fails to define the called symbol no longer replaces the module namespace or the snapshot — the caller gets AttributeError and the previous state survives. Also corrects the stale cache-layout line in how-it-works (dynamic snapshots live on a disjoint path since the namespace split). Smoke-tested against openai/gpt-4.1 (dynamic call + contract assertions).
…xception explore's user callables ran inline and unbounded — a while-True test hung the run and sys.exit() in a candidate killed the host. They now run through run_user_callable (daemon worker + per-variant timeout + BaseException containment), extracted to core/execution.py as the shared seam evolve already used; evolver imports it from there.
VariantResult.error_message kept only the first 80 chars (60 for compile errors), so ExplorationError.failures lost the diagnosis. Full text is now stored; the Rich table already truncates for display.
…syncio Spike result (2026-06-11, litellm 1.88.1, real gpt-4.1): per-call asyncio.run() still abandons litellm's Logging.async_success_handler coroutines — the loop dies before fire-and-forget logging runs. So the plan's conservative design applies: ONE persistent loop on a wishful-owned daemon thread. _run_async submits via run_coroutine_threadsafe and blocks, which also makes explore() safe inside a running loop (Jupyter) without nest_asyncio patching the host. The module-level coroutine-warning filter narrows from all coroutines to litellm's Logging.* only, so a user's own forgotten await still warns. nest-asyncio removed from dependencies and the mypy overrides. Smoke-tested against openai/gpt-4.1: two sequential explores plus one inside asyncio.run().
…d radius - compile_and_exec in core/execution.py is the one compile path for explorer and evolver, with a post-exec callback slot for deferred casefile writes (never inside an import lock). - evolve() returns EvolutionResult: callable, proxies the winner's __name__/__doc__/__wishful_source__/__wishful_evolution__ via __getattr__, sets __wrapped__ for inspect.signature, carries history/best_score. No accept() ships — spec-003 Open Decision 1 is unresolved and a no-op would be a trap. - __wishful_evolution__ is a versioned TypedDict (schema_version=1). - context_radius moves into Settings (configure/reset/env aware); set_context_radius is a thin wrapper. - explore's generation calls carry registered @wishful.type schemas and output bindings (verified end-to-end: gpt-4.1 produced the registered dataclass). - configure()/reset_defaults() mutate settings under a lock; 8-thread consistency test. - cache_dir resolves absolute at Settings construction, so os.chdir() can't move the cache even for import-only users. - MagicLoader.generate_fn is the injection seam for generation; the monkeypatch-tolerant resolver stays as documented fallback. - explore's CSV writer marked legacy pending spec-003 Open Decision 4. - conftest clears the type registry around every test. Smoke-tested against openai/gpt-4.1 (evolve wrapper + typed explore).
…#62 test debt - import wishful.static.a.b now fails at discovery time with an error that names the problem and a flattening hint — before the parent burns an LLM call (previously: parent generated, then an opaque 'X is not a package'). - mypy floor bumped to 2.1 (clean, no new findings). - #62 bundle: finder install() idempotence, _ensure_symbols fresh-generation failure path, cache helper lifecycle (has_cached/delete/snapshot paths), evolve argument validation, Windows path normalization in _is_user_frame, registry TypedDict-vs-dataclass discrimination and pydantic-v1 fallback. (The coroutine-warning filter was already narrowed to litellm's Logging.* in the owned-loop commit — GC-time warnings can't be scoped tighter.) Smoke-tested against openai/gpt-4.1: nested rejection in 0ms, zero cache artifacts.
…e race) - #44: __wishful_source__ is attached unconditionally to explore-returned functions (empty string over AttributeError) — the documented attribute is never absent. - #48 decided: evolve keeps the sync LLM path (worker threads + per-variant timeout bound the sync call; the async path belongs to explore's owned loop). Decision documented at the call site and pinned by a test that bombs if evolve ever reaches the async client. - The cross-process cache-race gap gets its test: two spawn-context writer processes hammer one cache path while the parent reads — os.replace atomicity means a reader sees a complete payload or a miss, never a torn file.
- 12_explore: return_all=True section (pick your own winner). - 14_evolve: keep_history/history_limit on the main call, EvolutionResult evidence (best_score, history), and an EvolutionError try/except demo. - 15_cli_and_config (new): CLI walkthrough incl. --json and exit codes, configure(model/temperature/max_tokens), reset_defaults(). LLM-free. - 16_safety_and_review (new): SecurityError on a planted poisoned cache file, allow_unsafe walkthrough, review=True TTY-gated demo with honest best-effort-blocklist framing. LLM-free. - 11_logging rewritten to earn its name (it was a mislabeled copy of the JSON example): debug, log_level, log_to_file, log_prompts demonstrated distinctly. - 09_context_shenanigans: docstring + section commentary. - README example count/tree updated (17 examples). Verified: 11/14/15/16 green under WISHFUL_FAKE_LLM=1; 09/12 green against real openai/gpt-4.1.
Every member starts with an underscore, which the first clause of __getattribute__ already catches — the set membership check was unreachable.
Validated P1s (independent validators + repros): - The owned explore loop now tracks its runner thread and owner pid: a dead thread or a fork is detected and a fresh loop starts instead of run_coroutine_threadsafe enqueueing work that never runs; _run_async waits with a watchdog poll so a mid-run thread death raises instead of hanging forever. - explore's test/benchmark callables are awaited via asyncio.to_thread: run_user_callable's worker.join no longer blocks the owned loop, so a candidate that itself calls explore() completes instead of deadlocking until timeout (verified vs gpt-4.1: 6.2s, was full-burn), and concurrent explores no longer starve. Verified P2s: - _commit_regeneration (shared by all three regen paths) rolls back the module namespace AND the previous cache/snapshot file when a symbol-defining generation raises at exec — the commit-guard contract now holds end to end (repro: module was gutted to _wishful_loader). - EvolutionResult.__getattr__ reads fn via __dict__ — copy/deepcopy/ unpickle no longer recurse infinitely; exported as wishful.EvolutionResult (parity with EvolutionError). - compile_and_exec contains module-level SystemExit (no imports needed, so the validator can't block it) as ValueError. - Settings.copy()/reset_defaults() are dataclasses.fields-driven; writer-lock docstrings no longer overclaim reader atomicity. - module_path/dynamic_snapshot_path read settings.cache_dir once, so a concurrent configure(cache_dir=...) can't trip a spurious escape error; cross-process race test gained a CI deadline. - Example 16 evicts the allow_unsafe demo module from sys.modules and pins its preconditions against ambient env. Docs/standards sync: AGENTS.md (deps, context_radius, public API, test/example counts, new test files), README count, CONCEPTS.md nested-wish entry, changelog 0.3.1-unreleased section covering every user-visible change since the 0.3.0 release, version bumped to 0.3.1 per AGENTS.md's per-commit patch-bump rule. Tests: test_evolve.py split (1132 -> 971 lines) into test_evolve_result.py; new test_execution.py unit-tests the seam incl. validate-before-exec canary; loop-death recovery, re-entrant explore, commit-guard rollback, and copy-semantics regressions pinned. 373 passed; ruff + mypy clean; loop fixes smoke-tested vs gpt-4.1.
evolve()'s return-type change (EvolutionResult) is breaking, so the post-0.3.0 work ships as a minor bump per pre-1.0 semver. 0.3.0 stays in the changelog as an internal milestone with a note that it was never published; 0.4.0 is the first public release of both batches.
… dep CI (every PR + main): ruff, mypy, bandit (B102/B110 acknowledged in pyproject — exec'ing generated code is the product), pytest with the 80% coverage gate on py3.12+3.13, the offline-safe examples sweep, sdist/wheel build with a clean-venv import check, and the docs build. Minimal permissions, concurrency cancellation, cached uv. release.yml replaces publish.yml: publishing now happens on a v* tag after a full gate re-run and a tag==pyproject version check — the old workflow published on every main push touching pyproject.toml, which with the bump-every-commit convention meant every merged PR would have shipped to PyPI. smoke.yml (manual): runs the credentialed real-model harness and uploads the metadata-only proof bundle. audit.yml: weekly + manifest-change pip-audit of the locked tree. docs-site: drop @astrojs/check — it pulled volar-service-emmet's git+ssh dependency (github:ramya-rao-a/css-parser), which broke npm ci locally (npm's git-dep preparation flag conflict) and could never install on CI runners. astro build doesn't use it; lockfile now has zero git deps. All CI commands verified locally (suite 87% coverage, build, docs: 11 pages).
First catch of the new audit gate: pip-audit flagged known vulnerabilities in transitive deps (aiohttp 3.13.2 et al., all with released fixes). uv lock --upgrade within existing manifest constraints; litellm stays at 1.88.1, so the owned-loop design verification is unaffected. Suite, ruff, mypy re-verified green; pip-audit clean locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Releases wishful 0.4.0 — the 0.3.0 release-readiness work plus the full post-0.3.0 quality plan, shipped together. 0.3.0 stays in the changelog as an internal milestone that was never published; 0.4.0 is the first public release of both batches (the
evolve()return-type change is breaking, hence the minor bump).Two plans executed end to end:
docs/proofs/0.3.0/), gpt-5.5 empty-content fix (max_tokens4096→16384 — reasoning models spend budget on hidden tokens), import-time LLM timeouts, cache-poisoning fixes, validator hardening with honest best-effort-blocklist framing, logging citizenship, argparse CLI with--json,WishfulErrorbase.SystemExitcontainment), untruncated failure messages, wishful-owned persistent event loop replacing nest_asyncio (litellm 1.88.1 spike showed per-callasyncio.run()still abandons its logging coroutines), spec-003 architecture seams (core/execution.pywithcompile_and_exec+ casefile callback slot,EvolutionResultwrapper, versioned__wishful_evolution__, settings-backedcontext_radius, injectableMagicLoader.generate_fn, absolutecache_dir), nested-wish rejection before any LLM spend, mypy 2.1, examples 15/16, test-debt closure.A 10-reviewer multi-agent code review ran on the final diff; all validated findings are fixed in
fix(review)(loop dead-thread/fork detection, re-entrant explore deadlock viaasyncio.to_thread, commit-guard rollback on exec failure,EvolutionResultcopy/pickle recursion guard,SystemExitcontainment in the shared exec path).Breaking changes (see changelog for the full list)
evolve()returnsEvolutionResult(callable wrapper; useresult.fnfor the bare function);evolve(verbose=)removed__wishful_evolution__gainedschema_versionwishful.settings.cache_diris always absolutelog_to_filedefaults off;WISHFUL_MODELbeatsDEFAULT_MODELTesting
docs/proofs/0.3.0/) and 4/4 against gpt-4.1 post-quality-worktest_execution.py(validate-before-exec canary,SystemExitcontainment),test_evolve_result.py(wrapper contract, copy semantics), loop-death recovery, commit-guard rollback, cross-process cache atomicity (spawn-context, CI deadline)Post-Deploy Monitoring & Validation
Library release (no deployed runtime). Validation window: first week of 0.4.0 adoption.
pip install wishful==0.4.0+import wishfulclean; examples 00–16 run green with a real key;wishful --versionprints 0.4.0.explore()(would implicate the owned-loop redesign — the watchdog should instead raiseRuntimeError: …event-loop thread died);RecursionErroraroundEvolutionResult;ImportError: nested wishful modulereports from users who relied on dotted wish names; complaints aboutevolve()return type from 0.2.x users (expected — documented breaking change).pip install wishful==0.2.*); no data migrations, cache files are forward-compatible (worst casewishful clear).Known Residuals (accepted advisories from the review)
errorrather thanfailedin CSV/display (cosmetic; full message preserved)explore()calls writing one module can lose the other's winner (pre-existing read-merge-write race; single-writer-per-module assumed until spec-003's evidence layout)configure_loggingruns outside the lockrun_user_callablecannot cancel a running thread — timed-out candidates run to completion concurrently (documented, inherent CPython limit)🤖 Generated with Claude Code