Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,20 @@ extern now : unit -> Datetime = datetime.datetime.now # import datetime; date
extern zeros : int -> Arr = np.zeros # import numpy as np; np.zeros(n)
```

**An undecidable module prefix is a compile error, not a guess.** A lowercase segment *after the
first* could be a submodule (`os.path`, `urllib.request`) or an object (`sys.stdout`,
`datetime.datetime`), and which one it is belongs to the running environment, not to the text. The
compiler used to guess with PEP 8 (maximal leading run of lowercase segments), which emitted
`import sys.stdout` for `sys.stdout.flush` and raised `ImportError` at runtime. It now refuses the
shape and names the fix: `` cannot tell which part of `sys.stdout.flush` names the module … declare it
with `extern import sys` — or `extern import sys.stdout` if `stdout` really is a module ``. A declared
`extern import` settles it and the error goes away, which is the escape hatch that already existed.
Capitalised segments still settle themselves (`pathlib.Path.read_text` needs nothing, since `Path` is
a class), and a two-segment target has nothing to guess. The alternative — emitting the deeper import
guarded by `try/except ImportError` — was rejected: it puts defensive machinery in the output for
something the author can state exactly, and a compile error that names the one line to add is the
better trade (`ROADMAP.md` finding #7).

A used target rooted at a declared path (longest match) — or at its alias name — imports the module
exactly as declared (trusted as written, the same signed-contract stance as the rest of the
boundary), and only otherwise does the heuristic decide (`Lowerer::extern_import_spec`). One
Expand Down
8 changes: 6 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,12 @@ on 2026-07-31; each entry records what was chosen and what was turned down with
not worth it. Rejections are silent, which is the real cost: a program keeps its recursion without
saying why.

7. **A dotted `extern` target imports the wrong prefix when a lowercase segment is not a module**
(S–M, reported 2026-07-31) — `extern flush : unit -> unit = sys.stdout.flush` emits
7. ~~**A dotted `extern` target imports the wrong prefix when a lowercase segment is not a module**~~
**CLOSED 2026-07-31** (was S–M, reported the same day) — fixed with option (b): the undecidable
shape is now a compile error naming the `extern import` to add (`types::undecidable_extern_segment`,
checked in `build_decls` so `pyfun check` reports it, not only `compile`). Two shipped externs in
`examples/interop/http_fetch.pyfun` needed the declaration the diagnostic asks for, which is the
expected cost of the trade. Original report below. — `extern flush : unit -> unit = sys.stdout.flush` emits
`import sys.stdout`, which raises `ImportError`. The *call* is always right; only the import line
is wrong. `lowering::extern_import` takes the **maximal leading run of lowercase-initial segments**
before the final name, following PEP 8 (packages lowercase, classes capitalised), so it succeeds
Expand Down
4 changes: 3 additions & 1 deletion examples/interop/http_fetch.pyfun
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
# compiler follows them through the pipeline instead of trusting an open `-> a`.
extern type Response
extern type Bytes
extern urlopen: string -> Response = urllib.request.urlopen # imports urllib.request
extern import urllib.request
extern urlopen: string -> Response = urllib.request.urlopen
extern readBody: Response -> Bytes = .read() # resp.read() — a method
extern decode: Bytes -> string -> string = codecs.decode

Expand All @@ -28,6 +29,7 @@ print (describe (fetch "://malformed")) # failed: ValueError
# The sibling of a method extern is a *property* read: `= .attr` with no `()`.
# `urlparse` returns a value whose parts are attributes — pure string parsing.
extern type Url
extern import urllib.parse
extern urlparse: string -> Url = urllib.parse.urlparse
extern scheme: Url -> string = .scheme
extern path: Url -> string = .path
Expand Down
26 changes: 2 additions & 24 deletions src/lowering/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3637,28 +3637,6 @@ fn is_unit_domain(ty: &TypeExpr) -> bool {
TypeExpr::Con(name, _, args) if name == "unit" && args.is_empty()))
}

/// Python builtin *type* names — available without an `import`, so a dotted extern
/// target rooted at one (`bytes.decode`, `int.from_bytes`) must not emit an import.
const PY_BUILTIN_TYPES: &[&str] = &[
"bool",
"int",
"float",
"complex",
"str",
"bytes",
"bytearray",
"memoryview",
"list",
"tuple",
"dict",
"set",
"frozenset",
"range",
"slice",
"object",
"type",
];

/// Lower a pinned `extern` keyword-argument literal to its Python IR expression.
/// A negative int/float is emitted as a `Neg` of the magnitude, matching how the
/// emitter renders unary minus (`compresslevel=-1`).
Expand Down Expand Up @@ -3827,7 +3805,7 @@ fn nullary_lambda(target: &[String], spec: Vec<(String, KwSource)>) -> PyExpr {
/// A target rooted at a builtin *type* (`bytes.decode`, `int.from_bytes`) imports
/// nothing — those names are always in scope.
fn extern_import(target: &[String]) -> Option<String> {
if target.len() < 2 || PY_BUILTIN_TYPES.contains(&target[0].as_str()) {
if target.len() < 2 || crate::types::PY_BUILTIN_TYPES.contains(&target[0].as_str()) {
return None;
}
let prefix = &target[..target.len() - 1];
Expand Down Expand Up @@ -3916,7 +3894,7 @@ fn py_value_name(name: &str) -> String {
// of them directly (`set([…])`, `dict(…)`, `list(…)`).
if PY_KEYWORDS.contains(&name)
|| PY_EMITTED_BUILTINS.contains(&name)
|| PY_BUILTIN_TYPES.contains(&name)
|| crate::types::PY_BUILTIN_TYPES.contains(&name)
{
format!("{name}_")
} else {
Expand Down
95 changes: 95 additions & 0 deletions src/types/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,54 @@ pub const PRELUDE: &[(&str, usize)] = &[
("snd", 1),
];

/// Python builtin *type* names — available without an `import`, so a dotted extern
/// target rooted at one (`bytes.decode`, `int.from_bytes`) must not emit an import.
pub const PY_BUILTIN_TYPES: &[&str] = &[
"bool",
"int",
"float",
"complex",
"str",
"bytes",
"bytearray",
"memoryview",
"list",
"tuple",
"dict",
"set",
"frozenset",
"range",
"slice",
"object",
"type",
];

/// The segment of a dotted `extern` target that makes its module prefix
/// **undecidable**, if any (`DESIGN.md` §6, `ROADMAP.md` finding #7).
///
/// A target mixes a module path with an attribute path (`urllib.request.urlopen`
/// is module `urllib.request` + attr `urlopen`; `sqlite3.Connection.execute` is
/// module `sqlite3` + attrs). PEP 8 tells them apart when the attribute is
/// capitalised, but a **lowercase** segment after the first could be either a
/// submodule (`os.path`, `urllib.request`) or an object (`sys.stdout`), and which
/// one it is belongs to the target environment, not to the text. Emitting a guess
/// produced `import sys.stdout`, which raises `ImportError` at runtime.
///
/// So the compiler refuses to guess and asks for an `extern import` instead. This
/// returns the first segment it cannot classify; `None` means the shape is
/// decidable (a single package, or a run stopped by a capitalised segment).
pub fn undecidable_extern_segment(target: &[String]) -> Option<&str> {
if target.len() < 3 || PY_BUILTIN_TYPES.contains(&target[0].as_str()) {
return None;
}
// Everything before the final referenced name is the candidate module path;
// the first segment is always the package, so only later ones are in doubt.
target[1..target.len() - 1]
.iter()
.find(|seg| seg.chars().next().is_some_and(char::is_lowercase))
.map(String::as_str)
}

/// The list prelude (`DESIGN.md` §6): functions over the eager `List a` type
/// (which lowers to a Python list — a dynamic array, so index/`len` are O(1),
/// prepend is O(n)). Like [`PRELUDE`], the `(name, arity)` pairs are the single
Expand Down Expand Up @@ -2579,6 +2627,24 @@ fn build_decls(
// variables are collected from the declared type (bare lowercase names, as in
// `type` decls) and generalized; the boundary is effectful-by-default, so the
// innermost arrow gets `io` unless the binding asserts `pure`.
// Which module an `extern import` declares, for the check below: the same
// longest-prefix match lowering uses to prefer a declaration over the
// heuristic, so the two agree on what counts as "declared".
let declared_imports: Vec<(Vec<String>, Option<String>)> = module
.items
.iter()
.filter_map(|item| match item {
Item::ExternImport { path, alias, .. } => Some((path.clone(), alias.clone())),
_ => None,
})
.collect();
let is_declared = |target: &[String]| {
declared_imports.iter().any(|(path, alias)| match alias {
Some(a) => target.first() == Some(a),
None => target.len() > path.len() && target.starts_with(path),
})
};

for item in &module.items {
let Item::Extern(decl) = item else { continue };
let span = decl.span.span();
Expand All @@ -2589,6 +2655,35 @@ fn build_decls(
});
continue;
}
// A lowercase segment in the middle of a dotted target could be a submodule
// or an object, and only the target environment knows which. Rather than
// guess and emit an import that may not exist, ask for the declaration that
// already settles it (`DESIGN.md` §6).
if decl.receiver.is_none()
&& let Some(seg) = undecidable_extern_segment(&decl.target)
&& !is_declared(&decl.target)
{
let target = decl.target.join(".");
let root = &decl.target[0];
let upto: Vec<&str> = decl
.target
.iter()
.take_while(|s| s.as_str() != seg)
.map(String::as_str)
.collect();
errors.push(TypeError {
message: format!(
"cannot tell which part of `{target}` names the module: `{seg}` is lowercase, \
so it could be a submodule (like `os.path`) or an object (like `sys.stdout`), \
and only the running environment knows which; declare it with \
`extern import {root}` — or `extern import {}` if `{seg}` really is a module",
upto.join(".") + "." + seg
),
span,
});
// Fall through: the extern still registers, so a real import problem
// does not also report every use of the name as unbound.
}
let mut var_map = HashMap::new();
collect_type_vars(&decl.ty, &mut var_map);
// Effect variables (`->{e}`) are an extern-only privilege: collected here
Expand Down
55 changes: 47 additions & 8 deletions tests/compile.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,14 @@ fn unused_extern_imports_nothing() {
#[test]
fn extern_in_submodule_imports_the_submodule() {
// A target inside a submodule must import the submodule, not just the top-level
// package — `import urllib` would leave `urllib.parse` unbound at runtime.
let py = pyfun::compile("extern q: string -> string = urllib.parse.quote\nlet r = q \"a b\"")
.unwrap();
// package — `import urllib` would leave `urllib.parse` unbound at runtime. The
// declaration is what says so; the compiler will not guess (`DESIGN.md` §6).
let py = pyfun::compile(
"extern import urllib.parse\n\
extern q: string -> string = urllib.parse.quote\n\
let r = q \"a b\"",
)
.unwrap();
assert!(py.contains("import urllib.parse"), "{py}");
assert!(py.contains("r = urllib.parse.quote("), "{py}");
}
Expand Down Expand Up @@ -363,15 +368,49 @@ fn extern_import_composes_with_pinned_kwargs() {
}

#[test]
fn import_heuristic_is_unchanged_without_a_declared_import() {
// With no `extern import`, the lowercase-prefix heuristic is untouched: a
// submodule target still imports the submodule, not just the top package.
let py = pyfun::compile("extern get: string -> a = urllib.request.urlopen\nlet r = get \"x\"")
.unwrap();
fn an_undecidable_target_is_rejected_rather_than_guessed() {
// `request` is lowercase, so it could be a submodule (it is) or an object
// (`sys.stdout` is one). The text cannot say which, and the answer belongs to
// the running environment, so the compiler asks instead of emitting an import
// that may not exist (`ROADMAP.md` finding #7).
let err = pyfun::compile("extern get: string -> a = urllib.request.urlopen\nlet r = get \"x\"")
.unwrap_err();
let msg = err.message();
assert!(msg.contains("cannot tell which part"), "{msg}");
assert!(msg.contains("extern import urllib.request"), "{msg}");
}

#[test]
fn a_declared_import_settles_an_undecidable_target() {
// The same target, with the one line the diagnostic asks for.
let py = pyfun::compile(
"extern import urllib.request\n\
extern get: string -> a = urllib.request.urlopen\n\
let r = get \"x\"",
)
.unwrap();
assert!(py.contains("import urllib.request"), "{py}");
assert!(py.contains("r = urllib.request.urlopen(\"x\")"), "{py}");
}

#[test]
fn a_capitalised_segment_needs_no_declaration() {
// PEP 8 settles this one: `Path` is a class, so the module is `pathlib` and
// nothing is in doubt.
let py = pyfun::compile(
"extern type P\nextern read: P -> string = pathlib.Path.read_text\nlet r = read p",
);
let msg = format!("{:?}", py.as_ref().err());
assert!(!msg.contains("cannot tell which part"), "{msg}");
}

#[test]
fn a_top_level_target_needs_no_declaration() {
// Two segments: the module is the first one, and there is nothing to guess.
let py = pyfun::compile("extern pure f: float -> float = math.fabs\nlet r = f 1.0").unwrap();
assert!(py.contains("import math"), "{py}");
}

#[test]
fn unused_extern_import_emits_nothing() {
// A declared module import is hoisted only when a target rooted at it is
Expand Down