What happens
While hunting for a TF-target database, the agent downloaded several files whose content didn't match their extension, and didn't notice until much later:
- A
.xlsx saved from a public article's supplementary-data URL was actually an HTML page; it was only caught when the file wouldn't open in Excel.
- A
.tsv.gz fetched from one source was an HTML document (file reported "HTML document text"), so gunzip failed.
- Another
.tsv.gz was an S3 <Error><Code>AccessDenied</Code> XML body saved as if it were gzipped data.
In each case the download "succeeded" (HTTP 200, non-zero file written), so the agent moved on and only discovered the mismatch a few steps later when something downstream broke.
Expected vs actual
- Expected: right after a download, validate the file against what it's supposed to be -- MIME type / magic bytes / a quick
file check -- and warn when a file labeled .xlsx/.gz/.tsv is actually HTML or XML.
- Actual: the file is saved under the requested name with no content check; mislabeled downloads slip through silently.
Suggested direction
A lightweight post-download sanity check would catch the whole class: sniff the first bytes (e.g. <!DOCTYPE/<html/<?xml => not the binary/table the agent asked for; PK magic for real xlsx; gzip magic 1f 8b for .gz) and surface a warning so the agent re-fetches or picks another source instead of building on a bad file. Relates to #282 (prefer server-side fetch-from-URL).
What happens
While hunting for a TF-target database, the agent downloaded several files whose content didn't match their extension, and didn't notice until much later:
.xlsxsaved from a public article's supplementary-data URL was actually an HTML page; it was only caught when the file wouldn't open in Excel..tsv.gzfetched from one source was an HTML document (filereported "HTML document text"), sogunzipfailed..tsv.gzwas an S3<Error><Code>AccessDenied</Code>XML body saved as if it were gzipped data.In each case the download "succeeded" (HTTP 200, non-zero file written), so the agent moved on and only discovered the mismatch a few steps later when something downstream broke.
Expected vs actual
filecheck -- and warn when a file labeled.xlsx/.gz/.tsvis actually HTML or XML.Suggested direction
A lightweight post-download sanity check would catch the whole class: sniff the first bytes (e.g.
<!DOCTYPE/<html/<?xml=> not the binary/table the agent asked for; PK magic for real xlsx; gzip magic1f 8bfor.gz) and surface a warning so the agent re-fetches or picks another source instead of building on a bad file. Relates to #282 (prefer server-side fetch-from-URL).