many minor improvements to field processing and reporting

Author: mroughan

ARCHITECTURE.md

@@ -26,6 +26,7 @@ important than maximal throughput.
    URL, or file led to the observation.
 6. Handle real-world bibliography messiness without treating every textual
    difference as an error.
+7. Prefer slow and accurate analysis.
 
 ## Non-Goals
 

CHANGELOG.md

@@ -2,6 +2,32 @@
 
 ## Unreleased
 
+### Reports
+- Added a documentation page explaining Markdown report flags, field comparison
+  flags, entry-type-specific required fields, normalization behavior, and fetch
+  manifests.
+- Reworked Markdown reports to reduce redundant checklist text. Each entry now
+  has a compact general-flags table for source discovery, provider errors,
+  required fields, comparison availability, PDF candidates, and confidence.
+  Field-level review flags now appear as a `Flag` column in the field
+  comparison table alongside importance, status, BibTeX value, source value,
+  and diagnostic note.
+- `@inproceedings` and related proceedings/chapter entries now compare the
+  container title as `booktitle`, not `journal`, in the default field
+  comparison set.
+- `@book` and related chapter entries with an `editor` but no `author` now
+  compare the creator field as `editor`, rather than reporting a missing
+  `author`.
+
+### Validation transparency
+- `check_bibliography` now emits a clear `@warn` when it falls back to
+  `CandidateProvider` (no fixture, no explicit `providers`, `use_apis=false`),
+  since that fallback only echoes each entry's own title/doi/url back as its
+  "source" and cannot validate anything against an independent record.
+- `compare_entry` now adds an explicit note to the `EntryReport` itself in
+  that same situation, so the limitation is visible in the generated
+  Markdown/INC report, not only in a log line a reader may not see.
+
 ### Architecture
 - Split monolithic `src/PaperFetch.jl` into focused include files:
   `normalize.jl`, `bib.jl`, `identifiers.jl`, `providers.jl`, `compare.jl`,
@@ -38,6 +64,39 @@
   API responses are written to and read from a local directory to support
   repeat runs without re-querying providers.
 
+### Bug fixes (round 4)
+- Fixed `pubmed_search_ids`: NCBI returns `idlist:[]` (not an omitted field)
+  for any search with zero hits, which is the common case for non-biomedical
+  DOIs. JSON3 parses an empty JSON array with no element-type hint as eltype
+  `Union{}`, so `String.(...)` over it produced a `Vector{Union{}}` instead of
+  `Vector{String}`. That failed to dispatch on
+  `pubmed_summary_records(::ApiProvider, ::Vector{String}; ...)`, so every
+  non-biomedical lookup was silently converted into a spurious
+  `pubmed-error`/`pubmed-search-error` source carrying a `MethodError`
+  instead of cleanly reporting "no PubMed hit". Switched to an explicitly
+  `String[...]`-typed comprehension so the empty case stays `Vector{String}`.
+- Fixed misleading "year" reason in `compare_entry`'s discarded-candidate
+  notes: the diagnostic listed "year" as a hard-mismatch reason whenever a
+  publication-year gap existed at all, even a gap of 0 or 1 that does not
+  meet the actual hard-mismatch threshold (`>=2`, or `>=3` for books). Added
+  `year_hard_mismatch` and reused it both in `source_hard_mismatch` and in the
+  note-building code so the explanation always matches the real cutoff.
+
+### Bug fixes (round 3)
+- Fixed `sources_for(::ApiProvider, ...)`: an identifier such as a DOI that
+  resolved successfully but pointed at the wrong work (mistyped or swapped
+  DOI) previously suppressed the title/author search fallback, because the
+  fallback only ran when *no* usable source was found at all. Added
+  `identifier_source_conflicts` (reusing the same hard-mismatch comparison as
+  `compare_entry`) so the fallback also runs when every identifier-resolved
+  source hard-mismatches the entry's title or author, letting the package find
+  the correct work under a different DOI.
+- Improved `compare_entry` diagnostics: discarded hard-mismatch candidates are
+  now reported in `EntryReport.notes` even when a reliable replacement source
+  is found, and an explicit note is added when the chosen best source was
+  found via title/author search under a DOI different from the one in the
+  bibliography, so the inconsistency stays visible to the reviewer.
+
 ### Bug fixes (round 2)
 - Fixed `openlibrary_isbn_records`: author objects from the ISBN endpoint only
   carry a key path (`/authors/OL1A`), not an inline name. Authors are now only

README.md

@@ -30,10 +30,14 @@ or separate AI-assisted editing task can improve the bibliography deliberately.
   page ranges, Unicode/LaTeX accents, DOI URL variants, author initials, and
   similar harmless differences. URL paths and queries keep their case, and
   reordered author lists are flagged for review rather than silently accepted.
+- Uses entry-type-aware comparisons: proceedings and chapter entries compare
+  `booktitle`, and edited books can use `editor` instead of `author`.
+- Matches URLs found in `url`, `note`, or `howpublished`, including LaTeX
+  `\url{...}` forms, when checking URL-backed references.
 - Writes Markdown and INC reports; INC is a spreadsheet-friendly CSV-like format
   handled by IncCSV.jl.
-- Optionally downloads PDFs from explicit PDF candidate URLs and writes a fetch
-  manifest.
+- Optionally downloads PDFs from explicit PDF candidate URLs and writes fetch
+  manifests.
 
 ## What PaperFetch Does Not Do
 
@@ -82,6 +86,11 @@ The Markdown report is meant for direct reading. The INC report is meant for
 spreadsheets and downstream tooling. CLI report names default to the input file
 stem; pass `--report-basename NAME` to choose a different basename.
 
+Each Markdown entry keeps the original BibTeX key, then shows general flags for
+source discovery, provider errors, required fields, PDF candidates, and
+confidence. Field-by-field comparisons include a `Flag` column so green, amber,
+red, and ignored review signals are visible next to the relevant value.
+
 ## Live API Checks
 
 Live provider lookup is opt-in:
@@ -120,11 +129,17 @@ Outputs include:
 
 - `paperfetch_out/references.md`
 - `paperfetch_out/references.inc`
+- `paperfetch_out/manifest.md`
 - `paperfetch_out/manifest.inc`
 - downloaded `*.pdf` files when candidate URLs are available and reachable
 
-Entries without PDF candidates are recorded as `skipped`, not as validation
-failures.
+`manifest.md` is the human-readable fetch table. `manifest.inc` is the
+spreadsheet/tooling manifest. Entries without PDF candidates are recorded as
+`skipped`, not as validation failures.
+
+The manifest records the reference key, compact title, fetch status, local file
+when downloaded, source URL, and a short diagnostic such as "no PDF candidate",
+"downloaded from ...", or a failed HTTP/content-type reason.
 
 ## Credential-Assisted Fetching
 
@@ -173,6 +188,8 @@ paths[:inc]
 results, manifest = fetch_pdfs(reports, "paperfetch_out")
 ```
 
+This writes both `paperfetch_out/manifest.md` and `paperfetch_out/manifest.inc`.
+
 For deterministic offline runs, pass a fixture instead of live APIs:
 
 ```julia
@@ -210,7 +227,8 @@ julia --project=. test/online/runtests.jl
 ## Documentation
 
 The documentation includes a quickstart, examples, API reference, and notes on
-live providers, caching, rate limiting, and building a stand-alone executable:
+live providers, report formats, fetch manifests, caching, rate limiting, and
+building a stand-alone executable:
 
 https://mroughan.github.io/PaperFetch.jl/dev
 

docs/make.jl

@@ -20,6 +20,7 @@ makedocs(;
         "Home" => "index.md",
         "Examples" => "examples.md",
         "Live Providers" => "providers.md",
+        "Reports and Manifests" => "reports.md",
         "Stand-Alone Executable" => "standalone.md",
         "API Reference" => "api.md",
     ],

docs/src/examples.md

@@ -54,6 +54,15 @@ Not every bibliography item is a journal article or has a PDF:
 PaperFetch.jl records the absence of a PDF candidate as a fetch status, not as
 a bibliography error.
 
+Book-like entries use book-specific fields during comparison. `@inbook` and
+`@incollection` compare container metadata as `booktitle`, and `@book`,
+`@inbook`, and `@incollection` entries can use `editor` as the creator field
+when no `author` field is present.
+
+URL-backed examples may place URLs in either `url` or common prose fields such
+as `note` and `howpublished`. LaTeX `\url{...}` macros are normalized before
+comparison.
+
 ## Plain DOI Lists
 
 Plain text input is also accepted:
@@ -77,6 +86,7 @@ julia --project=. -e 'using PaperFetch; PaperFetch.main()' -- \
   --email your.email@example.edu \
   --use-apis \
   --cache-dir .paperfetch_cache \
+  --rate-limit-seconds 0.05 \
   --outdir paperfetch_out
 ```
 
@@ -102,3 +112,8 @@ julia --project=. test/online/runtests.jl
 The online runner checks that each example returns at least one non-error
 source, has an exact normalized DOI match, writes reports, and finds at least
 one PDF candidate across the set.
+
+The generated Markdown report uses entry-level general flags plus field-level
+flags in the comparison table. Fetch mode also writes `manifest.md` and
+`manifest.inc`, which are the best first place to inspect why a PDF did or did
+not download.

docs/src/index.md

@@ -37,6 +37,11 @@ table with metadata, suitable for spreadsheets and downstream tooling. CLI
 report names default to the input file stem; pass `--report-basename NAME` to
 choose a different basename.
 
+Markdown reports preserve the original BibTeX keys. Each entry has a compact
+general-flags table for source discovery, provider errors, required fields, PDF
+candidates, and confidence, followed by a field comparison table with a `Flag`
+column for green, amber, red, or ignored review signals.
+
 ## Julia API Tutorial
 
 ```julia
@@ -65,15 +70,20 @@ Fetch mode uses explicit PDF candidate URLs from source metadata:
 results, manifest = fetch_pdfs(reports, "paperfetch_out")
 ```
 
-Entries without PDF candidates are recorded as `skipped`, not as validation
-failures.
+This writes `manifest.md`, a human-readable table of references and fetch
+diagnostics, plus `manifest.inc` for spreadsheet and tooling use. Entries
+without PDF candidates are recorded as `skipped`, not as validation failures.
 
 ## Design Commitments
 
 - BibTeX input is parsed, never rewritten.
 - DOI and similar identifiers use exact normalized comparison.
 - Titles, authors, pages, journal names, and other bibliographic text are
   compared with documented normalization.
+- Edited books may use `editor` instead of `author`; proceedings and chapter
+  entries compare their container as `booktitle`.
+- URLs found in `note` or `howpublished`, including LaTeX `\url{...}` macros,
+  can be matched to source URL metadata.
 - URL normalization preserves path and query case while canonicalizing DOI
   resolver links and HTTP(S) hosts.
 - Author order is treated as meaningful. Reordered author lists are marked for

docs/src/providers.md

@@ -40,18 +40,28 @@ PaperFetch.jl chooses lookup routes from the identifiers and entry type it can
 see:
 
 - DOI-backed entries can use Crossref, OpenAlex, Unpaywall, DataCite, Semantic
-  Scholar, PubMed, CORE, and Figshare.
+  Scholar, PubMed, CORE, and Figshare. DOI strings can be recovered from the
+  `doi` field, DOI resolver URLs, and common misplaced fields such as `note`,
+  `url`, and `howpublished`.
 - arXiv identifiers use the arXiv API.
 - PMID values use PubMed.
 - ISBN-backed books can use Open Library and Google Books.
-- Entries with only a title can use title-and-author search.
+- Entries with only a title, or entries whose identifier resolves to an
+  obviously different work, can use title-and-author fallback search.
 - URL-backed entries can check the URL and read common citation metadata from
-  the landing page.
+  the landing page. URLs can be taken from `url`, `note`, or `howpublished`,
+  including LaTeX `\url{...}` macros.
 
 Provider results are candidates, not automatic truth. The report records
 agreement, conflicts, missing fields, and provider errors so a person can decide
 what should be fixed in the original BibTeX.
 
+Book-like entries are handled with their expected fields. Proceedings and
+chapter entries compare `booktitle` rather than `journal`; edited books can
+compare `editor` as the creator when no `author` field is present. A large year
+gap for a book is treated as evidence that a provider may have returned a
+different edition.
+
 ## Caching And Rate Limiting
 
 `--cache-dir` stores API and landing-page responses by request URL and headers.

docs/src/reports.md

@@ -0,0 +1,115 @@
+# Reports and Manifests
+
+PaperFetch.jl reports are the main output of a check. They are designed to
+support review, not automatic editing: the input `.bib` file is never rewritten.
+
+## Check Reports
+
+`check` mode writes two files:
+
+- a Markdown report for direct human review;
+- an INC report for spreadsheets and downstream tooling.
+
+From the command line, report names default to the input file stem:
+
+```bash
+julia --project=. -e 'using PaperFetch; PaperFetch.main()' -- \
+  check examples/01_exact_article.bib \
+  --fixture examples/metadata_fixture.json \
+  --outdir paperfetch_out
+```
+
+This writes `paperfetch_out/01_exact_article.md` and
+`paperfetch_out/01_exact_article.inc`. Use `--report-basename NAME` to choose a
+different basename.
+
+Direct Julia API calls use `paperfetch_report` unless `basename` is supplied:
+
+```julia
+paths = write_reports(reports, "paperfetch_out"; basename="references")
+```
+
+## Markdown Report Layout
+
+Each entry section starts with the original BibTeX key. Keys are preserved as
+written, including underscores and punctuation.
+
+Each entry then has a general-flags table for entry-level review signals:
+
+- whether source metadata was found;
+- whether providers returned errors;
+- whether required fields are present;
+- whether any field comparisons were possible;
+- whether PDF candidates were discovered;
+- the confidence score.
+
+Field-level details are shown in a separate comparison table. The `Flag` column
+summarizes each field as green, amber, red, or ignored. This avoids repeating a
+full checklist while keeping the signal close to the field value being reviewed.
+
+## Required And Supplementary Fields
+
+Required fields depend on the BibTeX entry type. For example:
+
+- `@article` expects `author`, `title`, `journal`, and `year`;
+- `@inproceedings` expects `author`, `title`, `booktitle`, and `year`;
+- `@book` accepts either `author` or `editor`, plus `title`, `publisher`, and
+  `year`;
+- `@inbook` and `@incollection` accept either `author` or `editor` and compare
+  their container title as `booktitle`.
+
+Supplementary fields such as `doi`, `url`, `pages`, `volume`, `number`, `isbn`,
+and `edition` still appear in comparisons when present or when source metadata
+reports them, but their absence is usually marked amber rather than red.
+
+Common bibliography-manager fields such as `abstract`, `keywords`, `file`,
+`timestamp`, and similar local metadata are treated as ignored for reference-list
+validation.
+
+## Normalization
+
+Comparison is intentionally tolerant but explicit. Titles are normalized before
+comparison and before title-based search: braces, TeX-style quotes, common LaTeX
+formatting commands, accents, punctuation, case, and whitespace are normalized.
+
+Author and editor lists use the same name-normalization logic. Full names can
+match initials, accents are normalized, and `et al.` is treated as a review flag
+rather than an automatic conflict. Reordered creator lists are marked ambiguous
+because author order is often meaningful.
+
+DOIs are stricter. Bare DOI strings, `doi:` prefixes, `doi.org` URLs, and
+`dx.doi.org` URLs are canonicalized to the same DOI, but a different DOI remains
+a conflict.
+
+URLs are compared after canonicalizing hosts and DOI resolver links. URLs found
+inside `note` or `howpublished`, including LaTeX `\url{...}` macros, can be used
+as a fallback for a missing `url` field.
+
+## Fetch Manifests
+
+`fetch` mode first performs the same check workflow and then attempts only
+explicit PDF candidate URLs from source metadata:
+
+```bash
+julia --project=. -e 'using PaperFetch; PaperFetch.main()' -- \
+  fetch references.bib \
+  --email your.email@example.edu \
+  --use-apis \
+  --cache-dir .paperfetch_cache \
+  --outdir paperfetch_out
+```
+
+Fetch mode writes:
+
+- the normal Markdown and INC check reports;
+- `manifest.md`, a human-readable table of fetch outcomes;
+- `manifest.inc`, a spreadsheet/tooling manifest;
+- downloaded PDF files when a candidate URL succeeds.
+
+The manifest records the BibTeX key, a compact reference title, fetch status,
+local file path, source URL, and a short diagnostic. Entries with no PDF
+candidate are recorded as `skipped`; this is normal for websites, datasets,
+books, landing-page-only records, and many paywalled articles.
+
+Failed PDF candidates include diagnostics such as HTTP status, non-PDF content
+types, likely landing/login/paywall pages, or missing remote files.