Skip to content

feat(html): extract HTML meta, Dublin Core and Office tags - #406

Merged
swackhamer merged 1 commit into
mainfrom
feat/html-meta-tags
Aug 2, 2026
Merged

feat(html): extract HTML meta, Dublin Core and Office tags#406
swackhamer merged 1 commit into
mainfrom
feat/html-meta-tags

Conversation

@swackhamer

Copy link
Copy Markdown
Collaborator

HTML was already detectedFile:FileType said HTML — but had no parser and
no FileFormat variant, so every .html/.htm file fell through to the
plain-text fallback and was parsed as TXT. That produced five TEXT:*
statistics ExifTool never reports for an HTML file, and none of the 57 tags it
does.

This adds src/parsers/text/html.rs, a transcription of ExifTool 13.55's
HTML.pm (ProcessHTML), plus the FileFormat::HTML variant, its dispatch arm
and a detection gate.

Measured

combined-samples/HTML.html, per-file Group1:Name scoring, File/System/
ExifTool groups excluded, ExifTool JSON parsed with parse_float=str:

MATCHED VALUE-DIFF MISSING EXTRA
before 0 0 57 5
after 57 0 0 0

The five EXTRA were TEXT:Encoding, TEXT:FileSize, TEXT:LineCount,
TEXT:LineEnding and TEXT:WordCount — gone now that HTML is no longer treated
as plain text.

Run over the whole 207-file corpus with the same script and the same branch
built with and without the change, the delta is exactly the HTML file:
MATCHED +57, MISSING −57, EXTRA −5. HTML.html is the only file whose score
changed, and no file lost a matched tag.
(One unrelated file, XISF.xisf,
drifted by two in ExifTool's own tag count between the two runs; OxiDex matched
zero of its tags both times, so it is not affected either way.)

What the parser does

Five family-1 groups come out of one file, because a meta name carries its own
namespace and ExifTool routes each to a different tag table: HTML:Title (from
the TITLE element), HTTP-equiv:ContentType, HTML-dc:* (9), HTML-ncc:*
(20), HTML-prod:* (2) and HTML-office:* (24, from the MS-Office XML island
inside the <!--[if gte mso 9]> conditional comment).

The details that make values match rather than merely appear:

  • HTML-dc:Creator is a list. Two <meta name="dc:creator"> elements
    collapse into ["Phil Harvey","Another Creator"], while dc:language,
    dc:publisher and dc:subject are also List but hold one value each and so
    stay scalars — which is what ExifTool emits.
  • Charset. The file declares iso-8859-1; ExifTool maps that to its Latin
    charset, which is cp1252, not bare ISO 8859-1. HTML-office:Category is
    a catégory from a raw 0xE9. Values seen before the content-type element
    are not recoded — HTML.pm says so explicitly and this reproduces it rather
    than "fixing" it.
  • Two different unescapers. The meta path uses the full 253-entity HTML
    table (&alpha;HTML-dc:Subject = Greek: α β γ); the XML-island path
    uses only the five XML entities plus numeric references, so
    HTML-office:Description keeps its &#13; as a literal CR.
  • Conversions are per-tag, not per-kind. HTML-office:CreateDate goes
    through ConvertXMPDate to 2010:06:28 23:52:00Z, while HTML-dc:Date has
    no ValueConv and stays raw as the malformed 2007-30-01.
    HTML-office:TotalEditTime is ConvertTimeSpan($val, 60)1 minute.
  • Derived names go through two steps. <o:Checked_x0020_by> is not in the
    Office table, so HTML.pm mangles the name (hex escape → space →
    capitalise) and then AddTagToTable normalises it (strip illegal characters,
    ucfirst, Tag prefix when it is under two characters or does not start with
    a letter). Missing that second step reported unknownNsWeird_thing where
    ExifTool reports UnknownNsWeird_thing.
  • o:Revision and o:Version both map to RevisionNumber and neither is a
    list, so the later element wins (12.0), matching ExifTool.
  • No declared charset means no recoding, so a stray high byte reaches
    ExifTool's writer raw and its FixUTF8 turns each bad byte into one ?.
    Rust's lossy conversion collapses a whole invalid run into a single U+FFFD,
    which is a different string, so FixUTF8 is reproduced instead.

Verified against the Perl, not by hand

Three checks against ExifTool's own source, because a hand-invented table that
ships with tests asserting its own contents is the failure mode here:

  • Tag tables. All seven shipped tables (112 tags) were diffed key by key
    against Image::ExifTool::GetTagTable for HTML::{Main,dc,ncc,prod,vw96, equiv,Office} on tag ID, Name and List flag. Identical.
  • Entity table. The 253 entries were extracted from %entityNum, and the
    shipped Rust table was then re-parsed and diffed against a Perl eval of that
    block. Identical.
  • ConvertTimeSpan. Compared against the Perl for every integer
    o:TotalTime from 1 to 200000 — covering the seconds, minutes, hours and days
    branches. Byte-identical output, which settles %d truncation and %.1f
    rounding.

Nothing omitted

All 57 tags reproduce ExifTool's exact values, so nothing was dropped as
un-reproducible. One guard is in place for a case this file does not exercise:
ConvertTimeSpan's sub-minute branch would need Perl's %.15g number
stringification for a fractional o:TotalTime, and rather than print an
approximation under a real tag name the tag is omitted (AGENTS.md).

Detection, and why nothing else moved

The gate is ExifTool's own, from ProcessHTML: an XML declaration alone is not
enough — an actual HTML element must appear in the first 256 bytes. It runs
after the existing SVG and XML-plist root checks, so all four <?xml-rooted
formats keep their own detection.
test_xml_rooted_formats_keep_their_own_detection pins that ordering: HTML,
SVG, Plist, XMP and a plain RDF/XML document (which must stay TXT) each assert
their own FileFormat.

Verified unchanged on the corpus: XMP.svg → SVG, XMP.xmp → XMP,
PLIST-xml.plist → PLIST, XMP.xml / Geotag.xml / LNK.url → TXT,
Text1..5.txt → TXT, LNK.lnk → LNK, VCard.ics → ICS.

Beyond the corpus sample, nine hand-built probes were compared tag for tag with
ExifTool and all match with zero value differences: a single-line document,
CR-only line endings, a UTF-8 charset with a title spanning lines, an
undeclared charset with a raw high byte, an Office island with attributes and a
self-closing element, a document with no <head> at all, an HTML-4 doctype with
a three-value dc.subject list using the . namespace separator, a
windows-1252 document exercising the cp1252-only band, and an XHTML file
(which correctly still reports FileType: XHTML).

cargo test --workspace passes, cargo fmt --all and cargo clippy are clean.

🤖 Generated with Claude Code

HTML was detected but had no parser and no FileFormat variant, so every
.html file fell through to the plain-text fallback and was parsed as TXT:
five TEXT:* statistics ExifTool never reports for an HTML file, and none of
the 57 tags it does. combined-samples/HTML.html scored MATCHED 0, MISSING 57,
EXTRA 5.

This adds src/parsers/text/html.rs, a transcription of ExifTool 13.55's
HTML.pm (ProcessHTML), plus FileFormat::HTML, its dispatch arm and a
detection gate. The sample now scores MATCHED 57, MISSING 0, EXTRA 0. Across
the 207-file corpus it is the only file whose score changed, and no file lost
a matched tag.

One file yields five family-1 groups, because a meta name carries its own
namespace and ExifTool routes each to a different tag table: HTML:Title,
HTTP-equiv:ContentType, HTML-dc:*, HTML-ncc:*, HTML-prod:* and HTML-office:*
(the MS-Office XML island inside the mso conditional comment).

The details that make values match rather than merely appear:

- dc:creator is a Seq, so two META elements collapse into one list, while
  dc:language and dc:subject are also List but hold one value each and stay
  scalars -- which is what ExifTool emits.
- iso-8859-1 maps to ExifTool's Latin charset, which is cp1252 and not bare
  ISO 8859-1. Values seen before the content-type element are not recoded;
  HTML.pm says so explicitly and this reproduces it rather than fixing it.
- The meta path unescapes with the full 253-entity HTML table, the XML island
  with only the five XML entities plus numeric references, so
  HTML-office:Description keeps its &#13; as a literal CR.
- Conversions are per tag, not per kind: HTML-office:CreateDate goes through
  ConvertXMPDate to 2010:06:28 23:52:00Z while HTML-dc:Date has no ValueConv
  and stays raw as the malformed 2007-30-01.
- A tag the table does not declare is named by the module's own mangling and
  then by AddTagToTable's normalisation. Without that second step an unknown
  namespace reported unknownNsWeird_thing where ExifTool reports
  UnknownNsWeird_thing.
- With no declared charset ExifTool does not recode, so a stray high byte
  reaches its writer raw and FixUTF8 turns each bad byte into one '?'. Rust's
  lossy conversion collapses a whole invalid run into a single U+FFFD, which
  is a different string.

The tables are not hand-invented. All seven (112 tags) were diffed key by key
against Image::ExifTool::GetTagTable on tag ID, Name and List flag; the
253-entry entity table was diffed against a Perl eval of %entityNum; and
ConvertTimeSpan was compared with the Perl for every integer TotalTime from 1
to 200000. ConvertTimeSpan's sub-minute branch would need Perl's %.15g number
stringification for a fractional value, so that one case omits the tag rather
than print an approximation under a real tag name.

Detection uses ProcessHTML's own gate -- an XML declaration alone is not
enough, an HTML element must appear in the first 256 bytes -- and runs after
the existing SVG and XML-plist root checks, so all four <?xml-rooted formats
keep their own detection. test_xml_rooted_formats_keep_their_own_detection
pins that ordering.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@swackhamer
swackhamer force-pushed the feat/html-meta-tags branch from c0f91cb to 57b07f4 Compare August 2, 2026 09:07
@swackhamer
swackhamer merged commit 58a15fe into main Aug 2, 2026
6 of 7 checks passed
@swackhamer
swackhamer deleted the feat/html-meta-tags branch August 2, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant