feat(html): extract HTML meta, Dublin Core and Office tags - #406
Merged
Conversation
swackhamer
enabled auto-merge (squash)
August 2, 2026 03:47
HTML was detected but had no parser and no FileFormat variant, so every .html file fell through to the plain-text fallback and was parsed as TXT: five TEXT:* statistics ExifTool never reports for an HTML file, and none of the 57 tags it does. combined-samples/HTML.html scored MATCHED 0, MISSING 57, EXTRA 5. This adds src/parsers/text/html.rs, a transcription of ExifTool 13.55's HTML.pm (ProcessHTML), plus FileFormat::HTML, its dispatch arm and a detection gate. The sample now scores MATCHED 57, MISSING 0, EXTRA 0. Across the 207-file corpus it is the only file whose score changed, and no file lost a matched tag. One file yields five family-1 groups, because a meta name carries its own namespace and ExifTool routes each to a different tag table: HTML:Title, HTTP-equiv:ContentType, HTML-dc:*, HTML-ncc:*, HTML-prod:* and HTML-office:* (the MS-Office XML island inside the mso conditional comment). The details that make values match rather than merely appear: - dc:creator is a Seq, so two META elements collapse into one list, while dc:language and dc:subject are also List but hold one value each and stay scalars -- which is what ExifTool emits. - iso-8859-1 maps to ExifTool's Latin charset, which is cp1252 and not bare ISO 8859-1. Values seen before the content-type element are not recoded; HTML.pm says so explicitly and this reproduces it rather than fixing it. - The meta path unescapes with the full 253-entity HTML table, the XML island with only the five XML entities plus numeric references, so HTML-office:Description keeps its as a literal CR. - Conversions are per tag, not per kind: HTML-office:CreateDate goes through ConvertXMPDate to 2010:06:28 23:52:00Z while HTML-dc:Date has no ValueConv and stays raw as the malformed 2007-30-01. - A tag the table does not declare is named by the module's own mangling and then by AddTagToTable's normalisation. Without that second step an unknown namespace reported unknownNsWeird_thing where ExifTool reports UnknownNsWeird_thing. - With no declared charset ExifTool does not recode, so a stray high byte reaches its writer raw and FixUTF8 turns each bad byte into one '?'. Rust's lossy conversion collapses a whole invalid run into a single U+FFFD, which is a different string. The tables are not hand-invented. All seven (112 tags) were diffed key by key against Image::ExifTool::GetTagTable on tag ID, Name and List flag; the 253-entry entity table was diffed against a Perl eval of %entityNum; and ConvertTimeSpan was compared with the Perl for every integer TotalTime from 1 to 200000. ConvertTimeSpan's sub-minute branch would need Perl's %.15g number stringification for a fractional value, so that one case omits the tag rather than print an approximation under a real tag name. Detection uses ProcessHTML's own gate -- an XML declaration alone is not enough, an HTML element must appear in the first 256 bytes -- and runs after the existing SVG and XML-plist root checks, so all four <?xml-rooted formats keep their own detection. test_xml_rooted_formats_keep_their_own_detection pins that ordering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
swackhamer
force-pushed
the
feat/html-meta-tags
branch
from
August 2, 2026 09:07
c0f91cb to
57b07f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTML was already detected —
File:FileTypesaidHTML— but had no parser andno
FileFormatvariant, so every.html/.htmfile fell through to theplain-text fallback and was parsed as TXT. That produced five
TEXT:*statistics ExifTool never reports for an HTML file, and none of the 57 tags it
does.
This adds
src/parsers/text/html.rs, a transcription of ExifTool 13.55'sHTML.pm(ProcessHTML), plus theFileFormat::HTMLvariant, its dispatch armand a detection gate.
Measured
combined-samples/HTML.html, per-fileGroup1:Namescoring,File/System/ExifToolgroups excluded, ExifTool JSON parsed withparse_float=str:The five
EXTRAwereTEXT:Encoding,TEXT:FileSize,TEXT:LineCount,TEXT:LineEndingandTEXT:WordCount— gone now that HTML is no longer treatedas plain text.
Run over the whole 207-file corpus with the same script and the same branch
built with and without the change, the delta is exactly the HTML file:
MATCHED +57, MISSING −57, EXTRA −5.
HTML.htmlis the only file whose scorechanged, and no file lost a matched tag. (One unrelated file,
XISF.xisf,drifted by two in ExifTool's own tag count between the two runs; OxiDex matched
zero of its tags both times, so it is not affected either way.)
What the parser does
Five family-1 groups come out of one file, because a meta name carries its own
namespace and ExifTool routes each to a different tag table:
HTML:Title(fromthe
TITLEelement),HTTP-equiv:ContentType,HTML-dc:*(9),HTML-ncc:*(20),
HTML-prod:*(2) andHTML-office:*(24, from the MS-Office XML islandinside the
<!--[if gte mso 9]>conditional comment).The details that make values match rather than merely appear:
HTML-dc:Creatoris a list. Two<meta name="dc:creator">elementscollapse into
["Phil Harvey","Another Creator"], whiledc:language,dc:publisheranddc:subjectare alsoListbut hold one value each and sostay scalars — which is what ExifTool emits.
iso-8859-1; ExifTool maps that to itsLatincharset, which is cp1252, not bare ISO 8859-1.
HTML-office:Categoryisa catégoryfrom a raw0xE9. Values seen before the content-type elementare not recoded —
HTML.pmsays so explicitly and this reproduces it ratherthan "fixing" it.
table (
α→HTML-dc:Subject=Greek: α β γ); the XML-island pathuses only the five XML entities plus numeric references, so
HTML-office:Descriptionkeeps its as a literal CR.HTML-office:CreateDategoesthrough
ConvertXMPDateto2010:06:28 23:52:00Z, whileHTML-dc:Datehasno
ValueConvand stays raw as the malformed2007-30-01.HTML-office:TotalEditTimeisConvertTimeSpan($val, 60)→1 minute.<o:Checked_x0020_by>is not in theOffice table, so
HTML.pmmangles the name (hex escape → space →capitalise) and then
AddTagToTablenormalises it (strip illegal characters,ucfirst,Tagprefix when it is under two characters or does not start witha letter). Missing that second step reported
unknownNsWeird_thingwhereExifTool reports
UnknownNsWeird_thing.o:Revisionando:Versionboth map toRevisionNumberand neither is alist, so the later element wins (
12.0), matching ExifTool.ExifTool's writer raw and its
FixUTF8turns each bad byte into one?.Rust's lossy conversion collapses a whole invalid run into a single U+FFFD,
which is a different string, so
FixUTF8is reproduced instead.Verified against the Perl, not by hand
Three checks against ExifTool's own source, because a hand-invented table that
ships with tests asserting its own contents is the failure mode here:
against
Image::ExifTool::GetTagTableforHTML::{Main,dc,ncc,prod,vw96, equiv,Office}on tag ID,NameandListflag. Identical.%entityNum, and theshipped Rust table was then re-parsed and diffed against a Perl
evalof thatblock. Identical.
ConvertTimeSpan. Compared against the Perl for every integero:TotalTimefrom 1 to 200000 — covering the seconds, minutes, hours and daysbranches. Byte-identical output, which settles
%dtruncation and%.1frounding.
Nothing omitted
All 57 tags reproduce ExifTool's exact values, so nothing was dropped as
un-reproducible. One guard is in place for a case this file does not exercise:
ConvertTimeSpan's sub-minute branch would need Perl's%.15gnumberstringification for a fractional
o:TotalTime, and rather than print anapproximation under a real tag name the tag is omitted (AGENTS.md).
Detection, and why nothing else moved
The gate is ExifTool's own, from
ProcessHTML: an XML declaration alone is notenough — an actual
HTMLelement must appear in the first 256 bytes. It runsafter the existing SVG and XML-plist root checks, so all four
<?xml-rootedformats keep their own detection.
test_xml_rooted_formats_keep_their_own_detectionpins that ordering: HTML,SVG, Plist, XMP and a plain RDF/XML document (which must stay TXT) each assert
their own
FileFormat.Verified unchanged on the corpus:
XMP.svg→ SVG,XMP.xmp→ XMP,PLIST-xml.plist→ PLIST,XMP.xml/Geotag.xml/LNK.url→ TXT,Text1..5.txt→ TXT,LNK.lnk→ LNK,VCard.ics→ ICS.Beyond the corpus sample, nine hand-built probes were compared tag for tag with
ExifTool and all match with zero value differences: a single-line document,
CR-only line endings, a UTF-8 charset with a title spanning lines, an
undeclared charset with a raw high byte, an Office island with attributes and a
self-closing element, a document with no
<head>at all, an HTML-4 doctype witha three-value
dc.subjectlist using the.namespace separator, awindows-1252document exercising the cp1252-only band, and an XHTML file(which correctly still reports
FileType: XHTML).cargo test --workspacepasses,cargo fmt --allandcargo clippyare clean.🤖 Generated with Claude Code