Convert lib/PDF/Reuse.pm from latin-1 to UTF-8 (#27) - #34
Conversation
Ref #27 ## Reasoning This is the only line in the file whose non-ASCII bytes reach the output. The binary comment after the %PDF-1.4 header is four latin-1 bytes written literally into the source; every other non-ASCII byte in the file is inside a Swedish comment and cannot affect behaviour. Writing them as \xe2\xe3\xcf\xd3 makes the line pure ASCII while emitting the same four bytes, which is what allows the remaining 87 bytes to be transcoded in a following commit without any functional risk. Doing it in two commits rather than one keeps the part that could change output down to a single line with a byte-exact test on it. Verified equal before editing: both spellings unpack to 255044462d312e340a25e2e3cfd30a. The original was already backslash-escaping each high byte, which in a Perl double-quoted string is just the byte itself, so this is a change of spelling and not of value. ## Verification 40 tests pass. t/Reuse.t compares generated PDF output byte-for-byte against its __DATA__ block and is unchanged -- no expected-data edits, which is the condition the issue set for this being provably safe. Generated header inspected directly: 25 50 44 46 2d 31 2e 34 0a 25 e2 e3 cf d3, identical to before. Non-ASCII byte count in the file drops 91 -> 87, exactly the four marker bytes.
Closes #27 ## Reasoning With the header marker written as escapes in the previous commit, every remaining non-ASCII byte in the file was inside a Swedish comment -- verified, not assumed: zero non-ASCII survives outside a comment, and the only high bytes present were 0xd6 0xe4 0xe5 0xf6, the vowels Ö ä å ö. That is what makes this safe as a mechanical re-encode. The file is now valid UTF-8, so editing it through any ordinary text path no longer silently rewrites those bytes as U+FFFD -- which is what happened during the #26 work, adding 8 bytes to the PDF header and shifting every xref offset in the output. Not adding `use utf8;`. It changes how Perl interprets non-ASCII in identifiers and string literals, and there are none: after the escape change the file has non-ASCII only in comments, which the pragma does not affect. Adding it would be a no-op dressed as a precaution. The issue asked for this to be confirmed rather than assumed, so it was. Deliberately not included: lib/PDF/Util/reuseComponent_pl, which is also latin-1. Its single high byte (0xb4, an acute accent) sits inside a string literal passed to prText() rather than in a comment, so transcoding it would change what that utility draws on the page. It is a different problem from this one and wants its own decision -- filed separately. ## Verification The diff is purely an encoding change, and that is checkable rather than asserted: decoding the old bytes as latin-1 and the new bytes as UTF-8 yields identical text. 64 lines re-encoded, +87 bytes, exactly one per high byte. 40 tests pass. t/Reuse.t compares generated PDF output byte-for-byte against its __DATA__ block and needed no edit -- the condition #27 set for this being provably safe. Generated header still emits 25 e2 e3 cf d3. Comments read back correctly as UTF-8 (för, formulär, Övrigt, nollställs) with zero U+FFFD, so the original author's context survived rather than being flattened.
Coverage Report for CI Build 30504108715Warning No base build found for commit Coverage: 37.829%Details
Uncovered Changes
Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
|
CI: 14/14 green — run 30504108715. Checked specifically that the assertion which matters here actually executed rather than trusting the aggregate —
Windows is the one worth naming: an encoding change is exactly the class of thing that could behave differently there, and the generated PDF still matches the — CPAN Agent |
![vsits-codex-review-agent[bot]](https://avatars.githubusercontent.com/u/115404?s=60&v=4){kind=small}
There was a problem hiding this comment.
Codex review: approved.
I mechanically verified the encoding-only claim: fa4367a^:lib/PDF/Reuse.pm decoded as latin-1 matches 3da29cd:lib/PDF/Reuse.pm decoded as UTF-8 except for the isolated PDF marker spelling change. Generated PDFs at the pre-PR source and at 3da29cd have identical first 16 bytes: 25 50 44 46 2d 31 2e 34 0a 25 e2 e3 cf d3 0a 34, so the marker still emits the required binary comment bytes.
I found no non-ASCII outside comments at 3da29cd, so omitting use utf8; is correct for this change and avoids changing Perl source semantics. Adding it only for hypothetical future string literals would be speculative here; a future behavioral edit that introduces non-ASCII code or literals should make that decision in its own change.
I also checked packaging/POD-adjacent paths: Makefile.PL only uses VERSION_FROM and ABSTRACT_FROM, MANIFEST only lists the file, podchecker lib/PDF/Reuse.pm passes, and I found no in-distribution byte-reader disturbed by the comment transcode. The two-commit split is useful rather than ceremonial because it isolates the only output-bearing line before the file-wide transcode.
Verification: PERL5LIB=~/perl5/lib/perl5 perl Makefile.PL && make && make test passes, 40 tests.
vsits-codex-review-agent
Bot
added
reviewed-by-codex-agent
cnighswonger
added
ready-for-merge
2 participants
Two commits, deliberately split so the part that could change output is one line with a byte-exact test on it.
Closes #27
Commit 1 — the header marker as escapes
lib/PDF/Reuse.pmline 318 writes the PDF binary-comment marker as four literal latin-1 bytes. That is the only non-ASCII in the file that reaches the output; everything else is Swedish comments.Verified equal before editing — both spellings unpack to
255044462d312e340a25e2e3cfd30a. The original was already backslash-escaping each high byte, which in a double-quoted Perl string is just the byte, so this changes spelling and not value.Commit 2 — the transcode
With the marker encoding-proof, every remaining high byte was inside a comment. Verified rather than assumed:
0xd6 0xe4 0xe5 0xf6— the vowelsÖ ä å öThe diff is checkably encoding-only: decoding the old bytes as latin-1 and the new bytes as UTF-8 yields identical text. 64 lines re-encoded, +87 bytes, exactly one per high byte.
Why this matters
The file was not valid UTF-8, so editing it through any ordinary text path silently rewrote those 91 bytes as U+FFFD. That happened during the #26 work: it added 8 bytes to the PDF header, shifted every xref offset in the output, and surfaced as a
t/Reuse.tfailure pointing at offsets nowhere near the edited line.use utf8;deliberately not addedIt changes how Perl interprets non-ASCII in identifiers and string literals, and after commit 1 there are none — comments only, which the pragma does not affect. Adding it would be a no-op dressed as a precaution. #27 asked for this to be confirmed rather than assumed, so it was.
Verification
t/Reuse.tcompares generated PDF output byte-for-byte against its__DATA__block and needed no edit — the condition Convert lib/PDF/Reuse.pm from latin-1 to UTF-8 #27 set for this being provably safe.25 50 44 46 2d 31 2e 34 0a 25 e2 e3 cf d3.för,formulär,Övrigt,nollställs) with zero U+FFFD — the original author's context survived rather than being flattened to ASCII.Scope note — a second latin-1 file, filed separately as #33
Auditing the whole distribution turned up
lib/PDF/Util/reuseComponent_pl, which #27 did not mention. It is not the same problem: its single high byte (0xb4, acute accent) sits inside a string literal passed toprText(), not in a comment, so transcoding it would change what that utility draws. Filed as #33 with options rather than folded in here.Every other file in the distribution is already clean UTF-8.
Load-bearing: no. No behaviour change; the file's bytes are re-encoded and one line's spelling changed, with output proven identical.
— CPAN Agent