perf: cut allocations 2x in DOM traversal for ~1.6x faster conversion

[mscrivo]

Summary

Profiling Html2Text.convert (stackprof, wall + object modes) against this repo's own huge-msoffice.html fixture (1.4MB) showed:

• GC was 41% of wall time, driven by 1.37M object allocations for a single convert
• 573K allocations (42%) came from String#downcase and 375K (27%) from Nokogiri::XML::Node#node_name — node.name.downcase was being recomputed ~19 times per node across iterate_over, prefix_whitespace, suffix_whitespace and next_node_name
• another ~120K came from the per-node Array → map → compact → join in iterate_over, which also re-copies each subtree's text at every tree level

Changes

All behavior-preserving — output is byte-identical on every fixture in spec/examples/ (verified by a side-by-side harness during development, and the full suite passes unchanged):

• iterate_over computes node.name.downcase once per node and reuses it for all checks
• children are appended into a single mutable string instead of building an array and joining per node
• suffix_whitespace calls next_node_name once instead of twice for br/div
• merged the duplicate h1-h6/p case branches, removing the Lint/DuplicateBranch disables
• dropped no-op /im regex flags and simplified /\n\n\n*/ to /\n{3,}/

Method signatures and return types are deliberately unchanged, so subclasses that override prefix_whitespace, suffix_whitespace, iterate_over, wrap_link or image_text (supported since #30) are unaffected.

Benchmarks

Ruby 4.0.5, arm64 macOS, fixtures from spec/examples/, with and without YJIT (50-iteration warmup so YJIT compiles the hot paths):

Benchmark	master	this PR	improvement
huge-msoffice (1.4MB) ×5, no JIT	0.893s	0.545s	1.64x
huge-msoffice (1.4MB) ×5, YJIT	0.648s	0.469s	1.38x
full_email (15KB) ×500, no JIT	0.351s	0.280s	1.25x
full_email (15KB) ×500, YJIT	0.317s	0.248s	1.28x
allocations, 1 convert of 1.4MB	1,367,401	642,896	2.1x fewer

YJIT narrows the large-document gap somewhat (it cheapens the redundant method calls) but can't eliminate object allocations — allocation counts are identical with and without YJIT, and GC pressure was the dominant cost, so the win holds up under YJIT. The allocation reduction also lowers GC pressure process-wide for apps converting many documents.

New tests

• spec/subclass_spec.rb — pins the subclass extension points (prefix_whitespace, wrap_link, iterate_over, image_text, and self.convert dispatch), which were previously untested despite being an advertised feature
• spec/class_methods_spec.rb — unit specs for fix_newlines and replace_entities

All new tests were verified to pass against the old implementation as well (run in a clean master worktree), so they pin pre-existing behavior rather than new behavior. 64 examples, 0 failures; rubocop clean.

Not changed (noted for the record)

The div handling in prefix_whitespace compares node.parent.text.strip == node.text.strip, which serializes both subtrees for every nested div — O(n²) in the worst case. Stress tests with quote-chains up to depth 800 stayed fast in absolute terms, so it's left alone here; fixing it would need a structural check with subtle output-semantics risk.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR optimizes Html2Text’s DOM traversal to reduce per-node allocations during conversion, aiming to improve throughput and lower GC pressure on large HTML documents. It also adds targeted specs to pin key class methods and subclass extension points relied on by consumers.

Changes:

• Reduced allocations in iterate_over by caching the downcased node name per node and building child output via a single mutable string instead of array/map/compact/join.
• Minor whitespace/regex simplifications and reduced duplicate calls to next_node_name in suffix_whitespace.
• Added new specs for class methods (fix_newlines, replace_entities) and subclass override/dispatch behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
lib/html2text.rb	Refactors traversal and whitespace handling to reduce allocations and improve conversion performance.
spec/subclass_spec.rb	Adds specs asserting supported subclass override points and self.convert dispatch behavior.
spec/class_methods_spec.rb	Adds unit specs covering fix_newlines and replace_entities.
CHANGELOG.md	Documents the performance work and newly added specs in the Unreleased section.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.