Detect source charset before parsing (windows-1251 pages lose all non-ASCII text)

[SergeBrick]

The bug

extract_content() passes loadHTML() an explicit <?xml encoding="UTF-8"> declaration and then asks the parsed document what its encoding is:

if (!@$tmpdoc->loadHTML('<?xml encoding="UTF-8">' . $tmp))
    return false;

// this is the worst hack yet :(
if (strtolower($tmpdoc->encoding) != 'utf-8') {

$tmpdoc->encoding answers utf-8 because of that declaration, whatever the page really is, so the conversion below it is unreachable.

For a windows-1251 page the cp1251 bytes are then parsed as UTF-8 and every invalid sequence is dropped. The extracted article loses all of its Cyrillic and keeps Latin words, digits and punctuation — and that residue is non-empty, so process_article()'s check

$content_test = trim(strip_tags(Sanitizer::sanitize($extracted_content)));
if ($content_test) { ... $article["content"] = $extracted_content; }

accepts it and replaces the feed's own excerpt with it. The failure is invisible to any check based on length; a reader just sees an article of punctuation.

Reproduction

https://infostart.ru/1c/articles/2757168/ — Content-Type: text/html; charset=windows-1251.

	extract_content() length	chars in [\x{0400}-\x{04FF}]
before	8741	0
after	17618	8877

The broken decode was also halving the volume, not only destroying the script.

The change

Two commits, independent — the second is a policy change and can be dropped if you disagree with it.

1. Charset detection. Taken from the response Content-Type (UrlHelper::$fetch_last_content_type), falling back to the markup's own <meta charset>, and applied before parsing. Unknown charsets are left untouched, because mb_convert_encoding() raises ValueError on them in PHP 8. The now-dead $tmpdoc->encoding branch is removed.
2. Page size ceiling 500 KB → 1.5 MB, counted in bytes. The old ceiling rejects ordinary article pages: four of six sampled infostart.ru articles weigh 528–668 KB of markup and never reach Readability. Measured on PHP 8.5, one article at a time: a 668 KB page peaks at 8.0 MiB against memory_limit = 256M and takes 1.57 s wall, mostly fetch. mb_strlen() was also being called before any transcoding, on a payload of unknown encoding, where its result is not meaningful.

Regression check

Fourteen UTF-8 pages across seven sites — apnic.net, ipng.ch, saotn.org, williamlam.com, woshub.com, internet-lab.ru, blog.ipspace.net — return byte-identical extracted text before and after, with no exceptions raised. Six infostart.ru pages (windows-1251, four of them formerly over the ceiling) now extract with Cyrillic intact, 6141–21059 Cyrillic characters each.

Both changes have been running on a live tt-rss instance (PHP 8.5.7, supahgreg/tt-rss:latest) against 19 feeds since 2026-08-10.

🤖 Generated with Claude Code