Skip to content

fix: add image link to the discovery queue - #87

Open
SuoweiHu wants to merge 2 commits into
PhialsBasement:mainfrom
SuoweiHu:main
Open

fix: add image link to the discovery queue#87
SuoweiHu wants to merge 2 commits into
PhialsBasement:mainfrom
SuoweiHu:main

Conversation

@SuoweiHu

Copy link
Copy Markdown

Problem

Unless manually provided an image link to crawl in the first place, images almost never shown in the Images tab, despite the crawler extracting image metadata from HTML pages. For mainly two reasons:

  1. extract_links() only queued <a href> URLs — <img src> URLs were collected for the Links tab (collect_all_links()) and for per-page metadata (extract_images()), but were never added to the crawl discovery queue. So image URLs were never actually fetched as standalone resources.
  2. Default include_extensions excluded image types (jpg, jpeg, png, gif, webp, svg, ico, avif, bmp) — Even if image URLs had reached the queue, the _should_crawl_url() extension filter only allowed html, htm, php, asp, aspx, jsp, rejecting all image file extensions.

Changes

File
crawler.py Added jpg, jpeg, png, gif, webp, svg, ico, avif, bmp to the default include_extensions list in _get_default_config()
settings_manager.py Same image extensions added to the default includeExtensions string
settings.js Same image extensions added to the JS-side defaultSettings.includeExtensions
index.html Updated the <input> default value for the Include File Extensions field to include image types
link_manager.py Added a second pass in extract_links that iterates over <img src> tags after the existing <a href> loop. Each image URL is queued via the same should_crawl_callback filter to ensure they fulfil the includeExtension constraint.

Notes

Before/After Change Comparison (crawling the same URL):
2026-07-23T143206-optimized

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant