Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
2e9f616
Fix #124: Range with Volumes problem
ckorhonen Jan 7, 2026
831de12
Introduce runner orchestration and version check improvements
hollisticated-horse Oct 20, 2025
c106e6a
Enhance HTTP client with retries and user-agent handling
hollisticated-horse Oct 20, 2025
f927e4a
Refactor MangaDex scraper to use new HTTP helpers
hollisticated-horse Oct 20, 2025
18d721c
Refactor core image pipeline and tighten tests
hollisticated-horse Oct 20, 2025
d1a50f8
Improve logger safety and add formatted helpers
hollisticated-horse Oct 20, 2025
a5a7f30
Add golangci-lint configuration and workflow docs
hollisticated-horse Oct 20, 2025
4d1b798
Normalize obfuscated Comicextra image URLs
hollisticated-horse Oct 20, 2025
68047dc
Add optional HTTP mitigation for protected sources
hollisticated-horse Oct 20, 2025
40c84e2
Replace MangaKakalot network tests with fixtures
hollisticated-horse Oct 20, 2025
d9eea23
Replace site tests with fixture-backed servers
hollisticated-horse Oct 21, 2025
56da40a
feat: add request throttling
hollisticated-horse Oct 21, 2025
48dc7ec
feat(gui): improve download feedback
hollisticated-horse Oct 23, 2025
7021ddb
fix: check close errors
hollisticated-horse Oct 24, 2025
00ea753
fix(readallcomics): improve chapter/issue URL parsing and error handling
hollisticated-horse Feb 3, 2026
619b00b
fix(readcomiconline): enhance debug logging for image scraping
hollisticated-horse Feb 3, 2026
d7a6c61
fix(app): improve version check error handling
ProjectDislocate Mar 6, 2026
a84792f
chore: rename variables for better clarity
ProjectDislocate Mar 6, 2026
0b9a272
feat: validate output format
ProjectDislocate Mar 6, 2026
6cf972d
feat: comicinfo.xml support
ProjectDislocate Mar 6, 2026
35102aa
feat: improve mangadex metadata quality
ProjectDislocate Mar 6, 2026
3f4eb53
Merge remote-tracking branch 'ckorhonen/factory/issue-124' into metad…
ProjectDislocate Mar 6, 2026
3fd623e
fix: missing assert import
ProjectDislocate Mar 6, 2026
8808fa6
feat: customizable request timeout for all providers
ProjectDislocate Mar 6, 2026
a3346b0
feat: add author, lang, and desc to epubs
ProjectDislocate Mar 6, 2026
94f549a
feat: improve mangadex metadata and use locale title for epub
ProjectDislocate Mar 7, 2026
f4e0d0b
chore: cleanup todos
ProjectDislocate Mar 7, 2026
407f032
chore: mark mangadex as working
ProjectDislocate Mar 7, 2026
1e83449
feat: GetInfo now returns an error instead of silently failing
ProjectDislocate Mar 7, 2026
a442704
feat: add mangadex link to weblinks
ProjectDislocate Mar 7, 2026
8d2dbef
feat: log msgs are no longer voided during debug by progress bar
ProjectDislocate Mar 8, 2026
a1849ea
fix: duplicate AgeRating tag in comicinfo.xml
ProjectDislocate Mar 8, 2026
099b15b
fix: loader tests failing
ProjectDislocate Mar 8, 2026
f6165eb
fix: mangadex not returning tags and genres
ProjectDislocate Mar 8, 2026
4333837
feat: set comic format in comicinfo.xml
ProjectDislocate Mar 8, 2026
5d03d9f
feat: set author info from mangadex
ProjectDislocate Mar 8, 2026
369c056
feat: mangadex skip unavailable chapters
ProjectDislocate Mar 8, 2026
3b7f079
feat: ratings & localized titles from mangadex
ProjectDislocate Mar 8, 2026
2126df4
fix: using wrong tag for links
ProjectDislocate Mar 9, 2026
6625cc9
fix: invalid urls being saved
ProjectDislocate Mar 9, 2026
a4598ce
fix: genre output tag
ProjectDislocate Mar 9, 2026
0b41e10
feat: show issue num on progress bar
ProjectDislocate Mar 9, 2026
2edc342
chore: save todo items
ProjectDislocate Mar 9, 2026
b9bcff9
chore: ignore patch files
ProjectDislocate Apr 21, 2026
eab117e
feat: set custom header per request
ProjectDislocate Apr 21, 2026
f216adc
feat: limit file name length
ProjectDislocate Apr 21, 2026
ebdb8fe
fix: uninitiated SeriesMetadata fields
ProjectDislocate Apr 21, 2026
270e272
feat: improved debug logging
ProjectDislocate Apr 21, 2026
24c64c0
chore: bump deps
ProjectDislocate Apr 21, 2026
43df403
feat: dedupe probided urls
ProjectDislocate Apr 21, 2026
8b3beb8
feat: request caching
ProjectDislocate Apr 21, 2026
9e36ee4
fix: apparently web links are comma seperated
ProjectDislocate Apr 21, 2026
e2062cc
refactor: move http helpers to http package
ProjectDislocate Apr 21, 2026
ebe90fe
feat: inital switch to site registry
ProjectDislocate Apr 21, 2026
a550c8a
fix: tests missing proper client init
ProjectDislocate Apr 21, 2026
709969b
feat: detector uses new site registry
ProjectDislocate Apr 21, 2026
869a42c
feat: ensure issue num is never to long
ProjectDislocate Apr 21, 2026
d079620
fix: undo last commit
ProjectDislocate Apr 21, 2026
3ee11f2
fix: prevent trailing spaces in dir name
ProjectDislocate Apr 21, 2026
0451b07
fix: files overwriting each other
ProjectDislocate Apr 21, 2026
9163d4a
feat: add http proxy support
ProjectDislocate Apr 21, 2026
adc2ec7
refactor: make clear what the output format is
ProjectDislocate Apr 21, 2026
7c16f46
feat: add webp output support
ProjectDislocate Apr 21, 2026
dfa4172
refactor: type image format
ProjectDislocate Apr 21, 2026
7721032
refactor: tell SaveImage the provided image's format
ProjectDislocate Apr 21, 2026
0d5c871
feat: add libjpeg support
ProjectDislocate Apr 21, 2026
55cf4c9
fix: mangadex redownloading the same comics
ProjectDislocate Apr 21, 2026
b94ddf5
fix: prevent tripple periods
ProjectDislocate Apr 21, 2026
2225706
fix: use std lib to encode jpegs
ProjectDislocate Apr 21, 2026
d5db676
fix: dont recompute image format
ProjectDislocate Apr 21, 2026
bdc7ba7
feat: add chapter volume to filename
ProjectDislocate Apr 21, 2026
aec8851
feat: apply rate limiting to all requests
ProjectDislocate Apr 21, 2026
c79c9d6
feat: make retry an exponential backoff
ProjectDislocate Apr 21, 2026
0a81342
feat: disambiguate chapter and series title
ProjectDislocate Apr 21, 2026
dde3330
feat: cleaned names for file system, but leaves metadata alone
ProjectDislocate Apr 21, 2026
c147f93
chore: easily enable build flag in vscode for go lsp
ProjectDislocate Apr 21, 2026
5809833
fix: comicinfo.xml not being cleaned up
ProjectDislocate Apr 21, 2026
0204585
fix: make timelimit dynamic for download job
ProjectDislocate Apr 21, 2026
25fde59
feat: update default user agent
ProjectDislocate Apr 21, 2026
9fb0d33
feat: make timeout per request
ProjectDislocate Apr 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

*.patch
build/

# path where comics are downloaded
Expand Down
22 changes: 22 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
run:
timeout: 5m
tests: false
skip-dirs:
- comics

linters:
enable:
- errcheck
- govet
- staticcheck
- gosimple
- ineffassign
- typecheck

linters-settings:
errcheck:
check-type-assertions: true
check-blank: true

issues:
exclude-use-default: false
2 changes: 2 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ go:
before_install:
- sudo apt-get update
- sudo apt install libgl1-mesa-dev xorg-dev
- go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.59.1
- go install github.com/mattn/goveralls@latest

script:
- $(go env GOPATH)/bin/golangci-lint run
- go test -v ./...
- $GOPATH/bin/goveralls -service=travis-ci
7 changes: 7 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"gopls": {
"build.buildFlags": [
// "-tags=libjpeg"
]
},
}
22 changes: 22 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Repository Guidelines

## Project Structure & Module Organization
The project targets Go 1.23+ with a standard layout. `cmd/downloader` hosts the CLI entrypoint, while `cmd/gui` wraps the GUI built with Fyne; both depend on reusable packages under `pkg/`. Key subpackages include `pkg/core` for orchestration, `pkg/sites` for site-specific scrapers, and `pkg/util` for image and filesystem helpers. Supporting flag parsing and other internals live in `internal/`. Docs for contributors live in `docs/`, assets in `img/`, and release artifacts appear under `build/` when generated.

## Build, Test, and Development Commands
Run `go build -o ./bin/comics-downloader ./cmd/downloader` for a local CLI binary. The GUI can be built with `go build -o ./bin/comics-downloader-gui ./cmd/gui` once Fyne prerequisites are met. Cross-platform artifacts are automated via Make targets such as `make linux-x86-64-build` or the aggregate `make builds`. Use `fyne-cross windows -output comics-downloader-gui-windows.exe ./cmd/gui` when Docker-based cross-compilation is preferred. Static checks run via `make lint` (wraps `golangci-lint run` and matches CI), and `go test -v ./...` should pass before pushing; CI mirrors this command set and reports coverage with Coveralls.

For sites with aggressive bot protections, you can rotate request headers using `--user-agents "UA1,UA2"` and forward cookies such as `cf_clearance` with `--session-cookie "cf_clearance=...; other=value"` when invoking the CLI.

## Coding Style & Naming Conventions
Format all Go files with `gofmt` (invoked via `go fmt ./...`) to preserve canonical tab-indented style and import ordering. Follow idiomatic Go naming: exported identifiers use CamelCase, unexported use camelCase, constants are ALL_CAPS only when enumerations require clarity. Keep packages small and cohesive; prefer descriptive filenames like `mangadex.go` aligning with the site handled. Log through the existing Logrus helpers to stay consistent with current output.

## Testing Guidelines
Place tests alongside implementation in `*_test.go` files, using table-driven cases where practical. Run targeted suites with `go test ./pkg/sites` during feature work, and use `go test -cover ./...` to ensure coverage does not regress—Coveralls tracks the master branch. When scraping new sources, add integration-style tests that stub HTTP calls where possible to keep the suite deterministic.

## Commit & Pull Request Guidelines
Commit subjects should be concise and imperative (`add`, `fix`, `update`), optionally prefixed with scopes like `fix:` as seen in history. Commit early and keep diffs focused; avoid mixing GUI and CLI changes in one commit. Pull requests must target `master`, include a brief summary of the change, reference related issues, and note how to reproduce or test the update (logs, commands, screenshots for GUI tweaks). Ensure tests pass locally before requesting review.

## Security & Configuration Tips
- Use `--user-agents` to supply a comma-separated list of desktop/mobile agents; the downloader rotates them per request.
- When Cloudflare or other shields require clearance cookies, reuse your browser session via `--session-cookie` so authenticated requests succeed.
28 changes: 17 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,39 +1,42 @@
BUILD_TAGS ?=
GO_BUILD_TAG_FLAGS := $(if $(strip $(BUILD_TAGS)),-tags "$(BUILD_TAGS)",)

help: # this command
# [generating help from tasks header]
@egrep '^[A-Za-z0-9_-]+:' Makefile

osx-build-arm: # Creates Mac OSX
@GOOS=darwin go build -o build/comics-downloader-osx-arm ./cmd/downloader
@GOOS=darwin go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-osx-arm ./cmd/downloader

osx-build-x86-64: # Creates Mac OSX
@GOOS=darwin GOARCH=amd64 go build -o build/comics-downloader-osx-x86-64 ./cmd/downloader
@GOOS=darwin GOARCH=amd64 go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-osx-x86-64 ./cmd/downloader

windows-x86-64-build: # Creates Windows
@GOOS=windows GOARCH=amd64 go build -o build/comics-downloader-win-x86-64.exe ./cmd/downloader
@GOOS=windows GOARCH=amd64 go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-win-x86-64.exe ./cmd/downloader

windows-386-build: # Creates Windows
@GOOS=windows GOARCH=386 go build -o build/comics-downloader-win-386.exe ./cmd/downloader
@GOOS=windows GOARCH=386 go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-win-386.exe ./cmd/downloader

linux-x86-64-build: # Creates Linux
@GOOS=linux GOARCH=amd64 go build -o build/comics-downloader-linux-x86-64 ./cmd/downloader
@GOOS=linux GOARCH=amd64 go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-linux-x86-64 ./cmd/downloader

linux-386-build:
@GOOS=linux GOARCH=386 go build -o build/comics-downloader-linux-386 ./cmd/downloader
@GOOS=linux GOARCH=386 go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-linux-386 ./cmd/downloader

linux-arm-build: # Creates Linux ARM
@GOOS=linux GOARCH=arm go build -o build/comics-downloader-linux-arm ./cmd/downloader
@GOOS=linux GOARCH=arm go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-linux-arm ./cmd/downloader

linux-arm64-build: # Creates Linux ARM64
@GOOS=linux GOARCH=arm64 go build -o build/comics-downloader-linux-arm64 ./cmd/downloader
@GOOS=linux GOARCH=arm64 go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-linux-arm64 ./cmd/downloader

osx-gui-build: # Creates osx GUI
@GOOS=darwin go build -o build/comics-downloader-gui-osx ./cmd/gui
@GOOS=darwin go build $(GO_BUILD_TAG_FLAGS) -o build/comics-downloader-gui-osx ./cmd/gui

windows-gui-build: # Creates Window GUI executable
@fyne-cross windows -output comics-downloader-gui-windows.exe ./cmd/gui
@fyne-cross windows -output comics-downloader-gui-windows.exe $(GO_BUILD_TAG_FLAGS) ./cmd/gui

linux-gui-build: # Creates Linux Gui executable
@fyne-cross linux -output comics-downloader-gui ./cmd/gui
@fyne-cross linux -output comics-downloader-gui $(GO_BUILD_TAG_FLAGS) ./cmd/gui

builds: # Creates executables for OSX/Windows/Linux
@make linux-386-build
Expand All @@ -50,3 +53,6 @@ builds: # Creates executables for OSX/Windows/Linux

remove-builds: # Remove executables
@rm -rf build/

lint: # Run static analysis
@golangci-lint run
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
- https://readcomiconline.li/ ⚠️
- https://www.mangareader.tv/ ⚠️
- https://www.mangatown.com/ ⚠️
- https://mangadex.org/ ⚠️
- https://mangadex.org/
- https://mangakakalot.com/ ⚠️
- https://manganato.com/ ⚠️

Expand Down Expand Up @@ -199,6 +199,16 @@ Default is **jpg**.
./comics-downloader -url=[your url] -images-only -images-format=jpg
```

### Work Around Site Protection

Some sources employ rotating fingerprints or require a logged-in session. You can rotate multiple User-Agent strings and forward session cookies collected from your browser:

```bash
./comics-downloader -url=[your url] \
--user-agents="Mozilla/5.0 ...,Mozilla/5.0 (Macintosh; ...)" \
--session-cookie="cf_clearance=...; other=value"
```

### Avoid Default Folder Structure

The default folder structure that will be created is:
Expand Down
Loading