Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
bb2d3e1
Introduce runner orchestration and version check improvements
hollisticated-horse Oct 20, 2025
89021e0
Enhance HTTP client with retries and user-agent handling
hollisticated-horse Oct 20, 2025
dcea1b3
Refactor MangaDex scraper to use new HTTP helpers
hollisticated-horse Oct 20, 2025
dd6eb45
Refactor core image pipeline and tighten tests
hollisticated-horse Oct 20, 2025
ed7329d
Improve logger safety and add formatted helpers
hollisticated-horse Oct 20, 2025
cd2dc92
Add golangci-lint configuration and workflow docs
hollisticated-horse Oct 20, 2025
875a1f2
Normalize obfuscated Comicextra image URLs
hollisticated-horse Oct 20, 2025
1fa7761
Add optional HTTP mitigation for protected sources
hollisticated-horse Oct 20, 2025
3865e1d
Replace MangaKakalot network tests with fixtures
hollisticated-horse Oct 20, 2025
2a4ec6e
Replace site tests with fixture-backed servers
hollisticated-horse Oct 21, 2025
fb2972d
feat: add request throttling
hollisticated-horse Oct 21, 2025
3e488bb
feat(gui): improve download feedback
hollisticated-horse Oct 23, 2025
848cc98
fix: check close errors
hollisticated-horse Oct 24, 2025
ee6e3ce
chore: remove accidental screenshot file
hollisticated-horse Feb 3, 2026
f871c07
fix(readallcomics): improve chapter/issue URL parsing and error handling
hollisticated-horse Feb 3, 2026
4b528e1
fix(readcomiconline): enhance debug logging for image scraping
hollisticated-horse Feb 3, 2026
4f940ed
feat(util): add WebP image format support
hollisticated-horse Feb 3, 2026
aae4851
fix(logger): enable full level names in log output
hollisticated-horse Feb 3, 2026
77e57f8
fix(app): improve version check error handling
hollisticated-horse Feb 3, 2026
bf8d806
convert Makefile to Justfile
hollisticated-horse Feb 3, 2026
6849690
add GUI unit tests and test-gui Justfile recipe
Apr 7, 2026
816bd3b
add readcomicsonline.ru scraper with data-src and pages-var fallback
Apr 7, 2026
066a0a5
keep readcomiconline.li; add readcomicsonline.ru as separate scraper
Apr 7, 2026
03f666a
fix readcomiconline.li: bounds-safe deobfuscation, dedup, graceful er…
Apr 7, 2026
8c3ab2f
remove readcomiconline.li: server-side encrypted obfuscation, permane…
Apr 7, 2026
f7d5987
audit: check all supported sites for liveness
Apr 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
run:
timeout: 5m
tests: false
skip-dirs:
- comics

linters:
enable:
- errcheck
- govet
- staticcheck
- gosimple
- ineffassign
- typecheck

linters-settings:
errcheck:
check-type-assertions: true
check-blank: true

issues:
exclude-use-default: false
2 changes: 2 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ go:
before_install:
- sudo apt-get update
- sudo apt install libgl1-mesa-dev xorg-dev
- go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.59.1
- go install github.com/mattn/goveralls@latest

script:
- $(go env GOPATH)/bin/golangci-lint run
- go test -v ./...
- $GOPATH/bin/goveralls -service=travis-ci
22 changes: 22 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Repository Guidelines

## Project Structure & Module Organization
The project targets Go 1.23+ with a standard layout. `cmd/downloader` hosts the CLI entrypoint, while `cmd/gui` wraps the GUI built with Fyne; both depend on reusable packages under `pkg/`. Key subpackages include `pkg/core` for orchestration, `pkg/sites` for site-specific scrapers, and `pkg/util` for image and filesystem helpers. Supporting flag parsing and other internals live in `internal/`. Docs for contributors live in `docs/`, assets in `img/`, and release artifacts appear under `build/` when generated.

## Build, Test, and Development Commands
Run `go build -o ./bin/comics-downloader ./cmd/downloader` for a local CLI binary. The GUI can be built with `go build -o ./bin/comics-downloader-gui ./cmd/gui` once Fyne prerequisites are met. Cross-platform artifacts are automated via Make targets such as `make linux-x86-64-build` or the aggregate `make builds`. Use `fyne-cross windows -output comics-downloader-gui-windows.exe ./cmd/gui` when Docker-based cross-compilation is preferred. Static checks run via `make lint` (wraps `golangci-lint run` and matches CI), and `go test -v ./...` should pass before pushing; CI mirrors this command set and reports coverage with Coveralls.

For sites with aggressive bot protections, you can rotate request headers using `--user-agents "UA1,UA2"` and forward cookies such as `cf_clearance` with `--session-cookie "cf_clearance=...; other=value"` when invoking the CLI.

## Coding Style & Naming Conventions
Format all Go files with `gofmt` (invoked via `go fmt ./...`) to preserve canonical tab-indented style and import ordering. Follow idiomatic Go naming: exported identifiers use CamelCase, unexported use camelCase, constants are ALL_CAPS only when enumerations require clarity. Keep packages small and cohesive; prefer descriptive filenames like `mangadex.go` aligning with the site handled. Log through the existing Logrus helpers to stay consistent with current output.

## Testing Guidelines
Place tests alongside implementation in `*_test.go` files, using table-driven cases where practical. Run targeted suites with `go test ./pkg/sites` during feature work, and use `go test -cover ./...` to ensure coverage does not regress—Coveralls tracks the master branch. When scraping new sources, add integration-style tests that stub HTTP calls where possible to keep the suite deterministic.

## Commit & Pull Request Guidelines
Commit subjects should be concise and imperative (`add`, `fix`, `update`), optionally prefixed with scopes like `fix:` as seen in history. Commit early and keep diffs focused; avoid mixing GUI and CLI changes in one commit. Pull requests must target `master`, include a brief summary of the change, reference related issues, and note how to reproduce or test the update (logs, commands, screenshots for GUI tweaks). Ensure tests pass locally before requesting review.

## Security & Configuration Tips
- Use `--user-agents` to supply a comma-separated list of desktop/mobile agents; the downloader rotates them per request.
- When Cloudflare or other shields require clearance cookies, reuse your browser session via `--session-cookie` so authenticated requests succeed.
66 changes: 66 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# List available recipes
default:
@just --list

# Creates Mac OSX ARM binary
osx-build-arm:
GOOS=darwin go build -o build/comics-downloader-osx-arm ./cmd/downloader

# Creates Mac OSX x86-64 binary
osx-build-x86-64:
GOOS=darwin GOARCH=amd64 go build -o build/comics-downloader-osx-x86-64 ./cmd/downloader

# Creates Windows x86-64 binary
windows-x86-64-build:
GOOS=windows GOARCH=amd64 go build -o build/comics-downloader-win-x86-64.exe ./cmd/downloader

# Creates Windows 386 binary
windows-386-build:
GOOS=windows GOARCH=386 go build -o build/comics-downloader-win-386.exe ./cmd/downloader

# Creates Linux x86-64 binary
linux-x86-64-build:
GOOS=linux GOARCH=amd64 go build -o build/comics-downloader-linux-x86-64 ./cmd/downloader

# Creates Linux 386 binary
linux-386-build:
GOOS=linux GOARCH=386 go build -o build/comics-downloader-linux-386 ./cmd/downloader

# Creates Linux ARM binary
linux-arm-build:
GOOS=linux GOARCH=arm go build -o build/comics-downloader-linux-arm ./cmd/downloader

# Creates Linux ARM64 binary
linux-arm64-build:
GOOS=linux GOARCH=arm64 go build -o build/comics-downloader-linux-arm64 ./cmd/downloader

# Creates OSX GUI binary
osx-gui-build:
GOOS=darwin go build -o build/comics-downloader-gui-osx ./cmd/gui

# Creates Windows GUI executable
windows-gui-build:
fyne-cross windows -output comics-downloader-gui-windows.exe ./cmd/gui

# Creates Linux GUI executable
linux-gui-build:
fyne-cross linux -output comics-downloader-gui ./cmd/gui

# Creates executables for OSX/Windows/Linux
builds: linux-386-build linux-arm-build linux-arm64-build linux-x86-64-build osx-build-arm osx-build-x86-64 windows-x86-64-build windows-386-build osx-gui-build linux-gui-build windows-gui-build

# Remove build artifacts
remove-builds:
rm -rf build/

# Run static analysis
lint:
golangci-lint run

# Run GUI unit tests (headless, no display required)
test-gui:
go test -v ./cmd/gui/...

# Run the GUI
run-gui:
go run ./cmd/gui
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,6 @@ builds: # Creates executables for OSX/Windows/Linux

remove-builds: # Remove executables
@rm -rf build/

lint: # Run static analysis
@golangci-lint run
14 changes: 12 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

- https://comicextra.net/
- https://readallcomics.com/
- https://readcomiconline.li/ ⚠️
- https://readcomicsonline.ru/
- https://www.mangareader.tv/ ⚠️
- https://www.mangatown.com/ ⚠️
- https://mangadex.org/ ⚠️
Expand Down Expand Up @@ -92,7 +92,7 @@ Usage:
| http://www.comicextra.com/ | ✓ | ✗ | ✓ |
| http://www.mangatown.com/ | ✓ | ✗ | ✓ |
| https://mangadex.org/ | ✓ | ✓ | ✗ |
| https://readcomiconline.li/ | ✓ | ✗ | ✓ |
| https://readcomicsonline.ru/ | ✓ | ✗ | ✓ |
| https://www.mangareader.tv/ | ✓ | ✗ | ✓ |
| https://www.mangakalot.com/ | ✓ | ✗ | ✓ |
| https://www.manganato.com/ | ✓ | ✗ | ✓ |
Expand Down Expand Up @@ -199,6 +199,16 @@ Default is **jpg**.
./comics-downloader -url=[your url] -images-only -images-format=jpg
```

### Work Around Site Protection

Some sources employ rotating fingerprints or require a logged-in session. You can rotate multiple User-Agent strings and forward session cookies collected from your browser:

```bash
./comics-downloader -url=[your url] \
--user-agents="Mozilla/5.0 ...,Mozilla/5.0 (Macintosh; ...)" \
--session-cookie="cf_clearance=...; other=value"
```

### Avoid Default Folder Structure

The default folder structure that will be created is:
Expand Down
32 changes: 32 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# TODO

## Pending

### Site Audit — Remove Dead/Broken Sites

Site status as of 2026-04-07:

| Site | Domain | Status | Verdict |
|------|--------|--------|---------|
| readcomicsonline.ru | readcomicsonline.ru | ✅ HTTP 200, serves comics | **KEEP** |
| mangadex | api.mangadex.org | ✅ HTTP 200, API responds `pong` | **KEEP** |
| readallcomics | readallcomics.com | ✅ HTTP 200 (with browser UA) | **KEEP** |
| mangatown | mangatown.com | ✅ HTTP 200, site live | **KEEP** |
| comicextra | comicextra.com | ❌ Domain parked (Sedo parking page, not the comic site) | **REMOVE** |
| mangareader | mangareader.tv | ❌ DNS resolution failure — no address associated with hostname | **REMOVE** |
| mangakakalot | mangakakalot.com | ❌ HTTP 522 (Cloudflare origin timeout) — server unreachable | **REMOVE** |
| manganato | manganato.com | ❌ HTTP 522 (Cloudflare origin timeout) — server unreachable | **REMOVE** |

- [ ] Remove `comicextra` scraper: `pkg/sites/comicextra.go`, `pkg/sites/comicextra_test.go`, `pkg/sites/comicextra_deobfuscate_test.go`, `pkg/sites/deobfuscate.go` (if comicextra-only), update `pkg/sites/loader.go` switch, update README
- [ ] Remove `mangareader` scraper: `pkg/sites/mangareader.go`, `pkg/sites/mangareader_test.go`, update `pkg/sites/loader.go` switch, update README
- [ ] Remove `mangakakalot` scraper: `pkg/sites/mangakakalot.go`, `pkg/sites/mangakakalot_test.go`, update `pkg/sites/loader.go` switch, update README
- [ ] Remove `manganato` scraper: `pkg/sites/manganato.go`, `pkg/sites/manganato_test.go`, update `pkg/sites/loader.go` switch, update README
- [ ] Verify `deobfuscate.go` and `common.go` are not shared with kept sites before deleting
- [ ] Run `go test ./...` and `go build ./...` after removals to confirm clean compile
- [ ] Update README supported-sites table to reflect only active sites

## Completed
- [x] Add `readcomicsonline.ru` scraper (separate from `readcomiconline.li`)
- [x] Remove `readcomiconline.li` scraper: site uses server-side encrypted obfuscation, permanently broken (`pkg/sites/readcomicsonline.go`) with data-src primary path, JS `var pages` fallback, issue listing, `--all`, `--last` support, and full test coverage
- [x] Convert Makefile to Justfile
- [x] Add GUI unit tests (`cmd/gui/gui_test.go`) and `test-gui` Justfile recipe
Loading