Skip to content

feat: duplicate log entries into dedicated files per thread and base urls - #113

Draft
G-Rath wants to merge 2 commits into
mainfrom
split-logging
Draft

feat: duplicate log entries into dedicated files per thread and base urls#113
G-Rath wants to merge 2 commits into
mainfrom
split-logging

Conversation

@G-Rath

@G-Rath G-Rath commented Jul 14, 2025

Copy link
Copy Markdown
Contributor

This is an experiment into making it easier to dig into the logs by having us attempt to pipe logs into different files based on the thread and base url implicitly using context management - so far it seems to work ok and might even be completely thread safe, but it's possible I've missed something so I'm doing this as a draft for now.

Note that all the logs should still go into the existing single root log file

@G-Rath
G-Rath force-pushed the split-logging branch 2 times, most recently from 82c63cb to 33ab0ce Compare July 29, 2025 23:12

@jonholdsworth jonholdsworth left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suck it and see, I guess. Whether this is easier to visually parse or messier.

@eoinkelly

eoinkelly commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

I think adding a JSON log handler and including the base_url, url and viewport currently being processed in the log messages (when that is known) might be better. Something like:

// shown formatted here for easy reading, in real log this would be one line
{
  "message": "Non-empty fragment filter rejected https://www.foo.com/bar/?a=b",
  "asctime": "2026-08-07 12:53:42,182",
  "levelname": "INFO",
  "filename": "filters.py",
  "lineno": 328,
  "funcName": "run_url_filters",
  "threadName": "ThreadPoolExecutor-0_2",
  "name": "cwac",
  "baseUrl": "https://foo.com",
  "url": "https://foo.com/bar",
  "viewport": "800x600"
}

JSON is very friendly to further processing and we can use tools like jq to slice and dice it for ad-hoc queries.

This means the base_url and url is duplicated quite a lot in the log but the upside is that each log message is self contained and has all the context you need to understand where it came from.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants