feat: duplicate log entries into dedicated files per thread and base urls - #113
Draft
G-Rath wants to merge 2 commits into
Draft
feat: duplicate log entries into dedicated files per thread and base urls#113G-Rath wants to merge 2 commits into
G-Rath wants to merge 2 commits into
Conversation
G-Rath
force-pushed
the
split-logging
branch
2 times, most recently
from
July 29, 2025 23:12
82c63cb to
33ab0ce
Compare
jonholdsworth
approved these changes
Jul 30, 2025
jonholdsworth
left a comment
There was a problem hiding this comment.
Suck it and see, I guess. Whether this is easier to visually parse or messier.
a11ya11y
approved these changes
Jul 31, 2025
Contributor
|
I think adding a JSON log handler and including the base_url, url and viewport currently being processed in the log messages (when that is known) might be better. Something like: // shown formatted here for easy reading, in real log this would be one line
{
"message": "Non-empty fragment filter rejected https://www.foo.com/bar/?a=b",
"asctime": "2026-08-07 12:53:42,182",
"levelname": "INFO",
"filename": "filters.py",
"lineno": 328,
"funcName": "run_url_filters",
"threadName": "ThreadPoolExecutor-0_2",
"name": "cwac",
"baseUrl": "https://foo.com",
"url": "https://foo.com/bar",
"viewport": "800x600"
}JSON is very friendly to further processing and we can use tools like jq to slice and dice it for ad-hoc queries. This means the base_url and url is duplicated quite a lot in the log but the upside is that each log message is self contained and has all the context you need to understand where it came from. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an experiment into making it easier to dig into the logs by having us attempt to pipe logs into different files based on the thread and base url implicitly using context management - so far it seems to work ok and might even be completely thread safe, but it's possible I've missed something so I'm doing this as a draft for now.
Note that all the logs should still go into the existing single root log file