Skip to content

feat: change logging format - #358

Open
eoinkelly wants to merge 1 commit into
mainfrom
chore-improve-logs
Open

feat: change logging format#358
eoinkelly wants to merge 1 commit into
mainfrom
chore-improve-logs

Conversation

@eoinkelly

@eoinkelly eoinkelly commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Improve log output. Changes are:

  • Add the logger name field. Lets us distinguish between logs emitted from our code (name=cwac) and libraries e.g. all ultimate-sitemap-parser logs are name=usp.*
  • Simplify the format of each log line. All fields separated by single space, no padding of columns. This simplifies filtering the log.
  • remove funcName. The log entries contain filename and lineno. This is enough to find the line. funcName just adds noise.

Example log output after this change

2026-08-25T13:40:01+1200 INFO MainThread cwac output.py:118 ********************************************************************************
2026-08-25T13:40:01+1200 INFO MainThread cwac output.py:118 Centralised Web Accessibility Checker (CWAC)
2026-08-25T13:40:01+1200 INFO MainThread cwac output.py:118 Te Pūnaha Matihiko | Government Digital Delivery Agency
2026-08-25T13:40:01+1200 INFO MainThread cwac output.py:118 Run time: 2026-08-25 13:40:01
2026-08-25T13:40:01+1200 INFO MainThread cwac output.py:118 ********************************************************************************
(snip)
2026-08-25T13:40:01+1200 INFO MainThread cwac cwac.py:141 Number of websites to be scanned: 1
2026-08-25T13:40:05+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:70 Starting test https://www.ackama.com
2026-08-25T13:40:05+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:499 Starting crawl of https://www.ackama.com
2026-08-25T13:40:05+1200 INFO ThreadPoolExecutor-0_0 cwac browser.py:158 Quitting browser
2026-08-25T13:40:05+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:400 Fetching robots.txt https://www.ackama.com/robots.txt
2026-08-25T13:40:05+1200 INFO ThreadPoolExecutor-0_4 cwac browser.py:158 Quitting browser
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:412 Fetched robots.txt https://www.ackama.com/robots.txt
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:469 robots.txt result for https://www.ackama.com/. was allow
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_1 cwac browser.py:158 Quitting browser
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_7 cwac browser.py:158 Quitting browser
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_6 cwac browser.py:158 Quitting browser
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_5 cwac browser.py:158 Quitting browser
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_3 cwac browser.py:158 Quitting browser
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_2 cwac filters.py:251 https://www.ackama.com/. has status code 200
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_2 cwac audit_manager.py:214 run_audits called
2026-08-25T13:40:06+1200 INFO ThreadPoolExecutor-0_2 cwac audit_manager.py:242 Starting audit axe_core_audit on https://www.ackama.com/
(snip)
2026-08-25T13:40:12+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:633 Fetching sitemap for https://www.ackama.com
2026-08-25T13:40:12+1200 WARNING ThreadPoolExecutor-0_2 usp.tree tree.py:78 Assuming that the homepage of https://www.ackama.com is https://www.ackama.com/
2026-08-25T13:40:12+1200 INFO ThreadPoolExecutor-0_2 usp.fetch_parse fetch_parse.py:142 Fetching level 0 sitemap from https://www.ackama.com/robots.txt...
2026-08-25T13:40:12+1200 INFO ThreadPoolExecutor-0_2 usp.helpers helpers.py:173 Fetching URL https://www.ackama.com/robots.txt...
2026-08-25T13:40:12+1200 INFO ThreadPoolExecutor-0_2 usp.fetch_parse fetch_parse.py:215 Parsing sitemap from URL https://www.ackama.com/robots.txt...
2026-08-25T13:40:12+1200 INFO ThreadPoolExecutor-0_2 usp.fetch_parse fetch_parse.py:142 Fetching level 1 sitemap from https://www.ackama.com/sitemap_index.xml...
2026-08-25T13:40:12+1200 INFO ThreadPoolExecutor-0_2 usp.helpers helpers.py:173 Fetching URL https://www.ackama.com/sitemap_index.xml...
2026-08-25T13:40:12+1200 INFO ThreadPoolExecutor-0_2 usp.fetch_parse fetch_parse.py:215 Parsing sitemap from URL https://www.ackama.com/sitemap_index.xml...
(snip)
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:611 Found 510 urls from sitemaps
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:306 get_links skipped due to equality of: https://www.ackama.com https://www.ackama.com/
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:306 get_links skipped due to equality of: https://www.ackama.com https://www.ackama.com/
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac filters.py:329 Non-empty fragment filter rejected https://www.ackama.com/#main-content
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac filters.py:329 HTTPS only filter rejected http://ackama.com/services
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:306 get_links skipped due to equality of: https://www.ackama.com https://www.ackama.com/
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac filters.py:329 Whitelist filter rejected https://www.linkedin.com/company/ackama/
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac filters.py:329 Whitelist filter rejected https://github.com/ackama/
2026-08-25T13:40:17+1200 INFO ThreadPoolExecutor-0_2 cwac filters.py:329 Whitelist filter rejected https://maps.app.goo.gl/nFwwZRa2NtE9oiNP8
(snip)
2026-08-25T13:40:48+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:529 Max pages scanned reached https://www.ackama.com
2026-08-25T13:40:48+1200 INFO ThreadPoolExecutor-0_2 cwac crawler.py:629 Crawl exhausted all links https://www.ackama.com
2026-08-25T13:40:49+1200 INFO ThreadPoolExecutor-0_2 cwac browser.py:158 Quitting browser
2026-08-25T13:40:49+1200 INFO MainThread cwac cwac.py:50 All threads complete
2026-08-25T13:40:49+1200 INFO MainThread cwac cwac.py:168 CWAC complete!

@eoinkelly
eoinkelly marked this pull request as ready for review August 25, 2026 01:57
@eoinkelly
eoinkelly requested a review from G-Rath August 25, 2026 01:57
Comment thread config.py
# create a new formatter with our desired format
formatter = Formatter(
'[{%(asctime)s} %(levelname)-7s %(filename)10s : %(lineno)-4s] %(funcName)30s %(message)s %(threadName)s',
'%(asctime)s %(levelname)s %(threadName)s %(name)s %(filename)s:%(lineno)s %(message)s',

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finding a particular log message within our codebase isn't difficult via search so I think %(filename)s:%(lineno)s should be removed too.

I'm keeping it for now to minimise this change but my recommendation is to remove it and start using the name field to properly identify logical units within our own code e.g. cwac.crawler, cwac.browser. This would be a single line change to the top of each module e.g. in crawler.py:

- logger = logging.getLogger('cwac')
+ logger = logging.getLogger('cwac.crawler')

@G-Rath G-Rath left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're going to need to discuss this more - personally I do find the aligning some of the columns with padding makes it easier to glance over the lines, and the function names have been useful to have since they're what I have in my mental model not individual line numbers.

When it comes to logs, I'm usually of the opinion that you should log more not less, because filtering and parsing is usually very cheap whereas its often very hard to extract information after the fact.

I'm fine with the rest of your changes though, if you want to get them landed

@G-Rath G-Rath changed the title chore: Improve log output feat: change logging format Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants