How do I get more verbosity from logs?

[baccccccc]

I already run with log_level: 10 but I feel like that's not enough.

here's one example. I'm scraping a particular forum thread, and CDL crashes with

                    CRITICAL An error occurred, please report this to the developer with your logs file:                                                                                                                            logger.py:27
                    CRITICAL   An IPv4 address cannot be in brackets                                                                                                                                                                logger.py:27
                             ╭──────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────╮             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\cyberdrop_dl\scraper\crawlers\xenforo_crawler.py:209 in │             
                             │ post                                                                                                                                                                                               │             
                             │                                                                                                                                                                                                    │             
                             │   206 │   │   scrape_item.set_type(FORUM_POST, self.manager)                                                                                                                                       │             
                             │   207 │   │   posts_scrapers = [self.attachments, self.embeds, self.images, self.links,                                                                                                            │             
                             │       self.videos]                                                                                                                                                                                 │             
                             │   208 │   │   for scraper in posts_scrapers:                                                                                                                                                       │             
                             │ ❱ 209 │   │   │   await scraper(scrape_item, post)                                                                                                                                                 │             
                             │   210 │                                                                                                                                                                                            │             
                             │   211 │   async def links(self, scrape_item: ScrapeItem, post: ForumPost) -> None:                                                                                                                 │             
                             │   212 │   │   """Scrapes links from a post."""                                                                                                                                                     │             
                             │                                                                                                                                                                                                    │             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\cyberdrop_dl\scraper\crawlers\xenforo_crawler.py:216 in │             
                             │ links                                                                                                                                                                                              │             
                             │                                                                                                                                                                                                    │             
                             │   213 │   │   selector = post.selectors.links                                                                                                                                                      │             
                             │   214 │   │   links = post.content.select(selector.element)                                                                                                                                        │             
                             │   215 │   │   links = [link for link in links if self.is_valid_post_link(link)]                                                                                                                    │             
                             │ ❱ 216 │   │   await self.process_children(scrape_item, links, selector.attribute)                                                                                                                  │             
                             │   217 │                                                                                                                                                                                            │             
                             │   218 │   async def images(self, scrape_item: ScrapeItem, post: ForumPost) -> None:                                                                                                                │             
                             │   219 │   │   """Scrapes images from a post."""                                                                                                                                                    │             
                             │                                                                                                                                                                                                    │             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\cyberdrop_dl\scraper\crawlers\xenforo_crawler.py:275 in │             
                             │ process_children                                                                                                                                                                                   │             
                             │                                                                                                                                                                                                    │             
                             │   272 │   │   │   if not link_str or link_str.startswith("data:image/svg"):                                                                                                                        │             
                             │   273 │   │   │   │   continue                                                                                                                                                                     │             
                             │   274 │   │   │                                                                                                                                                                                    │             
                             │ ❱ 275 │   │   │   link = await self.get_absolute_link(link_str)                                                                                                                                    │             
                             │   276 │   │   │   link = self.filter_link(link)                                                                                                                                                    │             
                             │   277 │   │   │   if not link:                                                                                                                                                                     │             
                             │   278 │   │   │   │   continue                                                                                                                                                                     │             
                             │                                                                                                                                                                                                    │             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\cyberdrop_dl\scraper\crawlers\xenforo_crawler.py:311 in │             
                             │ _                                                                                                                                                                                                  │             
                             │                                                                                                                                                                                                    │             
                             │   308 │   │   text_to_replace = [(".th.", "."), (".md.", "."), ("ifr", "watch")]                                                                                                                   │             
                             │   309 │   │   for old, new in text_to_replace:                                                                                                                                                     │             
                             │   310 │   │   │   link_str = link_str.replace(old, new)                                                                                                                                            │             
                             │ ❱ 311 │   │   parsed_link = self.parse_url(link_str)                                                                                                                                               │             
                             │   312 │   │   return await self.get_absolute_link(parsed_link)                                                                                                                                     │             
                             │   313 │                                                                                                                                                                                            │             
                             │   314 │   async def handle_link(self, scrape_item: ScrapeItem, link: URL) -> None:                                                                                                                 │             
                             │                                                                                                                                                                                                    │             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\cyberdrop_dl\scraper\crawler.py:260 in parse_url        │             
                             │                                                                                                                                                                                                    │             
                             │   257 │   │   assert isinstance(link_str, str)                                                                                                                                                     │             
                             │   258 │   │   encoded = "%" in link_str                                                                                                                                                            │             
                             │   259 │   │   base = relative_to or self.primary_base_domain                                                                                                                                       │             
                             │ ❱ 260 │   │   new_url = URL(link_str, encoded=encoded)                                                                                                                                             │             
                             │   261 │   │   if not new_url.absolute:                                                                                                                                                             │             
                             │   262 │   │   │   new_url = base.join(new_url)                                                                                                                                                     │             
                             │   263 │   │   if not new_url.scheme:                                                                                                                                                               │             
                             │                                                                                                                                                                                                    │             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\yarl\_url.py:307 in __new__                             │             
                             │                                                                                                                                                                                                    │             
                             │    304 │   │   if strict is not None:  # pragma: no cover                                                                                                                                          │             
                             │    305 │   │   │   warnings.warn("strict parameter is ignored")                                                                                                                                    │             
                             │    306 │   │   if type(val) is str:                                                                                                                                                                │             
                             │ ❱  307 │   │   │   return pre_encoded_url(val) if encoded else encode_url(val)                                                                                                                     │             
                             │    308 │   │   if type(val) is cls:                                                                                                                                                                │             
                             │    309 │   │   │   return val                                                                                                                                                                      │             
                             │    310 │   │   if type(val) is SplitResult:                                                                                                                                                        │             
                             │                                                                                                                                                                                                    │             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\yarl\_url.py:148 in encode_url                          │             
                             │                                                                                                                                                                                                    │             
                             │    145 │   """Parse unencoded URL."""                                                                                                                                                              │             
                             │    146 │   cache: _InternalURLCache = {}                                                                                                                                                           │             
                             │    147 │   host: Union[str, None]                                                                                                                                                                  │             
                             │ ❱  148 │   scheme, netloc, path, query, fragment = split_url(url_str)                                                                                                                              │             
                             │    149 │   if not netloc:  # netloc                                                                                                                                                                │             
                             │    150 │   │   host = ""                                                                                                                                                                           │             
                             │    151 │   else:                                                                                                                                                                                   │             
                             │                                                                                                                                                                                                    │             
                             │ C:\Users\[REDACTED]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\yarl\_parse.py:76 in split_url                          │             
                             │                                                                                                                                                                                                    │             
                             │    73 │   │   │   │   if not re.match(r"\Av[a-fA-F0-9]+\..+\Z", bracketed_host):                                                                                                                   │             
                             │    74 │   │   │   │   │   raise ValueError("IPvFuture address is invalid")                                                                                                                         │             
                             │    75 │   │   │   elif ":" not in bracketed_host:                                                                                                                                                  │             
                             │ ❱  76 │   │   │   │   raise ValueError("An IPv4 address cannot be in brackets")                                                                                                                    │             
                             │    77 │   if has_hash:                                                                                                                                                                             │             
                             │    78 │   │   url, _, fragment = url.partition("#")                                                                                                                                                │             
                             │    79 │   if has_question_mark:                                                                                                                                                                    │             
                             ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯             
                             ValueError: An IPv4 address cannot be in brackets                                                                                                                                                                  

it would be nice to see the exact URL it does not like, or at least the exact forum post link where it occurs, right?

so, ether debug log level should include that info, or we need an even more verbose log option.

I'm a bit hesitant to file this as feature request because, chances are, I'm holding it wrong.

thoughts?

[NTFSvolume]

There no additional verbosity available. That's a low level error from a library. It's not a CDL error. We can't modify the message.

But this is the kind of errors that should never happen, which means the error it not actually accurate and the actual error is further up the logic chain. Whatever message gets printed here will be irrelevant. The only import detail is the trace-back to see to chain of methods that eventually raises the error.

You have to run the source code in an IDE to figure out what is actually happening.

TLDR: If you get a critical error, fill a bug report

[baccccccc]

yeah. I understand that you cannot give more info about the error itself. The existing error record is already long and probably helpful (for someone who understands the library code.)

my point was that, at debug level I would expect even more verbose messages from CDL itself like: “scraping post XYZ... done scraping post XYZ”. So that when errors like this happen, I can easily know where exactly it happens. So that I can find that post and take a guess about what went wrong, and maybe exclude this particular URL or do something else to mitigate the issue.

shit happens, and it would be unproductive to nudge you guys for every single typo someone does at forums while ham-fistedly typing the URLs :)

so, what I did instead is use a local HTTP proxy (Fiddler) to see where it fails. And it turned out the last forum page CDL requests before crashing was page 7 of particular forum thread.

then I went through the posts on this page one by one, trying to scrape them separately with scrape_single_forum_post: true. And voila, it was a post with a malformed link that looked literally like this:

http://[URL]https://gofile.io/d/HLUEeG
[/URL]

or, in the page source

<div><a href="http://[URL]https://gofile.io/d/HLUEeG" target="_blank" class="link link--external" rel="nofollow ugc noopener">http://[URL]https://gofile.io/d/HLUEeG</a></div>[/URL]

funny enough, this broken link was in a quote. And the original author has already corrected the typo in their post. It's just the quote where the link remains broken. (And it's a 404 anyways.)

now, it's probably impractical to ask you to handle this special case gracefully. But maybe the outcome is that we need an option to exclude a specific forum post from scraping?

[NTFSvolume]

Logging for those cases exists but you will only get it if you run CDL from source in an IDE.

It includes pager logs, posts logs, requests logs (every single requests, its URL and the response code), file system locks logs and local variable values from within a traceback. It's not enabled in production because that info is too verbose.

Maybe we can add an option to enable those logs in production. If that option is added you should never share those logs with anyone cause it will have raw information like cookies, auth credentials, IPs, etc..

[NTFSvolume]

It's now possible to get developer logs in v6.6.1

Set the environment variable CDL_DEBUG_LOG_FOLDER to any path and CDL will create the log file inside it, with the additional information mentioned above.