Hey! Thanks so much for the great work. Unfortunately I run into somewhat, at least for me, unexpected behaviour.
Description
When a User-agent: line contains a token that doesn't match the expected pattern (e.g. one starting with a digit like 5emeRue), robotspy silently drops the User-agent: line but continues applying the subsequent Disallow: rules to the previously open group. This causes silent misattribution of rules.
Reproduction
import robots
parser = robots.RobotsParser.from_string("""
User-agent: *
Disallow: /internal
User-agent: 5emeRue
Disallow: /
""")
print(parser.can_fetch("*", "/foo")) # Expected: True — Actual: FalseRoot cause
In parser.py, RE_AGENT and RE_PRODUCT only match tokens consisting of [a-zA-Z_-], which excludes digits. This is consistent with RFC 9309, which defines a product token as letters, hyphens, and underscores only.
However, when the regex fails to match, the User-agent: line is silently ignored while the parser leaves the previous group context open. The Disallow: rules that follow are then attributed to that previous group instead of being discarded.
Expected behavior
Regardless of whether the token is valid per RFC 9309, encountering a User-agent: line should always reset the current group context. The subsequent rules should either be grouped under an ignored/discarded group, or skipped entirely — but never silently merged into the preceding group.
Suggested fix
In parse_tokens(), when a User-agent: line is detected but the token fails validation, the parser should still close the current group and open a new (discarded) one, rather than leaving the previous group context open.
Hey! Thanks so much for the great work. Unfortunately I run into somewhat, at least for me, unexpected behaviour.
Description
When a
User-agent:line contains a token that doesn't match the expected pattern (e.g. one starting with a digit like5emeRue), robotspy silently drops theUser-agent:line but continues applying the subsequentDisallow:rules to the previously open group. This causes silent misattribution of rules.Reproduction
Root cause
In
parser.py,RE_AGENTandRE_PRODUCTonly match tokens consisting of[a-zA-Z_-], which excludes digits. This is consistent with RFC 9309, which defines a product token as letters, hyphens, and underscores only.However, when the regex fails to match, the
User-agent:line is silently ignored while the parser leaves the previous group context open. TheDisallow:rules that follow are then attributed to that previous group instead of being discarded.Expected behavior
Regardless of whether the token is valid per RFC 9309, encountering a
User-agent:line should always reset the current group context. The subsequent rules should either be grouped under an ignored/discarded group, or skipped entirely — but never silently merged into the preceding group.Suggested fix
In
parse_tokens(), when aUser-agent:line is detected but the token fails validation, the parser should still close the current group and open a new (discarded) one, rather than leaving the previous group context open.