[Bug]: Regex.all_matches fails for a regex without groups and drops nested or unmatched groups #40562

What happened?

beam.Regex.all_matches is documented to return all groups of a matching element, but it builds the output with range(m.lastindex + 1):

https://github.com/apache/beam/blob/d9e99ad211f/sdks/python/apache_beam/transforms/util.py#L2166-L2169

Match.lastindex is not the number of groups. It is the index of the last group that matched, and for nested groups it is the outer group that closed last. It is None when the regex has no groups. So:

import apache_beam as beam

with beam.Pipeline() as p:
  _ = (
      p
      | beam.Create(['ab'])
      | beam.Regex.all_matches(r'((a)(b))')
      | beam.Map(print))
# prints ['ab', 'ab']   expected ['ab', 'ab', 'a', 'b']

with beam.Pipeline() as p:
  _ = (
      p
      | beam.Create(['a'])
      | beam.Regex.all_matches(r'(a)(b)?')
      | beam.Map(print))
# prints ['a', 'a']   expected ['a', 'a', None]

with beam.Pipeline() as p:
  _ = (
      p
      | beam.Create(['aaa'])
      | beam.Regex.all_matches(r'a+')
      | beam.Map(print))
# TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
#   [while running 'all_matches(a+)/FlatMap(_process)']

So the output has a different number of groups depending on which optional groups matched, nested groups are lost, and a regex without groups fails the pipeline.

Expected: one entry for the entire match, then one per group of the regex (None for a group that did not take part in the match), which is what Java's Regex.AllMatches does (it loops over 0..groupCount() and adds null for such groups).

This is the same kind of problem as the one fixed for Regex.find_all(..., Regex.ALL) in #40399.

Reproduced on master at d9e99ad (Python 3.12, DirectRunner). The code has been the same since the transform was added.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner