Generated code fails with undefined: longrunning.InternalNewOperationWithMetadata

[suztomo]

I ran librarian generate -all with the latest googleapis SHA and the Kokoro build failed: https://fusion2.corp.google.com/invocations/13075e79-e911-4147-84cf-d118932abdaf/log :

+ gotestsum --packages=./... --junitfile sponge_log.xml --format standard-verbose -- -race --short --timeout 15m
go: downloading [cloud.google.com/go/longrunning](http://cloud.google.com/go/longrunning) v0.9.0
?   	[cloud.google.com/go/cloudtasks/apiv2/cloudtaskspb](http://cloud.google.com/go/cloudtasks/apiv2/cloudtaskspb)	[no test files]
testing: warning: no tests to run
PASS
?   	[cloud.google.com/go/cloudtasks/apiv2beta2/cloudtaskspb](http://cloud.google.com/go/cloudtasks/apiv2beta2/cloudtaskspb)	[no test files]
testing: warning: no tests to run
PASS
ok  	[cloud.google.com/go/cloudtasks/apiv2](http://cloud.google.com/go/cloudtasks/apiv2)	1.046s [no tests to run]
FAIL	[cloud.google.com/go/cloudtasks/apiv2beta3](http://cloud.google.com/go/cloudtasks/apiv2beta3) [build failed]
?   	[cloud.google.com/go/cloudtasks/apiv2beta3/cloudtaskspb](http://cloud.google.com/go/cloudtasks/apiv2beta3/cloudtaskspb)	[no test files]
?   	[cloud.google.com/go/cloudtasks/internal](http://cloud.google.com/go/cloudtasks/internal)	[no test files]
ok  	[cloud.google.com/go/cloudtasks/apiv2beta2](http://cloud.google.com/go/cloudtasks/apiv2beta2)	1.030s [no tests to run]

=== Failed
=== FAIL: apiv2beta3  (0.00s)
FAIL	[cloud.google.com/go/cloudtasks/apiv2beta3](http://cloud.google.com/go/cloudtasks/apiv2beta3) [build failed]

=== Errors
[apiv2beta3/cloud_tasks_client.go:1477](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=1477):21: undefined: longrunning.InternalNewOperationWithMetadata

[apiv2beta3/cloud_tasks_client.go:1527](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=1527):21: undefined: longrunning.InternalNewOperationWithMetadata

[apiv2beta3/cloud_tasks_client.go:2781](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=2781):21: undefined: longrunning.InternalNewOperationWithMetadata

[apiv2beta3/cloud_tasks_client.go:2901](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=2901):21: undefined: longrunning.InternalNewOperationWithMetadata

[apiv2beta3/cloud_tasks_client.go:3341](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=3341):20: undefined: longrunning.InternalNewOperationWithMetadata

[apiv2beta3/cloud_tasks_client.go:3350](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=3350):25: undefined: longrunning.InternalNewOperationWithMetadata

[apiv2beta3/cloud_tasks_client.go:3359](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=3359):20: undefined: longrunning.InternalNewOperationWithMetadata

[apiv2beta3/cloud_tasks_client.go:3368](https://cs.corp.google.com/piper///depot/google3/apiv2beta3/cloud_tasks_client.go?l=3368):25: undefined: longrunning.InternalNewOperationWithMetadata

Originally posted by @suztomo in #20188 (comment)

[suztomo]

@haphungw Would you add a comment about your findings and how you addressed #20188 (comment) ?

[haphungw]

@suztomo
@hongalex and I dug into this, and here's what we found:

The build failed because cloudtasks got a new LRO, but the generator bumped longrunning to v0.9.0 instead of v1.2.0, which meant the new tracing APIs were missing.

This happened because the generator relies on go mod tidy for new imports. Since cloudtasks/go.mod already depended on iam v1.11.0, go mod tidy fell back to iam boundary for longrunning (which is v0.9.0), missing the v1.2.0 requirement from librarian.yaml.

To unblock the PR, I just manually pushed a commit to the branch to pin cloudtasks/go.mod to cloud.google.com/go/longrunning v1.2.0, which turned it green.

Going forward, our generator either needs to check librarian.yaml when injecting new dependencies before running go mod tidy.
Or less ideally, we could just globally bump iam to v1.12.0 across the repo so the requirement is raised to v1.2.0.

[zhumin8]

Going forward, our generator either needs to check librarian.yaml when injecting new dependencies before running go mod tidy.

IIUC, this happend at a re-generation on googleapis commit update. There is no librarian.yaml change involved. Can you explain what info from librarian.yaml would help go gapic generator determine the version in this case?

[haphungw]

My note about librarian.yaml was regarding the post-processing phase. My assumption is when go mod tidy runs on the newly generated code, it detects the new longrunning import but defaults to v0.9.0, which is the minimum bound transitively inherited from the iam dep in cloudtasks.

Because librarian.yaml manages longrunning at v1.2.0, it seems like this intended version floor is passed when cloudtasks introduces the longrunning dep for the first time, causing go mod tidy to silently resolve to an older version and fail on the new tracing APIs.

If there was a configuration step I missed when introducing the tracing feature in longrunning that would have accounted for this, please let me know!

googleapis/gapic-generator-go#1776
#20107