v2.8.1 + Protobuf320, GRPC migration + cpu_count, xla

[h-vetinari]

• Closes Protobuf320 and GRPC migration #236
  • See Rebuild for libprotobuf320 #232
  • See Rebuild for grpc_cpp144 #228
• Closes v2.8.1 + add cpu_count, xla #235
  • Closes tracking "easy" updates and improvements (for patch release 2.8.1?) #234
  • Closes implement CPU_COUNT in bazel? #230
  • Closes Not a really a bug, but let's enable XLA in the builds going forward #223

CC @ngam @hmaarrfk

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

• It looks like the 'tensorflow-base' output doesn't have any tests.

[xhochy]

Looks good. I can kick off builds for this tomorrow!

[h-vetinari]

Looks good. I can kick off builds for this tomorrow!

That would be awesome! 🤩 ... and might be a record turn-around for a PR on this feedstock 😅

(though of course, I'm only riding on the coattails of previous work by @ngam & @hmaarrfk 😇)

[ngam]

All good with me, thanks for pushing along. Hopefully we will do the 2.9 soon too (I will start poking with it early next week)

[ngam]

@xhochy only potential issue to pay closer attention to here is XLA on osx. I believe if it doesn't work, it will just fail back safely (i.e. no clustering), but it is worth testing it. Everything else should be straightforward

[xhochy]

XLA is heavily used in jaxlib so I would guess that it will also work here.

[xhochy]

@hmaarrfk Builds are in https://anaconda.org/uwe.korn/repo and in https://anaconda.org/uwe.korn-tf-gpu/repo Can you do the usual review, copy, merge, please?

Logs:

• https://gist.github.com/xhochy/a3d7df9e736f998b0f54f4637196ab3b osx_arm64_abseil_cpp20210324.2grpc_cpp1.45python3.9.____cpython.log
• https://gist.github.com/xhochy/0a52e5270089c01147caf889449efae3 osx_arm64_abseil_cpp20210324.2grpc_cpp1.45python3.8.____cpython.log
• https://gist.github.com/xhochy/ef161b7af5fd95c1ac0917651e505835 osx_arm64_abseil_cpp20210324.2grpc_cpp1.45python3.10.____cpython.log
• https://gist.github.com/xhochy/388e0b65884a3dc92d4c8e5ad83951db osx_64_abseil_cpp20210324.2grpc_cpp1.45python3.9.____cpython.log
• https://gist.github.com/xhochy/2a0713ae46522bd0a1cd5df2fa190dcd osx_64_abseil_cpp20210324.2grpc_cpp1.45python3.8.____cpython.log
• https://gist.github.com/xhochy/da455ce68cde93dff05896b0626fb3a4 osx_64_abseil_cpp20210324.2grpc_cpp1.45python3.7.____cpython.log
• https://gist.github.com/xhochy/29af8b88837cd393c9417d62ffb5dcbe osx_64_abseil_cpp20210324.2grpc_cpp1.45python3.10.____cpython.log
• https://gist.github.com/xhochy/f74e41f1ed73a54db71b1cd41b963267 linux_64_abseil_cpp20210324.2c_compiler_version9cuda_compiler_version11.0cxx_compiler_version9grpc_cpp1.45python3.9.____cpython.log
• https://gist.github.com/xhochy/49a1c604de5c4dfae4e89e5f17dd0a00 linux_64_abseil_cpp20210324.2c_compiler_version9cuda_compiler_version11.0cxx_compiler_version9grpc_cpp1.45python3.8.____cpython.log
• https://gist.github.com/xhochy/c0f95cadfb0f5054835519007ff276af linux_64_abseil_cpp20210324.2c_compiler_version9cuda_compiler_version11.0cxx_compiler_version9grpc_cpp1.45python3.7.____cpython.log
• https://gist.github.com/xhochy/ba574c5ebe159e9c9093842fbe8ea8d1 linux_64_abseil_cpp20210324.2c_compiler_version9cuda_compiler_version11.0cxx_compiler_version9grpc_cpp1.45python3.10.____cpython.log
• https://gist.github.com/xhochy/86700cfad40053d403dea31f626ba981 linux_64_abseil_cpp20210324.2c_compiler_version7cuda_compiler_version10.2cxx_compiler_version7grpc_cpp1.45python3.9.____cpython.log
• https://gist.github.com/xhochy/397242ddd5e369bf34541e6ff81919f6 linux_64_abseil_cpp20210324.2c_compiler_version7cuda_compiler_version10.2cxx_compiler_version7grpc_cpp1.45python3.8.____cpython.log
• https://gist.github.com/xhochy/84936f4b6306415ac911f1fcc5a3f838 linux_64_abseil_cpp20210324.2c_compiler_version7cuda_compiler_version10.2cxx_compiler_version7grpc_cpp1.45python3.7.____cpython.log
• https://gist.github.com/xhochy/7943fb417911067690394562f4fa0543 linux_64_abseil_cpp20210324.2c_compiler_version7cuda_compiler_version10.2cxx_compiler_version7grpc_cpp1.45python3.10.____cpython.log
• https://gist.github.com/xhochy/dbe3865a2da84ee84129ff9a02c38de0 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_versionNonecxx_compiler_version10grpc_cpp1.45python3.9.____cpython.log
• https://gist.github.com/xhochy/c7a1e4370e8fbe702c5882b8d4310537 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_versionNonecxx_compiler_version10grpc_cpp1.45python3.8.____cpython.log
• https://gist.github.com/xhochy/513d6d6eacd05121fc4006e060db5ac7 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_versionNonecxx_compiler_version10grpc_cpp1.45python3.7.____cpython.log
• https://gist.github.com/xhochy/e70b54819a271c26cae6c96b3e642b67 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_versionNonecxx_compiler_version10grpc_cpp1.45python3.10.____cpython.log
• https://gist.github.com/xhochy/6c73c143cb2751fc9b8127d51ace1191 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.2cxx_compiler_version10grpc_cpp1.45python3.9.____cpython.log
• https://gist.github.com/xhochy/aed58aae3440d74e2ceb646319e9f671 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.2cxx_compiler_version10grpc_cpp1.45python3.8.____cpython.log
• https://gist.github.com/xhochy/4f886c6217293b25189a99a8b78ffae1 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.2cxx_compiler_version10grpc_cpp1.45python3.7.____cpython.log
• https://gist.github.com/xhochy/341d0d1b0103e7e3d5da4668a351c1d7 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.2cxx_compiler_version10grpc_cpp1.45python3.10.____cpython.log
• https://gist.github.com/xhochy/e7125faeb733b7ed92f190694dafdc7a linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.1cxx_compiler_version10grpc_cpp1.45python3.9.____cpython.log
• https://gist.github.com/xhochy/98bce59a4353fff96df29fa132a96d95 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.1cxx_compiler_version10grpc_cpp1.45python3.8.____cpython.log
• https://gist.github.com/xhochy/4fc417100f67a6aaf0e63da3cab46873 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.1cxx_compiler_version10grpc_cpp1.45python3.7.____cpython.log
• https://gist.github.com/xhochy/f2d4dd0e84b647430afe54f78e080dd9 linux_64_abseil_cpp20210324.2c_compiler_version10cuda_compiler_version11.1cxx_compiler_version10grpc_cpp1.45python3.10.____cpython.log

[ngam]

🚨 https://github.com/tensorflow/tensorflow/releases/tag/v2.9.1
https://github.com/tensorflow/tensorflow/releases

Looks like tensorflow isn't supposed to work with protobuf 3.20... but it is passing our tests somehow??

    'protobuf >= 3.9.2',
    # TODO(b/182876485): Protobuf 3.20 results in linker errors on Windows
    # Protobuf 4.0 is binary incompatible with what C++ TF uses.
    # We need ~1 quarter to update properly.
    # See also: https://github.com/tensorflow/tensorflow/issues/53234
    # See also: https://github.com/protocolbuffers/protobuf/issues/9954
    # See also: https://github.com/tensorflow/tensorflow/issues/56077
    # This is a temporary patch for now, to patch previous TF releases.
    'protobuf >= 3.9.2, < 3.20',

but it seems only applicable to windows...? just flagging this here for your consideration before finalizing this

[hmaarrfk]

"Luckily" for us we don't even package windows!

[njzjz]

3.20 is OK for us, but per tensorflow/tensorflow#56077, protobuf 4.0 (not released yet) will break tensorflow. We may add <4.0:

    'protobuf >= 3.9.2, < 4.0',

[hmaarrfk]

This isn't necessary as per protobuf and libprotobuf are pinned together
https://github.com/conda-forge/protobuf-feedstock/blob/main/recipe/meta.yaml#L29

[hmaarrfk]

thank you @xhochy

[ngam]

perfect 😅

thanks all, I was quite confused by their decision to release a patch for all releases but the note indicated it was only windows, and so I wasn't sure if this was needed beyond windows

[ngam]

we march forward, our linux (including cuda) builds are ready-ish, but not sure about the osx ones yet #240