Crash/Abort on Import (Due to Overlinking?)

[mharradon]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

As I was working through running the TensorFlow test suite for #489 I found that there was a crash on import. I since reproduced the issue on the released 2.19.1 builds cpu_py312h5fe9c63_55 and cuda128py311he515fd5_255. I've attached my installed packages for the CPU variant.

Example output:

>>> from tensorflow.python.eager import profiler_client
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1779051781.381708    1359 descriptor_database.cc:678] File already exists in database: tensorflow/core/example/feature.proto
F0000 00:00:1779051781.381785    1359 descriptor.cc:2523] Check failed: GeneratedDatabase()->Add(encoded_file_descriptor, size)
*** Check failure stack trace: ***
[1]    1359 IOT instruction (core dumped)  python

My understanding is that this is an issue due to overlinking where a protobuf header is imported multiple times.

I am hopeful @hmaarrfk may have already fixed this issue on #491. The import may serve as a useful test for future builds.

Thanks!

Installed packages

# Name                     Version          Build                    Channel
_openmp_mutex              4.5              20_gnu                   conda-forge
_x86_64-microarch-level    3                3_x86_64_v3              conda-forge
absl-py                    2.4.0            pyhd8ed1ab_0             conda-forge
astunparse                 1.6.3            pyhd8ed1ab_3             conda-forge
aws-c-auth                 0.10.1           ha62d5e7_3               conda-forge
aws-c-cal                  0.9.13           h2c9d079_1               conda-forge
aws-c-common               0.12.6           hb03c661_0               conda-forge
aws-c-compression          0.3.2            h8b1a151_0               conda-forge
aws-c-http                 0.10.13          h4bacb7b_0               conda-forge
aws-c-io                   0.26.3           h692f434_1               conda-forge
aws-c-s3                   0.12.2           he6ee468_1               conda-forge
aws-c-sdkutils             0.2.4            h8b1a151_4               conda-forge
aws-checksums              0.2.10           h8b1a151_0               conda-forge
backports.zstd             1.5.0            py312h90b7ffd_0          conda-forge
brotli-python              1.2.0            py312hdb49522_1          conda-forge
bzip2                      1.0.8            hda65f42_9               conda-forge
c-ares                     1.34.6           hb03c661_0               conda-forge
ca-certificates            2026.4.22        hbd8a1cb_0               conda-forge
cached-property            1.5.2            hd8ed1ab_1               conda-forge
cached_property            1.5.2            pyha770c72_1             conda-forge
certifi                    2026.4.22        pyhd8ed1ab_0             conda-forge
charset-normalizer         3.4.7            pyhd8ed1ab_0             conda-forge
flatbuffers                25.9.23          hb7d4c21_0               conda-forge
gast                       0.7.0            pyhd8ed1ab_0             conda-forge
giflib                     5.2.2            hd590300_0               conda-forge
google-pasta               0.2.0            pyhd8ed1ab_2             conda-forge
grpcio                     1.78.1           py312h39ee1c6_0          conda-forge
h2                         4.3.0            pyhcf101f3_0             conda-forge
h5py                       3.16.0           nompi_py312ha829cd9_102  conda-forge
hdf5                       2.1.0            nompi_h87a9417_105       conda-forge
hpack                      4.1.0            pyhd8ed1ab_0             conda-forge
hyperframe                 6.1.0            pyhd8ed1ab_0             conda-forge
icu                        78.3             h33c6efd_0               conda-forge
idna                       3.13             pyhcf101f3_0             conda-forge
importlib-metadata         8.8.0            pyhcf101f3_0             conda-forge
keras                      3.14.1           pyh753f3f9_0             conda-forge
keyutils                   1.6.3            hb9d3cd8_0               conda-forge
krb5                       1.22.2           ha1258a1_0               conda-forge
ld_impl_linux-64           2.45.1           default_hbd61a6d_102     conda-forge
libabseil                  20260107.1       cxx17_h7b12aa8_0         conda-forge
libaec                     1.1.5            h088129d_0               conda-forge
libblas                    3.11.0           7_h4a7cf45_openblas      conda-forge
libcblas                   3.11.0           7_h0358290_openblas      conda-forge
libcurl                    8.20.0           hcf29cc6_0               conda-forge
libedit                    3.1.20250104     pl5321h7949ede_0         conda-forge
libev                      4.33             hd590300_2               conda-forge
libexpat                   2.8.0            hecca717_0               conda-forge
libffi                     3.5.2            h3435931_0               conda-forge
libgcc                     15.2.0           he0feb66_19              conda-forge
libgcc-ng                  15.2.0           h69a702a_19              conda-forge
libgfortran                15.2.0           h69a702a_19              conda-forge
libgfortran5               15.2.0           h68bc16d_19              conda-forge
libgomp                    15.2.0           he0feb66_19              conda-forge
libgrpc                    1.78.1           h1d1128b_0               conda-forge
libjpeg-turbo              3.1.4.1          hb03c661_0               conda-forge
liblapack                  3.11.0           7_h47877c9_openblas      conda-forge
liblzma                    5.8.3            hb03c661_0               conda-forge
libml_dtypes-headers       0.5.4            h707e725_0               conda-forge
libnghttp2                 1.68.1           h877daf1_0               conda-forge
libnsl                     2.0.1            hb9d3cd8_1               conda-forge
libopenblas                0.3.33           pthreads_h94d23a6_0      conda-forge
libpng                     1.6.58           h421ea60_0               conda-forge
libprotobuf                6.33.5           h2b00c02_0               conda-forge
libre2-11                  2025.11.05       h0dc7533_1               conda-forge
libsqlite                  3.53.1           h0c1763c_0               conda-forge
libssh2                    1.11.1           hcf80075_0               conda-forge
libstdcxx                  15.2.0           h934c35e_19              conda-forge
libtensorflow_cc           2.19.1           cpu_h944eb50_55          conda-forge
libtensorflow_framework    2.19.1           cpu_he6e9716_55          conda-forge
libuuid                    2.42             h5347b49_0               conda-forge
libxcrypt                  4.4.36           hd590300_1               conda-forge
libzlib                    1.3.2            h25fd6f3_2               conda-forge
markdown                   3.10.2           pyhcf101f3_0             conda-forge
markdown-it-py             4.2.0            pyhd8ed1ab_0             conda-forge
markupsafe                 3.0.3            py312h8a5da7c_1          conda-forge
mdurl                      0.1.2            pyhd8ed1ab_1             conda-forge
ml_dtypes                  0.5.4            np2py312h0f77346_1       conda-forge
namex                      0.1.0            pyhd8ed1ab_0             conda-forge
ncurses                    6.6              hdb14827_0               conda-forge
numpy                      2.4.5            py312h33ff503_0          conda-forge
openssl                    3.6.2            h35e630c_0               conda-forge
opt_einsum                 3.4.0            pyhd8ed1ab_1             conda-forge
optree                     0.19.1           py312hd9148b4_0          conda-forge
packaging                  26.2             pyhc364b38_0             conda-forge
pip                        26.1.1           pyh8b19718_0             conda-forge
protobuf                   6.33.5           py312ha7b3241_2          conda-forge
pygments                   2.20.0           pyhd8ed1ab_0             conda-forge
pysocks                    1.7.1            pyha55dd90_7             conda-forge
python                     3.12.13          hd63d673_0_cpython       conda-forge
python-flatbuffers         25.9.23          pyh1e1bc0e_0             conda-forge
python_abi                 3.12             8_cp312                  conda-forge
re2                        2025.11.05       h5301d42_1               conda-forge
readline                   8.3              h853b02a_0               conda-forge
requests                   2.34.2           pyhcf101f3_0             conda-forge
rich                       15.0.0           pyhcf101f3_0             conda-forge
s2n                        1.7.2            hc5a330e_1               conda-forge
setuptools                 82.0.1           pyh332efcf_0             conda-forge
six                        1.17.0           pyhe01879c_1             conda-forge
snappy                     1.2.2            h03e3b7b_1               conda-forge
tensorboard                2.19.0           pyhd8ed1ab_0             conda-forge
tensorboard-data-server    0.7.0            py312h4eba8b5_4          conda-forge
tensorflow                 2.19.1           cpu_py312h5fe9c63_55     conda-forge
tensorflow-base            2.19.1           cpu_py312h0c7c805_55     conda-forge
termcolor                  3.3.0            pyhd8ed1ab_0             conda-forge
tk                         8.6.13           noxft_h366c992_103       conda-forge
typing-extensions          4.15.0           h396c80c_0               conda-forge
typing_extensions          4.15.0           pyhcf101f3_0             conda-forge
tzdata                     2025c            hc9c84f9_1               conda-forge
urllib3                    2.7.0            pyhd8ed1ab_0             conda-forge
werkzeug                   3.1.8            pyhcf101f3_0             conda-forge
wheel                      0.47.0           pyhd8ed1ab_0             conda-forge
wrapt                      2.1.2            py312h4c3975b_0          conda-forge
zipp                       3.23.1           pyhcf101f3_0             conda-forge
zstd                       1.5.7            hb78ec9c_6               conda-forge

Environment info

libmamba version : 2.3.3
          mamba version : 2.3.3
           curl version : libcurl/8.16.0 OpenSSL/3.6.2 zlib/1.3.2 zstd/1.5.7 libidn2/2.3.8 libssh2/1.11.1 nghttp2/1.58.0
     libarchive version : libarchive 3.8.7 zlib/1.3.2 liblzma/5.8.3 bz2lib/1.0.8 liblz4/1.9.4 libzstd/1.5.7 libxml2/2.14.4 openssl/3.5.6 libb2/bundled
       envs directories : /home/mharradon/.local/share/mamba/envs
          package cache : /home/mharradon/.local/share/mamba/pkgs
                          /home/mharradon/.mamba/pkgs
            environment : tf-testcpu (active)
           env location : /home/mharradon/.local/share/mamba/envs/tf-testcpu
      user config files : /home/mharradon/.mambarc
 populated config files : /home/mharradon/.condarc
       virtual packages : __unix=0=0
                          __linux=6.6.114=0
                          __glibc=2.35=0
                          __archspec=1=x86_64_v3
                          __cuda=13.1=0
               channels : https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
       base environment : /home/mharradon/.local/share/mamba
               platform : linux-64

[hmaarrfk]

Have you tried to add the CIs pull request you have?

Does it fail on our CIs

[ELundby45]

Note, profiler_client was removed from python/eager in 2.21

tensorflow/tensorflow@38379c1

[mharradon]

Ah, good catch @ELundby45! If I can find a reproducer that persists through 2.21 I'll update.

I see the crash in my local CUDA 12.9 build as well as the two released builds of 2.19.1 (CPU + CUDA 12.8).

I believe the test would fail post-build in CI - i.e. just adding from tensorflow.python.eager import profiler_client to test_tensorflow.py.

Running the full TF python test suite just requires running pytest tensorflow/python from the TensorFlow repo. I encountered the crash immediately - I'd like to have some sense of the runtime of that before suggesting conda-forge runs that suite in builds.

I'm currently testing a fix that removes _pywrap_cpu_feature_guard.so from pywrap_tensorflow_binaries packaging and distributes independently. No idea if this is a useful direction - if you would like me to push my attempts somewhere let me know. I'll update if I have any success.

Oh, are you asking if it works on #491? I have not gotten to testing that yet - I only noticed there are changes there that seem potentially related. I'll try that soon if I can (once I can free up my build machine!).

[isuruf]

It's a known crash that we couldn't figure out in #433. I spent quite a bit of hours into it and couldn't figure it out. At the end we decided it was not worth it to hold the 2.19 release.