Unable to run e2e_smoke_test successfully across multiple environments

[kmiskowicz]

Hi Circuit Training Team,
I’m a Master's student currently working on my thesis, where I aim to use your circuit-training project as a core component. Unfortunately, despite many attempts on different environments, I have not been able to successfully run even the basic e2e_smoke_test. I would be very grateful for any guidance or suggestions to help me move forward.
Here’s a summary of the environments I’ve tried and the issues encountered:

Local Machine 1
[MacOS 15.5, 16GB RAM, 8CPU, Chipset Model: Apple M3]
Method:

• Docker build (according to Preliminary Setup and Using the docker instructions) + interactive session

Problem:

• The Docker image builds but fails to install Python packages properly: for example TensorFlow is missing inside the container.

% docker run -it --rm -v ${REPO_ROOT}:/workspace --workdir /workspace circuit_training_cpu:r0.0.4 bash
root@8876760dabee:/workspace# python3.9 -m circuit_training.environment.environment_test
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/workspace/circuit_training/environment/environment_test.py", line 20, in <module>
    from circuit_training.environment import environment
  File "/workspace/circuit_training/environment/environment.py", line 24, in <module>
    from circuit_training.dreamplace import dreamplace_core
  File "/workspace/circuit_training/dreamplace/dreamplace_core.py", line 19, in <module>
    from circuit_training.dreamplace import dreamplace_util
  File "/workspace/circuit_training/dreamplace/dreamplace_util.py", line 23, in <module>
    import circuit_training.environment.placement_util as util
  File "/workspace/circuit_training/environment/placement_util.py", line 30, in <module>
    import tensorflow.io.gfile as gfile
ModuleNotFoundError: No module named 'tensorflow'

• Even if I tried to install it manually in the interactive session I got the following error (DREAMPlace submodule not found):
• Problem occurred for multiple stable branches (r0.0.4, r0.0.3)
• Docker version 28.0.4

Local Machine 2
[MacOS 14.7, 8GB RAM, 8CPU, Chipset Model: Intel Iris Plus Graphics]
Method:

• Docker build (according to Preliminary Setup and Using the docker instructions) + interactive session/run

Problem:

• The e2e_smoke_test script begins running, but gets stuck on the first training iteration.
• After several minutes (or sometimes hours), a SoftMacroPlacer place() timed out error appears.

console log:

docker run --rm -v ${REPO_ROOT}:/workspace --workdir /workspace circuit_training_gpu:core \
    bash tools/e2e_smoke_test.sh --root_dir /workspace/logs_toy_run --netlist_file /workspace/circuit_training/environment/test_data/toy_macro_stdcell/netlist.pb.txt --init_place /workspace/circuit_training/environment/test_data/toy_macro_stdcell/initial.plc
==========
== CUDA ==
==========
CUDA Version 11.8.0
Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .
--root_dir
/workspace/logs_toy_run
--netlist_file
/workspace/circuit_training/environment/test_data/toy_macro_stdcell/netlist.pb.txt
--init_place
/workspace/circuit_training/environment/test_data/toy_macro_stdcell/initial.plc
FYI: Local logs (--script_logs) cannot write to gcs. It is just a pipe.
Reverb server set to 127.0.0.1:8008
Starting Reveb Server in the background.
Logging reverb job  to /workspace/logs_toy_run/reverb.log.
Starting 4 collect jobs.
Start collect job 1 in the background...
Logging collect job 1 to /workspace/logs_toy_run/collect_1.log.
Start collect job 2 in the background...
Logging collect job 2 to /workspace/logs_toy_run/collect_2.log.
Start collect job 3 in the background...
Logging collect job 3 to /workspace/logs_toy_run/collect_3.log.
Start collect job 4 in the background...
Logging collect job 4 to /workspace/logs_toy_run/collect_4.log.
Start Training job in the background but logging to console.
It has been ~0m. Sleeping 60s waiting for error or end.
2025-05-20 10:07:46.699717: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-20 10:07:46.850638: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 10:07:47.674460: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-20 10:07:47.675404: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-20 10:07:47.821445: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-20 10:07:48.167014: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 10:07:48.171158: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-20 10:07:54.144668: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I0520 10:08:18.921119 140469920101504 train_ppo.py:292] global seed=111
W0520 10:08:18.935855 140469920101504 placement_util.py:249] block_name is not set. Please add the block_name in:
/workspace/circuit_training/environment/test_data/toy_macro_stdcell/netlist.pb.txt
or in:
/workspace/circuit_training/environment/test_data/toy_macro_stdcell/initial.plc
I0520 10:08:19.968621 140469920101504 environment.py:220] ***Num node to place***:2
I0520 10:08:19.974783 140469920101504 placement_util.py:508] node_order: descending_size_macro_first
I0520 10:08:19.990747 140469920101504 dreamplace_util.py:105] Update num_bins_x and num_bins_y: (256, 64)
I0520 10:08:20.185451 140469920101504 plc_converter.py:141] Node 7 is placed at (10.000000, 25.000000).
I0520 10:08:20.189099 140469920101504 plc_converter.py:141] Node 9 is placed at (10.000000, 50.000000).
I0520 10:08:20.198277 140469920101504 plc_converter.py:141] Node 11 is placed at (10.000000, 75.000000).
I0520 10:08:20.202397 140469920101504 plc_converter.py:141] Node 13 is placed at (190.000000, 25.000000).
I0520 10:08:20.209626 140469920101504 plc_converter.py:141] Node 15 is placed at (190.000000, 50.000000).
I0520 10:08:21.448081 140469920101504 PlaceDB.py:898] 
================================= Benchmark Statistics =================================
#nodes = 26, #terminals = 14, # terminal_NIs = 0, #movable = 12, #nets = 20
die area = (0, 0, 200, 100) 20000
row height = 10, site width = 10
num_bins = 256x64, bin sizes = 0.078125x0.15625
#pins = 50, #movable_pins = 30
total_movable_node_area = 24, total_fixed_node_area = 2400, total_space_area = 17600
utilization = 0.00136364, target_density = 0.85
use placeable_area = 17600 to compute fillers
total_filler_node_area = 14936, #fillers = 1494, filler sizes = 1x10
========================================================================================
I0520 10:08:21.457463 140469920101504 PlaceDB.py:593] reading benchmark takes 0.0320883 seconds
I0520 10:08:21.461522 140469920101504 environment.py:253] Run DP mix-sized to initialize the locations.
I0520 10:08:21.462296 140469920101504 placedb_plc.py:56] Reinitialized the PlaceDB.
I0520 10:08:21.467913 140469920101504 PlaceDB.py:898] 
================================= Benchmark Statistics =================================
#nodes = 26, #terminals = 12, # terminal_NIs = 0, #movable = 14, #nets = 20
die area = (0, 0, 200, 100) 20000
row height = 10, site width = 10
num_bins = 256x64, bin sizes = 0.078125x0.15625
#pins = 50, #movable_pins = 38
total_movable_node_area = 2424, total_fixed_node_area = 0, total_space_area = 17600
utilization = 0.137727, target_density = 0.85
use placeable_area = 20000 to compute fillers
total_filler_node_area = 14576, #fillers = 451, filler sizes = 3.23077x10
========================================================================================
I0520 10:08:21.468516 140469920101504 PlaceDB.py:593] reading benchmark takes 0.0024755 seconds
I0520 10:08:21.506047 140469920101504 BasicPlace.py:276] move cells to the center of layout with random noise
W0520 10:08:22.081337 140469920101504 BasicPlace.py:145] net weights are all the same, ignored
I0520 10:08:22.308836 140469920101504 PlaceObj.py:711] 256x64 bins #bins 256x64, bin sizes 0.078125x0.15625, max_num_bins = 41, padding = 0
I0520 10:08:22.310585 140469920101504 electric_potential.py:436] regard 0 cells as movable macros in global placement
W0520 10:08:22.334776 140469920101504 PlaceObj.py:864] quadratic density penalty is disabled, density weight update is forced to be based on HPWL
I0520 10:08:22.350704 140469920101504 NonLinearPlace.py:104] use nesterov optimizer
I0520 10:08:22.351149 140469920101504 NonLinearPlace.py:141] add 2.5% noise
I0520 10:08:22.903844 140469920101504 electric_potential.py:478] fixed density map: average 0, max 0, bin area 1.2207
I0520 10:08:25.423524 140469920101504 NonLinearPlace.py:149] nesterov initialization takes 3.2239 seconds
I0520 10:08:26.964675 140469920101504 NonLinearPlace.py:270] density_weight = 6.830810E-09
I0520 10:08:30.594659 140469920101504 NonLinearPlace.py:311] optimizer step 3356.983 ms
I0520 10:08:30.594987 140469920101504 NonLinearPlace.py:318] iteration    0, (   0,  0,  0), Obj 4.504703E+02, DensityWeight 0.000000E+00, HPWL 1.412111E+03, Overflow 5.582269E-01, MaxDensity 1.130E+01, gamma 9.375000E+01, time 270.486ms
I0520 10:08:30.595273 140469920101504 NonLinearPlace.py:327] full step 5170.683 ms
It has been ~1m. Sleeping 60s waiting for error or end.
I0520 10:08:43.993257 140469920101504 NonLinearPlace.py:311] optimizer step 13258.990 ms
...
I0520 10:08:52.839416 140469920101504 NonLinearPlace.py:318] iteration    4, (   4,  0,  0), Obj 9.792181E+02, DensityWeight 7.900402E-09, HPWL 1.039788E+03, Overflow 1.425231E-01, MaxDensity 1.819E+00, gamma 3.167984E+00, time 174.026ms
I0520 10:08:52.846931 140469920101504 NonLinearPlace.py:327] full step 5984.530 ms
I0520 10:08:52.848018 140469920101504 NonLinearPlace.py:609] optimizer nesterov takes 30.648 seconds
I0520 10:08:52.858402 140469920101504 dreamplace_core.py:50] Last Dreamplace metric: iteration    4, (   4,  0,  0), Obj 9.792181E+02, DensityWeight 8.291275E-09, HPWL 1.039788E+03, Overflow 1.425231E-01, MaxDensity 1.819E+00, gamma 1.165379E+00, time 174.026ms
/usr/local/lib/python3.9/dist-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.9/dist-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in divide
  ret = ret.dtype.type(ret / rcount)
I0520 10:08:52.934178 140469920101504 observation_extractor.py:349] Pad a tensor with shape (0,) by 42000
I0520 10:08:52.938057 140469920101504 observation_extractor.py:363] Pad a tensor with shape (14,) by 3500
I0520 10:09:00.807315 140469920101504 train_ppo_lib.py:212] Initialize iteration at: init_iteration 0.
I0520 10:09:00.815870 140469920101504 train_ppo_lib.py:227] Initialize train_step at 0
I0520 10:09:12.448040 140469920101504 signature_serialization.py:156] Function `function_with_signature` contains input name(s) 0/step_type, 0/reward, 0/discount, 0/observation/current_node, 0/observation/fake_net_heatmap, 0/observation/is_node_placed, 0/observation/locations_x, 0/observation/locations_y, 0/observation/mask, 0/observation/netlist_index, 12487, 12489, 12491, 12493, 12495, 12497, 12499, 12501, 12503, 12505, 12507, 12509, 12511, 12513, 12515, 12517, 12519, 12521, 12523, 12525, 12527, 12529, 12531, 12533, 12535, 12537, 12539, 12541, 12543, 12545, 12547, 12549, 12551, 12553 with unsupported characters which will be renamed to step_type, reward, discount, observation_current_node, observation_fake_net_heatmap, observation_is_node_placed, observation_locations_x, observation_locations_y, observation_mask, observation_netlist_index, unknown, unknown_0, unknown_1, unknown_2, unknown_3, unknown_4, unknown_5, unknown_6, unknown_7, unknown_8, unknown_9, unknown_10, unknown_11, unknown_12, unknown_13, unknown_14, unknown_15, unknown_16, unknown_17, unknown_18, unknown_19, unknown_20, unknown_21, unknown_22, unknown_23, unknown_24, unknown_25, unknown_26, unknown_27, unknown_28, unknown_29, unknown_30, unknown_31, unknown_32 in the SavedModel.
W0520 10:09:12.449027 140469920101504 function_type.py:509] `0/step_type` is not a valid tf.function parameter name. Sanitizing to `arg_0_step_type`.
W0520 10:09:12.449244 140469920101504 function_type.py:509] `0/reward` is not a valid tf.function parameter name. Sanitizing to `arg_0_reward`.
W0520 10:09:12.449594 140469920101504 function_type.py:509] `0/discount` is not a valid tf.function parameter name. Sanitizing to `arg_0_discount`.
W0520 10:09:12.449714 140469920101504 function_type.py:509] `0/observation/current_node` is not a valid tf.function parameter name. Sanitizing to `arg_0_observation_current_node`.
W0520 10:09:12.449866 140469920101504 function_type.py:509] `0/observation/fake_net_heatmap` is not a valid tf.function parameter name. Sanitizing to `arg_0_observation_fake_net_heatmap`.
I0520 10:09:12.661036 140469920101504 signature_serialization.py:156] Function `function_with_signature` contains input name(s) 12652 with unsupported characters which will be renamed to unknown in the SavedModel.
I0520 10:09:17.556622 140469920101504 save.py:289] Found untraced functions such as GrlPolicyModel_layer_call_fn, GrlPolicyModel_layer_call_and_return_conditional_losses, grl_value_model_layer_call_fn, grl_value_model_layer_call_and_return_conditional_losses, grl_model_layer_call_fn while saving (showing 5 of 21). These functions will not be directly callable after loading.
/usr/local/lib/python3.9/dist-packages/tensorflow/python/saved_model/nested_structure_coder.py:458: UserWarning: Encoding a StructuredValue with type tfp.distributions.Categorical_ACTTypeSpec; loading this StructuredValue will require that this type be imported and registered.
  warnings.warn("Encoding a StructuredValue with type %s; loading this "
INFO:tensorflow:Assets written to: /workspace/logs_toy_run/111/policies/policy/assets
I0520 10:09:18.826185 140469920101504 builder_impl.py:801] Assets written to: /workspace/logs_toy_run/111/policies/policy/assets
I0520 10:09:18.867345 140469920101504 fingerprinting_utils.py:49] Writing fingerprint to /workspace/logs_toy_run/111/policies/policy/fingerprint.pb
I0520 10:09:20.752289 140469920101504 signature_serialization.py:156] Function `function_with_signature` contains input name(s) 0/step_type, 0/reward, 0/discount, 0/observation/current_node, 0/observation/fake_net_heatmap, 0/observation/is_node_placed, 0/observation/locations_x, 0/observation/locations_y, 0/observation/mask, 0/observation/netlist_index, 19749, 19751, 19753, 19755, 19757, 19759, 19761, 19763, 19765, 19767, 19769, 19771, 19773, 19775, 19777, 19779, 19781, 19783, 19785, 19787, 19789, 19791, 19793, 19795, 19797, 19799, 19801, 19803, 19805, 19807, 19809, 19811, 19813, 19815 with unsupported characters which will be renamed to step_type, reward, discount, observation_current_node, observation_fake_net_heatmap, observation_is_node_placed, observation_locations_x, observation_locations_y, observation_mask, observation_netlist_index, unknown, unknown_0, unknown_1, unknown_2, unknown_3, unknown_4, unknown_5, unknown_6, unknown_7, unknown_8, unknown_9, unknown_10, unknown_11, unknown_12, unknown_13, unknown_14, unknown_15, unknown_16, unknown_17, unknown_18, unknown_19, unknown_20, unknown_21, unknown_22, unknown_23, unknown_24, unknown_25, unknown_26, unknown_27, unknown_28, unknown_29, unknown_30, unknown_31, unknown_32 in the SavedModel.
I0520 10:09:20.985277 140469920101504 signature_serialization.py:156] Function `function_with_signature` contains input name(s) 19922 with unsupported characters which will be renamed to unknown in the SavedModel.
I0520 10:09:33.492813 140469920101504 save.py:289] Found untraced functions such as GrlPolicyModel_layer_call_fn, GrlPolicyModel_layer_call_and_return_conditional_losses, grl_value_model_layer_call_fn, grl_value_model_layer_call_and_return_conditional_losses, grl_model_layer_call_fn while saving (showing 5 of 21). These functions will not be directly callable after loading.
/usr/local/lib/python3.9/dist-packages/tensorflow/python/saved_model/nested_structure_coder.py:458: UserWarning: Encoding a StructuredValue with type tfp.distributions.Categorical_ACTTypeSpec; loading this StructuredValue will require that this type be imported and registered.
  warnings.warn("Encoding a StructuredValue with type %s; loading this "
INFO:tensorflow:Assets written to: /workspace/logs_toy_run/111/policies/collect_policy/assets
I0520 10:09:35.469530 140469920101504 builder_impl.py:801] Assets written to: /workspace/logs_toy_run/111/policies/collect_policy/assets
I0520 10:09:35.722664 140469920101504 fingerprinting_utils.py:49] Writing fingerprint to /workspace/logs_toy_run/111/policies/collect_policy/fingerprint.pb
I0520 10:09:37.261806 140469920101504 signature_serialization.py:156] Function `function_with_signature` contains input name(s) 0/step_type, 0/reward, 0/discount, 0/observation/current_node, 0/observation/fake_net_heatmap, 0/observation/is_node_placed, 0/observation/locations_x, 0/observation/locations_y, 0/observation/mask, 0/observation/netlist_index, 28961, 28963, 28965, 28967, 28969, 28971, 28973, 28975, 28977, 28979, 28981, 28983, 28985, 28987, 28989, 28991, 28993, 28995, 28997, 28999, 29001, 29003, 29005, 29007, 29009, 29011, 29013, 29015, 29017, 29019, 29021, 29023, 29025, 29027 with unsupported characters which will be renamed to step_type, reward, discount, observation_current_node, observation_fake_net_heatmap, observation_is_node_placed, observation_locations_x, observation_locations_y, observation_mask, observation_netlist_index, unknown, unknown_0, unknown_1, unknown_2, unknown_3, unknown_4, unknown_5, unknown_6, unknown_7, unknown_8, unknown_9, unknown_10, unknown_11, unknown_12, unknown_13, unknown_14, unknown_15, unknown_16, unknown_17, unknown_18, unknown_19, unknown_20, unknown_21, unknown_22, unknown_23, unknown_24, unknown_25, unknown_26, unknown_27, unknown_28, unknown_29, unknown_30, unknown_31, unknown_32 in the SavedModel.
I0520 10:09:37.418963 140469920101504 signature_serialization.py:156] Function `function_with_signature` contains input name(s) 29126 with unsupported characters which will be renamed to unknown in the SavedModel.
It has been ~2m. Sleeping 60s waiting for error or end.
I0520 10:09:47.383197 140469920101504 save.py:289] Found untraced functions such as GrlPolicyModel_layer_call_fn, GrlPolicyModel_layer_call_and_return_conditional_losses, grl_value_model_layer_call_fn, grl_value_model_layer_call_and_return_conditional_losses, grl_model_layer_call_fn while saving (showing 5 of 21). These functions will not be directly callable after loading.
/usr/local/lib/python3.9/dist-packages/tensorflow/python/saved_model/nested_structure_coder.py:458: UserWarning: Encoding a StructuredValue with type tfp.distributions.Deterministic_ACTTypeSpec; loading this StructuredValue will require that this type be imported and registered.
  warnings.warn("Encoding a StructuredValue with type %s; loading this "
INFO:tensorflow:Assets written to: /workspace/logs_toy_run/111/policies/greedy_policy/assets
I0520 10:09:50.415745 140469920101504 builder_impl.py:801] Assets written to: /workspace/logs_toy_run/111/policies/greedy_policy/assets
I0520 10:09:50.486146 140469920101504 fingerprinting_utils.py:49] Writing fingerprint to /workspace/logs_toy_run/111/policies/greedy_policy/fingerprint.pb
It has been ~3m. Sleeping 60s waiting for error or end.
It has been ~4m. Sleeping 60s waiting for error or end.
I0520 10:11:44.087607 140469920101504 common.py:1073] No checkpoint available at /workspace/logs_toy_run/111/train/checkpoints
I0520 10:11:44.718514 140469920101504 train_ppo_lib.py:320] Created dataset for training_table_0
I0520 10:12:40.046255 140469920101504 learner.py:167] train steps per iteration: 536
I0520 10:12:40.048724 140469920101504 train_ppo_lib.py:357] Training. Iteration: 0
It has been ~5m. Sleeping 60s waiting for error or end.
It has been ~6m. Sleeping 60s waiting for error or end.
It has been ~7m. Sleeping 60s waiting for error or end.
It has been ~8m. Sleeping 60s waiting for error or end.
It has been ~9m. Sleeping 60s waiting for error or end.
It has been ~10m. Sleeping 60s waiting for error or end.
It has been ~11m. Sleeping 60s waiting for error or end.
It has been ~12m. Sleeping 60s waiting for error or end.
It has been ~13m. Sleeping 60s waiting for error or end.
It has been ~14m. Sleeping 60s waiting for error or end.
It has been ~15m. Sleeping 60s waiting for error or end.
It has been ~16m. Sleeping 60s waiting for error or end.
It has been ~17m. Sleeping 60s waiting for error or end.
It has been ~18m. Sleeping 60s waiting for error or end.
Collect job failed (SIGUSR1). Check /workspace/logs_toy_run/collect_*.log.
Exiting with code 8.

reverb log:

2025-05-20 10:07:46.699717: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-20 10:07:46.850638: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 10:07:47.674942: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-20 10:07:47.676432: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-20 10:07:47.821384: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-20 10:07:48.166757: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 10:07:48.171159: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-20 10:07:54.145639: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I0520 10:08:08.165468 139836571169920 train_utils.py:236] Waiting for `wait_predicate_fn`. Block execution. Sleeping for 2 seconds.
<42 same lines>
I0520 10:09:34.846357 139836571169920 train_utils.py:236] Waiting for `wait_predicate_fn`. Block execution. Sleeping for 2 seconds.
I0520 10:09:36.854888 139836571169920 train_utils.py:191] Checking for file /workspace/logs_toy_run/111/policies/collect_policy/saved_model.pb (not found)
I0520 10:09:36.856716 139836571169920 train_utils.py:250] The `wait_predicate_fn` returned `False`. Continue execution.
I0520 10:09:48.613471 139836571169920 ppo_reverb_server_lib.py:73] Signature of variables: 
{'policy_variables': ListWrapper([TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(17, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(17, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(17, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(12, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(56, 2048), dtype=tf.float32, name=None), TensorSpec(shape=(2048,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 16, 32), dtype=tf.float32, name=None), TensorSpec(shape=(16,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 8, 16), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 4, 8), dtype=tf.float32, name=None), TensorSpec(shape=(4,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 2, 4), dtype=tf.float32, name=None), TensorSpec(shape=(2,), dtype=tf.float32, name=None), TensorSpec(shape=(3, 3, 1, 2), dtype=tf.float32, name=None), TensorSpec(shape=(1,), dtype=tf.float32, name=None), TensorSpec(shape=(8, 1), dtype=tf.float32, name=None), TensorSpec(shape=(1,), dtype=tf.float32, name=None), TensorSpec(shape=(56, 32), dtype=tf.float32, name=None), TensorSpec(shape=(32,), dtype=tf.float32, name=None), TensorSpec(shape=(32, 8), dtype=tf.float32, name=None), TensorSpec(shape=(8,), dtype=tf.float32, name=None)]), 'train_step': TensorSpec(shape=(), dtype=tf.int64, name=None), 'model_id': TensorSpec(shape=(), dtype=tf.int64, name=None)}
I0520 10:09:48.625273 139836571169920 ppo_reverb_server_lib.py:80] Signature of experience: 
Trajectory(step_type=TensorSpec(shape=(None,), dtype=tf.int32, name='step_type'), observation={'is_node_placed': BoundedTensorSpec(shape=(None, 3500), dtype=tf.int32, name='observation/is_node_placed', minimum=array(0, dtype=int32), maximum=array(1, dtype=int32)), 'mask': BoundedTensorSpec(shape=(None, 16384), dtype=tf.int32, name='observation/mask', minimum=array(0, dtype=int32), maximum=array(1, dtype=int32)), 'netlist_index': BoundedTensorSpec(shape=(None, 1), dtype=tf.int32, name='observation/netlist_index', minimum=array(0, dtype=int32), maximum=array(0, dtype=int32)), 'locations_y': BoundedTensorSpec(shape=(None, 3500), dtype=tf.float32, name='observation/locations_y', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)), 'current_node': BoundedTensorSpec(shape=(None, 1), dtype=tf.int32, name='observation/current_node', minimum=array(0, dtype=int32), maximum=array(3499, dtype=int32)), 'fake_net_heatmap': BoundedTensorSpec(shape=(None, 16384), dtype=tf.float32, name='observation/fake_net_heatmap', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)), 'locations_x': BoundedTensorSpec(shape=(None, 3500), dtype=tf.float32, name='observation/locations_x', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32))}, action=BoundedTensorSpec(shape=(None,), dtype=tf.int64, name='action', minimum=array(0), maximum=array(16383)), policy_info={'value_prediction': TensorSpec(shape=(None,), dtype=tf.float32, name='value_prediction'), 'dist_params': {'logits': TensorSpec(shape=(None, 16384), dtype=tf.float32, name='GrlPolicyModel_logits')}}, next_step_type=TensorSpec(shape=(None,), dtype=tf.int32, name='step_type'), reward=TensorSpec(shape=(None,), dtype=tf.float32, name='reward'), discount=BoundedTensorSpec(shape=(None,), dtype=tf.float32, name='discount', minimum=array(0., dtype=float32), maximum=array(1., dtype=float32)))
[reverb/cc/platform/tfrecord_checkpointer.cc:162]  Initializing TFRecordCheckpointer in /tmp/tmplhpbjsfz.
[reverb/cc/platform/tfrecord_checkpointer.cc:565] Loading latest checkpoint from /tmp/tmplhpbjsfz
[reverb/cc/platform/default/server.cc:71] Started replay server on port 8008

collect job:

2025-05-20 10:07:46.699758: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-20 10:07:46.850654: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 10:07:47.674464: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-20 10:07:47.676453: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-20 10:07:47.821385: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-20 10:07:48.166596: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 10:07:48.171159: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-20 10:07:54.144700: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
W0520 10:08:19.128608 139895627211904 placement_util.py:249] block_name is not set. Please add the block_name in:
/workspace/circuit_training/environment/test_data/toy_macro_stdcell/netlist.pb.txt
or in:
/workspace/circuit_training/environment/test_data/toy_macro_stdcell/initial.plc
I0520 10:08:19.983546 139895627211904 environment.py:220] ***Num node to place***:2
I0520 10:08:20.012463 139895627211904 placement_util.py:508] node_order: descending_size_macro_first
I0520 10:08:20.054955 139895627211904 dreamplace_util.py:105] Update num_bins_x and num_bins_y: (256, 64)
I0520 10:08:20.216633 139895627211904 plc_converter.py:141] Node 7 is placed at (10.000000, 25.000000).
I0520 10:08:20.221116 139895627211904 plc_converter.py:141] Node 9 is placed at (10.000000, 50.000000).
I0520 10:08:20.234340 139895627211904 plc_converter.py:141] Node 11 is placed at (10.000000, 75.000000).
I0520 10:08:20.243139 139895627211904 plc_converter.py:141] Node 13 is placed at (190.000000, 25.000000).
I0520 10:08:20.247792 139895627211904 plc_converter.py:141] Node 15 is placed at (190.000000, 50.000000).
I0520 10:08:21.450447 139895627211904 PlaceDB.py:898]
================================= Benchmark Statistics =================================
#nodes = 26, #terminals = 14, # terminal_NIs = 0, #movable = 12, #nets = 20
die area = (0, 0, 200, 100) 20000
row height = 10, site width = 10
num_bins = 256x64, bin sizes = 0.078125x0.15625
#pins = 50, #movable_pins = 30
total_movable_node_area = 24, total_fixed_node_area = 2400, total_space_area = 17600
utilization = 0.00136364, target_density = 0.85
use placeable_area = 17600 to compute fillers
total_filler_node_area = 14936, #fillers = 1494, filler sizes = 1x10
========================================================================================
I0520 10:08:21.469202 139895627211904 PlaceDB.py:593] reading benchmark takes 0.0477154 seconds
I0520 10:08:21.473025 139895627211904 environment.py:253] Run DP mix-sized to initialize the locations.
I0520 10:08:21.474161 139895627211904 placedb_plc.py:56] Reinitialized the PlaceDB.
I0520 10:08:21.479808 139895627211904 PlaceDB.py:898]
================================= Benchmark Statistics =================================
#nodes = 26, #terminals = 12, # terminal_NIs = 0, #movable = 14, #nets = 20
die area = (0, 0, 200, 100) 20000
row height = 10, site width = 10
num_bins = 256x64, bin sizes = 0.078125x0.15625
#pins = 50, #movable_pins = 38
total_movable_node_area = 2424, total_fixed_node_area = 0, total_space_area = 17600
utilization = 0.137727, target_density = 0.85
use placeable_area = 20000 to compute fillers
total_filler_node_area = 14576, #fillers = 451, filler sizes = 3.23077x10
========================================================================================
I0520 10:08:21.482875 139895627211904 PlaceDB.py:593] reading benchmark takes 0.00434756 seconds
I0520 10:08:21.517792 139895627211904 BasicPlace.py:276] move cells to the center of layout with random noise
W0520 10:08:22.060704 139895627211904 BasicPlace.py:145] net weights are all the same, ignored
I0520 10:08:22.276005 139895627211904 PlaceObj.py:711] 256x64 bins #bins 256x64, bin sizes 0.078125x0.15625, max_num_bins = 41, padding = 0
I0520 10:08:22.337312 139895627211904 electric_potential.py:436] regard 0 cells as movable macros in global placement
W0520 10:08:22.338139 139895627211904 PlaceObj.py:864] quadratic density penalty is disabled, density weight update is forced to be based on HPWL
I0520 10:08:22.347890 139895627211904 NonLinearPlace.py:104] use nesterov optimizer
I0520 10:08:22.350552 139895627211904 NonLinearPlace.py:141] add 2.5% noise
I0520 10:08:22.813476 139895627211904 electric_potential.py:478] fixed density map: average 0, max 0, bin area 1.2207
I0520 10:08:25.621217 139895627211904 NonLinearPlace.py:149] nesterov initialization takes 3.40205 seconds
I0520 10:08:27.249626 139895627211904 NonLinearPlace.py:270] density_weight = 6.347933E-09
I0520 10:08:30.589321 139895627211904 NonLinearPlace.py:311] optimizer step 3151.712 ms
I0520 10:08:30.597144 139895627211904 NonLinearPlace.py:318] iteration    0, (   0,  0,  0), Obj 4.500859E+02, DensityWeight 0.000000E+00, HPWL 1.412990E+03, Overflow 5.582169E-01, MaxDensity 1.035E+01, gamma 9.375000E+01, time 186.910ms
I0520 10:08:30.601036 139895627211904 NonLinearPlace.py:327] full step 4978.452 ms
I0520 10:08:44.785067 139895627211904 NonLinearPlace.py:311] optimizer step 14010.433 ms
...
I0520 10:08:52.375026 139895627211904 NonLinearPlace.py:318] iteration    4, (   4,  0,  0), Obj 9.833179E+02, DensityWeight 7.341915E-09, HPWL 1.052723E+03, Overflow 1.476487E-01, MaxDensity 1.819E+00, gamma 3.805649E+00, time 142.219ms
I0520 10:08:52.375916 139895627211904 NonLinearPlace.py:327] full step 5063.685 ms
I0520 10:08:52.379190 139895627211904 NonLinearPlace.py:609] optimizer nesterov takes 30.160 seconds
I0520 10:08:52.391023 139895627211904 dreamplace_core.py:50] Last Dreamplace metric: iteration    4, (   4,  0,  0), Obj 9.833179E+02, DensityWeight 7.705157E-09, HPWL 1.052723E+03, Overflow 1.476487E-01, MaxDensity 1.819E+00, gamma 1.196347E+00, time 142.219ms
/usr/local/lib/python3.9/dist-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.9/dist-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in divide
  ret = ret.dtype.type(ret / rcount)
I0520 10:08:52.478971 139895627211904 observation_extractor.py:349] Pad a tensor with shape (0,) by 42000
I0520 10:08:52.523804 139895627211904 observation_extractor.py:363] Pad a tensor with shape (14,) by 3500
W0520 10:09:00.786846 139895627211904 ppo_agent.py:332] Only tf.keras.optimizers.Optimiers are well supported, got a non-TF2 optimizer: None
I0520 10:10:15.493735 139895627211904 ppo_collect_lib.py:144] Collecting at model_id: 0
I0520 10:10:22.383919 139895627211904 placedb_plc.py:56] Reinitialized the PlaceDB.
I0520 10:10:22.394182 139895627211904 PlaceDB.py:898]
================================= Benchmark Statistics =================================
#nodes = 26, #terminals = 14, # terminal_NIs = 0, #movable = 12, #nets = 20
die area = (0, 0, 200, 100) 20000
row height = 10, site width = 10
num_bins = 256x64, bin sizes = 0.078125x0.15625
#pins = 50, #movable_pins = 30
total_movable_node_area = 24, total_fixed_node_area = 2400, total_space_area = 17600
utilization = 0.00136364, target_density = 0.85
use placeable_area = 17600 to compute fillers
total_filler_node_area = 14936, #fillers = 1494, filler sizes = 1x10
========================================================================================
I0520 10:10:22.396546 139895627211904 PlaceDB.py:593] reading benchmark takes 0.00793886 seconds
I0520 10:10:22.427349 139895627211904 BasicPlace.py:276] move cells to the center of layout with random noise
W0520 10:10:22.600625 139895627211904 BasicPlace.py:145] net weights are all the same, ignored
I0520 10:10:22.621884 139895627211904 PlaceObj.py:711] 256x64 bins #bins 256x64, bin sizes 0.078125x0.15625, max_num_bins = 4, padding = 0
I0520 10:10:22.707161 139895627211904 electric_potential.py:436] regard 0 cells as movable macros in global placement
W0520 10:10:22.710636 139895627211904 PlaceObj.py:864] quadratic density penalty is disabled, density weight update is forced to be based on HPWL
I0520 10:10:22.714875 139895627211904 NonLinearPlace.py:104] use nesterov optimizer
I0520 10:10:22.717111 139895627211904 NonLinearPlace.py:141] add 2.5% noise
I0520 10:10:22.867953 139895627211904 electric_potential.py:478] fixed density map: average 0.124512, max 1.0376, bin area 1.2207
I0520 10:10:23.727374 139895627211904 NonLinearPlace.py:149] nesterov initialization takes 1.11368 seconds
I0520 10:10:24.285391 139895627211904 NonLinearPlace.py:270] density_weight = 4.716234E-09
I0520 10:10:25.816978 139895627211904 NonLinearPlace.py:311] optimizer step 1483.198 ms
I0520 10:10:25.821764 139895627211904 NonLinearPlace.py:318] iteration    0, (   0,  0,  0), Obj 5.269872E+02, DensityWeight 0.000000E+00, HPWL 1.656111E+03, Overflow 6.883214E-01, MaxDensity 8.904E+00, gamma 9.375000E+01, time 43.194ms
I0520 10:10:25.822677 139895627211904 NonLinearPlace.py:327] full step 2087.348 ms
I0520 10:10:32.409693 139895627211904 NonLinearPlace.py:311] optimizer step 6497.175 ms
...
I0520 10:10:44.062026 139895627211904 NonLinearPlace.py:318] iteration    3, (   3,  0,  0), Obj 1.506339E+03, DensityWeight 5.197049E-09, HPWL 1.513677E+03, Overflow 2.499992E-01, MaxDensity 1.486E+00, gamma 2.019774E+00, time 83.113ms
I0520 10:10:44.065448 139895627211904 NonLinearPlace.py:327] full step 1780.822 ms
I0520 10:10:44.087588 139895627211904 NonLinearPlace.py:609] optimizer nesterov takes 21.474 seconds
I0520 10:10:44.092856 139895627211904 dreamplace_core.py:50] Last Dreamplace metric: iteration    3, (   3,  0,  0), Obj 1.506339E+03, DensityWeight 5.454719E-09, HPWL 1.513677E+03, Overflow 2.499992E-01, MaxDensity 1.486E+00, gamma 2.019775E+00, time 83.113ms
I0520 10:10:46.019161 139895627211904 ppo_collect_lib.py:156] Current step: 0
I0520 10:10:46.027359 139895627211904 ppo_collect_lib.py:157] Current model_id: 0
I0520 10:10:46.032843 139895627211904 ppo_collect_lib.py:144] Collecting at model_id: 0
I0520 10:10:48.575352 139895627211904 BasicPlace.py:276] move cells to the center of layout with random noise
W0520 10:10:48.633654 139895627211904 BasicPlace.py:145] net weights are all the same, ignored
I0520 10:10:48.637322 139895627211904 PlaceObj.py:711] 256x64 bins #bins 256x64, bin sizes 0.078125x0.15625, max_num_bins = 4, padding = 0
I0520 10:10:48.644621 139895627211904 electric_potential.py:436] regard 0 cells as movable macros in global placement
W0520 10:10:48.648859 139895627211904 PlaceObj.py:864] quadratic density penalty is disabled, density weight update is forced to be based on HPWL
I0520 10:10:48.649452 139895627211904 NonLinearPlace.py:104] use nesterov optimizer
I0520 10:10:48.652932 139895627211904 NonLinearPlace.py:141] add 2.5% noise
I0520 10:10:48.763238 139895627211904 electric_potential.py:478] fixed density map: average 0.124512, max 1.0376, bin area 1.2207
I0520 10:10:51.492685 139895627211904 NonLinearPlace.py:149] nesterov initialization takes 2.85681 seconds
I0520 10:10:53.048169 139895627211904 NonLinearPlace.py:270] density_weight = 7.765560E-09
I0520 10:10:58.424235 139895627211904 NonLinearPlace.py:311] optimizer step 4877.851 ms
I0520 10:10:58.431993 139895627211904 NonLinearPlace.py:318] iteration    0, (   0,  0,  0), Obj 4.889251E+02, DensityWeight 0.000000E+00, HPWL 1.558857E+03, Overflow 7.831456E-01, MaxDensity 8.943E+00, gamma 9.375000E+01, time 482.733ms
I0520 10:10:58.432931 139895627211904 NonLinearPlace.py:327] full step 6918.902 ms
I0520 10:10:59.586692 139895627211904 NonLinearPlace.py:311] optimizer step 1059.123 ms
...
I0520 10:11:12.652804 139895627211904 NonLinearPlace.py:318] iteration    3, (   3,  0,  0), Obj 1.004180E+03, DensityWeight 8.557249E-09, HPWL 1.007382E+03, Overflow 2.131130E-01, MaxDensity 1.517E+00, gamma 1.672374E+00, time 120.045ms
I0520 10:11:12.657268 139895627211904 NonLinearPlace.py:327] full step 1789.831 ms
I0520 10:11:12.658546 139895627211904 NonLinearPlace.py:609] optimizer nesterov takes 24.023 seconds
I0520 10:11:12.662233 139895627211904 dreamplace_core.py:50] Last Dreamplace metric: iteration    3, (   3,  0,  0), Obj 1.004180E+03, DensityWeight 8.985111E-09, HPWL 1.007382E+03, Overflow 2.131130E-01, MaxDensity 1.517E+00, gamma 1.672374E+00, time 120.045ms
I0520 10:11:15.580829 139895627211904 ppo_collect_lib.py:156] Current step: 0
I0520 10:11:15.581913 139895627211904 ppo_collect_lib.py:157] Current model_id: 0
I0520 10:11:15.598651 139895627211904 ppo_collect_lib.py:144] Collecting at model_id: 0
I0520 10:11:18.551908 139895627211904 BasicPlace.py:276] move cells to the center of layout with random noise
W0520 10:11:18.621757 139895627211904 BasicPlace.py:145] net weights are all the same, ignored
I0520 10:11:18.653649 139895627211904 PlaceObj.py:711] 256x64 bins #bins 256x64, bin sizes 0.078125x0.15625, max_num_bins = 4, padding = 0
I0520 10:11:18.658501 139895627211904 electric_potential.py:436] regard 0 cells as movable macros in global placement
W0520 10:11:18.666754 139895627211904 PlaceObj.py:864] quadratic density penalty is disabled, density weight update is forced to be based on HPWL
I0520 10:11:18.669990 139895627211904 NonLinearPlace.py:104] use nesterov optimizer
I0520 10:11:18.703029 139895627211904 NonLinearPlace.py:141] add 2.5% noise
I0520 10:11:18.880851 139895627211904 electric_potential.py:478] fixed density map: average 0.124512, max 1.0376, bin area 1.2207
I0520 10:11:20.822913 139895627211904 NonLinearPlace.py:149] nesterov initialization takes 2.17087 seconds
I0520 10:11:22.235902 139895627211904 NonLinearPlace.py:270] density_weight = 9.725340E-09
I0520 10:11:25.517678 139895627211904 NonLinearPlace.py:311] optimizer step 3084.145 ms
…
I0520 10:25:19.172328 139895627211904 NonLinearPlace.py:318] iteration  216, ( 216,  0,  0), Obj 1.242919E+03, DensityWeight 1.393400E-04, HPWL 1.263650E+03, Overflow 3.103030E-01, MaxDensity 1.669E+00, gamma 2.833222E+00, time 84.916ms
I0520 10:25:19.173274 139895627211904 NonLinearPlace.py:327] full step 1187.224 ms
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/workspace/circuit_training/learning/ppo_collect.py", line 114, in <module>
    multiprocessing.handle_main(functools.partial(app.run, main))
  File "/usr/local/lib/python3.9/dist-packages/tf_agents/system/default/multiprocessing_core.py", line 76, in handle_main
    return app.run(parent_main_fn, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/absl/app.py", line 316, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.9/dist-packages/absl/app.py", line 261, in _run_main
    sys.exit(main(argv))
  File "/usr/local/lib/python3.9/dist-packages/absl/app.py", line 316, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.9/dist-packages/absl/app.py", line 261, in _run_main
    sys.exit(main(argv))
  File "/workspace/circuit_training/learning/ppo_collect.py", line 95, in main
    ppo_collect_lib.collect(
  File "/usr/local/lib/python3.9/dist-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/lib/python3.9/dist-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/usr/local/lib/python3.9/dist-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/workspace/circuit_training/learning/ppo_collect_lib.py", line 146, in collect
    collect_actor.run()
  File "/usr/local/lib/python3.9/dist-packages/tf_agents/train/actor.py", line 167, in run
    self._time_step, self._policy_state = self._driver.run(
  File "/usr/local/lib/python3.9/dist-packages/tf_agents/drivers/py_driver.py", line 120, in run
    next_time_step = self.env.step(action_step.action)
  File "/usr/local/lib/python3.9/dist-packages/tf_agents/environments/py_environment.py", line 238, in step
    self._current_time_step = self._step(action)
  File "/usr/local/lib/python3.9/dist-packages/tf_agents/environments/wrappers.py", line 672, in _step
    return self._env.step(clipped_actions)
 File "/usr/local/lib/python3.9/dist-packages/tf_agents/environments/py_environment.py", line 238, in step
    self._current_time_step = self._step(action)
  File "/usr/local/lib/python3.9/dist-packages/tf_agents/environments/gym_wrapper.py", line 233, in _step
    observation, reward, self._done, self._info = self._gym_env.step(action)
  File "/workspace/circuit_training/environment/environment.py", line 612, in step
    cost, info = self.call_analytical_placer_and_get_cost()
  File "/workspace/circuit_training/environment/environment.py", line 471, in call_analytical_placer_and_get_cost
    self.analytical_placer()
  File "/workspace/circuit_training/environment/environment.py", line 553, in analytical_placer
    converged = self._dreamplace.place()
  File "/workspace/circuit_training/dreamplace/dreamplace_core.py", line 65, in place
    return decorated_place()
  File "/usr/local/lib/python3.9/dist-packages/timeout_decorator/timeout_decorator.py", line 82, in new_function
    return function(*args, **kwargs)
  File "/workspace/circuit_training/dreamplace/dreamplace_core.py", line 62, in decorated_place
    return self._place()
  File "/workspace/circuit_training/dreamplace/dreamplace_core.py", line 49, in _place
    metrics = nonlinear_place(self.params, self.placedb_plc.placedb)
  File "/dreamplace/dreamplace/NonLinearPlace.py", line 448, in __call__
    one_descent_step(
  File "/dreamplace/dreamplace/NonLinearPlace.py", line 309, in one_descent_step
    optimizer.step()
  File "/usr/local/lib/python3.9/dist-packages/torch/optim/optimizer.py", line 140, in wrapper
    out = func(*args, **kwargs)
  File "/dreamplace/dreamplace/NesterovAcceleratedGradientOptimizer.py", line 121, in step
    f_kp1, g_kp1 = obj_and_grad_fn(v_kp1)
  File "/dreamplace/dreamplace/PlaceObj.py", line 387, in obj_and_grad_fn
    obj = self.obj_fn(pos)
  File "/dreamplace/dreamplace/PlaceObj.py", line 287, in obj_fn
    self.density = self.op_collections.density_op(pos)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/dreamplace/dreamplace/ops/electric_potential/electric_potential.py", line 521, in forward
    return ElectricPotentialFunction.apply(
  File "/dreamplace/dreamplace/ops/electric_potential/electric_potential.py", line 171, in forward
    ctx.field_map_y = idct_idxst.forward(auv_by_wu2_plus_wv2_wv)
  File "/dreamplace/dreamplace/ops/dct/dct2_fft2.py", line 117, in forward
    return IDCT_IDXSTFunction.apply(x, self.expkM, self.expkN, self.out, self.buf)
  File "/dreamplace/dreamplace/ops/dct/dct2_fft2.py", line 93, in forward
    dct2_fft2_cpp.idct_idxst(x, expkM, expkN, out, buf, torch.get_num_threads())
  File "/usr/local/lib/python3.9/dist-packages/timeout_decorator/timeout_decorator.py", line 69, in handler
    _raise_exception(timeout_exception, exception_message)
  File "/usr/local/lib/python3.9/dist-packages/timeout_decorator/timeout_decorator.py", line 47, in _raise_exception
    raise exception(exception_message)
timeout_decorator.timeout_decorator.TimeoutError: 'SoftMacroPlacer place() timed out.'
  In call to configurable 'collect' (<function collect at 0x7f3b107ef5e0>)

• Additionally, when running environment_test inside the container, it fails with a test error:

root@9b72becb570a:/workspace# python3.9 -m circuit_training.environment.environment_test
2025-05-20 04:48:33.633193: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-05-20 04:48:33.732123: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 04:48:34.197015: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-05-20 04:48:34.197149: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-20 04:48:34.273740: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-20 04:48:34.438634: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-05-20 04:48:34.441183: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-20 04:48:36.809153: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Running tests under Python 3.9.22: /usr/bin/python3.9
[ RUN      ] EnvironmentTest.test_action_space
...
[       OK ] EnvironmentTest.test_action_space
[ RUN      ] EnvironmentTest.test_create_and_obs_space
...
[       OK ] EnvironmentTest.test_create_and_obs_space
[ RUN      ] EnvironmentTest.test_infisible
...
[       OK ] EnvironmentTest.test_infisible
[ RUN      ] EnvironmentTest.test_save_file_train_step
I0520 04:48:46.700897 140521636689024 environment.py:220] ***Num node to place***:2
I0520 04:48:46.702336 140521636689024 placement_util.py:508] node_order: descending_size_macro_first
I0520 04:48:46.706506 140521636689024 observation_extractor.py:349] Pad a tensor with shape (5,) by 42000
I0520 04:48:46.706956 140521636689024 observation_extractor.py:363] Pad a tensor with shape (5,) by 3500
I0520 04:48:46.740284 140521636689024 placement_util.py:508] node_order: random
I0520 04:48:46.746101 140521636689024 dreamplace_util.py:105] Update num_bins_x and num_bins_y: (128, 128)
I0520 04:48:46.749952 140521636689024 plc_converter.py:141] Node 8 is placed at (170.000000, 230.000000).
I0520 04:48:46.751730 140521636689024 plc_converter.py:141] Node 2 is placed at (375.000000, 375.000000).
I0520 04:48:46.752928 140521636689024 plc_converter.py:141] Node 3 is placed at (125.000000, 125.000000).
I0520 04:48:46.754043 140521636689024 plc_converter.py:141] Node 0 is placed at (0.000000, 100.000000).
I0520 04:48:46.754911 140521636689024 plc_converter.py:141] Node 1 is placed at (499.000000, 499.000000).
/usr/local/lib/python3.9/dist-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.9/dist-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in divide
  ret = ret.dtype.type(ret / rcount)
INFO:tensorflow:time(__main__.EnvironmentTest.test_save_file_train_step): 0.17s
I0520 04:48:46.768812 140521636689024 test_util.py:2574] time(__main__.EnvironmentTest.test_save_file_train_step): 0.17s
[  FAILED  ] EnvironmentTest.test_save_file_train_step
[ RUN      ] EnvironmentTest.test_session
[  SKIPPED ] EnvironmentTest.test_session - Not a test.
[ RUN      ] EnvironmentTest.test_validate_circuite_env
...
[       OK ] EnvironmentTest.test_validate_circuite_env
[ RUN      ] EnvironmentTest.test_wrap_tfpy_environment
...
[       OK ] EnvironmentTest.test_wrap_tfpy_environment
======================================================================
ERROR: test_save_file_train_step (__main__.EnvironmentTest)
EnvironmentTest.test_save_file_train_step
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/circuit_training/environment/environment_test.py", line 158, in test_save_file_train_step
    obs, _, done, _ = env.step(action)
  File "/workspace/circuit_training/environment/environment.py", line 612, in step
    cost, info = self.call_analytical_placer_and_get_cost()
  File "/workspace/circuit_training/environment/environment.py", line 489, in call_analytical_placer_and_get_cost
    self._save_placement(cost)
  File "/workspace/circuit_training/environment/environment.py", line 440, in _save_placement
    self._run_cd()
  File "/workspace/circuit_training/environment/environment.py", line 402, in _run_cd
    cd = cd_placer.CoordinateDescentPlacer(plc=self._plc, cost_fn=cost_fn)
  File "/usr/local/lib/python3.9/dist-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/lib/python3.9/dist-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/usr/local/lib/python3.9/dist-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/workspace/circuit_training/environment/coordinate_descent_placer.py", line 151, in __init__
    self._dreamplace = dreamplace_core.SoftMacroPlacer(
  File "/usr/local/lib/python3.9/dist-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/lib/python3.9/dist-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/usr/local/lib/python3.9/dist-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/workspace/circuit_training/dreamplace/dreamplace_core.py", line 34, in __init__
    self.placedb_plc = placedb_plc.PlacedbPlc(plc, params, hard_macro_order)
  File "/workspace/circuit_training/dreamplace/placedb_plc.py", line 31, in __init__
    self.placedb(self.params)
  File "/dreamplace/dreamplace/PlaceDB.py", line 591, in __call__
    self.initialize(params)
  File "/dreamplace/dreamplace/PlaceDB.py", line 857, in initialize
    self.num_filler_nodes = int(round(self.total_filler_node_area / (filler_size_x * filler_size_y)))
ValueError: cannot convert float NaN to integer
  In call to configurable 'SoftMacroPlacer' (<class 'circuit_training.dreamplace.dreamplace_core.SoftMacroPlacer'>)
  In call to configurable 'CoordinateDescentPlacer' (<class 'circuit_training.environment.coordinate_descent_placer.CoordinateDescentPlacer'>)
----------------------------------------------------------------------
Ran 7 tests in 0.786s
FAILED (errors=1, skipped=1)

• It seems like the collect jobs are not sending data to Reverb — no progress is made even for toy netlists.
• Problem exists for multiple stable branches (r0.0.4, r0.0.3) as well as docker images with and without cuda
• Docker version 28.1.1

HPC Cluster (Slurm-based)
Method:

• Container system: Apptainer (Singularity-based) - I converted the Docker image to .sif on Local Machine 2 and uploaded it to the cluster.
• Local install using modules system and conda venv.
  Lib versions:
  GCCcore/11.3.0, GCC/11.3.0, CMake/3.23.1, Boost/1.79.0, CUDA/11.8.0, cuDNN/8.6.0.163-CUDA-11.8.0, Miniconda3/23.3.1-0
  conda env file:

name: /net/tscratch/people/plgkmiskowicz/conda_envs/circuit-training-gpu
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2025.4.26=hbd8a1cb_0
  - cairo=1.16.0=hb05425b_5
  - fontconfig=2.14.1=hef1e5e3_0
  - freetype=2.10.4=h0708190_1
  - glib=2.78.4=h6a678d5_0
  - glib-tools=2.78.4=h6a678d5_0
  - icu=58.2=hf484d3e_1000
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.4.4=h6a678d5_1
  - libgcc-ng=11.2.0=h1234567_1
  - libglib=2.78.4=hdc74915_0
  - libgomp=11.2.0=h1234567_1
  - libiconv=1.16=h516909a_0
  - libpng=1.6.39=h5eee18b_0
  - libstdcxx-ng=11.2.0=h1234567_1
  - libxcb=1.17.0=h9b100fa_0
  - libxml2=2.9.14=h74e7548_0
  - ncurses=6.4=h6a678d5_0
  - openssl=3.0.16=h5eee18b_0
  - pcre2=10.42=hebb0a14_1
  - pip=25.1=pyhc872135_2
  - pixman=0.40.0=h7f8727e_1
  - pthread-stubs=0.4=h36c2ea0_1001
  - python=3.9.21=he870216_1
  - readline=8.2=h5eee18b_0
  - setuptools=78.1.1=py39h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - tzdata=2025b=h04d1e81_0
  - wheel=0.45.1=py39h06a4308_0
  - xorg-libx11=1.8.12=h9b100fa_1
  - xorg-libxau=1.0.12=h9b100fa_0
  - xorg-libxdmcp=1.1.5=h9b100fa_0
  - xorg-libxext=1.3.6=h9b100fa_0
  - xorg-libxrender=0.9.12=h9b100fa_0
  - xorg-xorgproto=2024.1=h5eee18b_1
  - xz=5.6.4=h5eee18b_1
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - absl-py==2.2.2
      - astunparse==1.6.3
      - cachetools==5.5.2
      - cairocffi==1.7.1
      - certifi==2025.4.26
      - cffi==1.17.1
      - charset-normalizer==3.4.2
      - cloudpickle==3.1.1
      - contourpy==1.3.0
      - cycler==0.12.1
      - decorator==5.2.1
      - dm-reverb==0.14.0
      - dm-tree==0.1.8
      - easyprocess==1.1
      - entrypoint2==1.1
      - flatbuffers==25.2.10
      - fonttools==4.58.0
      - gast==0.6.0
      - gin-config==0.5.0
      - google-auth==2.40.1
      - google-auth-oauthlib==1.0.0
      - google-pasta==0.2.0
      - grpcio==1.71.0
      - gym==0.23.0
      - gym-notices==0.0.8
      - h5py==3.13.0
      - idna==3.10
      - importlib-metadata==8.7.0
      - importlib-resources==6.5.2
      - keras==2.15.0
      - kiwisolver==1.4.7
      - libclang==18.1.1
      - markdown==3.8
      - markdown-it-py==3.0.0
      - markupsafe==3.0.2
      - matplotlib==3.9.4
      - mdurl==0.1.2
      - ml-dtypes==0.3.2
      - namex==0.0.9
      - numpy==1.26.4
      - nvidia-cublas-cu11==11.10.3.66
      - nvidia-cuda-nvrtc-cu11==11.7.99
      - nvidia-cuda-runtime-cu11==11.7.99
      - nvidia-cudnn-cu11==8.5.0.96
      - oauthlib==3.2.2
      - opt-einsum==3.4.0
      - optree==0.15.0
      - packaging==25.0
      - patool==1.15.0
      - pillow==11.2.1
      - pkgconfig==1.5.5
      - portpicker==1.6.0
      - protobuf==4.25.7
      - psutil==7.0.0
      - pyasn1==0.6.1
      - pyasn1-modules==0.4.2
      - pycparser==2.22
      - pygame==2.1.3
      - pygments==2.19.1
      - pyparsing==3.2.3
      - python-dateutil==2.9.0.post0
      - pyunpack==0.3
      - requests==2.32.3
      - requests-oauthlib==2.0.0
      - rich==14.0.0
      - rlds==0.1.8
      - rsa==4.9.1
      - scipy==1.13.1
      - shapely==2.0.7
      - six==1.17.0
      - tensorboard==2.15.2
      - tensorboard-data-server==0.7.2
      - tensorflow==2.15.1
      - tensorflow-estimator==2.15.0
      - tensorflow-io-gcs-filesystem==0.37.1
      - tensorflow-probability==0.23.0
      - termcolor==3.1.0
      - tf-agents==0.19.0
      - tf-keras==2.15.0
      - timeout-decorator==0.5.0
      - torch==1.13.1
      - typing-extensions==4.5.0
      - urllib3==2.4.0
      - werkzeug==3.1.3
      - wrapt==1.14.1
      - zipp==3.21.0
prefix: /net/tscratch/people/plgkmiskowicz/conda_envs/circuit-training-gpu

Problem:

• The same issue as with e2e_smoke_test on Local Machine 2: the training hangs at iteration 0, no progress is made by collect jobs.
• I experimented with different numbers of collect jobs, running them both inside and outside the container, and varying compute resources — the behavior remained the same.
• Same issues for both the default and toy netlists.

I would greatly appreciate any advice or ideas on what might be going wrong. My goal is to get at least the toy example running end-to-end so that I can move forward with experiments. Particularly important for me will be to solve the problem associated with HPC (described in the section of Local Machine 2), because there I ultimately want to test the environment.
Please let me know:

• Any tips for making e2e_smoke_test pass
• Are there known issues with collect jobs, reverb communication or training stuck on the first iteration?
• Any tips for making the environment_test pass?

Thank you in advance for your time and support — and also for making this research open source!
Best regards,
Krzysiek

[metzkorn]

Both your machines run MacOS? From the README:

AlphaChip only supports Linux based OSes.

[kmiskowicz]

I wasn't sure if this refers to a non-docker installation.

Nevertheless, I tried to run the AWS smoke test where I set up EC2 with Ubuntu 22.04 and I was getting the same error.