InvalidArgumentError (see above for traceback): Nan in summary histogram for: DNC/FF_step/dense_1/bias_0

[SeekPoint]

Summary generated. Step 74400 Test cost == 31.212728500 Time == 8.57s
Summary generated. Step 74500 Test cost == 10.503337860 Time == 8.38s
Summary generated. Step 74600 Test cost == 749542.875000000 Time == 8.54s
Summary generated. Step 74700 Test cost == 866.475341797 Time == 8.72s
Summary generated. Step 74800 Test cost == 6.063397408 Time == 8.58s
Summary generated. Step 74900 Test cost == 19.894033432 Time == 8.50s
Summary generated. Step 75000 Test cost == 3009.345458984 Time == 8.53s
Model saved!
Summary generated. Step 75100 Test cost == 401.853302002 Time == 8.68s
Summary generated. Step 75200 Test cost == 201.970886230 Time == 8.54s
Summary generated. Step 75300 Test cost == 4504.419433594 Time == 8.59s
Summary generated. Step 75400 Test cost == 6352.577148438 Time == 8.56s
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1139, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1121, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: DNC/FF_step/dense_1/bias_0
[[Node: DNC/FF_step/dense_1/bias_0 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](DNC/FF_step/dense_1/bias_0/tag, DNC/FF_step/dense_1/bias/read)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 54, in
dnc.run_session(task, Hp, project_path) # , restore_path=restore_path)
File "/home/mldl/ub16_prj/DNC/src/controller.py", line 94, in run_session
mask: m})
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: DNC/FF_step/dense_1/bias_0
[[Node: DNC/FF_step/dense_1/bias_0 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](DNC/FF_step/dense_1/bias_0/tag, DNC/FF_step/dense_1/bias/read)]]

Caused by op 'DNC/FF_step/dense_1/bias_0', defined at:
File "main.py", line 54, in
dnc.run_session(task, Hp, project_path) # , restore_path=restore_path)
File "/home/mldl/ub16_prj/DNC/src/controller.py", line 57, in run_session
tf.summary.histogram(variable.name, variable)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/summary/summary.py", line 221, in histogram
tag=scope.rstrip('/'), values=values, name=scope)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 131, in _histogram_summary
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1269, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Nan in summary histogram for: DNC/FF_step/dense_1/bias_0
[[Node: DNC/FF_step/dense_1/bias_0 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](DNC/FF_step/dense_1/bias_0/tag, DNC/FF_step/dense_1/bias/read)]]

mldl@mldlUB1604:/ub16_prj/DNC/src$
mldl@mldlUB1604:/ub16_prj/DNC/src$
mldl@mldlUB1604:~/ub16_prj/DNC/src$ python3
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorflow as tf
tf.version
'1.2.1'

[bgavran]

Yeah, this is a well-known problem with DNCs, there are NaNs sometimes and there doesn't really seem to exist a way to fix it other than restarting the training process.

I do welcome a conversation about possible ways to solve the problem