Tensorflow:图表已完成,无法修改

我试图通过检查点保存变量,以便为我的程序引入容错.我试图通过使用MonitoredTrainingSession函数来实现这一目标.以下是我的配置: -

import tensorflow as tf

global_step = tf.Variable(10, trainable=False, name='global_step')
x = tf.constant(2)

with tf.device("/job:local/task:0"):
    y1 = tf.Variable(x + 300)

with tf.device("/job:local/task:1"):
    y2 = tf.Variable(x**2)

with tf.device("/job:local/task:2"):
    y3 = tf.Variable(5*x)

with tf.device("/job:local/task:3"):
    y0 = tf.Variable(x - 66)
    y = y0 + y1 + y2 + y3

model = tf.global_variables_initializer()
saver = tf.train.Saver(sharded=True)

chief = tf.train.ChiefSessionCreator(scaffold=None, master='grpc://localhost:2222', config=None, checkpoint_dir='/home/tensorflow/codes/checkpoints')
summary_hook = tf.train.SummarySaverHook(save_steps=None, save_secs=10, output_dir='/home/tensorflow/codes/savepoints', summary_writer=None, scaffold=None, summary_op=tf.summary.tensor_summary(name="y", tensor=y))
saver_hook = tf.train.CheckpointSaverHook(checkpoint_dir='/home/tensorflow/codes/checkpoints', save_secs=None, save_steps=True, saver=saver, checkpoint_basename='model.ckpt', scaffold=None)

# with tf.train.MonitoredSession(session_creator=ChiefSessionCreator,hooks=[saver_hook, summary_hook]) as sess:

with tf.train.MonitoredTrainingSession(master='grpc://localhost:2222', is_chief=True, checkpoint_dir='/home/tensorflow/codes/checkpoints',
    scaffold=None, hooks=[saver_hook,summary_hook], chief_only_hooks=None, save_checkpoint_secs=None, save_summaries_steps=True, config=None) as sess:

    while not sess.should_stop():
        sess.run(tf.global_variables_initializer())

    while not sess.should_stop():
        result = sess.run(y)
        print(result)

我得到以下无法解决的RuntimeError: -

Traceback (most recent call last):
  File "add_1.py", line 39, in 
    sess.run(tf.global_variables_initializer())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 1187, in global_variables_initializer
    return variables_initializer(global_variables())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 1169, in variables_initializer
    return control_flow_ops.group(*[v.initializer for v in var_list], name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2773, in group
    deps.append(_GroupControlDeps(dev, ops_on_device[dev]))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2721, in _GroupControlDeps
    return no_op(name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_control_flow_ops.py", line 186, in no_op
    result = _op_def_lib.apply_op("NoOp", name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2199, in create_op
    self._check_not_finalized()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1925, in _check_not_finalized
    raise RuntimeError("Graph is finalized and cannot be modified.")
RuntimeError: Graph is finalized and cannot be modified.

guinny.. 10

您的错误的根本原因似乎是MonitoredTrainingSession已完成(冻结)图表,您tf.global_variable_initializer()无法再修改它.

话虽如此,有很多事情需要注意:

1)为什么你试着在这里重复初始化所有变量？

while not sess.should_stop():
    sess.run(tf.global_variables_initializer())

2)似乎你的一些代码已经包含在内MonitoredTrainingSession,例如ChiefSessionCreator.你能再看看代码(https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/monitored_session.py#L243)或搜索它的样本用法,看看MonitoredTrainingSession应该如何使用？

1> guinny..：

---

您的错误的根本原因似乎是MonitoredTrainingSession已完成(冻结)图表,您tf.global_variable_initializer()无法再修改它.

话虽如此,有很多事情需要注意:

1)为什么你试着在这里重复初始化所有变量？

while not sess.should_stop():
    sess.run(tf.global_variables_initializer())

2)似乎你的一些代码已经包含在内MonitoredTrainingSession,例如ChiefSessionCreator.你能再看看代码(https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/monitored_session.py#L243)或搜索它的样本用法,看看MonitoredTrainingSession应该如何使用？

---

第二次运行它时,它会尝试从早期运行中加载检查点,这会缺少global_step.查看此主题(http://stackoverflow.com/questions/36113090/tensorflow-get-the-global-step-when-restoring-checkpoints)如何保存和恢复global_step.而这里(https://github.com/tensorflow/tensorflow/blob/b00fc538638f87ac45be9105057b9865f0f9418b/tensorflow/python/training/monitored_session_test.py#L206)如何初始化一个.

2> matwilso..：

---

这可能不建议用于您的用例,但可以取消图表的终结:

sess.graph._unsafe_unfinalize()

3> 小智..：

---

如果要在循环中初始化图形，则可以使用该函数在循环顶部创建新图形。

import tensorflow as tf

tf.reset_default_graph()
tf.Graph().as_default()