new object_detection error from today's commis: 'ManualStepLearningRate' object has no attribute 'warmup'

[jwnsu]

After pulled today's changes, got an error in previously working training (warmup was not in config file at all)

Traceback (most recent call last):
  File "object_detection/train.py", line 167, in <module>
    tf.app.run()
  File "/home/dsu/tf/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "object_detection/train.py", line 163, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/home/dsu/ai/tfmodels/research/object_detection/trainer.py", line 255, in train
    train_config.optimizer)
  File "/home/dsu/ai/tfmodels/research/object_detection/builders/optimizer_builder.py", line 50, in build
    learning_rate = _create_learning_rate(config.learning_rate)
  File "/home/dsu/ai/tfmodels/research/object_detection/builders/optimizer_builder.py", line 109, in _create_learning_rate
    learning_rate_sequence, config.warmup)
AttributeError: 'ManualStepLearningRate' object has no attribute 'warmup'

[learnbott]

I had the same problem, but then realized I need to update my pipeline.config to conform to the new manualsteplearningrate. After I made the adjustments I get this error:

python
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1022, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 233, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 212, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 422, in make_tensor_proto
    _GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 524, in _apply_op_helper
    values, as_ref=input_arg.is_ref).dtype.name
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1022, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 233, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 212, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 422, in make_tensor_proto
    _GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/repos/models/research/object_detection/train.py", line 167, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 124, in run
    _sys.exit(main(argv))
  File "/work/repos/models/research/object_detection/train.py", line 163, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "/work/repos/models/research/object_detection/trainer.py", line 255, in train
    train_config.optimizer)
  File "/work/repos/models/research/object_detection/builders/optimizer_builder.py", line 58, in build
    learning_rate = _create_learning_rate(config.learning_rate)
  File "/work/repos/models/research/object_detection/builders/optimizer_builder.py", line 109, in _create_learning_rate
    learning_rate_sequence, config.warmup)
  File "/work/repos/models/research/object_detection/utils/learning_schedules.py", line 169, in manual_stepping
    [0] * num_boundaries))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/array_ops.py", line 2540, in where
    return gen_math_ops._select(condition=condition, x=x, y=y, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 4043, in _select
    "Select", condition=condition, t=x, e=y, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 528, in _apply_op_helper
    (input_name, err))
ValueError: Tried to convert 't' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

[VinceKumar]

I get this error too

[learnbott]

I believe it is the same Python3 incompatibility that has crept up before (see #3443 ). The issue is with models/research/object_detection/utils/learning_schedules.py lines 167-169. Currently it is

rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
                                      range(num_boundaries),
                                      [0] * num_boundaries))

Wrap list() around the range() like this:

rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
                                     list(range(num_boundaries)),
                                      [0] * num_boundaries))

and you should be good to go. Mine is off and training.

@jwnsu make sure you change your pipeline.config from:

        manual_step_learning_rate {
          initial_learning_rate: 0.00001
          schedule {
            step: 0
            learning_rate: .0003
          }
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }

to the new slightly different:

        manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 900000
            learning_rate: .00003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }

[jrabary]

same problem as here #3605

[jwnsu]

manual_step_learning_rate is not used directly, seems a new bug, here is the config:

optimizer {
    momentum_optimizer: {
      learning_rate: {
        manual_step_learning_rate {
          initial_learning_rate: 0.0003
          schedule {
            step: 900000
            learning_rate: .0003
          }
          schedule {
            step: 1200000
            learning_rate: .000003
          }
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false

[allendoss]

@learnbott I was having the errors you had mentioned before:
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted []
The wraping of list worked for me. Thanks a ton!

[jlertle]

remember to recompile protobufs after upgrade:

# From tensorflow/models/research/
protoc object_detection/protos/*.proto --python_out=.

[hustc12]

I meet this issue too, but thanks to @learnbott , and now I can move on.

[roy-d]

you rock @learnbott

[FalcoGer]

when will this fix be merged?