Merged commit includes the following changes: (#6315)

236813471  by lzc:

    Internal change.

--
236507310  by lzc:

    Fix preprocess.random_resize_method config type issue. The target height and width will be passed as "size" to tf.image.resize_images which only accepts integer.

--
236409989  by Zhichao Lu:

    Config export_to_tpu from function parameter instead of HParams for TPU inference.

--
236403186  by Zhichao Lu:

    Make graph file names optional arguments.

--
236237072  by Zhichao Lu:

    Minor bugfix for keyword args.

--
236209602  by Zhichao Lu:

    Add support for PartitionedVariable to get_variables_available_in_checkpoint.

--
235828658  by Zhichao Lu:

    Automatically stop evaluation jobs when training is finished.

--
235817964  by Zhichao Lu:

    Add an optional process_metrics_fn callback to eval_util, it gets called
    with evaluation results once each evaluation is complete.

--
235788721  by lzc:

    Fix yml file tf runtime version.

--
235262897  by Zhichao Lu:

    Add keypoint support to the random_pad_image preprocessor method.

--
235257380  by Zhichao Lu:

    Support InputDataFields.groundtruth_confidences in retain_groundtruth(), retain_groundtruth_with_positive_classes(), filter_groundtruth_with_crowd_boxes(), filter_groundtruth_with_nan_box_coordinates(), filter_unrecognized_classes().

--
235109188  by Zhichao Lu:

    Fix bug in pad_input_data_to_static_shapes for num_additional_channels > 0; make color-specific data augmentation only touch RGB channels.

--
235045010  by Zhichao Lu:

    Don't slice class_predictions_with_background when add_background_class is false.

--
235026189  by lzc:

    Fix import in g3doc.

--
234863426  by Zhichao Lu:

    Added fixes in exporter to allow writing a checkpoint to a specified temporary directory.

--
234671886  by lzc:

    Internal Change.

--
234630803  by rathodv:

    Internal Change.

--
233985896  by Zhichao Lu:

    Add Neumann optimizer to object detection.

--
233560911  by Zhichao Lu:

    Add NAS-FPN object detection with Resnet and Mobilenet v2.

--
233513536  by Zhichao Lu:

    Export TPU compatible object detection model

--
233495772  by lzc:

    Internal change.

--
233453557  by Zhichao Lu:

    Create Keras-based SSD+MobilenetV1 for object detection.

--
233220074  by lzc:

    Update release notes date.

--
233165761  by Zhichao Lu:

    Support depth_multiplier and min_depth in _SSDResnetV1FpnFeatureExtractor.

--
233160046  by lzc:

    Internal change.

--
232926599  by Zhichao Lu:

    [tf.data] Switching tf.data functions to use `defun`, providing an escape hatch to continue using the legacy `Defun`.

    There are subtle differences between the implementation of `defun` and `Defun` (such as resources handling or control flow) and it is possible that input pipelines that use control flow or resources in their functions might be affected by this change. To migrate majority of existing pipelines to the recommended way of creating functions in TF 2.0 world, while allowing (a small number of) existing pipelines to continue relying on the deprecated behavior, this CL provides an escape hatch.

    If your input pipeline is affected by this CL, it should apply the escape hatch by replacing `foo.map(...)` with `foo.map_with_legacy_function(...)`.

--
232891621  by Zhichao Lu:

    Modify faster_rcnn meta architecture to normalize raw detections.

--
232875817  by Zhichao Lu:

    Make calibration a post-processing step.

    Specifically:
    - Move the calibration config from pipeline.proto --> post_processing.proto
    - Edit post_processing_builder.py to return a calibration function. If no calibration config is provided, it None.
    - Edit SSD and FasterRCNN meta architectures to optionally call the calibration function on detection scores after score conversion and before NMS.

--
232704481  by Zhichao Lu:

    Edit calibration builder to build a function that will be used within a detection model's `postprocess` method, after score conversion and before non-maxima suppression.

    Specific Edits:
    - The returned function now accepts class_predictions_with_background as its argument instead of detection_scores and detection_classes.
    - Class-specific calibration was temporarily removed, as it requires more significant refactoring. Will be added later.

--
232615379  by Zhichao Lu:

    Internal change

--
232483345  by ronnyvotel:

    Making the use of bfloat16 restricted to TPUs.

--
232399572  by Zhichao Lu:

    Edit calibration builder and proto to support class-agnostic calibration.

    Specifically:
    - Edit calibration protos to include path to relevant label map if required for class-specific calibration. Previously, label maps were inferred from other parts of the pipeline proto; this allows all information required by the builder stay within the calibration proto and remove extraneous information from being passed with class-agnostic calibration.
    - Add class-agnostic protos to the calibration config.

    Note that the proto supports sigmoid and linear interpolation parameters, but the builder currently only supports linear interpolation.

--
231613048  by Zhichao Lu:

    Add calibration builder for applying calibration transformations from output of object detection models.

    Specifically:
    - Add calibration proto to support sigmoid and isotonic regression (stepwise function) calibration.
    - Add a builder to support calibration from isotonic regression outputs.

--
231519786  by lzc:

    model_builder test refactor.
    - removed proto text boilerplate in each test case and let them call a create_default_proto function instead.
    - consolidated all separate ssd model creation tests into one.
    - consolidated all separate faster rcnn model creation tests into one.
    - used parameterized test for testing mask rcnn models and use_matmul_crop_and_resize
    - added all failures test.

--
231448169  by Zhichao Lu:

    Return static shape as a constant tensor.

--
231423126  by lzc:

    Add a release note for OID v4 models.

--
231401941  by Zhichao Lu:

    Adding correct labelmap for the models trained on Open Images V4 (*oid_v4
    config suffix).

--
231320357  by Zhichao Lu:

    Add scope to Nearest Neighbor Resize op so that it stays in the same name scope as the original resize ops.

--
231257699  by Zhichao Lu:

    Switch to using preserve_aspect_ratio in tf.image.resize_images rather than using a custom implementation.

--
231247368  by rathodv:

    Internal change.

--
231004874  by lzc:

    Update documentations to use tf 1.12 for object detection API.

--
230999911  by rathodv:

    Use tf.batch_gather instead of ops.batch_gather

--
230999720  by huizhongc:

    Fix weight equalization test in ops_test.

--
230984728  by rathodv:

    Internal update.

--
230929019  by lzc:

    Add an option to replace preprocess operation with placeholder for ssd feature extractor.

--
230845266  by lzc:

    Require tensorflow version 1.12 for object detection API and rename keras_applications to keras_models

--
230392064  by lzc:

    Add RetinaNet 101 checkpoint trained on OID v4 to detection model zoo.

--
230014128  by derekjchow:

    This file was re-located below the tensorflow/lite/g3doc/convert

--
229941449  by lzc:

    Update SSD mobilenet v2 quantized model download path.

--
229843662  by lzc:

    Add an option to use native resize tf op in fpn top-down feature map generation.

--
229636034  by rathodv:

    Add deprecation notice to a few old parameters in train.proto

--
228959078  by derekjchow:

    Remove duplicate elif case in _check_and_convert_legacy_input_config_key

--
228749719  by rathodv:

    Minor refactoring to make exporter's `build_detection_graph` method public.

--
228573828  by rathodv:

    Mofity model.postprocess to return raw detections and raw scores.

    Modify, post-process methods in core/model.py and the meta architectures to export raw detection (without any non-max suppression) and raw multiclass score logits for those detections.

--
228420670  by Zhichao Lu:

    Add shims for custom architectures for object detection models.

--
228241692  by Zhichao Lu:

    Fix the comment on "losses_mask" in "Loss" class.

--
228223810  by Zhichao Lu:

    Support other_heads' predictions in WeightSharedConvolutionalBoxPredictor. Also remove a few unused parameters and fix a couple of comments in convolutional_box_predictor.py.

--
228200588  by Zhichao Lu:

    Add Expected Calibration Error and an evaluator that calculates the metric for object detections.

--
228167740  by lzc:

    Add option to use bounded activations in FPN top-down feature map generation.

--
227767700  by rathodv:

    Internal.

--
226295236  by Zhichao Lu:

    Add Open Image V4 Resnet101-FPN training config to third_party

--
226254842  by Zhichao Lu:

    Fix typo in documentation.

--
225833971  by Zhichao Lu:

    Option to have no resizer in object detection model.

--
225824890  by lzc:

    Fixes p3 compatibility for model_lib.py

--
225760897  by menglong:

    normalizer should be at least 1.

--
225559842  by menglong:

    Add extra logic filtering unrecognized classes.

--
225379421  by lzc:

    Add faster_rcnn_inception_resnet_v2_atrous_oid_v4 config to third_party

--
225368337  by Zhichao Lu:

    Add extra logic filtering unrecognized classes.

--
225341095  by Zhichao Lu:

    Adding Open Images V4 models to OD API model zoo and corresponding configs to the
    configs.

--
225218450  by menglong:

    Add extra logic filtering unrecognized classes.

--
225057591  by Zhichao Lu:

    Internal change.

--
224895417  by rathodv:

    Internal change.

--
224209282  by Zhichao Lu:

    Add two data augmentations to object detection: (1) Self-concat (2) Absolute pads.

--
224073762  by Zhichao Lu:

    Do not create tf.constant until _generate() is actually called in the object detector.

--

PiperOrigin-RevId: 236813471

research/object_detection/README.md

@@ -99,6 +99,17 @@ reporting an issue.
 
 ## Release information
 
+### Feb 11, 2019
+
+We have released detection models trained on the [Open Images Dataset V4](https://storage.googleapis.com/openimages/web/challenge.html)
+in our detection model zoo, including
+
+* Faster R-CNN detector with Inception Resnet V2 feature extractor
+* SSD detector with MobileNet V2 feature extractor
+* SSD detector with ResNet 101 FPN feature extractor (aka RetinaNet-101)
+
+<b>Thanks to contributors</b>: Alina Kuznetsova, Yinxiao Li
+
 ### Sep 17, 2018
 
 We have released Faster R-CNN detectors with ResNet-50 / ResNet-101 feature

research/object_detection/anchor_generators/grid_anchor_generator.py

@@ -56,13 +56,10 @@ class GridAnchorGenerator(anchor_generator.AnchorGenerator):
     # Handle argument defaults
     if base_anchor_size is None:
       base_anchor_size = [256, 256]
-    base_anchor_size = tf.to_float(tf.convert_to_tensor(base_anchor_size))
     if anchor_stride is None:
       anchor_stride = [16, 16]
-    anchor_stride = tf.to_float(tf.convert_to_tensor(anchor_stride))
     if anchor_offset is None:
       anchor_offset = [0, 0]
-    anchor_offset = tf.to_float(tf.convert_to_tensor(anchor_offset))
 
     self._scales = scales
     self._aspect_ratios = aspect_ratios
@@ -108,6 +105,13 @@ class GridAnchorGenerator(anchor_generator.AnchorGenerator):
     if not all([isinstance(list_item, tuple) and len(list_item) == 2
                 for list_item in feature_map_shape_list]):
       raise ValueError('feature_map_shape_list must be a list of pairs.')
+    self._base_anchor_size = tf.to_float(tf.convert_to_tensor(
+        self._base_anchor_size))
+    self._anchor_stride = tf.to_float(tf.convert_to_tensor(
+        self._anchor_stride))
+    self._anchor_offset = tf.to_float(tf.convert_to_tensor(
+        self._anchor_offset))
+
     grid_height, grid_width = feature_map_shape_list[0]
     scales_grid, aspect_ratios_grid = ops.meshgrid(self._scales,
                                                    self._aspect_ratios)

research/object_detection/anchor_generators/multiple_grid_anchor_generator.py

@@ -60,7 +60,7 @@ class MultipleGridAnchorGenerator(anchor_generator.AnchorGenerator):
         outside list having the same number of entries as feature_map_shape_list
         (which is passed in at generation time).
       base_anchor_size: base anchor size as [height, width]
-                        (length-2 float tensor, default=[1.0, 1.0]).
+                        (length-2 float numpy or Tensor, default=[1.0, 1.0]).
                         The height and width values are normalized to the
                         minimum dimension of the input height and width, so that
                         when the base anchor height equals the base anchor
@@ -95,7 +95,7 @@ class MultipleGridAnchorGenerator(anchor_generator.AnchorGenerator):
       raise ValueError('box_specs_list is expected to be a '
                        'list of lists of pairs')
     if base_anchor_size is None:
-      base_anchor_size = tf.constant([256, 256], dtype=tf.float32)
+      base_anchor_size = [256, 256]
     self._base_anchor_size = base_anchor_size
     self._anchor_strides = anchor_strides
     self._anchor_offsets = anchor_offsets
@@ -211,10 +211,18 @@ class MultipleGridAnchorGenerator(anchor_generator.AnchorGenerator):
     min_im_shape = tf.minimum(im_height, im_width)
     scale_height = min_im_shape / im_height
     scale_width = min_im_shape / im_width
-    base_anchor_size = [
-        scale_height * self._base_anchor_size[0],
-        scale_width * self._base_anchor_size[1]
-    ]
+    if not tf.contrib.framework.is_tensor(self._base_anchor_size):
+      base_anchor_size = [
+          scale_height * tf.constant(self._base_anchor_size[0],
+                                     dtype=tf.float32),
+          scale_width * tf.constant(self._base_anchor_size[1],
+                                    dtype=tf.float32)
+      ]
+    else:
+      base_anchor_size = [
+          scale_height * self._base_anchor_size[0],
+          scale_width * self._base_anchor_size[1]
+      ]
     for feature_map_index, (grid_size, scales, aspect_ratios, stride,
                             offset) in enumerate(
                                 zip(feature_map_shape_list, self._scales,
@@ -304,7 +312,6 @@ def create_ssd_anchors(num_layers=6,
   """
   if base_anchor_size is None:
     base_anchor_size = [1.0, 1.0]
-  base_anchor_size = tf.constant(base_anchor_size, dtype=tf.float32)
   box_specs_list = []
   if scales is None or not scales:
     scales = [min_scale + (max_scale - min_scale) * i / (num_layers - 1)

research/object_detection/builders/anchor_generator_builder_test.py

@@ -47,14 +47,9 @@ class AnchorGeneratorBuilderTest(tf.test.TestCase):
                                grid_anchor_generator.GridAnchorGenerator))
     self.assertListEqual(anchor_generator_object._scales, [])
     self.assertListEqual(anchor_generator_object._aspect_ratios, [])
-    with self.test_session() as sess:
-      base_anchor_size, anchor_offset, anchor_stride = sess.run(
-          [anchor_generator_object._base_anchor_size,
-           anchor_generator_object._anchor_offset,
-           anchor_generator_object._anchor_stride])
-    self.assertAllEqual(anchor_offset, [0, 0])
-    self.assertAllEqual(anchor_stride, [16, 16])
-    self.assertAllEqual(base_anchor_size, [256, 256])
+    self.assertAllEqual(anchor_generator_object._anchor_offset, [0, 0])
+    self.assertAllEqual(anchor_generator_object._anchor_stride, [16, 16])
+    self.assertAllEqual(anchor_generator_object._base_anchor_size, [256, 256])
 
   def test_build_grid_anchor_generator_with_non_default_parameters(self):
     anchor_generator_text_proto = """
@@ -79,14 +74,9 @@ class AnchorGeneratorBuilderTest(tf.test.TestCase):
                                   [0.4, 2.2])
     self.assert_almost_list_equal(anchor_generator_object._aspect_ratios,
                                   [0.3, 4.5])
-    with self.test_session() as sess:
-      base_anchor_size, anchor_offset, anchor_stride = sess.run(
-          [anchor_generator_object._base_anchor_size,
-           anchor_generator_object._anchor_offset,
-           anchor_generator_object._anchor_stride])
-    self.assertAllEqual(anchor_offset, [30, 40])
-    self.assertAllEqual(anchor_stride, [10, 20])
-    self.assertAllEqual(base_anchor_size, [128, 512])
+    self.assertAllEqual(anchor_generator_object._anchor_offset, [30, 40])
+    self.assertAllEqual(anchor_generator_object._anchor_stride, [10, 20])
+    self.assertAllEqual(anchor_generator_object._base_anchor_size, [128, 512])
 
   def test_build_ssd_anchor_generator_with_defaults(self):
     anchor_generator_text_proto = """
@@ -114,10 +104,7 @@ class AnchorGeneratorBuilderTest(tf.test.TestCase):
         list(anchor_generator_object._aspect_ratios),
         [(1.0, 2.0, 0.5)] + 5 * [(1.0, 1.0)]):
       self.assert_almost_list_equal(expected_aspect_ratio, actual_aspect_ratio)
-
-    with self.test_session() as sess:
-      base_anchor_size = sess.run(anchor_generator_object._base_anchor_size)
-    self.assertAllClose(base_anchor_size, [1.0, 1.0])
+    self.assertAllClose(anchor_generator_object._base_anchor_size, [1.0, 1.0])
 
   def test_build_ssd_anchor_generator_with_custom_scales(self):
     anchor_generator_text_proto = """
@@ -194,9 +181,7 @@ class AnchorGeneratorBuilderTest(tf.test.TestCase):
         6 * [(1.0, 1.0)]):
       self.assert_almost_list_equal(expected_aspect_ratio, actual_aspect_ratio)
 
-    with self.test_session() as sess:
-      base_anchor_size = sess.run(anchor_generator_object._base_anchor_size)
-    self.assertAllClose(base_anchor_size, [1.0, 1.0])
+    self.assertAllClose(anchor_generator_object._base_anchor_size, [1.0, 1.0])
 
   def test_build_ssd_anchor_generator_with_non_default_parameters(self):
     anchor_generator_text_proto = """
@@ -241,9 +226,7 @@ class AnchorGeneratorBuilderTest(tf.test.TestCase):
         list(anchor_generator_object._anchor_offsets), [(8, 0), (16, 10)]):
       self.assert_almost_list_equal(expected_offsets, actual_offsets)
 
-    with self.test_session() as sess:
-      base_anchor_size = sess.run(anchor_generator_object._base_anchor_size)
-    self.assertAllClose(base_anchor_size, [1.0, 1.0])
+    self.assertAllClose(anchor_generator_object._base_anchor_size, [1.0, 1.0])
 
   def test_raise_value_error_on_empty_anchor_genertor(self):
     anchor_generator_text_proto = """

research/object_detection/builders/calibration_builder.py

+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Tensorflow ops to calibrate class predictions and background class."""
+
+import tensorflow as tf
+from object_detection.utils import shape_utils
+
+
+def _find_interval_containing_new_value(x, new_value):
+  """Find the index of x (ascending-ordered) after which new_value occurs."""
+  new_value_shape = shape_utils.combined_static_and_dynamic_shape(new_value)[0]
+  x_shape = shape_utils.combined_static_and_dynamic_shape(x)[0]
+  compare = tf.cast(tf.reshape(new_value, shape=(new_value_shape, 1)) >=
+                    tf.reshape(x, shape=(1, x_shape)),
+                    dtype=tf.int32)
+  diff = compare[:, 1:] - compare[:, :-1]
+  interval_idx = tf.argmin(diff, axis=1)
+  return interval_idx
+
+
+def _tf_linear_interp1d(x_to_interpolate, fn_x, fn_y):
+  """Tensorflow implementation of 1d linear interpolation.
+
+  Args:
+    x_to_interpolate: tf.float32 Tensor of shape (num_examples,) over which 1d
+      linear interpolation is performed.
+    fn_x: Monotonically-increasing, non-repeating tf.float32 Tensor of shape
+      (length,) used as the domain to approximate a function.
+    fn_y: tf.float32 Tensor of shape (length,) used as the range to approximate
+      a function.
+
+  Returns:
+    tf.float32 Tensor of shape (num_examples,)
+  """
+  x_pad = tf.concat([fn_x[:1] - 1, fn_x, fn_x[-1:] + 1], axis=0)
+  y_pad = tf.concat([fn_y[:1], fn_y, fn_y[-1:]], axis=0)
+  interval_idx = _find_interval_containing_new_value(x_pad, x_to_interpolate)
+
+  # Interpolate
+  alpha = (
+      (x_to_interpolate - tf.gather(x_pad, interval_idx)) /
+      (tf.gather(x_pad, interval_idx + 1) - tf.gather(x_pad, interval_idx)))
+  interpolation = ((1 - alpha) * tf.gather(y_pad, interval_idx) +
+                   alpha * tf.gather(y_pad, interval_idx + 1))
+
+  return interpolation
+
+
+def _function_approximation_proto_to_tf_tensors(x_y_pairs_message):
+  """Extracts (x,y) pairs from a XYPairs message.
+
+  Args:
+    x_y_pairs_message: calibration_pb2..XYPairs proto
+  Returns:
+    tf_x: tf.float32 tensor of shape (number_xy_pairs,) for function domain.
+    tf_y: tf.float32 tensor of shape (number_xy_pairs,) for function range.
+  """
+  tf_x = tf.convert_to_tensor([x_y_pair.x
+                               for x_y_pair
+                               in x_y_pairs_message.x_y_pair],
+                              dtype=tf.float32)
+  tf_y = tf.convert_to_tensor([x_y_pair.y
+                               for x_y_pair
+                               in x_y_pairs_message.x_y_pair],
+                              dtype=tf.float32)
+  return tf_x, tf_y
+
+
+def build(calibration_config):
+  """Returns a function that calibrates Tensorflow model scores.
+
+  All returned functions are expected to apply positive monotonic
+  transformations to inputs (i.e. score ordering is strictly preserved or
+  adjacent scores are mapped to the same score, but an input of lower value
+  should never be exceed an input of higher value after transformation).  For
+  class-agnostic calibration, positive monotonicity should hold across all
+  scores. In class-specific cases, positive monotonicity should hold within each
+  class.
+
+  Args:
+    calibration_config: calibration_pb2.CalibrationConfig proto.
+  Returns:
+    Function that that accepts class_predictions_with_background and calibrates
+    the output based on calibration_config's parameters.
+  Raises:
+    ValueError: No calibration builder defined for "Oneof" in
+      calibration_config.
+  """
+
+  # Linear Interpolation (usually used as a result of calibration via
+  # isotonic regression).
+  if calibration_config.WhichOneof('calibrator') == 'function_approximation':
+
+    def calibration_fn(class_predictions_with_background):
+      """Calibrate predictions via 1-d linear interpolation.
+
+      Predictions scores are linearly interpolated based on class-agnostic
+      function approximations. Note that the 0-indexed background class may
+      also transformed.
+
+      Args:
+        class_predictions_with_background: tf.float32 tensor of shape
+          [batch_size, num_anchors, num_classes + 1] containing scores on the
+          interval [0,1]. This is usually produced by a sigmoid or softmax layer
+          and the result of calling the `predict` method of a detection model.
+
+      Returns:
+        tf.float32 tensor of shape [batch_size, num_anchors, num_classes] if
+        background class is not present (else shape is
+        [batch_size, num_anchors, num_classes + 1]) on the interval [0, 1].
+      """
+      # Flattening Tensors and then reshaping at the end.
+      flat_class_predictions_with_background = tf.reshape(
+          class_predictions_with_background, shape=[-1])
+      fn_x, fn_y = _function_approximation_proto_to_tf_tensors(
+          calibration_config.function_approximation.x_y_pairs)
+      updated_scores = _tf_linear_interp1d(
+          flat_class_predictions_with_background, fn_x, fn_y)
+
+      # Un-flatten the scores
+      original_detections_shape = shape_utils.combined_static_and_dynamic_shape(
+          class_predictions_with_background)
+      calibrated_class_predictions_with_background = tf.reshape(
+          updated_scores,
+          shape=original_detections_shape,
+          name='calibrate_scores')
+      return calibrated_class_predictions_with_background
+
+  # TODO(zbeaver): Add sigmoid calibration and per-class isotonic regression.
+  else:
+    raise ValueError('No calibration builder defined for "Oneof" in '
+                     'calibration_config.')
+
+  return calibration_fn

research/object_detection/builders/calibration_builder_test.py

+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Tests for calibration_builder."""
+
+import numpy as np
+from scipy import interpolate
+import tensorflow as tf
+from object_detection.builders import calibration_builder
+from object_detection.protos import calibration_pb2
+
+
+class CalibrationBuilderTest(tf.test.TestCase):
+
+  def test_tf_linear_interp1d_map(self):
+    """Tests TF linear interpolation mapping to a single number."""
+    with self.test_session() as sess:
+      tf_x = tf.constant([0., 0.5, 1.])
+      tf_y = tf.constant([0.5, 0.5, 0.5])
+      new_x = tf.constant([0., 0.25, 0.5, 0.75, 1.])
+      tf_map_outputs = calibration_builder._tf_linear_interp1d(
+          new_x, tf_x, tf_y)
+      tf_map_outputs_np = sess.run([tf_map_outputs])
+    self.assertAllClose(tf_map_outputs_np, [[0.5, 0.5, 0.5, 0.5, 0.5]])
+
+  def test_tf_linear_interp1d_interpolate(self):
+    """Tests TF 1d linear interpolation not mapping to a single number."""
+    with self.test_session() as sess:
+      tf_x = tf.constant([0., 0.5, 1.])
+      tf_y = tf.constant([0.6, 0.7, 1.0])
+      new_x = tf.constant([0., 0.25, 0.5, 0.75, 1.])
+      tf_interpolate_outputs = calibration_builder._tf_linear_interp1d(
+          new_x, tf_x, tf_y)
+      tf_interpolate_outputs_np = sess.run([tf_interpolate_outputs])
+    self.assertAllClose(tf_interpolate_outputs_np, [[0.6, 0.65, 0.7, 0.85, 1.]])
+
+  @staticmethod
+  def _get_scipy_interp1d(new_x, x, y):
+    """Helper performing 1d linear interpolation using SciPy."""
+    interpolation1d_fn = interpolate.interp1d(x, y)
+    return interpolation1d_fn(new_x)
+
+  def _get_tf_interp1d(self, new_x, x, y):
+    """Helper performing 1d linear interpolation using Tensorflow."""
+    with self.test_session() as sess:
+      tf_interp_outputs = calibration_builder._tf_linear_interp1d(
+          tf.convert_to_tensor(new_x, dtype=tf.float32),
+          tf.convert_to_tensor(x, dtype=tf.float32),
+          tf.convert_to_tensor(y, dtype=tf.float32))
+      np_tf_interp_outputs = sess.run(tf_interp_outputs)
+    return np_tf_interp_outputs
+
+  def test_tf_linear_interp1d_against_scipy_map(self):
+    """Tests parity of TF linear interpolation with SciPy for simple mapping."""
+    length = 10
+    np_x = np.linspace(0, 1, length)
+
+    # Mapping all numbers to 0.5
+    np_y_map = np.repeat(0.5, length)
+
+    # Scipy and TF interpolations
+    test_data_np = np.linspace(0, 1, length * 10)
+    scipy_map_outputs = self._get_scipy_interp1d(test_data_np, np_x, np_y_map)
+    np_tf_map_outputs = self._get_tf_interp1d(test_data_np, np_x, np_y_map)
+    self.assertAllClose(scipy_map_outputs, np_tf_map_outputs)
+
+  def test_tf_linear_interp1d_against_scipy_interpolate(self):
+    """Tests parity of TF linear interpolation with SciPy."""
+    length = 10
+    np_x = np.linspace(0, 1, length)
+
+    # Requires interpolation over 0.5 to 1 domain
+    np_y_interp = np.linspace(0.5, 1, length)
+
+    # Scipy interpolation for comparison
+    test_data_np = np.linspace(0, 1, length * 10)
+    scipy_interp_outputs = self._get_scipy_interp1d(test_data_np, np_x,
+                                                    np_y_interp)
+    np_tf_interp_outputs = self._get_tf_interp1d(test_data_np, np_x,
+                                                 np_y_interp)
+    self.assertAllClose(scipy_interp_outputs, np_tf_interp_outputs)
+
+  @staticmethod
+  def _add_function_approximation_to_calibration_proto(calibration_proto,
+                                                       x_array,
+                                                       y_array,
+                                                       class_label):
+    """Adds a function approximation to calibration proto for a class label."""
+    # Per-class calibration.
+    if class_label:
+      label_function_approximation = (calibration_proto
+                                      .label_function_approximations
+                                      .label_xy_pairs_map[class_label])
+    # Class-agnostic calibration.
+    else:
+      label_function_approximation = (calibration_proto
+                                      .function_approximation
+                                      .x_y_pairs)
+    for x, y in zip(x_array, y_array):
+      x_y_pair_message = label_function_approximation.x_y_pair.add()
+      x_y_pair_message.x = x
+      x_y_pair_message.y = y
+
+  def test_class_agnostic_function_approximation(self):
+    """Ensures that calibration appropriate values, regardless of class."""
+    # Generate fake calibration proto. For this interpolation, any input on
+    # [0.0, 0.5] should be divided by 2 and any input on (0.5, 1.0] should have
+    # 0.25 subtracted from it.
+    class_agnostic_x = np.asarray([0.0, 0.5, 1.0])
+    class_agnostic_y = np.asarray([0.0, 0.25, 0.75])
+    calibration_config = calibration_pb2.CalibrationConfig()
+    self._add_function_approximation_to_calibration_proto(calibration_config,
+                                                          class_agnostic_x,
+                                                          class_agnostic_y,
+                                                          class_label=None)
+
+    od_graph = tf.Graph()
+    with self.test_session(graph=od_graph) as sess:
+      calibration_fn = calibration_builder.build(calibration_config)
+      # batch_size = 2, num_classes = 2, num_anchors = 2.
+      class_predictions_with_background = tf.constant(
+          [[[0.1, 0.2, 0.3],
+            [0.4, 0.5, 0.0]],
+           [[0.6, 0.7, 0.8],
+            [0.9, 1.0, 1.0]]], dtype=tf.float32)
+
+      # Everything should map to 0.5 if classes are ignored.
+      calibrated_scores = calibration_fn(class_predictions_with_background)
+      calibrated_scores_np = sess.run(calibrated_scores)
+    self.assertAllClose(calibrated_scores_np, [[[0.05, 0.1, 0.15],
+                                                [0.2, 0.25, 0.0]],
+                                               [[0.35, 0.45, 0.55],
+                                                [0.65, 0.75, 0.75]]])
+
+if __name__ == '__main__':
+  tf.test.main()

research/object_detection/builders/dataset_builder.py

@@ -117,6 +117,7 @@ def build(input_reader_config, batch_size=None, transform_input_data_fn=None):
       label_map_proto_file = input_reader_config.label_map_path
     decoder = tf_example_decoder.TfExampleDecoder(
         load_instance_masks=input_reader_config.load_instance_masks,
+        load_multiclass_scores=input_reader_config.load_multiclass_scores,
         instance_mask_type=input_reader_config.mask_type,
         label_map_proto_file=label_map_proto_file,
         use_display_name=input_reader_config.use_display_name,
@@ -140,9 +141,12 @@ def build(input_reader_config, batch_size=None, transform_input_data_fn=None):
       num_parallel_calls = batch_size * input_reader_config.num_parallel_batches
     else:
       num_parallel_calls = input_reader_config.num_parallel_map_calls
-    dataset = dataset.map(
-        process_fn,
-        num_parallel_calls=num_parallel_calls)
+    # TODO(b/123952794): Migrate to V2 function.
+    if hasattr(dataset, 'map_with_legacy_function'):
+      data_map_fn = dataset.map_with_legacy_function
+    else:
+      data_map_fn = dataset.map
+    dataset = data_map_fn(process_fn, num_parallel_calls=num_parallel_calls)
     if batch_size:
       dataset = dataset.apply(
           tf.contrib.data.batch_and_drop_remainder(batch_size))

research/object_detection/builders/image_resizer_builder.py

@@ -102,6 +102,14 @@ def build(image_resizer_config):
         method=method)
     if not fixed_shape_resizer_config.convert_to_grayscale:
       return image_resizer_fn
+  elif image_resizer_oneof == 'identity_resizer':
+    def image_resizer_fn(image, masks=None, **kwargs):
+      del kwargs
+      if masks is None:
+        return [image, tf.shape(image)]
+      else:
+        return [image, masks, tf.shape(image)]
+    return image_resizer_fn
   else:
     raise ValueError(
         'Invalid image resizer option: \'%s\'.' % image_resizer_oneof)

research/object_detection/builders/image_resizer_builder_test.py

@@ -104,6 +104,17 @@ class ImageResizerBuilderTest(tf.test.TestCase):
         input_shape, image_resizer_text_proto)
     self.assertEqual(output_shape, expected_output_shape)
 
+  def test_identity_resizer_returns_expected_shape(self):
+    image_resizer_text_proto = """
+      identity_resizer {
+      }
+    """
+    input_shape = (10, 20, 3)
+    expected_output_shape = (10, 20, 3)
+    output_shape = self._shape_of_resized_random_image_given_text_proto(
+        input_shape, image_resizer_text_proto)
+    self.assertEqual(output_shape, expected_output_shape)
+
   def test_raises_error_on_invalid_input(self):
     invalid_input = 'invalid_input'
     with self.assertRaises(ValueError):

research/object_detection/builders/model_builder.py

@@ -44,6 +44,7 @@ from object_detection.models.ssd_inception_v2_feature_extractor import SSDIncept
 from object_detection.models.ssd_inception_v3_feature_extractor import SSDInceptionV3FeatureExtractor
 from object_detection.models.ssd_mobilenet_v1_feature_extractor import SSDMobileNetV1FeatureExtractor
 from object_detection.models.ssd_mobilenet_v1_fpn_feature_extractor import SSDMobileNetV1FpnFeatureExtractor
+from object_detection.models.ssd_mobilenet_v1_keras_feature_extractor import SSDMobileNetV1KerasFeatureExtractor
 from object_detection.models.ssd_mobilenet_v1_ppn_feature_extractor import SSDMobileNetV1PpnFeatureExtractor
 from object_detection.models.ssd_mobilenet_v2_feature_extractor import SSDMobileNetV2FeatureExtractor
 from object_detection.models.ssd_mobilenet_v2_fpn_feature_extractor import SSDMobileNetV2FpnFeatureExtractor
@@ -76,6 +77,7 @@ SSD_FEATURE_EXTRACTOR_CLASS_MAP = {
 }
 
 SSD_KERAS_FEATURE_EXTRACTOR_CLASS_MAP = {
+    'ssd_mobilenet_v1_keras': SSDMobileNetV1KerasFeatureExtractor,
     'ssd_mobilenet_v2_keras': SSDMobileNetV2KerasFeatureExtractor
 }
 
@@ -187,6 +189,12 @@ def _build_ssd_feature_extractor(feature_extractor_config,
           override_base_feature_extractor_hyperparams
   }
 
+  if feature_extractor_config.HasField('replace_preprocessor_with_placeholder'):
+    kwargs.update({
+        'replace_preprocessor_with_placeholder':
+            feature_extractor_config.replace_preprocessor_with_placeholder
+    })
+
   if is_keras_extractor:
     kwargs.update({
         'conv_hyperparams': conv_hyperparams,

research/object_detection/builders/optimizer_builder.py

@@ -16,6 +16,8 @@
 """Functions to build DetectionModel training optimizers."""
 
 import tensorflow as tf
+
+
 from object_detection.utils import learning_schedules
 
 
@@ -59,6 +61,7 @@ def build(optimizer_config):
     summary_vars.append(learning_rate)
     optimizer = tf.train.AdamOptimizer(learning_rate)
 
+
   if optimizer is None:
     raise ValueError('Optimizer %s not supported.' % optimizer_type)
 

research/object_detection/builders/post_processing_builder.py

-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,6 +17,7 @@
 import functools
 
 import tensorflow as tf
+from object_detection.builders import calibration_builder
 from object_detection.core import post_processing
 from object_detection.protos import post_processing_pb2
 
@@ -24,8 +25,8 @@ from object_detection.protos import post_processing_pb2
 def build(post_processing_config):
   """Builds callables for post-processing operations.
 
-  Builds callables for non-max suppression and score conversion based on the
-  configuration.
+  Builds callables for non-max suppression, score conversion, and (optionally)
+  calibration based on the configuration.
 
   Non-max suppression callable takes `boxes`, `scores`, and optionally
   `clip_window`, `parallel_iterations` `masks, and `scope` as inputs. It returns
@@ -35,8 +36,10 @@ def build(post_processing_config):
 
   Score converter callable should be called with `input` tensor. The callable
   returns the output from one of 3 tf operations based on the configuration -
-  tf.identity, tf.sigmoid or tf.nn.softmax. See tensorflow documentation for
-  argument and return value descriptions.
+  tf.identity, tf.sigmoid or tf.nn.softmax. If a calibration config is provided,
+  score_converter also applies calibration transformations, as defined in
+  calibration_builder.py. See tensorflow documentation for argument and return
+  value descriptions.
 
   Args:
     post_processing_config: post_processing.proto object containing the
@@ -57,6 +60,10 @@ def build(post_processing_config):
   score_converter_fn = _build_score_converter(
       post_processing_config.score_converter,
       post_processing_config.logit_scale)
+  if post_processing_config.HasField('calibration_config'):
+    score_converter_fn = _build_calibrated_score_converter(
+        score_converter_fn,
+        post_processing_config.calibration_config)
   return non_max_suppressor_fn, score_converter_fn
 
 
@@ -122,3 +129,32 @@ def _build_score_converter(score_converter_config, logit_scale):
   if score_converter_config == post_processing_pb2.PostProcessing.SOFTMAX:
     return _score_converter_fn_with_logit_scale(tf.nn.softmax, logit_scale)
   raise ValueError('Unknown score converter.')
+
+
+def _build_calibrated_score_converter(score_converter_fn, calibration_config):
+  """Wraps a score_converter_fn, adding a calibration step.
+
+  Builds a score converter function witha calibration transformation according
+  to calibration_builder.py. Calibration applies positive monotonic
+  transformations to inputs (i.e. score ordering is strictly preserved or
+  adjacent scores are mapped to the same score). When calibration is
+  class-agnostic, the highest-scoring class remains unchanged, unless two
+  adjacent scores are mapped to the same value and one class arbitrarily
+  selected to break the tie. In per-class calibration, it's possible (though
+  rare in practice) that the highest-scoring class will change, since positive
+  monotonicity is only required to hold within each class.
+
+  Args:
+    score_converter_fn: callable that takes logit scores as input.
+    calibration_config: post_processing_pb2.PostProcessing.calibration_config.
+
+  Returns:
+    Callable calibrated score coverter op.
+  """
+  calibration_fn = calibration_builder.build(calibration_config)
+  def calibrated_score_converter_fn(logits):
+    converted_logits = score_converter_fn(logits)
+    return calibration_fn(converted_logits)
+  calibrated_score_converter_fn.__name__ = (
+      'calibrate_with_%s' % calibration_config.WhichOneof('calibrator'))
+  return calibrated_score_converter_fn

research/object_detection/builders/post_processing_builder_test.py

@@ -47,7 +47,8 @@ class PostProcessingBuilderTest(tf.test.TestCase):
     """
     post_processing_config = post_processing_pb2.PostProcessing()
     text_format.Merge(post_processing_text_proto, post_processing_config)
-    _, score_converter = post_processing_builder.build(post_processing_config)
+    _, score_converter = post_processing_builder.build(
+        post_processing_config)
     self.assertEqual(score_converter.__name__, 'identity_with_logit_scale')
 
     inputs = tf.constant([1, 1], tf.float32)
@@ -102,6 +103,36 @@ class PostProcessingBuilderTest(tf.test.TestCase):
     _, score_converter = post_processing_builder.build(post_processing_config)
     self.assertEqual(score_converter.__name__, 'softmax_with_logit_scale')
 
+  def test_build_calibrator_with_nonempty_config(self):
+    """Test that identity function used when no calibration_config specified."""
+    # Calibration config maps all scores to 0.5.
+    post_processing_text_proto = """
+      score_converter: SOFTMAX
+      calibration_config {
+        function_approximation {
+          x_y_pairs {
+              x_y_pair {
+                x: 0.0
+                y: 0.5
+              }
+              x_y_pair {
+                x: 1.0
+                y: 0.5
+              }}}}"""
+    post_processing_config = post_processing_pb2.PostProcessing()
+    text_format.Merge(post_processing_text_proto, post_processing_config)
+    _, calibrated_score_conversion_fn = post_processing_builder.build(
+        post_processing_config)
+    self.assertEqual(calibrated_score_conversion_fn.__name__,
+                     'calibrate_with_function_approximation')
+
+    input_scores = tf.constant([1, 1], tf.float32)
+    outputs = calibrated_score_conversion_fn(input_scores)
+    with self.test_session() as sess:
+      calibrated_scores = sess.run(outputs)
+      expected_calibrated_scores = sess.run(tf.constant([0.5, 0.5], tf.float32))
+      self.assertAllClose(calibrated_scores, expected_calibrated_scores)
+
 
 if __name__ == '__main__':
   tf.test.main()

research/object_detection/builders/preprocessor_builder.py

@@ -191,10 +191,10 @@ def build(preprocessor_step_config):
 
     pad_color = config.pad_color or None
     if pad_color:
-      if len(pad_color) == 3:
-        pad_color = tf.to_float([x for x in config.pad_color])
-      else:
-        raise ValueError('pad_color should have 3 elements (RGB) if set!')
+      if len(pad_color) != 3:
+        tf.logging.warn('pad_color should have 3 elements (RGB) if set!')
+
+      pad_color = tf.to_float([x for x in config.pad_color])
     return (preprocessor.random_pad_image,
             {
                 'min_image_size': min_image_size,
@@ -202,6 +202,25 @@ def build(preprocessor_step_config):
                 'pad_color': pad_color,
             })
 
+  if step_type == 'random_absolute_pad_image':
+    config = preprocessor_step_config.random_absolute_pad_image
+
+    max_height_padding = config.max_height_padding or 1
+    max_width_padding = config.max_width_padding or 1
+
+    pad_color = config.pad_color or None
+    if pad_color:
+      if len(pad_color) != 3:
+        tf.logging.warn('pad_color should have 3 elements (RGB) if set!')
+
+      pad_color = tf.to_float([x for x in config.pad_color])
+
+    return (preprocessor.random_absolute_pad_image,
+            {
+                'max_height_padding': max_height_padding,
+                'max_width_padding': max_width_padding,
+                'pad_color': pad_color,
+            })
   if step_type == 'random_crop_pad_image':
     config = preprocessor_step_config.random_crop_pad_image
     min_padded_size_ratio = config.min_padded_size_ratio
@@ -210,9 +229,13 @@ def build(preprocessor_step_config):
     max_padded_size_ratio = config.max_padded_size_ratio
     if max_padded_size_ratio and len(max_padded_size_ratio) != 2:
       raise ValueError('max_padded_size_ratio should have 2 elements if set!')
-    pad_color = config.pad_color
-    if pad_color and len(pad_color) != 3:
-      raise ValueError('pad_color should have 3 elements if set!')
+    pad_color = config.pad_color or None
+    if pad_color:
+      if len(pad_color) != 3:
+        tf.logging.warn('pad_color should have 3 elements (RGB) if set!')
+
+      pad_color = tf.to_float([x for x in config.pad_color])
+
     kwargs = {
         'min_object_covered': config.min_object_covered,
         'aspect_ratio_range': (config.min_aspect_ratio,
@@ -221,13 +244,12 @@ def build(preprocessor_step_config):
         'overlap_thresh': config.overlap_thresh,
         'clip_boxes': config.clip_boxes,
         'random_coef': config.random_coef,
+        'pad_color': pad_color,
     }
     if min_padded_size_ratio:
       kwargs['min_padded_size_ratio'] = tuple(min_padded_size_ratio)
     if max_padded_size_ratio:
       kwargs['max_padded_size_ratio'] = tuple(max_padded_size_ratio)
-    if pad_color:
-      kwargs['pad_color'] = tuple(pad_color)
     return (preprocessor.random_crop_pad_image, kwargs)
 
   if step_type == 'random_resize_method':
@@ -247,6 +269,13 @@ def build(preprocessor_step_config):
                 'method': method
             })
 
+  if step_type == 'random_self_concat_image':
+    config = preprocessor_step_config.random_self_concat_image
+    return (preprocessor.random_self_concat_image, {
+        'concat_vertical_probability': config.concat_vertical_probability,
+        'concat_horizontal_probability': config.concat_horizontal_probability
+    })
+
   if step_type == 'ssd_random_crop':
     config = preprocessor_step_config.ssd_random_crop
     if config.operations:

research/object_detection/builders/preprocessor_builder_test.py

@@ -254,6 +254,23 @@ class PreprocessorBuilderTest(tf.test.TestCase):
         'pad_color': None,
     })
 
+  def test_build_random_absolute_pad_image(self):
+    preprocessor_text_proto = """
+    random_absolute_pad_image {
+      max_height_padding: 50
+      max_width_padding: 100
+    }
+    """
+    preprocessor_proto = preprocessor_pb2.PreprocessingStep()
+    text_format.Merge(preprocessor_text_proto, preprocessor_proto)
+    function, args = preprocessor_builder.build(preprocessor_proto)
+    self.assertEqual(function, preprocessor.random_absolute_pad_image)
+    self.assertEqual(args, {
+        'max_height_padding': 50,
+        'max_width_padding': 100,
+        'pad_color': None,
+    })
+
   def test_build_random_crop_pad_image(self):
     preprocessor_text_proto = """
     random_crop_pad_image {
@@ -278,6 +295,7 @@ class PreprocessorBuilderTest(tf.test.TestCase):
         'overlap_thresh': 0.5,
         'clip_boxes': False,
         'random_coef': 0.125,
+        'pad_color': None,
     })
 
   def test_build_random_crop_pad_image_with_optional_parameters(self):
@@ -295,9 +313,6 @@ class PreprocessorBuilderTest(tf.test.TestCase):
       min_padded_size_ratio: 0.75
       max_padded_size_ratio: 0.5
       max_padded_size_ratio: 0.75
-      pad_color: 0.5
-      pad_color: 0.5
-      pad_color: 1.0
     }
     """
     preprocessor_proto = preprocessor_pb2.PreprocessingStep()
@@ -313,7 +328,7 @@ class PreprocessorBuilderTest(tf.test.TestCase):
         'random_coef': 0.125,
         'min_padded_size_ratio': (0.5, 0.75),
         'max_padded_size_ratio': (0.5, 0.75),
-        'pad_color': (0.5, 0.5, 1.0)
+        'pad_color': None,
     })
 
   def test_build_random_crop_to_aspect_ratio(self):
@@ -409,6 +424,20 @@ class PreprocessorBuilderTest(tf.test.TestCase):
     self.assertEqual(function, preprocessor.subtract_channel_mean)
     self.assertEqual(args, {'means': [1.0, 2.0, 3.0]})
 
+  def test_random_self_concat_image(self):
+    preprocessor_text_proto = """
+    random_self_concat_image {
+      concat_vertical_probability: 0.5
+      concat_horizontal_probability: 0.25
+    }
+    """
+    preprocessor_proto = preprocessor_pb2.PreprocessingStep()
+    text_format.Merge(preprocessor_text_proto, preprocessor_proto)
+    function, args = preprocessor_builder.build(preprocessor_proto)
+    self.assertEqual(function, preprocessor.random_self_concat_image)
+    self.assertEqual(args, {'concat_vertical_probability': 0.5,
+                            'concat_horizontal_probability': 0.25})
+
   def test_build_ssd_random_crop(self):
     preprocessor_text_proto = """
     ssd_random_crop {

research/object_detection/builders/region_similarity_calculator_builder.py

@@ -53,7 +53,7 @@ def build(region_similarity_calculator_config):
     return region_similarity_calculator.NegSqDistSimilarity()
   if similarity_calculator == 'thresholded_iou_similarity':
     return region_similarity_calculator.ThresholdedIouSimilarity(
-        region_similarity_calculator_config.thresholded_iou_similarity.threshold
-    )
+        region_similarity_calculator_config.thresholded_iou_similarity
+        .iou_threshold)
 
   raise ValueError('Unknown region similarity calculator.')

research/object_detection/core/losses.py

@@ -61,8 +61,8 @@ class Loss(object):
         shouldn't be factored into the loss.
       losses_mask: A [batch] boolean tensor that indicates whether losses should
         be applied to individual images in the batch. For elements that
-        are True, corresponding prediction, target, and weight tensors will be
-        removed prior to loss computation. If None, no filtering will take place
+        are False, corresponding prediction, target, and weight tensors will not
+        contribute to loss computation. If None, no filtering will take place
         prior to loss computation.
       scope: Op scope name. Defaults to 'Loss' if None.
       **params: Additional keyword arguments for specific implementations of

research/object_detection/core/model.py

@@ -54,15 +54,14 @@ By default, DetectionModels produce bounding box detections; However, we support
 a handful of auxiliary annotations associated with each bounding box, namely,
 instance masks and keypoints.
 """
-from abc import ABCMeta
-from abc import abstractmethod
+import abc
 
 from object_detection.core import standard_fields as fields
 
 
 class DetectionModel(object):
   """Abstract base class for detection models."""
-  __metaclass__ = ABCMeta
+  __metaclass__ = abc.ABCMeta
 
   def __init__(self, num_classes):
     """Constructor.
@@ -112,7 +111,7 @@ class DetectionModel(object):
     """
     return field in self._groundtruth_lists
 
-  @abstractmethod
+  @abc.abstractmethod
   def preprocess(self, inputs):
     """Input preprocessing.
 
@@ -155,7 +154,7 @@ class DetectionModel(object):
     """
     pass
 
-  @abstractmethod
+  @abc.abstractmethod
   def predict(self, preprocessed_inputs, true_image_shapes):
     """Predict prediction tensors from inputs tensor.
 
@@ -175,10 +174,14 @@ class DetectionModel(object):
     """
     pass
 
-  @abstractmethod
+  @abc.abstractmethod
   def postprocess(self, prediction_dict, true_image_shapes, **params):
     """Convert predicted output tensors to final detections.
 
+    This stage typically performs a few things such as
+    * Non-Max Suppression to remove overlapping detection boxes.
+    * Score conversion and background class removal.
+
     Outputs adhere to the following conventions:
     * Classes are integers in [0, num_classes); background classes are removed
       and the first non-background class is mapped to 0. If the model produces
@@ -212,10 +215,20 @@ class DetectionModel(object):
           (optional)
         keypoints: [batch, max_detections, num_keypoints, 2] (optional)
         num_detections: [batch]
+
+        In addition to the above fields this stage also outputs the following
+        raw tensors:
+
+        raw_detection_boxes: [batch, total_detections, 4] tensor containing
+          all detection boxes from `prediction_dict` in the format
+          [ymin, xmin, ymax, xmax] and normalized co-ordinates.
+        raw_detection_scores: [batch, total_detections,
+          num_classes_with_background] tensor of class score logits for
+          raw detection boxes.
     """
     pass
 
-  @abstractmethod
+  @abc.abstractmethod
   def loss(self, prediction_dict, true_image_shapes):
     """Compute scalar loss tensors with respect to provided groundtruth.
 
@@ -296,7 +309,7 @@ class DetectionModel(object):
       self._groundtruth_lists[
           fields.InputDataFields.is_annotated] = is_annotated_list
 
-  @abstractmethod
+  @abc.abstractmethod
   def regularization_losses(self):
     """Returns a list of regularization losses for this model.
 
@@ -308,7 +321,7 @@ class DetectionModel(object):
     """
     pass
 
-  @abstractmethod
+  @abc.abstractmethod
   def restore_map(self, fine_tune_checkpoint_type='detection'):
     """Returns a map of variables to load from a foreign checkpoint.
 
@@ -332,7 +345,7 @@ class DetectionModel(object):
     """
     pass
 
-  @abstractmethod
+  @abc.abstractmethod
   def updates(self):
     """Returns a list of update operators for this model.
 

research/object_detection/core/post_processing_test.py

@@ -57,7 +57,53 @@ class MulticlassNonMaxSuppressionTest(test_case.TestCase):
       self.assertAllClose(nms_scores_output, exp_nms_scores)
       self.assertAllClose(nms_classes_output, exp_nms_classes)
 
-  # TODO(bhattad): Remove conditional after CMLE moves to TF 1.9
+  def test_multiclass_nms_select_with_shared_boxes_pad_to_max_output_size(self):
+    boxes = np.array([[[0, 0, 1, 1]],
+                      [[0, 0.1, 1, 1.1]],
+                      [[0, -0.1, 1, 0.9]],
+                      [[0, 10, 1, 11]],
+                      [[0, 10.1, 1, 11.1]],
+                      [[0, 100, 1, 101]],
+                      [[0, 1000, 1, 1002]],
+                      [[0, 1000, 1, 1002.1]]], np.float32)
+    scores = np.array([[.9, 0.01], [.75, 0.05],
+                       [.6, 0.01], [.95, 0],
+                       [.5, 0.01], [.3, 0.01],
+                       [.01, .85], [.01, .5]], np.float32)
+    score_thresh = 0.1
+    iou_thresh = .5
+    max_size_per_class = 4
+    max_output_size = 5
+
+    exp_nms_corners = [[0, 10, 1, 11],
+                       [0, 0, 1, 1],
+                       [0, 1000, 1, 1002],
+                       [0, 100, 1, 101]]
+    exp_nms_scores = [.95, .9, .85, .3]
+    exp_nms_classes = [0, 0, 1, 0]
+
+    def graph_fn(boxes, scores):
+      nms, num_valid_nms_boxes = post_processing.multiclass_non_max_suppression(
+          boxes,
+          scores,
+          score_thresh,
+          iou_thresh,
+          max_size_per_class,
+          max_total_size=max_output_size,
+          pad_to_max_output_size=True)
+      return [nms.get(), nms.get_field(fields.BoxListFields.scores),
+              nms.get_field(fields.BoxListFields.classes), num_valid_nms_boxes]
+
+    [nms_corners_output, nms_scores_output, nms_classes_output,
+     num_valid_nms_boxes] = self.execute(graph_fn, [boxes, scores])
+
+    self.assertEqual(num_valid_nms_boxes, 4)
+    self.assertAllClose(nms_corners_output[0:num_valid_nms_boxes],
+                        exp_nms_corners)
+    self.assertAllClose(nms_scores_output[0:num_valid_nms_boxes],
+                        exp_nms_scores)
+    self.assertAllClose(nms_classes_output[0:num_valid_nms_boxes],
+                        exp_nms_classes)
 
   def test_multiclass_nms_select_with_shared_boxes_given_keypoints(self):
     boxes = tf.constant([[[0, 0, 1, 1]],

research/object_detection/core/preprocessor_cache.py

@@ -51,6 +51,7 @@ class PreprocessorCache(object):
   ADD_BLACK_PATCH = 'add_black_patch'
   SELECTOR = 'selector'
   SELECTOR_TUPLES = 'selector_tuples'
+  SELF_CONCAT_IMAGE = 'self_concat_image'
   SSD_CROP_SELECTOR_ID = 'ssd_crop_selector_id'
   SSD_CROP_PAD_SELECTOR_ID = 'ssd_crop_pad_selector_id'
 
@@ -60,7 +61,8 @@ class PreprocessorCache(object):
                 ADJUST_HUE, ADJUST_SATURATION, DISTORT_COLOR, STRICT_CROP_IMAGE,
                 CROP_IMAGE, PAD_IMAGE, CROP_TO_ASPECT_RATIO, RESIZE_METHOD,
                 PAD_TO_ASPECT_RATIO, BLACK_PATCHES, ADD_BLACK_PATCH, SELECTOR,
-                SELECTOR_TUPLES, SSD_CROP_SELECTOR_ID, SSD_CROP_PAD_SELECTOR_ID]
+                SELECTOR_TUPLES, SELF_CONCAT_IMAGE, SSD_CROP_SELECTOR_ID,
+                SSD_CROP_PAD_SELECTOR_ID]
 
   def __init__(self):
     self._history = defaultdict(dict)
@@ -99,4 +101,3 @@ class PreprocessorCache(object):
     if function_id not in self._VALID_FNS:
       raise ValueError('Function id not recognized: %s.' % str(function_id))
     self._history[function_id][key] = value
-

research/object_detection/core/preprocessor_test.py

@@ -2071,6 +2071,96 @@ class PreprocessorTest(tf.test.TestCase):
       self.assertTrue(np.all((boxes_[:, 3] - boxes_[:, 1]) >= (
           padded_boxes_[:, 3] - padded_boxes_[:, 1])))
 
+  def testRandomPadImageWithKeypoints(self):
+    preprocessing_options = [(preprocessor.normalize_image, {
+        'original_minval': 0,
+        'original_maxval': 255,
+        'target_minval': 0,
+        'target_maxval': 1
+    })]
+
+    images = self.createTestImages()
+    boxes = self.createTestBoxes()
+    labels = self.createTestLabels()
+    keypoints = self.createTestKeypoints()
+    tensor_dict = {
+        fields.InputDataFields.image: images,
+        fields.InputDataFields.groundtruth_boxes: boxes,
+        fields.InputDataFields.groundtruth_classes: labels,
+        fields.InputDataFields.groundtruth_keypoints: keypoints,
+    }
+    tensor_dict = preprocessor.preprocess(tensor_dict, preprocessing_options)
+    images = tensor_dict[fields.InputDataFields.image]
+
+    preprocessing_options = [(preprocessor.random_pad_image, {})]
+    padded_tensor_dict = preprocessor.preprocess(tensor_dict,
+                                                 preprocessing_options)
+
+    padded_images = padded_tensor_dict[fields.InputDataFields.image]
+    padded_boxes = padded_tensor_dict[
+        fields.InputDataFields.groundtruth_boxes]
+    padded_keypoints = padded_tensor_dict[
+        fields.InputDataFields.groundtruth_keypoints]
+    boxes_shape = tf.shape(boxes)
+    padded_boxes_shape = tf.shape(padded_boxes)
+    keypoints_shape = tf.shape(keypoints)
+    padded_keypoints_shape = tf.shape(padded_keypoints)
+    images_shape = tf.shape(images)
+    padded_images_shape = tf.shape(padded_images)
+
+    with self.test_session() as sess:
+      (boxes_shape_, padded_boxes_shape_, keypoints_shape_,
+       padded_keypoints_shape_, images_shape_, padded_images_shape_, boxes_,
+       padded_boxes_, keypoints_, padded_keypoints_) = sess.run(
+           [boxes_shape, padded_boxes_shape, keypoints_shape,
+            padded_keypoints_shape, images_shape, padded_images_shape, boxes,
+            padded_boxes, keypoints, padded_keypoints])
+      self.assertAllEqual(boxes_shape_, padded_boxes_shape_)
+      self.assertAllEqual(keypoints_shape_, padded_keypoints_shape_)
+      self.assertTrue((images_shape_[1] >= padded_images_shape_[1] * 0.5).all)
+      self.assertTrue((images_shape_[2] >= padded_images_shape_[2] * 0.5).all)
+      self.assertTrue((images_shape_[1] <= padded_images_shape_[1]).all)
+      self.assertTrue((images_shape_[2] <= padded_images_shape_[2]).all)
+      self.assertTrue(np.all((boxes_[:, 2] - boxes_[:, 0]) >= (
+          padded_boxes_[:, 2] - padded_boxes_[:, 0])))
+      self.assertTrue(np.all((boxes_[:, 3] - boxes_[:, 1]) >= (
+          padded_boxes_[:, 3] - padded_boxes_[:, 1])))
+      self.assertTrue(np.all((keypoints_[1, :, 0] - keypoints_[0, :, 0]) >= (
+          padded_keypoints_[1, :, 0] - padded_keypoints_[0, :, 0])))
+      self.assertTrue(np.all((keypoints_[1, :, 1] - keypoints_[0, :, 1]) >= (
+          padded_keypoints_[1, :, 1] - padded_keypoints_[0, :, 1])))
+
+  def testRandomAbsolutePadImage(self):
+    images = self.createTestImages()
+    boxes = self.createTestBoxes()
+    labels = self.createTestLabels()
+    tensor_dict = {
+        fields.InputDataFields.image: tf.to_float(images),
+        fields.InputDataFields.groundtruth_boxes: boxes,
+        fields.InputDataFields.groundtruth_classes: labels,
+    }
+
+    height_padding = 10
+    width_padding = 20
+    preprocessing_options = [(preprocessor.random_absolute_pad_image, {
+        'max_height_padding': height_padding,
+        'max_width_padding': width_padding})]
+    padded_tensor_dict = preprocessor.preprocess(tensor_dict,
+                                                 preprocessing_options)
+
+    original_shape = tf.shape(images)
+    final_shape = tf.shape(padded_tensor_dict[fields.InputDataFields.image])
+
+    with self.test_session() as sess:
+      _, height, width, _ = sess.run(original_shape)
+      for _ in range(100):
+        output_shape = sess.run(final_shape)
+
+        self.assertTrue(output_shape[1] >= height)
+        self.assertTrue(output_shape[1] < height + height_padding)
+        self.assertTrue(output_shape[2] >= width)
+        self.assertTrue(output_shape[2] < width + width_padding)
+
   def testRandomCropPadImageWithCache(self):
     preprocess_options = [(preprocessor.normalize_image, {
         'original_minval': 0,
@@ -2693,6 +2783,95 @@ class PreprocessorTest(tf.test.TestCase):
 
       self.assertAllEqual([0, 1, 1, 0, 1], one_hot)
 
+  def testRandomSelfConcatImage(self):
+    tf.set_random_seed(24601)
+
+    images = self.createTestImages()
+    boxes = self.createTestBoxes()
+    labels = self.createTestLabels()
+    weights = self.createTestGroundtruthWeights()
+    confidences = weights
+    scores = self.createTestMultiClassScores()
+
+    tensor_dict = {
+        fields.InputDataFields.image: tf.to_float(images),
+        fields.InputDataFields.groundtruth_boxes: boxes,
+        fields.InputDataFields.groundtruth_classes: labels,
+        fields.InputDataFields.groundtruth_weights: weights,
+        fields.InputDataFields.groundtruth_confidences: confidences,
+        fields.InputDataFields.multiclass_scores: scores,
+    }
+
+    preprocessing_options = [(preprocessor.random_self_concat_image, {
+        'concat_vertical_probability': 0.5,
+        'concat_horizontal_probability': 0.5,
+        'seed': 24601,
+    })]
+    func_arg_map = preprocessor.get_default_func_arg_map(
+        True, True, True)
+    output_tensor_dict = preprocessor.preprocess(
+        tensor_dict, preprocessing_options, func_arg_map=func_arg_map)
+
+    final_shape = tf.shape(output_tensor_dict[fields.InputDataFields.image])[
+        1:3]
+
+    with self.test_session() as sess:
+      outputs = []
+
+      augment_height_only = False
+      augment_width_only = False
+
+      for _ in range(50):
+        original_boxes = sess.run(boxes)
+        shape, new_boxes, new_labels, new_confidences, new_scores = sess.run(
+            [final_shape,
+             output_tensor_dict[fields.InputDataFields.groundtruth_boxes],
+             output_tensor_dict[fields.InputDataFields.groundtruth_classes],
+             output_tensor_dict[fields.InputDataFields.groundtruth_confidences],
+             output_tensor_dict[fields.InputDataFields.multiclass_scores],
+            ])
+        shape = np.array(shape)
+        outputs.append(shape)
+
+        if np.array_equal(shape, [8, 4]):
+          augment_height_only = True
+          self.assertEqual(
+              new_boxes.shape[0], 2 * boxes.shape[0])
+
+          self.assertAllClose(new_boxes[:2, :] * [2.0, 1.0, 2.0, 1.0],
+                              original_boxes)
+          self.assertAllClose(
+              (new_boxes[2:, :] - [0.5, 0.0, 0.5, 0.0]) * [
+                  2.0, 1.0, 2.0, 1.0],
+              original_boxes)
+        elif np.array_equal(shape, [4, 8]):
+          augment_width_only = True
+          self.assertEqual(
+              new_boxes.shape[0], 2 * boxes.shape[0])
+
+          self.assertAllClose(new_boxes[:2, :] * [1.0, 2.0, 1.0, 2.0],
+                              original_boxes)
+          self.assertAllClose(
+              (new_boxes[2:, :] - [0.0, 0.5, 0.0, 0.5]) * [
+                  1.0, 2.0, 1.0, 2.0],
+              original_boxes)
+
+        augmentation_factor = new_boxes.shape[0] / boxes.shape[0].value
+        self.assertEqual(new_labels.shape[0],
+                         labels.shape[0].value * augmentation_factor)
+        self.assertEqual(new_confidences.shape[0],
+                         confidences.shape[0].value * augmentation_factor)
+        self.assertEqual(new_scores.shape[0],
+                         scores.shape[0].value * augmentation_factor)
+
+      max_height = max(x[0] for x in outputs)
+      max_width = max(x[1] for x in outputs)
+
+      self.assertEqual(max_height, 8)
+      self.assertEqual(max_width, 8)
+      self.assertEqual(augment_height_only, True)
+      self.assertEqual(augment_width_only, True)
+
   def testSSDRandomCropWithCache(self):
     preprocess_options = [
         (preprocessor.normalize_image, {

research/object_detection/core/standard_fields.py

@@ -114,6 +114,9 @@ class DetectionResultFields(object):
     detection_boundaries: contains an object boundary for each detection box.
     detection_keypoints: contains detection keypoints for each detection box.
     num_detections: number of detections in the batch.
+    raw_detection_boxes: contains decoded detection boxes without Non-Max
+      suppression.
+    raw_detection_scores: contains class score logits for raw detection boxes.
   """
 
   source_id = 'source_id'
@@ -125,6 +128,8 @@ class DetectionResultFields(object):
   detection_boundaries = 'detection_boundaries'
   detection_keypoints = 'detection_keypoints'
   num_detections = 'num_detections'
+  raw_detection_boxes = 'raw_detection_boxes'
+  raw_detection_scores = 'raw_detection_scores'
 
 
 class BoxListFields(object):

research/object_detection/core/target_assigner.py

@@ -166,6 +166,10 @@ class TargetAssigner(object):
         num_gt_boxes = groundtruth_boxes.num_boxes()
       groundtruth_weights = tf.ones([num_gt_boxes], dtype=tf.float32)
 
+    # set scores on the gt boxes
+    scores = 1 - groundtruth_labels[:, 0]
+    groundtruth_boxes.add_field(fields.BoxListFields.scores, scores)
+
     with tf.control_dependencies(
         [unmatched_shape_assert, labels_and_box_shapes_assert]):
       match_quality_matrix = self._similarity_calc.compare(groundtruth_boxes,

research/object_detection/data_decoders/tf_example_decoder.py

@@ -131,7 +131,8 @@ class TfExampleDecoder(data_decoder.DataDecoder):
                use_display_name=False,
                dct_method='',
                num_keypoints=0,
-               num_additional_channels=0):
+               num_additional_channels=0,
+               load_multiclass_scores=False):
     """Constructor sets keys_to_features and items_to_handlers.
 
     Args:
@@ -153,6 +154,8 @@ class TfExampleDecoder(data_decoder.DataDecoder):
         example, the jpeg library does not have that specific option.
       num_keypoints: the number of keypoints per object.
       num_additional_channels: how many additional channels to use.
+      load_multiclass_scores: Whether to load multiclass scores associated with
+        boxes.
 
     Raises:
       ValueError: If `instance_mask_type` option is not one of
@@ -205,6 +208,7 @@ class TfExampleDecoder(data_decoder.DataDecoder):
             tf.VarLenFeature(tf.int64),
         'image/object/weight':
             tf.VarLenFeature(tf.float32),
+
     }
     # We are checking `dct_method` instead of passing it directly in order to
     # ensure TF version 1.6 compatibility.
@@ -251,7 +255,13 @@ class TfExampleDecoder(data_decoder.DataDecoder):
             slim_example_decoder.Tensor('image/object/group_of')),
         fields.InputDataFields.groundtruth_weights: (
             slim_example_decoder.Tensor('image/object/weight')),
+
     }
+    if load_multiclass_scores:
+      self.keys_to_features[
+          'image/object/class/multiclass_scores'] = tf.VarLenFeature(tf.float32)
+      self.items_to_handlers[fields.InputDataFields.multiclass_scores] = (
+          slim_example_decoder.Tensor('image/object/class/multiclass_scores'))
     if num_additional_channels > 0:
       self.keys_to_features[
           'image/additional_channels/encoded'] = tf.FixedLenFeature(
@@ -355,6 +365,9 @@ class TfExampleDecoder(data_decoder.DataDecoder):
         shape [None, None, None] containing instance masks.
       fields.InputDataFields.groundtruth_image_classes - 1D uint64 of shape
         [None] containing classes for the boxes.
+      fields.InputDataFields.multiclass_scores - 1D float32 tensor of shape
+        [None * num_classes] containing flattened multiclass scores for
+        groundtruth boxes.
     """
     serialized_example = tf.reshape(tf_example_string_tensor, shape=[])
     decoder = slim_example_decoder.TFExampleDecoder(self.keys_to_features,

research/object_detection/data_decoders/tf_example_decoder_test.py

@@ -374,6 +374,43 @@ class TfExampleDecoderTest(tf.test.TestCase):
     self.assertAllEqual(bbox_classes,
                         tensor_dict[fields.InputDataFields.groundtruth_classes])
 
+  def testDecodeMultiClassScores(self):
+    image_tensor = np.random.randint(256, size=(4, 5, 3)).astype(np.uint8)
+    encoded_jpeg = self._EncodeImage(image_tensor)
+    bbox_ymins = [0.0, 4.0]
+    bbox_xmins = [1.0, 5.0]
+    bbox_ymaxs = [2.0, 6.0]
+    bbox_xmaxs = [3.0, 7.0]
+    flattened_multiclass_scores = [100., 50.] + [20., 30.]
+    example = tf.train.Example(
+        features=tf.train.Features(
+            feature={
+                'image/encoded':
+                    dataset_util.bytes_feature(encoded_jpeg),
+                'image/format':
+                    dataset_util.bytes_feature('jpeg'),
+                'image/object/class/multiclass_scores':
+                    dataset_util.float_list_feature(flattened_multiclass_scores
+                                                   ),
+                'image/object/bbox/ymin':
+                    dataset_util.float_list_feature(bbox_ymins),
+                'image/object/bbox/xmin':
+                    dataset_util.float_list_feature(bbox_xmins),
+                'image/object/bbox/ymax':
+                    dataset_util.float_list_feature(bbox_ymaxs),
+                'image/object/bbox/xmax':
+                    dataset_util.float_list_feature(bbox_xmaxs),
+            })).SerializeToString()
+
+    example_decoder = tf_example_decoder.TfExampleDecoder(
+        load_multiclass_scores=True)
+    tensor_dict = example_decoder.decode(tf.convert_to_tensor(example))
+    with self.test_session() as sess:
+      tensor_dict = sess.run(tensor_dict)
+
+    self.assertAllEqual(flattened_multiclass_scores,
+                        tensor_dict[fields.InputDataFields.multiclass_scores])
+
   def testDecodeObjectLabelNoText(self):
     image_tensor = np.random.randint(256, size=(4, 5, 3)).astype(np.uint8)
     encoded_jpeg = self._EncodeImage(image_tensor)
@@ -417,6 +454,51 @@ class TfExampleDecoderTest(tf.test.TestCase):
     self.assertAllEqual(bbox_classes,
                         tensor_dict[fields.InputDataFields.groundtruth_classes])
 
+  def testDecodeObjectLabelWithText(self):
+    image_tensor = np.random.randint(256, size=(4, 5, 3)).astype(np.uint8)
+    encoded_jpeg = self._EncodeImage(image_tensor)
+    bbox_classes_text = ['cat', 'dog']
+    # Annotation label gets overridden by labelmap id.
+    annotated_bbox_classes = [3, 4]
+    expected_bbox_classes = [1, 2]
+    example = tf.train.Example(
+        features=tf.train.Features(
+            feature={
+                'image/encoded':
+                    dataset_util.bytes_feature(encoded_jpeg),
+                'image/format':
+                    dataset_util.bytes_feature('jpeg'),
+                'image/object/class/text':
+                    dataset_util.bytes_list_feature(bbox_classes_text),
+                'image/object/class/label':
+                    dataset_util.int64_list_feature(annotated_bbox_classes),
+            })).SerializeToString()
+    label_map_string = """
+      item {
+        id:1
+        name:'cat'
+      }
+      item {
+        id:2
+        name:'dog'
+      }
+    """
+    label_map_path = os.path.join(self.get_temp_dir(), 'label_map.pbtxt')
+    with tf.gfile.Open(label_map_path, 'wb') as f:
+      f.write(label_map_string)
+
+    example_decoder = tf_example_decoder.TfExampleDecoder(
+        label_map_proto_file=label_map_path)
+    tensor_dict = example_decoder.decode(tf.convert_to_tensor(example))
+
+    init = tf.tables_initializer()
+    with self.test_session() as sess:
+      sess.run(init)
+      tensor_dict = sess.run(tensor_dict)
+
+    self.assertAllEqual(expected_bbox_classes,
+                        tensor_dict[fields.InputDataFields.groundtruth_classes])
+
   def testDecodeObjectLabelUnrecognizedName(self):
     image_tensor = np.random.randint(256, size=(4, 5, 3)).astype(np.uint8)
     encoded_jpeg = self._EncodeImage(image_tensor)
@@ -501,6 +583,50 @@ class TfExampleDecoderTest(tf.test.TestCase):
     self.assertAllEqual([3, 1],
                         tensor_dict[fields.InputDataFields.groundtruth_classes])
 
+  def testDecodeObjectLabelUnrecognizedNameWithMappingWithDisplayName(self):
+    image_tensor = np.random.randint(256, size=(4, 5, 3)).astype(np.uint8)
+    encoded_jpeg = self._EncodeImage(image_tensor)
+    bbox_classes_text = ['cat', 'cheetah']
+    bbox_classes_id = [5, 6]
+    example = tf.train.Example(
+        features=tf.train.Features(
+            feature={
+                'image/encoded':
+                    dataset_util.bytes_feature(encoded_jpeg),
+                'image/format':
+                    dataset_util.bytes_feature('jpeg'),
+                'image/object/class/text':
+                    dataset_util.bytes_list_feature(bbox_classes_text),
+                'image/object/class/label':
+                    dataset_util.int64_list_feature(bbox_classes_id),
+            })).SerializeToString()
+
+    label_map_string = """
+      item {
+        name:'/m/cat'
+        id:3
+        display_name:'cat'
+      }
+      item {
+        name:'/m/dog'
+        id:1
+        display_name:'dog'
+      }
+    """
+    label_map_path = os.path.join(self.get_temp_dir(), 'label_map.pbtxt')
+    with tf.gfile.Open(label_map_path, 'wb') as f:
+      f.write(label_map_string)
+    example_decoder = tf_example_decoder.TfExampleDecoder(
+        label_map_proto_file=label_map_path)
+    tensor_dict = example_decoder.decode(tf.convert_to_tensor(example))
+
+    with self.test_session() as sess:
+      sess.run(tf.tables_initializer())
+      tensor_dict = sess.run(tensor_dict)
+
+    self.assertAllEqual([3, -1],
+                        tensor_dict[fields.InputDataFields.groundtruth_classes])
+
   def testDecodeObjectLabelWithMappingWithName(self):
     image_tensor = np.random.randint(256, size=(4, 5, 3)).astype(np.uint8)
     encoded_jpeg = self._EncodeImage(image_tensor)

research/object_detection/eval_util.py

@@ -15,6 +15,7 @@
 """Common utility functions for evaluation."""
 import collections
 import os
+import re
 import time
 
 import numpy as np
@@ -233,7 +234,8 @@ def _run_checkpoint_once(tensor_dict,
                          save_graph=False,
                          save_graph_dir='',
                          losses_dict=None,
-                         eval_export_path=None):
+                         eval_export_path=None,
+                         process_metrics_fn=None):
   """Evaluates metrics defined in evaluators and returns summaries.
 
   This function loads the latest checkpoint in checkpoint_dirs and evaluates
@@ -275,6 +277,12 @@ def _run_checkpoint_once(tensor_dict,
     losses_dict: optional dictionary of scalar detection losses.
     eval_export_path: Path for saving a json file that contains the detection
       results in json format.
+    process_metrics_fn: a callback called with evaluation results after each
+      evaluation is done.  It could be used e.g. to back up checkpoints with
+      best evaluation scores, or to call an external system to update evaluation
+      results in order to drive best hyper-parameter search.  Parameters are:
+      int checkpoint_number, Dict[str, ObjectDetectionEvalMetrics] metrics,
+      str checkpoint_file path.
 
   Returns:
     global_step: the count of global steps.
@@ -291,6 +299,7 @@ def _run_checkpoint_once(tensor_dict,
   sess.run(tf.global_variables_initializer())
   sess.run(tf.local_variables_initializer())
   sess.run(tf.tables_initializer())
+  checkpoint_file = None
   if restore_fn:
     restore_fn(sess)
   else:
@@ -370,6 +379,15 @@ def _run_checkpoint_once(tensor_dict,
 
       for key, value in iter(aggregate_result_losses_dict.items()):
         all_evaluator_metrics['Losses/' + key] = np.mean(value)
+      if process_metrics_fn and checkpoint_file:
+        m = re.search(r'model.ckpt-(\d+)$', checkpoint_file)
+        if not m:
+          tf.logging.error('Failed to parse checkpoint number from: %s',
+                           checkpoint_file)
+        else:
+          checkpoint_number = int(m.group(1))
+          process_metrics_fn(checkpoint_number, all_evaluator_metrics,
+                             checkpoint_file)
   sess.close()
   return (global_step, all_evaluator_metrics)
 
@@ -385,11 +403,13 @@ def repeated_checkpoint_run(tensor_dict,
                             num_batches=1,
                             eval_interval_secs=120,
                             max_number_of_evaluations=None,
+                            max_evaluation_global_step=None,
                             master='',
                             save_graph=False,
                             save_graph_dir='',
                             losses_dict=None,
-                            eval_export_path=None):
+                            eval_export_path=None,
+                            process_metrics_fn=None):
   """Periodically evaluates desired tensors using checkpoint_dirs or restore_fn.
 
   This function repeatedly loads a checkpoint and evaluates a desired
@@ -425,6 +445,7 @@ def repeated_checkpoint_run(tensor_dict,
     eval_interval_secs: the number of seconds between each evaluation run.
     max_number_of_evaluations: the max number of iterations of the evaluation.
       If the value is left as None the evaluation continues indefinitely.
+    max_evaluation_global_step: global step when evaluation stops.
     master: the location of the Tensorflow session.
     save_graph: whether or not the Tensorflow graph is saved as a pbtxt file.
     save_graph_dir: where to save on disk the Tensorflow graph. If store_graph
@@ -432,6 +453,12 @@ def repeated_checkpoint_run(tensor_dict,
     losses_dict: optional dictionary of scalar detection losses.
     eval_export_path: Path for saving a json file that contains the detection
       results in json format.
+    process_metrics_fn: a callback called with evaluation results after each
+      evaluation is done.  It could be used e.g. to back up checkpoints with
+      best evaluation scores, or to call an external system to update evaluation
+      results in order to drive best hyper-parameter search.  Parameters are:
+      int checkpoint_number, Dict[str, ObjectDetectionEvalMetrics] metrics,
+      str checkpoint_file path.
 
   Returns:
     metrics: A dictionary containing metric names and values in the latest
@@ -443,7 +470,10 @@ def repeated_checkpoint_run(tensor_dict,
   """
   if max_number_of_evaluations and max_number_of_evaluations <= 0:
     raise ValueError(
-        '`number_of_steps` must be either None or a positive number.')
+        '`max_number_of_evaluations` must be either None or a positive number.')
+  if max_evaluation_global_step and max_evaluation_global_step <= 0:
+    raise ValueError(
+        '`max_evaluation_global_step` must be either None or positive.')
 
   if not checkpoint_dirs:
     raise ValueError('`checkpoint_dirs` must have at least one entry.')
@@ -475,8 +505,13 @@ def repeated_checkpoint_run(tensor_dict,
           save_graph,
           save_graph_dir,
           losses_dict=losses_dict,
-          eval_export_path=eval_export_path)
+          eval_export_path=eval_export_path,
+          process_metrics_fn=process_metrics_fn)
       write_metrics(metrics, global_step, summary_dir)
+      if (max_evaluation_global_step and
+          global_step >= max_evaluation_global_step):
+        tf.logging.info('Finished evaluation!')
+        break
     number_of_evaluations += 1
 
     if (max_number_of_evaluations and

research/object_detection/export_inference_graph.py

@@ -39,6 +39,12 @@ and the following output nodes returned by the model.postprocess(..):
       [batch, num_boxes] containing class scores for the detections.
   * `detection_classes`: Outputs float32 tensors of the form
       [batch, num_boxes] containing classes for the detections.
+  * `raw_detection_boxes`: Outputs float32 tensors of the form
+      [batch, raw_num_boxes, 4] containing detection boxes without
+      post-processing.
+  * `raw_detection_scores`: Outputs float32 tensors of the form
+      [batch, raw_num_boxes, num_classes_with_background] containing class score
+      logits for raw detection boxes.
   * `detection_masks`: Outputs float32 tensors of the form
       [batch, num_boxes, mask_height, mask_width] containing predicted instance
       masks for each box if its present in the dictionary of postprocessed

research/object_detection/export_tflite_ssd_graph_lib.py

@@ -154,7 +154,9 @@ def export_tflite_graph(pipeline_config,
                         max_detections,
                         max_classes_per_detection,
                         detections_per_class=100,
-                        use_regular_nms=False):
+                        use_regular_nms=False,
+                        binary_graph_name='tflite_graph.pb',
+                        txt_graph_name='tflite_graph.pbtxt'):
   """Exports a tflite compatible graph and anchors for ssd detection model.
 
   Anchors are written to a tensor and tflite compatible graph
@@ -174,6 +176,8 @@ def export_tflite_graph(pipeline_config,
     for NonMaxSuppression per class
     use_regular_nms: Flag to set postprocessing op to use Regular NMS instead
       of Fast NMS.
+    binary_graph_name: Name of the exported graph file in binary format.
+    txt_graph_name: Name of the exported graph file in text format.
 
   Raises:
     ValueError: if the pipeline config contains models other than ssd or uses an
@@ -304,9 +308,9 @@ def export_tflite_graph(pipeline_config,
     # Return frozen without adding post-processing custom op
     transformed_graph_def = frozen_graph_def
 
-  binary_graph = os.path.join(output_dir, 'tflite_graph.pb')
+  binary_graph = os.path.join(output_dir, binary_graph_name)
   with tf.gfile.GFile(binary_graph, 'wb') as f:
     f.write(transformed_graph_def.SerializeToString())
-  txt_graph = os.path.join(output_dir, 'tflite_graph.pbtxt')
+  txt_graph = os.path.join(output_dir, txt_graph_name)
   with tf.gfile.GFile(txt_graph, 'w') as f:
     f.write(str(transformed_graph_def))

research/object_detection/exporter.py

@@ -19,11 +19,7 @@ import tempfile
 import tensorflow as tf
 from tensorflow.contrib.quantize.python import graph_matcher
 from tensorflow.core.protobuf import saver_pb2
-from tensorflow.python.client import session
-from tensorflow.python.platform import gfile
-from tensorflow.python.saved_model import signature_constants
-from tensorflow.python.tools import freeze_graph
-from tensorflow.python.training import saver as saver_lib
+from tensorflow.python.tools import freeze_graph  # pylint: disable=g-direct-tensorflow-import
 from object_detection.builders import graph_rewriter_builder
 from object_detection.builders import model_builder
 from object_detection.core import standard_fields as fields
@@ -73,7 +69,8 @@ def rewrite_nn_resize_op(is_quantized=False):
     nn_resize = tf.image.resize_nearest_neighbor(
         projection_op.outputs[0],
         add_op.outputs[0].shape.dims[1:3],
-        align_corners=False)
+        align_corners=False,
+        name=os.path.split(reshape_2_op.name)[0] + '/resize_nearest_neighbor')
 
     for index, op_input in enumerate(add_op.inputs):
       if op_input == reshape_2_op.outputs[0]:
@@ -207,6 +204,8 @@ def add_output_tensor_nodes(postprocessed_tensors,
   label_id_offset = 1
   boxes = postprocessed_tensors.get(detection_fields.detection_boxes)
   scores = postprocessed_tensors.get(detection_fields.detection_scores)
+  raw_boxes = postprocessed_tensors.get(detection_fields.raw_detection_boxes)
+  raw_scores = postprocessed_tensors.get(detection_fields.raw_detection_scores)
   classes = postprocessed_tensors.get(
       detection_fields.detection_classes) + label_id_offset
   keypoints = postprocessed_tensors.get(detection_fields.detection_keypoints)
@@ -221,6 +220,12 @@ def add_output_tensor_nodes(postprocessed_tensors,
       classes, name=detection_fields.detection_classes)
   outputs[detection_fields.num_detections] = tf.identity(
       num_detections, name=detection_fields.num_detections)
+  if raw_boxes is not None:
+    outputs[detection_fields.raw_detection_boxes] = tf.identity(
+        raw_boxes, name=detection_fields.raw_detection_boxes)
+  if raw_scores is not None:
+    outputs[detection_fields.raw_detection_scores] = tf.identity(
+        raw_scores, name=detection_fields.raw_detection_scores)
   if keypoints is not None:
     outputs[detection_fields.detection_keypoints] = tf.identity(
         keypoints, name=detection_fields.detection_keypoints)
@@ -252,7 +257,7 @@ def write_saved_model(saved_model_path,
     outputs: A tensor dictionary containing the outputs of a DetectionModel.
   """
   with tf.Graph().as_default():
-    with session.Session() as sess:
+    with tf.Session() as sess:
 
       tf.import_graph_def(frozen_graph_def, name='')
 
@@ -268,12 +273,15 @@ def write_saved_model(saved_model_path,
           tf.saved_model.signature_def_utils.build_signature_def(
               inputs=tensor_info_inputs,
               outputs=tensor_info_outputs,
-              method_name=signature_constants.PREDICT_METHOD_NAME))
+              method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME
+          ))
 
       builder.add_meta_graph_and_variables(
-          sess, [tf.saved_model.tag_constants.SERVING],
+          sess,
+          [tf.saved_model.tag_constants.SERVING],
           signature_def_map={
-              signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
+              tf.saved_model.signature_constants
+              .DEFAULT_SERVING_SIGNATURE_DEF_KEY:
                   detection_signature,
           },
       )
@@ -289,9 +297,9 @@ def write_graph_and_checkpoint(inference_graph_def,
     node.device = ''
   with tf.Graph().as_default():
     tf.import_graph_def(inference_graph_def, name='')
-    with session.Session() as sess:
-      saver = saver_lib.Saver(saver_def=input_saver_def,
-                              save_relative_paths=True)
+    with tf.Session() as sess:
+      saver = tf.train.Saver(
+          saver_def=input_saver_def, save_relative_paths=True)
       saver.restore(sess, trained_checkpoint_prefix)
       saver.save(sess, model_path)
 
@@ -308,8 +316,8 @@ def _get_outputs_from_inputs(input_tensors, detection_model,
                                  output_collection_name)
 
 
-def _build_detection_graph(input_type, detection_model, input_shape,
-                           output_collection_name, graph_hook_fn):
+def build_detection_graph(input_type, detection_model, input_shape,
+                          output_collection_name, graph_hook_fn):
   """Build the detection graph."""
   if input_type not in input_placeholder_fn_map:
     raise ValueError('Unknown input type: {}'.format(input_type))
@@ -343,7 +351,8 @@ def _export_inference_graph(input_type,
                             input_shape=None,
                             output_collection_name='inference_op',
                             graph_hook_fn=None,
-                            write_inference_graph=False):
+                            write_inference_graph=False,
+                            temp_checkpoint_prefix=''):
   """Export helper."""
   tf.gfile.MakeDirs(output_directory)
   frozen_graph_path = os.path.join(output_directory,
@@ -351,7 +360,7 @@ def _export_inference_graph(input_type,
   saved_model_path = os.path.join(output_directory, 'saved_model')
   model_path = os.path.join(output_directory, 'model.ckpt')
 
-  outputs, placeholder_tensor = _build_detection_graph(
+  outputs, placeholder_tensor = build_detection_graph(
       input_type=input_type,
       detection_model=detection_model,
       input_shape=input_shape,
@@ -361,12 +370,13 @@ def _export_inference_graph(input_type,
   profile_inference_graph(tf.get_default_graph())
   saver_kwargs = {}
   if use_moving_averages:
-    # This check is to be compatible with both version of SaverDef.
-    if os.path.isfile(trained_checkpoint_prefix):
-      saver_kwargs['write_version'] = saver_pb2.SaverDef.V1
-      temp_checkpoint_prefix = tempfile.NamedTemporaryFile().name
-    else:
-      temp_checkpoint_prefix = tempfile.mkdtemp()
+    if not temp_checkpoint_prefix:
+      # This check is to be compatible with both version of SaverDef.
+      if os.path.isfile(trained_checkpoint_prefix):
+        saver_kwargs['write_version'] = saver_pb2.SaverDef.V1
+        temp_checkpoint_prefix = tempfile.NamedTemporaryFile().name
+      else:
+        temp_checkpoint_prefix = tempfile.mkdtemp()
     replace_variable_values_with_moving_averages(
         tf.get_default_graph(), trained_checkpoint_prefix,
         temp_checkpoint_prefix)
@@ -388,7 +398,7 @@ def _export_inference_graph(input_type,
                                         'inference_graph.pbtxt')
     for node in inference_graph_def.node:
       node.device = ''
-    with gfile.GFile(inference_graph_path, 'wb') as f:
+    with tf.gfile.GFile(inference_graph_path, 'wb') as f:
       f.write(str(inference_graph_def))
 
   if additional_output_tensor_names is not None:
@@ -486,4 +496,3 @@ def profile_inference_graph(graph):
   tf.contrib.tfprof.model_analyzer.print_model_analysis(
       graph,
       tfprof_options=tfprof_flops_option)
-

research/object_detection/exporter_test.py

@@ -61,7 +61,14 @@ class FakeModel(model.DetectionModel):
                                            [0.9, 0.0]], tf.float32),
           'detection_classes': tf.constant([[0, 1],
                                             [1, 0]], tf.float32),
-          'num_detections': tf.constant([2, 1], tf.float32)
+          'num_detections': tf.constant([2, 1], tf.float32),
+          'raw_detection_boxes': tf.constant([[[0.0, 0.0, 0.5, 0.5],
+                                               [0.5, 0.5, 0.8, 0.8]],
+                                              [[0.5, 0.5, 1.0, 1.0],
+                                               [0.0, 0.5, 0.0, 0.5]]],
+                                             tf.float32),
+          'raw_detection_scores': tf.constant([[0.7, 0.6],
+                                               [0.9, 0.5]], tf.float32),
       }
       if self._add_detection_keypoints:
         postprocessed_tensors['detection_keypoints'] = tf.constant(
@@ -612,7 +619,7 @@ class ExportInferenceGraphTest(tf.test.TestCase):
       pipeline_config.eval_config.use_moving_averages = False
       detection_model = model_builder.build(pipeline_config.model,
                                             is_training=False)
-      outputs, _ = exporter._build_detection_graph(
+      outputs, _ = exporter.build_detection_graph(
           input_type='tf_example',
           detection_model=detection_model,
           input_shape=None,
@@ -760,7 +767,7 @@ class ExportInferenceGraphTest(tf.test.TestCase):
       pipeline_config.eval_config.use_moving_averages = False
       detection_model = model_builder.build(pipeline_config.model,
                                             is_training=False)
-      outputs, placeholder_tensor = exporter._build_detection_graph(
+      outputs, placeholder_tensor = exporter.build_detection_graph(
           input_type='tf_example',
           detection_model=detection_model,
           input_shape=None,
@@ -893,7 +900,7 @@ class ExportInferenceGraphTest(tf.test.TestCase):
       pipeline_config.eval_config.use_moving_averages = False
       detection_model = model_builder.build(pipeline_config.model,
                                             is_training=False)
-      exporter._build_detection_graph(
+      exporter.build_detection_graph(
           input_type='tf_example',
           detection_model=detection_model,
           input_shape=None,
@@ -917,13 +924,16 @@ class ExportInferenceGraphTest(tf.test.TestCase):
         tf_example = od_graph.get_tensor_by_name('tf_example:0')
         boxes = od_graph.get_tensor_by_name('detection_boxes:0')
         scores = od_graph.get_tensor_by_name('detection_scores:0')
+        raw_boxes = od_graph.get_tensor_by_name('raw_detection_boxes:0')
+        raw_scores = od_graph.get_tensor_by_name('raw_detection_scores:0')
         classes = od_graph.get_tensor_by_name('detection_classes:0')
         keypoints = od_graph.get_tensor_by_name('detection_keypoints:0')
         masks = od_graph.get_tensor_by_name('detection_masks:0')
         num_detections = od_graph.get_tensor_by_name('num_detections:0')
-        (boxes_np, scores_np, classes_np, keypoints_np, masks_np,
-         num_detections_np) = sess.run(
-             [boxes, scores, classes, keypoints, masks, num_detections],
+        (boxes_np, scores_np, raw_boxes_np, raw_scores_np, classes_np,
+         keypoints_np, masks_np, num_detections_np) = sess.run(
+             [boxes, scores, raw_boxes, raw_scores, classes, keypoints, masks,
+              num_detections],
              feed_dict={tf_example: tf_example_np})
         self.assertAllClose(boxes_np, [[[0.0, 0.0, 0.5, 0.5],
                                         [0.5, 0.5, 0.8, 0.8]],
@@ -931,6 +941,12 @@ class ExportInferenceGraphTest(tf.test.TestCase):
                                         [0.0, 0.0, 0.0, 0.0]]])
         self.assertAllClose(scores_np, [[0.7, 0.6],
                                         [0.9, 0.0]])
+        self.assertAllClose(raw_boxes_np, [[[0.0, 0.0, 0.5, 0.5],
+                                            [0.5, 0.5, 0.8, 0.8]],
+                                           [[0.5, 0.5, 1.0, 1.0],
+                                            [0.0, 0.5, 0.0, 0.5]]])
+        self.assertAllClose(raw_scores_np, [[0.7, 0.6],
+                                            [0.9, 0.5]])
         self.assertAllClose(classes_np, [[1, 2],
                                          [2, 1]])
         self.assertAllClose(keypoints_np, np.arange(48).reshape([2, 2, 6, 2]))

research/object_detection/g3doc/detection_model_zoo.md

@@ -57,7 +57,7 @@ Some remarks on frozen inference graphs:
   a detector (and discarding the part past that point), which negatively impacts
   standard mAP metrics.
 * Our frozen inference graphs are generated using the
-  [v1.8.0](https://github.com/tensorflow/tensorflow/tree/v1.8.0)
+  [v1.12.0](https://github.com/tensorflow/tensorflow/tree/v1.12.0)
   release version of Tensorflow and we do not guarantee that these will work
   with other versions; this being said, each frozen inference graph can be
   regenerated using your current version of Tensorflow by re-running the
@@ -78,7 +78,7 @@ Some remarks on frozen inference graphs:
 | [ssd_mobilenet_v1_fpn_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gz) | 56 | 32 | Boxes |
 | [ssd_resnet_50_fpn_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gz) | 76 | 35 | Boxes |
 | [ssd_mobilenet_v2_coco](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz) | 31 | 22 | Boxes |
-| [ssd_mobilenet_v2_quantized_coco](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_quantized_300x300_coco_2018_09_14.tar.gz) | 29 | 22 | Boxes |
+| [ssd_mobilenet_v2_quantized_coco](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_quantized_300x300_coco_2019_01_03.tar.gz) | 29 | 22 | Boxes |
 | [ssdlite_mobilenet_v2_coco](http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz) | 27 | 22 | Boxes |
 | [ssd_inception_v2_coco](http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2018_01_28.tar.gz) | 42 | 24 | Boxes |
 | [faster_rcnn_inception_v2_coco](http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_v2_coco_2018_01_28.tar.gz) | 58 | 28 | Boxes |
@@ -110,10 +110,15 @@ Model name
 
 Model name                                                                                                                                                                                    | Speed (ms) | Open Images mAP@0.5[^2] | Outputs
 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------: | :---------------------: | :-----:
-[faster_rcnn_inception_resnet_v2_atrous_oidv2](http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_resnet_v2_atrous_oid_2018_01_28.tar.gz)                           | 727        | 37                      | Boxes
+[faster_rcnn_inception_resnet_v2_atrous_oidv2](http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_resnet_v2_atrous_oid_2018_01_28.tar.gz)                           | 727        | 37                     | Boxes
 [faster_rcnn_inception_resnet_v2_atrous_lowproposals_oidv2](http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_resnet_v2_atrous_lowproposals_oid_2018_01_28.tar.gz) | 347        |                         | Boxes
 [facessd_mobilenet_v2_quantized_open_image_v4](http://download.tensorflow.org/models/object_detection/facessd_mobilenet_v2_quantized_320x320_open_image_v4.tar.gz) [^3]                       | 20         | 73 (faces)              | Boxes
 
+Model name                                                                                                                                                                                    | Speed (ms) | Open Images mAP@0.5[^4] | Outputs
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------: | :---------------------: | :-----:
+[faster_rcnn_inception_resnet_v2_atrous_oidv4](http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_resnet_v2_atrous_oid_v4_2018_12_12.tar.gz)                         | 425        | 54                  | Boxes
+[ssd_mobilenetv2_oidv4](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_oid_v4_2018_12_12.tar.gz)                                                                       | 89         | 36                | Boxes
+[ssd_resnet_101_fpn_oidv4](http://download.tensorflow.org/models/object_detection/ssd_resnet101_v1_fpn_shared_box_predictor_oid_512x512_sync_2019_01_20.tar.gz)                                                                       | 237         | 38                | Boxes
 ## iNaturalist Species-trained models
 
 Model name                                                                                                                                                        | Speed (ms) | Pascal mAP@0.5 | Outputs
@@ -129,8 +134,10 @@ Model name
 [faster_rcnn_resnet101_ava_v2.1](http://download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_ava_v2.1_2018_04_30.tar.gz) | 93  | 11              | Boxes
 
 
-[^1]: See [MSCOCO evaluation protocol](http://cocodataset.org/#detections-eval).
+[^1]: See [MSCOCO evaluation protocol](http://cocodataset.org/#detections-eval). The COCO mAP numbers here are evaluated on COCO 14 minival set (note that our split is different from COCO 17 Val). A full list of image ids used in our split could be fould [here](https://github.com/tensorflow/models/blob/master/research/object_detection/data/mscoco_minival_ids.txt).
+
 
-[^2]: This is PASCAL mAP with a slightly different way of true positives computation: see [Open Images evaluation protocol](evaluation_protocols.md#open-images).
+[^2]: This is PASCAL mAP with a slightly different way of true positives computation: see [Open Images evaluation protocols](evaluation_protocols.md), oid_V2_detection_metrics.
 [^3]: Non-face boxes are dropped during training and non-face groundtruth boxes are ignored when evaluating.
+[^4]: This is Open Images Challenge metric: see [Open Images evaluation protocols](evaluation_protocols.md), oid_challenge_detection_metrics.
 

research/object_detection/g3doc/installation.md

@@ -11,7 +11,7 @@ Tensorflow Object Detection API depends on the following libraries:
 *   tf Slim (which is included in the "tensorflow/models/research/" checkout)
 *   Jupyter notebook
 *   Matplotlib
-*   Tensorflow (>=1.9.0)
+*   Tensorflow (>=1.12.0)
 *   Cython
 *   contextlib2
 *   cocoapi

research/object_detection/g3doc/running_on_cloud.md

@@ -44,7 +44,7 @@ job using GPUs. A sample YAML file is given below:
 
 ```
 trainingInput:
-  runtimeVersion: "1.9"
+  runtimeVersion: "1.12"
   scaleTier: CUSTOM
   masterType: standard_gpu
   workerCount: 9
@@ -73,7 +73,7 @@ following command:
 ```bash
 # From tensorflow/models/research/
 gcloud ml-engine jobs submit training object_detection_`date +%m_%d_%Y_%H_%M_%S` \
-    --runtime-version 1.9 \
+    --runtime-version 1.12 \
     --job-dir=gs://${MODEL_DIR} \
     --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz \
     --module-name object_detection.model_main \
@@ -93,7 +93,7 @@ Google Cloud Storage.
 Users can monitor the progress of their training job on the [ML Engine
 Dashboard](https://console.cloud.google.com/mlengine/jobs).
 
-Note: This sample is supported for use with 1.9 runtime version.
+Note: This sample is supported for use with 1.12 runtime version.
 
 ## Running a TPU Training Job on CMLE
 
@@ -105,7 +105,7 @@ gcloud ml-engine jobs submit training `whoami`_object_detection_`date +%m_%d_%Y_
 --job-dir=gs://${MODEL_DIR} \
 --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz \
 --module-name object_detection.model_tpu_main \
---runtime-version 1.9 \
+--runtime-version 1.12 \
 --scale-tier BASIC_TPU \
 --region us-central1 \
 -- \
@@ -133,7 +133,7 @@ job:
 
 ```bash
 gcloud ml-engine jobs submit training object_detection_eval_`date +%m_%d_%Y_%H_%M_%S` \
-    --runtime-version 1.9 \
+    --runtime-version 1.12 \
     --job-dir=gs://${MODEL_DIR} \
     --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz \
     --module-name object_detection.model_main \

research/object_detection/g3doc/running_pets.md

@@ -208,7 +208,7 @@ For running the training Cloud ML job, we'll configure the cluster to use 5
 training jobs and three parameters servers. The
 configuration file can be found at `object_detection/samples/cloud/cloud.yml`.
 
-Note: The code sample below is supported for use with 1.9 runtime version.
+Note: The code sample below is supported for use with 1.12 runtime version.
 
 To start training and evaluation, execute the following command from the
 `tensorflow/models/research/` directory:
@@ -216,7 +216,7 @@ To start training and evaluation, execute the following command from the
 ```bash
 # From tensorflow/models/research/
 gcloud ml-engine jobs submit training `whoami`_object_detection_pets_`date +%m_%d_%Y_%H_%M_%S` \
-    --runtime-version 1.9 \
+    --runtime-version 1.12 \
     --job-dir=gs://${YOUR_GCS_BUCKET}/model_dir \
     --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz \
     --module-name object_detection.model_main \

research/object_detection/g3doc/using_your_own_dataset.md

@@ -76,7 +76,7 @@ def create_cat_tf_example(encoded_cat_image_data):
       'image/width': dataset_util.int64_feature(width),
       'image/filename': dataset_util.bytes_feature(filename),
       'image/source_id': dataset_util.bytes_feature(filename),
-      'image/encoded': dataset_util.bytes_feature(encoded_cat_image_data),
+      'image/encoded': dataset_util.bytes_feature(encoded_image_data),
       'image/format': dataset_util.bytes_feature(image_format),
       'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
       'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),

research/object_detection/inputs.py

@@ -53,6 +53,7 @@ def transform_input_data(tensor_dict,
                          data_augmentation_fn=None,
                          merge_multiple_boxes=False,
                          retain_original_image=False,
+                         use_multiclass_scores=False,
                          use_bfloat16=False):
   """A single function that is responsible for all input data transformations.
 
@@ -87,25 +88,37 @@ def transform_input_data(tensor_dict,
       and classes for a given image if the boxes are exactly the same.
     retain_original_image: (optional) whether to retain original image in the
       output dictionary.
+    use_multiclass_scores: whether to use multiclass scores as
+      class targets instead of one-hot encoding of `groundtruth_classes`.
     use_bfloat16: (optional) a bool, whether to use bfloat16 in training.
 
   Returns:
     A dictionary keyed by fields.InputDataFields containing the tensors obtained
     after applying all the transformations.
   """
+  # Reshape flattened multiclass scores tensor into a 2D tensor of shape
+  # [num_boxes, num_classes].
+  if fields.InputDataFields.multiclass_scores in tensor_dict:
+    tensor_dict[fields.InputDataFields.multiclass_scores] = tf.reshape(
+        tensor_dict[fields.InputDataFields.multiclass_scores], [
+            tf.shape(tensor_dict[fields.InputDataFields.groundtruth_boxes])[0],
+            num_classes
+        ])
   if fields.InputDataFields.groundtruth_boxes in tensor_dict:
     tensor_dict = util_ops.filter_groundtruth_with_nan_box_coordinates(
         tensor_dict)
-  if fields.InputDataFields.image_additional_channels in tensor_dict:
-    channels = tensor_dict[fields.InputDataFields.image_additional_channels]
-    tensor_dict[fields.InputDataFields.image] = tf.concat(
-        [tensor_dict[fields.InputDataFields.image], channels], axis=2)
+    tensor_dict = util_ops.filter_unrecognized_classes(tensor_dict)
 
   if retain_original_image:
     tensor_dict[fields.InputDataFields.original_image] = tf.cast(
         image_resizer_fn(tensor_dict[fields.InputDataFields.image], None)[0],
         tf.uint8)
 
+  if fields.InputDataFields.image_additional_channels in tensor_dict:
+    channels = tensor_dict[fields.InputDataFields.image_additional_channels]
+    tensor_dict[fields.InputDataFields.image] = tf.concat(
+        [tensor_dict[fields.InputDataFields.image], channels], axis=2)
+
   # Apply data augmentation ops.
   if data_augmentation_fn is not None:
     tensor_dict = data_augmentation_fn(tensor_dict)
@@ -136,6 +149,11 @@ def transform_input_data(tensor_dict,
   tensor_dict[fields.InputDataFields.groundtruth_classes] = tf.one_hot(
       zero_indexed_groundtruth_classes, num_classes)
 
+  if use_multiclass_scores:
+    tensor_dict[fields.InputDataFields.groundtruth_classes] = tensor_dict[
+        fields.InputDataFields.multiclass_scores]
+    tensor_dict.pop(fields.InputDataFields.multiclass_scores, None)
+
   if fields.InputDataFields.groundtruth_confidences in tensor_dict:
     groundtruth_confidences = tensor_dict[
         fields.InputDataFields.groundtruth_confidences]
@@ -172,6 +190,9 @@ def pad_input_data_to_static_shapes(tensor_dict, max_num_boxes, num_classes,
                                     spatial_image_shape=None):
   """Pads input tensors to static shapes.
 
+  In case num_additional_channels > 0, we assume that the additional channels
+  have already been concatenated to the base image.
+
   Args:
     tensor_dict: Tensor dictionary of input data
     max_num_boxes: Max number of groundtruth boxes needed to compute shapes for
@@ -186,7 +207,8 @@ def pad_input_data_to_static_shapes(tensor_dict, max_num_boxes, num_classes,
     tensors in the dataset.
 
   Raises:
-    ValueError: If groundtruth classes is neither rank 1 nor rank 2.
+    ValueError: If groundtruth classes is neither rank 1 nor rank 2, or if we
+      detect that additional channels have not been concatenated yet.
   """
 
   if not spatial_image_shape or spatial_image_shape == [-1, -1]:
@@ -198,14 +220,27 @@ def pad_input_data_to_static_shapes(tensor_dict, max_num_boxes, num_classes,
   if fields.InputDataFields.image_additional_channels in tensor_dict:
     num_additional_channels = tensor_dict[
         fields.InputDataFields.image_additional_channels].shape[2].value
-  num_image_channels = 3
+
+  # We assume that if num_additional_channels > 0, then it has already been
+  # concatenated to the base image (but not the ground truth).
+  num_channels = 3
   if fields.InputDataFields.image in tensor_dict:
-    num_image_channels = tensor_dict[fields.InputDataFields
-                                     .image].shape[2].value
+    num_channels = tensor_dict[fields.InputDataFields.image].shape[2].value
+
+  if num_additional_channels:
+    if num_additional_channels >= num_channels:
+      raise ValueError(
+          'Image must be already concatenated with additional channels.')
+
+    if (fields.InputDataFields.original_image in tensor_dict and
+        tensor_dict[fields.InputDataFields.original_image].shape[2].value ==
+        num_channels):
+      raise ValueError(
+          'Image must be already concatenated with additional channels.')
+
   padding_shapes = {
-      # Additional channels are merged before batching.
       fields.InputDataFields.image: [
-          height, width, num_image_channels + num_additional_channels
+          height, width, num_channels
       ],
       fields.InputDataFields.original_image_spatial_shape: [2],
       fields.InputDataFields.image_additional_channels: [
@@ -231,16 +266,14 @@ def pad_input_data_to_static_shapes(tensor_dict, max_num_boxes, num_classes,
       fields.InputDataFields.groundtruth_label_types: [max_num_boxes],
       fields.InputDataFields.groundtruth_label_weights: [max_num_boxes],
       fields.InputDataFields.true_image_shape: [3],
-      fields.InputDataFields.multiclass_scores: [
-          max_num_boxes, num_classes + 1 if num_classes is not None else None
-      ],
       fields.InputDataFields.groundtruth_image_classes: [num_classes],
       fields.InputDataFields.groundtruth_image_confidences: [num_classes],
   }
 
   if fields.InputDataFields.original_image in tensor_dict:
     padding_shapes[fields.InputDataFields.original_image] = [
-        height, width, num_image_channels + num_additional_channels
+        height, width, tensor_dict[fields.InputDataFields.
+                                   original_image].shape[2].value
     ]
   if fields.InputDataFields.groundtruth_keypoints in tensor_dict:
     tensor_shape = (
@@ -294,11 +327,14 @@ def augment_input_data(tensor_dict, data_augmentation_options):
                            in tensor_dict)
   include_label_confidences = (fields.InputDataFields.groundtruth_confidences
                                in tensor_dict)
+  include_multiclass_scores = (fields.InputDataFields.multiclass_scores in
+                               tensor_dict)
   tensor_dict = preprocessor.preprocess(
       tensor_dict, data_augmentation_options,
       func_arg_map=preprocessor.get_default_func_arg_map(
           include_label_weights=include_label_weights,
           include_label_confidences=include_label_confidences,
+          include_multiclass_scores=include_multiclass_scores,
           include_instance_masks=include_instance_masks,
           include_keypoints=include_keypoints))
   tensor_dict[fields.InputDataFields.image] = tf.squeeze(
@@ -472,6 +508,7 @@ def create_train_input_fn(train_config, train_input_config,
           data_augmentation_fn=data_augmentation_fn,
           merge_multiple_boxes=train_config.merge_multiple_label_boxes,
           retain_original_image=train_config.retain_original_images,
+          use_multiclass_scores=train_config.use_multiclass_scores,
           use_bfloat16=train_config.use_bfloat16)
 
       tensor_dict = pad_input_data_to_static_shapes(

research/object_detection/inputs_test.py

@@ -105,6 +105,48 @@ class InputsTest(test_case.TestCase, parameterized.TestCase):
         tf.float32,
         labels[fields.InputDataFields.groundtruth_confidences].dtype)
 
+  def test_faster_rcnn_resnet50_train_input_with_additional_channels(self):
+    """Tests the training input function for FasterRcnnResnet50."""
+    configs = _get_configs_for_model('faster_rcnn_resnet50_pets')
+    model_config = configs['model']
+    configs['train_input_config'].num_additional_channels = 2
+    configs['train_config'].retain_original_images = True
+    model_config.faster_rcnn.num_classes = 37
+    train_input_fn = inputs.create_train_input_fn(
+        configs['train_config'], configs['train_input_config'], model_config)
+    features, labels = _make_initializable_iterator(train_input_fn()).get_next()
+
+    self.assertAllEqual([1, None, None, 5],
+                        features[fields.InputDataFields.image].shape.as_list())
+    self.assertAllEqual(
+        [1, None, None, 3],
+        features[fields.InputDataFields.original_image].shape.as_list())
+    self.assertEqual(tf.float32, features[fields.InputDataFields.image].dtype)
+    self.assertAllEqual([1],
+                        features[inputs.HASH_KEY].shape.as_list())
+    self.assertEqual(tf.int32, features[inputs.HASH_KEY].dtype)
+    self.assertAllEqual(
+        [1, 100, 4],
+        labels[fields.InputDataFields.groundtruth_boxes].shape.as_list())
+    self.assertEqual(tf.float32,
+                     labels[fields.InputDataFields.groundtruth_boxes].dtype)
+    self.assertAllEqual(
+        [1, 100, model_config.faster_rcnn.num_classes],
+        labels[fields.InputDataFields.groundtruth_classes].shape.as_list())
+    self.assertEqual(tf.float32,
+                     labels[fields.InputDataFields.groundtruth_classes].dtype)
+    self.assertAllEqual(
+        [1, 100],
+        labels[fields.InputDataFields.groundtruth_weights].shape.as_list())
+    self.assertEqual(tf.float32,
+                     labels[fields.InputDataFields.groundtruth_weights].dtype)
+    self.assertAllEqual(
+        [1, 100, model_config.faster_rcnn.num_classes],
+        labels[fields.InputDataFields.groundtruth_confidences].shape.as_list())
+    self.assertEqual(
+        tf.float32,
+        labels[fields.InputDataFields.groundtruth_confidences].dtype)
+
   @parameterized.parameters(
       {'eval_batch_size': 1},
       {'eval_batch_size': 8}
@@ -595,6 +637,72 @@ class DataTransformationFnTest(test_case.TestCase):
         transformed_inputs[fields.InputDataFields.groundtruth_confidences],
         [[0, 0, 1], [1, 0, 0]])
 
+  def test_returns_correct_labels_with_unrecognized_class(self):
+    tensor_dict = {
+        fields.InputDataFields.image:
+            tf.constant(np.random.rand(4, 4, 3).astype(np.float32)),
+        fields.InputDataFields.groundtruth_boxes:
+            tf.constant(
+                np.array([[0, 0, 1, 1], [.2, .2, 4, 4], [.5, .5, 1, 1]],
+                         np.float32)),
+        fields.InputDataFields.groundtruth_area:
+            tf.constant(np.array([.5, .4, .3])),
+        fields.InputDataFields.groundtruth_classes:
+            tf.constant(np.array([3, -1, 1], np.int32)),
+        fields.InputDataFields.groundtruth_keypoints:
+            tf.constant(
+                np.array([[[.1, .1]], [[.2, .2]], [[.5, .5]]],
+                         np.float32)),
+        fields.InputDataFields.groundtruth_keypoint_visibilities:
+            tf.constant([True, False, True]),
+        fields.InputDataFields.groundtruth_instance_masks:
+            tf.constant(np.random.rand(3, 4, 4).astype(np.float32)),
+        fields.InputDataFields.groundtruth_is_crowd:
+            tf.constant([False, True, False]),
+        fields.InputDataFields.groundtruth_difficult:
+            tf.constant(np.array([0, 0, 1], np.int32))
+    }
+
+    num_classes = 3
+    input_transformation_fn = functools.partial(
+        inputs.transform_input_data,
+        model_preprocess_fn=_fake_model_preprocessor_fn,
+        image_resizer_fn=_fake_image_resizer_fn,
+        num_classes=num_classes)
+    with self.test_session() as sess:
+      transformed_inputs = sess.run(
+          input_transformation_fn(tensor_dict=tensor_dict))
+
+    self.assertAllClose(
+        transformed_inputs[fields.InputDataFields.groundtruth_classes],
+        [[0, 0, 1], [1, 0, 0]])
+    self.assertAllEqual(
+        transformed_inputs[fields.InputDataFields.num_groundtruth_boxes], 2)
+    self.assertAllClose(
+        transformed_inputs[fields.InputDataFields.groundtruth_area], [.5, .3])
+    self.assertAllEqual(
+        transformed_inputs[fields.InputDataFields.groundtruth_confidences],
+        [[0, 0, 1], [1, 0, 0]])
+    self.assertAllClose(
+        transformed_inputs[fields.InputDataFields.groundtruth_boxes],
+        [[0, 0, 1, 1], [.5, .5, 1, 1]])
+    self.assertAllClose(
+        transformed_inputs[fields.InputDataFields.groundtruth_keypoints],
+        [[[.1, .1]], [[.5, .5]]])
+    self.assertAllEqual(
+        transformed_inputs[
+            fields.InputDataFields.groundtruth_keypoint_visibilities],
+        [True, True])
+    self.assertAllEqual(
+        transformed_inputs[
+            fields.InputDataFields.groundtruth_instance_masks].shape, [2, 4, 4])
+    self.assertAllEqual(
+        transformed_inputs[fields.InputDataFields.groundtruth_is_crowd],
+        [False, False])
+    self.assertAllEqual(
+        transformed_inputs[fields.InputDataFields.groundtruth_difficult],
+        [0, 1])
+
   def test_returns_correct_merged_boxes(self):
     tensor_dict = {
         fields.InputDataFields.image:
@@ -885,7 +993,7 @@ class PadInputDataToStaticShapesFnTest(test_case.TestCase):
   def test_images_and_additional_channels(self):
     input_tensor_dict = {
         fields.InputDataFields.image:
-            tf.placeholder(tf.float32, [None, None, 3]),
+            tf.placeholder(tf.float32, [None, None, 5]),
         fields.InputDataFields.image_additional_channels:
             tf.placeholder(tf.float32, [None, None, 2]),
     }
@@ -895,6 +1003,8 @@ class PadInputDataToStaticShapesFnTest(test_case.TestCase):
         num_classes=3,
         spatial_image_shape=[5, 6])
 
+    # pad_input_data_to_static_shape assumes that image is already concatenated
+    # with additional channels.
     self.assertAllEqual(
         padded_tensor_dict[fields.InputDataFields.image].shape.as_list(),
         [5, 6, 5])
@@ -902,6 +1012,22 @@ class PadInputDataToStaticShapesFnTest(test_case.TestCase):
         padded_tensor_dict[fields.InputDataFields.image_additional_channels]
         .shape.as_list(), [5, 6, 2])
 
+  def test_images_and_additional_channels_errors(self):
+    input_tensor_dict = {
+        fields.InputDataFields.image:
+            tf.placeholder(tf.float32, [None, None, 3]),
+        fields.InputDataFields.image_additional_channels:
+            tf.placeholder(tf.float32, [None, None, 2]),
+        fields.InputDataFields.original_image:
+            tf.placeholder(tf.float32, [None, None, 3]),
+    }
+    with self.assertRaises(ValueError):
+      _ = inputs.pad_input_data_to_static_shapes(
+          tensor_dict=input_tensor_dict,
+          max_num_boxes=3,
+          num_classes=3,
+          spatial_image_shape=[5, 6])
+
   def test_gray_images(self):
     input_tensor_dict = {
         fields.InputDataFields.image:
@@ -920,10 +1046,12 @@ class PadInputDataToStaticShapesFnTest(test_case.TestCase):
   def test_gray_images_and_additional_channels(self):
     input_tensor_dict = {
         fields.InputDataFields.image:
-            tf.placeholder(tf.float32, [None, None, 1]),
+            tf.placeholder(tf.float32, [None, None, 3]),
         fields.InputDataFields.image_additional_channels:
             tf.placeholder(tf.float32, [None, None, 2]),
     }
+    # pad_input_data_to_static_shape assumes that image is already concatenated
+    # with additional channels.
     padded_tensor_dict = inputs.pad_input_data_to_static_shapes(
         tensor_dict=input_tensor_dict,
         max_num_boxes=3,

research/object_detection/meta_architectures/faster_rcnn_meta_arch.py

@@ -92,8 +92,8 @@ configured in the meta architecture:
   non-max suppression and normalize them. In this case, the `postprocess` method
   skips both `_postprocess_rpn` and `_postprocess_box_classifier`.
 """
-from abc import abstractmethod
-from functools import partial
+import abc
+import functools
 import tensorflow as tf
 
 from object_detection.anchor_generators import grid_anchor_generator
@@ -138,7 +138,7 @@ class FasterRCNNFeatureExtractor(object):
     self._reuse_weights = reuse_weights
     self._weight_decay = weight_decay
 
-  @abstractmethod
+  @abc.abstractmethod
   def preprocess(self, resized_inputs):
     """Feature-extractor specific preprocessing (minus image resizing)."""
     pass
@@ -162,7 +162,7 @@ class FasterRCNNFeatureExtractor(object):
     with tf.variable_scope(scope, values=[preprocessed_inputs]):
       return self._extract_proposal_features(preprocessed_inputs, scope)
 
-  @abstractmethod
+  @abc.abstractmethod
   def _extract_proposal_features(self, preprocessed_inputs, scope):
     """Extracts first stage RPN features, to be overridden."""
     pass
@@ -185,7 +185,7 @@ class FasterRCNNFeatureExtractor(object):
         scope, values=[proposal_feature_maps], reuse=tf.AUTO_REUSE):
       return self._extract_box_classifier_features(proposal_feature_maps, scope)
 
-  @abstractmethod
+  @abc.abstractmethod
   def _extract_box_classifier_features(self, proposal_feature_maps, scope):
     """Extracts second stage box classifier features, to be overridden."""
     pass
@@ -770,7 +770,7 @@ class FasterRCNNMetaArch(model.DetectionModel):
           representing the features for each proposal.
     """
     image_shape_2d = self._image_batch_shape_2d(image_shape)
-    proposal_boxes_normalized, _, num_proposals = self._postprocess_rpn(
+    proposal_boxes_normalized, _, num_proposals, _, _ = self._postprocess_rpn(
         rpn_box_encodings, rpn_objectness_predictions_with_background,
         anchors, image_shape_2d, true_image_shapes)
 
@@ -1080,7 +1080,7 @@ class FasterRCNNMetaArch(model.DetectionModel):
         anchors_boxlist, clip_window)
     def _batch_gather_kept_indices(predictions_tensor):
       return shape_utils.static_or_dynamic_map_fn(
-          partial(tf.gather, indices=keep_indices),
+          functools.partial(tf.gather, indices=keep_indices),
           elems=predictions_tensor,
           dtype=tf.float32,
           parallel_iterations=self._parallel_iterations,
@@ -1148,17 +1148,22 @@ class FasterRCNNMetaArch(model.DetectionModel):
 
     with tf.name_scope('FirstStagePostprocessor'):
       if self._number_of_stages == 1:
-        proposal_boxes, proposal_scores, num_proposals = self._postprocess_rpn(
-            prediction_dict['rpn_box_encodings'],
-            prediction_dict['rpn_objectness_predictions_with_background'],
-            prediction_dict['anchors'],
-            true_image_shapes,
-            true_image_shapes)
+        (proposal_boxes, proposal_scores, num_proposals, raw_proposal_boxes,
+         raw_proposal_scores) = self._postprocess_rpn(
+             prediction_dict['rpn_box_encodings'],
+             prediction_dict['rpn_objectness_predictions_with_background'],
+             prediction_dict['anchors'], true_image_shapes, true_image_shapes)
         return {
-            fields.DetectionResultFields.detection_boxes: proposal_boxes,
-            fields.DetectionResultFields.detection_scores: proposal_scores,
+            fields.DetectionResultFields.detection_boxes:
+                proposal_boxes,
+            fields.DetectionResultFields.detection_scores:
+                proposal_scores,
             fields.DetectionResultFields.num_detections:
                 tf.to_float(num_proposals),
+            fields.DetectionResultFields.raw_detection_boxes:
+                raw_proposal_boxes,
+            fields.DetectionResultFields.raw_detection_scores:
+                raw_proposal_scores
         }
 
     # TODO(jrru): Remove mask_predictions from _post_process_box_classifier.
@@ -1266,6 +1271,11 @@ class FasterRCNNMetaArch(model.DetectionModel):
       num_proposals: A Tensor of type `int32`. A 1-D tensor of shape [batch]
         representing the number of proposals predicted for each image in
         the batch.
+      raw_detection_boxes: [batch, total_detections, 4] tensor with decoded
+        proposal boxes before Non-Max Suppression.
+      raw_detection_score: [batch, total_detections,
+        num_classes_with_background] tensor of class score logits for
+        raw proposal boxes.
     """
     rpn_box_encodings_batch = tf.expand_dims(rpn_box_encodings_batch, axis=2)
     rpn_encodings_shape = shape_utils.combined_static_and_dynamic_shape(
@@ -1274,13 +1284,13 @@ class FasterRCNNMetaArch(model.DetectionModel):
         tf.expand_dims(anchors, 0), [rpn_encodings_shape[0], 1, 1])
     proposal_boxes = self._batch_decode_boxes(rpn_box_encodings_batch,
                                               tiled_anchor_boxes)
-    proposal_boxes = tf.squeeze(proposal_boxes, axis=2)
+    raw_proposal_boxes = tf.squeeze(proposal_boxes, axis=2)
     rpn_objectness_softmax_without_background = tf.nn.softmax(
         rpn_objectness_predictions_with_background_batch)[:, :, 1]
     clip_window = self._compute_clip_window(image_shapes)
     (proposal_boxes, proposal_scores, _, _, _,
      num_proposals) = self._first_stage_nms_fn(
-         tf.expand_dims(proposal_boxes, axis=2),
+         tf.expand_dims(raw_proposal_boxes, axis=2),
          tf.expand_dims(rpn_objectness_softmax_without_background, axis=2),
          clip_window=clip_window)
     if self._is_training:
@@ -1304,7 +1314,13 @@ class FasterRCNNMetaArch(model.DetectionModel):
       return normalized_boxes_per_image
     normalized_proposal_boxes = shape_utils.static_or_dynamic_map_fn(
         normalize_boxes, elems=[proposal_boxes, image_shapes], dtype=tf.float32)
-    return normalized_proposal_boxes, proposal_scores, num_proposals
+    raw_normalized_proposal_boxes = shape_utils.static_or_dynamic_map_fn(
+        normalize_boxes,
+        elems=[raw_proposal_boxes, image_shapes],
+        dtype=tf.float32)
+    return (normalized_proposal_boxes, proposal_scores, num_proposals,
+            raw_normalized_proposal_boxes,
+            rpn_objectness_predictions_with_background_batch)
 
   def _sample_box_classifier_batch(
       self,
@@ -1576,6 +1592,11 @@ class FasterRCNNMetaArch(model.DetectionModel):
           (optional) [batch, max_detections, mask_height, mask_width]. Note
           that a pixel-wise sigmoid score converter is applied to the detection
           masks.
+        `raw_detection_boxes`: [batch, total_detections, 4] tensor with decoded
+          detection boxes before Non-Max Suppression.
+        `raw_detection_score`: [batch, total_detections,
+          num_classes_with_background] tensor of multi-class score logits for
+          raw detection boxes.
     """
     refined_box_encodings_batch = tf.reshape(
         refined_box_encodings,
@@ -1589,11 +1610,11 @@ class FasterRCNNMetaArch(model.DetectionModel):
     )
     refined_decoded_boxes_batch = self._batch_decode_boxes(
         refined_box_encodings_batch, proposal_boxes)
-    class_predictions_with_background_batch = (
+    class_predictions_with_background_batch_normalized = (
         self._second_stage_score_conversion_fn(
             class_predictions_with_background_batch))
     class_predictions_batch = tf.reshape(
-        tf.slice(class_predictions_with_background_batch,
+        tf.slice(class_predictions_with_background_batch_normalized,
                  [0, 0, 1], [-1, -1, -1]),
         [-1, self.max_num_proposals, self.num_classes])
     clip_window = self._compute_clip_window(image_shapes)
@@ -1614,11 +1635,51 @@ class FasterRCNNMetaArch(model.DetectionModel):
          change_coordinate_frame=True,
          num_valid_boxes=num_proposals,
          masks=mask_predictions_batch)
+    if refined_decoded_boxes_batch.shape[2] > 1:
+      class_ids = tf.expand_dims(
+          tf.argmax(class_predictions_with_background_batch[:, :, 1:], axis=2,
+                    output_type=tf.int32),
+          axis=-1)
+      raw_detection_boxes = tf.squeeze(
+          tf.batch_gather(refined_decoded_boxes_batch, class_ids), axis=2)
+    else:
+      raw_detection_boxes = tf.squeeze(refined_decoded_boxes_batch, axis=2)
+
+    def normalize_and_clip_boxes(args):
+      """Normalize and clip boxes."""
+      boxes_per_image = args[0]
+      image_shape = args[1]
+      normalized_boxes_per_image = box_list_ops.to_normalized_coordinates(
+          box_list.BoxList(boxes_per_image),
+          image_shape[0],
+          image_shape[1],
+          check_range=False).get()
+
+      normalized_boxes_per_image = box_list_ops.clip_to_window(
+          box_list.BoxList(normalized_boxes_per_image),
+          tf.constant([0.0, 0.0, 1.0, 1.0], tf.float32),
+          filter_nonoverlapping=False).get()
+
+      return normalized_boxes_per_image
+
+    raw_normalized_detection_boxes = shape_utils.static_or_dynamic_map_fn(
+        normalize_and_clip_boxes,
+        elems=[raw_detection_boxes, image_shapes],
+        dtype=tf.float32)
+
     detections = {
-        fields.DetectionResultFields.detection_boxes: nmsed_boxes,
-        fields.DetectionResultFields.detection_scores: nmsed_scores,
-        fields.DetectionResultFields.detection_classes: nmsed_classes,
-        fields.DetectionResultFields.num_detections: tf.to_float(num_detections)
+        fields.DetectionResultFields.detection_boxes:
+            nmsed_boxes,
+        fields.DetectionResultFields.detection_scores:
+            nmsed_scores,
+        fields.DetectionResultFields.detection_classes:
+            nmsed_classes,
+        fields.DetectionResultFields.num_detections:
+            tf.to_float(num_detections),
+        fields.DetectionResultFields.raw_detection_boxes:
+            raw_normalized_detection_boxes,
+        fields.DetectionResultFields.raw_detection_scores:
+            class_predictions_with_background_batch
     }
     if nmsed_masks is not None:
       detections[fields.DetectionResultFields.detection_masks] = nmsed_masks
@@ -1769,7 +1830,8 @@ class FasterRCNNMetaArch(model.DetectionModel):
           back_prop=True))
 
       # Normalize by number of examples in sampled minibatch
-      normalizer = tf.reduce_sum(batch_sampled_indices, axis=1)
+      normalizer = tf.maximum(
+          tf.reduce_sum(batch_sampled_indices, axis=1), 1.0)
       batch_one_hot_targets = tf.one_hot(
           tf.to_int32(batch_cls_targets), depth=2)
       sampled_reg_indices = tf.multiply(batch_sampled_indices,

research/object_detection/meta_architectures/faster_rcnn_meta_arch_test.py

@@ -85,6 +85,68 @@ class FasterRCNNMetaArchTest(
       self.assertTrue(np.amax(detections_out['detection_masks'] <= 1.0))
       self.assertTrue(np.amin(detections_out['detection_masks'] >= 0.0))
 
+  def test_postprocess_second_stage_only_inference_mode_with_calibration(self):
+    model = self._build_model(
+        is_training=False, number_of_stages=2, second_stage_batch_size=6,
+        calibration_mapping_value=0.5)
+
+    batch_size = 2
+    total_num_padded_proposals = batch_size * model.max_num_proposals
+    proposal_boxes = tf.constant(
+        [[[1, 1, 2, 3],
+          [0, 0, 1, 1],
+          [.5, .5, .6, .6],
+          4*[0], 4*[0], 4*[0], 4*[0], 4*[0]],
+         [[2, 3, 6, 8],
+          [1, 2, 5, 3],
+          4*[0], 4*[0], 4*[0], 4*[0], 4*[0], 4*[0]]], dtype=tf.float32)
+    num_proposals = tf.constant([3, 2], dtype=tf.int32)
+    refined_box_encodings = tf.zeros(
+        [total_num_padded_proposals, model.num_classes, 4], dtype=tf.float32)
+    class_predictions_with_background = tf.ones(
+        [total_num_padded_proposals, model.num_classes+1], dtype=tf.float32)
+    image_shape = tf.constant([batch_size, 36, 48, 3], dtype=tf.int32)
+
+    mask_height = 2
+    mask_width = 2
+    mask_predictions = 30. * tf.ones(
+        [total_num_padded_proposals, model.num_classes,
+         mask_height, mask_width], dtype=tf.float32)
+    exp_detection_masks = np.array([[[[1, 1], [1, 1]],
+                                     [[1, 1], [1, 1]],
+                                     [[1, 1], [1, 1]],
+                                     [[1, 1], [1, 1]],
+                                     [[1, 1], [1, 1]]],
+                                    [[[1, 1], [1, 1]],
+                                     [[1, 1], [1, 1]],
+                                     [[1, 1], [1, 1]],
+                                     [[1, 1], [1, 1]],
+                                     [[0, 0], [0, 0]]]])
+
+    _, true_image_shapes = model.preprocess(tf.zeros(image_shape))
+    detections = model.postprocess({
+        'refined_box_encodings': refined_box_encodings,
+        'class_predictions_with_background': class_predictions_with_background,
+        'num_proposals': num_proposals,
+        'proposal_boxes': proposal_boxes,
+        'image_shape': image_shape,
+        'mask_predictions': mask_predictions
+    }, true_image_shapes)
+    with self.test_session() as sess:
+      detections_out = sess.run(detections)
+      self.assertAllEqual(detections_out['detection_boxes'].shape, [2, 5, 4])
+      # All scores map to 0.5, except for the final one, which is pruned.
+      self.assertAllClose(detections_out['detection_scores'],
+                          [[0.5, 0.5, 0.5, 0.5, 0.5],
+                           [0.5, 0.5, 0.5, 0.5, 0.0]])
+      self.assertAllClose(detections_out['detection_classes'],
+                          [[0, 0, 0, 1, 1], [0, 0, 1, 1, 0]])
+      self.assertAllClose(detections_out['num_detections'], [5, 4])
+      self.assertAllClose(detections_out['detection_masks'],
+                          exp_detection_masks)
+      self.assertTrue(np.amax(detections_out['detection_masks'] <= 1.0))
+      self.assertTrue(np.amin(detections_out['detection_masks'] >= 0.0))
+
   def test_postprocess_second_stage_only_inference_mode_with_shared_boxes(self):
     model = self._build_model(
         is_training=False, number_of_stages=2, second_stage_batch_size=6)
@@ -190,6 +252,7 @@ class FasterRCNNMetaArchTest(
               set([
                   'detection_boxes', 'detection_scores', 'detection_classes',
                   'detection_masks', 'num_detections', 'mask_predictions',
+                  'raw_detection_boxes', 'raw_detection_scores'
               ])))
       for key in expected_shapes:
         self.assertAllEqual(tensor_dict_out[key].shape, expected_shapes[key])
@@ -276,7 +339,7 @@ class FasterRCNNMetaArchTest(
           self.assertAllEqual(tensor_dict_out[key].shape, expected_shapes[key])
 
         anchors_shape_out = tensor_dict_out['anchors'].shape
-        self.assertEqual(2, len(anchors_shape_out))
+        self.assertLen(anchors_shape_out, 2)
         self.assertEqual(4, anchors_shape_out[1])
         num_anchors_out = anchors_shape_out[0]
         self.assertAllEqual(tensor_dict_out['rpn_box_encodings'].shape,

research/object_detection/meta_architectures/faster_rcnn_meta_arch_test_lib.py

@@ -165,7 +165,8 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
                    use_matmul_crop_and_resize=False,
                    clip_anchors_to_image=False,
                    use_matmul_gather_in_matcher=False,
-                   use_static_shapes=False):
+                   use_static_shapes=False,
+                   calibration_mapping_value=None):
 
     def image_resizer_fn(image, masks=None):
       """Fake image resizer function."""
@@ -244,7 +245,9 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
     first_stage_localization_loss_weight = 1.0
     first_stage_objectness_loss_weight = 1.0
 
+    post_processing_config = post_processing_pb2.PostProcessing()
     post_processing_text_proto = """
+      score_converter: IDENTITY
       batch_non_max_suppression {
         score_threshold: -20.0
         iou_threshold: 1.0
@@ -253,18 +256,31 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
         use_static_shapes: """ +'{}'.format(use_static_shapes) + """
       }
     """
-    post_processing_config = post_processing_pb2.PostProcessing()
+    if calibration_mapping_value:
+      calibration_text_proto = """
+      calibration_config {
+        function_approximation {
+          x_y_pairs {
+            x_y_pair {
+              x: 0.0
+              y: %f
+            }
+            x_y_pair {
+              x: 1.0
+              y: %f
+              }}}}""" % (calibration_mapping_value, calibration_mapping_value)
+      post_processing_text_proto = (post_processing_text_proto
+                                    + ' ' + calibration_text_proto)
     text_format.Merge(post_processing_text_proto, post_processing_config)
+    second_stage_non_max_suppression_fn, second_stage_score_conversion_fn = (
+        post_processing_builder.build(post_processing_config))
 
     second_stage_target_assigner = target_assigner.create_target_assigner(
         'FasterRCNN', 'detection',
         use_matmul_gather=use_matmul_gather_in_matcher)
-    second_stage_non_max_suppression_fn, _ = post_processing_builder.build(
-        post_processing_config)
     second_stage_sampler = sampler.BalancedPositiveNegativeSampler(
         positive_fraction=1.0, is_static=use_static_shapes)
 
-    second_stage_score_conversion_fn = tf.identity
     second_stage_localization_loss_weight = 1.0
     second_stage_classification_loss_weight = 1.0
     if softmax_second_stage_classification_loss:
@@ -336,6 +352,10 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
             predict_masks=predict_masks,
             masks_are_class_agnostic=masks_are_class_agnostic), **common_kwargs)
 
+  @parameterized.parameters(
+      {'use_static_shapes': False},
+      {'use_static_shapes': True}
+  )
   def test_predict_gives_correct_shapes_in_inference_mode_first_stage_only(
       self, use_static_shapes=False):
     batch_size = 2
@@ -457,6 +477,10 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
             prediction_out['rpn_objectness_predictions_with_background'].shape,
             (batch_size, num_anchors_out, 2))
 
+  @parameterized.parameters(
+      {'use_static_shapes': False},
+      {'use_static_shapes': True}
+  )
   def test_predict_correct_shapes_in_inference_mode_two_stages(
       self, use_static_shapes=False):
 
@@ -578,6 +602,10 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
         for key in expected_shapes:
           self.assertAllEqual(tensor_dict_out[key].shape, expected_shapes[key])
 
+  @parameterized.parameters(
+      {'use_static_shapes': False},
+      {'use_static_shapes': True}
+  )
   def test_predict_gives_correct_shapes_in_train_mode_both_stages(
       self,
       use_static_shapes=False):
@@ -670,6 +698,12 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
     self.assertAllEqual(results[8].shape,
                         expected_shapes['rpn_box_predictor_features'])
 
+  @parameterized.parameters(
+      {'use_static_shapes': False, 'pad_to_max_dimension': None},
+      {'use_static_shapes': True, 'pad_to_max_dimension': None},
+      {'use_static_shapes': False, 'pad_to_max_dimension': 56},
+      {'use_static_shapes': True, 'pad_to_max_dimension': 56}
+  )
   def test_postprocess_first_stage_only_inference_mode(
       self, use_static_shapes=False, pad_to_max_dimension=None):
     batch_size = 2
@@ -696,9 +730,9 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
           rpn_objectness_predictions_with_background,
           'rpn_features_to_crop': rpn_features_to_crop,
           'anchors': anchors}, true_image_shapes)
-      return (proposals['num_detections'],
-              proposals['detection_boxes'],
-              proposals['detection_scores'])
+      return (proposals['num_detections'], proposals['detection_boxes'],
+              proposals['detection_scores'], proposals['raw_detection_boxes'],
+              proposals['raw_detection_scores'])
 
     anchors = np.array(
         [[0, 0, 16, 16],
@@ -741,6 +775,12 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
     expected_proposal_scores = [[1, 1, 0, 0, 0, 0, 0, 0],
                                 [1, 1, 0, 0, 0, 0, 0, 0]]
     expected_num_proposals = [4, 4]
+    expected_raw_proposal_boxes = [[[0., 0., 0.5, 0.5], [0., 0.5, 0.5, 1.],
+                                    [0.5, 0., 1., 0.5], [0.5, 0.5, 1., 1.]],
+                                   [[0., 0., 0.5, 0.5], [0., 0.5, 0.5, 1.],
+                                    [0.5, 0., 1., 0.5], [0.5, 0.5, 1., 1.]]]
+    expected_raw_scores = [[[-10., 13.], [10., -10.], [10., -11.], [-10., 12.]],
+                           [[10., -10.], [-10., 13.], [-10., 12.], [10., -11.]]]
 
     self.assertAllClose(results[0], expected_num_proposals)
     for indx, num_proposals in enumerate(expected_num_proposals):
@@ -748,6 +788,8 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
                           expected_proposal_boxes[indx][0:num_proposals])
       self.assertAllClose(results[2][indx][0:num_proposals],
                           expected_proposal_scores[indx][0:num_proposals])
+    self.assertAllClose(results[3], expected_raw_proposal_boxes)
+    self.assertAllClose(results[4], expected_raw_scores)
 
   def _test_postprocess_first_stage_only_train_mode(self,
                                                     pad_to_max_dimension=None):
@@ -801,9 +843,17 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
     expected_proposal_scores = [[1, 1],
                                 [1, 1]]
     expected_num_proposals = [2, 2]
-
-    expected_output_keys = set(['detection_boxes', 'detection_scores',
-                                'num_detections'])
+    expected_raw_proposal_boxes = [[[0., 0., 0.5, 0.5], [0., 0.5, 0.5, 1.],
+                                    [0.5, 0., 1., 0.5], [0.5, 0.5, 1., 1.]],
+                                   [[0., 0., 0.5, 0.5], [0., 0.5, 0.5, 1.],
+                                    [0.5, 0., 1., 0.5], [0.5, 0.5, 1., 1.]]]
+    expected_raw_scores = [[[-10., 13.], [-10., 12.], [-10., 11.], [-10., 10.]],
+                           [[-10., 13.], [-10., 12.], [-10., 11.], [-10., 10.]]]
+
+    expected_output_keys = set([
+        'detection_boxes', 'detection_scores', 'num_detections',
+        'raw_detection_boxes', 'raw_detection_scores'
+    ])
     self.assertEqual(set(proposals.keys()), expected_output_keys)
 
     with self.test_session() as sess:
@@ -817,6 +867,10 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
                           expected_proposal_scores)
       self.assertAllEqual(proposals_out['num_detections'],
                           expected_num_proposals)
+    self.assertAllClose(proposals_out['raw_detection_boxes'],
+                        expected_raw_proposal_boxes)
+    self.assertAllClose(proposals_out['raw_detection_scores'],
+                        expected_raw_scores)
 
   def test_postprocess_first_stage_only_train_mode(self):
     self._test_postprocess_first_stage_only_train_mode()
@@ -824,6 +878,12 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
   def test_postprocess_first_stage_only_train_mode_padded_image(self):
     self._test_postprocess_first_stage_only_train_mode(pad_to_max_dimension=56)
 
+  @parameterized.parameters(
+      {'use_static_shapes': False, 'pad_to_max_dimension': None},
+      {'use_static_shapes': True, 'pad_to_max_dimension': None},
+      {'use_static_shapes': False, 'pad_to_max_dimension': 56},
+      {'use_static_shapes': True, 'pad_to_max_dimension': 56}
+  )
   def test_postprocess_second_stage_only_inference_mode(
       self, use_static_shapes=False, pad_to_max_dimension=None):
     batch_size = 2
@@ -854,10 +914,10 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
           'num_proposals': num_proposals,
           'proposal_boxes': proposal_boxes,
       }, true_image_shapes)
-      return (detections['num_detections'],
-              detections['detection_boxes'],
-              detections['detection_scores'],
-              detections['detection_classes'])
+      return (detections['num_detections'], detections['detection_boxes'],
+              detections['detection_scores'], detections['detection_classes'],
+              detections['raw_detection_boxes'],
+              detections['raw_detection_scores'])
 
     proposal_boxes = np.array(
         [[[1, 1, 2, 3],
@@ -867,6 +927,7 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
          [[2, 3, 6, 8],
           [1, 2, 5, 3],
           4*[0], 4*[0], 4*[0], 4*[0], 4*[0], 4*[0]]], dtype=np.float32)
+
     num_proposals = np.array([3, 2], dtype=np.int32)
     refined_box_encodings = np.zeros(
         [total_num_padded_proposals, num_classes, 4], dtype=np.float32)
@@ -887,6 +948,15 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
     expected_num_detections = [5, 4]
     expected_detection_classes = [[0, 0, 0, 1, 1], [0, 0, 1, 1, 0]]
     expected_detection_scores = [[1, 1, 1, 1, 1], [1, 1, 1, 1, 0]]
+    h = float(image_shape[1])
+    w = float(image_shape[2])
+    expected_raw_detection_boxes = np.array(
+        [[[1 / h, 1 / w, 2 / h, 3 / w], [0, 0, 1 / h, 1 / w],
+          [.5 / h, .5 / w, .6 / h, .6 / w], 4 * [0], 4 * [0], 4 * [0], 4 * [0],
+          4 * [0]],
+         [[2 / h, 3 / w, 6 / h, 8 / w], [1 / h, 2 / w, 5 / h, 3 / w], 4 * [0],
+          4 * [0], 4 * [0], 4 * [0], 4 * [0], 4 * [0]]],
+        dtype=np.float32)
 
     self.assertAllClose(results[0], expected_num_detections)
 
@@ -896,6 +966,9 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
       self.assertAllClose(results[3][indx][0:num_proposals],
                           expected_detection_classes[indx][0:num_proposals])
 
+    self.assertAllClose(results[4], expected_raw_detection_boxes)
+    self.assertAllClose(results[5],
+                        class_predictions_with_background.reshape([-1, 8, 3]))
     if not use_static_shapes:
       self.assertAllEqual(results[1].shape, [2, 5, 4])
 
@@ -1268,6 +1341,13 @@ class FasterRCNNMetaArchTestBase(test_case.TestCase, parameterized.TestCase):
           'Loss/BoxClassifierLoss/classification_loss'], 0)
       self.assertAllClose(loss_dict_out['Loss/BoxClassifierLoss/mask_loss'], 0)
 
+  @parameterized.parameters(
+      {'use_static_shapes': False, 'shared_boxes': False},
+      {'use_static_shapes': False, 'shared_boxes': True},
+
+      {'use_static_shapes': True, 'shared_boxes': False},
+      {'use_static_shapes': True, 'shared_boxes': True},
+  )
   def test_loss_full_zero_padded_proposals_nonzero_loss_with_two_images(
       self, use_static_shapes=False, shared_boxes=False):
     batch_size = 2

research/object_detection/meta_architectures/rfcn_meta_arch.py

@@ -288,7 +288,7 @@ class RFCNMetaArch(faster_rcnn_meta_arch.FasterRCNNMetaArch):
     """
     image_shape_2d = tf.tile(tf.expand_dims(image_shape[1:], 0),
                              [image_shape[0], 1])
-    proposal_boxes_normalized, _, num_proposals = self._postprocess_rpn(
+    proposal_boxes_normalized, _, num_proposals, _, _ = self._postprocess_rpn(
         rpn_box_encodings, rpn_objectness_predictions_with_background,
         anchors, image_shape_2d, true_image_shapes)
 

research/object_detection/meta_architectures/ssd_meta_arch.py

@@ -17,8 +17,7 @@
 General tensorflow implementation of convolutional Multibox/SSD detection
 models.
 """
-from abc import abstractmethod
-
+import abc
 import tensorflow as tf
 
 from object_detection.core import box_list
@@ -80,7 +79,7 @@ class SSDFeatureExtractor(object):
   def is_keras_model(self):
     return False
 
-  @abstractmethod
+  @abc.abstractmethod
   def preprocess(self, resized_inputs):
     """Preprocesses images for feature extraction (minus image resizing).
 
@@ -98,7 +97,7 @@ class SSDFeatureExtractor(object):
     """
     pass
 
-  @abstractmethod
+  @abc.abstractmethod
   def extract_features(self, preprocessed_inputs):
     """Extracts features from preprocessed inputs.
 
@@ -196,7 +195,7 @@ class SSDKerasFeatureExtractor(tf.keras.Model):
   def is_keras_model(self):
     return True
 
-  @abstractmethod
+  @abc.abstractmethod
   def preprocess(self, resized_inputs):
     """Preprocesses images for feature extraction (minus image resizing).
 
@@ -214,7 +213,7 @@ class SSDKerasFeatureExtractor(tf.keras.Model):
     """
     raise NotImplementedError
 
-  @abstractmethod
+  @abc.abstractmethod
   def _extract_features(self, preprocessed_inputs):
     """Extracts features from preprocessed inputs.
 
@@ -552,8 +551,10 @@ class SSDMetaArch(model.DetectionModel):
         5) anchors: 2-D float tensor of shape [num_anchors, 4] containing
           the generated anchors in normalized coordinates.
     """
-    batchnorm_updates_collections = (None if self._inplace_batchnorm_update
-                                     else tf.GraphKeys.UPDATE_OPS)
+    if self._inplace_batchnorm_update:
+      batchnorm_updates_collections = None
+    else:
+      batchnorm_updates_collections = tf.GraphKeys.UPDATE_OPS
     if self._feature_extractor.is_keras_model:
       feature_maps = self._feature_extractor(preprocessed_inputs)
     else:
@@ -648,14 +649,22 @@ class SSDMetaArch(model.DetectionModel):
 
     Returns:
       detections: a dictionary containing the following fields
-        detection_boxes: [batch, max_detections, 4]
-        detection_scores: [batch, max_detections]
-        detection_classes: [batch, max_detections]
+        detection_boxes: [batch, max_detections, 4] tensor with post-processed
+          detection boxes.
+        detection_scores: [batch, max_detections] tensor with scalar scores for
+          post-processed detection boxes.
+        detection_classes: [batch, max_detections] tensor with classes for
+          post-processed detection classes.
         detection_keypoints: [batch, max_detections, num_keypoints, 2] (if
           encoded in the prediction_dict 'box_encodings')
         detection_masks: [batch_size, max_detections, mask_height, mask_width]
           (optional)
         num_detections: [batch]
+        raw_detection_boxes: [batch, total_detections, 4] tensor with decoded
+          detection boxes before Non-Max Suppression.
+        raw_detection_score: [batch, total_detections,
+          num_classes_with_background] tensor of multi-class score logits for
+          raw detection boxes.
     Raises:
       ValueError: if prediction_dict does not contain `box_encodings` or
         `class_predictions_with_background` fields.
@@ -700,11 +709,18 @@ class SSDMetaArch(model.DetectionModel):
            additional_fields=additional_fields,
            masks=prediction_dict.get('mask_predictions'))
       detection_dict = {
-          fields.DetectionResultFields.detection_boxes: nmsed_boxes,
-          fields.DetectionResultFields.detection_scores: nmsed_scores,
-          fields.DetectionResultFields.detection_classes: nmsed_classes,
+          fields.DetectionResultFields.detection_boxes:
+              nmsed_boxes,
+          fields.DetectionResultFields.detection_scores:
+              nmsed_scores,
+          fields.DetectionResultFields.detection_classes:
+              nmsed_classes,
           fields.DetectionResultFields.num_detections:
-              tf.to_float(num_detections)
+              tf.to_float(num_detections),
+          fields.DetectionResultFields.raw_detection_boxes:
+              tf.squeeze(detection_boxes, axis=2),
+          fields.DetectionResultFields.raw_detection_scores:
+              class_predictions
       }
       if (nmsed_additional_fields is not None and
           fields.BoxListFields.keypoints in nmsed_additional_fields):
@@ -1049,9 +1065,9 @@ class SSDMetaArch(model.DetectionModel):
       mined_cls_loss: a float scalar with sum of classification losses from
         selected hard examples.
     """
-    class_predictions = tf.slice(
-        prediction_dict['class_predictions_with_background'], [0, 0,
-                                                               1], [-1, -1, -1])
+    class_predictions = prediction_dict['class_predictions_with_background']
+    if self._add_background_class:
+      class_predictions = tf.slice(class_predictions, [0, 0, 1], [-1, -1, -1])
 
     decoded_boxes, _ = self._batch_decode(prediction_dict['box_encodings'])
     decoded_box_tensors_list = tf.unstack(decoded_boxes)

research/object_detection/meta_architectures/ssd_meta_arch_test_lib.py

@@ -16,7 +16,9 @@
 
 import functools
 import tensorflow as tf
+from google.protobuf import text_format
 
+from object_detection.builders import post_processing_builder
 from object_detection.core import anchor_generator
 from object_detection.core import balanced_positive_negative_sampler as sampler
 from object_detection.core import box_list
@@ -25,6 +27,7 @@ from object_detection.core import post_processing
 from object_detection.core import region_similarity_calculator as sim_calc
 from object_detection.core import target_assigner
 from object_detection.meta_architectures import ssd_meta_arch
+from object_detection.protos import calibration_pb2
 from object_detection.protos import model_pb2
 from object_detection.utils import ops
 from object_detection.utils import test_case
@@ -125,7 +128,8 @@ class SSDMetaArchTestBase(test_case.TestCase):
       use_keras=False,
       predict_mask=False,
       use_static_shapes=False,
-      nms_max_size_per_class=5):
+      nms_max_size_per_class=5,
+      calibration_mapping_value=None):
     is_training = False
     num_classes = 1
     mock_anchor_generator = MockAnchorGenerator2x2()
@@ -156,6 +160,24 @@ class SSDMetaArchTestBase(test_case.TestCase):
         max_size_per_class=nms_max_size_per_class,
         max_total_size=nms_max_size_per_class,
         use_static_shapes=use_static_shapes)
+    score_conversion_fn = tf.identity
+    calibration_config = calibration_pb2.CalibrationConfig()
+    if calibration_mapping_value:
+      calibration_text_proto = """
+      function_approximation {
+        x_y_pairs {
+            x_y_pair {
+              x: 0.0
+              y: %f
+            }
+            x_y_pair {
+              x: 1.0
+              y: %f
+            }}}""" % (calibration_mapping_value, calibration_mapping_value)
+      text_format.Merge(calibration_text_proto, calibration_config)
+      score_conversion_fn = (
+          post_processing_builder._build_calibrated_score_converter(  # pylint: disable=protected-access
+              tf.identity, calibration_config))
     classification_loss_weight = 1.0
     localization_loss_weight = 1.0
     negative_class_weight = 1.0
@@ -201,7 +223,7 @@ class SSDMetaArchTestBase(test_case.TestCase):
         encode_background_as_zeros=encode_background_as_zeros,
         image_resizer_fn=image_resizer_fn,
         non_max_suppression_fn=non_max_suppression_fn,
-        score_conversion_fn=tf.identity,
+        score_conversion_fn=score_conversion_fn,
         classification_loss=classification_loss,
         localization_loss=localization_loss,
         classification_loss_weight=classification_loss_weight,