diff --git a/.github/ISSUE_TEMPLATE/caffe2paddle.md b/.github/ISSUE_TEMPLATE/caffe2paddle.md new file mode 100644 index 0000000000000000000000000000000000000000..d650fcfbd85663600f57c2055211fa9c2d4b20d4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/caffe2paddle.md @@ -0,0 +1,8 @@ +--- +name: caffe2paddle +about: Caffe模型转换至PaddlePaddle，请说明Caffe模型的来源，模型类型（例如图像分类、目标检测等） +--- + +Caffe模型转换至PaddlePaddle，请说明Caffe模型的来源，模型类型（例如图像分类、目标检测等） + +如有原模型文件或github链接，如方便可一并附上，方便开发人员分析。 diff --git a/.github/ISSUE_TEMPLATE/onnx2paddle.md b/.github/ISSUE_TEMPLATE/onnx2paddle.md new file mode 100644 index 0000000000000000000000000000000000000000..de8a7aebce34994f1269adad7d326a71851e9700 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/onnx2paddle.md @@ -0,0 +1,7 @@ +--- +name: onnx2paddle +about: ONNX模型转换至PaddlePaddle，请说明ONNX模型的来源，模型类型（例如图像分类、目标检测等) +--- +ONNX模型转换至PaddlePaddle，请说明ONNX模型的来源，模型类型（例如图像分类、目标检测等） + +如有原模型文件或github链接，如方便可一并附上，方便开发人员分析。 diff --git a/.github/ISSUE_TEMPLATE/other.md b/.github/ISSUE_TEMPLATE/other.md new file mode 100644 index 0000000000000000000000000000000000000000..7202cce897890f3d4a8e970cc4e379ba5db02821 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/other.md @@ -0,0 +1,4 @@ +--- +name: 其它类型 +about: 例如Bug类，需求建议类 +--- diff --git a/.github/ISSUE_TEMPLATE/paddle2onnx.md b/.github/ISSUE_TEMPLATE/paddle2onnx.md new file mode 100644 index 0000000000000000000000000000000000000000..4f589a66df1a653995d17d0d6e422dff3764bcd7 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/paddle2onnx.md @@ -0,0 +1,7 @@ +--- +name: paddle2onnx +about: Paddle模型转换至ONNX，请说明Paddle模型的来源，模型类型（例如图像分类、目标检测等） +--- +Paddle模型转换至ONNX，请说明Paddle模型的来源，模型类型（例如图像分类、目标检测等） + +如有原模型文件或github链接，如方便可一并附上，方便开发人员分析，同时建议说明模型转换的使用场景:) diff --git a/.github/ISSUE_TEMPLATE/tensorflow2paddle.md b/.github/ISSUE_TEMPLATE/tensorflow2paddle.md new file mode 100644 index 0000000000000000000000000000000000000000..88af18bf67a737206b6757ec55f1d581d62460b3 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/tensorflow2paddle.md @@ -0,0 +1,8 @@ +--- +name: tensorflow2paddle +about: TensorFlow模型转换至PaddlePaddle，请说明TensorFlow模型的来源，模型类型（例如图像分类、目标检测等） +--- + +TensorFlow模型转换至PaddlePaddle，请说明TensorFlow模型的来源，模型类型（例如图像分类、目标检测等） + +如有原模型文件或github链接，可以一并附上，方便开发人员分析。 diff --git a/FAQ.md b/FAQ.md index 695c160cb61d0095125a844de16e5fac84c0b16d..a66ac9b0f6c06652f96af7d417effedbf2261730 100644 --- a/FAQ.md +++ b/FAQ.md @@ -11,3 +11,6 @@ x2paddle -f tensorflow -m tf.pb -s pd-model --without_data_format_optimization - ``` > 1. 目前Tensorflow的CV模型大部分均为`NHWC`的输入格式，而Paddle的默认输入格式为`NCHW`，因此X2Paddle在转换过程中，会对如`axis`， `shape`等参数进行转换，适应Paddle的NCHW格式。但在这种情况下，可能会由于TensorFlow模型太复杂，导致出错。 指定`--without_data_format_optimization`后，会停止对`axis`，`shape`等参数的优化（这可能会带来一定数量的transpose操作） + +**Q3. ONNX模型转换过程中，提示『Unknown shape for input tensor[tensor name: "input"] -> shape: ['batch', 'sequence']， Please define shape of input here』** +A：该提示信息表示从ONNX的模型中获取到输入tensor(tensor名为"input:)的shape是语义象征性的['batch', 'sequence']，而不是dim为int类型的shape，从而可能会因为部分node的shape无法推理，导致转换失败。所以用户可以尝试手动在提示后输入详细的shape信息，如:-1,3,224,224 其中-1表示Batch diff --git a/README.md b/README.md index 0481de7e7c483c60ab38b00c9f96b8707d0bc658..075308aaab8cf9c2d6977a31624ce13f114285d3 100644 --- a/README.md +++ b/README.md @@ -10,12 +10,12 @@ X2Paddle在多个主流的CV模型上，测试过TensorFlow/Caffe/ONNX模型的 ## 环境依赖 python == 2.7 | python >= 3.5 -paddlepaddle >= 1.6.0 +paddlepaddle >= 1.8.0 **按需安装以下依赖** tensorflow ： tensorflow == 1.14.0 caffe ： 无 -onnx ： onnx == 1.6.0 onnxruntime == 1.0.0 +onnx ： onnx == 1.6.0 ## 安装 ### 安装方式一（推荐） @@ -61,6 +61,8 @@ x2paddle --framework=paddle2onnx --model=paddle_infer_model_dir --save_dir=onnx_ |--without_data_format_optimization | **[可选]** For TensorFlow, 当指定该参数时，关闭NHWC->NCHW的优化，见[文档Q2](FAQ.md) | |--define_input_shape | **[可选]** For TensorFlow, 当指定该参数时，强制用户输入每个Placeholder的shape，见[文档Q2](FAQ.md) | |--params_merge | **[可选]** 当指定该参数时，转换完成后，inference_model中的所有模型参数将合并保存为一个文件__params__ | +|--onnx_opset | **[可选]** 当framework为paddle2onnx时，该参数可设置转换为ONNX的OpSet版本，目前支持9、10、11，默认为10 | + ## 使用转换后的模型 diff --git a/op_list.md b/op_list.md index 3c4002c3ba8d7f2bf1ab88ee183f46d675af9b03..a69a54bef1ad763602a350dd894f816279a7d531 100644 --- a/op_list.md +++ b/op_list.md @@ -1,5 +1,5 @@ # X2Paddle支持OP列表 -> 目前X2Paddle支持40+的TensorFlow OP，30+的Caffe Layer，覆盖了大部分CV分类模型常用的操作。我们在如下列表中给出了目前X2Paddle支持的全部OP。 +> 目前X2Paddle支持50+的TensorFlow OP，30+的Caffe Layer，覆盖了大部分CV分类模型常用的操作。我们在如下列表中给出了目前X2Paddle支持的全部OP。 **注：** 目前，部分OP暂未支持，如您在转换过程中出现OP不支持的情况，可自行添加或反馈给我们。欢迎通过[ISSUE反馈](https://github.com/PaddlePaddle/X2Paddle/issues/new)的方式告知我们(模型名，代码实现或模型获取方式)，我们会及时跟进：） @@ -20,7 +20,7 @@ | 41 | Cast | 42 | Split | 43 | Squeeze | 44 | ResizeNearestNeighbor | | 45 | Softmax | 46 | Range | 47 | ConcatV2 | 48 | MirrorPad | | 49 | Identity | 50 | GreaterEqual | 51 | StopGradient | 52 | Minimum | -| 53 | RadnomUniform | | | | | | | +| 53 | RadnomUniform | 54 | Fill | 55 | Floor | 56 | DepthToSpace | ## Caffe diff --git a/x2paddle/__init__.py b/x2paddle/__init__.py index ed9d4d87b643f72ff4a7d8458a1d89f2693423b2..c8fef6e4cce341ed356df3a3a27a5655e67cb0ff 100644 --- a/x2paddle/__init__.py +++ b/x2paddle/__init__.py @@ -1 +1,17 @@ __version__ = "0.7.4" + +from .core.program import PaddleProgram + +program = PaddleProgram() + +name_counter = dict() + + +def gen_name(op_name, var_name): + name = "{}_{}".format(op_name, var_name) + if name not in name_counter: + name_counter[name] = 0 + else: + name_counter[name] += 1 + name = name + "_" + str(name_counter[name]) + return name diff --git a/x2paddle/convert.py b/x2paddle/convert.py index 28a303bda9ea81899f03419dc9f538a5e5967a3d..358986911c64ba39864d7f4eb6f023ed1926e938 100644 --- a/x2paddle/convert.py +++ b/x2paddle/convert.py @@ -13,6 +13,7 @@ # limitations under the License. from six import text_type as _text_type +from x2paddle import program import argparse import sys @@ -75,6 +76,12 @@ def arg_parser(): action="store_true", default=False, help="define input shape for tf model") + parser.add_argument( + "--onnx_opset", + "-oo", + type=int, + default=10, + help="when paddle2onnx set onnx opset version to export") parser.add_argument( "--params_merge", "-pm", @@ -114,29 +121,10 @@ def tf2paddle(model_path, print("Now translating model from tensorflow to paddle.") model = TFDecoder(model_path, define_input_shape=define_input_shape) - if not without_data_format_optimization: - mapper = TFOpMapper(model) - optimizer = TFOptimizer(mapper) - # neccesary optimization - optimizer.delete_redundance_code() - # optimizer below is experimental - optimizer.optimize_elementwise_op() - optimizer.merge_activation() - optimizer.merge_bias() - optimizer.optimize_sub_graph() -# optimizer.merge_batch_norm() -# optimizer.merge_prelu() - else: - mapper = TFOpMapperNHWC(model) - optimizer = TFOptimizer(mapper) - optimizer.delete_redundance_code() - optimizer.strip_graph() - optimizer.merge_activation() - optimizer.merge_bias() - optimizer.make_nchw_input_output() - optimizer.remove_transpose() - mapper.save_inference_model(save_dir, params_merge) + mapper = TFOpMapperNHWC(model) + program.build() + program.gen_model(save_dir) def caffe2paddle(proto, weight, save_dir, caffe_proto, params_merge=False): @@ -172,24 +160,26 @@ def onnx2paddle(model_path, save_dir, params_merge=False): return print("Now translating model from onnx to paddle.") - from x2paddle.op_mapper.onnx_op_mapper import ONNXOpMapper + from x2paddle.op_mapper.onnx2paddle.onnx_op_mapper import ONNXOpMapper from x2paddle.decoder.onnx_decoder import ONNXDecoder from x2paddle.optimizer.onnx_optimizer import ONNXOptimizer - import onnxruntime model = ONNXDecoder(model_path) - mapper = ONNXOpMapper(model, save_dir) + mapper = ONNXOpMapper(model) + print("Model optimizing ...") optimizer = ONNXOptimizer(mapper) + print("Model optimized.") - optimizer.delete_redundance_code() + print("Paddle model and code generating ...") mapper.save_inference_model(save_dir, params_merge) + print("Paddle model and code generated.") -def paddle2onnx(model_path, save_dir): +def paddle2onnx(model_path, save_dir, opset_version=10): from x2paddle.decoder.paddle_decoder import PaddleDecoder - from x2paddle.op_mapper.paddle_op_mapper import PaddleOpMapper + from x2paddle.op_mapper.paddle2onnx.paddle_op_mapper import PaddleOpMapper model = PaddleDecoder(model_path, '__model__', '__params__') mapper = PaddleOpMapper() - mapper.convert(model.program, save_dir) + mapper.convert(model.program, save_dir, opset_number=opset_version) def main(): @@ -211,18 +201,6 @@ def main(): assert args.framework is not None, "--framework is not defined(support tensorflow/caffe/onnx)" assert args.save_dir is not None, "--save_dir is not defined" - if args.framework == "onnx": - try: - import onnxruntime as rt - version = rt.__version__ - if version != '1.0.0': - print("[ERROR] onnxruntime==1.0.0 is required") - return - except: - print( - "[ERROR] onnxruntime is not installed, use \"pip install onnxruntime==1.0.0\"." - ) - try: import paddle v0, v1, v2 = paddle.__version__.split('.') @@ -261,13 +239,14 @@ def main(): elif args.framework == "onnx": assert args.model is not None, "--model should be defined while translating onnx model" params_merge = False + if args.params_merge: params_merge = True onnx2paddle(args.model, args.save_dir, params_merge) elif args.framework == "paddle2onnx": assert args.model is not None, "--model should be defined while translating paddle model to onnx" - paddle2onnx(args.model, args.save_dir) + paddle2onnx(args.model, args.save_dir, args.onnx_opset) else: raise Exception( diff --git a/x2paddle/core/fluid_code.py b/x2paddle/core/fluid_code.py index 72ea10788e9519f8ec4256c13220969b3a0b4959..dbf66ef9b89602b692377b993798f338d78482b4 100644 --- a/x2paddle/core/fluid_code.py +++ b/x2paddle/core/fluid_code.py @@ -25,6 +25,7 @@ class Layer(object): self.inputs = dict() self.output = None self.is_custom_layer = False + self.use_fluid = False def get_code(self): layer_code = "" @@ -38,6 +39,8 @@ class Layer(object): layer_code = layer_code + self.op + "(" elif self.op == "=": layer_code = layer_code + elif self.use_fluid: + layer_code = layer_code + "fluid." + self.op + "(" else: layer_code = layer_code + "fluid.layers." + self.op + "(" @@ -108,9 +111,11 @@ class FluidCode(object): inputs, output, param_attr=None, + use_fluid=False, is_custom_layer=False): layer = Layer() layer.op = op + layer.use_fluid = use_fluid layer.is_custom_layer = is_custom_layer if inputs is not None: layer.inputs = inputs diff --git a/x2paddle/core/op_mapper.py b/x2paddle/core/op_mapper.py index d6649a54dc2cde78054781dc5b4b00b2b8522e61..9e6d24162eecdfd858de03d4dabb783edffc94fb 100644 --- a/x2paddle/core/op_mapper.py +++ b/x2paddle/core/op_mapper.py @@ -29,11 +29,14 @@ def export_paddle_param(param, param_name, dir): "bool": [framework_pb2.VarType.BOOL, None] } shape = param.shape + if str(param.dtype) in ['uint8', 'uint_8', 'bool']: + param = param.astype('int64') if len(shape) == 0: assert param.size == 1, "Unexpected situation happend!" shape = [1] - assert str(param.dtype) in dtype_map, "Unknown dtype of params." - + assert str( + param.dtype) in dtype_map, "Unknown dtype {} of params: {}.".format( + str(param.dtype), param_name) fp = open(os.path.join(dir, param_name), 'wb') numpy.array([0], dtype='int32').tofile(fp) numpy.array([0], dtype='int64').tofile(fp) @@ -52,6 +55,24 @@ def export_paddle_param(param, param_name, dir): def run_net(param_dir="./"): import os inputs, outputs = x2paddle_net() + + ops = fluid.default_main_program().global_block().ops + used_vars = list() + for op in ops: + used_vars += op.input_arg_names + + tmp = list() + for input in inputs: + if isinstance(input, list): + for ipt in input: + if ipt.name not in used_vars: + continue + tmp.append(ipt) + else: + if input.name not in used_vars: + continue + tmp.append(input) + inputs = tmp for i, out in enumerate(outputs): if isinstance(out, list): for out_part in out: @@ -119,12 +140,30 @@ class OpMapper(object): import model try: inputs, outputs = model.x2paddle_net() + + ops = fluid.default_main_program().global_block().ops + used_vars = list() + for op in ops: + used_vars += op.input_arg_names + for i, out in enumerate(outputs): if isinstance(out, list): for out_part in out: outputs.append(out_part) del outputs[i] - input_names = [input.name for input in inputs] + + input_names = list() + for input in inputs: + if isinstance(input, list): + for ipt in input: + if ipt.name not in used_vars: + continue + input_names.append(ipt.name) + else: + if input.name not in used_vars: + continue + input_names.append(input.name) + exe = fluid.Executor(fluid.CPUPlace()) exe.run(fluid.default_startup_program()) diff --git a/x2paddle/core/program.py b/x2paddle/core/program.py new file mode 100644 index 0000000000000000000000000000000000000000..7316f22b49351012c1ce4c0f6fcbd6f037de4a53 --- /dev/null +++ b/x2paddle/core/program.py @@ -0,0 +1,225 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import print_function +from __future__ import division +import paddle.fluid as fluid +from paddle.fluid.proto import framework_pb2 +import numpy +import collections +import sys +import os +import six + + +class PaddleLayer(object): + def __init__(self, kernel, inputs, outputs, **kwargs): + assert isinstance( + inputs, + dict), "parameter 'inputs' for PaddleLayer should be type of dict" + assert isinstance( + outputs, + list), "parameter 'outputs' for PaddleLayer should be type of list" + for k, v in inputs.items(): + if isinstance(v, list): + for i in v: + assert isinstance( + i, six.string_types + ), "value in inputs should be type of string or list of string" + else: + assert isinstance(v, six.string_types) or isinstance( + v, list + ), "value in inputs should be type of string or list of string" + for v in outputs: + assert isinstance( + v, six. + string_types), "elements in outputs should be type of string" + self.kernel = kernel + self.inputs = inputs + self.outputs = outputs + self.attrs = kwargs + + +class PaddleProgram(object): + def __init__(self): + self.layers = list() + self.edges_out = dict() + self.edges_in = dict() + self.inputs = list() + self.outputs = list() + self.parameters = dict() + + def clear(self): + self.layers = list() + self.edges_out = dict() + self.edges_in = dict() + self.inputs = list() + self.outputs = list() + self.parameters = dict() + + def add_layer(self, kernel, inputs, outputs, **kwargs): + layer = PaddleLayer(kernel, inputs, outputs, **kwargs) + index = len(self.layers) + self.layers.append(layer) + return index + + def build(self): + outputs_from_nodes = dict() + for i, layer in enumerate(self.layers): + for input_key, input_var in layer.inputs.items(): + vs = input_var + if not isinstance(vs, list): + vs = [vs] + for v in vs: + assert v in outputs_from_nodes, "Couldn't find {} in previous layers, the layers should be make by topological sort".format( + v) + in_layer_index = outputs_from_nodes[v] + if in_layer_index not in self.edges_out: + self.edges_out[in_layer_index] = list() + self.edges_out[in_layer_index].append(i) + + if i not in self.edges_in: + self.edges_in[i] = list() + self.edges_in[i].append(in_layer_index) + for output in layer.outputs: + outputs_from_nodes[output] = i + + def get_layer_outputs(self, i): + return self.edges_out[i] + + def get_layer_inputs(self, i): + return self.edges_in[i] + + def gen_code(self, code_dir): + def write_code(f, code_list, indent=0): + indent_blank = " " * indent + for code_line in code_list: + if code_line.strip() == "": + f.write('\n') + else: + f.write(indent_blank + code_line + '\n') + + if not os.path.exists(code_dir): + os.makedirs(code_dir) + f = open(os.path.join(code_dir, 'x2paddle_model.py'), 'w') + + write_code( + f, [ + "from paddle.fluid.initializer import Constant", + "from paddle.fluid.param_attr import ParamAttr", + "import paddle.fluid as fluid" + "", "def x2paddle_net():" + ], + indent=0) + for i, layer in enumerate(self.layers): + edges_in = self.edges_in.get(i, []) + edges_out = self.edges_out.get(i, []) + if len(edges_in) == 0 and len(edges_out) == 0: + continue + + line = "" + + if len(layer.outputs) == 1: + line = layer.outputs[0] + else: + for output in layer.outputs: + line += "{}, ".format(output) + line = line.strip(", ") + + line += " = {}(".format(layer.kernel) + for k, v in layer.inputs.items(): + if isinstance(v, list): + line += "{}=[{}], ".format(k, ", ".join(v)) + else: + line += "{}={}, ".format(k, v) + for k, v in layer.attrs.items(): + line += "{}={}, ".format(k, v) + line = line.strip(", ") + line += ")" + write_code(f, [line], indent=1) + + write_code( + f, [ + "return [{}], [{}]".format(", ".join(self.inputs), + ", ".join(self.outputs)) + ], + indent=1) + f.close() + + def gen_model(self, save_dir): + code_dir = os.path.join(save_dir, 'model_with_code') + infer_dir = os.path.join(save_dir, 'inference_model') + self.gen_code(code_dir) + sys.path.append(code_dir) + import x2paddle_model + scope = fluid.Scope() + startup_program = fluid.Program() + main_program = fluid.Program() + with fluid.scope_guard(scope): + with fluid.program_guard(main_program, startup_program): + inputs, outputs = x2paddle_model.x2paddle_net() + exe = fluid.Executor(fluid.CPUPlace()) + exe.run(startup_program) + + param_dir = os.path.join(code_dir, 'weights') + for k, v in self.parameters.items(): + if scope.find_var(k): + self.dump_parameter(k, v, param_dir) + + def if_exist(var): + b = os.path.exists( + os.path.join(os.path.join(param_dir, var.name))) + return b + + fluid.io.load_vars( + exe, param_dir, main_program, predicate=if_exist) + fluid.io.save_inference_model( + dirname=infer_dir, + feeded_var_names=[i.name for i in inputs], + target_vars=outputs, + executor=exe) + + def dump_parameter(self, param_name, param, save_dir): + if not os.path.exists(save_dir): + os.makedirs(save_dir) + dtype_map = { + "int16": [framework_pb2.VarType.INT16, 'h'], + "int32": [framework_pb2.VarType.INT32, 'i'], + "int64": [framework_pb2.VarType.INT64, 'q'], + "float16": [framework_pb2.VarType.FP16, 'e'], + "float32": [framework_pb2.VarType.FP32, 'f'], + "float64": [framework_pb2.VarType.FP64, 'd'], + "bool": [framework_pb2.VarType.BOOL, None] + } + shape = param.shape + if str(param.dtype) in ['uint8', 'uint_8', 'bool']: + param = param.astype('int64') + if len(shape) == 0: + assert param.size == 1, "Unexpected situation happend!" + shape = [1] + assert str( + param.dtype) in dtype_map, "Unknown dtype {} of params: {}.".format( + str(param.dtype), param_name) + fp = open(os.path.join(save_dir, param_name), 'wb') + numpy.array([0], dtype='int32').tofile(fp) + numpy.array([0], dtype='int64').tofile(fp) + numpy.array([0], dtype='int32').tofile(fp) + tensor_desc = framework_pb2.VarType.TensorDesc() + tensor_desc.data_type = dtype_map[str(param.dtype)][0] + tensor_desc.dims.extend(shape) + desc_size = tensor_desc.ByteSize() + numpy.array([desc_size], dtype='int32').tofile(fp) + fp.write(tensor_desc.SerializeToString()) + param.tofile(fp) + fp.close() diff --git a/x2paddle/decoder/onnx_decoder.py b/x2paddle/decoder/onnx_decoder.py index 922c6629582ab62843359c8f9219f9e83cb2aa13..280b5b4f1a27a892e243283c6798164c86d73bb7 100644 --- a/x2paddle/decoder/onnx_decoder.py +++ b/x2paddle/decoder/onnx_decoder.py @@ -14,6 +14,7 @@ from x2paddle.core.graph import GraphNode, Graph from x2paddle.core.fluid_code import FluidCode +from x2paddle.decoder.onnx_shape_inference import SymbolicShapeInference from onnx.checker import ValidationError from onnx.checker import check_model from onnx.utils import polish_model @@ -53,7 +54,7 @@ class ONNXGraphNode(GraphNode): convert ONNX node attributes to dict """ return { - attr.name: self.get_attribute_value2(attr) + attr.name: self.get_attribute_value(attr) for attr in self.layer.attribute } @@ -64,7 +65,7 @@ class ONNXGraphNode(GraphNode): return None return self.attr_map['value'] - def get_attribute_value2(self, attr): + def get_attribute_value(self, attr): """ get_attribute_value enhanced """ @@ -130,43 +131,90 @@ class ONNXGraphDataNode(GraphNode): class ONNXGraph(Graph): def __init__(self, onnx_model): - super(ONNXGraph, self).__init__(onnx_model.graph) - self.onnx_model = onnx_model + super(ONNXGraph, self).__init__(onnx_model) + self.fixed_input_shape = {} self.initializer = {} self.place_holder_nodes = list() + self.value_infos = {} + self.graph = onnx_model.graph self.get_place_holder_nodes() - self.value_infos = self.inferred_model_value_info(self.model) - self.results_of_inference = dict() + print("shape inferencing ...") + self.graph = SymbolicShapeInference.infer_shapes( + onnx_model, fixed_input_shape=self.fixed_input_shape) + print("shape inferenced.") + self.build() + self.collect_value_infos() + self.allocate_shapes() def get_inner_nodes(self): """ generate inner node of ONNX model """ inner_nodes = [] - if not isinstance(self.model, onnx.GraphProto): + if not isinstance(self.graph, onnx.GraphProto): logger.error('graph is not a GraphProto instance') return - for initializer in self.model.initializer: + for initializer in self.graph.initializer: name = initializer.name inner_nodes.append(name) return inner_nodes + def get_symbolic_shape(self, dims): + shape = [] + for dim in dims: + if dim.HasField('dim_param'): + shape.append(dim.dim_param) + else: + shape.append(dim.dim_value) + return shape + + def check_input_shape(self, vi): + if vi.type.HasField('tensor_type'): + for dim in vi.type.tensor_type.shape.dim: + if dim.HasField( + 'dim_param') and vi.name not in self.fixed_input_shape: + shape = self.get_symbolic_shape( + vi.type.tensor_type.shape.dim) + print( + "Unknown shape for input tensor[tensor name: '{}'] -> shape: {}, Please define shape of input here,\nNote:you can use visualization tools like Netron to check input shape." + .format(vi.name, shape)) + right_shape_been_input = False + while not right_shape_been_input: + try: + shape = raw_input( + "Shape of Input(e.g. -1,3,224,224), enter 'N' to skip: " + ) + except: + shape = input( + "Shape of Input(e.g. -1,3,224,224), enter 'N' to skip: " + ) + if shape.count("-1") > 1: + print("Only 1 dimension can be -1, type again:)") + else: + right_shape_been_input = True + if shape == 'N': + break + shape = [int(dim) for dim in shape.strip().split(',')] + assert shape.count(-1) <= 1, "Only one dimension can be -1" + self.fixed_input_shape[vi.name] = shape + break + def get_place_holder_nodes(self): """ generate place_holder node of ONNX model """ inner_nodes = self.get_inner_nodes() - input_nodes = [value.name for value in self.model.input] - for ipt_data in input_nodes: - if ipt_data not in inner_nodes: - self.place_holder_nodes.append(ipt_data) + for ipt_vi in self.graph.input: + if ipt_vi.name not in inner_nodes: + self.check_input_shape(ipt_vi) + self.place_holder_nodes.append(ipt_vi.name) def get_output_nodes(self): """ generate output_nodes node of ONNX model """ inner_nodes = self.get_inner_nodes() - output_nodes = [value.name for value in self.model.output] + output_nodes = [value.name for value in self.graph.output] for opt_data in output_nodes: if opt_data not in inner_nodes: self.output_nodes.append(opt_data) @@ -183,11 +231,11 @@ class ONNXGraph(Graph): """ build topo_sort of ONNX model """ - for layer in self.model.node: + for layer in self.graph.node: node = ONNXGraphNode(layer) self.node_map[layer.name] = node - for layer in self.model.input: + for layer in self.graph.input: if layer.name not in self.node_map: is_place_holder = self.is_place_holder_nodes(layer.name) self.node_map[layer.name] = ONNXGraphDataNode( @@ -196,7 +244,7 @@ class ONNXGraph(Graph): is_global_input=is_place_holder) #set data node's weight - for initializer in self.model.initializer: + for initializer in self.graph.initializer: name = initializer.name weight = to_array(initializer) if name in self.node_map: @@ -228,7 +276,7 @@ class ONNXGraph(Graph): continue if in_node not in self.node_map: flag = 0 - for nd in self.model.node: + for nd in self.graph.node: for idx, opt in enumerate(nd.output): if opt == in_node: self.connect(nd.name, layer_name) @@ -256,81 +304,68 @@ class ONNXGraph(Graph): ipt_node.index = node.which_child[ipt_node.layer_name] return ipt_node - def graph_weights(self, graph): + def graph_weights(self): """ generator for weights """ - if not isinstance(graph, onnx.GraphProto): + if not isinstance(self.graph, onnx.GraphProto): logger.error('graph is not a GraphProto instance') return - for initializer in graph.initializer: + for initializer in self.graph.initializer: name = initializer.name weight = to_array(initializer) yield name, weight - def inferred_model_value_info(self, graph): + def collect_value_infos(self): """ collect value/type info for an ONNX model """ - assert isinstance(graph, + assert isinstance(self.graph, onnx.GraphProto), 'model is not a ModelProto instance' - value_info = Dict() - for item in graph.value_info: - value_info[item.name] = { + for item in self.graph.value_info: + self.value_infos[item.name] = { 'dtype': TENSOR_TYPE_TO_NP_TYPE[item.type.tensor_type.elem_type], 'shape': [dim.dim_value for dim in item.type.tensor_type.shape.dim], 'external': False } - for item in graph.input: - assert item.name not in value_info - value_info[item.name] = { - 'dtype': - TENSOR_TYPE_TO_NP_TYPE[item.type.tensor_type.elem_type], - 'shape': - [dim.dim_value for dim in item.type.tensor_type.shape.dim], - 'external': True - } - for item in graph.output: - assert item.name not in value_info - value_info[item.name] = { - 'dtype': - TENSOR_TYPE_TO_NP_TYPE[item.type.tensor_type.elem_type], - 'shape': - [dim.dim_value for dim in item.type.tensor_type.shape.dim], - 'external': True - } - return value_info + + def allocate_shapes(self): + """ + run shape inference + """ + for layer in self.graph.node: + node = self.node_map[layer.name] + for opt in layer.output: + if opt in self.value_infos: + value_info = self.value_infos[opt] + #if len(value_info['shape']) == 0 or value_info[ + # 'dtype'] is None or 0 in value_info['shape']: + # #TODO add node shape inference + node.dtype = value_info['dtype'] + node.out_shapes.append(value_info['shape']) + else: + node.out_shapes.append([]) class ONNXDecoder(object): def __init__(self, onnx_model): - model = onnx.load(onnx_model) + onnx_model = onnx.load(onnx_model) print('model ir_version: {}, op version: {}'.format( - model.ir_version, model.opset_import[0].version)) - if model.opset_import[0].version < 9: - _logger.warning( - 'Now, onnx2paddle support convert onnx model opset_verison == 9,' - 'opset_verison of your onnx model is %d < 9,' - 'some operator maybe unsuccessful in convertion.', - model.opset_import[0].version) - - check_model(model) - self.check_model_running_state(onnx_model) - - model = onnx.shape_inference.infer_shapes(model) - model = self.optimize_model_skip_op_for_inference(model) - model = self.optimize_model_strip_initializer(model) - self.standardize_variable_name(model.graph) - - self.model = model - graph = model.graph - self.onnx_graph = ONNXGraph(model) - self.onnx_graph.build() + onnx_model.ir_version, onnx_model.opset_import[0].version)) + self.op_set = onnx_model.opset_import[0].version + + check_model(onnx_model) + + onnx_model = self.optimize_model_skip_op(onnx_model) + onnx_model = self.optimize_model_strip_initializer(onnx_model) + onnx_model = self.optimize_node_name(onnx_model) + self.graph = ONNXGraph(onnx_model) + #self.onnx_model = onnx_model def build_value_refs(self, nodes): """ @@ -373,14 +408,13 @@ class ONNXDecoder(object): processed += 1 return processed - def optimize_model_skip_op_for_inference(self, model, op_list=None): + def optimize_model_skip_op(self, model, op_list=None): """ skip ops can be bypassed for inference """ + nodes = model.graph.node if op_list is None: op_list = ['Dropout'] - - nodes = model.graph.node input_refs, output_refs = self.build_value_refs(nodes) ret = type(model)() ret.CopyFrom(model) @@ -473,38 +507,11 @@ class ONNXDecoder(object): name = name.replace(s, '_') return 'x2paddle_' + name - def check_model_running_state(self, model_path): - import onnxruntime as rt - model = onnx.load(model_path) - model = onnx.shape_inference.infer_shapes(model) - if len(model.graph.value_info) < len(model.graph.node) - 1: - _logger.warning( - 'During conversion of your model, some operators will be assignd node.out_shape==None, ' - 'refer to https://github.com/onnx/onnx/blob/master/docs/ShapeInference.md' - ) - try: - datatype_map = { - 'tensor(int64)': 'int', - 'tensor(float)': 'float32', - 'tensor(int32)': 'int32' - } - input_dict = {} - sess = rt.InferenceSession(model_path) - for ipt in sess.get_inputs(): - datatype = datatype_map[ipt.type] - input_dict[ipt.name] = np.random.random(ipt.shape).astype( - datatype) - - res = sess.run(None, input_feed=input_dict) - except: - raise Exception( - "onnxruntime inference onnx model failed, Please confirm the correctness of onnx model by onnxruntime, if onnx model is correct, please submit issue in github." - ) - - def standardize_variable_name(self, graph): + def optimize_node_name(self, model): """ standardize variable name for paddle's code """ + graph = model.graph for initializer in graph.initializer: initializer.name = self.make_variable_name(initializer.name) for ipt in graph.input: @@ -523,3 +530,4 @@ class ONNXDecoder(object): node.input[i] = self.make_variable_name(node.input[i]) for i in range(len(node.output)): node.output[i] = self.make_variable_name(node.output[i]) + return model diff --git a/x2paddle/decoder/onnx_shape_inference.py b/x2paddle/decoder/onnx_shape_inference.py new file mode 100644 index 0000000000000000000000000000000000000000..303fc290deb6aef43cd8a9f6fd53078ff77d1564 --- /dev/null +++ b/x2paddle/decoder/onnx_shape_inference.py @@ -0,0 +1,1604 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Reference Code from https://github.com/microsoft/onnxruntime, Licensed under the MIT License. + +# -*- coding: UTF-8 -*- +import argparse +import numpy as np +import onnx +import sys +from onnx import helper, numpy_helper, shape_inference +import sympy + +from packaging import version +assert version.parse(onnx.__version__) >= version.parse("1.5.0") + + +def get_attribute(node, attr_name, default_value=None): + found = [attr for attr in node.attribute if attr.name == attr_name] + if found: + return helper.get_attribute_value(found[0]) + return default_value + + +def get_dim_from_type_proto(dim): + return getattr(dim, dim.WhichOneof('value')) if type( + dim.WhichOneof('value')) == str else None + + +def get_shape_from_type_proto(type_proto): + return [ + get_dim_from_type_proto(d) for d in type_proto.tensor_type.shape.dim + ] + + +def get_shape_from_sympy_shape(sympy_shape): + sympy_shape = [ + None if i is None else (int(i) if is_literal(i) else str(i)) + for i in sympy_shape + ] + return sympy_shape + + +def is_literal(dim): + return type(dim) in [int, np.int64, np.int32, sympy.Integer] or ( + hasattr(dim, 'is_number') and + dim.is_number) # or (hasattr(dim, 'is_integer') and dim.is_integer) + + +def handle_negative_axis(axis, rank): + assert axis < rank and axis >= -rank + return axis if axis >= 0 else rank + axis + + +def get_opset(mp, domain=['', 'onnx', 'ai.onnx']): + if type(domain) != list: + domain = [domain] + for opset in mp.opset_import: + if opset.domain in domain: + return opset.version + return None + + +def as_scalar(x): + if type(x) == list: + assert len(x) == 1 + return x[0] + elif type(x) == np.ndarray: + return np.asscalar(x) + else: + return x + + +def as_list(x, keep_none): + if type(x) == list: + return x + elif type(x) == np.ndarray: + return list(x) + elif keep_none and x is None: + return None + else: + return [x] + + +def sympy_reduce_product(x): + if type(x) == list: + value = sympy.Integer(1) + for v in x: + value = value * v + else: + value = x + return value + + +class SymbolicShapeInference: + def __init__(self, int_max, auto_merge, guess_output_rank, verbose): + self.dispatcher_ = { + 'Add': self._infer_symbolic_compute_ops, + 'ArrayFeatureExtractor': self._infer_ArrayFeatureExtractor, + 'AveragePool': self._infer_Pool, + 'Cast': self._infer_Cast, + 'CategoryMapper': self._infer_CategoryMapper, + 'Compress': self._infer_Compress, + 'Concat': self._infer_Concat, + 'ConstantOfShape': self._infer_ConstantOfShape, + 'Conv': self._infer_Conv, + 'CumSum': self._pass_on_shape_and_type, + 'Div': self._infer_symbolic_compute_ops, + 'Expand': self._infer_Expand, + 'Equal': self._infer_symbolic_compute_ops, + 'Gather': self._infer_Gather, + 'GatherElements': self._infer_GatherElements, + 'GatherND': self._infer_GatherND, + 'If': self._infer_If, + 'Loop': self._infer_Loop, + 'MatMul': self._infer_MatMul, + 'MatMulInteger16': self._infer_MatMulInteger, + 'MaxPool': self._infer_Pool, + 'Max': self._infer_symbolic_compute_ops, + 'Min': self._infer_symbolic_compute_ops, + 'Mul': self._infer_symbolic_compute_ops, + 'NonMaxSuppression': self._infer_NonMaxSuppression, + 'NonZero': self._infer_NonZero, + 'OneHot': self._infer_OneHot, + 'Pad': self._infer_Pad, + 'Range': self._infer_Range, + 'ReduceProd': self._infer_ReduceProd, + 'Reshape': self._infer_Reshape, + 'Resize': self._infer_Resize, + 'Round': self._pass_on_shape_and_type, + 'Scan': self._infer_Scan, + 'ScatterElements': self._infer_ScatterElements, + 'Shape': self._infer_Shape, + 'Size': self._infer_Size, + 'Slice': self._infer_Slice, + 'Split': self._infer_Split, + 'Squeeze': self._infer_Squeeze, + 'Sub': self._infer_symbolic_compute_ops, + 'Tile': self._infer_Tile, + 'TopK': self._infer_TopK, + 'Unsqueeze': self._infer_Unsqueeze, + 'Where': self._infer_symbolic_compute_ops, + 'Transpose': self._infer_Transpose, + 'ZipMap': self._infer_ZipMap + } + self.run_ = True + self.suggested_merge_ = {} + self.symbolic_dims_ = {} + self.input_symbols_ = {} + self.auto_merge_ = auto_merge + self.guess_output_rank_ = guess_output_rank + self.verbose_ = verbose + self.int_max_ = int_max + + def _add_suggested_merge(self, symbols, apply=False): + assert all([(type(s) == str and s in self.symbolic_dims_) or + is_literal(s) for s in symbols]) + symbols = set(symbols) + for k, v in self.suggested_merge_.items(): + if k in symbols: + symbols.remove(k) + symbols.add(v) + map_to = None + # if there is literal, map to it first + for s in symbols: + if is_literal(s): + map_to = s + break + # when no literals, map to input symbolic dims, then existing symbolic dims + if map_to is None: + for s in symbols: + if s in self.input_symbols_: + map_to = s + break + if map_to is None: + for s in symbols: + if type(self.symbolic_dims_[s]) == sympy.Symbol: + map_to = s + break + # when nothing to map to, use the shorter one + if map_to is None: + if self.verbose_ > 0: + print( + 'Potential unsafe merge between symbolic expressions: ({})'. + format(','.join(symbols))) + symbols_list = list(symbols) + lens = [len(s) for s in symbols_list] + map_to = symbols_list[lens.index(min(lens))] + symbols.remove(map_to) + + for s in symbols: + if s == map_to: + continue + if is_literal(map_to) and is_literal(s): + assert int(map_to) == int(s) + self.suggested_merge_[s] = int(map_to) if is_literal( + map_to) else map_to + for k, v in self.suggested_merge_.items(): + if v == s: + self.suggested_merge_[k] = map_to + if apply and self.auto_merge_: + self._apply_suggested_merge() + + def _apply_suggested_merge(self, graph_input_only=False): + if not self.suggested_merge_: + return + for i in list(self.out_mp_.graph.input) + ( + [] if graph_input_only else list(self.out_mp_.graph.value_info)): + for d in i.type.tensor_type.shape.dim: + if d.dim_param in self.suggested_merge_: + v = self.suggested_merge_[d.dim_param] + if is_literal(v): + d.dim_value = int(v) + else: + d.dim_param = v + + def _preprocess(self, in_mp, input_shapes=None): + out_mp = onnx.ModelProto() + out_mp.CopyFrom(in_mp) + out_mp.graph.ClearField('node') + self.out_mp_ = out_mp + + defined = set([ + i.name + for i in list(in_mp.graph.input) + list(in_mp.graph.initializer) + ]) + pending_nodes = [] + + # returns True if no more ready nodes + def _insert_ready_nodes(): + ready_nodes = [ + pn for pn in pending_nodes + if all([i in defined for i in pn.input if i]) + ] + for rn in ready_nodes: + self.out_mp_.graph.node.add().CopyFrom(rn) + for o in rn.output: + defined.add(o) + pending_nodes.remove(rn) + return not ready_nodes + + # constant op -> initializer, topological sort + for in_n in in_mp.graph.node: + if in_n.op_type == 'Constant': + t = get_attribute(in_n, 'value') + t.name = in_n.output[0] + self.out_mp_.graph.initializer.add().CopyFrom(t) + defined.add(t.name) + else: + pending_nodes.append(in_n) + _insert_ready_nodes() + + while pending_nodes: + if _insert_ready_nodes(): + break + + if pending_nodes and self.verbose_ > 0: + print('SymbolicShapeInference: orphaned nodes discarded: ') + print( + * [n.op_type + ': ' + n.output[0] for n in pending_nodes], + sep='\n') + if input_shapes is not None: + for input_name, shape in input_shapes.items(): + for idx in range(len(self.out_mp_.graph.input)): + if self.out_mp_.graph.input[idx].name == input_name: + value_info = self.out_mp_.graph.input[idx] + del self.out_mp_.graph.input[idx] + self.out_mp_.graph.input.append( + helper.make_tensor_value_info( + value_info.name, + value_info.type.tensor_type.elem_type, shape)) + self.initializers_ = dict( + [(i.name, i) for i in self.out_mp_.graph.initializer]) + self.known_vi_ = dict( + [(i.name, i) for i in list(self.out_mp_.graph.input)]) + self.known_vi_.update( + dict([(i.name, helper.make_tensor_value_info(i.name, i.data_type, + list(i.dims))) + for i in self.out_mp_.graph.initializer])) + + def _merge_symbols(self, dims): + if not all([type(d) == str for d in dims]): + if self.auto_merge_: + assert len( + dims + ) == 2 # only allow symbol->int merge in binary ops for now + is_int = [is_literal(d) for d in dims] + if sum(is_int) == 1: + int_dim = is_int.index(1) + if self.verbose_ > 0: + print('dim {} has been merged with value {}'.format( + dims[1 - int_dim], dims[int_dim])) + self._check_merged_dims(dims, allow_broadcast=False) + return dims[int_dim] + else: + if self.verbose_ > 0: + print('dim {} has been mergd with dim {}'.format(dims[ + 0], dims[1])) + return dims[0] + else: + return None + if all([d == dims[0] for d in dims]): + return dims[0] + merged = [ + self.suggested_merge_[d] if d in self.suggested_merge_ else d + for d in dims + ] + if all([d == merged[0] for d in merged]): + assert merged[0] in self.symbolic_dims_ + return merged[0] + else: + return None + + # broadcast from right to left, and merge symbolic dims if needed + def _broadcast_shapes(self, shape1, shape2): + new_shape = [] + rank1 = len(shape1) + rank2 = len(shape2) + new_rank = max(rank1, rank2) + for i in range(new_rank): + dim1 = shape1[rank1 - 1 - i] if i < rank1 else 1 + dim2 = shape2[rank2 - 1 - i] if i < rank2 else 1 + if dim1 == 1 or dim1 == dim2: + new_dim = dim2 + elif dim2 == 1: + new_dim = dim1 + else: + new_dim = self._merge_symbols([dim1, dim2]) + if not new_dim: + # warning about unsupported broadcast when not auto merge + # note that auto merge has the risk of incorrectly merge symbols while one of them being 1 + # for example, 'a' = 1, 'b' = 5 at runtime is valid broadcasting, but with auto merge 'a' == 'b' + if self.auto_merge_: + self._add_suggested_merge([dim1, dim2], apply=True) + else: + print('unsupported broadcast between ' + str(dim1) + ' ' + + str(dim2)) + new_shape = [new_dim] + new_shape + return new_shape + + def _get_shape(self, node, idx): + name = node.input[idx] + shape = [] + if name in self.known_vi_: + shape = get_shape_from_type_proto(self.known_vi_[name].type) + elif name in self.initializers_: + assert name in self.initializers_ + shape = list(self.initializers_[name].dims) + return shape + + def _get_initializer_value(self, node, idx): + name = node.input[idx] + if name in self.initializers_: + value = numpy_helper.to_array(self.initializers_[name]) + return value + else: + return False + + def _get_shape_rank(self, node, idx): + return len(self._get_shape(node, idx)) + + def _get_sympy_shape(self, node, idx): + sympy_shape = [] + for d in self._get_shape(node, idx): + if type(d) is str: + sympy_shape.append(self.symbolic_dims_[d] if d in + self.symbolic_dims_ else sympy.Symbol( + d, integer=True)) + else: + assert None != d + sympy_shape.append(d) + return sympy_shape + + def _get_value(self, node, idx): + name = node.input[idx] + assert name in self.sympy_data_ or name in self.initializers_ + return self.sympy_data_[ + name] if name in self.sympy_data_ else numpy_helper.to_array( + self.initializers_[name]) + + def _try_get_value(self, node, idx): + if idx >= len(node.input): + return None + name = node.input[idx] + if name in self.sympy_data_ or name in self.initializers_: + return self._get_value(node, idx) + return None + + def _update_computed_dims(self, new_sympy_shape): + for i, new_dim in enumerate(new_sympy_shape): + if not is_literal(new_dim) and not type(new_dim) == str: + str_dim = str(new_dim) + if str_dim in self.suggested_merge_: + new_sympy_shape[i] = self.symbolic_dims_[ + self.suggested_merge_[str_dim]] + else: + # add new_dim if it's a computational expression + if not str(new_dim) in self.symbolic_dims_: + self.symbolic_dims_[str(new_dim)] = new_dim + + def _onnx_infer_single_node(self, node): + # skip onnx shape inference for Scan/Loop + skip_infer = node.op_type in ['Scan', 'Loop'] + if not skip_infer: + # run single node inference with self.known_vi_ shapes + # note that inference rely on initializer values is not handled + # as we don't copy initializer weights to tmp_graph for inference speed purpose + tmp_graph = helper.make_graph( + [node], 'tmp', [self.known_vi_[i] for i in node.input if i], [ + helper.make_tensor_value_info(i, onnx.TensorProto.UNDEFINED, + None) for i in node.output + ]) + self.tmp_mp_.graph.CopyFrom(tmp_graph) + self.tmp_mp_ = shape_inference.infer_shapes(self.tmp_mp_) + for i_o in range(len(node.output)): + o = node.output[i_o] + vi = self.out_mp_.graph.value_info.add() + if not skip_infer: + vi.CopyFrom(self.tmp_mp_.graph.output[i_o]) + self.known_vi_[o] = vi + + def _onnx_infer_subgraph(self, node, subgraph, use_node_input=True): + if self.verbose_ > 2: + print('Inferencing subgraph of node {} with output({}...): {}'. + format(node.name, node.output[0], node.op_type)) + # node inputs are not passed directly to the subgraph + # it's up to the node dispatcher to prepare subgraph input + # for example, with Scan/Loop, subgraph input shape would be trimmed from node input shape + # besides, inputs in subgraph could shadow implicit inputs + subgraph_inputs = set([ + i.name for i in list(subgraph.initializer) + list(subgraph.input) + ]) + subgraph_implicit_input = set([ + name for name in self.known_vi_.keys() + if not name in subgraph_inputs + ]) + tmp_graph = helper.make_graph( + list(subgraph.node), 'tmp', + list(subgraph.input) + + [self.known_vi_[i] for i in subgraph_implicit_input], [ + helper.make_tensor_value_info(i.name, + onnx.TensorProto.UNDEFINED, None) + for i in subgraph.output + ]) + tmp_graph.initializer.extend([ + i for i in self.out_mp_.graph.initializer + if i.name in subgraph_implicit_input + ]) + tmp_graph.initializer.extend(subgraph.initializer) + self.tmp_mp_.graph.CopyFrom(tmp_graph) + + symbolic_shape_inference = SymbolicShapeInference( + self.int_max_, self.auto_merge_, self.guess_output_rank_, + self.verbose_) + all_shapes_inferred = False + symbolic_shape_inference._preprocess(self.tmp_mp_) + symbolic_shape_inference.suggested_merge_ = self.suggested_merge_.copy() + while symbolic_shape_inference.run_: + all_shapes_inferred = symbolic_shape_inference._infer_impl( + self.tmp_mp_, self.sympy_data_.copy()) + symbolic_shape_inference._update_output_from_vi() + if use_node_input: + # if subgraph uses node input, it needs to update to merged dims + subgraph.ClearField('input') + subgraph.input.extend( + symbolic_shape_inference.out_mp_.graph.input[:len(node.input)]) + subgraph.ClearField('output') + subgraph.output.extend(symbolic_shape_inference.out_mp_.graph.output) + subgraph.ClearField('value_info') + subgraph.value_info.extend( + symbolic_shape_inference.out_mp_.graph.value_info) + subgraph.ClearField('node') + subgraph.node.extend(symbolic_shape_inference.out_mp_.graph.node) + # for new symbolic dims from subgraph output, add to main graph symbolic dims + subgraph_shapes = [ + get_shape_from_type_proto(o.type) + for o in symbolic_shape_inference.out_mp_.graph.output + ] + subgraph_new_symbolic_dims = set([ + d for s in subgraph_shapes + if s for d in s if type(d) == str and not d in self.symbolic_dims_ + ]) + new_dims = {} + for d in subgraph_new_symbolic_dims: + assert d in symbolic_shape_inference.symbolic_dims_ + new_dims[d] = symbolic_shape_inference.symbolic_dims_[d] + self.symbolic_dims_.update(new_dims) + return symbolic_shape_inference + + def _get_int_values(self, node, broadcast=False): + values = [self._try_get_value(node, i) for i in range(len(node.input))] + if all([v is not None for v in values]): + # some shape compute is in floating point, cast to int for sympy + for i, v in enumerate(values): + if type(v) != np.ndarray: + continue + if len(v.shape) > 1: + new_v = None # ignore value for rank > 1 + elif len(v.shape) == 0: + new_v = int(np.asscalar(v)) + else: + assert len(v.shape) == 1 + new_v = [int(vv) for vv in v] + values[i] = new_v + values_len = [len(v) if type(v) == list else 0 for v in values] + max_len = max(values_len) + if max_len >= 1 and broadcast: + # broadcast + for i, v in enumerate(values): + if v is None: + continue # don't broadcast if value is unknown + if type(v) == list: + if len(v) < max_len: + values[i] = v * max_len + else: + assert len(v) == max_len + else: + values[i] = [v] * max_len + return values + + def _compute_on_sympy_data(self, node, op_func): + assert len(node.output) == 1 + values = self._get_int_values(node, broadcast=True) + if all([v is not None for v in values]): + new_shape = [] + is_list = [type(v) == list for v in values] + as_list = any(is_list) + if as_list: + data = [op_func(vs) for vs in zip(*values)] + self.sympy_data_[node.output[0]] = data + new_shape = np.array(data).shape + else: + data = op_func(values) + self.sympy_data_[node.output[0]] = data + new_shape = np.array(data).shape + vi = self.known_vi_[node.output[0]] + #print(node.output[0]) + #print(new_shape) + #vi.CopyFrom(helper.make_tensor_value_info(node.output[0], self.known_vi_[node.input[0]].type.tensor_type.elem_type, list(new_shape))) + + def _pass_on_sympy_data(self, node): + assert len(node.input) == 1 or node.op_type == 'Reshape' + self._compute_on_sympy_data(node, lambda x: x[0]) + + def _pass_on_shape_and_type(self, node): + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, + self._get_shape(node, 0))) + + def _new_symbolic_dim(self, prefix, dim): + new_dim = '{}_d{}'.format(prefix, dim) + if new_dim in self.suggested_merge_: + v = self.suggested_merge_[new_dim] + new_dim = sympy.Integer(int(v)) if is_literal(v) else v + else: + self.symbolic_dims_[new_dim] = sympy.Symbol(new_dim, integer=True) + return new_dim + + def _new_symbolic_dim_from_output(self, node, out_idx=0, dim=0): + return self._new_symbolic_dim('{}{}_o{}_'.format( + node.op_type, list(self.out_mp_.graph.node).index(node), out_idx), + dim) + + def _new_symbolic_shape(self, rank, node, out_idx=0): + return [ + self._new_symbolic_dim_from_output(node, out_idx, i) + for i in range(rank) + ] + + def _compute_conv_pool_shape(self, node): + sympy_shape = self._get_sympy_shape(node, 0) + if len(node.input) > 1: + W_shape = self._get_sympy_shape(node, 1) + rank = len(W_shape) - 2 # number of spatial axes + kernel_shape = W_shape[-rank:] + sympy_shape[1] = W_shape[0] + else: + W_shape = None + kernel_shape = get_attribute(node, 'kernel_shape') + rank = len(kernel_shape) + + assert len(sympy_shape) == rank + 2 + + # only need to symbolic shape inference if input has symbolic dims in spatial axes + is_symbolic_dims = [not is_literal(i) for i in sympy_shape[-rank:]] + + if not any(is_symbolic_dims): + shape = get_shape_from_type_proto(self.known_vi_[node.output[0]] + .type) + if len(shape) > 0: + assert len(sympy_shape) == len(shape) + sympy_shape[-rank:] = [sympy.Integer(d) for d in shape[-rank:]] + return sympy_shape + + dilations = get_attribute(node, 'dilations', [1] * rank) + strides = get_attribute(node, 'strides', [1] * rank) + effective_kernel_shape = [(k - 1) * d + 1 + for k, d in zip(kernel_shape, dilations)] + pads = get_attribute(node, 'pads') + if pads is None: + pads = [0] * (2 * rank) + auto_pad = get_attribute(node, 'auto_pad', + b'NOTSET').decode('utf-8') + if auto_pad != 'VALID' and auto_pad != 'NOTSET': + try: + residual = [ + sympy.Mod(d, s) + for d, s in zip(sympy_shape[-rank:], strides) + ] + total_pads = [ + max(0, (k - s) if r == 0 else (k - r)) + for k, s, r in zip(effective_kernel_shape, strides, + residual) + ] + except TypeError: # sympy may throw TypeError: cannot determine truth value of Relational + total_pads = [ + max(0, (k - s)) + for k, s in zip(effective_kernel_shape, strides) + ] # assuming no residual if sympy throws error + elif auto_pad == 'VALID': + total_pads = [] + else: + total_pads = [0] * rank + else: + assert len(pads) == 2 * rank + total_pads = [p1 + p2 for p1, p2 in zip(pads[:rank], pads[rank:])] + + ceil_mode = get_attribute(node, 'ceil_mode', 0) + for i in range(rank): + effective_input_size = sympy_shape[-rank + i] + if len(total_pads) > 0: + effective_input_size = effective_input_size + total_pads[i] + if ceil_mode: + strided_kernel_positions = sympy.ceiling( + (effective_input_size - effective_kernel_shape[i]) / + strides[i]) + else: + strided_kernel_positions = ( + effective_input_size - effective_kernel_shape[i] + ) // strides[i] + sympy_shape[-rank + i] = strided_kernel_positions + 1 + return sympy_shape + + def _check_merged_dims(self, dims, allow_broadcast=True): + if allow_broadcast: + dims = [d for d in dims if not (is_literal(d) and int(d) <= 1)] + if not all([d == dims[0] for d in dims]): + self._add_suggested_merge(dims, apply=True) + + def _compute_matmul_shape(self, node, output_dtype=None): + lhs_shape = self._get_shape(node, 0) + rhs_shape = self._get_shape(node, 1) + lhs_rank = len(lhs_shape) + rhs_rank = len(rhs_shape) + lhs_reduce_dim = 0 + rhs_reduce_dim = 0 + assert lhs_rank > 0 and rhs_rank > 0 + if lhs_rank == 1 and rhs_rank == 1: + new_shape = [] + elif lhs_rank == 1: + rhs_reduce_dim = -2 + new_shape = rhs_shape[:rhs_reduce_dim] + [rhs_shape[-1]] + elif rhs_rank == 1: + lhs_reduce_dim = -1 + new_shape = lhs_shape[:lhs_reduce_dim] + else: + lhs_reduce_dim = -1 + rhs_reduce_dim = -2 + new_shape = self._broadcast_shapes( + lhs_shape[:-2], rhs_shape[:-2]) + [lhs_shape[-2] + ] + [rhs_shape[-1]] + # merge reduce dim + self._check_merged_dims( + [lhs_shape[lhs_reduce_dim], rhs_shape[rhs_reduce_dim]], + allow_broadcast=False) + if output_dtype is None: + # infer output_dtype from input type when not specified + output_dtype = self.known_vi_[node.input[ + 0]].type.tensor_type.elem_type + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], output_dtype, + new_shape)) + + def _infer_ArrayFeatureExtractor(self, node): + data_shape = self._get_shape(node, 0) + indices_shape = self._get_shape(node, 1) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, data_shape[:-1] + + indices_shape)) + + def _infer_symbolic_compute_ops(self, node): + funcs = { + 'Add': lambda l: l[0] + l[1], + 'Div': lambda l: l[0] // l[1], # integer div in sympy + 'Equal': lambda l: l[0] == l[1], + 'Max': + lambda l: l[1] if is_literal(l[0]) and int(l[0]) < -self.int_max_ else (l[0] if is_literal(l[1]) and int(l[1]) < -self.int_max_ else sympy.Max(l[0], l[1])), + 'Min': + lambda l: l[1] if is_literal(l[0]) and int(l[0]) > self.int_max_ else (l[0] if is_literal(l[1]) and int(l[1]) > self.int_max_ else sympy.Min(l[0], l[1])), + 'Mul': lambda l: l[0] * l[1], + 'Sub': lambda l: l[0] - l[1], + 'Where': lambda l: l[1] if l[0] else l[2] + } + assert node.op_type in funcs + self._compute_on_sympy_data(node, funcs[node.op_type]) + + def _infer_Cast(self, node): + self._pass_on_sympy_data(node) + + def _infer_CategoryMapper(self, node): + input_type = self.known_vi_[node.input[0]].type.tensor_type.elem_type + if input_type == onnx.TensorProto.STRING: + output_type = onnx.TensorProto.INT64 + else: + output_type = onnx.TensorProto.STRING + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], output_type, + self._get_shape(node, 0))) + + def _infer_Transpose(self, node): + input_shape = self._get_shape(node, 0) + perm = get_attribute(node, 'perm') + output_shape = np.array(input_shape)[perm].tolist() + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, output_shape)) + + def _infer_Compress(self, node): + input_shape = self._get_shape(node, 0) + # create a new symbolic dimension for Compress output + compress_len = self._new_symbolic_dim_from_output(node) + axis = get_attribute(node, 'axis') + if axis == None: + # when axis is not specified, input is flattened before compress so output is 1D + output_shape = [compress_len] + else: + output_shape = input_shape + output_shape[handle_negative_axis(axis, len( + input_shape))] = compress_len + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, output_shape)) + + def _infer_Concat(self, node): + if any([i in self.sympy_data_ for i in node.input]): + values = self._get_int_values(node) + if all([v is not None for v in values]): + assert 0 == get_attribute(node, 'axis') + self.sympy_data_[node.output[0]] = [] + for i in range(len(node.input)): + value = values[i] + if type(value) == list: + self.sympy_data_[node.output[0]].extend(value) + else: + self.sympy_data_[node.output[0]].append(value) + + sympy_shape = self._get_sympy_shape(node, 0) + axis = handle_negative_axis( + get_attribute(node, 'axis'), len(sympy_shape)) + for i_idx in range(1, len(node.input)): + input_shape = self._get_sympy_shape(node, i_idx) + if input_shape: + sympy_shape[axis] = sympy_shape[axis] + input_shape[axis] + self._update_computed_dims(sympy_shape) + # merge symbolic dims for non-concat axes + for d in range(len(sympy_shape)): + if d == axis: + continue + dims = [ + self._get_shape(node, i_idx)[d] + for i_idx in range(len(node.input)) + if self._get_shape(node, i_idx) + ] + if all([d == dims[0] for d in dims]): + continue + merged = self._merge_symbols(dims) + if type(merged) == str: + sympy_shape[d] = self.symbolic_dims_[merged] if merged else None + else: + sympy_shape[d] = merged + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], self.known_vi_[node.input[0]].type.tensor_type. + elem_type, get_shape_from_sympy_shape(sympy_shape))) + + def _infer_Conv(self, node): + sympy_shape = self._compute_conv_pool_shape(node) + self._update_computed_dims(sympy_shape) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], vi.type.tensor_type.elem_type, + get_shape_from_sympy_shape(sympy_shape))) + + def _infer_ConstantOfShape(self, node): + sympy_shape = self._get_int_values(node)[0] + vi = self.known_vi_[node.output[0]] + if sympy_shape is not None: + if type(sympy_shape) != list: + sympy_shape = [sympy_shape] + self._update_computed_dims(sympy_shape) + # update sympy data if output type is int, and shape is known + if vi.type.tensor_type.elem_type == onnx.TensorProto.INT64 and all( + [is_literal(x) for x in sympy_shape]): + self.sympy_data_[node.output[0]] = np.ones( + [int(x) for x in sympy_shape], + dtype=np.int64) * numpy_helper.to_array( + get_attribute(node, 'value', 0)) + else: + # create new dynamic shape + sympy_shape = self._new_symbolic_shape( + self._get_shape_rank(node, 0), node) + + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], vi.type.tensor_type.elem_type, + get_shape_from_sympy_shape(sympy_shape))) + + def _infer_Expand(self, node): + expand_to_shape = self._try_get_value(node, 1) + if expand_to_shape is not None: + # new_shape's dim can come from shape value + self._update_computed_dims(expand_to_shape) + shape = self._get_shape(node, 0) + new_shape = self._broadcast_shapes( + shape, get_shape_from_sympy_shape(expand_to_shape)) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, new_shape)) + + def _infer_Gather(self, node): + data_shape = self._get_shape(node, 0) + axis = handle_negative_axis( + get_attribute(node, 'axis', 0), len(data_shape)) + indices_shape = self._get_shape(node, 1) + #if indices_shape == []: + # value = self._get_initializer_value(node, 1) + # if isinstance(value.tolist(), int): + # indices_shape = [1] + new_shape = data_shape[:axis] + indices_shape + data_shape[axis + 1:] + #print(new_shape) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[ + 0], vi.type.tensor_type.elem_type, new_shape)) + if node.input[0] in self.sympy_data_: + assert 0 == get_attribute(node, 'axis', + 0) # only handle 1D sympy compute + idx = self._get_value(node, 1) + data = self.sympy_data_[node.input[0]] + if type(data) == list: + if type(idx) == np.ndarray and len(idx.shape) == 1: + self.sympy_data_[node.output[0]] = [ + data[int(i)] for i in idx + ] + else: + self.sympy_data_[node.output[0]] = data[int(idx)] + else: + assert idx == 0 + self.sympy_data_[node.output[0]] = data + + def _infer_GatherElements(self, node): + indices_shape = self._get_shape(node, 1) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, indices_shape)) + + def _infer_GatherND(self, node): + data_shape = self._get_shape(node, 0) + data_rank = len(data_shape) + indices_shape = self._get_shape(node, 1) + indices_rank = len(indices_shape) + last_index_dimension = indices_shape[-1] + assert is_literal( + last_index_dimension) and last_index_dimension <= data_rank + new_shape = indices_shape[:-1] + data_shape[last_index_dimension:] + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, new_shape)) + + def _infer_If(self, node): + # special case for constant condition, in case there are mismatching shape from the non-executed branch + subgraphs = [ + get_attribute(node, 'then_branch'), + get_attribute(node, 'else_branch') + ] + cond = self._try_get_value(node, 0) + if cond is not None: + if cond > 0: + subgraphs[1].CopyFrom(subgraphs[0]) + else: + subgraphs[0].CopyFrom(subgraphs[1]) + + for i_sub, subgraph in enumerate(subgraphs): + subgraph_infer = self._onnx_infer_subgraph( + node, subgraph, use_node_input=False) + for i_out in range(len(node.output)): + vi = self.known_vi_[node.output[i_out]] + if i_sub == 0: + vi.CopyFrom(subgraph.output[i_out]) + vi.name = node.output[i_out] + else: + assert all([ + d1 == d2 + for d1, d2 in zip(vi.type.tensor_type.shape.dim, + subgraph.output[ + i_out].type.tensor_type.shape.dim) + ]) + # pass on sympy data from subgraph, if cond is constant + if cond is not None and i_sub == (0 if cond > 0 else 1): + if subgraph.output[ + i_out].name in subgraph_infer.sympy_data_: + self.sympy_data_[vi.name] = subgraph_infer.sympy_data_[ + subgraph.output[i_out].name] + + def _infer_Loop(self, node): + subgraph = get_attribute(node, 'body') + assert len(subgraph.input) == len(node.input) + for i, si in enumerate(subgraph.input): + subgraph_name = si.name + si.CopyFrom(self.known_vi_[node.input[i]]) + si.name = subgraph_name + self._onnx_infer_subgraph(node, subgraph) + # create a new symbolic dimension for iteration dependent dimension + loop_iter_dim = self._new_symbolic_dim_from_output(node) + num_loop_carried = len(node.input) - 2 + for i in range(len(node.output)): + vi = self.known_vi_[node.output[i]] + vi.CopyFrom( + subgraph.output[i + 1] + ) # first subgraph output is condition, not in node output + if i >= num_loop_carried: + subgraph_vi_dim = subgraph.output[i + + 1].type.tensor_type.shape.dim + vi.type.tensor_type.shape.ClearField('dim') + vi_dim = vi.type.tensor_type.shape.dim + vi_dim.add().dim_param = loop_iter_dim + vi_dim.extend(list(subgraph_vi_dim)) + vi.name = node.output[i] + + def _infer_MatMul(self, node): + self._compute_matmul_shape(node) + + def _infer_MatMulInteger(self, node): + self._compute_matmul_shape(node, onnx.TensorProto.INT32) + + def _infer_NonMaxSuppression(self, node): + selected = self._new_symbolic_dim_from_output(node) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[ + 0], onnx.TensorProto.INT64, [selected, 3])) + + def _infer_NonZero(self, node): + input_rank = self._get_shape_rank(node, 0) + # create a new symbolic dimension for NonZero output + nz_len = self._new_symbolic_dim_from_output(node, 0, 1) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[ + 0], vi.type.tensor_type.elem_type, [input_rank, nz_len])) + + def _infer_OneHot(self, node): + shape = self._get_shape(node, 0) + axis = get_attribute(node, 'axis', -1) + axis = handle_negative_axis(axis, len(shape) + 1) + new_shape = shape[:axis] + [self._new_symbolic_dim_from_output(node) + ] + shape[axis:] + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[2]].type.tensor_type.elem_type, new_shape)) + + def _infer_Pad(self, node): + if get_opset(self.out_mp_) <= 10: + pads = get_attribute(node, 'pads') + else: + pads = self._try_get_value(node, 1) + + vi = self.known_vi_[node.output[0]] + output_shape = get_shape_from_type_proto(vi.type) + if len(output_shape) == 0 or None in output_shape: + sympy_shape = self._get_sympy_shape(node, 0) + rank = len(sympy_shape) + if pads is not None: + assert len(pads) == 2 * rank + new_sympy_shape = [ + d + pad_up + pad_down + for d, pad_up, pad_down in zip(sympy_shape, pads[:rank], + pads[rank:]) + ] + self._update_computed_dims(new_sympy_shape) + else: + # dynamic pads, create new symbolic dimensions + new_sympy_shape = self._new_symbolic_shape(rank, node) + output_tp = self.known_vi_[node.input[0]].type.tensor_type.elem_type + vi.CopyFrom( + helper.make_tensor_value_info(node.output[ + 0], output_tp, get_shape_from_sympy_shape(new_sympy_shape))) + + def _infer_Pool(self, node): + sympy_shape = self._compute_conv_pool_shape(node) + self._update_computed_dims(sympy_shape) + for o in node.output: + if not o: + continue + vi = self.known_vi_[o] + vi.CopyFrom( + helper.make_tensor_value_info(o, vi.type.tensor_type.elem_type, + get_shape_from_sympy_shape( + sympy_shape))) + + def _infer_Range(self, node): + vi = self.known_vi_[node.output[0]] + input_data = self._get_int_values(node) + if all([i is not None for i in input_data]): + start = as_scalar(input_data[0]) + limit = as_scalar(input_data[1]) + delta = as_scalar(input_data[2]) + new_sympy_shape = [ + sympy.Max(sympy.ceiling((limit - start) / delta), 0) + ] + else: + new_dim = self._new_symbolic_dim_from_output(node) + new_sympy_shape = [self.symbolic_dims_[new_dim]] + self._update_computed_dims(new_sympy_shape) + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], self.known_vi_[node.input[0]].type.tensor_type. + elem_type, get_shape_from_sympy_shape(new_sympy_shape))) + + def _infer_ReduceProd(self, node): + axes = get_attribute(node, 'axes') + keep_dims = get_attribute(node, 'keepdims') + if keep_dims == 0 and axes == [0]: + data = self._get_int_values(node)[0] + if data is not None: + self.sympy_data_[node.output[0]] = sympy_reduce_product(data) + + def _infer_Reshape(self, node): + shape_value = self._try_get_value(node, 1) + vi = self.known_vi_[node.output[0]] + if shape_value is None: + shape_shape = self._get_shape(node, 1) + assert len(shape_shape) == 1 + shape_rank = shape_shape[0] + assert is_literal(shape_rank) + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], vi.type.tensor_type.elem_type, + get_shape_from_sympy_shape( + self._new_symbolic_shape(shape_rank, node)))) + else: + input_shape = self._get_shape(node, 0) + input_sympy_shape = self._get_sympy_shape(node, 0) + total = int(1) + for d in input_sympy_shape: + total = total * d + new_sympy_shape = [] + deferred_dim_idx = -1 + non_deferred_size = int(1) + for i, d in enumerate(shape_value): + if type(d) == sympy.Symbol: + new_sympy_shape.append(d) + elif d == 0: + new_sympy_shape.append(input_sympy_shape[i]) + non_deferred_size = non_deferred_size * input_sympy_shape[i] + else: + new_sympy_shape.append(d) + if d == -1: + deferred_dim_idx = i + elif d != 0: + non_deferred_size = non_deferred_size * d + + assert new_sympy_shape.count(-1) < 2 + if -1 in new_sympy_shape: + new_dim = total // non_deferred_size + new_sympy_shape[deferred_dim_idx] = new_dim + self._update_computed_dims(new_sympy_shape) + + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], vi.type.tensor_type.elem_type, + get_shape_from_sympy_shape(new_sympy_shape))) + + self._pass_on_sympy_data(node) + + def _infer_Resize(self, node): + vi = self.known_vi_[node.output[0]] + input_sympy_shape = self._get_sympy_shape(node, 0) + if get_opset(self.out_mp_) <= 10: + scales = self._try_get_value(node, 1) + if scales is not None: + new_sympy_shape = [ + sympy.simplify(sympy.floor(d * s)) + for d, s in zip(input_sympy_shape, scales) + ] + self._update_computed_dims(new_sympy_shape) + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], self.known_vi_[node.input[ + 0]].type.tensor_type.elem_type, + get_shape_from_sympy_shape(new_sympy_shape))) + else: + roi = self._try_get_value(node, 1) + scales = self._try_get_value(node, 2) + sizes = self._try_get_value(node, 3) + if sizes is not None: + new_sympy_shape = [ + sympy.simplify(sympy.floor(s)) for s in sizes + ] + self._update_computed_dims(new_sympy_shape) + elif roi is not None and scales is not None: + rank = len(scales) + assert len(roi) == 2 * rank + roi_start = list(roi)[:rank] + roi_end = list(roi)[rank:] + scales = list(scales) + new_sympy_shape = [ + sympy.simplify(sympy.floor(d * (end - start) * scale)) + for d, start, end, scale in zip(input_sympy_shape, + roi_start, roi_end, scales) + ] + self._update_computed_dims(new_sympy_shape) + else: + new_sympy_shape = self._new_symbolic_shape( + self._get_shape_rank(node, 0), node) + + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, + get_shape_from_sympy_shape( + new_sympy_shape))) + + def _infer_Scan(self, node): + subgraph = get_attribute(node, 'body') + num_scan_inputs = get_attribute(node, 'num_scan_inputs') + scan_input_axes = get_attribute(node, 'scan_input_axes', + [0] * num_scan_inputs) + num_scan_states = len(node.input) - num_scan_inputs + scan_input_axes = [ + handle_negative_axis( + ax, self._get_shape_rank(node, i + num_scan_states)) + for i, ax in enumerate(scan_input_axes) + ] + # We may have cases where the subgraph has optionial inputs that appear in both subgraph's input and initializer, + # but not in the node's input. In such cases, the input model might be invalid, but let's skip those optional inputs. + assert len(subgraph.input) >= len(node.input) + subgraph_inputs = subgraph.input[:len(node.input)] + for i, si in enumerate(subgraph_inputs): + subgraph_name = si.name + si.CopyFrom(self.known_vi_[node.input[i]]) + if i >= num_scan_states: + scan_input_dim = si.type.tensor_type.shape.dim + scan_input_dim.remove(scan_input_dim[scan_input_axes[ + i - num_scan_states]]) + si.name = subgraph_name + self._onnx_infer_subgraph(node, subgraph) + num_scan_outputs = len(node.output) - num_scan_states + scan_output_axes = get_attribute(node, 'scan_output_axes', + [0] * num_scan_outputs) + scan_input_dim = get_shape_from_type_proto(self.known_vi_[node.input[ + -1]].type)[scan_input_axes[-1]] + for i, o in enumerate(node.output): + vi = self.known_vi_[o] + if i >= num_scan_states: + shape = get_shape_from_type_proto(subgraph.output[i].type) + new_dim = handle_negative_axis( + scan_output_axes[i - num_scan_states], len(shape) + 1) + shape = shape[:new_dim] + [scan_input_dim] + shape[new_dim:] + vi.CopyFrom( + helper.make_tensor_value_info(o, subgraph.output[ + i].type.tensor_type.elem_type, shape)) + else: + vi.CopyFrom(subgraph.output[i]) + vi.name = o + + def _infer_ScatterElements(self, node): + data_shape = self._get_shape(node, 0) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[0], self.known_vi_[ + node.input[0]].type.tensor_type.elem_type, data_shape)) + + def _infer_Shape(self, node): + self.sympy_data_[node.output[0]] = self._get_sympy_shape(node, 0) + + def _infer_Size(self, node): + sympy_shape = self._get_sympy_shape(node, 0) + self.sympy_data_[node.output[0]] = sympy_reduce_product(sympy_shape) + self.known_vi_[node.output[0]].CopyFrom( + helper.make_tensor_value_info(node.output[0], + onnx.TensorProto.INT64, [])) + + def _infer_Slice(self, node): + if get_opset(self.out_mp_) <= 9: + axes = get_attribute(node, 'axes') + starts = get_attribute(node, 'starts') + ends = get_attribute(node, 'ends') + steps = [1] * len(axes) + else: + starts = as_list(self._try_get_value(node, 1), keep_none=True) + ends = as_list(self._try_get_value(node, 2), keep_none=True) + axes = self._try_get_value(node, 3) + steps = self._try_get_value(node, 4) + if axes is None and not (starts is None and ends is None): + axes = list( + range(0, len(starts if starts is not None else ends))) + if steps is None and not (starts is None and ends is None): + steps = [1] * len(starts if starts is not None else ends) + axes = as_list(axes, keep_none=True) + steps = as_list(steps, keep_none=True) + + new_sympy_shape = self._get_sympy_shape(node, 0) + if starts is None or ends is None: + if axes is None: + for i in range(len(new_sympy_shape)): + new_sympy_shape[i] = self._new_symbolic_dim_from_output( + node, 0, i) + else: + new_sympy_shape = get_shape_from_sympy_shape(new_sympy_shape) + for i in axes: + new_sympy_shape[i] = self._new_symbolic_dim_from_output( + node, 0, i) + else: + for i, s, e, t in zip(axes, starts, ends, steps): + idx = handle_negative_axis(i, len(new_sympy_shape)) + if is_literal(e): + if e >= self.int_max_: + e = new_sympy_shape[i] + elif e <= -self.int_max_: + e = 0 if s > 0 else -1 + elif is_literal(new_sympy_shape[i]): + if e < 0: + e = e + new_sympy_shape[i] + e = min(e, new_sympy_shape[i]) + else: + if e > 0: + e = sympy.Min( + e, new_sympy_shape[i] + ) if e > 1 else e #special case for slicing first to make computation easier + else: + e = new_sympy_shape[i] + e + else: + if is_literal(new_sympy_shape[i]): + e = sympy.Min(e, new_sympy_shape[i]) + else: + try: + if e >= new_sympy_shape[i]: + e = new_sympy_shape[i] + except Exception: + print( + 'Unable to determine if {} <= {}, treat as equal' + .format(e, new_sympy_shape[i])) + e = new_sympy_shape[i] + + if is_literal(s) and int(s) < 0: + s = new_sympy_shape[i] + s + + new_sympy_shape[idx] = (e - s + t + (-1 if t > 0 else 1)) // t + + self._update_computed_dims(new_sympy_shape) + + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], vi.type.tensor_type.elem_type, + get_shape_from_sympy_shape(new_sympy_shape))) + + # handle sympy_data if needed, for slice in shape computation + if node.input[0] in self.sympy_data_: + assert [0] == axes + assert len(starts) == 1 + assert len(ends) == 1 + self.sympy_data_[node.output[0]] = self.sympy_data_[node.input[0]][ + starts[0]:ends[0]] + + def _infer_Split(self, node): + input_sympy_shape = self._get_sympy_shape(node, 0) + axis = handle_negative_axis( + get_attribute(node, 'axis', 0), len(input_sympy_shape)) + split = get_attribute(node, 'split') + if not split: + num_outputs = len(node.output) + split = [input_sympy_shape[axis] / + sympy.Integer(num_outputs)] * num_outputs + self._update_computed_dims(split) + else: + split = [sympy.Integer(s) for s in split] + + for i_o in range(len(split)): + vi = self.known_vi_[node.output[i_o]] + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[i_o], self.known_vi_[node.input[ + 0]].type.tensor_type.elem_type, + get_shape_from_sympy_shape(input_sympy_shape[:axis] + [ + split[i_o] + ] + input_sympy_shape[axis + 1:]))) + self.known_vi_[vi.name] = vi + + def _infer_Squeeze(self, node): + self._pass_on_sympy_data(node) + + def _infer_Tile(self, node): + repeats_value = self._get_value(node, 1) + input_sympy_shape = self._get_sympy_shape(node, 0) + new_sympy_shape = [] + for i, d in enumerate(input_sympy_shape): + new_dim = d * repeats_value[i] + new_sympy_shape.append(new_dim) + self._update_computed_dims(new_sympy_shape) + vi = self.known_vi_[node.output[0]] + vi.CopyFrom( + helper.make_tensor_value_info( + node.output[0], vi.type.tensor_type.elem_type, + get_shape_from_sympy_shape(new_sympy_shape))) + + def _infer_TopK(self, node): + rank = self._get_shape_rank(node, 0) + axis = handle_negative_axis(get_attribute(node, 'axis', -1), rank) + new_shape = self._get_shape(node, 0) + + if get_opset(self.out_mp_) <= 9: + k = get_attribute(node, 'k') + else: + k = self._get_int_values(node)[1] + + if k == None: + k = self._new_symbolic_dim_from_output(node) + else: + k = as_scalar(k) + + if type(k) in [int, str]: + new_shape[axis] = k + else: + new_sympy_shape = self._get_sympy_shape(node, 0) + new_sympy_shape[axis] = k + self._update_computed_dims( + new_sympy_shape + ) # note that TopK dim could be computed in sympy_data, so need to update computed_dims when it enters shape + new_shape = get_shape_from_sympy_shape(new_sympy_shape) + + for i_o in range(len(node.output)): + vi = self.known_vi_[node.output[i_o]] + vi.CopyFrom( + helper.make_tensor_value_info(node.output[ + i_o], vi.type.tensor_type.elem_type, new_shape)) + + def _infer_Unsqueeze(self, node): + self._pass_on_sympy_data(node) + + def _infer_ZipMap(self, node): + map_key_type = None + if get_attribute(node, 'classlabels_int64s') is not None: + map_key_type = onnx.TensorProto.INT64 + elif get_attribute(node, 'classlabels_strings') is not None: + map_key_type = onnx.TensorProto.STRING + + assert map_key_type is not None + new_vi = onnx.ValueInfoProto() + new_vi.name = node.output[0] + new_vi.type.sequence_type.elem_type.map_type.value_type.tensor_type.elem_type = onnx.TensorProto.FLOAT + new_vi.type.sequence_type.elem_type.map_type.key_type = map_key_type + vi = self.known_vi_[node.output[0]] + vi.CopyFrom(new_vi) + + def _infer_impl(self, in_mp, start_sympy_data={}): + self.sympy_data_ = start_sympy_data + self.out_mp_.graph.ClearField('value_info') + self._apply_suggested_merge(graph_input_only=True) + self.input_symbols_ = set() + for i in self.out_mp_.graph.input: + input_dims = i.type.tensor_type.shape.dim + for i_dim in range(len(input_dims)): + if get_dim_from_type_proto(input_dims[i_dim]) is None: + # some models use None for symbolic dim in input, replace it with a string + input_dims[i_dim].dim_param = self._new_symbolic_dim(i.name, + i_dim) + self.input_symbols_.update([ + d for d in get_shape_from_type_proto(i.type) if type(d) == str + ]) + + for s in self.input_symbols_: + if s in self.suggested_merge_: + s_merge = self.suggested_merge_[s] + assert s_merge in self.symbolic_dims_ + self.symbolic_dims_[s] = self.symbolic_dims_[s_merge] + else: + self.symbolic_dims_[s] = sympy.Symbol(s, integer=True) + + # create a temporary ModelProto for single node inference + # note that we remove initializer to have faster inference + # for tensor ops like Reshape/Tile/Expand that read initializer, we need to do sympy computation based inference anyways + self.tmp_mp_ = onnx.ModelProto() + self.tmp_mp_.CopyFrom(self.out_mp_) + self.tmp_mp_.graph.ClearField('initializer') + + for node in self.out_mp_.graph.node: + assert all([i in self.known_vi_ for i in node.input if i]) + self._onnx_infer_single_node(node) + if node.op_type in self.dispatcher_: + self.dispatcher_[node.op_type](node) + if self.verbose_ > 2: + print(node.op_type + ': ' + node.name) + for i, name in enumerate(node.input): + print(' Input {}: {} {}€5€5€5€5€5'.format( + i, name, 'initializer' + if name in self.initializers_ else '')) + + # onnx automatically merge dims with value, i.e. Mul(['aaa', 'bbb'], [1000, 1]) -> [1000, 'bbb'] + # symbolic shape inference needs to apply merge of 'aaa' -> 1000 in this case + if node.op_type in [ + 'Add', 'Sub', 'Mul', 'Div', 'MatMul', 'MatMulInteger', + 'MatMulInteger16', 'Where', 'Sum' + ]: + vi = self.known_vi_[node.output[0]] + out_rank = len(get_shape_from_type_proto(vi.type)) + in_shapes = [ + self._get_shape(node, i) for i in range(len(node.input)) + ] + for d in range(out_rank - (2 if node.op_type in [ + 'MatMul', 'MatMulInteger', 'MatMulInteger16' + ] else 0)): + in_dims = [ + s[len(s) - out_rank + d] for s in in_shapes + if len(s) + d >= out_rank + ] + if len(in_dims) > 1: + self._check_merged_dims(in_dims, allow_broadcast=True) + for i_o in range(len(node.output)): + vi = self.known_vi_[node.output[i_o]] + out_type = vi.type + out_type_kind = out_type.WhichOneof('value') + # only TensorProto and SparseTensorProto have shape + if out_type_kind != 'tensor_type' and out_type_kind != 'sparse_tensor_type': + continue + out_shape = get_shape_from_type_proto(vi.type) + out_type_undefined = out_type.tensor_type.elem_type == onnx.TensorProto.UNDEFINED + if self.verbose_ > 2: + print(' {}: {} {}'.format(node.output[ + i_o], str(out_shape), vi.type.tensor_type.elem_type)) + if node.output[i_o] in self.sympy_data_: + print(' Sympy Data: ' + str(self.sympy_data_[ + node.output[i_o]])) + + if None in out_shape or out_type_undefined: + if self.auto_merge_: + if node.op_type in [ + 'Add', 'Sub', 'Mul', 'Div', 'MatMul', + 'MatMulInteger', 'MatMulInteger16', 'Concat', + 'Where', 'Sum' + ]: + shapes = [ + self._get_shape(node, i) + for i in range(len(node.input)) + ] + if node.op_type in [ + 'MatMul', 'MatMulInteger', 'MatMulInteger16' + ]: + # only support auto merge for MatMul for dim < rank-2 when rank > 2 + assert len(shapes[0]) > 2 and dim_idx[0] < len( + shapes[0]) - 2 + assert len(shapes[1]) > 2 and dim_idx[1] < len( + shapes[1]) - 2 + elif node.op_type == 'Expand': + # auto merge for cases like Expand([min(batch, 1), min(seq, 512)], [batch, seq]) + shapes = [ + self._get_shape(node, 0), + self._get_value(node, 1) + ] + else: + shapes = [] + + if shapes: + for idx in range(len(out_shape)): + if out_shape[idx] is not None: + continue + dim_idx = [ + len(s) - len(out_shape) + idx + for s in shapes + ] + assert all([d >= 0 for d in dim_idx]) + self._add_suggested_merge([ + s[i] if is_literal(s[i]) else str(s[i]) + for s, i in zip(shapes, dim_idx) + ]) + self.run_ = True + else: + self.run_ = False + else: + self.run_ = False + + # create new dynamic dims for ops not handled by symbolic shape inference + if self.run_ == False and not node.op_type in self.dispatcher_: + is_unknown_op = (out_type_undefined and + len(out_shape) == 0) + if is_unknown_op: + # unknown op to ONNX, maybe from higher opset or other domain + # only guess the output rank from input 0 when using guess_output_rank option + out_rank = self._get_shape_rank( + node, 0) if self.guess_output_rank_ else -1 + else: + # valid ONNX op, but not handled by symbolic shape inference, just assign dynamic shape + out_rank = len(out_shape) + + if out_rank >= 0: + new_shape = self._new_symbolic_shape(out_rank, node, + i_o) + vi.CopyFrom( + helper.make_tensor_value_info( + vi.name, self.known_vi_[node.input[ + 0]].type.tensor_type.elem_type, + get_shape_from_sympy_shape(new_shape))) + + if self.verbose_ > 0: + if is_unknown_op: + print( + "Possible unknown op: {} node: {}, guessing {} shape" + .format(node.op_type, node.name, + vi.name)) + if self.verbose_ > 2: + print(' {}: {} {}'.format( + node.output[i_o], + str(new_shape), + vi.type.tensor_type.elem_type)) + + self.run_ = True + continue # continue the inference after guess, no need to stop as no merge is needed + + if self.verbose_ > 0 or not self.auto_merge_ or out_type_undefined: + print('Stopping at incomplete shape inference at ' + + node.op_type + ': ' + node.name) + print('node inputs:') + for i in node.input: + print(self.known_vi_[i]) + print('node outputs:') + for o in node.output: + print(self.known_vi_[o]) + if self.auto_merge_ and not out_type_undefined: + print('Merging: ' + str(self.suggested_merge_)) + return False + self.run_ = False + return True + + def _update_output_from_vi(self): + for output in self.out_mp_.graph.output: + if output.name in self.known_vi_: + tmp_output = self.known_vi_[output.name] + output.CopyFrom(tmp_output) + + @staticmethod + def infer_shapes(in_mp, + int_max=2**31 - 1, + fixed_input_shape=None, + auto_merge=True, + guess_output_rank=False, + verbose=0): + if get_opset(in_mp) < 7: + print('Only support shape inferencing models of opset 7 and above.') + return + symbolic_shape_inference = SymbolicShapeInference( + int_max, auto_merge, guess_output_rank, verbose) + all_shapes_inferred = False + symbolic_shape_inference._preprocess( + in_mp, input_shapes=fixed_input_shape) + try: + while symbolic_shape_inference.run_: + all_shapes_inferred = symbolic_shape_inference._infer_impl( + in_mp) + symbolic_shape_inference._update_output_from_vi() + if not all_shapes_inferred: + print('!' * 10) + symbolic_shape_inference.out_mp_ = shape_inference.infer_shapes( + symbolic_shape_inference.out_mp_) + #onnx.save(symbolic_shape_inference.out_mp_, 'tmp.onnx') + except: + print('Stopping at incomplete shape inference') + symbolic_shape_inference.out_mp_ = shape_inference.infer_shapes( + symbolic_shape_inference.out_mp_) + return symbolic_shape_inference.out_mp_.graph diff --git a/x2paddle/decoder/tf_decoder.py b/x2paddle/decoder/tf_decoder.py index dcc534241a06b1d70e462bfe16d3a7da5b82b2fd..e7d63aa39a009e4b8dda3b9e625a8e81a045f640 100644 --- a/x2paddle/decoder/tf_decoder.py +++ b/x2paddle/decoder/tf_decoder.py @@ -48,7 +48,7 @@ class TFGraphNode(GraphNode): @property def out_shapes(self): - if self.layer_type == "OneShotIterator": + if self.layer_type == "OneShotIterator" or self.layer_type == "IteratorV2": values = self.layer.attr["output_shapes"].list.shape else: values = self.layer.attr["_output_shapes"].list.shape @@ -68,7 +68,8 @@ class TFGraphNode(GraphNode): if dtype == 0: dtype = self.layer.attr['output_types'].list.type[0] if dtype not in self.dtype_map: - raise Exception("Dtype[{}] not in dtype_map".format(dtype)) + raise Exception("Dtype[{}] of node({}) not in dtype_map".format( + dtype, self.layer.name)) return self.dtype_map[dtype] @property @@ -88,6 +89,16 @@ class TFGraphNode(GraphNode): field = getattr(attr, attr.WhichOneof('value')) return tensor_util.MakeNdarray(field) + @property + def name(self): + multi_out_ops = ['Split', 'SplitV', 'IteratorV2'] + if self.layer_type in multi_out_ops: + if self.layer_name.count(':') > 0: + return self.layer_name.replace(':', '_p') + else: + return "{}_p0".format(self.layer_name) + return self.layer_name + def get_attr(self, name): if name not in self.layer.attr: return None @@ -114,16 +125,20 @@ class TFGraph(Graph): def __init__(self, model, data_format="NHWC"): super(TFGraph, self).__init__(model) self.identity_map = dict() - self.multi_out_ops = ['Split', 'SplitV'] + self.multi_out_ops = ['Split', 'SplitV', 'IteratorV2'] self.tf_data_format = data_format def build(self): for layer in self.model.node: + if layer.op == 'Assert': + continue self.node_map[layer.name.replace('/', '_').replace( '-', '_')] = TFGraphNode( layer, data_format=self.tf_data_format) for layer_name, node in self.node_map.items(): + if node.layer_type == 'Const': + continue for in_node in node.layer.input: in_node = in_node.replace('/', '_').replace('-', '_').replace( '^', '') @@ -139,6 +154,14 @@ class TFGraph(Graph): super(TFGraph, self).build() + for layer in self.model.node: + if layer.op == 'Assert': + for ipt in layer.input: + ipt_name = ipt.replace('-', '_').replace('/', '_') + if ipt_name in self.output_nodes: + idx = self.output_nodes.index(ipt_name) + del self.output_nodes[idx] + # tensorflow graph optimize self._remove_isolated_node() self._optimize_dialiation_conv() @@ -272,7 +295,6 @@ class TFGraph(Graph): def data_format_propagation(self, node): current_node = self.node_map[node.layer_name] - current_node = node.tf_data_format outputs = current_node.outputs if len(outputs) == 0: return @@ -322,7 +344,7 @@ class TFDecoder(object): graph_def = cp.deepcopy(graph_def) input_map = dict() for layer in graph_def.node: - if layer.op != "Placeholder" and layer.op != "OneShotIterator": + if layer.op != "Placeholder" and layer.op != "OneShotIterator" and layer.op != "IteratorV2": continue graph_node = TFGraphNode(layer) dtype = graph_node.layer.attr['dtype'].type @@ -403,7 +425,7 @@ class TFDecoder(object): else: value = graph_node.layer.attr["shape"].shape shape = [dim.size for dim in value.dim] - self.input_info[graph_node.layer_name] = (shape, dtype) + self.input_info[layer.name] = (shape, dtype) return input_map diff --git a/x2paddle/op_mapper/paddle_custom_layer/__init__.py b/x2paddle/op_mapper/onnx2paddle/__init__.py similarity index 100% rename from x2paddle/op_mapper/paddle_custom_layer/__init__.py rename to x2paddle/op_mapper/onnx2paddle/__init__.py diff --git a/x2paddle/op_mapper/onnx2paddle/onnx_op_mapper.py b/x2paddle/op_mapper/onnx2paddle/onnx_op_mapper.py new file mode 100644 index 0000000000000000000000000000000000000000..bbb960899f79e95a5b4276e6a179acc8a7a02b5a --- /dev/null +++ b/x2paddle/op_mapper/onnx2paddle/onnx_op_mapper.py @@ -0,0 +1,91 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from x2paddle.op_mapper.onnx2paddle.opset9 import OpSet9, custom_layers +from x2paddle.core.op_mapper import OpMapper +from x2paddle.decoder.onnx_decoder import ONNXGraph, ONNXGraphNode, ONNXGraphDataNode + + +class ONNXOpMapper(OpMapper): + def __init__(self, decoder): + super(ONNXOpMapper, self).__init__() + self.support_op_sets = [9, ] + self.default_op_set = 9 + self.graph = decoder.graph + self.opset = self.create_opset(decoder) + if not self.op_checker(): + raise Exception("Model are not supported yet.") + #mapping op + print("Total nodes: {}".format( + sum([ + isinstance(node, ONNXGraphNode) + for name, node in self.graph.node_map.items() + ]))) + + print("Nodes converting ...") + for node_name in self.graph.topo_sort: + node = self.graph.get_node(node_name) + op = node.layer_type + if hasattr(self.opset, op): + func = getattr(self.opset, op) + func(node) + elif op in self.opset.default_op_mapping: + self.opset.directly_map(node) + elif op in custom_layers: + self.opset.deal_custom_layer(node) + elif op in self.opset.elementwise_ops: + self.opset.elementwise_map(node) + print("Nodes converted.") + self.weights = self.opset.weights + self.omit_nodes = self.opset.omit_nodes + self.used_custom_layers = self.opset.used_custom_layers + + def op_checker(self): + unsupported_ops = set() + for node_name in self.graph.topo_sort: + node = self.graph.get_node(node_name) + op = node.layer_type + if not hasattr(self.opset, op) and \ + op not in self.opset.default_op_mapping and \ + op not in custom_layers and \ + op not in self.opset.elementwise_ops: + unsupported_ops.add(op) + if len(unsupported_ops) == 0: + return True + else: + print("There are {} ops not supported yet, list as below".format( + len(unsupported_ops))) + for op in unsupported_ops: + print(op) + return False + + def create_opset(self, decoder): + run_op_set = self.default_op_set + opset = '' + if decoder.op_set in self.support_op_sets: + opset = 'OpSet' + str(decoder.op_set) + elif decoder.op_set < self.default_op_set: + opset = 'OpSet' + str(self.default_op_set) + else: + for op_set in self.support_op_sets: + if decoder.op_set > op_set: + run_op_set = op_set + else: + break + opset = 'OpSet' + str(run_op_set) + print( + 'Now, onnx2paddle support convert onnx model opset_verison {},' + 'opset_verison of your onnx model is {}, automatically treated as op_set: {}.' + .format(self.support_op_sets, decoder.op_set, run_op_set)) + return eval(opset)(decoder) diff --git a/x2paddle/op_mapper/onnx2paddle/opset9/__init__.py b/x2paddle/op_mapper/onnx2paddle/opset9/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..8563f99e1be3f04916bcc1c02caf68ffbd6ad5a1 --- /dev/null +++ b/x2paddle/op_mapper/onnx2paddle/opset9/__init__.py @@ -0,0 +1,2 @@ +from .opset import OpSet9 +from .custom_layer import custom_layers diff --git a/x2paddle/op_mapper/onnx_custom_layer/__init__.py b/x2paddle/op_mapper/onnx2paddle/opset9/custom_layer/__init__.py similarity index 97% rename from x2paddle/op_mapper/onnx_custom_layer/__init__.py rename to x2paddle/op_mapper/onnx2paddle/opset9/custom_layer/__init__.py index 3c074350a63e9e24cf1eba441da5a420868b5bb7..24fe91780ae8d19bfa55b9d86e951570a2a044c9 100644 --- a/x2paddle/op_mapper/onnx_custom_layer/__init__.py +++ b/x2paddle/op_mapper/onnx2paddle/opset9/custom_layer/__init__.py @@ -13,10 +13,6 @@ # limitations under the License. from .register import get_registered_layers -#custom layer import begins - -from . import InstanceNormalization -#custom layer import ends custom_layers = get_registered_layers() diff --git a/x2paddle/op_mapper/onnx_custom_layer/register.py b/x2paddle/op_mapper/onnx2paddle/opset9/custom_layer/register.py similarity index 100% rename from x2paddle/op_mapper/onnx_custom_layer/register.py rename to x2paddle/op_mapper/onnx2paddle/opset9/custom_layer/register.py diff --git a/x2paddle/op_mapper/onnx_op_mapper.py b/x2paddle/op_mapper/onnx2paddle/opset9/opset.py similarity index 75% rename from x2paddle/op_mapper/onnx_op_mapper.py rename to x2paddle/op_mapper/onnx2paddle/opset9/opset.py index a50670b29f774a496dc0d6da450560848a47abed..e1ebdf23b5506bba3cbb659f598e9eab029a0cd8 100644 --- a/x2paddle/op_mapper/onnx_op_mapper.py +++ b/x2paddle/op_mapper/onnx2paddle/opset9/opset.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License" # you may not use this file except in compliance with the License. @@ -12,27 +12,23 @@ # See the License for the specific language governing permissions and # limitations under the License. +from x2paddle.decoder.onnx_decoder import ONNXGraph, ONNXGraphNode, ONNXGraphDataNode from x2paddle.core.graph import GraphNode -from x2paddle.core.op_mapper import OpMapper from x2paddle.core.fluid_code import Layer from x2paddle.core.fluid_code import FluidCode -from x2paddle.decoder.onnx_decoder import ONNXGraph, ONNXGraphNode, ONNXGraphDataNode -from x2paddle.op_mapper.onnx_directly_map import default_op_mapping_field_values -from x2paddle.op_mapper.onnx_directly_map import default_op_mapping -from x2paddle.op_mapper.onnx_directly_map import default_ioa_constraint -from x2paddle.op_mapper.onnx_custom_layer import * from x2paddle.core.util import string +from x2paddle.op_mapper.onnx2paddle.opset9.custom_layer import * +from functools import reduce import numpy as np import onnx import onnx.numpy_helper as numpy_helper from onnx.mapping import TENSOR_TYPE_TO_NP_TYPE import logging as _logging -from collections import OrderedDict as _dict +from collections import OrderedDict import math import os import shutil -from functools import reduce -import onnxruntime as rt + _logger = _logging.getLogger(__name__) @@ -52,7 +48,24 @@ def get_same_padding(in_size, kernel_size, stride): return [pad0, pad1] -class ONNXOpMapper(OpMapper): +def print_mapping_info(func): + def run_mapping(*args, **kwargs): + node = args[1] + try: + res = func(*args, **kwargs) + except: + print("convert failed node:{}, op_type is {}".format( + node.layer_name[9:], node.layer_type)) + raise + else: + #print("convert successfully node:{}, op_type is {}".format( + # node.layer_name[9:], node.layer_type)) + return res + + return run_mapping + + +class OpSet9(): elementwise_ops = { 'Add': 'elementwise_add', 'Div': 'elementwise_div', @@ -61,154 +74,87 @@ class ONNXOpMapper(OpMapper): 'Pow': 'elementwise_pow', } - def __init__(self, decoder, save_dir): - super(ONNXOpMapper, self).__init__() - self.decoder = decoder - self.graph = decoder.onnx_graph + default_op_mapping_field_values = OrderedDict() + default_op_mapping_field_values['FLUID_OP'] = '' + default_op_mapping_field_values['FLUID_INPUT_ARGS'] = None + default_op_mapping_field_values['FLUID_OUTPUT_ARGS'] = None + default_op_mapping_field_values['ATTR_MAPPING'] = dict() + default_op_mapping_field_values['DEFAULTS'] = dict() + default_op_mapping_field_values['INPUT_PERM'] = None + default_op_mapping_field_values['OUTPUT_PERM'] = None + default_op_mapping_field_values['FILL_NAME_FIELD'] = True + + default_op_mapping = { + 'Shape': ['shape', ['X'], ['Out']], + 'Clip': [ + 'clip', ['X'], ['Out'], dict(), dict( + min=(np.asarray( + [255, 255, 127, 255], dtype=np.uint8).view(np.float32)[0]), + max=(np.asarray( + [255, 255, 127, 127], dtype=np.uint8).view(np.float32)[0]), + ) + ], + 'Erf': ['erf', ['X'], ['Out']], + 'Ceil': ['ceil', ['X'], ['Out']], + 'ReduceMean': [ + 'reduce_mean', ['X'], ['Out'], dict( + axes='dim', keepdims='keep_dim'), dict(keep_dim=1) + ], + 'ReduceSum': [ + 'reduce_sum', ['X'], ['Out'], dict( + axes='dim', keepdims='keep_dim'), dict(keep_dim=1) + ], + 'ReduceMin': [ + 'reduce_min', ['X'], ['Out'], dict( + axes='dim', keepdims='keep_dim'), dict(keep_dim=1) + ], + 'ReduceMax': [ + 'reduce_max', ['X'], ['Out'], dict( + axes='dim', keepdims='keep_dim'), dict(keep_dim=1) + ], + #active function + 'Relu': ['relu', ['X'], ['Out']], + 'LeakyRelu': ['leaky_relu', ['X'], ['Out'], dict(), dict(alpha=.01)], + 'Elu': ['elu', ['X'], ['Out'], dict(), dict(alpha=1.)], + 'ThresholdedRelu': [ + 'thresholded_relu', ['X'], ['Out'], dict(alpha='threshold'), + dict(alpha=1.) + ], + 'Tanh': ['tanh', ['X'], ['Out']], + 'Sigmoid': ['sigmoid', ['X'], ['Out']], + 'HardSigmoid': [ + 'hard_sigmoid', ['X'], ['Out'], dict( + alpha='slope', beta='offset'), dict( + slope=.2, offset=.5) + ], + 'Softsign': ['softsign', ['X'], ['Out']], + 'Softplus': ['softplus', ['X'], ['Out']], + 'Exp': ['exp', ['X'], ['Out']], + 'Softmax': ['softmax', ['X'], ['Out'], dict(), dict(axis=1)], + 'Sqrt': ['sqrt', ['X'], ['Out']], + 'Floor': ['floor', ['X'], ['Out']], + 'Abs': ['abs', ['X'], ['Out']], + } + + default_ioa_constraint = {} + + def __init__(self, decoder): + super(OpSet9, self).__init__() + self.graph = decoder.graph self.input_shapes = [] self.weights = dict() self.omit_nodes = list() self.used_custom_layers = dict() - self.is_inference = False - self.tmp_data_dir = os.path.join(save_dir, 'tmp_data') - self.tmp_outputs_dict = {} - self.get_output_shapes() - - if not self.op_checker(): - raise Exception("Model are not supported yet.") - - #mapping op - print("Total nodes: {}".format( - sum([ - isinstance(node, ONNXGraphNode) - for name, node in self.graph.node_map.items() - ]))) - for node_name in self.graph.topo_sort: - node = self.graph.get_node(node_name) - op = node.layer_type - if hasattr(self, op): - func = getattr(self, op) - func(node) - elif op in default_op_mapping: - self.directly_map(node) - elif op in custom_layers: - self.deal_custom_layer(node) - elif op in self.elementwise_ops: - self.elementwise_map(node) - - self.remove_tmp_data() - - def op_checker(self): - unsupported_ops = set() - for node_name in self.graph.topo_sort: - node = self.graph.get_node(node_name) - op = node.layer_type - if not hasattr(self, op) and \ - op not in default_op_mapping and \ - op not in custom_layers and \ - op not in self.elementwise_ops: - unsupported_ops.add(op) - if len(unsupported_ops) == 0: - return True - else: - print("There are {} ops not supported yet, list as below".format( - len(unsupported_ops))) - for op in unsupported_ops: - print(op) - return False - - def get_results_of_inference(self, model, value_infos, data_nodes): - if not os.path.exists(self.tmp_data_dir): - os.makedirs(self.tmp_data_dir) - inputs_dict = {} - for data_node in data_nodes: - value_info = value_infos[data_node] - shape = value_info['shape'] - for i, dim_shape in enumerate(shape): - if dim_shape == 0 and i == 0: - shape[i] = 1 - if dim_shape == 0 and i != 0: - assert 'shape of input is not assigned' - ipt = np.random.random(shape).astype(value_info['dtype']) - inputs_dict[data_node] = ipt - - model = onnx.shape_inference.infer_shapes(model) - outputs = [] - - for value_info in model.graph.value_info: - outputs.append(value_info.name) - - model.graph.ClearField('output') - model.graph.output.MergeFrom(model.graph.value_info) - onnx.save(model, - os.path.join(self.tmp_data_dir, 'onnx_model_infer.onnx')) - sess = rt.InferenceSession( - os.path.join(self.tmp_data_dir, 'onnx_model_infer.onnx')) - res = sess.run(None, input_feed=inputs_dict) - self.tmp_outputs_dict = dict(zip(outputs, res)) - - return - - def get_dynamic_shape(self, layer): - """ - get dynamic shape from infer_result - """ - if layer not in self.tmp_outputs_dict: - return [None, None, None] - output = self.tmp_outputs_dict[layer] - return output.tolist(), output.dtype, output.shape - - def get_output_shapes(self): - """ - build topo_sort of ONNX model - """ - nodes = self.decoder.model.graph.node - node_map = self.decoder.onnx_graph.node_map - value_infos = self.decoder.onnx_graph.value_infos - onnx_model = self.decoder.model - for layer in nodes: - node = node_map[layer.name] - for opt in layer.output: - if opt in value_infos: - value_info = value_infos[opt] - if len(value_info['shape']) == 0 or value_info[ - 'dtype'] is None or 0 in value_info['shape']: - if self.is_inference == False: - self.get_results_of_inference( - onnx_model, value_infos, - self.decoder.onnx_graph.place_holder_nodes) - self.is_inference = True - _, dtype, shape = self.get_dynamic_shape(opt) - node.out_shapes.append(shape) - node.dtype = dtype - else: - node.dtype = value_info['dtype'] - node.out_shapes.append(value_info['shape']) - else: - if self.is_inference == False: - self.get_results_of_inference( - onnx_model, value_infos, - self.decoder.onnx_graph.place_holder_nodes) - self.is_inference = True - _, dtype, shape = self.get_dynamic_shape(opt) - node.dtype = dtype - node.out_shapes.append(shape) - - def remove_tmp_data(self): - """ - remove temporarily generated file - """ - if os.path.exists(self.tmp_data_dir): - import shutil - shutil.rmtree(self.tmp_data_dir) + @print_mapping_info def directly_map(self, node, name='', *args, **kwargs): inputs = node.layer.input outputs = node.layer.output op_type = node.layer_type attrs = node.attr_map - info = default_op_mapping[op_type] - info.extend(list(default_op_mapping_field_values.values())[len(info):]) + info = self.default_op_mapping[op_type] + info.extend( + list(self.default_op_mapping_field_values.values())[len(info):]) ( fluid_op, fluid_input_args, @@ -219,8 +165,8 @@ class ONNXOpMapper(OpMapper): output_perm, fill_name_field, ) = info - if fluid_op in default_ioa_constraint: - for predicate, message in default_ioa_constraint[fluid_op]: + if fluid_op in self.default_ioa_constraint: + for predicate, message in self.default_ioa_constraint[fluid_op]: assert predicate(inputs, outputs, attrs), message mapped_attrs = { @@ -243,11 +189,18 @@ class ONNXOpMapper(OpMapper): map(lambda i: outputs[i], output_perm)) attr = fluid_attrs assert len(val_inps) == 1, 'directly_map error with multi inputs' - if fluid_op not in ['shape']: + if fluid_op not in ['shape', 'erf']: attr['name'] = string(node.layer_name) node.fluid_code.add_layer( fluid_op, inputs=val_inps[0], output=val_outs[0], param_attr=attr) + if fluid_op in ['shape']: + node.fluid_code.add_layer( + 'cast', + inputs=val_outs[0], + output=val_outs[0], + param_attr={'dtype': string('int64')}) + @print_mapping_info def deal_custom_layer(self, node): op = node.layer_type custom_code, func = make_custom_layer(node) @@ -268,6 +221,7 @@ class ONNXOpMapper(OpMapper): self.used_custom_layers[op + '_child_func'] = child_func_code + @print_mapping_info def elementwise_map(self, node): assert node.layer_type in self.elementwise_ops op_type = self.elementwise_ops[node.layer_type] @@ -279,6 +233,7 @@ class ONNXOpMapper(OpMapper): if len(val_x_shape) < len(val_y_shape): val_x, val_y = val_y, val_x + val_y_shape, val_x_shape = val_x_shape, val_y_shape str_y_shape = ','.join(str(e) for e in val_y_shape) str_x_shape = ','.join(str(e) for e in val_x_shape) @@ -310,6 +265,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( op_type, inputs=inputs, output=node, param_attr=attr) + @print_mapping_info def place_holder(self, node): self.input_shapes.append(node.out_shapes[0]) @@ -329,6 +285,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( "data", inputs=None, output=node, param_attr=attr) + @print_mapping_info def create_parameter(self, node, parameter=None): if parameter is not None: node = parameter @@ -341,11 +298,24 @@ class ONNXOpMapper(OpMapper): 'dtype': string(dtype), 'shape': shape, 'name': string(node.layer_name), - 'attr': string(node.layer_name), 'default_initializer': 'Constant(0.0)' } - node.fluid_code.add_layer( - "create_parameter", inputs=None, output=node, param_attr=attr) + if dtype == 'bool': + attr['dtype'] = string('int64') + node.fluid_code.add_layer( + "create_parameter", inputs=None, output=node, param_attr=attr) + node.fluid_code.add_layer( + "cast", + inputs=node, + output=node, + param_attr={'dtype': string('bool')}) + elif dtype == 'uint8': + attr['dtype'] = string('float32') + node.fluid_code.add_layer( + "create_parameter", inputs=None, output=node, param_attr=attr) + else: + node.fluid_code.add_layer( + "create_parameter", inputs=None, output=node, param_attr=attr) def _pad_if_asymmetric(self, node, pads, val_name): # pads: SSEE assert len(pads) & 1 == 0 @@ -362,41 +332,13 @@ class ONNXOpMapper(OpMapper): def _interpolate(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) - val_scales = self.graph.get_input_node(node, idx=1, copy=True) - val_y = self.graph.get_node(node.layer.output[0], copy=True) - out_shape = val_y.out_shapes[0] - if out_shape is not None: - assert len(out_shape) == 4, 'only 4-D Tensor as X and Y supported' - out_shape = out_shape[2:] - - scales = _const_weight_or_none(val_scales) + if node.layer_type == 'Resize': + val_scales = self.graph.get_input_node(node, idx=2, copy=True) + elif node.layer_type == 'Upsample': + val_scales = self.graph.get_input_node(node, idx=1, copy=True) - if isinstance(val_scales, ONNXGraphNode): - scales, _, _ = self.get_dynamic_shape(val_scales.layer_name) attr = {'name': string(node.layer_name)} - use_scales = True - if scales is not None: - try: - assert len(scales) == 4, 'only 4-D Tensor as X and Y supported' - assert scales[0] == 1 and scales[ - 1] == 1, 'only scale on (NC)HW supported' - assert scales[2] == scales[ - 3], 'only aspect-ratio-invariant scale supported' - except: - use_scales = False - scale = scales[2] if scales else None - if scale is None: - assert out_shape, 'neither scales nor output shape is available' - else: - if out_shape is None: - in_shape = val_x.out_shapes[0] - assert in_shape is not None, 'out_shape required but not inferrable' - assert len( - in_shape) == 4, 'only 4-D Tensor as X and Y supported' - out_shape = [in_shape[2] * scale, in_shape[3] * scale] - mode = node.get_attr('mode', 'nearest') - fluid_op = 'resize_{}'.format(mode) if 'linear' in mode: print( @@ -404,14 +346,14 @@ class ONNXOpMapper(OpMapper): ) fluid_op = 'resize_bilinear' - if use_scales and scale is not None: - attr['scale'] = scale - else: - attr['out_shape'] = out_shape - node.fluid_code.add_layer( - fluid_op, inputs=val_x, output=node, param_attr=attr) + fluid_op, + inputs={'input': val_x, + 'scale': val_scales}, + output=node, + param_attr=attr) + @print_mapping_info def RoiAlign(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_rois = self.graph.get_input_node(node, idx=1, copy=True) @@ -433,6 +375,7 @@ class ONNXOpMapper(OpMapper): output=node, param_attr=attr) + @print_mapping_info def MaxRoiPool(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_rois = self.graph.get_input_node(node, idx=1, copy=True) @@ -451,6 +394,7 @@ class ONNXOpMapper(OpMapper): output=node, param_attr=attr) + @print_mapping_info def Pad(self, node, op_independent=True): val_x = self.graph.get_input_node(node, idx=0, copy=True) pads = node.get_attr('pads') @@ -497,17 +441,23 @@ class ONNXOpMapper(OpMapper): param_attr=attr) return node.layer_name + '_paded' + @print_mapping_info def Unsqueeze(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) axes = node.get_attr('axes') + attr = {'axes': axes, 'name': string(node.layer_name)} if len(val_x.out_shapes[0]) == 0: - node.fluid_code.add_layer( - 'assign', inputs=val_x, output=node, param_attr=None) + if node.layer_name: + node.fluid_code.add_layer( + 'reshape', + inputs=val_x, + output=node, + param_attr={'shape': [1]}) else: - attr = {'axes': axes, 'name': string(node.layer_name)} node.fluid_code.add_layer( 'unsqueeze', inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def Shrink(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) bias = node.get_attr('bias') @@ -527,6 +477,7 @@ class ONNXOpMapper(OpMapper): output=node, param_attr=None) + @print_mapping_info def Constant(self, node): val_output = self.graph.get_node(node.layer.output[0], copy=True) @@ -557,24 +508,42 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( 'fill_constant', inputs=None, output=node, param_attr=attr) else: + if dtype.name == 'uint8': + dtype = 'int64' value = np.reshape(value, shape) self.weights[node.layer_name] = value attr = { 'dtype': string(dtype), 'shape': shape, 'name': string(node.layer_name), - 'attr': string(node.layer_name), 'default_initializer': 'Constant(0.0)' } node.fluid_code.add_layer( "create_parameter", inputs=None, output=node, param_attr=attr) + @print_mapping_info def Resize(self, node): self._interpolate(node) + @print_mapping_info def Upsample(self, node): self._interpolate(node) + @print_mapping_info + def InstanceNormalization(self, node): + val_x = self.graph.get_input_node(node, idx=0, copy=True) + val_scale = self.graph.get_input_node(node, idx=1, copy=True) + val_b = self.graph.get_input_node(node, idx=2, copy=True) + epsilon = node.get_attr('epsilon', 1e-5) + attr = { + 'epsilon': epsilon, + 'param_attr': string(val_scale.layer_name), + 'bias_attr': string(val_b.layer_name) + } + node.fluid_code.add_layer( + "instance_norm", inputs=val_x, output=node, param_attr=attr) + + @print_mapping_info def Expand(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_shape = self.graph.get_input_node(node, idx=1, copy=True) @@ -598,13 +567,14 @@ class ONNXOpMapper(OpMapper): output=node.layer_name, param_attr=attr) + @print_mapping_info def Gather(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) indices = self.graph.get_input_node(node, idx=1, copy=True) indices_shape = indices.out_shapes[0] axis = node.get_attr('axis', 0) - assert len( - indices_shape) <= 2, "Gather op don't support dim of indice >2 " + #assert len( + # indices_shape) <= 2, "Gather op don't support dim of indice >2 " if axis == 0 and len(indices_shape) <= 1: node.fluid_code.add_layer( 'gather', @@ -630,13 +600,55 @@ class ONNXOpMapper(OpMapper): param_attr=None) node.fluid_code.add_layer( 'transpose', inputs=node, output=node, param_attr=attr_trans) - elif len(indices_shape) > 1: + elif axis == 0 and len(indices_shape) > 1: + if val_x.out_shapes[0] is not None and isinstance( + val_x, ONNXGraphDataNode): + node.fluid_code.add_layer( + 'embedding', + inputs=indices, + output=node, + use_fluid=True, + param_attr={ + 'param_attr': string(val_x.layer_name), + 'size': val_x.out_shapes[0] + }) + else: + from functools import reduce + #indices_shape = [1,7] + reshape_shape = reduce(lambda x, y: x * y, indices_shape) + indices_reshape = indices.layer_name + '_shape' + node.fluid_code.add_layer( + 'reshape', + inputs=indices, + output=indices_reshape, + param_attr={'shape': [reshape_shape, ]}) + + perm = list(range(len(val_x.out_shapes[0]))) + node.fluid_code.add_layer( + 'gather', + inputs={'input': val_x, + 'index': indices_reshape}, + output=node, + param_attr=None) + val_x_shape = val_x.out_shapes[0] + reshaped_shape = [] + for i in perm: + reshaped_shape.append(indices_shape[i]) + for i in val_x_shape[:axis] + val_x_shape[axis + 1:]: + reshaped_shape.append(i) + node.fluid_code.add_layer( + 'reshape', + inputs=node, + output=node, + param_attr={'shape': reshaped_shape}) + elif axis > 0 and len(indices_shape) > 1: from functools import reduce reshape_shape = reduce(lambda x, y: x * y, indices_shape) + indices_reshape = indices.layer_name + '_shape' node.fluid_code.add_layer( 'reshape', inputs=indices, - output=indices, + output=indices_reshape, param_attr={'shape': [reshape_shape, ]}) perm = list(range(len(val_x.out_shapes[0]))) @@ -651,7 +663,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( 'gather', inputs={'input': name_trans, - 'index': indices}, + 'index': indices_reshape}, output=node, param_attr=None) node.fluid_code.add_layer( @@ -668,69 +680,102 @@ class ONNXOpMapper(OpMapper): output=node, param_attr={'shape': reshaped_shape}) + @print_mapping_info + def Range(self, node): + val_start = self.graph.get_input_node(node, idx=0, copy=True) + val_limit = self.graph.get_input_node(node, idx=1, copy=True) + val_delta = self.graph.get_input_node(node, idx=2, copy=True) + dtype = val_start.dtype + inputs = {'start': val_start, 'end': val_limit, 'step': val_delta} + node.fluid_code.add_layer( + 'range', + inputs=inputs, + output=node, + param_attr={'dtype': string(dtype)}) + + @print_mapping_info def Slice(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) starts, ends, axes, steps = None, None, None, None + attr = {} if len(node.inputs) > 1: starts = self.graph.get_input_node(node, idx=1, copy=True) ends = self.graph.get_input_node(node, idx=2, copy=True) if len(node.inputs) > 3: axes = self.graph.get_input_node(node, idx=3, copy=True) - self.omit_nodes.append(axes.layer_name) axes = _const_weight_or_none(axes) if len(node.inputs) > 4: steps = self.graph.get_input_node(node, idx=4, copy=True) - self.omit_nodes.append(steps.layer_name) steps = _const_weight_or_none(steps) - - self.omit_nodes.append(starts.layer_name) - self.omit_nodes.append(ends.layer_name) - starts = _const_weight_or_none(starts).copy() - ends = _const_weight_or_none(ends).copy() + if steps is not None: + assert steps == 1, "Only support convert op:Slice, which attribute:steps == 1" + attr = { + "axes": axes, + "starts": starts.layer_name, + "ends": ends.layer_name + } + starts_value = _const_weight_or_none(starts) + ends_value = _const_weight_or_none(ends) + if starts_value is not None and ends_value is not None: + self.omit_nodes.append(starts.layer_name) + self.omit_nodes.append(ends.layer_name) + ends_value = ends_value.copy() + for idx in range(len(ends_value)): + if ends_value[idx] > 2**31 - 1: + ends_value[idx] = 2**31 - 1 + attr = { + "axes": axes, + "starts": starts_value, + "ends": ends_value + } + else: + if starts.dtype != 'int32': + node.fluid_code.add_layer( + 'cast', + inputs=starts, + output=starts, + param_attr={'dtype': string('int32')}) + if ends.dtype != 'int32': + node.fluid_code.add_layer( + 'cast', + inputs=ends, + output=ends, + param_attr={'dtype': string('int32')}) else: starts = node.get_attr('starts') ends = node.get_attr('ends') axes = node.get_attr('axes') + for idx in range(len(ends)): + if ends[idx] > 2**31 - 1: + ends[idx] = 2**31 - 1 + attr = {"axes": axes, "starts": starts, "ends": ends} - val_y = self.graph.get_node(node.layer.output[0], copy=True) - - shape = val_x.out_shapes[0] - - if shape is not None: - for idx, value in enumerate(starts): - if value > shape[axes[idx]]: - starts[idx] = shape[axes[idx]] - for idx, value in enumerate(ends): - if value > shape[axes[idx]]: - ends[idx] = shape[axes[idx]] - attr = {"axes": axes, "starts": starts, "ends": ends} node.fluid_code.add_layer( 'slice', inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def ConstantOfShape(self, node): val_shape = self.graph.get_input_node(node, idx=0, copy=True) val_y = self.graph.get_node(node.layer.output[0], copy=True) - shape = _const_weight_or_none(val_shape) - - if shape is None: - shape = node.out_shapes[0] - - assert shape is not None, ( - 'given shape is neither const value nor deductible from output, ' - 'this is not supported') value = node.get_attr('value') dtype = value.dtype value = value.tolist() + assert len(value) == 1, ('given value not Scalar, shape of value > 1, ' + 'this is not supported') if len(value) == 1: - shape = [1] value = value[0] if dtype.name == 'int64': dtype = 'int32' - attr = {'shape': shape, 'dtype': string(dtype), 'value': value} + attr = { + 'shape': val_shape.layer_name, + 'dtype': string(dtype), + 'value': value + } node.fluid_code.add_layer( 'fill_constant', inputs=None, output=node, param_attr=attr) + @print_mapping_info def Split(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_y = self.graph.get_node(node.layer.output[0], copy=True) @@ -747,46 +792,53 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( 'split', inputs=val_x, output=val_y, param_attr=attr) + @print_mapping_info def Reshape(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_shape = self.graph.get_input_node(node, idx=1, copy=True) val_reshaped = self.graph.get_node(node.layer.output[0], copy=True) - shape = None - - if isinstance(val_shape, ONNXGraphDataNode): - self.omit_nodes.append(val_shape.layer_name) - - attr = {'name': string(node.layer_name)} - # catch dynamic graph shape - if isinstance(val_shape, ONNXGraphNode): - shape, _, _ = self.get_dynamic_shape(val_shape.layer_name) - if val_shape.dtype == 'int64': - val_shape_cast = val_shape.layer_name + '_cast' - node.fluid_code.add_layer( - 'cast', - inputs=val_shape, - output=val_shape_cast, - param_attr={'dtype': string('int32')}) - - attr['actual_shape'] = val_shape_cast - else: - attr['actual_shape'] = val_shape - - if shape is None: - shape = val_reshaped.out_shapes[0] + attr = {} + shape_value = _const_weight_or_none(val_shape) + shape_dims = len(val_shape.out_shapes[0]) - if shape is None: - shape = [1, -1] - _logger.warning( - 'in %s(%s -> Reshape -> %s): ' - 'input "shape" not inferred, use [1, -1] as dummy value, ' - 'the behavior of Paddle fluid maybe undefined', node.layer_name, - val_x.layer_name, val_reshaped.layer_name) - - attr['shape'] = shape - node.fluid_code.add_layer( - 'reshape', inputs=val_x, output=node, param_attr=attr) + if shape_value is not None: + node.fluid_code.add_layer( + 'reshape', + inputs={'x': val_x}, + output=node, + param_attr={'shape': shape_value.tolist()}) + elif val_shape.dtype == 'int64': + val_shape_cast = val_shape.layer_name + '_cast' + node.fluid_code.add_layer( + 'cast', + inputs=val_shape, + output=val_shape_cast, + param_attr={'dtype': string('int32')}) + node.fluid_code.add_layer( + 'reshape', + inputs=val_shape_cast, + output=val_shape_cast, + param_attr={'shape': val_shape.out_shapes[0]}) + node.fluid_code.add_layer( + 'reshape', + inputs={'x': val_x, + 'shape': val_shape_cast}, + output=node, + param_attr=attr) + else: + node.fluid_code.add_layer( + 'reshape', + inputs=val_shape, + output=val_shape, + param_attr={'shape': val_shape.out_shapes[0]}) + node.fluid_code.add_layer( + 'reshape', + inputs={'x': val_x, + 'shape': val_shape}, + output=node, + param_attr=attr) + @print_mapping_info def Cast(self, node): val_input = self.graph.get_input_node(node, idx=0, copy=True) val_output = self.graph.get_node(node.layer.output[0], copy=True) @@ -802,6 +854,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( 'cast', inputs=val_input, output=node, param_attr=attr) + @print_mapping_info def AveragePool(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) @@ -838,6 +891,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( fluid_op, inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def Concat(self, node): inputs = [] for i in range(len(node.layer.input)): @@ -851,6 +905,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( 'concat', inputs=inputs, output=node, param_attr=attr) + @print_mapping_info def Flatten(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) axis = node.get_attr('axis', 1) @@ -858,6 +913,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( 'flatten', inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def Gemm(self, node): val_a = self.graph.get_input_node(node, idx=0, copy=True) val_b = self.graph.get_input_node(node, idx=1, copy=True) @@ -907,6 +963,7 @@ class ONNXOpMapper(OpMapper): output=node, param_attr=attr) + @print_mapping_info def Sum(self, node): val_inps = node.layer.input inputs = { @@ -926,6 +983,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( "elementwise_add", inputs=inputs, output=node) + @print_mapping_info def MatMul(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_y = self.graph.get_input_node(node, idx=1, copy=True) @@ -934,6 +992,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( "matmul", inputs=inputs, output=node, param_attr=attr) + @print_mapping_info def BatchNormalization(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_scale = self.graph.get_input_node(node, idx=1, copy=True) @@ -966,6 +1025,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( "batch_norm", inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def Transpose(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) perm = node.get_attr('perm') @@ -973,12 +1033,14 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( "transpose", inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def Relu(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) attr = {"name": string(node.layer_name)} node.fluid_code.add_layer( "relu", inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def PRelu(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_slope = self.graph.get_input_node(node, idx=1, copy=True) @@ -996,13 +1058,22 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( "prelu", inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def Squeeze(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) axes = node.get_attr('axes') attr = {'axes': axes, "name": string(node.layer_name)} - node.fluid_code.add_layer( - "squeeze", inputs=val_x, output=node, param_attr=attr) + if len(val_x.out_shapes[0]) == 1: + node.fluid_code.add_layer( + "cast", + inputs=val_x, + output=node, + param_attr={'dtype': string(val_x.dtype)}) + else: + node.fluid_code.add_layer( + "squeeze", inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def Equal(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_y = self.graph.get_input_node(node, idx=1, copy=True) @@ -1013,6 +1084,18 @@ class ONNXOpMapper(OpMapper): output=node, param_attr=None) + @print_mapping_info + def Greater(self, node): + val_x = self.graph.get_input_node(node, idx=0, copy=True) + val_y = self.graph.get_input_node(node, idx=1, copy=True) + node.fluid_code.add_layer( + "greater_than", + inputs={'x': val_x, + 'y': val_y}, + output=node, + param_attr=None) + + @print_mapping_info def Where(self, node): condition = self.graph.get_input_node(node, idx=0, copy=True) val_x = self.graph.get_input_node(node, idx=1, copy=True) @@ -1059,45 +1142,42 @@ class ONNXOpMapper(OpMapper): output=node, param_attr=None) + @print_mapping_info def NonZero(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) - where_name = node.layer_name + '_where' - node.fluid_code.add_layer( - "where", inputs=val_x.layer_name + '!=0', output=where_name) - dims = len(val_x.out_shapes[0]) - elements_count_val_x = reduce(lambda x, y: x * y, val_x.out_shapes[0]) - flatten_names = [] - for dim in range(dims): - slice_name = node.layer_name + '_slice' + str(dim) - flatten_name = node.layer_name + '_flatten' + str(dim) - flatten_names.append(flatten_name) - attr = { - 'axes': list(range(dims)), - 'starts': [0, dim], - 'ends': [elements_count_val_x, dim + 1] - } + val_x_dim = len(val_x.out_shapes[0]) + print(val_x.layer_name, val_x.out_shapes[0]) + if val_x_dim == 1: + node.fluid_code.add_layer("nonzero", inputs=val_x, output=val_x) node.fluid_code.add_layer( - "slice", inputs=where_name, output=slice_name, param_attr=attr) + "transpose", + inputs=val_x, + output=node, + param_attr={'perm': [1, 0]}) + if val_x_dim > 1: + node.fluid_code.add_layer("nonzero", inputs=val_x, output=val_x) node.fluid_code.add_layer( - "flatten", - inputs=slice_name, - output=flatten_name, - param_attr={'axis': 0}) - node.fluid_code.add_layer( - "concat", inputs=flatten_names, output=node, - param_attr={'axis': 0}) + "split", + inputs=val_x, + output=val_x, + param_attr={'num_or_sections': 1, + 'dim': val_x_dim}) + node.fluid_code.add_layer("concat", inputs=val_x, output=node) + @print_mapping_info def Identity(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) node.fluid_code.add_layer("assign", inputs=val_x, output=node) + @print_mapping_info def Tile(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_repeats = self.graph.get_input_node(node, idx=1, copy=True) repeats = _const_weight_or_none(val_repeats) - assert repeats is not None, 'for OP:Tile, only const repeats supported' - if isinstance(repeats, int): + if repeats is None: + repeats = val_repeats.layer_name + elif isinstance(repeats, int): repeats = [repeats] attr = { @@ -1107,9 +1187,9 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( "expand", inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def MaxPool(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) - auto_pad = node.get_attr('auto_pad', 'NOTSET') assert node.get_attr( "dilations") is None, 'only dilations = 0 is supported' # optional @@ -1148,16 +1228,7 @@ class ONNXOpMapper(OpMapper): def _global_pool(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_y = self.graph.get_node(node.layer.output[0], copy=True) - input_shape = val_x.out_shapes[0] - output_shape = val_y.out_shapes[0] - assert input_shape is not None or output_shape is not None, 'poolnd not inferred' # N - if input_shape: - poolnd = len(input_shape) - 2 # NC... - elif output_shape: - poolnd = len(output_shape) - 2 # NC... - assert 2 <= poolnd <= 3, 'only pool2d and pool3d is supported' - fluid_op = 'pool{}d'.format(poolnd) - + fluid_op = 'pool2d' pool_type = None if node.layer.op_type == 'GlobalMaxPool': pool_type = 'max' @@ -1172,12 +1243,15 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( fluid_op, inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def GlobalMaxPool(self, node): self._global_pool(node) + @print_mapping_info def GlobalAveragePool(self, node): self._global_pool(node) + @print_mapping_info def Conv(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_w = self.graph.get_input_node(node, idx=1, copy=True) @@ -1229,6 +1303,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( fluid_op, inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def ConvTranspose(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_w = self.graph.get_input_node(node, idx=1, copy=True) @@ -1280,6 +1355,7 @@ class ONNXOpMapper(OpMapper): node.fluid_code.add_layer( fluid_op, inputs=val_x, output=node, param_attr=attr) + @print_mapping_info def GRU(self, node): val_x = self.graph.get_input_node(node, idx=0, copy=True) val_w = self.graph.get_input_node(node, idx=1, copy=True) @@ -1303,8 +1379,6 @@ class ONNXOpMapper(OpMapper): val_xh = self.graph.get_input_node( node, idx=5 - miss_arg_num, copy=True) - data, dtype, shape = self.get_dynamic_shape(val_x.layer_name) - x_shape = val_x.out_shapes[0] assert x_shape[1] == 1, 'only X with batch_size = 1 supported' @@ -1394,8 +1468,8 @@ class ONNXOpMapper(OpMapper): inputs=val_b, output=var_bi + ',' + var_bh, param_attr={ - 'axis': 1, - 'split': [hidden_size * 3, hidden_size * 3], + 'dim': 1, + 'num_or_sections': [hidden_size * 3, hidden_size * 3], 'name': string(node.layer_name + '.b/split') }) var_bi0 = node.layer_name + '_bi0' @@ -1407,7 +1481,7 @@ class ONNXOpMapper(OpMapper): 'name': string(var_bi0)}) node.fluid_code.add_layer( - 'elmentwise_add', + 'elementwise_add', inputs=[var_mm, var_bi0], output=var_fc, param_attr={ diff --git a/x2paddle/op_mapper/onnx_custom_layer/InstanceNormalization.py b/x2paddle/op_mapper/onnx_custom_layer/InstanceNormalization.py deleted file mode 100644 index f47273206d3108316e5a3a8a72da5a1f96f8fe9e..0000000000000000000000000000000000000000 --- a/x2paddle/op_mapper/onnx_custom_layer/InstanceNormalization.py +++ /dev/null @@ -1,56 +0,0 @@ -# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License" -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from .register import register - - -def InstanceNormalization_shape(input_shape): - return input_shape - - -def InstanceNormalization_layer(inputs, name=None): - # TODO(lvmengsi@baidu.com): Check the accuracy when using fluid.layers.layer_norm. - epsilon = 1e-5 - input_ = inputs[0] - mean = fluid.layers.reduce_mean(input_, dim=[2, 3], keep_dim=True) - var = fluid.layers.reduce_mean( - fluid.layers.square(input_ - mean), dim=[2, 3], keep_dim=True) - if name is not None: - scale_name = name + "_scale" - offset_name = name + "_offset" - - scale_param = inputs[1] - offset_param = inputs[2] - scale = fluid.layers.create_parameter( - name=scale_param.name, shape=input_.shape[1:2], dtype="float32") - offset = fluid.layers.create_parameter( - name=offset_param.name, shape=input_.shape[1:2], dtype="float32") - - tmp = fluid.layers.elementwise_mul(x=(input_ - mean), y=scale, axis=1) - tmp = tmp / fluid.layers.sqrt(var + epsilon) - tmp = fluid.layers.elementwise_add(tmp, offset, axis=1) - return tmp - - -def InstanceNormalization_weights(name, data=None): - weights_name = [name + '_scale'] - return weights_name - - -register( - kind='InstanceNormalization', - shape=InstanceNormalization_shape, - layer=InstanceNormalization_layer, - child_func=None, - weights=InstanceNormalization_weights) diff --git a/x2paddle/op_mapper/onnx_directly_map.py b/x2paddle/op_mapper/onnx_directly_map.py deleted file mode 100644 index a1765f064153dca3e698857065a23b6650254f33..0000000000000000000000000000000000000000 --- a/x2paddle/op_mapper/onnx_directly_map.py +++ /dev/null @@ -1,82 +0,0 @@ -# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License" -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from collections import OrderedDict as _dict -import numpy as _np - -default_op_mapping_field_values = _dict() -default_op_mapping_field_values['FLUID_OP'] = '' -default_op_mapping_field_values['FLUID_INPUT_ARGS'] = None -default_op_mapping_field_values['FLUID_OUTPUT_ARGS'] = None -default_op_mapping_field_values['ATTR_MAPPING'] = dict() -default_op_mapping_field_values['DEFAULTS'] = dict() -default_op_mapping_field_values['INPUT_PERM'] = None -default_op_mapping_field_values['OUTPUT_PERM'] = None -default_op_mapping_field_values['FILL_NAME_FIELD'] = True - -default_op_mapping = { - 'Shape': ['shape', ['X'], ['Out']], - 'Clip': [ - 'clip', ['X'], ['Out'], dict(), dict( - min=(_np.asarray( - [255, 255, 127, 255], dtype=_np.uint8).view(_np.float32)[0]), - max=(_np.asarray( - [255, 255, 127, 127], dtype=_np.uint8).view(_np.float32)[0]), ) - ], - 'Erf': ['erf', ['X'], ['Out']], - 'Ceil': ['ceil', ['X'], ['Out']], - 'ReduceMean': [ - 'reduce_mean', ['X'], ['Out'], dict( - axes='dim', keepdims='keep_dim'), dict(keep_dim=1) - ], - 'ReduceSum': [ - 'reduce_sum', ['X'], ['Out'], dict( - axes='dim', keepdims='keep_dim'), dict(keep_dim=1) - ], - 'ReduceMin': [ - 'reduce_min', ['X'], ['Out'], dict( - axes='dim', keepdims='keep_dim'), dict(keep_dim=1) - ], - 'ReduceMax': [ - 'reduce_max', ['X'], ['Out'], dict( - axes='dim', keepdims='keep_dim'), dict(keep_dim=1) - ], - #active function - 'Relu': ['relu', ['X'], ['Out']], - 'LeakyRelu': ['leaky_relu', ['X'], ['Out'], dict(), dict(alpha=.01)], - 'Elu': ['elu', ['X'], ['Out'], dict(), dict(alpha=1.)], - 'ThresholdedRelu': [ - 'thresholded_relu', ['X'], ['Out'], dict(alpha='threshold'), - dict(alpha=1.) - ], - 'Tanh': ['tanh', ['X'], ['Out']], - 'Sigmoid': ['sigmoid', ['X'], ['Out']], - 'HardSigmoid': [ - 'hard_sigmoid', ['X'], ['Out'], dict( - alpha='slope', beta='offset'), dict( - slope=.2, offset=.5) - ], - 'Softsign': ['softsign', ['X'], ['Out']], - 'Softplus': ['softplus', ['X'], ['Out']], - 'Exp': ['exp', ['X'], ['Out']], - 'Softmax': ['softmax', ['X'], ['Out'], dict(), dict(axis=1)], - 'Sqrt': ['sqrt', ['X'], ['Out']], - 'Floor': ['floor', ['X'], ['Out']], - 'Abs': ['abs', ['X'], ['Out']], -} - -default_ioa_constraint = { - 'Gather': [(lambda i, o, a: a.get('axis', 0) == 0, - 'only axis = 0 is supported')], -} diff --git a/x2paddle/op_mapper/paddle2onnx/__init__.py b/x2paddle/op_mapper/paddle2onnx/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/x2paddle/op_mapper/paddle2onnx/opset10/__init__.py b/x2paddle/op_mapper/paddle2onnx/opset10/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/x2paddle/op_mapper/paddle2onnx/opset10/opset.py b/x2paddle/op_mapper/paddle2onnx/opset10/opset.py new file mode 100644 index 0000000000000000000000000000000000000000..72925ddec33c922f5cc6de9140a926552b02a36b --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset10/opset.py @@ -0,0 +1,61 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math +import sys +import x2paddle +import os +import numpy as np +import paddle.fluid.core as core +import paddle.fluid as fluid +import onnx +from onnx import helper, onnx_pb +from x2paddle.op_mapper.paddle2onnx.opset9.opset import OpSet9 + + +class OpSet10(OpSet9): + def __init__(self): + super(OpSet10, self).__init__() + + def slice(self, op, block): + axes = op.attr('axes') + starts = op.attr('starts') + ends = op.attr('ends') + axes_name = self.get_name(op.type, 'axes') + starts_name = self.get_name(op.type, 'starts') + ends_name = self.get_name(op.type, 'ends') + + axes_node = self.make_constant_node(axes_name, + onnx_pb.TensorProto.INT64, axes) + starts_node = self.make_constant_node(starts_name, + onnx_pb.TensorProto.INT64, starts) + ends_node = self.make_constant_node(ends_name, + onnx_pb.TensorProto.INT64, ends) + node = helper.make_node( + "Slice", + inputs=[op.input('Input')[0], starts_name, ends_name, axes_name], + outputs=op.output('Out'), ) + return [starts_node, ends_node, axes_node, node] + + def im2sequence(self, op, block): + from .paddle_custom_layer.im2sequence import im2sequence + return im2sequence(op, block) + + def yolo_box(self, op, block): + from .paddle_custom_layer.yolo_box import yolo_box + return yolo_box(op, block) + + def multiclass_nms(self, op, block): + from .paddle_custom_layer.multiclass_nms import multiclass_nms + return multiclass_nms(op, block) diff --git a/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/__init__.py b/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/im2sequence.py b/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/im2sequence.py new file mode 100644 index 0000000000000000000000000000000000000000..84144ca1fdacbc18cc640370352b0ff726b45f0b --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/im2sequence.py @@ -0,0 +1,22 @@ +# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import onnx +import numpy as np +from onnx import onnx_pb, helper +from x2paddle.op_mapper.paddle2onnx.opset9.paddle_custom_layer.im2sequence import im2sequence as im2sequence9 + + +def im2sequence(op, block): + return im2sequence9(op, block) diff --git a/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/multiclass_nms.py b/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/multiclass_nms.py new file mode 100644 index 0000000000000000000000000000000000000000..ef57b76f65d41ad6c62730db84b6873b50afdee7 --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/multiclass_nms.py @@ -0,0 +1,32 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math +import sys +import os +import numpy as np +import paddle.fluid.core as core +import paddle.fluid as fluid +import onnx +import warnings +from onnx import helper, onnx_pb +from x2paddle.op_mapper.paddle2onnx.opset9.paddle_custom_layer.multiclass_nms import multiclass_nms as multiclass_nms9 + + +def multiclass_nms(op, block): + """ + Convert the paddle multiclass_nms to onnx op. + This op is get the select boxes from origin boxes. + """ + return multiclass_nms9(op, block) diff --git a/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/yolo_box.py b/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/yolo_box.py new file mode 100644 index 0000000000000000000000000000000000000000..563c5b1db2f57f99eeb1c29efad687bc5a0cfab1 --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset10/paddle_custom_layer/yolo_box.py @@ -0,0 +1,22 @@ +# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import onnx +import numpy as np +from onnx import onnx_pb, helper +from x2paddle.op_mapper.paddle2onnx.opset9.paddle_custom_layer.yolo_box import yolo_box as yolo_box9 + + +def yolo_box(op, block): + return yolo_box9(op, block) diff --git a/x2paddle/op_mapper/paddle2onnx/opset11/__init__.py b/x2paddle/op_mapper/paddle2onnx/opset11/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/x2paddle/op_mapper/paddle2onnx/opset11/opset.py b/x2paddle/op_mapper/paddle2onnx/opset11/opset.py new file mode 100644 index 0000000000000000000000000000000000000000..4ec88b5b7d8544f41f545e6758f0803f47b8ebac --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset11/opset.py @@ -0,0 +1,276 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math +import sys +import x2paddle +import os +import numpy as np +import paddle.fluid.core as core +import paddle.fluid as fluid +import onnx +from onnx import helper, onnx_pb +from x2paddle.op_mapper.paddle2onnx.opset10.opset import OpSet10 + + +class OpSet11(OpSet10): + def __init__(self): + super(OpSet11, self).__init__() + + def relu6(self, op, block): + min_name = self.get_name(op.type, 'min') + max_name = self.get_name(op.type, 'max') + min_node = self.make_constant_node(min_name, onnx_pb.TensorProto.FLOAT, + 0) + max_node = self.make_constant_node(max_name, onnx_pb.TensorProto.FLOAT, + op.attr('threshold')) + node = helper.make_node( + 'Clip', + inputs=[op.input('X')[0], min_name, max_name], + outputs=op.output('Out'), ) + return [min_node, max_node, node] + + def pad2d(self, op, block): + x_shape = block.var(op.input('X')[0]).shape + paddings = op.attr('paddings') + onnx_pads = [] + #TODO support pads is Variable + if op.attr('data_format') == 'NCHW': + pads = [ + 0, 0, paddings[0], paddings[2], 0, 0, paddings[1], paddings[3] + ] + else: + pads = [ + 0, paddings[0], paddings[2], 0, 0, paddings[1], paddings[3], 0 + ] + pads_name = self.get_name(op.type, 'pads') + pads_node = self.make_constant_node(pads_name, + onnx_pb.TensorProto.INT64, pads) + constant_value_name = self.get_name(op.type, 'constant_value') + constant_value_node = self.make_constant_node(constant_value_name, + onnx_pb.TensorProto.FLOAT, + op.attr('pad_value')) + node = helper.make_node( + 'Pad', + inputs=op.input('X') + [pads_name, constant_value_name], + outputs=op.output('Out'), + mode=op.attr('mode')) + return [pads_node, constant_value_node, node] + + def bilinear_interp(self, op, block): + input_names = op.input_names + coordinate_transformation_mode = '' + align_corners = op.attr('align_corners') + align_mode = op.attr('align_mode') + if align_corners: + coordinate_transformation_mode = 'align_corners' + elif align_mode == 1: + coordinate_transformation_mode = 'asymmetric' + else: + coordinate_transformation_mode = 'half_pixel' + + if ('OutSize' in input_names and len(op.input('OutSize')) > 0) or ( + 'SizeTensor' in input_names and + len(op.input('SizeTensor')) > 0): + node_list = list() + roi_node = self.make_constant_node( + self.get_name(op.type, 'roi'), onnx_pb.TensorProto.FLOAT, + [1, 1, 1, 1, 1, 1, 1, 1]) + roi_name = self.get_name(op.type, 'roi') + roi_node = self.make_constant_node( + roi_name, onnx_pb.TensorProto.FLOAT, [1, 1, 1, 1, 1, 1, 1, 1]) + empty_name = self.get_name(op.type, 'empty') + empty_tensor = helper.make_tensor( + empty_name, + onnx_pb.TensorProto.FLOAT, (0, ), + np.array([]).astype('float32'), + raw=False) + empty_node = helper.make_node( + 'Constant', [], outputs=[empty_name], value=empty_tensor) + shape_name0 = self.get_name(op.type, 'shape') + shape_node0 = helper.make_node( + 'Shape', inputs=op.input('X'), outputs=[shape_name0]) + starts_name = self.get_name(op.type, 'slice.starts') + starts_node = self.make_constant_node( + starts_name, onnx_pb.TensorProto.INT64, [0]) + ends_name = self.get_name(op.type, 'slice.ends') + ends_node = self.make_constant_node(ends_name, + onnx_pb.TensorProto.INT64, [2]) + shape_name1 = self.get_name(op.type, 'shape') + shape_node1 = helper.make_node( + 'Slice', + inputs=[shape_name0, starts_name, ends_name], + outputs=[shape_name1]) + node_list.extend([ + roi_node, empty_node, shape_node0, starts_node, ends_node, + shape_node1 + ]) + if 'OutSize' in input_names and len(op.input('OutSize')) > 0: + cast_shape_name = self.get_name(op.type, "shape.cast") + cast_shape_node = helper.make_node( + 'Cast', + inputs=op.input('OutSize'), + outputs=[cast_shape_name], + to=onnx_pb.TensorProto.INT64) + node_list.append(cast_shape_node) + else: + concat_shape_name = self.get_name(op.type, "shape.concat") + concat_shape_node = helper.make_node( + "Concat", + inputs=op.input('SizeTensor'), + outputs=[concat_shape_name], + axis=0) + cast_shape_name = self.get_name(op.type, "shape.cast") + cast_shape_node = helper.make_node( + 'Cast', + inputs=[concat_shape_name], + outputs=[cast_shape_name], + to=onnx_pb.TensorProto.INT64) + node_list.extend([concat_shape_node, cast_shape_node]) + shape_name3 = self.get_name(op.type, "shape.concat") + shape_node3 = helper.make_node( + 'Concat', + inputs=[shape_name1, cast_shape_name], + outputs=[shape_name3], + axis=0) + result_node = helper.make_node( + 'Resize', + inputs=[op.input('X')[0], roi_name, empty_name, shape_name3], + outputs=op.output('Out'), + mode='linear', + coordinate_transformation_mode=coordinate_transformation_mode) + node_list.extend([shape_node3, result_node]) + return node_list + elif 'Scale' in input_names and len(op.input('Scale')) > 0: + node = helper.make_node( + 'Resize', + inputs=[op.input('X')[0], op.input('Scale')[0]], + outputs=op.output('Out'), + mode='linear', + coordinate_transformation_mode=coordinate_transformation_mode) + else: + out_shape = [op.attr('out_h'), op.attr('out_w')] + scale = op.attr('scale') + if out_shape.count(-1) > 0: + scale_name = self.get_name(op.type, 'scale') + scale_node = self.make_constant_node(scale_name, + onnx_pb.TensorProto.FLOAT, + [1, 1, scale, scale]) + roi_name = self.get_name(op.type, 'roi') + roi_node = self.make_constant_node(roi_name, + onnx_pb.TensorProto.FLOAT, + [1, 1, 1, 1, 1, 1, 1, 1]) + node = helper.make_node( + 'Resize', + inputs=[op.input('X')[0], roi_name, scale_name], + outputs=op.output('Out'), + mode='nearest', + coordinate_transformation_mode=coordinate_transformation_mode + ) + return [scale_node, roi_node, node] + else: + raise Exception("Unexpected situation happend") + return node + + def nearest_interp(self, op, block): + input_names = op.input_names + coordinate_transformation_mode = '' + align_corners = op.attr('align_corners') + if align_corners: + coordinate_transformation_mode = 'align_corners' + else: + coordinate_transformation_mode = 'asymmetric' + if 'OutSize' in input_names and len(op.input('OutSize')) > 0: + node = helper.make_node( + 'Resize', + inputs=[op.input('X')[0], '', op.input('OutSize')[0]], + outputs=op.output('Out'), + mode='nearest', + coordinate_transformation_mode=coordinate_transformation_mode) + elif 'Scale' in input_names and len(op.input('Scale')) > 0: + node = helper.make_node( + 'Resize', + inputs=[op.input('X')[0], op.input('Scale')[0]], + outputs=op.output('Out'), + mode='nearest', + coordinate_transformation_mode=coordinate_transformation_mode) + else: + out_shape = [op.attr('out_h'), op.attr('out_w')] + scale = op.attr('scale') + if out_shape.count(-1) > 0: + scale_name = self.get_name(op.type, 'scale') + scale_node = self.make_constant_node(scale_name, + onnx_pb.TensorProto.FLOAT, + [1, 1, scale, scale]) + roi_name = self.get_name(op.type, 'roi') + roi_node = self.make_constant_node(roi_name, + onnx_pb.TensorProto.FLOAT, + [1, 1, 1, 1, 1, 1, 1, 1]) + node = helper.make_node( + 'Resize', + inputs=[op.input('X')[0], roi_name, scale_name], + outputs=op.output('Out'), + mode='nearest', + coordinate_transformation_mode=coordinate_transformation_mode + ) + return [scale_node, roi_node, node] + else: + raise Exception("Unexpected situation happend") + return node + + def hard_swish(self, op, block): + min_name = self.get_name(op.type, 'min') + max_name = self.get_name(op.type, 'max') + scale_name = self.get_name(op.type, 'scale') + offset_name = self.get_name(op.type, 'offset') + min_node = self.make_constant_node(min_name, onnx_pb.TensorProto.FLOAT, + 0) + max_node = self.make_constant_node(max_name, onnx_pb.TensorProto.FLOAT, + op.attr('threshold')) + scale_node = self.make_constant_node(scale_name, + onnx_pb.TensorProto.FLOAT, + op.attr('scale')) + offset_node = self.make_constant_node(offset_name, + onnx_pb.TensorProto.FLOAT, + op.attr('offset')) + + name0 = self.get_name(op.type, 'add') + node0 = helper.make_node( + 'Add', inputs=[op.input('X')[0], offset_name], outputs=[name0]) + name1 = self.get_name(op.type, 'relu') + node1 = helper.make_node( + 'Clip', + inputs=[name0, min_name, max_name], + outputs=[name1], ) + name2 = self.get_name(op.type, 'mul') + node2 = helper.make_node( + 'Mul', inputs=[op.input('X')[0], name1], outputs=[name2]) + node3 = helper.make_node( + 'Div', inputs=[name2, scale_name], outputs=op.output('Out')) + return [ + min_node, max_node, scale_node, offset_node, node0, node1, node2, + node3 + ] + + def im2sequence(self, op, block): + from .paddle_custom_layer.im2sequence import im2sequence + return im2sequence(op, block) + + def yolo_box(self, op, block): + from .paddle_custom_layer.yolo_box import yolo_box + return yolo_box(op, block) + + def multiclass_nms(self, op, block): + from .paddle_custom_layer.multiclass_nms import multiclass_nms + return multiclass_nms(op, block) diff --git a/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/__init__.py b/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/im2sequence.py b/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/im2sequence.py new file mode 100644 index 0000000000000000000000000000000000000000..be7aa3423d06c781413a87a9ca37cafb9bda5af4 --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/im2sequence.py @@ -0,0 +1,22 @@ +# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import onnx +import numpy as np +from onnx import onnx_pb, helper +from x2paddle.op_mapper.paddle2onnx.opset10.paddle_custom_layer.im2sequence import im2sequence as im2sequence10 + + +def im2sequence(op, block): + return im2sequence10(op, block) diff --git a/x2paddle/op_mapper/paddle_custom_layer/multiclass_nms.py b/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/multiclass_nms.py similarity index 92% rename from x2paddle/op_mapper/paddle_custom_layer/multiclass_nms.py rename to x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/multiclass_nms.py index 5d30f651bbe2e51a2d328a81ceb8c52391374d60..fa578cc65d5c36aec5e006394c38e4a3f3f469b6 100644 --- a/x2paddle/op_mapper/paddle_custom_layer/multiclass_nms.py +++ b/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/multiclass_nms.py @@ -125,7 +125,7 @@ def multiclass_nms(op, block): vals=[value])) node_list.append(node) - # Ine this code block, we will deocde the raw score data, reshape N * C * M to 1 * N*C*M + # In this code block, we will deocde the raw score data, reshape N * C * M to 1 * N*C*M # and the same time, decode the select indices to 1 * D, gather the select_indices outputs_gather_1 = [result_name + "@gather_1"] node_gather_1 = onnx.helper.make_node( @@ -405,12 +405,36 @@ def multiclass_nms(op, block): inputs_concat_final_results = outputs_cast_topk_class + outputs_unsqueeze_topk_scores +\ outputs_gather_select_boxes - outputs_concat_final_results = outputs['Out'] - node_concat_final_results = onnx.helper.make_node( + outputs_sort_by_socre_results = [result_name + "@concat_topk_scores"] + node_sort_by_socre_results = onnx.helper.make_node( 'Concat', inputs=inputs_concat_final_results, - outputs=outputs_concat_final_results, + outputs=outputs_sort_by_socre_results, axis=2) - node_list.append(node_concat_final_results) + node_list.append(node_sort_by_socre_results) + # select topk classes indices + outputs_squeeze_cast_topk_class = [result_name + "@squeeze_cast_topk_class"] + node_squeeze_cast_topk_class = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_cast_topk_class, + outputs=outputs_squeeze_cast_topk_class, + axes=[0, 2]) + node_list.append(node_squeeze_cast_topk_class) + outputs_topk_select_classes_indices = [result_name + "@topk_select_topk_classes_scores",\ + result_name + "@topk_select_topk_classes_indices"] + node_topk_select_topk_indices = onnx.helper.make_node( + 'TopK', + inputs=outputs_squeeze_cast_topk_class + outputs_cast_topk_indices, + outputs=outputs_topk_select_classes_indices, + largest=0) + node_list.append(node_topk_select_topk_indices) + outputs_concat_final_results = outputs['Out'] + node_concat_final_results = onnx.helper.make_node( + 'Gather', + inputs=outputs_sort_by_socre_results + + [outputs_topk_select_classes_indices[1]], + outputs=outputs_concat_final_results, + axis=1) + node_list.append(node_concat_final_results) return node_list diff --git a/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/yolo_box.py b/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/yolo_box.py new file mode 100644 index 0000000000000000000000000000000000000000..5e674d5217693227d4eec33b58f7d252296947f1 --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset11/paddle_custom_layer/yolo_box.py @@ -0,0 +1,855 @@ +# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import onnx +import numpy as np +from onnx import onnx_pb, helper + +MAX_FLOAT = np.asarray([255, 255, 127, 127], dtype=np.uint8).view(np.float32)[0] + + +def get_old_name(arg, name_prefix=''): + prefix_index = arg.find(name_prefix) + + if prefix_index != -1: + last_prefix = arg[len(name_prefix):] + else: + last_prefix = arg + idx = last_prefix.find('@') + if idx != -1: + last_prefix = last_prefix[:idx] + return name_prefix + last_prefix + + +def yolo_box(op, block): + inputs = dict() + outputs = dict() + attrs = dict() + for name in op.input_names: + inputs[name] = op.input(name) + for name in op.output_names: + outputs[name] = op.output(name) + for name in op.attr_names: + attrs[name] = op.attr(name) + model_name = outputs['Boxes'][0] + input_shape = block.vars[get_old_name(inputs['X'][0])].shape + image_size = inputs['ImgSize'] + input_height = input_shape[2] + input_width = input_shape[3] + + class_num = attrs['class_num'] + anchors = attrs['anchors'] + num_anchors = int(len(anchors)) // 2 + downsample_ratio = attrs['downsample_ratio'] + input_size = input_height * downsample_ratio + conf_thresh = attrs['conf_thresh'] + conf_thresh_mat = np.ones([num_anchors * input_height * + input_width]) * conf_thresh + + node_list = [] + im_outputs = [] + + x_shape = [1, num_anchors, 5 + class_num, input_height, input_width] + name_x_shape = [model_name + "@x_shape"] + node_x_shape = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_x_shape, + value=onnx.helper.make_tensor( + name=name_x_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[5], + vals=x_shape)) + node_list.append(node_x_shape) + + outputs_x_reshape = [model_name + "@reshape"] + node_x_reshape = onnx.helper.make_node( + 'Reshape', inputs=inputs['X'] + name_x_shape, outputs=outputs_x_reshape) + node_list.append(node_x_reshape) + + outputs_x_transpose = [model_name + "@x_transpose"] + node_x_transpose = onnx.helper.make_node( + 'Transpose', + inputs=outputs_x_reshape, + outputs=outputs_x_transpose, + perm=[0, 1, 3, 4, 2]) + node_list.append(node_x_transpose) + + range_x = [] + range_y = [] + for i in range(0, input_width): + range_x.append(i) + for j in range(0, input_height): + range_y.append(j) + + name_range_x = [model_name + "@range_x"] + node_range_x = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_range_x, + value=onnx.helper.make_tensor( + name=name_range_x[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=[input_width], + vals=range_x)) + node_list.append(node_range_x) + + name_range_y = [model_name + "@range_y"] + node_range_y = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_range_y, + value=onnx.helper.make_tensor( + name=name_range_y[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=[input_height], + vals=range_y)) + node_list.append(node_range_y) + + range_x_new_shape = [1, input_width] + range_y_new_shape = [input_height, 1] + + name_range_x_new_shape = [model_name + "@range_x_new_shape"] + node_range_x_new_shape = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_range_x_new_shape, + value=onnx.helper.make_tensor( + name=name_range_x_new_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[len(range_x_new_shape)], + vals=range_x_new_shape)) + node_list.append(node_range_x_new_shape) + + name_range_y_new_shape = [model_name + "@range_y_new_shape"] + node_range_y_new_shape = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_range_y_new_shape, + value=onnx.helper.make_tensor( + name=name_range_y_new_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[len(range_y_new_shape)], + vals=range_y_new_shape)) + node_list.append(node_range_y_new_shape) + + outputs_range_x_reshape = [model_name + "@range_x_reshape"] + node_range_x_reshape = onnx.helper.make_node( + 'Reshape', + inputs=name_range_x + name_range_x_new_shape, + outputs=outputs_range_x_reshape) + node_list.append(node_range_x_reshape) + + outputs_range_y_reshape = [model_name + "@range_y_reshape"] + node_range_y_reshape = onnx.helper.make_node( + 'Reshape', + inputs=name_range_y + name_range_y_new_shape, + outputs=outputs_range_y_reshape) + node_list.append(node_range_y_reshape) + + outputs_grid_x = [model_name + "@grid_x"] + node_grid_x = onnx.helper.make_node( + "Tile", + inputs=outputs_range_x_reshape + name_range_y_new_shape, + outputs=outputs_grid_x) + node_list.append(node_grid_x) + + outputs_grid_y = [model_name + "@grid_y"] + node_grid_y = onnx.helper.make_node( + "Tile", + inputs=outputs_range_y_reshape + name_range_x_new_shape, + outputs=outputs_grid_y) + node_list.append(node_grid_y) + + outputs_box_x = [model_name + "@box_x"] + outputs_box_y = [model_name + "@box_y"] + outputs_box_w = [model_name + "@box_w"] + outputs_box_h = [model_name + "@box_h"] + outputs_conf = [model_name + "@conf"] + outputs_prob = [model_name + "@prob"] + + node_split_input = onnx.helper.make_node( + "Split", + inputs=outputs_x_transpose, + outputs=outputs_box_x + outputs_box_y + outputs_box_w\ + + outputs_box_h + outputs_conf + outputs_prob, + axis=-1, + split=[1, 1, 1, 1, 1, class_num]) + node_list.append(node_split_input) + + outputs_box_x_sigmoid = [model_name + "@box_x_sigmoid"] + outputs_box_y_sigmoid = [model_name + "@box_y_sigmoid"] + + node_box_x_sigmoid = onnx.helper.make_node( + "Sigmoid", inputs=outputs_box_x, outputs=outputs_box_x_sigmoid) + node_list.append(node_box_x_sigmoid) + + node_box_y_sigmoid = onnx.helper.make_node( + "Sigmoid", inputs=outputs_box_y, outputs=outputs_box_y_sigmoid) + node_list.append(node_box_y_sigmoid) + + outputs_box_x_squeeze = [model_name + "@box_x_squeeze"] + outputs_box_y_squeeze = [model_name + "@box_y_squeeze"] + + node_box_x_squeeze = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_box_x_sigmoid, + outputs=outputs_box_x_squeeze, + axes=[4]) + node_list.append(node_box_x_squeeze) + + node_box_y_squeeze = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_box_y_sigmoid, + outputs=outputs_box_y_squeeze, + axes=[4]) + node_list.append(node_box_y_squeeze) + + outputs_box_x_add_grid = [model_name + "@box_x_add_grid"] + outputs_box_y_add_grid = [model_name + "@box_y_add_grid"] + + node_box_x_add_grid = onnx.helper.make_node( + "Add", + inputs=outputs_grid_x + outputs_box_x_squeeze, + outputs=outputs_box_x_add_grid) + node_list.append(node_box_x_add_grid) + + node_box_y_add_grid = onnx.helper.make_node( + "Add", + inputs=outputs_grid_y + outputs_box_y_squeeze, + outputs=outputs_box_y_add_grid) + node_list.append(node_box_y_add_grid) + + name_input_h = [model_name + "@input_h"] + name_input_w = [model_name + "@input_w"] + + node_input_h = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_input_h, + value=onnx.helper.make_tensor( + name=name_input_w[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[input_height])) + node_list.append(node_input_h) + + node_input_w = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_input_w, + value=onnx.helper.make_tensor( + name=name_input_w[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[input_width])) + node_list.append(node_input_w) + + outputs_box_x_encode = [model_name + "@box_x_encode"] + outputs_box_y_encode = [model_name + "@box_y_encode"] + + node_box_x_encode = onnx.helper.make_node( + 'Div', + inputs=outputs_box_x_add_grid + name_input_w, + outputs=outputs_box_x_encode) + node_list.append(node_box_x_encode) + + node_box_y_encode = onnx.helper.make_node( + 'Div', + inputs=outputs_box_y_add_grid + name_input_h, + outputs=outputs_box_y_encode) + node_list.append(node_box_y_encode) + + name_anchor_tensor = [model_name + "@anchor_tensor"] + node_anchor_tensor = onnx.helper.make_node( + "Constant", + inputs=[], + outputs=name_anchor_tensor, + value=onnx.helper.make_tensor( + name=name_anchor_tensor[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=[len(anchors)], + vals=anchors)) + node_list.append(node_anchor_tensor) + + anchor_shape = [int(num_anchors), 2] + name_anchor_shape = [model_name + "@anchor_shape"] + node_anchor_shape = onnx.helper.make_node( + "Constant", + inputs=[], + outputs=name_anchor_shape, + value=onnx.helper.make_tensor( + name=name_anchor_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[2], + vals=anchor_shape)) + node_list.append(node_anchor_shape) + + outputs_anchor_tensor_reshape = [model_name + "@anchor_tensor_reshape"] + node_anchor_tensor_reshape = onnx.helper.make_node( + "Reshape", + inputs=name_anchor_tensor + name_anchor_shape, + outputs=outputs_anchor_tensor_reshape) + node_list.append(node_anchor_tensor_reshape) + + name_input_size = [model_name + "@input_size"] + node_input_size = onnx.helper.make_node( + "Constant", + inputs=[], + outputs=name_input_size, + value=onnx.helper.make_tensor( + name=name_input_size[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[input_size])) + node_list.append(node_input_size) + + outputs_anchors_div_input_size = [model_name + "@anchors_div_input_size"] + node_anchors_div_input_size = onnx.helper.make_node( + "Div", + inputs=outputs_anchor_tensor_reshape + name_input_size, + outputs=outputs_anchors_div_input_size) + node_list.append(node_anchors_div_input_size) + + outputs_anchor_w = [model_name + "@anchor_w"] + outputs_anchor_h = [model_name + "@anchor_h"] + + node_anchor_split = onnx.helper.make_node( + 'Split', + inputs=outputs_anchors_div_input_size, + outputs=outputs_anchor_w + outputs_anchor_h, + axis=1, + split=[1, 1]) + node_list.append(node_anchor_split) + + new_anchor_shape = [1, int(num_anchors), 1, 1] + name_new_anchor_shape = [model_name + "@new_anchor_shape"] + node_new_anchor_shape = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_new_anchor_shape, + value=onnx.helper.make_tensor( + name=name_new_anchor_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[len(new_anchor_shape)], + vals=new_anchor_shape)) + node_list.append(node_new_anchor_shape) + + outputs_anchor_w_reshape = [model_name + "@anchor_w_reshape"] + outputs_anchor_h_reshape = [model_name + "@anchor_h_reshape"] + + node_anchor_w_reshape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_anchor_w + name_new_anchor_shape, + outputs=outputs_anchor_w_reshape) + node_list.append(node_anchor_w_reshape) + + node_anchor_h_reshape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_anchor_h + name_new_anchor_shape, + outputs=outputs_anchor_h_reshape) + node_list.append(node_anchor_h_reshape) + + outputs_box_w_squeeze = [model_name + "@box_w_squeeze"] + node_box_w_squeeze = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_box_w, + outputs=outputs_box_w_squeeze, + axes=[4]) + node_list.append(node_box_w_squeeze) + + outputs_box_h_squeeze = [model_name + "@box_h_squeeze"] + node_box_h_squeeze = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_box_h, + outputs=outputs_box_h_squeeze, + axes=[4]) + node_list.append(node_box_h_squeeze) + + outputs_box_w_exp = [model_name + "@box_w_exp"] + node_box_w_exp = onnx.helper.make_node( + "Exp", inputs=outputs_box_w_squeeze, outputs=outputs_box_w_exp) + node_list.append(node_box_w_exp) + + outputs_box_h_exp = [model_name + "@box_h_exp"] + node_box_h_exp = onnx.helper.make_node( + "Exp", inputs=outputs_box_h_squeeze, outputs=outputs_box_h_exp) + node_list.append(node_box_h_exp) + + outputs_box_w_encode = [model_name + "box_w_encode"] + outputs_box_h_encode = [model_name + "box_h_encode"] + + node_box_w_encode = onnx.helper.make_node( + 'Mul', + inputs=outputs_box_w_exp + outputs_anchor_w_reshape, + outputs=outputs_box_w_encode) + node_list.append(node_box_w_encode) + + node_box_h_encode = onnx.helper.make_node( + 'Mul', + inputs=outputs_box_h_exp + outputs_anchor_h_reshape, + outputs=outputs_box_h_encode) + node_list.append(node_box_h_encode) + + outputs_conf_sigmoid = [model_name + "@conf_sigmoid"] + node_conf_sigmoid = onnx.helper.make_node( + 'Sigmoid', inputs=outputs_conf, outputs=outputs_conf_sigmoid) + node_list.append(node_conf_sigmoid) + + name_conf_thresh = [model_name + "@conf_thresh"] + node_conf_thresh = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_conf_thresh, + value=onnx.helper.make_tensor( + name=name_conf_thresh[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=[num_anchors * input_height * input_width], + vals=conf_thresh_mat)) + node_list.append(node_conf_thresh) + + conf_shape = [1, int(num_anchors), input_height, input_width, 1] + name_conf_shape = [model_name + "@conf_shape"] + node_conf_shape = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_conf_shape, + value=onnx.helper.make_tensor( + name=name_conf_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[len(conf_shape)], + vals=conf_shape)) + node_list.append(node_conf_shape) + + outputs_conf_thresh_reshape = [model_name + "@conf_thresh_reshape"] + node_conf_thresh_reshape = onnx.helper.make_node( + 'Reshape', + inputs=name_conf_thresh + name_conf_shape, + outputs=outputs_conf_thresh_reshape) + node_list.append(node_conf_thresh_reshape) + + outputs_conf_sub = [model_name + "@conf_sub"] + node_conf_sub = onnx.helper.make_node( + 'Sub', + inputs=outputs_conf_sigmoid + outputs_conf_thresh_reshape, + outputs=outputs_conf_sub) + node_list.append(node_conf_sub) + + outputs_conf_clip = [model_name + "@conf_clip"] + node_conf_clip = onnx.helper.make_node( + 'Clip', inputs=outputs_conf_sub, outputs=outputs_conf_clip) + node_list.append(node_conf_clip) + + zeros = [0] + name_zeros = [model_name + "@zeros"] + node_zeros = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_zeros, + value=onnx.helper.make_tensor( + name=name_zeros[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=zeros)) + node_list.append(node_zeros) + + outputs_conf_clip_bool = [model_name + "@conf_clip_bool"] + node_conf_clip_bool = onnx.helper.make_node( + 'Greater', + inputs=outputs_conf_clip + name_zeros, + outputs=outputs_conf_clip_bool) + node_list.append(node_conf_clip_bool) + + outputs_conf_clip_cast = [model_name + "@conf_clip_cast"] + node_conf_clip_cast = onnx.helper.make_node( + 'Cast', + inputs=outputs_conf_clip_bool, + outputs=outputs_conf_clip_cast, + to=1) + node_list.append(node_conf_clip_cast) + + outputs_conf_set_zero = [model_name + "@conf_set_zero"] + node_conf_set_zero = onnx.helper.make_node( + 'Mul', + inputs=outputs_conf_sigmoid + outputs_conf_clip_cast, + outputs=outputs_conf_set_zero) + node_list.append(node_conf_set_zero) + + outputs_prob_sigmoid = [model_name + "@prob_sigmoid"] + node_prob_sigmoid = onnx.helper.make_node( + 'Sigmoid', inputs=outputs_prob, outputs=outputs_prob_sigmoid) + node_list.append(node_prob_sigmoid) + + new_shape = [1, int(num_anchors), input_height, input_width, 1] + name_new_shape = [model_name + "@new_shape"] + node_new_shape = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_new_shape, + value=onnx.helper.make_tensor( + name=name_new_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[len(new_shape)], + vals=new_shape)) + node_list.append(node_new_shape) + + outputs_conf_new_shape = [model_name + "@_conf_new_shape"] + node_conf_new_shape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_conf_set_zero + name_new_shape, + outputs=outputs_conf_new_shape) + node_list.append(node_conf_new_shape) + + outputs_score = [model_name + "@score"] + node_score = onnx.helper.make_node( + 'Mul', + inputs=outputs_prob_sigmoid + outputs_conf_new_shape, + outputs=outputs_score) + node_list.append(node_score) + + outputs_conf_bool = [model_name + "@conf_bool"] + node_conf_bool = onnx.helper.make_node( + 'Greater', + inputs=outputs_conf_new_shape + name_zeros, + outputs=outputs_conf_bool) + node_list.append(node_conf_bool) + + outputs_box_x_new_shape = [model_name + "@box_x_new_shape"] + node_box_x_new_shape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_box_x_encode + name_new_shape, + outputs=outputs_box_x_new_shape) + node_list.append(node_box_x_new_shape) + + outputs_box_y_new_shape = [model_name + "@box_y_new_shape"] + node_box_y_new_shape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_box_y_encode + name_new_shape, + outputs=outputs_box_y_new_shape) + node_list.append(node_box_y_new_shape) + + outputs_box_w_new_shape = [model_name + "@box_w_new_shape"] + node_box_w_new_shape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_box_w_encode + name_new_shape, + outputs=outputs_box_w_new_shape) + node_list.append(node_box_w_new_shape) + + outputs_box_h_new_shape = [model_name + "@box_h_new_shape"] + node_box_h_new_shape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_box_h_encode + name_new_shape, + outputs=outputs_box_h_new_shape) + node_list.append(node_box_h_new_shape) + + outputs_pred_box = [model_name + "@pred_box"] + node_pred_box = onnx.helper.make_node( + 'Concat', + inputs=outputs_box_x_new_shape + outputs_box_y_new_shape + \ + outputs_box_w_new_shape + outputs_box_h_new_shape, + outputs=outputs_pred_box, + axis=4) + node_list.append(node_pred_box) + + outputs_conf_cast = [model_name + "conf_cast"] + node_conf_cast = onnx.helper.make_node( + 'Cast', inputs=outputs_conf_bool, outputs=outputs_conf_cast, to=1) + node_list.append(node_conf_cast) + + outputs_pred_box_mul_conf = [model_name + "@pred_box_mul_conf"] + node_pred_box_mul_conf = onnx.helper.make_node( + 'Mul', + inputs=outputs_pred_box + outputs_conf_cast, + outputs=outputs_pred_box_mul_conf) + node_list.append(node_pred_box_mul_conf) + + box_shape = [1, int(num_anchors) * input_height * input_width, 4] + name_box_shape = [model_name + "@box_shape"] + node_box_shape = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_box_shape, + value=onnx.helper.make_tensor( + name=name_box_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[len(box_shape)], + vals=box_shape)) + node_list.append(node_box_shape) + + outputs_pred_box_new_shape = [model_name + "@pred_box_new_shape"] + node_pred_box_new_shape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_pred_box_mul_conf + name_box_shape, + outputs=outputs_pred_box_new_shape) + node_list.append(node_pred_box_new_shape) + + outputs_pred_box_x = [model_name + "@_pred_box_x"] + outputs_pred_box_y = [model_name + "@_pred_box_y"] + outputs_pred_box_w = [model_name + "@_pred_box_w"] + outputs_pred_box_h = [model_name + "@_pred_box_h"] + + node_pred_box_split = onnx.helper.make_node( + 'Split', + inputs=outputs_pred_box_new_shape, + outputs=outputs_pred_box_x + outputs_pred_box_y + outputs_pred_box_w + + outputs_pred_box_h, + axis=2) + node_list.append(node_pred_box_split) + + name_number_two = [model_name + "@number_two"] + node_number_two = onnx.helper.make_node( + "Constant", + inputs=[], + outputs=name_number_two, + value=onnx.helper.make_tensor( + name=name_number_two[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[2])) + node_list.append(node_number_two) + + outputs_half_w = [model_name + "@half_w"] + node_half_w = onnx.helper.make_node( + "Div", + inputs=outputs_pred_box_w + name_number_two, + outputs=outputs_half_w) + node_list.append(node_half_w) + + outputs_half_h = [model_name + "@half_h"] + node_half_h = onnx.helper.make_node( + "Div", + inputs=outputs_pred_box_h + name_number_two, + outputs=outputs_half_h) + node_list.append(node_half_h) + + outputs_pred_box_x1 = [model_name + "@pred_box_x1"] + node_pred_box_x1 = onnx.helper.make_node( + 'Sub', + inputs=outputs_pred_box_x + outputs_half_w, + outputs=outputs_pred_box_x1) + node_list.append(node_pred_box_x1) + + outputs_pred_box_y1 = [model_name + "@pred_box_y1"] + node_pred_box_y1 = onnx.helper.make_node( + 'Sub', + inputs=outputs_pred_box_y + outputs_half_h, + outputs=outputs_pred_box_y1) + node_list.append(node_pred_box_y1) + + outputs_pred_box_x2 = [model_name + "@pred_box_x2"] + node_pred_box_x2 = onnx.helper.make_node( + 'Add', + inputs=outputs_pred_box_x + outputs_half_w, + outputs=outputs_pred_box_x2) + node_list.append(node_pred_box_x2) + + outputs_pred_box_y2 = [model_name + "@pred_box_y2"] + node_pred_box_y2 = onnx.helper.make_node( + 'Add', + inputs=outputs_pred_box_y + outputs_half_h, + outputs=outputs_pred_box_y2) + node_list.append(node_pred_box_y2) + + outputs_sqeeze_image_size = [model_name + "@sqeeze_image_size"] + node_sqeeze_image_size = onnx.helper.make_node( + "Squeeze", + axes=[0], + inputs=image_size, + outputs=outputs_sqeeze_image_size) + node_list.append(node_sqeeze_image_size) + + output_img_height = [model_name + "@img_height"] + output_img_width = [model_name + "@img_width"] + node_image_size_split = onnx.helper.make_node( + "Split", + inputs=outputs_sqeeze_image_size, + outputs=output_img_height + output_img_width, + axis=-1, + split=[1, 1]) + node_list.append(node_image_size_split) + + output_img_width_cast = [model_name + "@img_width_cast"] + node_img_width_cast = onnx.helper.make_node( + 'Cast', inputs=output_img_width, outputs=output_img_width_cast, to=1) + node_list.append(node_img_width_cast) + + output_img_height_cast = [model_name + "@img_height_cast"] + node_img_height_cast = onnx.helper.make_node( + 'Cast', inputs=output_img_height, outputs=output_img_height_cast, to=1) + node_list.append(node_img_height_cast) + + outputs_pred_box_x1_decode = [model_name + "@pred_box_x1_decode"] + outputs_pred_box_y1_decode = [model_name + "@pred_box_y1_decode"] + outputs_pred_box_x2_decode = [model_name + "@pred_box_x2_decode"] + outputs_pred_box_y2_decode = [model_name + "@pred_box_y2_decode"] + + node_pred_box_x1_decode = onnx.helper.make_node( + 'Mul', + inputs=outputs_pred_box_x1 + output_img_width_cast, + outputs=outputs_pred_box_x1_decode) + node_list.append(node_pred_box_x1_decode) + + node_pred_box_y1_decode = onnx.helper.make_node( + 'Mul', + inputs=outputs_pred_box_y1 + output_img_height_cast, + outputs=outputs_pred_box_y1_decode) + node_list.append(node_pred_box_y1_decode) + + node_pred_box_x2_decode = onnx.helper.make_node( + 'Mul', + inputs=outputs_pred_box_x2 + output_img_width_cast, + outputs=outputs_pred_box_x2_decode) + node_list.append(node_pred_box_x2_decode) + + node_pred_box_y2_decode = onnx.helper.make_node( + 'Mul', + inputs=outputs_pred_box_y2 + output_img_height_cast, + outputs=outputs_pred_box_y2_decode) + node_list.append(node_pred_box_y2_decode) + + name_number_one = [model_name + "@one"] + node_number_one = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_number_one, + value=onnx.helper.make_tensor( + name=name_number_one[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[1])) + node_list.append(node_number_one) + + output_new_img_height = [model_name + "@new_img_height"] + node_new_img_height = onnx.helper.make_node( + 'Sub', + inputs=output_img_height_cast + name_number_one, + outputs=output_new_img_height) + node_list.append(node_new_img_height) + + output_new_img_width = [model_name + "@new_img_width"] + node_new_img_width = onnx.helper.make_node( + 'Sub', + inputs=output_img_width_cast + name_number_one, + outputs=output_new_img_width) + node_list.append(node_new_img_width) + + outputs_pred_box_x2_sub_w = [model_name + "@pred_box_x2_sub_w"] + node_pred_box_x2_sub_w = onnx.helper.make_node( + 'Sub', + inputs=outputs_pred_box_x2_decode + output_new_img_width, + outputs=outputs_pred_box_x2_sub_w) + node_list.append(node_pred_box_x2_sub_w) + + outputs_pred_box_y2_sub_h = [model_name + "@pred_box_y2_sub_h"] + node_pred_box_y2_sub_h = onnx.helper.make_node( + 'Sub', + inputs=outputs_pred_box_y2_decode + output_new_img_height, + outputs=outputs_pred_box_y2_sub_h) + node_list.append(node_pred_box_y2_sub_h) + + outputs_pred_box_x1_clip = [model_name + "@pred_box_x1_clip"] + outputs_pred_box_y1_clip = [model_name + "@pred_box_y1_clip"] + outputs_pred_box_x2_clip = [model_name + "@pred_box_x2_clip"] + outputs_pred_box_y2_clip = [model_name + "@pred_box_y2_clip"] + + min_const_name = model_name + "@pred_box_min_const" + max_const_name = model_name + "@pred_box_max_const" + + min_const = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=[min_const_name], + value=onnx.helper.make_tensor( + name=min_const_name, + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[0.0])) + node_list.append(min_const) + + max_const = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=[max_const_name], + value=onnx.helper.make_tensor( + name=max_const_name, + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[MAX_FLOAT])) + node_list.append(max_const) + + node_pred_box_x1_clip = onnx.helper.make_node( + 'Clip', + inputs=outputs_pred_box_x1_decode + [min_const_name, max_const_name], + outputs=outputs_pred_box_x1_clip) + node_list.append(node_pred_box_x1_clip) + + node_pred_box_y1_clip = onnx.helper.make_node( + 'Clip', + inputs=outputs_pred_box_y1_decode + [min_const_name, max_const_name], + outputs=outputs_pred_box_y1_clip) + node_list.append(node_pred_box_y1_clip) + + node_pred_box_x2_clip = onnx.helper.make_node( + 'Clip', + inputs=outputs_pred_box_x2_sub_w + [min_const_name, max_const_name], + outputs=outputs_pred_box_x2_clip) + node_list.append(node_pred_box_x2_clip) + + node_pred_box_y2_clip = onnx.helper.make_node( + 'Clip', + inputs=outputs_pred_box_y2_sub_h + [min_const_name, max_const_name], + outputs=outputs_pred_box_y2_clip) + node_list.append(node_pred_box_y2_clip) + + outputs_pred_box_x2_res = [model_name + "@box_x2_res"] + node_pred_box_x2_res = onnx.helper.make_node( + 'Sub', + inputs=outputs_pred_box_x2_decode + outputs_pred_box_x2_clip, + outputs=outputs_pred_box_x2_res) + node_list.append(node_pred_box_x2_res) + + outputs_pred_box_y2_res = [model_name + "@box_y2_res"] + node_pred_box_y2_res = onnx.helper.make_node( + 'Sub', + inputs=outputs_pred_box_y2_decode + outputs_pred_box_y2_clip, + outputs=outputs_pred_box_y2_res) + node_list.append(node_pred_box_y2_res) + + node_pred_box_result = onnx.helper.make_node( + 'Concat', + inputs=outputs_pred_box_x1_clip + outputs_pred_box_y1_clip + + outputs_pred_box_x2_res + outputs_pred_box_y2_res, + outputs=outputs['Boxes'], + axis=-1) + node_list.append(node_pred_box_result) + + score_shape = [1, input_height * input_width * int(num_anchors), class_num] + name_score_shape = [model_name + "@score_shape"] + node_score_shape = onnx.helper.make_node( + "Constant", + inputs=[], + outputs=name_score_shape, + value=onnx.helper.make_tensor( + name=name_score_shape[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[len(score_shape)], + vals=score_shape)) + node_list.append(node_score_shape) + + node_score_new_shape = onnx.helper.make_node( + 'Reshape', + inputs=outputs_score + name_score_shape, + outputs=outputs['Scores']) + node_list.append(node_score_new_shape) + return node_list diff --git a/x2paddle/op_mapper/paddle2onnx/opset9/__init__.py b/x2paddle/op_mapper/paddle2onnx/opset9/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/x2paddle/op_mapper/paddle_op_mapper.py b/x2paddle/op_mapper/paddle2onnx/opset9/opset.py similarity index 78% rename from x2paddle/op_mapper/paddle_op_mapper.py rename to x2paddle/op_mapper/paddle2onnx/opset9/opset.py index 0ba7ad682528b4062dea381964835271f0177432..850e3d0430d8bcb05c209560765b29d61a022edf 100644 --- a/x2paddle/op_mapper/paddle_op_mapper.py +++ b/x2paddle/op_mapper/paddle2onnx/opset9/opset.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License" # you may not use this file except in compliance with the License. @@ -23,7 +23,7 @@ import onnx from onnx import helper, onnx_pb -class PaddleOpMapper(object): +class OpSet9(object): def __init__(self): self.paddle_onnx_dtype_map = { core.VarDesc.VarType.FP32: onnx_pb.TensorProto.FLOAT, @@ -34,63 +34,8 @@ class PaddleOpMapper(object): core.VarDesc.VarType.INT64: onnx_pb.TensorProto.INT64, core.VarDesc.VarType.BOOL: onnx_pb.TensorProto.BOOL } - self.name_counter = dict() - def convert(self, program, save_dir): - weight_nodes = self.convert_weights(program) - op_nodes = list() - input_nodes = list() - output_nodes = list() - - unsupported_ops = set() - - print("Translating PaddlePaddle to ONNX...\n") - for block in program.blocks: - for i, op in enumerate(block.ops): - sys.stdout.write( - "\rTotal:{}, Current:{} : {} ".format( - len(block.ops), i + 1, op.type)) - sys.stdout.flush() - if not hasattr(self, op.type): - unsupported_ops.add(op.type) - continue - if len(unsupported_ops) > 0: - continue - node = getattr(self, op.type)(op, block) - if op.type == 'feed': - input_nodes.append(node) - elif op.type == 'fetch': - output_nodes.append(node) - else: - if isinstance(node, list): - op_nodes = op_nodes + node - else: - op_nodes.append(node) - - if len(unsupported_ops) > 0: - print("\nThere's {} ops are not supported yet".format( - len(unsupported_ops))) - for op in unsupported_ops: - print("=========== {} ===========".format(op)) - return - - graph = helper.make_graph( - nodes=weight_nodes + op_nodes, - name='onnx_model_from_paddle', - initializer=[], - inputs=input_nodes, - outputs=output_nodes) - model = helper.make_model(graph, producer_name='X2Paddle') - onnx.checker.check_model(model) - - if not os.path.isdir(save_dir): - os.makedirs(save_dir) - with open(os.path.join(save_dir, 'x2paddle_model.onnx'), 'wb') as f: - f.write(model.SerializeToString()) - print("\nTranslated model saved in {}".format( - os.path.join(save_dir, 'x2paddle_model.onnx'))) - def get_name(self, op_name, var_name): name = 'p2o.{}.{}'.format(op_name, var_name) if name not in self.name_counter: @@ -99,6 +44,21 @@ class PaddleOpMapper(object): self.name_counter[name] += 1 return name + '.{}'.format(self.name_counter[name]) + def make_constant_node(self, name, dtype, value=None): + if isinstance(value, list): + dims = (len(value), ) + elif value is None: + dims = () + value = [] + else: + dims = () + value = [value] + tensor = helper.make_tensor( + name=name, data_type=dtype, dims=dims, vals=value) + node = helper.make_node( + 'Constant', inputs=[], outputs=[name], value=tensor) + return node + def convert_weights(self, program): var_names = program.global_block().vars nodes = list() @@ -119,21 +79,6 @@ class PaddleOpMapper(object): nodes.append(node) return nodes - def make_constant_node(self, name, dtype, value=None): - if isinstance(value, list): - dims = (len(value), ) - elif value is None: - dims = () - value = [] - else: - dims = () - value = [value] - tensor = helper.make_tensor( - name=name, data_type=dtype, dims=dims, vals=value) - node = helper.make_node( - 'Constant', inputs=[], outputs=[name], value=tensor) - return node - def conv2d(self, op, block): kernel_shape = block.var(op.input('Filter')[0]).shape node = helper.make_node( @@ -251,6 +196,8 @@ class PaddleOpMapper(object): pool_type[op.attr('pooling_type')][1], inputs=op.input('X'), outputs=op.output('Out'), ) + elif op.attr('adaptive'): + raise Excpetion("ONNX cannot support adaptive pool") else: input_shape = block.var(op.input('X')[0]).shape k_size = op.attr('ksize') @@ -268,6 +215,28 @@ class PaddleOpMapper(object): pads=op.attr('paddings') + op.attr('paddings')) return node + def pad2d(self, op, block): + x_shape = block.var(op.input('X')[0]).shape + paddings = op.attr('paddings') + onnx_pads = [] + if op.attr('data_format') == 'NCHW': + pads = [ + 0, 0, paddings[0], paddings[2], 0, 0, paddings[1], paddings[3] + ] + else: + pads = [ + 0, paddings[0], paddings[2], 0, 0, paddings[1], paddings[3], 0 + ] + #TODO support pads is Variable + node = helper.make_node( + 'Pad', + inputs=op.input('X'), + outputs=op.output('Out'), + mode=op.attr('mode'), + value=op.attr('pad_value'), + pads=pads) + return node + def softmax(self, op, block): axis = op.attr('axis') shape = block.var(op.output('Out')[0]).shape @@ -397,17 +366,14 @@ class PaddleOpMapper(object): return self.conv2d(op, block) def relu6(self, op, block): - min_name = self.get_name(op.type, 'min') - max_name = self.get_name(op.type, 'max') - min_node = self.make_constant_node(min_name, onnx_pb.TensorProto.FLOAT, - 0) - max_node = self.make_constant_node(max_name, onnx_pb.TensorProto.FLOAT, - op.attr('threshold')) + threshold = op.attr('threshold') node = helper.make_node( 'Clip', - inputs=[op.input('X')[0], min_name, max_name], - outputs=op.output('Out'), ) - return [min_node, max_node, node] + inputs=[op.input('X')[0]], + outputs=op.output('Out'), + max=threshold, + min=0.0) + return [node] def shape(self, op, block): node = helper.make_node( @@ -435,21 +401,14 @@ class PaddleOpMapper(object): axes = op.attr('axes') starts = op.attr('starts') ends = op.attr('ends') - axes_name = self.get_name(op.type, 'axes') - starts_name = self.get_name(op.type, 'starts') - ends_name = self.get_name(op.type, 'ends') - - axes_node = self.make_constant_node(axes_name, - onnx_pb.TensorProto.INT64, axes) - starts_node = self.make_constant_node(starts_name, - onnx_pb.TensorProto.INT64, starts) - ends_node = self.make_constant_node(ends_name, - onnx_pb.TensorProto.INT64, ends) node = helper.make_node( "Slice", inputs=[op.input('Input')[0], starts_name, ends_name, axes_name], - outputs=op.output('Out'), ) - return [starts_node, ends_node, axes_node, node] + outputs=op.output('Out'), + axes=axes, + starts=starts, + ends=ends) + return [node] def fill_constant(self, op, block): value = op.attr('value') @@ -544,27 +503,15 @@ class PaddleOpMapper(object): def bilinear_interp(self, op, block): input_names = op.input_names - coordinate_transformation_mode = 'half_pixel' - if op.attr('align_corners'): - coordinate_transformation_mode = 'align_corners' + input_shape = block.vars[op.input('X')[0]].shape + if op.attr('align_corners') or op.attr('align_mode') == 0: + raise Exception( + "Resize in onnx(opset<=10) only support coordinate_transformation_mode: 'asymmetric'." + ) if ('OutSize' in input_names and len(op.input('OutSize')) > 0) or ( 'SizeTensor' in input_names and len(op.input('SizeTensor')) > 0): node_list = list() - roi_node = self.make_constant_node( - self.get_name(op.type, 'roi'), onnx_pb.TensorProto.FLOAT, - [1, 1, 1, 1, 1, 1, 1, 1]) - roi_name = self.get_name(op.type, 'roi') - roi_node = self.make_constant_node( - roi_name, onnx_pb.TensorProto.FLOAT, [1, 1, 1, 1, 1, 1, 1, 1]) - empty_name = self.get_name(op.type, 'empty') - empty_tensor = helper.make_tensor( - empty_name, - onnx_pb.TensorProto.FLOAT, (0, ), - np.array([]).astype('float32'), - raw=False) - empty_node = helper.make_node( - 'Constant', [], outputs=[empty_name], value=empty_tensor) shape_name0 = self.get_name(op.type, 'shape') shape_node0 = helper.make_node( 'Shape', inputs=op.input('X'), outputs=[shape_name0]) @@ -579,16 +526,7 @@ class PaddleOpMapper(object): 'Slice', inputs=[shape_name0, starts_name, ends_name], outputs=[shape_name1]) - node_list.extend([ - roi_node, empty_node, shape_node0, starts_node, ends_node, - shape_node1 - ]) - # shape_name2 = self.get_name(op.type, "shape.cast") - # shape_node2 = helper.make_node( - # 'Cast', - # inputs=op.input('OutSize'), - # outputs=[shape_name2], - # to=onnx_pb.TensorProto.INT64) + node_list.extend([shape_node0, starts_node, ends_node, shape_node1]) if 'OutSize' in input_names and len(op.input('OutSize')) > 0: cast_shape_name = self.get_name(op.type, "shape.cast") cast_shape_node = helper.make_node( @@ -598,7 +536,8 @@ class PaddleOpMapper(object): to=onnx_pb.TensorProto.INT64) node_list.append(cast_shape_node) else: - concat_shape_name = self.get_name(op.type, "shape.concat") + concat_shape_name = self.get_name( + op.type, op.output('Out')[0] + "shape.concat") concat_shape_node = helper.make_node( "Concat", inputs=op.input('SizeTensor'), @@ -611,27 +550,46 @@ class PaddleOpMapper(object): outputs=[cast_shape_name], to=onnx_pb.TensorProto.INT64) node_list.extend([concat_shape_node, cast_shape_node]) - shape_name3 = self.get_name(op.type, "shape.concat") - shape_node3 = helper.make_node( + shape_name2 = self.get_name(op.type, "shape.concat") + shape_node2 = helper.make_node( 'Concat', inputs=[shape_name1, cast_shape_name], - outputs=[shape_name3], + outputs=[shape_name2], axis=0) + node_list.append(shape_node2) + cast_shape_name2 = self.get_name(op.type, "shape.cast") + cast_shape_node2 = helper.make_node( + 'Cast', + inputs=[shape_name2], + outputs=[cast_shape_name2], + to=onnx_pb.TensorProto.FLOAT) + node_list.append(cast_shape_node2) + cast_shape_name0 = self.get_name(op.type, "shape.cast") + cast_shape_node0 = helper.make_node( + 'Cast', + inputs=[shape_name0], + outputs=[cast_shape_name0], + to=onnx_pb.TensorProto.FLOAT) + node_list.append(cast_shape_node0) + outputs_h_w_scales = op.output('Out')[0] + "@out_hw_scales" + node_h_w_scales = helper.make_node( + 'Div', + inputs=[cast_shape_name2, cast_shape_name0], + outputs=[outputs_h_w_scales]) + node_list.append(node_h_w_scales) result_node = helper.make_node( 'Resize', - inputs=[op.input('X')[0], roi_name, empty_name, shape_name3], + inputs=[op.input('X')[0], outputs_h_w_scales], outputs=op.output('Out'), - mode='linear', - coordinate_transformation_mode=coordinate_transformation_mode) - node_list.extend([shape_node3, result_node]) + mode='linear') + node_list.extend([result_node]) return node_list elif 'Scale' in input_names and len(op.input('Scale')) > 0: node = helper.make_node( 'Resize', inputs=[op.input('X')[0], op.input('Scale')[0]], outputs=op.output('Out'), - mode='linear', - coordinate_transformation_mode=coordinate_transformation_mode) + mode='linear') else: out_shape = [op.attr('out_h'), op.attr('out_w')] scale = op.attr('scale') @@ -640,41 +598,34 @@ class PaddleOpMapper(object): scale_node = self.make_constant_node(scale_name, onnx_pb.TensorProto.FLOAT, [1, 1, scale, scale]) - roi_name = self.get_name(op.type, 'roi') - roi_node = self.make_constant_node(roi_name, - onnx_pb.TensorProto.FLOAT, - [1, 1, 1, 1, 1, 1, 1, 1]) node = helper.make_node( 'Resize', - inputs=[op.input('X')[0], roi_name, scale_name], + inputs=[op.input('X')[0], scale_name], outputs=op.output('Out'), - mode='nearest', - coordinate_transformation_mode=coordinate_transformation_mode - ) - return [scale_node, roi_node, node] + mode='linear') + return [scale_node, node] else: raise Exception("Unexpected situation happend") return node def nearest_interp(self, op, block): input_names = op.input_names - coordinate_transformation_mode = 'half_pixel' if op.attr('align_corners'): - coordinate_transformation_mode = 'align_corners' + raise Exception( + "Resize in onnx(opset<=10) only support coordinate_transformation_mode: 'asymmetric'." + ) if 'OutSize' in input_names and len(op.input('OutSize')) > 0: node = helper.make_node( 'Resize', - inputs=[op.input('X')[0], '', op.input('OutSize')[0]], + inputs=[op.input('X')[0], op.input('OutSize')[0]], outputs=op.output('Out'), - mode='nearest', - coordinate_transformation_mode=coordinate_transformation_mode) + mode='nearest') elif 'Scale' in input_names and len(op.input('Scale')) > 0: node = helper.make_node( 'Resize', inputs=[op.input('X')[0], op.input('Scale')[0]], outputs=op.output('Out'), - mode='nearest', - coordinate_transformation_mode=coordinate_transformation_mode) + mode='nearest') else: out_shape = [op.attr('out_h'), op.attr('out_w')] scale = op.attr('scale') @@ -683,18 +634,12 @@ class PaddleOpMapper(object): scale_node = self.make_constant_node(scale_name, onnx_pb.TensorProto.FLOAT, [1, 1, scale, scale]) - roi_name = self.get_name(op.type, 'roi') - roi_node = self.make_constant_node(roi_name, - onnx_pb.TensorProto.FLOAT, - [1, 1, 1, 1, 1, 1, 1, 1]) node = helper.make_node( 'Resize', - inputs=[op.input('X')[0], roi_name, scale_name], + inputs=[op.input('X')[0], scale_name], outputs=op.output('Out'), - mode='nearest', - coordinate_transformation_mode=coordinate_transformation_mode - ) - return [scale_node, roi_node, node] + mode='nearest') + return [scale_node, node] else: raise Exception("Unexpected situation happend") return node @@ -711,14 +656,8 @@ class PaddleOpMapper(object): return node def hard_swish(self, op, block): - min_name = self.get_name(op.type, 'min') - max_name = self.get_name(op.type, 'max') scale_name = self.get_name(op.type, 'scale') offset_name = self.get_name(op.type, 'offset') - min_node = self.make_constant_node(min_name, onnx_pb.TensorProto.FLOAT, - 0) - max_node = self.make_constant_node(max_name, onnx_pb.TensorProto.FLOAT, - op.attr('threshold')) scale_node = self.make_constant_node(scale_name, onnx_pb.TensorProto.FLOAT, op.attr('scale')) @@ -730,19 +669,20 @@ class PaddleOpMapper(object): node0 = helper.make_node( 'Add', inputs=[op.input('X')[0], offset_name], outputs=[name0]) name1 = self.get_name(op.type, 'relu') + min_value = op.attr('min') + max_value = op.attr('max') node1 = helper.make_node( 'Clip', - inputs=[name0, min_name, max_name], - outputs=[name1], ) + inputs=[name0], + outputs=[name1], + max=max_value, + min=min_value) name2 = self.get_name(op.type, 'mul') node2 = helper.make_node( 'Mul', inputs=[op.input('X')[0], name1], outputs=[name2]) node3 = helper.make_node( 'Div', inputs=[name2, scale_name], outputs=op.output('Out')) - return [ - min_node, max_node, scale_node, offset_node, node0, node1, node2, - node3 - ] + return [scale_node, offset_node, node0, node1, node2, node3] def elementwise_mul(self, op, block): axis = op.attr('axis') @@ -818,3 +758,11 @@ class PaddleOpMapper(object): def im2sequence(self, op, block): from .paddle_custom_layer.im2sequence import im2sequence return im2sequence(op, block) + + def yolo_box(self, op, block): + from .paddle_custom_layer.yolo_box import yolo_box + return yolo_box(op, block) + + def multiclass_nms(self, op, block): + from .paddle_custom_layer.multiclass_nms import multiclass_nms + return multiclass_nms(op, block) diff --git a/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/__init__.py b/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/x2paddle/op_mapper/paddle_custom_layer/im2sequence.py b/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/im2sequence.py similarity index 84% rename from x2paddle/op_mapper/paddle_custom_layer/im2sequence.py rename to x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/im2sequence.py index aeb4a9ceddb280295aafb2ebcafa3a25f8767d75..ab5a90d3b2a64f5c61fad957c0f0c4e1835bb165 100644 --- a/x2paddle/op_mapper/paddle_custom_layer/im2sequence.py +++ b/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/im2sequence.py @@ -1,3 +1,17 @@ +# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + import onnx import numpy as np from onnx import onnx_pb, helper diff --git a/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/multiclass_nms.py b/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/multiclass_nms.py new file mode 100644 index 0000000000000000000000000000000000000000..58d8e0c9d1c9e780b62cfba578973512e1214ba8 --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/multiclass_nms.py @@ -0,0 +1,447 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math +import sys +import os +import numpy as np +import paddle.fluid.core as core +import paddle.fluid as fluid +import onnx +import warnings +from onnx import helper, onnx_pb + + +def multiclass_nms(op, block): + """ + Convert the paddle multiclass_nms to onnx op. + This op is get the select boxes from origin boxes. + """ + inputs = dict() + outputs = dict() + attrs = dict() + for name in op.input_names: + inputs[name] = op.input(name) + for name in op.output_names: + outputs[name] = op.output(name) + for name in op.attr_names: + attrs[name] = op.attr(name) + + result_name = outputs['Out'][0] + background = attrs['background_label'] + normalized = attrs['normalized'] + if normalized == False: + warnings.warn( + 'The parameter normalized of multiclass_nms OP of Paddle is False, which has diff with ONNX. \ + Please set normalized=True in multiclass_nms of Paddle') + + #convert the paddle attribute to onnx tensor + name_score_threshold = [outputs['Out'][0] + "@score_threshold"] + name_iou_threshold = [outputs['Out'][0] + "@iou_threshold"] + name_keep_top_k = [outputs['Out'][0] + '@keep_top_k'] + name_keep_top_k_2D = [outputs['Out'][0] + '@keep_top_k_1D'] + + node_score_threshold = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_score_threshold, + value=onnx.helper.make_tensor( + name=name_score_threshold[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[float(attrs['score_threshold'])])) + + node_iou_threshold = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_iou_threshold, + value=onnx.helper.make_tensor( + name=name_iou_threshold[0] + "@const", + data_type=onnx.TensorProto.FLOAT, + dims=(), + vals=[float(attrs['nms_threshold'])])) + + node_keep_top_k = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_keep_top_k, + value=onnx.helper.make_tensor( + name=name_keep_top_k[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=(), + vals=[np.int64(attrs['keep_top_k'])])) + + node_keep_top_k_2D = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_keep_top_k_2D, + value=onnx.helper.make_tensor( + name=name_keep_top_k_2D[0] + "@const", + data_type=onnx.TensorProto.INT64, + dims=[1, 1], + vals=[np.int64(attrs['keep_top_k'])])) + + # the paddle data format is x1,y1,x2,y2 + kwargs = {'center_point_box': 0} + + name_select_nms = [outputs['Out'][0] + "@select_index"] + node_select_nms= onnx.helper.make_node( + 'NonMaxSuppression', + inputs=inputs['BBoxes'] + inputs['Scores'] + name_keep_top_k +\ + name_iou_threshold + name_score_threshold, + outputs=name_select_nms) + # step 1 nodes select the nms class + node_list = [ + node_score_threshold, node_iou_threshold, node_keep_top_k, + node_keep_top_k_2D, node_select_nms + ] + + # create some const value to use + name_const_value = [result_name+"@const_0", + result_name+"@const_1",\ + result_name+"@const_2",\ + result_name+"@const_-1"] + value_const_value = [0, 1, 2, -1] + for name, value in zip(name_const_value, value_const_value): + node = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=[name], + value=onnx.helper.make_tensor( + name=name + "@const", + data_type=onnx.TensorProto.INT64, + dims=[1], + vals=[value])) + node_list.append(node) + + # In this code block, we will deocde the raw score data, reshape N * C * M to 1 * N*C*M + # and the same time, decode the select indices to 1 * D, gather the select_indices + outputs_gather_1 = [result_name + "@gather_1"] + node_gather_1 = onnx.helper.make_node( + 'Gather', + inputs=name_select_nms + [result_name + "@const_1"], + outputs=outputs_gather_1, + axis=1) + node_list.append(node_gather_1) + + outputs_squeeze_gather_1 = [result_name + "@sequeeze_gather_1"] + node_squeeze_gather_1 = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_gather_1, + outputs=outputs_squeeze_gather_1, + axes=[1]) + node_list.append(node_squeeze_gather_1) + + outputs_gather_2 = [result_name + "@gather_2"] + node_gather_2 = onnx.helper.make_node( + 'Gather', + inputs=name_select_nms + [result_name + "@const_2"], + outputs=outputs_gather_2, + axis=1) + node_list.append(node_gather_2) + + #slice the class is not 0 + if background == 0: + outputs_nonzero = [result_name + "@nonzero"] + node_nonzero = onnx.helper.make_node( + 'NonZero', inputs=outputs_squeeze_gather_1, outputs=outputs_nonzero) + node_list.append(node_nonzero) + else: + name_thresh = [result_name + "@thresh"] + node_thresh = onnx.helper.make_node( + 'Constant', + inputs=[], + outputs=name_thresh, + value=onnx.helper.make_tensor( + name=name_thresh[0] + "@const", + data_type=onnx.TensorProto.INT32, + dims=[1], + vals=[-1])) + node_list.append(node_thresh) + + outputs_cast = [result_name + "@cast"] + node_cast = onnx.helper.make_node( + 'Cast', inputs=outputs_squeeze_gather_1, outputs=outputs_cast, to=6) + node_list.append(node_cast) + + outputs_greater = [result_name + "@greater"] + node_greater = onnx.helper.make_node( + 'Greater', + inputs=outputs_cast + name_thresh, + outputs=outputs_greater) + node_list.append(node_greater) + + outputs_nonzero = [result_name + "@nonzero"] + node_nonzero = onnx.helper.make_node( + 'NonZero', inputs=outputs_greater, outputs=outputs_nonzero) + node_list.append(node_nonzero) + + outputs_gather_1_nonzero = [result_name + "@gather_1_nonzero"] + node_gather_1_nonzero = onnx.helper.make_node( + 'Gather', + inputs=outputs_gather_1 + outputs_nonzero, + outputs=outputs_gather_1_nonzero, + axis=0) + node_list.append(node_gather_1_nonzero) + + outputs_gather_2_nonzero = [result_name + "@gather_2_nonzero"] + node_gather_2_nonzero = onnx.helper.make_node( + 'Gather', + inputs=outputs_gather_2 + outputs_nonzero, + outputs=outputs_gather_2_nonzero, + axis=0) + node_list.append(node_gather_2_nonzero) + + # reshape scores N * C * M to (N*C*M) * 1 + outputs_reshape_scores_rank1 = [result_name + "@reshape_scores_rank1"] + node_reshape_scores_rank1 = onnx.helper.make_node( + "Reshape", + inputs=inputs['Scores'] + [result_name + "@const_-1"], + outputs=outputs_reshape_scores_rank1) + node_list.append(node_reshape_scores_rank1) + + # get the shape of scores + outputs_shape_scores = [result_name + "@shape_scores"] + node_shape_scores = onnx.helper.make_node( + 'Shape', inputs=inputs['Scores'], outputs=outputs_shape_scores) + node_list.append(node_shape_scores) + + # gather the index: 2 shape of scores + outputs_gather_scores_dim1 = [result_name + "@gather_scores_dim1"] + node_gather_scores_dim1 = onnx.helper.make_node( + 'Gather', + inputs=outputs_shape_scores + [result_name + "@const_2"], + outputs=outputs_gather_scores_dim1, + axis=0) + node_list.append(node_gather_scores_dim1) + + # mul class * M + outputs_mul_classnum_boxnum = [result_name + "@mul_classnum_boxnum"] + node_mul_classnum_boxnum = onnx.helper.make_node( + 'Mul', + inputs=outputs_gather_1_nonzero + outputs_gather_scores_dim1, + outputs=outputs_mul_classnum_boxnum) + node_list.append(node_mul_classnum_boxnum) + + # add class * M * index + outputs_add_class_M_index = [result_name + "@add_class_M_index"] + node_add_class_M_index = onnx.helper.make_node( + 'Add', + inputs=outputs_mul_classnum_boxnum + outputs_gather_2_nonzero, + outputs=outputs_add_class_M_index) + node_list.append(node_add_class_M_index) + + # Squeeze the indices to 1 dim + outputs_squeeze_select_index = [result_name + "@squeeze_select_index"] + node_squeeze_select_index = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_add_class_M_index, + outputs=outputs_squeeze_select_index, + axes=[0, 2]) + node_list.append(node_squeeze_select_index) + + # gather the data from flatten scores + outputs_gather_select_scores = [result_name + "@gather_select_scores"] + node_gather_select_scores = onnx.helper.make_node('Gather', + inputs=outputs_reshape_scores_rank1 + \ + outputs_squeeze_select_index, + outputs=outputs_gather_select_scores, + axis=0) + node_list.append(node_gather_select_scores) + + # get nums to input TopK + outputs_shape_select_num = [result_name + "@shape_select_num"] + node_shape_select_num = onnx.helper.make_node( + 'Shape', + inputs=outputs_gather_select_scores, + outputs=outputs_shape_select_num) + node_list.append(node_shape_select_num) + + outputs_gather_select_num = [result_name + "@gather_select_num"] + node_gather_select_num = onnx.helper.make_node( + 'Gather', + inputs=outputs_shape_select_num + [result_name + "@const_0"], + outputs=outputs_gather_select_num, + axis=0) + node_list.append(node_gather_select_num) + + outputs_unsqueeze_select_num = [result_name + "@unsqueeze_select_num"] + node_unsqueeze_select_num = onnx.helper.make_node( + 'Unsqueeze', + inputs=outputs_gather_select_num, + outputs=outputs_unsqueeze_select_num, + axes=[0]) + node_list.append(node_unsqueeze_select_num) + + outputs_concat_topK_select_num = [result_name + "@conat_topK_select_num"] + node_conat_topK_select_num = onnx.helper.make_node( + 'Concat', + inputs=outputs_unsqueeze_select_num + name_keep_top_k_2D, + outputs=outputs_concat_topK_select_num, + axis=0) + node_list.append(node_conat_topK_select_num) + + outputs_cast_concat_topK_select_num = [ + result_name + "@concat_topK_select_num" + ] + node_outputs_cast_concat_topK_select_num = onnx.helper.make_node( + 'Cast', + inputs=outputs_concat_topK_select_num, + outputs=outputs_cast_concat_topK_select_num, + to=6) + node_list.append(node_outputs_cast_concat_topK_select_num) + # get min(topK, num_select) + outputs_compare_topk_num_select = [result_name + "@compare_topk_num_select"] + node_compare_topk_num_select = onnx.helper.make_node( + 'ReduceMin', + inputs=outputs_cast_concat_topK_select_num, + outputs=outputs_compare_topk_num_select, + keepdims=0) + node_list.append(node_compare_topk_num_select) + + # unsqueeze the indices to 1D tensor + outputs_unsqueeze_topk_select_indices = [ + result_name + "@unsqueeze_topk_select_indices" + ] + node_unsqueeze_topk_select_indices = onnx.helper.make_node( + 'Unsqueeze', + inputs=outputs_compare_topk_num_select, + outputs=outputs_unsqueeze_topk_select_indices, + axes=[0]) + node_list.append(node_unsqueeze_topk_select_indices) + + # cast the indices to INT64 + outputs_cast_topk_indices = [result_name + "@cast_topk_indices"] + node_cast_topk_indices = onnx.helper.make_node( + 'Cast', + inputs=outputs_unsqueeze_topk_select_indices, + outputs=outputs_cast_topk_indices, + to=7) + node_list.append(node_cast_topk_indices) + + # select topk scores indices + outputs_topk_select_topk_indices = [result_name + "@topk_select_topk_values",\ + result_name + "@topk_select_topk_indices"] + node_topk_select_topk_indices = onnx.helper.make_node( + 'TopK', + inputs=outputs_gather_select_scores + outputs_cast_topk_indices, + outputs=outputs_topk_select_topk_indices) + node_list.append(node_topk_select_topk_indices) + + # gather topk label, scores, boxes + outputs_gather_topk_scores = [result_name + "@gather_topk_scores"] + node_gather_topk_scores = onnx.helper.make_node( + 'Gather', + inputs=outputs_gather_select_scores + + [outputs_topk_select_topk_indices[1]], + outputs=outputs_gather_topk_scores, + axis=0) + node_list.append(node_gather_topk_scores) + + outputs_gather_topk_class = [result_name + "@gather_topk_class"] + node_gather_topk_class = onnx.helper.make_node( + 'Gather', + inputs=outputs_gather_1_nonzero + + [outputs_topk_select_topk_indices[1]], + outputs=outputs_gather_topk_class, + axis=1) + node_list.append(node_gather_topk_class) + + # gather the boxes need to gather the boxes id, then get boxes + outputs_gather_topk_boxes_id = [result_name + "@gather_topk_boxes_id"] + node_gather_topk_boxes_id = onnx.helper.make_node( + 'Gather', + inputs=outputs_gather_2_nonzero + + [outputs_topk_select_topk_indices[1]], + outputs=outputs_gather_topk_boxes_id, + axis=1) + node_list.append(node_gather_topk_boxes_id) + + # squeeze the gather_topk_boxes_id to 1 dim + outputs_squeeze_topk_boxes_id = [result_name + "@squeeze_topk_boxes_id"] + node_squeeze_topk_boxes_id = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_gather_topk_boxes_id, + outputs=outputs_squeeze_topk_boxes_id, + axes=[0, 2]) + node_list.append(node_squeeze_topk_boxes_id) + + outputs_gather_select_boxes = [result_name + "@gather_select_boxes"] + node_gather_select_boxes = onnx.helper.make_node( + 'Gather', + inputs=inputs['BBoxes'] + outputs_squeeze_topk_boxes_id, + outputs=outputs_gather_select_boxes, + axis=1) + node_list.append(node_gather_select_boxes) + + # concat the final result + # before concat need to cast the class to float + outputs_cast_topk_class = [result_name + "@cast_topk_class"] + node_cast_topk_class = onnx.helper.make_node( + 'Cast', + inputs=outputs_gather_topk_class, + outputs=outputs_cast_topk_class, + to=1) + node_list.append(node_cast_topk_class) + + outputs_unsqueeze_topk_scores = [result_name + "@unsqueeze_topk_scores"] + node_unsqueeze_topk_scores = onnx.helper.make_node( + 'Unsqueeze', + inputs=outputs_gather_topk_scores, + outputs=outputs_unsqueeze_topk_scores, + axes=[0, 2]) + node_list.append(node_unsqueeze_topk_scores) + + inputs_concat_final_results = outputs_cast_topk_class + outputs_unsqueeze_topk_scores +\ + outputs_gather_select_boxes + outputs_sort_by_socre_results = [result_name + "@concat_topk_scores"] + node_sort_by_socre_results = onnx.helper.make_node( + 'Concat', + inputs=inputs_concat_final_results, + outputs=outputs_sort_by_socre_results, + axis=2) + node_list.append(node_sort_by_socre_results) + + # select topk classes indices + outputs_squeeze_cast_topk_class = [result_name + "@squeeze_cast_topk_class"] + node_squeeze_cast_topk_class = onnx.helper.make_node( + 'Squeeze', + inputs=outputs_cast_topk_class, + outputs=outputs_squeeze_cast_topk_class, + axes=[0, 2]) + node_list.append(node_squeeze_cast_topk_class) + outputs_neg_squeeze_cast_topk_class = [ + result_name + "@neg_squeeze_cast_topk_class" + ] + node_neg_squeeze_cast_topk_class = onnx.helper.make_node( + 'Neg', + inputs=outputs_squeeze_cast_topk_class, + outputs=outputs_neg_squeeze_cast_topk_class) + node_list.append(node_neg_squeeze_cast_topk_class) + outputs_topk_select_classes_indices = [result_name + "@topk_select_topk_classes_scores",\ + result_name + "@topk_select_topk_classes_indices"] + node_topk_select_topk_indices = onnx.helper.make_node( + 'TopK', + inputs=outputs_neg_squeeze_cast_topk_class + outputs_cast_topk_indices, + outputs=outputs_topk_select_classes_indices) + node_list.append(node_topk_select_topk_indices) + outputs_concat_final_results = outputs['Out'] + node_concat_final_results = onnx.helper.make_node( + 'Gather', + inputs=outputs_sort_by_socre_results + + [outputs_topk_select_classes_indices[1]], + outputs=outputs_concat_final_results, + axis=1) + node_list.append(node_concat_final_results) + return node_list diff --git a/x2paddle/op_mapper/paddle_custom_layer/yolo_box.py b/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/yolo_box.py similarity index 97% rename from x2paddle/op_mapper/paddle_custom_layer/yolo_box.py rename to x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/yolo_box.py index a1e49e77b32ce1d64c11ca35ac69ed6cb20ee51c..c1a37030302b94a111a019c3c8713cdbc1b02364 100644 --- a/x2paddle/op_mapper/paddle_custom_layer/yolo_box.py +++ b/x2paddle/op_mapper/paddle2onnx/opset9/paddle_custom_layer/yolo_box.py @@ -1,3 +1,17 @@ +# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + import onnx import numpy as np from onnx import onnx_pb, helper diff --git a/x2paddle/op_mapper/paddle2onnx/paddle_op_mapper.py b/x2paddle/op_mapper/paddle2onnx/paddle_op_mapper.py new file mode 100644 index 0000000000000000000000000000000000000000..1ce2ec5e7093d6d5302e673c4400fe0a87d66583 --- /dev/null +++ b/x2paddle/op_mapper/paddle2onnx/paddle_op_mapper.py @@ -0,0 +1,108 @@ +# Copyright (c) 2019 PaddlePaddle Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License" +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import math +import sys +import x2paddle +import os +import numpy as np +import paddle.fluid.core as core +import paddle.fluid as fluid +import onnx +from onnx import helper, onnx_pb +from x2paddle.op_mapper.paddle2onnx.opset9.opset import OpSet9 +from x2paddle.op_mapper.paddle2onnx.opset10.opset import OpSet10 +from x2paddle.op_mapper.paddle2onnx.opset11.opset import OpSet11 + + +class PaddleOpMapper(object): + def __init__(self): + self.support_opsets = [9, 10, 11] + self.default_opset = 10 + self.name_counter = dict() + self.op_set = None + + def convert(self, program, save_dir, opset_number=10): + self.op_set = self.create_opset(opset_number) + weight_nodes = self.op_set.convert_weights(program) + op_nodes = list() + input_nodes = list() + output_nodes = list() + unsupported_ops = set() + + print("Translating PaddlePaddle to ONNX...\n") + for block in program.blocks: + for i, op in enumerate(block.ops): + sys.stdout.write("\rTotal:{}, Current:{} : {} ".format( + len(block.ops), i + 1, op.type)) + sys.stdout.flush() + if not hasattr(self.op_set, op.type): + unsupported_ops.add(op.type) + continue + if len(unsupported_ops) > 0: + continue + node = getattr(self.op_set, op.type)(op, block) + if op.type == 'feed': + print(node.name) + input_nodes.append(node) + elif op.type == 'fetch': + output_nodes.append(node) + else: + if isinstance(node, list): + op_nodes = op_nodes + node + else: + op_nodes.append(node) + + if len(unsupported_ops) > 0: + print("\nThere's {} ops are not supported yet".format( + len(unsupported_ops))) + for op in unsupported_ops: + print("=========== {} ===========".format(op)) + return + + graph = helper.make_graph( + nodes=weight_nodes + op_nodes, + name='onnx_model_from_paddle', + initializer=[], + inputs=input_nodes, + outputs=output_nodes) + opset_imports = [helper.make_opsetid("", opset_number)] + model = helper.make_model( + graph, producer_name='X2Paddle', opset_imports=opset_imports) + onnx.checker.check_model(model) + + if not os.path.isdir(save_dir): + os.makedirs(save_dir) + with open(os.path.join(save_dir, 'x2paddle_model.onnx'), 'wb') as f: + f.write(model.SerializeToString()) + print("\nTranslated model saved in {}".format( + os.path.join(save_dir, 'x2paddle_model.onnx'))) + + def create_opset(self, opset_number): + run_opset = self.default_opset + opset = '' + if opset_number in self.support_opsets: + run_opset = opset_number + else: + for support_opset_number in self.support_opsets: + if support_opset_number < opset_number: + run_opset = support_opset_number + else: + break + print( + 'Now, onnx2paddle support convert onnx model opset_verison {},' + 'opset_verison of your onnx model is {}, automatically treated as op_set: {}.' + .format(self.support_opsets, opset_number, run_opset)) + opset = 'OpSet' + str(run_opset) + return eval(opset)() diff --git a/x2paddle/op_mapper/tf_op_mapper_nhwc.py b/x2paddle/op_mapper/tf_op_mapper_nhwc.py index 6f36c8dbaaccdc03ea3c482a1a011bd3a6b275ee..53e52a4b177665e0b376f93b3da21423ed9ecc48 100644 --- a/x2paddle/op_mapper/tf_op_mapper_nhwc.py +++ b/x2paddle/op_mapper/tf_op_mapper_nhwc.py @@ -15,6 +15,10 @@ from x2paddle.decoder.tf_decoder import TFGraph from x2paddle.core.op_mapper import OpMapper from x2paddle.core.util import * +from x2paddle import program +from x2paddle import gen_name +import traceback +import math import inspect import numpy import sys @@ -35,7 +39,6 @@ class TFOpMapperNHWC(OpMapper): directly_map_ops = { 'Relu': ['relu'], 'Relu6': ['relu6'], - 'Shape': ['shape'], 'Abs': ['abs'], 'Sigmoid': ['sigmoid'], 'Exp': ['exp'], @@ -46,7 +49,9 @@ class TFOpMapperNHWC(OpMapper): 'Softplus': ['softplus'], 'LeakyRelu': ['leaky_relu', { 'alpha': 'alpha' - }] + }], + 'Floor': ['floor'], + 'Erf': ['erf'] } elementwise_ops = { 'Add': 'elementwise_add', @@ -54,6 +59,9 @@ class TFOpMapperNHWC(OpMapper): 'RealDiv': 'elementwise_div', 'Sub': 'elementwise_sub', 'Maximum': 'elementwise_max', + 'Minimum': 'elementwise_min', + 'LessEqual': 'less_equal', + 'GreaterEqual': 'greater_equal', 'Mul': 'elementwise_mul', 'FloorDiv': 'elementwise_floordiv' } @@ -63,18 +71,25 @@ class TFOpMapperNHWC(OpMapper): self.decoder = decoder self.graph = decoder.tf_graph self.weights = dict() - self.batch_node = None self.omit_nodes = list() self.used_custom_layers = dict() + program.clear() not_placeholder = list() for name in self.graph.input_nodes: - if self.graph.get_node(name).layer_type != "Placeholder": + if self.graph.get_node( + name).layer_type != "Placeholder" and self.graph.get_node( + name + ).layer_type != "OneShotIterator" and self.graph.get_node( + name).layer_type != "IteratorV2": not_placeholder.append(name) for name in not_placeholder: idx = self.graph.input_nodes.index(name) del self.graph.input_nodes[idx] + program.inputs = self.graph.input_nodes + program.outputs = self.graph.output_nodes + unsupported_ops = set() sys.stderr.write("Total nodes: {}\n".format(len(self.graph.topo_sort))) for i, node_name in enumerate(self.graph.topo_sort): @@ -95,31 +110,23 @@ class TFOpMapperNHWC(OpMapper): func = getattr(self, op) try: func(node) - except: + except Exception as e: unsupported_ops.add(op) + print("\n{}\n".format(traceback.format_exc())) else: unsupported_ops.add(op) if len(unsupported_ops) > 0: - print("========= {} OPs are not supported yet ===========".format( + print("\n========= {} OPs are not supported yet ===========".format( len(unsupported_ops))) for op in unsupported_ops: print("========== {} ============".format(op)) sys.exit(-1) sys.stderr.write("\nDone!\n") - def add_omit_nodes(self, in_node_name, out_node_name): - in_node = self.graph.get_node(in_node_name) - out_node = self.graph.get_node(out_node_name) - index = in_node.outputs.index(out_node_name) - del in_node.outputs[index] - index = out_node.inputs.index(in_node_name) - del out_node.inputs[index] - self.omit_nodes.append(in_node.layer_name) - def directly_map(self, node): assert node.layer_type in self.directly_map_ops op_info = self.directly_map_ops[node.layer_type] - input = self.graph.get_node(node.layer.input[0], copy=True) + input = self.graph.get_node(node.layer.input[0]) attr = dict() for param in op_info[1:]: tf_param_name = list(param.keys())[0] @@ -127,128 +134,35 @@ class TFOpMapperNHWC(OpMapper): tf_param = node.get_attr(tf_param_name) attr[pd_param_name] = tf_param - if len(input.out_shapes[0]) == 4 and op_info[0] != 'shape': - attr1 = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - 'transpose', inputs=input, output=node, param_attr=attr1) - input = node - node.fluid_code.add_layer( - op_info[0], inputs=input, output=node, param_attr=attr) - input = node - attr2 = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - 'transpose', inputs=input, output=node, param_attr=attr2) - else: - node.fluid_code.add_layer( - op_info[0], inputs=input, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.layers.{}".format(op_info[0]), + inputs={"x": input.name}, + outputs=[node.name], + **attr) def elementwise_map(self, node): assert node.layer_type in self.elementwise_ops op_type = self.elementwise_ops[node.layer_type] - x = self.graph.get_node(node.layer.input[0], copy=True) - y = self.graph.get_node(node.layer.input[1], copy=True) - x_shape = x.out_shapes[0] - y_shape = y.out_shapes[0] - if len(x_shape) == 0: - x_shape = [1] - if len(y_shape) == 0: - y_shape = [1] - # incomplement broadcasting support for paddle - x_input = x - y_input = y - if len(x_shape) < len(y_shape): - unrevertable_ops = [ - "elementwise_sub", "elementwise_div", "elementwise_floordiv", - "elementwise_mod", "elementwise_pow" - ] - if op_type not in unrevertable_ops: - x_input = y - y_input = x - x_shape = y.out_shapes[0] - if len(x_shape) == 0: - x_shape = [1] - y_shape = x.out_shapes[0] - if len(y_shape) == 0: - y_shape = [1] - else: - raise Exception("Unexpected situation happend") - - if len(x_shape) == 4 and len(y_shape) == 1: - inputs = {"x": x_input, "y": y_input} - node.fluid_code.add_layer(op_type, inputs=inputs, output=node) - return - - is_sub_seq = True - for i in range(len(y_shape)): - index = -1 * i - 1 - if y_shape[index] != x_shape[index]: - is_sub_seq = False - if not is_sub_seq: - x_expand_times = [1] * len(x_shape) - y_expand_times = [1] * len(y_shape) - x_need_expand = False - y_need_expand = False - for i in range(len(y_shape)): - index = -1 * i - 1 - if y_shape[index] != x_shape[index]: - if y_shape[index] == 1: - y_expand_times[index] = x_shape[index] - y_need_expand = True - elif x_shape[index] == 1: - x_expand_times[index] = y_shape[index] - x_need_expand = True - else: - raise Exception("Unexpected situation happend") - if x_need_expand: - attr = {"expand_times": x_expand_times} - node.fluid_code.add_layer( - "expand", inputs=x_input, output="x_tmp", param_attr=attr) - x_input = "x_tmp" - if y_need_expand: - attr = {"expand_times": y_expand_times} - node.fluid_code.add_layer( - "expand", inputs=y_input, output="y_tmp", param_attr=attr) - y_input = "y_tmp" - if len(x_shape) == 4 and len(y_shape) == 4: - node.fluid_code.add_layer( - "transpose", - inputs=x_input, - output=x_input, - param_attr={'perm': [0, 3, 1, 2]}) - node.fluid_code.add_layer( - "transpose", - inputs=y_input, - output=y_input, - param_attr={'perm': [0, 3, 1, 2]}) - inputs = {"x": x_input, "y": y_input} - node.fluid_code.add_layer( - op_type, inputs=inputs, output=node, param_attr=None) - node.fluid_code.add_layer( - "transpose", - inputs=node, - output=node, - param_attr={'perm': [0, 2, 3, 1]}) - else: - inputs = {"x": x_input, "y": y_input} - node.fluid_code.add_layer( - op_type, inputs=inputs, output=node, param_attr=None) + x = self.graph.get_node(node.layer.input[0]) + y = self.graph.get_node(node.layer.input[1]) + program.add_layer( + kernel="fluid.layers.{}".format(op_type), + inputs={"x": x.name, + "y": y.name}, + outputs=[node.name]) def Placeholder(self, node): shape = node.out_shapes[0] assert len(shape) != 0, "Unknown shape of input nodes[{}].".format( node.layer_name) dtype = node.dtype - if shape[0] < 0: - self.batch_node = node - attr = { - 'dtype': string(dtype), - 'shape': shape, - 'name': string(node.layer_name), - 'append_batch_size': False - } - - node.fluid_code.add_layer( - "data", inputs=None, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.data", + inputs={}, + outputs=[node.name], + dtype=string(dtype), + shape=shape, + name=string(node.name)) def Const(self, node): shape = node.out_shapes[0] @@ -260,477 +174,475 @@ class TFOpMapperNHWC(OpMapper): shape = [1] initializer = "Constant({})".format(value) - self.weights[node.layer_name] = node.value - - attr = { - 'dtype': string(dtype), - 'shape': shape, - 'name': string(node.layer_name), - 'default_initializer': initializer - } - node.fluid_code.add_layer( - "create_parameter", inputs=None, output=node, param_attr=attr) + program.parameters[node.name] = node.value + program.add_layer( + kernel="fluid.layers.create_parameter", + inputs={}, + outputs=[node.name], + dtype=string(dtype), + shape=shape, + name=string(node.name), + default_initializer=initializer) def Transpose(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - perm = self.graph.get_node(node.layer.input[1], copy=True) + input = self.graph.get_node(node.layer.input[0]) + perm = self.graph.get_node(node.layer.input[1]) assert perm.layer_type == "Const", "Perm of transpose OP should be Const" - del self.weights[perm.layer_name.replace('/', '_')] - perm.fluid_code.clear() perm = perm.value.tolist() - attr = {'perm': perm} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[node.name], + perm=perm) + + def Fill(self, node): + dims = self.graph.get_node(node.layer.input[0], copy=True) + input_value = self.graph.get_node(node.layer.input[1], copy=True) + assert input_value.layer_type == "Const", "Value of fill OP should be Const" + + input_value = input_value.value + input_dtype = string(input_value.dtype) + program.add_layer( + "fluid.layers.fill_constant", + inputs={}, + outputs=[node.name], + shape=dims, + dtype=string(input_dtype), + value=input_value) + + def DepthToSpace(self, node): + input = self.graph.get_node(node.layer.input[0], copy=True) + + block_size = node.get_attr("block_size") + data_format = node.get_attr("data_format").decode() + n, h, w, c = input.out_shapes[0] + + input_name = input.name + if data_format == "NHWC": + transpose_name = gen_name("depth_to_space", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + input_name = transpose_name + + shape = [0, block_size * block_size, -1, h, w] + reshape_name = gen_name("depth_to_space", "reshape") + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": input_name}, + outputs=[reshape_name], + shape=shape) + + transpose_name = gen_name("depth_to_space", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": reshape_name}, + outputs=[transpose_name], + perm=[0, 2, 1, 3, 4]) + + reshape_name = gen_name("depth_to_space", "reshape") + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": transpose_name}, + outputs=[reshape_name], + shape=[0, c, h, w]) + + program.add_layer( + kernel="fluid.layers.pixed_shuffle", + inputs={"input": reshape_name}, + outputs=[node.name], + upscale_factor=block_size) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) def MaxPool(self, node): input = self.graph.get_node(node.layer.input[0], copy=True) - in_shape = input.out_shapes[0] - if in_shape.count(-1) > 2: - in_shape = self.decoder.infer_tensor(input).shape - k_size = node.get_attr("ksize") strides = node.get_attr("strides") data_format = node.get_attr("data_format").decode() pad_mode = node.get_attr("padding").decode() - channel_first = data_format == "NCHW" - if not channel_first: - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - in_shape = [in_shape[i] for i in [0, 3, 1, 2]] + input_name = input.name + if data_format == "NHWC": + transpose_name = gen_name("max_pool", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) strides = [strides[i] for i in [0, 3, 1, 2]] k_size = [k_size[i] for i in [0, 3, 1, 2]] - input = node - - attr = { - "pool_size": k_size[2:4], - "pool_type": string("max"), - "pool_stride": strides[2:4], - "pool_padding": string(pad_mode) - } - node.fluid_code.add_layer( - "pool2d", inputs=input, output=node, param_attr=attr) - - if not channel_first: - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) + input_name = transpose_name + + program.add_layer( + kernel="fluid.layers.pool2d", + inputs={"input": input_name}, + outputs=[node.name], + pool_size=k_size[2:4], + pool_type=string("max"), + pool_stride=strides[2:4], + pool_padding=string(pad_mode)) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) def Conv2D(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - kernel = self.graph.get_node(node.layer.input[1], copy=True) - self.add_omit_nodes(kernel.layer_name, node.layer_name) + input = self.graph.get_node(node.layer.input[0]) + kernel = self.graph.get_node(node.layer.input[1]) - in_shape = input.out_shapes[0] - if in_shape.count(-1) > 2: - in_shape = self.decoder.infer_tensor(input).shape k_size = kernel.out_shapes[0] - if k_size.count(-1) > 2: - k_size = self.decoder.infer_tensor(kernel).shape - strides = node.get_attr("strides") dilations = node.get_attr("dilations") data_format = node.get_attr("data_format").decode() pad_mode = node.get_attr("padding").decode() - channel_first = data_format == "NCHW" if kernel.layer_type == 'Const': kernel_value = kernel.value + kernel_weight_name = kernel.layer_name.replace('/', '_') else: kernel_value = self.decoder.infer_tensor(kernel) - self.weights[kernel.layer_name.replace('/', '_')] = numpy.transpose( - kernel_value, (3, 2, 0, 1)) + if kernel.layer_type == 'Split': + kernel_weight_name = "{}_{}_kernel".format(node.layer_name, + kernel.layer_name) + else: + kernel_weight_name = kernel.layer_name.replace('/', '_') + program.parameters[kernel_weight_name] = numpy.transpose(kernel_value, + (3, 2, 0, 1)) - if not channel_first: - in_shape = [in_shape[i] for i in [0, 3, 1, 2]] + input_name = input.name + if data_format == "NHWC": strides = [strides[i] for i in [0, 3, 1, 2]] dilations = [dilations[i] for i in [0, 3, 1, 2]] - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - input = node - - attr = { - "bias_attr": False, - "param_attr": string(kernel.layer_name), - "num_filters": k_size[3], - "filter_size": k_size[0:2], - "stride": strides[2:4], - "dilation": dilations[2:4], - "padding": string(pad_mode) - } - - if hasattr(node, 'dilation') and attr['dilation'] == [1, 1]: - if len(node.dilation) == 1: - attr['dilation'] = [1, node.dilation[0]] - - node.fluid_code.add_layer( - "conv2d", inputs=input, output=node, param_attr=attr) - if not channel_first: - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) + transpose_name = gen_name("conv2d", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + input_name = transpose_name + + program.add_layer( + kernel="fluid.layers.conv2d", + inputs={"input": input_name}, + outputs=[node.name], + bias_attr=False, + param_attr=string(kernel_weight_name), + num_filters=k_size[3], + filter_size=k_size[0:2], + stride=strides[2:4], + dilation=dilations[2:4], + padding=string(pad_mode)) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) def BiasAdd(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - bias = self.graph.get_node(node.layer.input[1], copy=True) - inputs = {"x": input, "y": bias} - node.fluid_code.add_layer( - "elementwise_add", inputs=inputs, output=node, param_attr=None) + input = self.graph.get_node(node.layer.input[0]) + bias = self.graph.get_node(node.layer.input[1]) + program.add_layer( + kernel="fluid.layers.elementwise_add", + inputs={"x": input.name, + "y": bias.name}, + outputs=[node.name]) def FusedBatchNorm(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - gamma = self.graph.get_node(node.layer.input[1], copy=True) - beta = self.graph.get_node(node.layer.input[2], copy=True) - moving_mean = self.graph.get_node(node.layer.input[3], copy=True) - moving_var = self.graph.get_node(node.layer.input[4], copy=True) + input = self.graph.get_node(node.layer.input[0]) + gamma = self.graph.get_node(node.layer.input[1]) + beta = self.graph.get_node(node.layer.input[2]) + moving_mean = self.graph.get_node(node.layer.input[3]) + moving_var = self.graph.get_node(node.layer.input[4]) data_format = node.get_attr("data_format").decode() - channel_first = data_format == "NCHW" assert gamma.layer_type == "Const" assert beta.layer_type == "Const" assert moving_mean.layer_type == "Const" assert moving_var.layer_type == "Const" - self.add_omit_nodes(gamma.layer_name, node.layer_name) - self.add_omit_nodes(beta.layer_name, node.layer_name) - self.add_omit_nodes(moving_mean.layer_name, node.layer_name) - self.add_omit_nodes(moving_var.layer_name, node.layer_name) - - if not channel_first: - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - input = node - - attr = { - "epsilon": node.get_attr("epsilon"), - "param_attr": string(gamma.layer_name), - "bias_attr": string(beta.layer_name), - "moving_mean_name": string(moving_mean.layer_name), - "moving_variance_name": string(moving_var.layer_name), - "is_test": True - } - - node.fluid_code.add_layer( - "batch_norm", inputs=input, output=node, param_attr=attr) - - if not channel_first: - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) + + input_name = input.name + if data_format == "NHWC": + transpose_name = gen_name("batch_norm", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + input_name = transpose_name + + program.add_layer( + kernel="fluid.layers.batch_norm", + inputs={"input": input_name}, + outputs=[node.name], + epsilon=node.get_attr("epsilon"), + param_attr=string(gamma.name), + bias_attr=string(beta.name), + moving_mean_name=string(moving_mean.name), + moving_variance_name=string(moving_var.name), + is_test=True) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) + + def Mean(self, node): + input = self.graph.get_node(node.layer.input[0]) + reduce_idx = self.graph.get_node(node.layer.input[1]) + assert reduce_idx.layer_type == "Const", "Only support Const parameter[reduce_idx]" + dims = reduce_idx.value.tolist() + keep_dims = node.get_attr("keep_dims") + + program.add_layer( + kernel="fluid.layers.reduce_mean", + inputs={"input": input.name}, + outputs=[node.name], + dim=dims, + keep_dim=keep_dims) + + def Reshape(self, node): + input = self.graph.get_node(node.layer.input[0]) + param = self.graph.get_node(node.layer.input[1]) + if param.layer_type == "Const": + shape = param.value.tolist() + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": input.name}, + outputs=[node.name], + shape=shape) + else: + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": input.name, + "shape": param.name}, + outputs=[node.name]) + if param.layer_type != "Const": + out_shape = numpy.array(node.out_shapes[0]) + if (out_shape > 0).any(): + out_shape[out_shape < 0] = 0 + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": node.name}, + outputs=[node.name], + shape=out_shape.tolist()) + + def Pad(self, node): + input = self.graph.get_node(node.layer.input[0]) + paddings = self.graph.get_node(node.layer.input[1]) + assert paddings.layer_type == "Const", "Padding should be Const" + paddings = paddings.value.flatten().tolist() + + if len(input.out_shapes[0]) == 4: + if paddings[0] + paddings[1] + paddings[6] + paddings[7] == 0: + new_padding = paddings[2:6] + transpose_name = gen_name("pad", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + program.add_layer( + kernel="fluid.layers.pad2d", + inputs={"input": transpose_name}, + outputs=[node.name], + paddings=new_padding) + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) + return + + program.add_layer( + kernel="fluid.layers.pad", + inputs={"input": input.name}, + outputs=[node.name], + paddings=paddings) + + def Squeeze(self, node): + input = self.graph.get_node(node.layer.input[0]) + squeeze_dims = node.get_attr('squeeze_dims') + program.add_layer( + kernel="fluid.layers.squeeze", + inputs={"input": input.name}, + outputs=[node.name], + axes=squeeze_dims) + + def Softmax(self, node): + input = self.graph.get_node(node.layer.input[0]) + axis = node.get_attr("axis") + program.add_layer( + kernel="fluid.layers.softmax", + inputs={"input": input.name}, + outputs=[node.name], + axis=axis) + + def Shape(self, node): + input = self.graph.get_node(node.layer.input[0]) + program.add_layer( + kernel="fluid.layers.shape", + inputs={"input": input.name}, + outputs=[node.name]) + + def ArgMax(self, node): + input = self.graph.get_node(node.layer.input[0]) + axis = self.graph.get_node(node.layer.input[1]) + assert axis.layer_type == "Const", "ArgMax only support Const parameter" + axis = axis.value + program.add_layer( + kernel="fluid.layers.argmax", + inputs={"x": input.name}, + outputs=[node.name], + axis=axis) + + def MatMul(self, node): + x = self.graph.get_node(node.layer.input[0]) + y = self.graph.get_node(node.layer.input[1]) + transpose_a = node.get_attr('transpose_a') + transpose_b = node.get_attr('transpose_b') + if transpose_a is None: + transpose_a = node.get_attr('adj_x') + if transpose_b is None: + transpose_b = node.get_attr('adj_y') + program.add_layer( + kernel="fluid.layers.matmul", + inputs={"x": x.name, + "y": y.name}, + outputs=[node.name], + transpose_x=transpose_a, + transpose_y=transpose_b) def DepthwiseConv2dNative(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - kernel = self.graph.get_node(node.layer.input[1], copy=True) + input = self.graph.get_node(node.layer.input[0]) + kernel = self.graph.get_node(node.layer.input[1]) assert kernel.layer_type == "Const", "Kernel of DepthwiseConv2DNative should be Const" - self.add_omit_nodes(kernel.layer_name, node.layer_name) in_shape = input.out_shapes[0] - if in_shape.count(-1) > 2: - in_shape = self.decoder.infer_tensor(input).shape k_size = kernel.out_shapes[0] - if k_size.count(-1) > 2: - k_size = self.decoder.infer_tensor(kernel).shape - strides = node.get_attr("strides") dilations = node.get_attr("dilations") data_format = node.get_attr("data_format").decode() pad_mode = node.get_attr("padding").decode() - channel_first = data_format == "NCHW" - self.weights[kernel.layer_name.replace('/', '_')] = numpy.transpose( - kernel.value, (2, 3, 0, 1)) + program.parameters[kernel.layer_name.replace( + '/', '_')] = numpy.transpose(kernel.value, (2, 3, 0, 1)) - if not channel_first: + input_name = input.name + if data_format == "NHWC": in_shape = [in_shape[i] for i in [0, 3, 1, 2]] strides = [strides[i] for i in [0, 3, 1, 2]] dilations = [dilations[i] for i in [0, 3, 1, 2]] - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - input = node - - attr = { - "bias_attr": False, - "param_attr": string(kernel.layer_name), - "num_filters": in_shape[1], - "filter_size": k_size[0:2], - "stride": strides[2:4], - "dilation": dilations[2:4], - "groups": k_size[3] * in_shape[1], - "use_cudnn": False, - "padding": string(pad_mode) - } - node.fluid_code.add_layer( - "conv2d", inputs=input, output=node, param_attr=attr) - - if not channel_first: - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) - - def Reshape(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - param = self.graph.get_node(node.layer.input[1], copy=True) - is_variable = False - if param.layer_type == "Const": - attr = {"shape": param.value.tolist()} - self.add_omit_nodes(param.layer_name, node.layer_name) - else: - # Here is a trick method to solove tensor parameter in tensorflow - shape = self.decoder.infer_shape_tensor(param, node.out_shapes[0]) - if shape.count(-1) <= 1: - attr = {"shape": shape} - self.add_omit_nodes(param.layer_name, node.layer_name) - else: - assert len(param.out_shapes[ - 0]) == 1, "Unexpected situation of shape parameter" - attr = {"shape": [-1]} - node.fluid_code.add_layer( - "reshape", - inputs=param, - output="shape_param", - param_attr=attr) - attr = {"num_or_sections": param.out_shapes[0][0], "dim": 0} - node.fluid_code.add_layer( - "split", inputs="shape_param", output=node, param_attr=attr) - new_param = "[" - for i in range(param.out_shapes[0][0]): - new_param += (node.layer_name + "[{}]".format(i) + ", ") - new_param = new_param.strip(", ") + "]" - attr = {"shape": new_param} - is_variable = True - # to change [192, -1]->[-1, 192], allways put -1 in the first dimension - # optimization for Paddle-Lite - in_shape = input.out_shapes[0] - if not is_variable and in_shape.count(-1) < 1: - total_size = 1 - for i in range(len(in_shape)): - total_size *= in_shape[i] - for i in range(len(attr["shape"])): - if attr["shape"][i] == 0: - attr["shape"][i] = in_shape[i] - if attr["shape"][i] != -1: - total_size /= attr["shape"][i] - if attr["shape"].count(-1) > 0: - index = attr["shape"].index(-1) - attr["shape"][index] = int(total_size) - attr["shape"][0] = -1 - - node.fluid_code.add_layer( - "reshape", inputs=input, output=node, param_attr=attr) + transpose_name = gen_name('depthwise_conv2d', 'transpose') + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + input_name = transpose_name + + program.add_layer( + kernel="fluid.layers.conv2d", + inputs={"input": input_name}, + outputs=[node.name], + num_filters=in_shape[1], + filter_size=k_size[0:2], + stride=strides[2:4], + dilation=dilations[2:4], + groups=k_size[3] * in_shape[1], + padding=string(pad_mode), + param_attr=string(kernel.layer_name), + bias_attr=False) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) def AvgPool(self, node): input = self.graph.get_node(node.layer.input[0], copy=True) - in_shape = input.out_shapes[0] - if in_shape.count(-1) > 2: - in_shape = self.decoder.infer_tensor(input).shape - k_size = node.get_attr("ksize") strides = node.get_attr("strides") data_format = node.get_attr("data_format").decode() pad_mode = node.get_attr("padding").decode() - channel_first = data_format == "NCHW" - if not channel_first: - in_shape = [in_shape[i] for i in [0, 3, 1, 2]] + input_name = input.name + if data_format == "NHWC": + transpose_name = gen_name("avg_pool", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) strides = [strides[i] for i in [0, 3, 1, 2]] k_size = [k_size[i] for i in [0, 3, 1, 2]] - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - input = node - - attr = { - "pool_size": k_size[2:4], - "pool_type": string("avg"), - "pool_stride": strides[2:4], - "pool_padding": string(pad_mode) - } - node.fluid_code.add_layer( - "pool2d", inputs=input, output=node, param_attr=attr) - - if not channel_first: - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) - - def SplitV(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - num_sections = self.graph.get_node(node.layer.input[1], copy=True) - dim = self.graph.get_node(node.layer.input[2], copy=True) - assert num_sections.layer_type == "Const" - assert dim.layer_type == "Const" - self.add_omit_nodes(num_sections.layer_name, node.layer_name) - self.add_omit_nodes(dim.layer_name, node.layer_name) - dim = dim.value - attr = { - "num_or_sections": num_sections.value.tolist(), - "dim": dim.value - } - node.fluid_code.add_layer( - "split", inputs=input, output=node, param_attr=attr) - - def ConcatV2(self, node): - inputs = [ - self.graph.get_node( - name, copy=True) for name in node.layer.input[:-1] - ] - axis = self.graph.get_node(node.layer.input[-1], copy=True) - assert axis.layer_type == "Const" - self.add_omit_nodes(axis.layer_name, node.layer_name) - axis = axis.value - if axis < 0: - axis += len(inputs[0].out_shapes[0]) - - attr = {"axis": axis} - node.fluid_code.add_layer( - "concat", inputs=inputs, output=node, param_attr=attr) - - def Tile(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - expand_times = self.graph.get_node(node.layer.input[1], copy=True) - self.add_omit_nodes(expand_times.layer_name, node.layer_name) - if expand_times.layer_type == "Const": - expand_times = expand_times.value.tolist() - else: - expand_times = self.decoder.infer_shape_tensor(expand_times) - for i in range(len(expand_times)): - if expand_times[i] < 0: - expand_times[i] = 1 - attr = {"expand_times": expand_times} - node.fluid_code.add_layer( - "expand", inputs=input, output=node, param_attr=attr) + input_name = transpose_name + + program.add_layer( + kernel="fluid.layers.pool2d", + inputs={"input": input_name}, + outputs=[node.name], + pool_size=k_size[2:4], + pool_type=string("avg"), + pool_stride=strides[2:4], + pool_padding=string(pad_mode)) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) def Pack(self, node): - inputs = [ - self.graph.get_node( - name, copy=True) for name in node.layer.input - ] + inputs = [self.graph.get_node(name) for name in node.layer.input] axis = node.get_attr("axis") - attr = {"axis": axis} - node.fluid_code.add_layer( - "stack", inputs=inputs, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.layers.stack", + inputs={"x": [i.name for i in inputs]}, + outputs=[node.name], + axis=axis) - def Pad(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - paddings = self.graph.get_node(node.layer.input[1], copy=True) - assert paddings.layer_type == "Const", "Padding should be Const" - self.add_omit_nodes(paddings.layer_name, node.layer_name) - paddings = paddings.value.flatten().tolist() - data_format = input.tf_data_format - - if len(input.out_shapes[0]) == 4: - new_padding = None - if input.tf_data_format == "NHWC": - if paddings[0] + paddings[1] + paddings[6] + paddings[7] == 0: - new_padding = paddings[2:6] - else: - if paddings[0] + paddings[1] + paddings[2] + paddings[3] == 0: - new_padding = paddings[4:] - if new_padding is not None: - if input.tf_data_format == "NHWC": - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - input = node - attr = {"paddings": new_padding} - node.fluid_code.add_layer( - "pad2d", inputs=input, output=node, param_attr=attr) - if input.tf_data_format == "NHWC": - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) - - return - - attr = {"paddings": paddings} - node.fluid_code.add_layer( - "pad", inputs=input, output=node, param_attr=attr) - - def Range(self, node): - start = self.graph.get_node(node.layer.input[0], copy=True) - limit = self.graph.get_node(node.layer.input[1], copy=True) - delta = self.graph.get_node(node.layer.input[2], copy=True) - self.add_omit_nodes(start.layer_name, node.layer_name) - self.add_omit_nodes(limit.layer_name, node.layer_name) - self.add_omit_nodes(delta.layer_name, node.layer_name) - if start.layer_type == "Const": - start = start.value - else: - start = self.decoder.infer_tensor(start) - if limit.layer_type == "Const": - limit = limit.value - else: - limit = self.decoder.infer_tensor(limit) - if delta.layer_type == "Const": - delta = delta.value - else: - delta = self.decoder.infer_tensor(delta) - dtype = node.dtype - inputs = { - "start": start, - "end": limit, - "step": delta, - } - attr = {"dtype": string(node.dtype)} - node.fluid_code.add_layer( - "range", inputs=inputs, output=node, param_attr=attr) - - def Mean(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - reduce_idx = self.graph.get_node(node.layer.input[1], copy=True) - assert reduce_idx.layer_type == "Const", "Only support Const parameter[reduce_idx]" - dims = reduce_idx.value.tolist() - keep_dims = node.get_attr("keep_dims") - - attr = {"dim": dims, "keep_dim": keep_dims} - node.fluid_code.add_layer( - "reduce_mean", inputs=input, output=node, param_attr=attr) - - def MatMul(self, node): - x = self.graph.get_node(node.layer.input[0], copy=True) - y = self.graph.get_node(node.layer.input[1], copy=True) - transpose_a = node.get_attr('transpose_a') - transpose_b = node.get_attr('transpose_b') - inputs = {"x": x, "y": y} - # fix paddle shape infer problem - # should be removed after paddle 1.6 - if x.out_shapes[0][-1] < 0 and y.out_shapes[0][0] > 0: - shape = x.out_shapes[0] - shape[-1] = y.out_shapes[0][0] - attr = {"shape": shape} - node.fluid_code.add_layer( - "reshape", inputs=x, output=x, param_attr=attr) - attr = {"transpose_x": transpose_a, "transpose_y": transpose_b} - node.fluid_code.add_layer( - "matmul", inputs=inputs, output=node, param_attr=attr) - - def ArgMax(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - axis = self.graph.get_node(node.layer.input[1], copy=True) - assert axis.layer_type == "Const", "ArgMax only support Const parameter" - self.add_omit_nodes(axis.layer_name, node.layer_name) + def ConcatV2(self, node): + inputs = [self.graph.get_node(name) for name in node.layer.input[:-1]] + axis = self.graph.get_node(node.layer.input[-1]) + assert axis.layer_type == "Const", "axis for ConcatV2 must be type Const" axis = axis.value - attr = {"axis": axis} - node.fluid_code.add_layer( - "argmax", inputs=input, output=node, param_attr=attr) + if axis < 0: + axis += len(inputs[0].out_shapes[0]) + program.add_layer( + kernel="fluid.layers.concat", + inputs={"input": [i.name for i in inputs]}, + outputs=[node.name], + axis=axis) def StridedSlice(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - begin = self.graph.get_node(node.layer.input[1], copy=True) - end = self.graph.get_node(node.layer.input[2], copy=True) - strides = self.graph.get_node(node.layer.input[3], copy=True) + input = self.graph.get_node(node.layer.input[0]) + begin = self.graph.get_node(node.layer.input[1]) + end = self.graph.get_node(node.layer.input[2]) + strides = self.graph.get_node(node.layer.input[3]) assert begin.layer_type == "Const" assert end.layer_type == "Const" assert strides.layer_type == "Const" - self.add_omit_nodes(begin.layer_name, node.layer_name) - self.add_omit_nodes(end.layer_name, node.layer_name) - self.add_omit_nodes(strides.layer_name, node.layer_name) strides = strides.value.tolist() assert len(set(strides)) == 1 and strides[ 0] == 1, "Only support strides be 1 in StridedSlice OP" @@ -779,65 +691,228 @@ class TFOpMapperNHWC(OpMapper): else: new_end.append(end[i]) - attr = { - "axes": [i for i in range(len(new_begin))], - "starts": new_begin, - "ends": new_end - } - node.fluid_code.add_layer( - "slice", inputs=input, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.layers.slice", + inputs={"input": input.name}, + outputs=[node.name], + axes=[i for i in range(len(new_begin))], + starts=new_begin, + ends=new_end) if len(new_axes) > 0: - attr = {"axes": new_axes} - node.fluid_code.add_layer( - "unsqueeze", inputs=node, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.layers.unsqueeze", + inputs={"x": node.name}, + outputs=[node.name], + axes=new_axes) if len(shrink_axes) > 0: if len(input.out_shapes[0]) + len(new_axes) <= 1: pass else: - attr = {"axes": shrink_axes} - node.fluid_code.add_layer( - "squeeze", inputs=node, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.layers.unsqueeze", + inputs={"x": node.name}, + outputs=[node.name], + axes=new_axes) + + def Split(self, node): + dim = self.graph.get_node(node.layer.input[0]) + input = self.graph.get_node(node.layer.input[1]) + assert dim.layer_type == "Const" + num_split = node.get_attr('num_split') + dim = dim.value + + program.add_layer( + kernel="fluid.layers.split", + inputs={"input": input.name}, + outputs=[ + "{}_p{}".format(node.layer_name, i) for i in range(num_split) + ], + num_or_sections=num_split, + dim=dim) def Slice(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - begin = self.graph.get_node(node.layer.input[1], copy=True) - size = self.graph.get_node(node.layer.input[2], copy=True) - self.add_omit_nodes(begin.layer_name, node.layer_name) - self.add_omit_nodes(size.layer_name, node.layer_name) + input = self.graph.get_node(node.layer.input[0]) + begin = self.graph.get_node(node.layer.input[1]) + size = self.graph.get_node(node.layer.input[2]) + + inputs = {"x": input.name} + attrs = {} if begin.layer_type == "Const": begin = begin.value.tolist() + attrs['offsets'] = begin else: - begin = self.decoder.infer_tensor(begin).tolist() - if size.layer_type == "const": + shape = begin.out_shapes[0] + reshape_name = gen_name("slice", "reshape") + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": begin.name}, + outputs=[reshape_name], + shape=shape) + inputs['offsets'] = reshape_name + if size.layer_type == "Const": size = size.value.tolist() + attrs['shape'] = size else: - size = self.decoder.infer_tensor(size).tolist() + shape = size.out_shapes[0] + reshape_name = gen_name("slice", "reshape") + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": size.name}, + outputs=[reshape_name], + shape=shape) + inputs['shape'] = reshape_name + program.add_layer( + kernel="fluid.layers.crop_tensor", + inputs=inputs, + outputs=[node.name], + **attrs) - for i in range(len(size)): - if size[i] < 0: - size[i] = 99999999 - else: - size[i] = size[i] + begin[i] + def ResizeNearestNeighbor(self, node): + input = self.graph.get_node(node.layer.input[0]) + resize_shape = self.graph.get_node(node.layer.input[1]) + data_format = "NHWC" + inputs = {"input": input.name} + attrs = {"align_corners": node.get_attr("align_corners")} + + if resize_shape.layer_type == "Const": + resize_shape = resize_shape.value.tolist() + attrs["out_shape"] = resize_shape + else: + shape = resize_shape.out_shapes[0] + reshape_name = gen_name("resize_nearest", "reshape") + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": resize_shape.name}, + outputs=[reshape_name], + shape=shape) + inputs["out_shape"] = reshape_name + + if data_format == "NHWC": + transpose_name = gen_name("resize_nearest", "reshape") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + inputs["input"] = transpose_name + + program.add_layer( + kernel="fluid.layers.resize_nearest", + inputs=inputs, + outputs=[node.name], + **attrs) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) + + def ResizeBilinear(self, node): + input = self.graph.get_node(node.layer.input[0]) + resize_shape = self.graph.get_node(node.layer.input[1]) + data_format = "NHWC" + inputs = {"input": input.name} + attrs = {"align_corners": node.get_attr("align_corners")} + + if resize_shape.layer_type == "Const": + resize_shape = resize_shape.value.tolist() + attrs["out_shape"] = resize_shape + else: + shape = resize_shape.out_shapes[0] + reshape_name = gen_name("resize_bilinear", "reshape") + program.add_layer( + kernel="fluid.layers.reshape", + inputs={"x": resize_shape.name}, + outputs=[reshape_name], + shape=shape) + inputs["out_shape"] = reshape_name + + if data_format == "NHWC": + transpose_name = gen_name("resize_bilinear", "reshape") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + inputs["input"] = transpose_name + + program.add_layer( + kernel="fluid.layers.resize_bilinear", + inputs=inputs, + outputs=[node.name], + **attrs) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) - attr = { - "axes": [i for i in range(len(size))], - "starts": begin, - "ends": size - } + def Cast(self, node): + input = self.graph.get_node(node.layer.input[0]) + dtype = node.dtype + program.add_layer( + kernel="fluid.layers.cast", + inputs={"x": input.name}, + outputs=[node.name], + dtype=string(dtype)) + + def Sum(self, node): + input = self.graph.get_node(node.layer.input[0]) + reduce_idx = self.graph.get_node(node.layer.input[1]) + assert reduce_idx.layer_type == "Const", "Only support Const parameter[reduce_idx]" + keep_dims = node.get_attr("keep_dims") + dim = reduce_idx.value.tolist() - node.fluid_code.add_layer( - "slice", inputs=input, output=node, param_attr=attr) + program.add_layer( + kernel="fluid.layers.reduce_sum", + inputs={"input": input.name}, + outputs=[node.name], + dim=dim, + keep_dim=keep_dims) + + def Max(self, node): + input = self.graph.get_node(node.layer.input[0]) + reduce_idx = self.graph.get_node(node.layer.input[1]) + assert reduce_idx.layer_type == "Const", "Only support Const parameter[reduce_idx]" + keep_dims = node.get_attr("keep_dims") + dim = reduce_idx.value.tolist() + program.add_layer( + kernel="fluid.layers.reduce_max", + inputs={"input": input.name}, + outputs=[node.name], + dim=dim, + keep_dim=keep_dims) + + def RandomUniform(self, node): + shape = self.graph.get_node(node.layer.input[0], copy=True) + if shape.layer_type == "Const": + shape = shape.value.tolist() + program.add_layer( + kernel="fluid.layers.uniform_random", + inputs={}, + outputs=[node.name], + shape=shape, + min=0.0, + max=0.9999) + else: + program.add_layer( + kernel="fluid.layers.uniform_random", + inputs={'shape': shape.name}, + outputs=[node.name], + min=0.0, + max=0.9999) def Conv2DBackpropInput(self, node): - out_shape = self.graph.get_node(node.layer.input[0], copy=True) - kernel = self.graph.get_node(node.layer.input[1], copy=True) - input = self.graph.get_node(node.layer.input[2], copy=True) + out_shape = self.graph.get_node(node.layer.input[0]) + kernel = self.graph.get_node(node.layer.input[1]) + input = self.graph.get_node(node.layer.input[2]) assert kernel.layer_type == "Const", "Kernel of Conv2DBackpropInput should be Const" - self.add_omit_nodes(kernel.layer_name, node.layer_name) - self.add_omit_nodes(out_shape.layer_name, node.layer_name) - if out_shape.layer_type == "Const": out_shape = out_shape.value.tolist() else: @@ -855,201 +930,39 @@ class TFOpMapperNHWC(OpMapper): strides = node.get_attr("strides") dilations = node.get_attr("dilations") data_format = node.get_attr("data_format").decode() - channel_first = data_format == "NCHW" - self.weights[kernel.layer_name.replace('/', '_')] = numpy.transpose( - kernel.value, (3, 2, 0, 1)) - if not channel_first: + program.parameters[kernel.layer_name.replace( + '/', '_')] = numpy.transpose(kernel.value, (3, 2, 0, 1)) + + input_name = input.name + if data_format == "NHWC": in_shape = [in_shape[i] for i in [0, 3, 1, 2]] strides = [strides[i] for i in [0, 3, 1, 2]] dilations = [dilations[i] for i in [0, 3, 1, 2]] - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - input = node - else: - self.data_format_propagation(node) - - attr = { - "bias_attr": False, - "param_attr": string(kernel.layer_name), - "num_filters": k_size[2], - "filter_size": k_size[0:2], - "stride": strides[2:4], - "dilation": dilations[2:4], - "padding": string(pad_mode), - "output_size": out_shape[1:3] - } - node.fluid_code.add_layer( - "conv2d_transpose", inputs=input, output=node, param_attr=attr) - - if not channel_first: - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) - - def Max(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - reduce_idx = self.graph.get_node(node.layer.input[1], copy=True) - assert reduce_idx.layer_type == "Const", "Only support Const parameter[reduce_idx]" - keep_dims = node.get_attr("keep_dims") - dim = reduce_idx.value.tolist() - - attr = {"dim": dim, "keep_dim": keep_dims} - node.fluid_code.add_layer( - "reduce_max", inputs=input, output=node, param_attr=attr) - - def Sum(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - reduce_idx = self.graph.get_node(node.layer.input[1], copy=True) - assert reduce_idx.layer_type == "Const", "Only support Const parameter[reduce_idx]" - keep_dims = node.get_attr("keep_dims") - dim = reduce_idx.value.tolist() - - attr = {"dim": dim, "keep_dim": keep_dims} - node.fluid_code.add_layer( - "reduce_sum", inputs=input, output=node, param_attr=attr) - - def Cast(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - dtype = node.dtype_map[node.get_attr('DstT')] - attr = {"dtype": string(dtype)} - node.fluid_code.add_layer( - "cast", inputs=input, output=node, param_attr=attr) - - def Split(self, node): - dim = self.graph.get_node(node.layer.input[0], copy=True) - input = self.graph.get_node(node.layer.input[1], copy=True) - assert dim.layer_type == "Const" - self.add_omit_nodes(dim.layer_name, node.layer_name) - num_split = node.get_attr('num_split') - dim = dim.value - - attr = {"num_or_sections": num_split, "dim": dim} - node.fluid_code.add_layer( - "split", inputs=input, output=node, param_attr=attr) - - def Squeeze(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - squeeze_dims = node.get_attr('squeeze_dims') - attr = {"axes": squeeze_dims} - node.fluid_code.add_layer( - "squeeze", inputs=input, output=node, param_attr=attr) - - def Softmax(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - axis = node.get_attr("axis") - attr = {"axis": axis} - node.fluid_code.add_layer( - "softmax", inputs=input, output=node, param_attr=attr) - - def ResizeNearestNeighbor(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - resize_shape = self.graph.get_node(node.layer.input[1], copy=True) - self.add_omit_nodes(resize_shape.layer_name, node.layer_name) - if resize_shape.layer_type == "Const": - resize_shape = resize_shape.value.tolist() - else: - resize_shape = self.decoder.infer_shape_tensor(resize_shape, - node.out_shapes[0]) - align_corners = node.get_attr("align_corners") - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - attr = {"align_corners": align_corners, "out_shape": resize_shape} - node.fluid_code.add_layer( - "resize_nearest", inputs=node, output=node, param_attr=attr) - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) - - def ResizeBilinear(self, node): - input = self.graph.get_node(node.layer.input[0], copy=True) - resize_shape = self.graph.get_node(node.layer.input[1], copy=True) - self.add_omit_nodes(resize_shape.layer_name, node.layer_name) - if resize_shape.layer_type == "Const": - resize_shape = resize_shape.value.tolist() - else: - resize_shape = self.decoder.infer_shape_tensor(resize_shape, - node.out_shapes[0]) - align_corners = node.get_attr("align_corners") - attr = {"perm": [0, 3, 1, 2]} - node.fluid_code.add_layer( - "transpose", inputs=input, output=node, param_attr=attr) - attr = { - "align_corners": align_corners, - "out_shape": resize_shape, - "align_mode": 1 - } - node.fluid_code.add_layer( - "resize_bilinear", inputs=node, output=node, param_attr=attr) - attr = {"perm": [0, 2, 3, 1]} - node.fluid_code.add_layer( - "transpose", inputs=node, output=node, param_attr=attr) - - def GreaterEqual(self, node): - x = self.graph.get_node(node.layer.input[0], copy=True) - y = self.graph.get_node(node.layer.input[1], copy=True) - inputs = {"x": x, "y": y} - node.fluid_code.add_layer( - "greater_equal", inputs=inputs, output=node, param_attr=None) - - def RandomUniform(self, node): - shape = self.graph.get_node(node.layer.input[0], copy=True) - self.add_omit_nodes(shape.layer_name, node.layer_name) - if shape.layer_type == "Const": - shape = shape.value.tolist() - else: - shape = self.decoder.infer_shape_tensor(shape) - attr = {"shape": shape, "min": 0.0, "max": 0.9999} - - if shape[0] < 0: - input = self.batch_node - node.fluid_code.add_layer( - "uniform_random_batch_size_like", - inputs=input, - output=node, - param_attr=attr) - else: - node.fluid_code.add_layer( - "uniform_random", inputs=None, output=node, param_attr=attr) - - def SquaredDifference(self, node): - x = self.graph.get_node(node.layer.input[0], copy=True) - y = self.graph.get_node(node.layer.input[1], copy=True) - inputs = {"x": x, "y": y} - node.fluid_code.add_layer( - "elementwise_sub", inputs=inputs, output=node, param_attr=None) - inputs = {"x": node, "y": node} - node.fluid_code.add_layer( - "elementwise_mul", inputs=inputs, output=node, param_attr=None) - - def ExpandDims(self, node): - x = self.graph.get_node(node.layer.input[0], copy=True) - y = self.graph.get_node(node.layer.input[1], copy=True) - if y.layer_type == 'Const': - dim = y.value.tolist() - else: - dim = self.decoder.infer_tensor(y) - self.add_omit_nodes(y.layer_name, node.layer_name) - attr = {'axes': [dim]} - node.fluid_code.add_layer( - "unsqueeze", inputs=x, output=node, param_attr=attr) - - def BatchToSpaceND(self, node): - x = self.graph.get_node(node.layer.input[0], copy=True) - y = self.graph.get_node(node.layer.input[1], copy=True) - if hasattr(node, 'skip') and node.skip: - node.fluid_code.add_layer( - "=", inputs=x, output=node, param_attr=None) - else: - raise Exception("BatchToSpaceND is not supported") - - def SpaceToBatchND(self, node): - x = self.graph.get_node(node.layer.input[0], copy=True) - y = self.graph.get_node(node.layer.input[1], copy=True) - if hasattr(node, 'skip') and node.skip: - node.fluid_code.add_layer( - "=", inputs=x, output=node, param_attr=None) - else: - raise Exception("SpaceToBatchND is not supported") + transpose_name = gen_name("conv2dbackpropinput", "transpose") + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": input.name}, + outputs=[transpose_name], + perm=[0, 3, 1, 2]) + input_name = transpose_name + + program.add_layer( + kernel="fluid.layers.conv2d_transpose", + inputs={"input": input_name}, + outputs=[node.name], + bias_attr=False, + param_attr=string(kernel.layer_name), + num_filters=k_size[2], + filter_size=k_size[0:2], + stride=strides[2:4], + dilation=dilations[2:4], + padding=string(pad_mode), + output_size=out_shape[1:3]) + + if data_format == "NHWC": + program.add_layer( + kernel="fluid.layers.transpose", + inputs={"x": node.name}, + outputs=[node.name], + perm=[0, 2, 3, 1]) diff --git a/x2paddle/optimizer/onnx_optimizer.py b/x2paddle/optimizer/onnx_optimizer.py index a8f851b6c5ea6140c53b91b5d20a6bbf3aa3046f..4059258267310094d238bc01fe6b70d38dc5e3f1 100644 --- a/x2paddle/optimizer/onnx_optimizer.py +++ b/x2paddle/optimizer/onnx_optimizer.py @@ -13,7 +13,6 @@ # limitations under the License. # TODO useless node remove -from x2paddle.op_mapper.onnx_op_mapper import ONNXOpMapper class ONNXOptimizer(object): diff --git a/x2paddle/optimizer/tf_optimizer.py b/x2paddle/optimizer/tf_optimizer.py index 8b63df18017295aefb25155d85e0c1595edec88a..6d3c0cdd017c6d046451e5837d2b75ef649cd6a8 100644 --- a/x2paddle/optimizer/tf_optimizer.py +++ b/x2paddle/optimizer/tf_optimizer.py @@ -236,26 +236,18 @@ class TFOptimizer(object): def remove_transpose(self): graph_copy = cp.deepcopy(self.graph) - nhwc_insensitive_ops = [ - 'Relu', 'Relu6', 'Abs', 'Sigmoid', 'Exp', 'Rsqrt', 'swish_f32', - 'LeakyRelu', 'Cast', 'Tanh' - ] elementwise_ops = [ 'Sub', 'Add', 'RealDiv', 'Maximum', 'Mul', 'FloorDiv', - 'GreaterEqual' - ] - optimize_ops = [ - 'Conv2D', 'MaxPool', 'FusedBatchNorm', 'DepthwiseConv2dNative', - 'AvgPool', 'Pad', 'Conv2DBackpropInput', 'ResizeNearestNeighbor', - 'ResizeBilinear', "Placeholder" + 'GreateerEqual' ] can_be_optimized_ops = [ 'Conv2D', 'MaxPool', 'FusedBatchNorm', 'DepthwiseConv2dNative', 'AvgPool', 'Pad', 'Conv2DBackpropInput', 'ResizeNearestNeighbor', - 'ResizeBilinear', "Placeholder", 'Relu', 'Relu6', 'Abs', 'Sigmoid', - 'Exp', 'Rsqrt', 'swish_f32', 'LeakyRelu', 'Cast', 'Tanh' + 'Placeholder', 'Relu', 'Relu6', 'Abs', 'Sigmoid', 'Exp', 'Rsqrt', + 'swish_f32', 'LeakyRelu', 'Cast', 'Tanh' ] - + # These ops may have one more Variable input + can_be_optimized_special_ops = ['ResizeBilinear'] for node_name in self.graph.topo_sort: node = graph_copy.get_node(node_name) if node is None: @@ -278,9 +270,10 @@ class TFOptimizer(object): 0].param_attr["perm"] != [0, 3, 1, 2]: can_be_removed = False break - elif out_node.layer_type in elementwise_ops: + elif out_node.layer_type in elementwise_ops or out_node.layer_type in can_be_optimized_special_ops: can_be_removed = False break + if can_be_removed and len(node.fluid_code.layers) > 1: true_node = self.graph.get_node(node_name) if true_node.layer_type == "Placeholder": @@ -298,6 +291,7 @@ class TFOptimizer(object): -2].output = true_node.fluid_code.layers[-1].output node.removed = True del true_node.fluid_code.layers[-1] + for out_name in output_names: out_node = self.graph.get_node(out_name) out_node.fluid_code.layers[ diff --git a/x2paddle_model_zoo.md b/x2paddle_model_zoo.md index edcdd173af8277ae3c804e8a438b729418dd5b84..d2a95c735b1a80a9680e0b9b39958bed1b6702f2 100644 --- a/x2paddle_model_zoo.md +++ b/x2paddle_model_zoo.md @@ -1,5 +1,5 @@ # X2Paddle模型测试库 -> 目前X2Paddle支持40+的TensorFlow OP，40+的Caffe Layer，覆盖了大部分CV分类模型常用的操作。我们在如下模型列表中测试了X2Paddle的转换。 +> 目前X2Paddle支持50+的TensorFlow OP，40+的Caffe Layer，覆盖了大部分CV分类模型常用的操作。我们在如下模型列表中测试了X2Paddle的转换。 **注：** 受限于不同框架的差异，部分模型可能会存在目前无法转换的情况，如TensorFlow中包含控制流的模型，NLP模型等。对于CV常见的模型，如若您发现无法转换或转换失败，存在较大diff等问题，欢迎通过[ISSUE反馈](https://github.com/PaddlePaddle/X2Paddle/issues/new)的方式告知我们(模型名，代码实现或模型获取方式)，我们会及时跟进：） @@ -22,7 +22,8 @@ | UNet | [code1](https://github.com/jakeret/tf_unet )/[code2](https://github.com/lyatdawn/Unet-Tensorflow) |-| |MTCNN | [code](https://github.com/AITTSMD/MTCNN-Tensorflow) |-| |YOLO-V3| [code](https://github.com/YunYang1994/tensorflow-yolov3) | 转换需要关闭NHWC->NCHW的优化，见[文档Q2](FAQ.md) | -|Inception_ResNet_V2| [code](https://github.com/tensorflow/models/tree/master/research/slim/nets) | - | +| FALSR | [code](https://github.com/xiaomi-automl/FALSR) | - | +| DCSCN | [code](https://modelzoo.co/model/dcscn-super-resolution) | - | ## Caffe @@ -48,8 +49,8 @@ ## ONNX **注：** 部分模型来源于PyTorch，PyTorch的转换可参考[pytorch_to_onnx.md](pytorch_to_onnx.md) -| 模型 | 来源 | operator version| -|-------|--------|---------| +| 模型 | 来源 | operator version|备注| +|-------|--------|---------|---------| | ResNet18 | [torchvison.model.resnet18](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py) |9| | ResNet34 | [torchvison.model.resnet34](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py) |9| | ResNet50 | [torchvison.model.resnet50](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py) |9| @@ -65,4 +66,6 @@ | mNASNet | [pytorch(personal practice)](https://github.com/rwightman/gen-efficientnet-pytorch) |9| | EfficientNet | [pytorch(personal practice)](https://github.com/rwightman/gen-efficientnet-pytorch) |9| | SqueezeNet | [onnx official](https://s3.amazonaws.com/download.onnx/models/opset_9/squeezenet.tar.gz) |9| -|Ultra-Light-Fast-Generic-Face-Detector-1MB| [onnx_model](https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB/tree/master/models/onnx)| | +|Ultra-Light-Fast-Generic-Face-Detector-1MB| [onnx_model](https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB/tree/master/models/onnx)|9 | +|BERT| [pytorch(huggingface)](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb)|11|转换时需指定input shape，见[文档Q3](FAQ.md)| +|GPT2| [pytorch(huggingface)](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb)|11|转换时需指定input shape，见[文档Q3](FAQ.md)|