Merge pull request #1461 from RainFrost1/whole_chain

完善test_tipc文档

test_tipc/README.md

@@ -24,15 +24,15 @@
 | :--- | :--- |  :----:  | :--------: |  :----  |   :----  |   :----  |
 | ResNet     |ResNet50_vd | 分类  | 支持 | 多机多卡 <br> 混合精度 | FPGM裁剪 <br> PACT量化|  |
 | MobileNetV3     |MobileNetV3_large_x1_0 | 分类  | 支持 | 多机多卡 <br> 混合精度 | FPGM裁剪 <br> PACT量化|  |
-| PPLCNet     |PPLCNet_x2_5 | 分类  | 支持 | 多机多卡 <br> 混合精度 | FPGM裁剪 <br> PACT量化|  |
+| PPLCNet     |PPLCNet_x2_5 | 分类  | 支持 | 多机多卡 <br> 混合精度 | - |  |
 
-## 3. 一键测试工具使用
+## 3. 测试工具简介
 ### 目录介绍
 ```
 ./test_tipc/
 ├── common_func.sh                      #test_*.sh会调用到的公共函数
 ├── config     # 配置文件目录
-│   ├── MobileNetV3_large_x1_0   # MobileNetV3系列模型测试配置文件目录
+│   ├── MobileNetV3         # MobileNetV3系列模型测试配置文件目录
 │   │   ├── MobileNetV3_large_x1_0_train_infer_python.txt                                    #基础训练预测配置文件
 │   │   ├── MobileNetV3_large_x1_0_train_linux_gpu_fleet_amp_infer_python_linux_gpu_cpu.txt  #多机多卡训练预测配置文件
 │   │   └── MobileNetV3_large_x1_0_train_linux_gpu_normal_amp_infer_python_linux_gpu_cpu.txt #混合精度训练预测配置文件
@@ -50,7 +50,7 @@
 └── test_train_inference_python.sh      # 测试python训练预测的主程序
 ```
 
-### 测试流程
+### 测试流程概述
 使用本工具，可以测试不同功能的支持情况，以及预测结果是否对齐，测试流程如下：
 <div align="center">
     <img src="docs/test.png" width="800">
@@ -60,16 +60,49 @@
 2. 运行要测试的功能对应的测试脚本`test_*.sh`，产出log，由log可以看到不同配置是否运行成功；
 3. 用`compare_results.py`对比log中的预测结果和预存在results目录下的结果，判断预测精度是否符合预期（在误差范围内）。
 
-其中，有4个测试主程序，功能如下：
-- `test_train_inference_python.sh`：测试基于Python的模型训练、评估、推理等基本功能，包括裁剪、量化、蒸馏。
-- `test_inference_cpp.sh`：测试基于C++的模型推理。待支持
-- `test_serving.sh`：测试基于Paddle Serving的服务化部署功能。待支持
-- `test_lite.sh`：测试基于Paddle-Lite的端侧预测部署功能。待支持
+测试单项功能仅需两行命令，**如需测试不同模型/功能，替换配置文件即可**，命令格式如下：
+
+```shell
+# 功能：准备数据
+# 格式：bash + 运行脚本 + 参数1: 配置文件选择 + 参数2: 模式选择
+bash test_tipc/prepare.sh  configs/[model_name]/[params_file_name]  [Mode]
+
+# 功能：运行测试
+# 格式：bash + 运行脚本 + 参数1: 配置文件选择 + 参数2: 模式选择
+bash test_tipc/test_train_inference_python.sh configs/[model_name]/[params_file_name]  [Mode]
+
+```
+
+例如，测试基本训练预测功能的`lite_train_lite_infer`模式，运行：
+
+```shell
+# 准备数据
+bash test_tipc/prepare.sh ./test_tipc/configs/MobileNetV3/MobileNetV3_large_x1_0_train_infer_python.txt 'lite_train_lite_infer'
+# 运行测试
+bash test_tipc/test_train_inference_python.sh ./test_tipc/configs/MobileNetV3/MobileNetV3_large_x1_0_train_infer_python.txt 'lite_train_lite_infer'
+
+```
+
+关于本示例命令的更多信息可查看[基础训练预测使用文档](docs/test_train_inference_python.md)。
+
+### 配置文件命名规范
+
+在`configs`目录下，**按模型系列划分为子目录**，子目录中存放所有该模型系列测试需要用到的配置文件，如`MobileNetV3`文件夹下存放了所有`MobileNetV3`系列模型的配置文件。配置文件的命名遵循如下规范：
+
+1. 基础训练预测配置简单命名为：`ModelName_train_infer_python.txt`，表示**Linux环境下单机、不使用混合精度训练+python预测**，其完整命名对应`ModelName_train_linux_gpu_normal_normal_infer_python_linux_gpu_cpu.txt`，由于本配置文件使用频率较高，这里进行了名称简化。其中`ModelName`指具体模型名称
+2. 其他带训练配置命名格式为：`ModelName_train_训练硬件环境(linux_gpu/linux_dcu/…)_是否多机(fleet/normal)_是否混合精度(amp/normal)_预测模式(infer/lite/serving/js)_语言(cpp/python/java)_预测硬件环境(ModelName_linux_gpu/mac/jetson/opencl_arm_gpu/...).txt`。如，linux gpu下多机多卡+混合精度链条测试对应配置 `ModelName_train_linux_gpu_fleet_amp_infer_python_linux_gpu_cpu.txt`，linux dcu下基础训练预测对应配置 `ModelName_train_linux_dcu_normal_normal_infer_python_linux_dcu.txt`。
+3. 仅预测的配置（如serving、lite等）命名格式：`ModelName_model_训练硬件环境(ModelName_linux_gpu/linux_dcu/…)_是否多机(fleet/normal)_是否混合精度(amp/normal)_(infer/lite/serving/js)_语言(cpp/python/java)_预测硬件环境(ModelName_linux_gpu/mac/jetson/opencl_arm_gpu/...).txt`，即，与2相比，仅第二个字段从train换为model，测试时模型直接下载获取，这里的“训练硬件环境”表示所测试的模型是在哪种环境下训练得到的。
+
+**根据上述命名规范，可以直接从子目录名称和配置文件名找到需要测试的场景和功能对应的配置文件。**
 
 <a name="more"></a>
-#### 更多教程
+
+## 4 开始测试
+
 各功能测试中涉及混合精度、裁剪、量化等训练相关，及mkldnn、Tensorrt等多种预测相关参数配置，请点击下方相应链接了解更多细节和使用教程：  
-[test_train_inference_python 使用](docs/test_train_inference_python.md)  
-[test_inference_cpp 使用](docs/test_inference_cpp.md)  
-[test_serving 使用](docs/test_serving.md)  
-[test_lite 使用](docs/test_lite.md)
+
+- [test_train_inference_python 使用](docs/test_train_inference_python.md)：测试基于Python的模型训练、评估、推理等基本功能，包括裁剪、量化、蒸馏。
+- [test_inference_cpp 使用](docs/test_inference_cpp.md) ：测试基于C++的模型推理。
+- [test_serving 使用](docs/test_serving.md) ：测试基于Paddle Serving的服务化部署功能。
+- [test_lite_arm_cpu_cpp 使用](docs/test_lite_arm_cpu_cpp.md): 测试基于Paddle-Lite的ARM CPU端c++预测部署功能.
+- [test_paddle2onnx 使用](docs/test_paddle2onnx.md)：测试Paddle2ONNX的模型转化功能，并验证正确性。

test_tipc/prepare.sh

@@ -76,7 +76,7 @@ elif [ ${MODE} = "whole_train_whole_infer" ];then
     ln -s whole_chain_CIFAR100 ILSVRC2012
     cd ILSVRC2012 
     mv train.txt train_list.txt
-    mv val.txt val_list.txt
+    mv test.txt val_list.txt
     cd ../../
 fi
 

test_tipc/prepare_lite_cpp.sh

+#!/bin/bash
+source ./test_tipc/common_func.sh
+FILENAME=$1
+dataline=$(cat ${FILENAME})
+# parser params
+IFS=$'\n'
+lines=(${dataline})
+IFS=$'\n'
+
+inference_cmd=$(func_parser_value "${lines[1]}")
+DEVICE=$(func_parser_value "${lines[2]}")
+det_lite_model_list=$(func_parser_value "${lines[3]}")
+rec_lite_model_list=$(func_parser_value "${lines[4]}")
+cls_lite_model_list=$(func_parser_value "${lines[5]}")
+
+if [[ $inference_cmd =~ "det" ]];then
+    lite_model_list=${det_lite_model_list}
+elif [[ $inference_cmd =~ "rec" ]];then
+    lite_model_list=(${rec_lite_model_list[*]} ${cls_lite_model_list[*]})
+elif [[ $inference_cmd =~ "system" ]];then
+    lite_model_list=(${det_lite_model_list[*]} ${rec_lite_model_list[*]} ${cls_lite_model_list[*]})
+else
+    echo "inference_cmd is wrong, please check."
+    exit 1
+fi
+
+if [ ${DEVICE} = "ARM_CPU" ];then
+    valid_targets="arm"
+    paddlelite_url="https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.10-rc/inference_lite_lib.android.armv8.gcc.c++_shared.with_extra.with_cv.tar.gz"
+    end_index="66"
+elif [ ${DEVICE} = "ARM_GPU_OPENCL" ];then
+    valid_targets="opencl"
+    paddlelite_url="https://github.com/PaddlePaddle/Paddle-Lite/releases/download/v2.10-rc/inference_lite_lib.armv8.clang.with_exception.with_extra.with_cv.opencl.tar.gz"
+    end_index="71"
+else
+    echo "DEVICE only suport ARM_CPU, ARM_GPU_OPENCL."
+    exit 2    
+fi
+
+# prepare lite .nb model
+pip install paddlelite==2.10-rc
+current_dir=${PWD}
+IFS="|"
+model_path=./inference_models
+
+for model in ${lite_model_list[*]}; do
+    if [[ $model =~ "PP-OCRv2" ]];then
+        inference_model_url=https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/${model}.tar
+    elif [[ $model =~ "v2.0" ]];then
+        inference_model_url=https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/${model}.tar
+    else 
+        echo "Model is wrong, please check."
+        exit 3
+    fi
+    inference_model=${inference_model_url##*/}
+    wget -nc  -P ${model_path} ${inference_model_url}
+    cd ${model_path} && tar -xf ${inference_model} && cd ../
+    model_dir=${model_path}/${inference_model%.*}
+    model_file=${model_dir}/inference.pdmodel
+    param_file=${model_dir}/inference.pdiparams
+    paddle_lite_opt --model_dir=${model_dir} --model_file=${model_file} --param_file=${param_file} --valid_targets=${valid_targets} --optimize_out=${model_dir}_opt
+done
+
+# prepare test data
+data_url=https://paddleocr.bj.bcebos.com/dygraph_v2.0/test/icdar2015_lite.tar
+model_path=./inference_models
+inference_model=${inference_model_url##*/}
+data_file=${data_url##*/}
+wget -nc  -P ./inference_models ${inference_model_url}
+wget -nc  -P ./test_data ${data_url}
+cd ./inference_models && tar -xf ${inference_model} && cd ../
+cd ./test_data && tar -xf ${data_file} && rm ${data_file} && cd ../
+
+# prepare lite env
+paddlelite_zipfile=$(echo $paddlelite_url | awk -F "/" '{print $NF}')
+paddlelite_file=${paddlelite_zipfile:0:${end_index}}
+wget ${paddlelite_url} && tar -xf ${paddlelite_zipfile}
+mkdir -p  ${paddlelite_file}/demo/cxx/ocr/test_lite
+cp -r ${model_path}/*_opt.nb test_data ${paddlelite_file}/demo/cxx/ocr/test_lite
+cp ppocr/utils/ppocr_keys_v1.txt deploy/lite/config.txt ${paddlelite_file}/demo/cxx/ocr/test_lite
+cp -r ./deploy/lite/* ${paddlelite_file}/demo/cxx/ocr/
+cp ${paddlelite_file}/cxx/lib/libpaddle_light_api_shared.so ${paddlelite_file}/demo/cxx/ocr/test_lite
+cp ${FILENAME} test_tipc/test_lite_arm_cpp.sh test_tipc/common_func.sh ${paddlelite_file}/demo/cxx/ocr/test_lite
+cd ${paddlelite_file}/demo/cxx/ocr/
+git clone https://github.com/cuicheng01/AutoLog.git
+
+# make
+make -j
+sleep 1
+make -j
+cp ocr_db_crnn test_lite && cp test_lite/libpaddle_light_api_shared.so test_lite/libc++_shared.so
+tar -cf test_lite.tar ./test_lite && cp test_lite.tar ${current_dir} && cd ${current_dir}
+rm -rf ${paddlelite_file}* && rm -rf ${model_path}

test_tipc/test_inference_cpp.sh

+#!/bin/bash
+source test_tipc/common_func.sh
+
+FILENAME=$1
+dataline=$(awk 'NR==1, NR==16{print}'  $FILENAME)
+
+# parser params
+IFS=$'\n'
+lines=(${dataline})
+
+# parser cpp inference model 
+model_name=$(func_parser_value "${lines[1]}")
+use_opencv=$(func_parser_value "${lines[2]}")
+cpp_infer_model_dir_list=$(func_parser_value "${lines[3]}")
+cpp_infer_is_quant=$(func_parser_value "${lines[4]}")
+# parser cpp inference 
+inference_cmd=$(func_parser_value "${lines[5]}")
+cpp_use_gpu_key=$(func_parser_key "${lines[6]}")
+cpp_use_gpu_list=$(func_parser_value "${lines[6]}")
+cpp_use_mkldnn_key=$(func_parser_key "${lines[7]}")
+cpp_use_mkldnn_list=$(func_parser_value "${lines[7]}")
+cpp_cpu_threads_key=$(func_parser_key "${lines[8]}")
+cpp_cpu_threads_list=$(func_parser_value "${lines[8]}")
+cpp_batch_size_key=$(func_parser_key "${lines[9]}")
+cpp_batch_size_list=$(func_parser_value "${lines[9]}")
+cpp_use_trt_key=$(func_parser_key "${lines[10]}")
+cpp_use_trt_list=$(func_parser_value "${lines[10]}")
+cpp_precision_key=$(func_parser_key "${lines[11]}")
+cpp_precision_list=$(func_parser_value "${lines[11]}")
+cpp_infer_model_key=$(func_parser_key "${lines[12]}")
+cpp_image_dir_key=$(func_parser_key "${lines[13]}")
+cpp_infer_img_dir=$(func_parser_value "${lines[13]}")
+cpp_infer_key1=$(func_parser_key "${lines[14]}")
+cpp_infer_value1=$(func_parser_value "${lines[14]}")
+cpp_benchmark_key=$(func_parser_key "${lines[15]}")
+cpp_benchmark_value=$(func_parser_value "${lines[15]}")
+
+LOG_PATH="./test_tipc/output"
+mkdir -p ${LOG_PATH}
+status_log="${LOG_PATH}/results_cpp.log"
+
+
+function func_cpp_inference(){
+    IFS='|'
+    _script=$1
+    _model_dir=$2
+    _log_path=$3
+    _img_dir=$4
+    _flag_quant=$5
+    # inference 
+    for use_gpu in ${cpp_use_gpu_list[*]}; do
+        if [ ${use_gpu} = "False" ] || [ ${use_gpu} = "cpu" ]; then
+            for use_mkldnn in ${cpp_use_mkldnn_list[*]}; do
+                if [ ${use_mkldnn} = "False" ] && [ ${_flag_quant} = "True" ]; then
+                    continue
+                fi
+                for threads in ${cpp_cpu_threads_list[*]}; do
+                    for batch_size in ${cpp_batch_size_list[*]}; do
+                        precision="fp32"
+                        if [ ${use_mkldnn} = "False" ] && [ ${_flag_quant} = "True" ]; then
+                            precison="int8"
+                        fi
+                        _save_log_path="${_log_path}/cpp_infer_cpu_usemkldnn_${use_mkldnn}_threads_${threads}_precision_${precision}_batchsize_${batch_size}.log"
+                        set_infer_data=$(func_set_params "${cpp_image_dir_key}" "${_img_dir}")
+                        set_benchmark=$(func_set_params "${cpp_benchmark_key}" "${cpp_benchmark_value}")
+                        set_batchsize=$(func_set_params "${cpp_batch_size_key}" "${batch_size}")
+                        set_cpu_threads=$(func_set_params "${cpp_cpu_threads_key}" "${threads}")
+                        set_model_dir=$(func_set_params "${cpp_infer_model_key}" "${_model_dir}")
+                        set_infer_params1=$(func_set_params "${cpp_infer_key1}" "${cpp_infer_value1}")
+                        command="${_script} ${cpp_use_gpu_key}=${use_gpu} ${cpp_use_mkldnn_key}=${use_mkldnn} ${set_cpu_threads} ${set_model_dir} ${set_batchsize} ${set_infer_data} ${set_benchmark} ${set_infer_params1} > ${_save_log_path} 2>&1 "
+                        eval $command
+                        last_status=${PIPESTATUS[0]}
+                        eval "cat ${_save_log_path}"
+                        status_check $last_status "${command}" "${status_log}"
+                    done
+                done
+            done
+        elif [ ${use_gpu} = "True" ] || [ ${use_gpu} = "gpu" ]; then
+            for use_trt in ${cpp_use_trt_list[*]}; do
+                for precision in ${cpp_precision_list[*]}; do
+                    if [[ ${_flag_quant} = "False" ]] && [[ ${precision} =~ "int8" ]]; then
+                        continue
+                    fi 
+                    if [[ ${precision} =~ "fp16" || ${precision} =~ "int8" ]] && [ ${use_trt} = "False" ]; then
+                        continue
+                    fi
+                    if [[ ${use_trt} = "False" || ${precision} =~ "int8" ]] && [ ${_flag_quant} = "True" ]; then
+                        continue
+                    fi
+                    for batch_size in ${cpp_batch_size_list[*]}; do
+                        _save_log_path="${_log_path}/cpp_infer_gpu_usetrt_${use_trt}_precision_${precision}_batchsize_${batch_size}.log"
+                        set_infer_data=$(func_set_params "${cpp_image_dir_key}" "${_img_dir}")
+                        set_benchmark=$(func_set_params "${cpp_benchmark_key}" "${cpp_benchmark_value}")
+                        set_batchsize=$(func_set_params "${cpp_batch_size_key}" "${batch_size}")
+                        set_tensorrt=$(func_set_params "${cpp_use_trt_key}" "${use_trt}")
+                        set_precision=$(func_set_params "${cpp_precision_key}" "${precision}")
+                        set_model_dir=$(func_set_params "${cpp_infer_model_key}" "${_model_dir}")
+                        set_infer_params1=$(func_set_params "${cpp_infer_key1}" "${cpp_infer_value1}")
+                        command="${_script} ${cpp_use_gpu_key}=${use_gpu} ${set_tensorrt} ${set_precision} ${set_model_dir} ${set_batchsize} ${set_infer_data} ${set_benchmark} ${set_infer_params1} > ${_save_log_path} 2>&1 "
+                        eval $command
+                        last_status=${PIPESTATUS[0]}
+                        eval "cat ${_save_log_path}"
+                        status_check $last_status "${command}" "${status_log}"
+                        
+                    done
+                done
+            done
+        else
+            echo "Does not support hardware other than CPU and GPU Currently!"
+        fi
+    done
+}
+
+
+cd deploy/cpp_infer
+if [ ${use_opencv} = "True" ]; then
+    if [ -d "opencv-3.4.7/opencv3/" ] && [ $(md5sum opencv-3.4.7.tar.gz | awk -F ' ' '{print $1}') = "faa2b5950f8bee3f03118e600c74746a" ];then
+        echo "################### build opencv skipped ###################"
+    else
+        echo "################### build opencv ###################"
+        rm -rf opencv-3.4.7.tar.gz opencv-3.4.7/
+        wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/test/opencv-3.4.7.tar.gz
+        tar -xf opencv-3.4.7.tar.gz
+
+        cd opencv-3.4.7/
+        install_path=$(pwd)/opencv3
+
+        rm -rf build
+        mkdir build
+        cd build
+
+        cmake .. \
+            -DCMAKE_INSTALL_PREFIX=${install_path} \
+            -DCMAKE_BUILD_TYPE=Release \
+            -DBUILD_SHARED_LIBS=OFF \
+            -DWITH_IPP=OFF \
+            -DBUILD_IPP_IW=OFF \
+            -DWITH_LAPACK=OFF \
+            -DWITH_EIGEN=OFF \
+            -DCMAKE_INSTALL_LIBDIR=lib64 \
+            -DWITH_ZLIB=ON \
+            -DBUILD_ZLIB=ON \
+            -DWITH_JPEG=ON \
+            -DBUILD_JPEG=ON \
+            -DWITH_PNG=ON \
+            -DBUILD_PNG=ON \
+            -DWITH_TIFF=ON \
+            -DBUILD_TIFF=ON
+
+        make -j
+        make install
+        cd ../
+        echo "################### build opencv finished ###################"
+    fi
+fi
+
+
+echo "################### build PaddleOCR demo ####################"
+if [ ${use_opencv} = "True" ]; then
+    OPENCV_DIR=$(pwd)/opencv-3.4.7/opencv3/
+else
+    OPENCV_DIR=''
+fi
+LIB_DIR=$(pwd)/Paddle/build/paddle_inference_install_dir/
+CUDA_LIB_DIR=$(dirname `find /usr -name libcudart.so`)
+CUDNN_LIB_DIR=$(dirname `find /usr -name libcudnn.so`)
+
+BUILD_DIR=build
+rm -rf ${BUILD_DIR}
+mkdir ${BUILD_DIR}
+cd ${BUILD_DIR}
+cmake .. \
+    -DPADDLE_LIB=${LIB_DIR} \
+    -DWITH_MKL=ON \
+    -DWITH_GPU=OFF \
+    -DWITH_STATIC_LIB=OFF \
+    -DWITH_TENSORRT=OFF \
+    -DOPENCV_DIR=${OPENCV_DIR} \
+    -DCUDNN_LIB=${CUDNN_LIB_DIR} \
+    -DCUDA_LIB=${CUDA_LIB_DIR} \
+    -DTENSORRT_DIR=${TENSORRT_DIR} \
+
+make -j
+cd ../../../
+echo "################### build PaddleOCR demo finished ###################"
+
+
+# set cuda device
+GPUID=$2
+if [ ${#GPUID} -le 0 ];then
+    env=" "
+else
+    env="export CUDA_VISIBLE_DEVICES=${GPUID}"
+fi
+set CUDA_VISIBLE_DEVICES
+eval $env
+
+
+echo "################### run test ###################"
+export Count=0
+IFS="|"
+infer_quant_flag=(${cpp_infer_is_quant})
+for infer_model in ${cpp_infer_model_dir_list[*]}; do
+    #run inference
+    is_quant=${infer_quant_flag[Count]}
+    func_cpp_inference "${inference_cmd}" "${infer_model}" "${LOG_PATH}" "${cpp_infer_img_dir}" ${is_quant}
+    Count=$(($Count + 1))
+done

test_tipc/test_inference_jeston.sh

+#!/bin/bash
+source test_tipc/common_func.sh
+source test_tipc/test_train_inference_python.sh
+
+FILENAME=$1
+# MODE be one of ['whole_infer']
+MODE=$2
+
+dataline=$(awk 'NR==1, NR==17{print}'  $FILENAME)
+
+# parser params
+IFS=$'\n'
+lines=(${dataline})
+
+model_name=$(func_parser_value "${lines[1]}")
+python=$(func_parser_value "${lines[2]}")
+
+infer_model_dir_list=$(func_parser_value "${lines[3]}")
+infer_export_list=$(func_parser_value "${lines[4]}")
+infer_is_quant=$(func_parser_value "${lines[5]}")
+# parser inference 
+inference_py=$(func_parser_value "${lines[6]}")
+use_gpu_key=$(func_parser_key "${lines[7]}")
+use_gpu_list=$(func_parser_value "${lines[7]}")
+use_mkldnn_key=$(func_parser_key "${lines[8]}")
+use_mkldnn_list=$(func_parser_value "${lines[8]}")
+cpu_threads_key=$(func_parser_key "${lines[9]}")
+cpu_threads_list=$(func_parser_value "${lines[9]}")
+batch_size_key=$(func_parser_key "${lines[10]}")
+batch_size_list=$(func_parser_value "${lines[10]}")
+use_trt_key=$(func_parser_key "${lines[11]}")
+use_trt_list=$(func_parser_value "${lines[11]}")
+precision_key=$(func_parser_key "${lines[12]}")
+precision_list=$(func_parser_value "${lines[12]}")
+infer_model_key=$(func_parser_key "${lines[13]}")
+image_dir_key=$(func_parser_key "${lines[14]}")
+infer_img_dir=$(func_parser_value "${lines[14]}")
+save_log_key=$(func_parser_key "${lines[15]}")
+benchmark_key=$(func_parser_key "${lines[16]}")
+benchmark_value=$(func_parser_value "${lines[16]}")
+infer_key1=$(func_parser_key "${lines[17]}")
+infer_value1=$(func_parser_value "${lines[17]}")
+
+
+LOG_PATH="./test_tipc/output"
+mkdir -p ${LOG_PATH}
+status_log="${LOG_PATH}/results_python.log"
+
+
+if [ ${MODE} = "whole_infer" ]; then
+    GPUID=$3
+    if [ ${#GPUID} -le 0 ];then
+        env=" "
+    else
+        env="export CUDA_VISIBLE_DEVICES=${GPUID}"
+    fi
+    # set CUDA_VISIBLE_DEVICES
+    eval $env
+    export Count=0
+    IFS="|"
+    infer_run_exports=(${infer_export_list})
+    infer_quant_flag=(${infer_is_quant})
+    for infer_model in ${infer_model_dir_list[*]}; do
+        # run export
+        if [ ${infer_run_exports[Count]} != "null" ];then
+            save_infer_dir=$(dirname $infer_model)
+            set_export_weight=$(func_set_params "${export_weight}" "${infer_model}")
+            set_save_infer_key=$(func_set_params "${save_infer_key}" "${save_infer_dir}")
+            export_cmd="${python} ${infer_run_exports[Count]} ${set_export_weight} ${set_save_infer_key}"
+            echo ${infer_run_exports[Count]} 
+            echo  $export_cmd
+            eval $export_cmd
+            status_export=$?
+            status_check $status_export "${export_cmd}" "${status_log}"
+        else
+            save_infer_dir=${infer_model}
+        fi
+        #run inference
+        is_quant=${infer_quant_flag[Count]}
+        if [ ${MODE} = "klquant_infer" ]; then
+            is_quant="True"
+        fi
+        func_inference "${python}" "${inference_py}" "${save_infer_dir}" "${LOG_PATH}" "${infer_img_dir}" ${is_quant}
+        Count=$(($Count + 1))
+    done
+fi
+

test_tipc/test_lite_arm_cpp.sh

+#!/bin/bash
+source ./common_func.sh
+export LD_LIBRARY_PATH=${PWD}:$LD_LIBRARY_PATH
+
+FILENAME=$1
+dataline=$(cat $FILENAME)
+# parser params
+IFS=$'\n'
+lines=(${dataline})
+
+# parser lite inference
+inference_cmd=$(func_parser_value "${lines[1]}")
+runtime_device=$(func_parser_value "${lines[2]}")
+det_model_list=$(func_parser_value "${lines[3]}")
+rec_model_list=$(func_parser_value "${lines[4]}")
+cls_model_list=$(func_parser_value "${lines[5]}")
+cpu_threads_list=$(func_parser_value "${lines[6]}")
+det_batch_size_list=$(func_parser_value "${lines[7]}")
+rec_batch_size_list=$(func_parser_value "${lines[8]}")
+infer_img_dir_list=$(func_parser_value "${lines[9]}")
+config_dir=$(func_parser_value "${lines[10]}")
+rec_dict_dir=$(func_parser_value "${lines[11]}")
+benchmark_value=$(func_parser_value "${lines[12]}")
+
+if [[ $inference_cmd =~ "det" ]]; then
+    lite_model_list=${det_lite_model_list}
+elif [[ $inference_cmd =~ "rec" ]]; then
+    lite_model_list=(${rec_lite_model_list[*]} ${cls_lite_model_list[*]})
+elif [[ $inference_cmd =~ "system" ]]; then
+    lite_model_list=(${det_lite_model_list[*]} ${rec_lite_model_list[*]} ${cls_lite_model_list[*]})
+else
+    echo "inference_cmd is wrong, please check."
+    exit 1
+fi
+
+LOG_PATH="./output"
+mkdir -p ${LOG_PATH}
+status_log="${LOG_PATH}/results.log"
+
+
+function func_test_det(){
+    IFS='|'
+    _script=$1
+    _det_model=$2
+    _log_path=$3
+    _img_dir=$4
+    _config=$5
+    if [[ $_det_model =~ "slim" ]]; then
+        precision="INT8"
+    else
+        precision="FP32"
+    fi
+
+    # lite inference
+    for num_threads in ${cpu_threads_list[*]}; do
+	for det_batchsize in ${det_batch_size_list[*]}; do
+            _save_log_path="${_log_path}/lite_${_det_model}_runtime_device_${runtime_device}_precision_${precision}_det_batchsize_${det_batchsize}_threads_${num_threads}.log"
+            command="${_script} ${_det_model} ${runtime_device} ${precision} ${num_threads} ${det_batchsize}  ${_img_dir} ${_config} ${benchmark_value} > ${_save_log_path} 2>&1"
+            eval ${command}
+            status_check $? "${command}" "${status_log}"
+        done
+    done
+}
+
+function func_test_rec(){
+    IFS='|'
+    _script=$1
+    _rec_model=$2
+    _cls_model=$3
+    _log_path=$4
+    _img_dir=$5
+    _config=$6
+    _rec_dict_dir=$7
+
+    if [[ $_det_model =~ "slim" ]]; then
+        _precision="INT8"
+    else
+        _precision="FP32"
+    fi
+
+    # lite inference
+    for num_threads in ${cpu_threads_list[*]}; do
+	for rec_batchsize in ${rec_batch_size_list[*]}; do
+            _save_log_path="${_log_path}/lite_${_rec_model}_${cls_model}_runtime_device_${runtime_device}_precision_${_precision}_rec_batchsize_${rec_batchsize}_threads_${num_threads}.log"
+            command="${_script} ${_rec_model} ${_cls_model} ${runtime_device} ${_precision} ${num_threads} ${rec_batchsize}  ${_img_dir} ${_config} ${_rec_dict_dir} ${benchmark_value} > ${_save_log_path} 2>&1"
+            eval ${command}
+            status_check $? "${command}" "${status_log}"
+        done
+    done
+}
+
+function func_test_system(){
+    IFS='|'
+    _script=$1
+    _det_model=$2
+    _rec_model=$3
+    _cls_model=$4
+    _log_path=$5
+    _img_dir=$6
+    _config=$7
+    _rec_dict_dir=$8
+    if [[ $_det_model =~ "slim" ]]; then
+        _precision="INT8"
+    else
+        _precision="FP32"
+    fi
+
+    # lite inference
+    for num_threads in ${cpu_threads_list[*]}; do
+	for det_batchsize in ${det_batch_size_list[*]}; do
+	   for rec_batchsize in ${rec_batch_size_list[*]}; do
+                _save_log_path="${_log_path}/lite_${_det_model}_${_rec_model}_${_cls_model}_runtime_device_${runtime_device}_precision_${_precision}_det_batchsize_${det_batchsize}_rec_batchsize_${rec_batchsize}_threads_${num_threads}.log"
+                command="${_script} ${_det_model} ${_rec_model} ${_cls_model} ${runtime_device} ${_precision} ${num_threads} ${det_batchsize}  ${_img_dir} ${_config} ${_rec_dict_dir} ${benchmark_value} > ${_save_log_path} 2>&1"
+               eval ${command}
+               status_check $? "${command}" "${status_log}"
+	    done
+        done
+    done
+}
+
+
+echo "################### run test ###################"
+
+if [[ $inference_cmd =~ "det" ]]; then
+    IFS="|"
+    det_model_list=(${det_model_list[*]})
+
+    for i in {0..1}; do
+        #run lite inference
+        for img_dir in ${infer_img_dir_list[*]}; do
+            func_test_det "${inference_cmd}" "${det_model_list[i]}_opt.nb" "${LOG_PATH}" "${img_dir}" "${config_dir}"
+        done
+    done
+
+elif [[ $inference_cmd =~ "rec" ]]; then
+    IFS="|"
+    rec_model_list=(${rec_model_list[*]})
+    cls_model_list=(${cls_model_list[*]})
+
+    for i in {0..1}; do
+        #run lite inference
+        for img_dir in ${infer_img_dir_list[*]}; do
+            func_test_rec "${inference_cmd}" "${rec_model}_opt.nb" "${cls_model_list[i]}_opt.nb" "${LOG_PATH}" "${img_dir}" "${rec_dict_dir}" "${config_dir}"
+        done
+    done
+
+elif [[ $inference_cmd =~ "system" ]]; then
+    IFS="|"
+    det_model_list=(${det_model_list[*]})
+    rec_model_list=(${rec_model_list[*]})
+    cls_model_list=(${cls_model_list[*]})
+
+    for i in {0..1}; do
+	#run lite inference
+        for img_dir in ${infer_img_dir_list[*]}; do
+            func_test_system "${inference_cmd}" "${det_model_list[i]}_opt.nb" "${rec_model_list[i]}_opt.nb" "${cls_model_list[i]}_opt.nb" "${LOG_PATH}" "${img_dir}" "${config_dir}" "${rec_dict_dir}"
+        done
+    done
+fi

test_tipc/test_paddle2onnx.sh

+#!/bin/bash
+source test_tipc/common_func.sh 
+
+FILENAME=$1
+
+dataline=$(cat ${FILENAME})
+lines=(${dataline})
+# common params
+model_name=$(func_parser_value "${lines[1]}")
+python=$(func_parser_value "${lines[2]}")
+
+
+# parser params
+dataline=$(awk 'NR==1, NR==12{print}'  $FILENAME)
+IFS=$'\n'
+lines=(${dataline})
+
+# parser paddle2onnx
+model_name=$(func_parser_value "${lines[1]}")
+python=$(func_parser_value "${lines[2]}")
+padlle2onnx_cmd=$(func_parser_value "${lines[3]}")
+infer_model_dir_key=$(func_parser_key "${lines[4]}")
+infer_model_dir_value=$(func_parser_value "${lines[4]}")
+model_filename_key=$(func_parser_key "${lines[5]}")
+model_filename_value=$(func_parser_value "${lines[5]}")
+params_filename_key=$(func_parser_key "${lines[6]}")
+params_filename_value=$(func_parser_value "${lines[6]}")
+save_file_key=$(func_parser_key "${lines[7]}")
+save_file_value=$(func_parser_value "${lines[7]}")
+opset_version_key=$(func_parser_key "${lines[8]}")
+opset_version_value=$(func_parser_value "${lines[8]}")
+enable_onnx_checker_key=$(func_parser_key "${lines[9]}")
+enable_onnx_checker_value=$(func_parser_value "${lines[9]}")
+# parser onnx inference 
+inference_py=$(func_parser_value "${lines[10]}")
+use_gpu_key=$(func_parser_key "${lines[11]}")
+use_gpu_value=$(func_parser_value "${lines[11]}")
+det_model_key=$(func_parser_key "${lines[12]}")
+image_dir_key=$(func_parser_key "${lines[13]}")
+image_dir_value=$(func_parser_value "${lines[13]}")
+
+
+LOG_PATH="./test_tipc/output"
+mkdir -p ./test_tipc/output
+status_log="${LOG_PATH}/results_paddle2onnx.log"
+
+
+function func_paddle2onnx(){
+    IFS='|'
+    _script=$1
+
+    # paddle2onnx
+    _save_log_path="${LOG_PATH}/paddle2onnx_infer_cpu.log"
+    set_dirname=$(func_set_params "${infer_model_dir_key}" "${infer_model_dir_value}")
+    set_model_filename=$(func_set_params "${model_filename_key}" "${model_filename_value}")
+    set_params_filename=$(func_set_params "${params_filename_key}" "${params_filename_value}")
+    set_save_model=$(func_set_params "${save_file_key}" "${save_file_value}")
+    set_opset_version=$(func_set_params "${opset_version_key}" "${opset_version_value}")
+    set_enable_onnx_checker=$(func_set_params "${enable_onnx_checker_key}" "${enable_onnx_checker_value}")
+    trans_model_cmd="${padlle2onnx_cmd} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_save_model} ${set_opset_version} ${set_enable_onnx_checker}"
+    eval $trans_model_cmd
+    last_status=${PIPESTATUS[0]}
+    status_check $last_status "${trans_model_cmd}" "${status_log}"
+    # python inference
+    set_gpu=$(func_set_params "${use_gpu_key}" "${use_gpu_value}")
+    set_model_dir=$(func_set_params "${det_model_key}" "${save_file_value}")
+    set_img_dir=$(func_set_params "${image_dir_key}" "${image_dir_value}")
+    infer_model_cmd="${python} ${inference_py} ${set_gpu} ${set_img_dir} ${set_model_dir} --use_onnx=True > ${_save_log_path} 2>&1 "
+    eval $infer_model_cmd
+    status_check $last_status "${infer_model_cmd}" "${status_log}"
+}
+
+
+echo "################### run test ###################"
+
+export Count=0
+IFS="|"
+func_paddle2onnx 
\ No newline at end of file

test_tipc/test_serving.sh

+#!/bin/bash
+source test_tipc/common_func.sh
+
+FILENAME=$1
+dataline=$(awk 'NR==1, NR==18{print}'  $FILENAME)
+
+# parser params
+IFS=$'\n'
+lines=(${dataline})
+
+# parser serving
+model_name=$(func_parser_value "${lines[1]}")
+python=$(func_parser_value "${lines[2]}")
+trans_model_py=$(func_parser_value "${lines[3]}")
+infer_model_dir_key=$(func_parser_key "${lines[4]}")
+infer_model_dir_value=$(func_parser_value "${lines[4]}")
+model_filename_key=$(func_parser_key "${lines[5]}")
+model_filename_value=$(func_parser_value "${lines[5]}")
+params_filename_key=$(func_parser_key "${lines[6]}")
+params_filename_value=$(func_parser_value "${lines[6]}")
+serving_server_key=$(func_parser_key "${lines[7]}")
+serving_server_value=$(func_parser_value "${lines[7]}")
+serving_client_key=$(func_parser_key "${lines[8]}")
+serving_client_value=$(func_parser_value "${lines[8]}")
+serving_dir_value=$(func_parser_value "${lines[9]}")
+web_service_py=$(func_parser_value "${lines[10]}")
+web_use_gpu_key=$(func_parser_key "${lines[11]}")
+web_use_gpu_list=$(func_parser_value "${lines[11]}")
+web_use_mkldnn_key=$(func_parser_key "${lines[12]}")
+web_use_mkldnn_list=$(func_parser_value "${lines[12]}")
+web_cpu_threads_key=$(func_parser_key "${lines[13]}")
+web_cpu_threads_list=$(func_parser_value "${lines[13]}")
+web_use_trt_key=$(func_parser_key "${lines[14]}")
+web_use_trt_list=$(func_parser_value "${lines[14]}")
+web_precision_key=$(func_parser_key "${lines[15]}")
+web_precision_list=$(func_parser_value "${lines[15]}")
+pipeline_py=$(func_parser_value "${lines[16]}")
+image_dir_key=$(func_parser_key "${lines[17]}")
+image_dir_value=$(func_parser_value "${lines[17]}")
+
+LOG_PATH="../../test_tipc/output"
+mkdir -p ./test_tipc/output
+status_log="${LOG_PATH}/results_serving.log"
+
+function func_serving(){
+    IFS='|'
+    _python=$1
+    _script=$2
+    _model_dir=$3
+    # pdserving
+    set_dirname=$(func_set_params "${infer_model_dir_key}" "${infer_model_dir_value}")
+    set_model_filename=$(func_set_params "${model_filename_key}" "${model_filename_value}")
+    set_params_filename=$(func_set_params "${params_filename_key}" "${params_filename_value}")
+    set_serving_server=$(func_set_params "${serving_server_key}" "${serving_server_value}")
+    set_serving_client=$(func_set_params "${serving_client_key}" "${serving_client_value}")
+    set_image_dir=$(func_set_params "${image_dir_key}" "${image_dir_value}")
+    trans_model_cmd="${python} ${trans_model_py} ${set_dirname} ${set_model_filename} ${set_params_filename} ${set_serving_server} ${set_serving_client}"
+    eval $trans_model_cmd
+    cd ${serving_dir_value}
+    echo $PWD
+    unset https_proxy
+    unset http_proxy
+    for python in ${python[*]}; do
+        if [ ${python} = "cpp"]; then
+            for use_gpu in ${web_use_gpu_list[*]}; do
+                if [ ${use_gpu} = "null" ]; then
+                    web_service_cpp_cmd="${python} -m paddle_serving_server.serve --model ppocr_det_mobile_2.0_serving/ ppocr_rec_mobile_2.0_serving/ --port 9293"
+                    eval $web_service_cmd
+                    sleep 2s
+                    _save_log_path="${LOG_PATH}/server_infer_cpp_cpu_pipeline_usemkldnn_False_threads_4_batchsize_1.log"
+                    pipeline_cmd="${python} ocr_cpp_client.py ppocr_det_mobile_2.0_client/ ppocr_rec_mobile_2.0_client/"
+                    eval $pipeline_cmd
+                    status_check $last_status "${pipeline_cmd}" "${status_log}"
+                    sleep 2s
+                    ps ux | grep -E 'web_service|pipeline' | awk '{print $2}' | xargs kill -s 9
+                else
+                    web_service_cpp_cmd="${python} -m paddle_serving_server.serve --model ppocr_det_mobile_2.0_serving/ ppocr_rec_mobile_2.0_serving/ --port 9293 --gpu_id=0"
+                    eval $web_service_cmd
+                    sleep 2s
+                    _save_log_path="${LOG_PATH}/server_infer_cpp_cpu_pipeline_usemkldnn_False_threads_4_batchsize_1.log"
+                    pipeline_cmd="${python} ocr_cpp_client.py ppocr_det_mobile_2.0_client/ ppocr_rec_mobile_2.0_client/"
+                    eval $pipeline_cmd
+                    status_check $last_status "${pipeline_cmd}" "${status_log}"
+                    sleep 2s
+                    ps ux | grep -E 'web_service|pipeline' | awk '{print $2}' | xargs kill -s 9                
+                fi
+            done
+        else
+            # python serving
+            for use_gpu in ${web_use_gpu_list[*]}; do
+                echo ${ues_gpu}
+                if [ ${use_gpu} = "null" ]; then
+                    for use_mkldnn in ${web_use_mkldnn_list[*]}; do
+                        if [ ${use_mkldnn} = "False" ]; then
+                            continue
+                        fi
+                        for threads in ${web_cpu_threads_list[*]}; do
+                            set_cpu_threads=$(func_set_params "${web_cpu_threads_key}" "${threads}")
+                            web_service_cmd="${python} ${web_service_py} ${web_use_gpu_key}=${use_gpu} ${web_use_mkldnn_key}=${use_mkldnn} ${set_cpu_threads} &"
+                            eval $web_service_cmd
+                            sleep 2s
+                            for pipeline in ${pipeline_py[*]}; do
+                                _save_log_path="${LOG_PATH}/server_infer_cpu_${pipeline%_client*}_usemkldnn_${use_mkldnn}_threads_${threads}_batchsize_1.log"
+                                pipeline_cmd="${python} ${pipeline} ${set_image_dir} > ${_save_log_path} 2>&1 "
+                                eval $pipeline_cmd
+                                last_status=${PIPESTATUS[0]}
+                                eval "cat ${_save_log_path}"
+                                status_check $last_status "${pipeline_cmd}" "${status_log}"
+                                sleep 2s
+                            done
+                            ps ux | grep -E 'web_service|pipeline' | awk '{print $2}' | xargs kill -s 9
+                        done
+                    done
+                elif [ ${use_gpu} = "0" ]; then
+                    for use_trt in ${web_use_trt_list[*]}; do
+                        for precision in ${web_precision_list[*]}; do
+                            if [[ ${_flag_quant} = "False" ]] && [[ ${precision} =~ "int8" ]]; then
+                                continue
+                            fi
+                            if [[ ${precision} =~ "fp16" || ${precision} =~ "int8" ]] && [ ${use_trt} = "False" ]; then
+                                continue
+                            fi
+                            if [[ ${use_trt} = "False" || ${precision} =~ "int8" ]] && [[ ${_flag_quant} = "True" ]]; then
+                                continue
+                            fi
+                            set_tensorrt=$(func_set_params "${web_use_trt_key}" "${use_trt}")
+                            set_precision=$(func_set_params "${web_precision_key}" "${precision}")
+                            web_service_cmd="${python} ${web_service_py} ${web_use_gpu_key}=${use_gpu} ${set_tensorrt} ${set_precision} & "
+                            eval $web_service_cmd
+                        
+                            sleep 2s
+                            for pipeline in ${pipeline_py[*]}; do
+                                _save_log_path="${LOG_PATH}/server_infer_gpu_${pipeline%_client*}_usetrt_${use_trt}_precision_${precision}_batchsize_1.log"
+                                pipeline_cmd="${python} ${pipeline} ${set_image_dir}> ${_save_log_path} 2>&1"
+                                eval $pipeline_cmd
+                                last_status=${PIPESTATUS[0]}
+                                eval "cat ${_save_log_path}"
+                                status_check $last_status "${pipeline_cmd}" "${status_log}"
+                                sleep 2s
+                            done
+                            ps ux | grep -E 'web_service|pipeline' | awk '{print $2}' | xargs kill -s 9
+                        done
+                    done
+                else
+                    echo "Does not support hardware other than CPU and GPU Currently!"
+                fi
+            done
+        fi
+    done
+}
+
+
+# set cuda device
+GPUID=$2
+if [ ${#GPUID} -le 0 ];then
+    env=" "
+else
+    env="export CUDA_VISIBLE_DEVICES=${GPUID}"
+fi
+set CUDA_VISIBLE_DEVICES
+eval $env
+
+
+echo "################### run test ###################"
+
+export Count=0
+IFS="|"
+func_serving "${web_service_cmd}"