diff --git a/doc/howto/cross_compiling/cross_compiling_for_raspberry_cn.md b/doc/howto/cross_compiling/cross_compiling_for_raspberry_cn.md index 085b5dda1615a9af918b59870db460fcc5acdcca..026c0c6f3b2a2ca322d063f38e1736a010e1197e 100644 --- a/doc/howto/cross_compiling/cross_compiling_for_raspberry_cn.md +++ b/doc/howto/cross_compiling/cross_compiling_for_raspberry_cn.md @@ -1,39 +1,36 @@ # 构建Raspberry Pi平台上的PaddlePaddle库 -对于Rasspberry Pi系统,用户可通过ssh等方式登录到Raspberry Pi系统上,按照[源码编译PaddlePaddle](http://www.paddlepaddle.org/doc_cn/getstarted/build_and_install/cmake/build_from_source_cn.html)相关文档所述,直接编译Raspberry Pi平台上适用的PaddlePaddle库。 +通常有两个方法来构建基于 Rasspberry Pi 的版本: -用户也可以在自己熟悉的开发平台上,通过交叉编译的方式来编译。这篇文档将以Linux x86-64平台为例,介绍交叉编译Raspberry Pi平台上适用的PaddlePaddle的方法和步骤。 +1. 通过ssh等方式登录到Raspberry Pi系统上来构建。所需的开发工具和第三方库可以参考 [`/Dockerfile`](https://github.com/PaddlePaddle/Paddle/blob/develop/Dockerfile)。 -## 准备交叉编译环境 +1. 另一个方法是交叉编译。这篇文档介绍在 Linux/x64 上交叉编译Raspberry Pi平台上适用的PaddlePaddle的方法和步骤。 -从源码交叉编译PaddlePaddle,用户需要提前准备好交叉编译环境。用户可自行前往[github](https://github.com/raspberrypi/tools)下载Raspberry Pi平台使用的C/C++交叉编译工具链,也可通过以下命令获取: +## 安装交叉编译器 + +克隆下面 Github repo ```bash git clone https://github.com/raspberrypi/tools.git ``` -该github仓库中包含若干个预编译好的、针对不同平台的编译工具。宿主机是Linux x86-64环境,则需选用`arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64`下的作为编译工具,所使用的编译器为arm-linux-gnueabihf-gcc 4.8.3。 - -注意,该编译工具链需要系统glibc支持2.14以上。 +即可在 `./tools/tree/master/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64` 目录里找到交叉编译器 arm-linux-gnueabihf-gcc 4.8.3。运行该编译工具链需要一台 Linux x64 机器上以及 2.14版本以上的 glibc。 ## 配置交叉编译参数 -CMake系统对交叉编译提供了支持[cmake-toolchains](https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling)。为了简化cmake配置,PaddlePaddle为交叉编译提供了工具链配置文档[cmake/cross_compiling/raspberry_pi.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/cross_compiling/raspberry_pi.cmake),以提供一些默认的编译器和编译参数相关配置。 +CMake[支持交叉编译](https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling)。PaddlePaddle for Raspberry Pi的配置信息在[cmake/cross_compiling/raspberry_pi.cmake](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/cross_compiling/raspberry_pi.cmake)。 交叉编译Raspberry Pi版本PaddlePaddle库时,有一些必须配置的参数: -- `CMAKE_SYSTEM_NAME`,CMake编译的目标平台,必须配置为`RPi`。在设置`CMAKE_SYSTEM_NAME=RPi`后,PaddlePaddle的CMake系统才认为在是在交叉编译Raspberry Pi系统的版本,并自动编译宿主机版protoc可执行文件、目标机版protobuf库、以及目标机版OpenBLAS库。 - -Raspberry Pi平台可选配置参数: +- `CMAKE_SYSTEM_NAME`:CMake编译的目标平台,必须配置为`RPi`。在设置`CMAKE_SYSTEM_NAME=RPi`后,PaddlePaddle的CMake系统才认为在是在交叉编译Raspberry Pi系统的版本,并自动编译宿主机版protoc可执行文件、目标机版protobuf库、以及目标机版OpenBLAS库。 -- `RPI_TOOLCHAIN`,编译工具链所在的绝对路径,或者相对于构建目录的相对路径。PaddlePaddle的CMake系统将根据该值自动设置需要使用的交叉编译器;否则,用户需要在cmake时手动设置这些值。无默认值。 -- `RPI_ARM_NEON`,是否使用NEON指令。目前必须设置成`ON`,默认值为`ON`。 +- `RPI_TOOLCHAIN`:编译工具链所在的绝对路径,或者相对于构建目录的相对路径。PaddlePaddle的CMake系统将根据该值自动设置需要使用的交叉编译器;否则,用户需要在cmake时手动设置这些值。无默认值。 -其他配置参数: +- `RPI_ARM_NEON`:是否使用NEON指令。目前必须设置成`ON`,默认值为`ON`。 - `HOST_C/CXX_COMPILER`,宿主机的C/C++编译器。在编译宿主机版protoc可执行文件和目标机版OpenBLAS库时需要用到。默认设置成环境变量`CC`的值;若环境变量`CC`没有设置,则设置成`cc`编译器。 -cmake参数如下; +一个常用的CMake配置如下: ``` cmake -DCMAKE_SYSTEM_NAME=RPi \ @@ -47,7 +44,9 @@ cmake -DCMAKE_SYSTEM_NAME=RPi \ .. ``` -用户还可根据自己的需求设置其他编译参数。比如希望最小化生成的库的大小,可以设置`CMAKE_BUILD_TYPE`为`MinSizeRel`;若希望最快的执行速度,则可设置`CMAKE_BUILD_TYPE`为`Release`。亦可以通过手动设置`CMAKE_C/CXX_FLAGS_MINSIZEREL/RELEASE`来影响PaddlePaddle的编译过程。 +其中`WITH_C_API=ON`表示需要构建推理库。 + +用户还可根据自己的需求设置其他编译参数。比如希望最小化生成的库的大小,可以设置`CMAKE_BUILD_TYPE`为`MinSizeRel`;若希望最快的执行速度,则可设置`CMAKE_BUILD_TYPE`为`Release`。 ## 编译和安装 @@ -60,6 +59,4 @@ make install 注意:如果你曾经在源码目录下编译过其他平台的PaddlePaddle库,请先使用`rm -rf`命令删除`third_party`目录和`build`目录,以确保所有的第三方依赖库和PaddlePaddle代码都是针对新的CMake配置重新编译的。 -执行完安装命令后,由于上一步cmake配置中`WITH_C_API`设置为`ON`,`your/path/to/install`目录中会包含`include`和`lib`目录,其中`include`中包含C-API的头文件,`lib`中包含一个Raspberry Pi版本的库。 - -更多的编译配置见[源码编译PaddlePaddle](http://www.paddlepaddle.org/doc_cn/getstarted/build_and_install/cmake/build_from_source_cn.html)相关文档。 +执行完安装命令后,,`your/path/to/install`目录中会包含`include`和`lib`目录,其中`include`中包含C-API的头文件,`lib`中包含一个Raspberry Pi版本的库。 diff --git a/doc/howto/cross_compiling/cross_compiling_for_raspberry_en.md b/doc/howto/cross_compiling/cross_compiling_for_raspberry_en.md new file mode 100644 index 0000000000000000000000000000000000000000..09ac4733ec98c598dfd62f22beaf838320dc7531 --- /dev/null +++ b/doc/howto/cross_compiling/cross_compiling_for_raspberry_en.md @@ -0,0 +1,62 @@ +# Build PaddlePaddle for Raspberry Pi + +You may use any of the following two approaches to build the inference library of PaddlePaddle for Raspberry Pi: + +1. Build using SSH: Log in to a Raspberry Pi using SSH and build the library. The required development tools and third-party dependencies are listed in here: [`/Dockerfile`](https://github.com/PaddlePaddle/Paddle/blob/develop/Dockerfile). + +1. Cross-compile: We talk about how to cross-compile PaddlePaddle for Raspberry Pi on a Linux/x64 machine, in more detail in this article. + +## The Cross-Compiling Toolchain + +Step 1. Clone the Github repo by running the following command. + +```bash +git clone https://github.com/raspberrypi/tools.git +``` + +Step 2. Use the pre-built cross-compiler found in `./tools/tree/master/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64`. To run it on a Linux computer, glibc version >= 2.14 is needed. + +## CMake Arguments + +CMake supports [cross-compiling](https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling). All CMake configuration arguments required for the cross-compilation for Raspberry Pi can be found in [`cmake/cross_compiling/raspberry_pi.cmake`](https://github.com/PaddlePaddle/Paddle/blob/develop/cmake/cross_compiling/raspberry_pi.cmake). + +Some important arguments that need to be set: + +- `CMAKE_SYSTEM_NAME`: The target platform. Must be `RPi`. + +- `RPI_TOOLCHAIN`: The absolute path of the cross-compiling toolchain. + +- `RPI_ARM_NEON`: Use ARM NEON Intrinsics. This is a required argument and set default to `ON`. + +- `HOST_C/CXX_COMPILER`: The C/C++ compiler for the host. It is used to build building tools running on the host, for example, protoc. + +A commonly-used CMake configuration is as follows: + +``` +cmake -DCMAKE_SYSTEM_NAME=RPi \ + -DRPI_TOOLCHAIN=your/path/to/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64 \ + -DRPI_ARM_NEON=ON \ + -DCMAKE_INSTALL_PREFIX=your/path/to/install \ + -DWITH_GPU=OFF \ + -DWITH_C_API=ON \ + -DWITH_PYTHON=OFF \ + -DWITH_SWIG_PY=OFF \ + .. +``` + +To build the inference library, please set the argument WITH_API to ON: `WITH_C_API=ON`. + +You can add more arguments. For example, to minimize the size of the generated inference library, you may use `CMAKE_BUILD_TYPE=MinSizeRel`. For performance optimization, you may use `CMAKE_BUILD_TYPE=Release`. + +## Build and Install + +The following commands build the inference library of PaddlePaddle for Raspberry Pi and third-party dependencies. + +```bash +make +make install +``` + + The intermediate files will be stored in `build`. Third-party libraries will be located in `build/third_party`. If you have already built it for other platforms like Android or iOS, you may want to clear these directories by running the command: `rm -rf build`. + +The infernece library will be in `your/path/to/install/lib`, with related header files in `your/path/to/install/include`. diff --git a/go/.gitignore b/go/.gitignore index 000e1fd55b63b8e532308b787c2708a6c3e5ac87..398d70ca375ffceccdbfc82a4851a6830ca31264 100644 --- a/go/.gitignore +++ b/go/.gitignore @@ -1,2 +1,3 @@ vendor/ .glide/ +proto/*.go diff --git a/go/glide.lock b/go/glide.lock index ce654d36364f8078a493651d8d8b141532eea26d..d15fc934dbe511389cc92ce95cededa41ba32b4d 100644 --- a/go/glide.lock +++ b/go/glide.lock @@ -1,5 +1,5 @@ -hash: 51d9e2e46d7fd9173ff11ecada40f7b7728756be18d5e2f032535f66465e6e15 -updated: 2017-10-24T15:04:09.987751592-07:00 +hash: 107c058cf5c9163a75d40eef2273a793c36112683c25d72aa8288827fdde3a19 +updated: 2017-10-30T03:46:19.137696069Z imports: - name: github.com/alecthomas/gometalinter version: bae2f1293d092fd8167939d5108d1b025eaef9de diff --git a/go/glide.yaml b/go/glide.yaml index ba253f8bebef0ddab810a8303ab1fbe541defbdf..c5d66694acd0f45de5002391a7953b7491eaf2bc 100644 --- a/go/glide.yaml +++ b/go/glide.yaml @@ -30,3 +30,4 @@ import: version: v2.13 - package: github.com/go-stack/stack version: v1.6.0 +- package: github.com/golang/protobuf diff --git a/go/proto/.gitignore b/go/proto/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..5e7d2734cfc60289debf74293817c0a8f572ff32 --- /dev/null +++ b/go/proto/.gitignore @@ -0,0 +1,4 @@ +# Ignore everything in this directory +* +# Except this file +!.gitignore diff --git a/go/pserver/CMakeLists.txt b/go/pserver/CMakeLists.txt index 4fe0a8cb021e8dbf443c8f33bfb046e228a2fd8d..9ac05199e7ab76c21275838092c0afbdf2612b77 100644 --- a/go/pserver/CMakeLists.txt +++ b/go/pserver/CMakeLists.txt @@ -13,5 +13,5 @@ # limitations under the License. # if(WITH_TESTING) - go_test(pserver_test DEPS paddle_go_optimizer) + go_test(pserver_test DEPS paddle_go_optimizer gen_proto_go) endif() diff --git a/go/pserver/service.go b/go/pserver/service.go index f703d99a29ae9f5310ef36a7492b729c4c892937..7484ec90b1a3a9e67fa798741a9dfeb580c51f1a 100644 --- a/go/pserver/service.go +++ b/go/pserver/service.go @@ -17,6 +17,7 @@ package pserver import ( "bufio" "bytes" + "encoding/binary" "encoding/gob" "encoding/json" "errors" @@ -26,11 +27,15 @@ import ( "os" "path" "strconv" + "strings" "sync" "time" + "github.com/golang/protobuf/proto" uuid "github.com/satori/go.uuid" + pb "github.com/PaddlePaddle/Paddle/go/proto" + log "github.com/inconshreveable/log15" ) @@ -65,6 +70,46 @@ type Parameter struct { Content []byte } +func float32ToString(b []byte) string { + f := make([]float32, len(b)/4) + buf := bytes.NewReader(b) + err := binary.Read(buf, binary.LittleEndian, &f) + if err != nil { + return "" + } + return fmt.Sprintf("%v", f) +} + +func float32ByteToString(c []byte) string { + var a []byte + var b []byte + if len(c) <= 80 { + a = c + } else { + a = c[0:40] + b = c[len(c)-40:] + } + + var s string + s = float32ToString(a) + + if b == nil { + return s + } + + s = strings.Replace(s, "]", "", -1) + "..." + strings.Replace(float32ToString(b), "[", "", -1) + return s +} + +func (p Parameter) String() string { + if p.ElementType != Float32 { + return fmt.Sprintf("name:%v ElementType:%v", + p.Name, p.ElementType) + } + + return float32ByteToString(p.Content) +} + // ParameterWithConfig contains the parameter and the configuration. type ParameterWithConfig struct { Param Parameter @@ -189,7 +234,9 @@ func (s *Service) InitParam(paramWithConfigs ParameterWithConfig, _ *int) error default: } - // TODO(helin): parse parameter config + c := &pb.OptimizerConfig{} + proto.Unmarshal(paramWithConfigs.Config, c) + log.Debug(fmt.Sprintf("OptimizerConfig:%v", c)) s.mu.Lock() defer s.mu.Unlock() @@ -239,7 +286,8 @@ func (s *Service) SendGrad(g Gradient, _ *int) error { select { case <-s.initialized: default: - log.Warn("received gradient before initialization.", "name", g.Name, "size", len(g.Content), "type", g.ElementType) + log.Warn("received gradient before initialization.", + "name", g.Name, "size", len(g.Content), "type", g.ElementType) return errors.New(Uninitialized) } @@ -248,10 +296,14 @@ func (s *Service) SendGrad(g Gradient, _ *int) error { o, ok := s.optMap[g.Name] if !ok { + log.Warn("received gradient but can't find name.", + "name", g.Name, "size", len(g.Content), "type", g.ElementType) return fmt.Errorf("parameter: %s does not exist", g.Name) } - log.Info("received gradient from trainer, updating gradient.", "name", g.Name, "size", len(g.Content), "type", g.ElementType) + log.Debug(Parameter(g).String()) + log.Info("received gradient from trainer, updating gradient.", + "name", g.Name, "size", len(g.Content), "type", g.ElementType) return o.UpdateParameter(g) } @@ -277,7 +329,7 @@ func (s *Service) GetParam(name string, parameter *Parameter) error { parameter.Name = name parameter.ElementType = opt.elementType parameter.Content = opt.GetWeights() - + log.Debug(parameter.String()) log.Info("sending parameter to the trainer", "name", parameter.Name, "size", len(parameter.Content), "type", parameter.ElementType) return nil } diff --git a/go/pserver/service_test.go b/go/pserver/service_test.go index b6f4566eb78cf797e3738afa5f86f5c4e8090d85..58a743e1fadff9d629f682d660e661013c33ac8a 100644 --- a/go/pserver/service_test.go +++ b/go/pserver/service_test.go @@ -15,6 +15,7 @@ package pserver_test import ( + "fmt" "io/ioutil" "reflect" "sync" @@ -178,3 +179,33 @@ func TestBlockUntilInitialized(t *testing.T) { wg.Wait() } + +func TestGradientString(t *testing.T) { + g := pserver.Parameter{} + g.ElementType = pserver.Float32 + g.Content = []byte{0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40} + if g.String() != "[3.3702806e+12 2.142699 3.3702806e+12 2.142699]" { + t.Fatal("get float data error!") + } + + g.Content = []byte{0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40, + 0x18, 0x2d, 0x44, 0x54, 0xfb, 0x21, 0x09, 0x40} + if g.String() != "[3.3702806e+12 2.142699 3.3702806e+12 2.142699 3.3702806e+12 2.142699 3.3702806e+12 2.142699 3.3702806e+12 2.142699...3.3702806e+12 2.142699 3.3702806e+12 2.142699 3.3702806e+12 2.142699 3.3702806e+12 2.142699 3.3702806e+12 2.142699]" { + t.Fatal("get float data error!", g.String()) + } + fmt.Println(g) +} diff --git a/paddle/operators/accuracy_op.cc b/paddle/operators/accuracy_op.cc index eb8bce8da70a128bd1e0d36540dce5e296540629..88958e1634c51c5d217c22333f2aaf4a5adf89fd 100644 --- a/paddle/operators/accuracy_op.cc +++ b/paddle/operators/accuracy_op.cc @@ -32,7 +32,8 @@ class AccuracyOp : public framework::OperatorWithKernel { auto inference_dim = ctx->GetInputDim("Inference"); auto label_dim = ctx->GetInputDim("Label"); - PADDLE_ENFORCE_EQ(label_dim.size(), 1, "label must be a vector"); + PADDLE_ENFORCE_EQ(label_dim.size(), 2, "label's rank must be 2."); + PADDLE_ENFORCE_EQ(label_dim[1], 1, "label's second dimension must be 1"); PADDLE_ENFORCE_EQ(inference_dim[0], label_dim[0], "inference size must be the same as label size"); @@ -68,7 +69,8 @@ information, or not. But the output only shares the LoD with input `Inference`. } // namespace paddle namespace ops = paddle::operators; -REGISTER_OP_WITHOUT_GRADIENT(accuracy, ops::AccuracyOp, ops::AccuracyOpMaker); +REGISTER_OPERATOR(accuracy, ops::AccuracyOp, ops::AccuracyOpMaker, + paddle::framework::EmptyGradOpMaker); REGISTER_OP_CPU_KERNEL( accuracy, ops::AccuracyKernel, ops::AccuracyKernel); diff --git a/paddle/operators/activation_op.h b/paddle/operators/activation_op.h index e4c6b2e09cd71f00a2ef73173205b9066c34fcf5..ddd966e26c9abad0d83f8b5c6e3e7d9ad65158a8 100644 --- a/paddle/operators/activation_op.h +++ b/paddle/operators/activation_op.h @@ -547,6 +547,7 @@ struct ELUGradFunctor : public BaseActivationFunctor { } }; +// FIXME(qijun) https://github.com/PaddlePaddle/Paddle/issues/5198 template struct PowFunctor : public BaseActivationFunctor { float factor; diff --git a/paddle/operators/top_k_op.cc b/paddle/operators/top_k_op.cc index d5c2c91a5fb0f639ea84d13e27de8271218da54f..ac9257259548dc4223efac239d8362f69366850c 100644 --- a/paddle/operators/top_k_op.cc +++ b/paddle/operators/top_k_op.cc @@ -52,7 +52,11 @@ class TopkOpMaker : public framework::OpProtoAndCheckerMaker { AddOutput("Out", "The output tensor of Topk op"); AddOutput("Indices", "The indices of Topk elements of input"); AddComment( - R"DOC(If the input is a vector (1d tensor), finds the k largest entries in the vector and outputs their values and indices as vectors. Thus values[j] is the j-th largest entry in input, and its index is indices[j]. + R"DOC(If the input is a vector (1d tensor), + finds the k largest entries in the vector + and outputs their values and indices as vectors. + Thus values[j] is the j-th largest entry in input, + and its index is indices[j]. For matrices, computes the top k entries in each row. )DOC"); AddAttr("k", @@ -66,6 +70,7 @@ class TopkOpMaker : public framework::OpProtoAndCheckerMaker { } // namespace paddle namespace ops = paddle::operators; -REGISTER_OP_WITHOUT_GRADIENT(top_k, ops::TopkOp, ops::TopkOpMaker); +REGISTER_OPERATOR(top_k, ops::TopkOp, ops::TopkOpMaker, + paddle::framework::EmptyGradOpMaker); REGISTER_OP_CPU_KERNEL(top_k, ops::TopkKernel); diff --git a/paddle/platform/CMakeLists.txt b/paddle/platform/CMakeLists.txt index eb850b658583f2256629d63fdb64248dbf249937..bd86a9fe268c277065cd450f91b544def6c4d32f 100644 --- a/paddle/platform/CMakeLists.txt +++ b/paddle/platform/CMakeLists.txt @@ -9,7 +9,6 @@ cc_test(place_test SRCS place_test.cc DEPS place glog gflags) add_subdirectory(dynload) cc_test(enforce_test SRCS enforce_test.cc DEPS stringpiece) -cc_test(environment_test SRCS environment_test.cc DEPS stringpiece) IF(WITH_GPU) set(GPU_CTX_DEPS dynload_cuda dynamic_loader) diff --git a/paddle/platform/environment.h b/paddle/platform/environment.h deleted file mode 100644 index 4edcce932edc61453cef74f2c4ee0f72496b3677..0000000000000000000000000000000000000000 --- a/paddle/platform/environment.h +++ /dev/null @@ -1,60 +0,0 @@ -/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. */ - -#pragma once - -#include -#include -#include - -#include "paddle/platform/enforce.h" -#include "paddle/string/piece.h" - -extern char** environ; // for environment variables - -namespace paddle { -namespace platform { - -inline void SetEnvVariable(const std::string& name, const std::string& value) { - PADDLE_ENFORCE_NE(setenv(name.c_str(), value.c_str(), 1), -1, - "Failed to set environment variable %s=%s", name, value); -} - -inline void UnsetEnvVariable(const std::string& name) { - PADDLE_ENFORCE_NE(unsetenv(name.c_str()), -1, - "Failed to unset environment variable %s", name); -} - -inline bool IsEnvVarDefined(const std::string& name) { - return std::getenv(name.c_str()) != nullptr; -} - -inline std::string GetEnvValue(const std::string& name) { - PADDLE_ENFORCE(IsEnvVarDefined(name), - "Tried to access undefined environment variable %s", name); - return std::getenv(name.c_str()); -} - -inline std::vector GetAllEnvVariables() { - std::vector vars; - for (auto var = environ; *var != nullptr; ++var) { - auto tail = string::Index(*var, "="); - auto name = string::SubStr(*var, 0, tail).ToString(); - vars.push_back(name); - } - return vars; -} - -} // namespace platform -} // namespace paddle diff --git a/paddle/platform/environment_test.cc b/paddle/platform/environment_test.cc deleted file mode 100644 index 5f136527215d6a676cfa1a3b08f09dfd3ab24a90..0000000000000000000000000000000000000000 --- a/paddle/platform/environment_test.cc +++ /dev/null @@ -1,54 +0,0 @@ -/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. */ - -#include "paddle/platform/environment.h" - -#include "glog/logging.h" -#include "gtest/gtest.h" - -TEST(ENVIRONMENT, ACCESS) { - namespace platform = paddle::platform; - namespace string = paddle::string; - - platform::SetEnvVariable("PADDLE_USE_ENV", "TRUE"); - - EXPECT_TRUE(platform::IsEnvVarDefined("PADDLE_USE_ENV")); - EXPECT_EQ(platform::GetEnvValue("PADDLE_USE_ENV"), "TRUE"); - - platform::UnsetEnvVariable("PADDLE_USE_ENV"); - EXPECT_FALSE(platform::IsEnvVarDefined("PADDLE_USE_ENV")); - - platform::SetEnvVariable("PADDLE_USE_ENV1", "Hello "); - platform::SetEnvVariable("PADDLE_USE_ENV2", "World, "); - platform::SetEnvVariable("PADDLE_USE_ENV3", "PaddlePaddle!"); - - std::string env_info; - auto vars = platform::GetAllEnvVariables(); - for_each(vars.begin(), vars.end(), [&](const std::string& var) { - env_info += platform::GetEnvValue(var); - }); - - EXPECT_TRUE(string::Contains(env_info, "Hello World, PaddlePaddle!")); - platform::UnsetEnvVariable("PADDLE_USE_ENV1"); - platform::UnsetEnvVariable("PADDLE_USE_ENV2"); - platform::UnsetEnvVariable("PADDLE_USE_ENV3"); - - env_info.clear(); - vars = platform::GetAllEnvVariables(); - for_each(vars.begin(), vars.end(), [&](const std::string& var) { - env_info += platform::GetEnvValue(var); - }); - - EXPECT_FALSE(string::Contains(env_info, "Hello World, PaddlePaddle!")); - EXPECT_FALSE(platform::IsEnvVarDefined("PADDLE_USE_ENV1")); - EXPECT_FALSE(platform::IsEnvVarDefined("PADDLE_USE_ENV2")); - EXPECT_FALSE(platform::IsEnvVarDefined("PADDLE_USE_ENV3")); -} diff --git a/paddle/platform/gpu_info.cc b/paddle/platform/gpu_info.cc index 0cab5ffc5609bbd6fd08c74329d8370fb95f8102..f3455a8733862c91eaece629b6684d446672336c 100644 --- a/paddle/platform/gpu_info.cc +++ b/paddle/platform/gpu_info.cc @@ -17,7 +17,6 @@ limitations under the License. */ #include "gflags/gflags.h" #include "paddle/platform/enforce.h" -#include "paddle/platform/environment.h" DEFINE_double(fraction_of_gpu_memory_to_use, 0.95, "Default use 95% of GPU memory for PaddlePaddle," @@ -75,13 +74,6 @@ size_t GpuMaxChunkSize() { GpuMemoryUsage(available, total); - if (IsEnvVarDefined(kEnvFractionGpuMemoryToUse)) { - auto val = std::stod(GetEnvValue(kEnvFractionGpuMemoryToUse)); - PADDLE_ENFORCE_GT(val, 0.0); - PADDLE_ENFORCE_LE(val, 1.0); - FLAGS_fraction_of_gpu_memory_to_use = val; - } - // Reserving the rest memory for page tables, etc. size_t reserving = (1 - FLAGS_fraction_of_gpu_memory_to_use) * total; diff --git a/paddle/pybind/pybind.cc b/paddle/pybind/pybind.cc index 4baff895daa1f26fe3f344372f2686fbcbb82098..2a0075356ed1e0f0b3501ac681c5e3a1bf37e2ca 100644 --- a/paddle/pybind/pybind.cc +++ b/paddle/pybind/pybind.cc @@ -15,6 +15,7 @@ limitations under the License. */ #include "paddle/pybind/protobuf.h" #include // for call_once +#include #include "gflags/gflags.h" #include "paddle/framework/backward.h" #include "paddle/framework/executor.h" @@ -42,9 +43,9 @@ limitations under the License. */ namespace paddle { namespace pybind { -static size_t UniqueIntegerGenerator() { - static std::atomic generator; - return generator.fetch_add(1); +static size_t UniqueIntegerGenerator(const std::string &prefix) { + static std::unordered_map> generators; + return generators[prefix].fetch_add(1); } std::once_flag gflags_init_flag; diff --git a/proto/CMakeLists.txt b/proto/CMakeLists.txt index 5d898d860cfc6dc26eaf5a81d8aed6d757ed5831..556bcd1d7e60c27fece43de666e9531ab4203414 100644 --- a/proto/CMakeLists.txt +++ b/proto/CMakeLists.txt @@ -27,3 +27,30 @@ foreach(filename ${proto_filenames}) endforeach() add_custom_target(gen_proto_py ALL DEPENDS ${PROTO_GEN_PY}) + + +if (WITH_GOLANG) + add_custom_target(protoc-gen-go) + add_custom_command(TARGET protoc-gen-go + COMMAND go + ARGS "get" "-u" "github.com/golang/protobuf/protoc-gen-go") + + set(PROTO_GEN_GO) + file(GLOB proto_filenames . OptimizerConfig.proto) + foreach(filename ${proto_filenames}) + message(STATUS ${filename}) + get_filename_component(ABS_FIL ${filename} ABSOLUTE) + get_filename_component(FIL_WE ${filename} NAME_WE) + set(CUR_PROTO_GEN_GO + ${PADDLE_SOURCE_DIR}/paddle/go/proto/${FIL_WE}.pb.go) + set(PROTO_GEN_GO + ${CUR_PROTO_GEN_GO} + ${PROTO_GEN_GO}) + add_custom_command(OUTPUT ${CUR_PROTO_GEN_GO} + COMMAND ${PROTOBUF_PROTOC_EXECUTABLE} + ARGS "--go_out=${PADDLE_SOURCE_DIR}/go/proto" + "-I" ${CMAKE_CURRENT_SOURCE_DIR} ${ABS_FIL} + DEPENDS ${ABS_FIL} protoc protoc-gen-go) + endforeach() + add_custom_target(gen_proto_go ALL DEPENDS ${PROTO_GEN_GO}) +endif() diff --git a/python/paddle/v2/framework/framework.py b/python/paddle/v2/framework/framework.py index 43101c9ddad76b7c1c322130dc0362a5c8ea4336..f8d2f67410a6c06a1642180d2d62c881ec6bda3d 100644 --- a/python/paddle/v2/framework/framework.py +++ b/python/paddle/v2/framework/framework.py @@ -119,8 +119,9 @@ class Variable(object): @staticmethod def _unique_var_name_(): - uid = core.unique_integer() # unique during whole process. - return "_generated_var_%d" % uid + prefix = "_generated_var" + uid = core.unique_integer(prefix) # unique during whole process. + return "_".join([prefix, str(uid)]) @staticmethod def _convert_np_dtype_to_dtype_(np_dtype): diff --git a/python/paddle/v2/framework/layer_helper.py b/python/paddle/v2/framework/layer_helper.py index 1f72c9bc7b0ceda1dd954703fcc10c77a3e5ed25..d96dbe172c22617182e7ebf4aab175c6142352b7 100644 --- a/python/paddle/v2/framework/layer_helper.py +++ b/python/paddle/v2/framework/layer_helper.py @@ -8,7 +8,7 @@ from paddle.v2.framework.framework import Variable, g_program, \ def unique_name(prefix): - uid = core.unique_integer() # unique during whole process. + uid = core.unique_integer(prefix) # unique during whole process. return "_".join([prefix, str(uid)]) diff --git a/python/paddle/v2/framework/layers.py b/python/paddle/v2/framework/layers.py index 57723c4d5aa09a6756b2959aa45e716d1c1dc140..4727d139a28541e06a2dcbcb7bf28510ee123775 100644 --- a/python/paddle/v2/framework/layers.py +++ b/python/paddle/v2/framework/layers.py @@ -5,7 +5,7 @@ import re __all__ = [ 'fc', 'data', 'cross_entropy', 'conv2d', 'pool2d', 'embedding', 'concat', - 'StaticRNN', 'cast', 'sequence_conv', 'sequence_pool' + 'StaticRNN', 'cast', 'sequence_conv', 'sequence_pool', 'accuracy' ] @@ -225,13 +225,30 @@ def square_error_cost(input, label, **kwargs): square_out = helper.create_tmp_variable(dtype=input.data_type) helper.append_op( - type='pow', - inputs={'X': [minus_out]}, - outputs={'Y': [square_out]}, - attrs={'factor': 2.0}) + type='square', inputs={'X': [minus_out]}, outputs={'Y': [square_out]}) return square_out +def accuracy(input, label, k=1, **kwargs): + helper = LayerHelper("accuracy", **kwargs) + topk_out = helper.create_tmp_variable(dtype=input.data_type) + topk_indices = helper.create_tmp_variable(dtype="int64") + helper.append_op( + type="top_k", + inputs={"X": [input]}, + outputs={"Out": [topk_out], + "Indices": [topk_indices]}, + attrs={"k": k}) + acc_out_dtype = kwargs.get("out_dtype", "float32") + acc_out = helper.create_tmp_variable(dtype=acc_out_dtype) + helper.append_op( + type="accuracy", + inputs={"Inference": [topk_indices], + "Label": [label]}, + outputs={"Accuracy": [acc_out]}) + return acc_out + + def sequence_conv(input, num_filters, name=None, diff --git a/python/paddle/v2/framework/tests/op_test.py b/python/paddle/v2/framework/tests/op_test.py index 50360e6e729df2957a5c7fe871100b5a53bd9305..2e6710b5fcfe5a531067498e38a4cb93d3165602 100644 --- a/python/paddle/v2/framework/tests/op_test.py +++ b/python/paddle/v2/framework/tests/op_test.py @@ -281,7 +281,8 @@ class OpTest(unittest.TestCase): type(sub_out)) for sub_out_name, expect in sub_out: idx = find_actual(sub_out_name, fetch_list) - actual_t = np.array(outs[idx]) + actual = outs[idx] + actual_t = np.array(actual) expect_t = expect[0] \ if isinstance(expect, tuple) else expect self.assertTrue( @@ -291,11 +292,12 @@ class OpTest(unittest.TestCase): str(place)) if isinstance(expect, tuple): self.assertListEqual( - actual_t.lod(), expect[1], "Output (" + sub_out_name - + ") has different lod at " + str(place)) + actual.lod(), expect[1], "Output (" + sub_out_name + + ") has different lod at " + str(place)) else: idx = find_actual(out_name, fetch_list) - actual_t = outs[idx] + actual = outs[idx] + actual_t = np.array(actual) expect = self.outputs[out_name] expect_t = expect[0] if isinstance(expect, tuple) else expect self.assertTrue( @@ -303,7 +305,7 @@ class OpTest(unittest.TestCase): actual_t, expect_t, atol=atol), "Output (" + out_name + ") has diff at " + str(place)) if isinstance(expect, tuple): - self.assertListEqual(actual_t.lod(), expect[1], + self.assertListEqual(actual.lod(), expect[1], "Output (" + out_name + ") has different lod at " + str(place)) diff --git a/python/paddle/v2/framework/tests/test_accuracy_op.py b/python/paddle/v2/framework/tests/test_accuracy_op.py index 02be9a02910bee3eae63e12cceaa51cf53591539..f17edd44aefa870ce704552077d91e661a9167de 100644 --- a/python/paddle/v2/framework/tests/test_accuracy_op.py +++ b/python/paddle/v2/framework/tests/test_accuracy_op.py @@ -8,12 +8,12 @@ class TestAccuracyOp(OpTest): self.op_type = "accuracy" n = 8192 infer = np.random.randint(0, 2, (n, 1)).astype("int") - label = np.random.randint(0, 2, (n, )).astype("int") + label = np.random.randint(0, 2, (n, 1)).astype("int") self.inputs = {'Inference': infer, "Label": label} num_correct = 0 for rowid in xrange(n): for ele in infer[rowid]: - if ele == label[rowid]: + if ele == label[rowid][0]: num_correct += 1 break self.outputs = { diff --git a/python/paddle/v2/framework/tests/test_image_classification_layer.py b/python/paddle/v2/framework/tests/test_image_classification_layer.py index 7411689b615aeaa900fecb8c5ae00cf92203289c..b4eda13552e60f009ec910e3d21e9d77107133a1 100644 --- a/python/paddle/v2/framework/tests/test_image_classification_layer.py +++ b/python/paddle/v2/framework/tests/test_image_classification_layer.py @@ -37,7 +37,7 @@ class TestLayer(unittest.TestCase): layers.batch_norm( input=images, program=program, init_program=init_program) - #print str(program) + # print str(program) def test_dropout_layer(self): program = Program() @@ -53,7 +53,7 @@ class TestLayer(unittest.TestCase): program=program, init_program=init_program) - #print str(program) + # print str(program) def test_img_conv_group(self): program = Program() diff --git a/python/paddle/v2/framework/tests/test_recognize_digits_conv.py b/python/paddle/v2/framework/tests/test_recognize_digits_conv.py index a9b6c8410e2af36e6928b2fac919398473611728..92b1d0542619b765cc32c98f59604cfc73d7d6d4 100644 --- a/python/paddle/v2/framework/tests/test_recognize_digits_conv.py +++ b/python/paddle/v2/framework/tests/test_recognize_digits_conv.py @@ -51,12 +51,14 @@ predict = layers.fc(input=conv_pool_2, cost = layers.cross_entropy( input=predict, label=label, program=program, init_program=init_program) avg_cost = layers.mean(x=cost, program=program) +accuracy = layers.accuracy( + input=predict, label=label, program=program, init_program=init_program) sgd_optimizer = optimizer.SGDOptimizer(learning_rate=0.001) opts = sgd_optimizer.minimize(avg_cost) BATCH_SIZE = 50 -PASS_NUM = 1 +PASS_NUM = 3 train_reader = paddle.batch( paddle.reader.shuffle( paddle.dataset.mnist.train(), buf_size=500), @@ -83,10 +85,11 @@ for pass_id in range(PASS_NUM): outs = exe.run(program, feed={"pixel": tensor_img, "label": tensor_y}, - fetch_list=[avg_cost]) - + fetch_list=[avg_cost, accuracy]) loss = np.array(outs[0]) + acc = np.array(outs[1]) - if loss < 10.0: - exit(0) # if avg cost less than 10.0, we think our code is good. + if loss < 10.0 and acc > 0.9: + # if avg cost less than 10.0 and accuracy is larger than 0.9, we think our code is good. + exit(0) exit(1) diff --git a/python/paddle/v2/trainer.py b/python/paddle/v2/trainer.py index b68fd0d5a97a7993ddd0a1d947304fa5428c01b8..db01ab7374eca18b6063dc634da5ef83c4bc9adc 100644 --- a/python/paddle/v2/trainer.py +++ b/python/paddle/v2/trainer.py @@ -205,7 +205,8 @@ class SGD(object): """ Testing method. Will test input data. - :param reader: A reader that reads and yeilds data items. + :param reader: A batch reader that reads and yeilds data items, + it should be a paddle.v2.batch. :type reader: collections.Iterable :param feeding: Feeding is a map of neural network input name and array index that reader returns.