Skip to content

P
PaddleDetection

PaddlePaddle / PaddleDetection
大约 1 年 前同步成功

通知 694
Star 11112
Fork 2696
P
PaddleDetection

体验新版 GitCode，发现更多精彩内容 >>

提交 96615fe3 编写于 作者: L Luo Tao
浏览文件
操作

merge develop, fix conflict

上级 496d64eb 4f1bf30d
展开全部 隐藏空白更改
内联 并排
Showing with 13472 addition and 9010 deletion
.pre-commit-config.yaml
浏览文件 @ 96615fe3
......@@ -7,18 +7,14 @@
hooks:
- id: yapf
- repo: https://github.com/pre-commit/pre-commit-hooks
sha: 4ef03c4223ad322c7adaa6c6c0efb26b57df3b71
sha: 7539d8bd1a00a3c1bfd34cdb606d3a6372e83469
hooks:
- id: check-added-large-files
- id: check-merge-conflict
- id: check-symlinks
- id: detect-private-key
- id: end-of-file-fixer
# TODO(yuyang): trailing whitespace has some bugs on markdown
# files now, please not add it to pre-commit hook now
# - id: trailing-whitespace
#
# TODO(yuyang): debug-statements not fit for Paddle, because
# not all of our python code is runnable. Some are used for
# documenation
# - id: debug-statements
- repo: https://github.com/PaddlePaddle/clang-format-pre-commit-hook.git
sha: 28c0ea8a67a3e2dbbf4822ef44e85b63a0080a29
hooks:
- id: clang-formater
.travis.yml
浏览文件 @ 96615fe3
......@@ -42,7 +42,7 @@ addons:
before_install:
- |
if [ ${JOB} == "BUILD_AND_TEST" ]; then
if ! git diff --name-only $TRAVIS_COMMIT_RANGE | grep -qvE '(\.md$)'
if ! git diff --name-only $TRAVIS_COMMIT_RANGE | grep -qvE '(\.md$)|(\.rst$)|(\.jpg$)|(\.png$)'
then
echo "Only markdown docs were updated, stopping build process."
exit
......
CMakeLists.txt
浏览文件 @ 96615fe3
......@@ -2,8 +2,8 @@ cmake_minimum_required(VERSION 2.8)
project(paddle CXX C)
set(PADDLE_MAJOR_VERSION 0)
set(PADDLE_MINOR_VERSION 8)
set(PADDLE_PATCH_VERSION 0b3)
set(PADDLE_MINOR_VERSION 9)
set(PADDLE_PATCH_VERSION 0a0)
set(PADDLE_VERSION ${PADDLE_MAJOR_VERSION}.${PADDLE_MINOR_VERSION}.${PADDLE_PATCH_VERSION})
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake")
......@@ -36,6 +36,7 @@ option(WITH_RDMA "Compile PaddlePaddle with rdma support" OFF)
option(WITH_GLOG "Compile PaddlePaddle use glog, otherwise use a log implement internally" ${LIBGLOG_FOUND})
option(WITH_GFLAGS "Compile PaddlePaddle use gflags, otherwise use a flag implement internally" ${GFLAGS_FOUND})
option(WITH_TIMER "Compile PaddlePaddle use timer" OFF)
option(WITH_PROFILER "Compile PaddlePaddle use gpu profiler" OFF)
option(WITH_TESTING "Compile and run unittest for PaddlePaddle" ${GTEST_FOUND})
option(WITH_DOC "Compile PaddlePaddle with documentation" OFF)
option(WITH_SWIG_PY "Compile PaddlePaddle with py PaddlePaddle prediction api" ${SWIG_FOUND})
......@@ -115,7 +116,6 @@ else()
endif(WITH_AVX)
if(WITH_DSO)
set(CUDA_LIBRARIES "")
add_definitions(-DPADDLE_USE_DSO)
endif(WITH_DSO)
......@@ -135,6 +135,10 @@ if(NOT WITH_TIMER)
add_definitions(-DPADDLE_DISABLE_TIMER)
endif(NOT WITH_TIMER)
if(NOT WITH_PROFILER)
add_definitions(-DPADDLE_DISABLE_PROFILER)
endif(NOT WITH_PROFILER)
if(WITH_AVX)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${AVX_FLAG}")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${AVX_FLAG}")
......
README.md
浏览文件 @ 96615fe3
# PaddlePaddle
[![Build Status](https://travis-ci.org/baidu/Paddle.svg?branch=master)](https://travis-ci.org/baidu/Paddle)
[![Coverage Status](https://coveralls.io/repos/github/baidu/Paddle/badge.svg?branch=develop)](https://coveralls.io/github/baidu/Paddle?branch=develop)
[![Join the chat at https://gitter.im/PaddlePaddle/Deep_Learning](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/PaddlePaddle/Deep_Learning?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
[![Build Status](https://travis-ci.org/PaddlePaddle/Paddle.svg?branch=develop)](https://travis-ci.org/PaddlePaddle/Paddle)
[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat)](http://www.paddlepaddle.org/)
[![Documentation Status](https://img.shields.io/badge/中文文档-最新-brightgreen.svg)](http://www.paddlepaddle.org/cn/index.html)
[![Coverage Status](https://coveralls.io/repos/github/PaddlePaddle/Paddle/badge.svg?branch=develop)](https://coveralls.io/github/PaddlePaddle/Paddle?branch=develop)
[![Release](https://img.shields.io/github/release/PaddlePaddle/Paddle.svg)](https://github.com/PaddlePaddle/Paddle/releases)
[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE)
Welcome to the PaddlePaddle GitHub.
......@@ -14,7 +17,7 @@ developed by Baidu scientists and engineers for the purpose of applying deep
learning to many products at Baidu.
Our vision is to enable deep learning for everyone via PaddlePaddle.
Please refer to our [release announcement](https://github.com/baidu/Paddle/releases) to track the latest feature of PaddlePaddle.
Please refer to our [release announcement](https://github.com/PaddlePaddle/Paddle/releases) to track the latest feature of PaddlePaddle.
## Features
......@@ -26,15 +29,15 @@ Please refer to our [release announcement](https://github.com/baidu/Paddle/relea
connection.
- **Efficiency**
In order to unleash the power of heterogeneous computing resource,
optimization occurs at different levels of PaddlePaddle, including
computing, memory, architecture and communication. The following are some
examples:
- Optimized math operations through SSE/AVX intrinsics, BLAS libraries
(e.g. MKL, ATLAS, cuBLAS) or customized CPU/GPU kernels.
- Highly optimized recurrent networks which can handle **variable-length**
(e.g. MKL, ATLAS, cuBLAS) or customized CPU/GPU kernels.
- Highly optimized recurrent networks which can handle **variable-length**
sequence without padding.
- Optimized local and distributed training for models with high dimensional
sparse data.
......@@ -57,41 +60,39 @@ Please refer to our [release announcement](https://github.com/baidu/Paddle/relea
## Installation
Check out the [Install Guide](http://paddlepaddle.org/doc/build/) to install from
pre-built packages (**docker image**, **deb package**) or
pre-built packages (**docker image**, **deb package**) or
directly build on **Linux** and **Mac OS X** from the source code.
## Documentation
Both [English Docs](http://paddlepaddle.org/doc/) and [Chinese Docs](http://paddlepaddle.org/doc_cn/) are provided for our users and developers.
- [Quick Start](http://paddlepaddle.org/doc/demo/quick_start/index_en) <br>
You can follow the quick start tutorial to learn how use PaddlePaddle
step-by-step.
- [Example and Demo](http://paddlepaddle.org/doc/demo/) <br>
We provide five demos, including: image classification, sentiment analysis,
sequence to sequence model, recommendation, semantic role labeling.
sequence to sequence model, recommendation, semantic role labeling.
- [Distributed Training](http://paddlepaddle.org/doc/cluster) <br>
This system supports training deep learning models on multiple machines
with data parallelism.
- [Python API](http://paddlepaddle.org/doc/ui/) <br>
PaddlePaddle supports using either Python interface or C++ to build your
system. We also use SWIG to wrap C++ source code to create a user friendly
interface for Python. You can also use SWIG to create interface for your
favorite programming language.
- [How to Contribute](http://paddlepaddle.org/doc/build/contribute_to_paddle.html) <br>
We sincerely appreciate your interest and contributions. If you would like to
contribute, please read the contribution guide.
contribute, please read the contribution guide.
- [Source Code Documents](http://paddlepaddle.org/doc/source/) <br>
## Ask Questions
Please join the [**gitter chat**](https://gitter.im/PaddlePaddle/Deep_Learning) or send email to
**paddle-dev@baidu.com** to ask questions and talk about methods and models.
Framework development discussions and
bug reports are collected on [Issues](https://github.com/baidu/paddle/issues).
You are welcome to submit questions and bug reports as [Github Issues](https://github.com/PaddlePaddle/Paddle/issues).
## Copyright and License
PaddlePaddle is provided under the [Apache-2.0 license](LICENSE).
demo/image_classification/train.sh
浏览文件 @ 96615fe3
......@@ -24,7 +24,7 @@ paddle train \
--test_all_data_in_one_period=1 \
--use_gpu=1 \
--trainer_count=1 \
--num_passes=200 \
--num_passes=300 \
--save_dir=$output \
2>&1 | tee $log
......
demo/model_zoo/embedding/pre_DictAndModel.sh
浏览文件 @ 96615fe3
......@@ -18,7 +18,5 @@ set -x
# download the dictionary and pretrained model
for file in baidu.dict model_32.emb model_64.emb model_128.emb model_256.emb
do
# following is the google drive address
# you can also directly download from https://pan.baidu.com/s/1o8q577s
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/embedding/$file --no-check-certificate
wget http://paddlepaddle.bj.bcebos.com/model_zoo/embedding/$file
done
demo/model_zoo/resnet/get_model.sh
浏览文件 @ 96615fe3
......@@ -24,9 +24,7 @@ echo "Downloading ResNet models..."
for file in resnet_50.tar.gz resnet_101.tar.gz resnet_152.tar.gz mean_meta_224.tar.gz
do
# following is the google drive address
# you can also directly download from https://pan.baidu.com/s/1o8q577s
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/imagenet/$file --no-check-certificate
wget http://paddlepaddle.bj.bcebos.com/model_zoo/imagenet/$file
tar -xvf $file
rm $file
done
......
demo/quick_start/preprocess.sh
浏览文件 @ 96615fe3
......@@ -23,7 +23,7 @@ set -e
export LC_ALL=C
UNAME_STR=`uname`
if [[ ${UNAME_STR} == 'Linux' ]]; then
if [ ${UNAME_STR} == 'Linux' ]; then
SHUF_PROG='shuf'
else
SHUF_PROG='gshuf'
......
demo/semantic_role_labeling/data/extract_dict_feature.py
浏览文件 @ 96615fe3
......@@ -17,24 +17,15 @@ import os
from optparse import OptionParser
def extract_dict_features(pair_file, feature_file, src_dict_file,
tgt_dict_file):
src_dict = set()
tgt_dict = set()
with open(pair_file) as fin, open(feature_file, 'w') as feature_out, open(
src_dict_file, 'w') as src_dict_out, open(tgt_dict_file,
'w') as tgt_dict_out:
def extract_dict_features(pair_file, feature_file):
with open(pair_file) as fin, open(feature_file, 'w') as feature_out:
for line in fin:
sentence, labels = line.strip().split('\t')
sentence, predicate, labels = line.strip().split('\t')
sentence_list = sentence.split()
labels_list = labels.split()
src_dict.update(sentence_list)
tgt_dict.update(labels_list)
verb_index = labels_list.index('B-V')
verb_feature = sentence_list[verb_index]
mark = [0] * len(labels_list)
if verb_index > 0:
......@@ -42,47 +33,50 @@ def extract_dict_features(pair_file, feature_file, src_dict_file,
ctx_n1 = sentence_list[verb_index - 1]
else:
ctx_n1 = 'bos'
ctx_n1_feature = ctx_n1
if verb_index > 1:
mark[verb_index - 2] = 1
ctx_n2 = sentence_list[verb_index - 2]
else:
ctx_n2 = 'bos'
mark[verb_index] = 1
ctx_0_feature = sentence_list[verb_index]
ctx_0 = sentence_list[verb_index]
if verb_index < len(labels_list) - 2:
mark[verb_index + 1] = 1
ctx_p1 = sentence_list[verb_index + 1]
else:
ctx_p1 = 'eos'
ctx_p1_feature = ctx_p1
if verb_index < len(labels_list) - 3:
mark[verb_index + 2] = 1
ctx_p2 = sentence_list[verb_index + 2]
else:
ctx_p2 = 'eos'
feature_str = sentence + '\t' \
+ verb_feature + '\t' \
+ ctx_n1_feature + '\t' \
+ ctx_0_feature + '\t' \
+ ctx_p1_feature + '\t' \
+ predicate + '\t' \
+ ctx_n2 + '\t' \
+ ctx_n1 + '\t' \
+ ctx_0 + '\t' \
+ ctx_p1 + '\t' \
+ ctx_p2 + '\t' \
+ ' '.join([str(i) for i in mark]) + '\t' \
+ labels
feature_out.write(feature_str + '\n')
src_dict_out.write('<unk>\n')
src_dict_out.write('\n'.join(list(src_dict)))
tgt_dict_out.write('\n'.join(list(tgt_dict)))
if __name__ == '__main__':
usage = '-p pair_file -f feature_file -s source dictionary -t target dictionary '
usage = '-p pair_file -f feature_file'
parser = OptionParser(usage)
parser.add_option('-p', dest='pair_file', help='the pair file')
parser.add_option(
'-f', dest='feature_file', help='the file to store feature')
parser.add_option(
'-s', dest='src_dict', help='the file to store source dictionary')
parser.add_option(
'-t', dest='tgt_dict', help='the file to store target dictionary')
parser.add_option('-f', dest='feature_file', help='the feature file')
(options, args) = parser.parse_args()
extract_dict_features(options.pair_file, options.feature_file,
options.src_dict, options.tgt_dict)
extract_dict_features(options.pair_file, options.feature_file)
demo/semantic_role_labeling/data/extract_pairs.py
浏览文件 @ 96615fe3
......@@ -51,7 +51,7 @@ def read_sentences(words_file):
for line in fin:
line = line.strip()
if line == '':
sentences.append(s.lower())
sentences.append(s)
s = ''
else:
s += line + ' '
......@@ -64,6 +64,11 @@ def transform_labels(sentences, labels):
if len(labels[i]) == 1:
continue
else:
verb_list = []
for x in labels[i][0]:
if x !='-':
verb_list.append(x)
for j in xrange(1, len(labels[i])):
label_list = labels[i][j]
current_tag = 'O'
......@@ -88,8 +93,7 @@ def transform_labels(sentences, labels):
is_in_bracket = True
else:
print 'error:', ll
sen_lab_pair.append((sentences[i], label_seq))
sen_lab_pair.append((sentences[i], verb_list[j-1], label_seq))
return sen_lab_pair
......@@ -97,9 +101,9 @@ def write_file(sen_lab_pair, output_file):
with open(output_file, 'w') as fout:
for x in sen_lab_pair:
sentence = x[0]
label_seq = ' '.join(x[1])
assert len(sentence.split()) == len(x[1])
fout.write(sentence + '\t' + label_seq + '\n')
label_seq = ' '.join(x[2])
assert len(sentence.split()) == len(x[2])
fout.write(sentence + '\t' + x[1]+'\t' +label_seq + '\n')
if __name__ == '__main__':
......
demo/semantic_role_labeling/data/get_data.sh
浏览文件 @ 96615fe3
......@@ -14,6 +14,10 @@
# limitations under the License.
set -e
wget http://www.cs.upc.edu/~srlconll/conll05st-tests.tar.gz
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/semantic_role_labeling/verbDict.txt --no-check-certificate
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/semantic_role_labeling/targetDict.txt --no-check-certificate
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/semantic_role_labeling/wordDict.txt --no-check-certificate
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/semantic_role_labeling/emb --no-check-certificate
tar -xzvf conll05st-tests.tar.gz
rm conll05st-tests.tar.gz
cp ./conll05st-release/test.wsj/words/test.wsj.words.gz .
......@@ -22,4 +26,4 @@ gunzip test.wsj.words.gz
gunzip test.wsj.props.gz
python extract_pairs.py -w test.wsj.words -p test.wsj.props -o test.wsj.seq_pair
python extract_dict_feature.py -p test.wsj.seq_pair -f feature -s src.dict -t tgt.dict
python extract_dict_feature.py -p test.wsj.seq_pair -f feature
demo/semantic_role_labeling/dataprovider.py
浏览文件 @ 96615fe3
......@@ -17,41 +17,52 @@ from paddle.trainer.PyDataProvider2 import *
UNK_IDX = 0
def hook(settings, word_dict, label_dict, **kwargs):
def hook(settings, word_dict, label_dict, predicate_dict, **kwargs):
settings.word_dict = word_dict
settings.label_dict = label_dict
settings.predicate_dict = predicate_dict
#all inputs are integral and sequential type
settings.slots = [
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)), integer_value_sequence(2),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(predicate_dict)),
integer_value_sequence(2),
integer_value_sequence(len(label_dict))
]
@provider(init_hook=hook)
def process(obj, file_name):
def get_batch_size(yeild_data):
return len(yeild_data[0])
@provider(init_hook=hook, should_shuffle=True, calc_batch_size=get_batch_size,
can_over_batch_size=False, cache=CacheType.CACHE_PASS_IN_MEM)
def process(settings, file_name):
with open(file_name, 'r') as fdata:
for line in fdata:
sentence, predicate, ctx_n1, ctx_0, ctx_p1, mark, label = \
sentence, predicate, ctx_n2, ctx_n1, ctx_0, ctx_p1, ctx_p2, mark, label = \
line.strip().split('\t')
words = sentence.split()
sen_len = len(words)
word_slot = [obj.word_dict.get(w, UNK_IDX) for w in words]
word_slot = [settings.word_dict.get(w, UNK_IDX) for w in words]
predicate_slot = [obj.word_dict.get(predicate, UNK_IDX)] * sen_len
ctx_n1_slot = [obj.word_dict.get(ctx_n1, UNK_IDX)] * sen_len
ctx_0_slot = [obj.word_dict.get(ctx_0, UNK_IDX)] * sen_len
ctx_p1_slot = [obj.word_dict.get(ctx_p1, UNK_IDX)] * sen_len
predicate_slot = [settings.predicate_dict.get(predicate)] * sen_len
ctx_n2_slot = [settings.word_dict.get(ctx_n2, UNK_IDX)] * sen_len
ctx_n1_slot = [settings.word_dict.get(ctx_n1, UNK_IDX)] * sen_len
ctx_0_slot = [settings.word_dict.get(ctx_0, UNK_IDX)] * sen_len
ctx_p1_slot = [settings.word_dict.get(ctx_p1, UNK_IDX)] * sen_len
ctx_p2_slot = [settings.word_dict.get(ctx_p2, UNK_IDX)] * sen_len
marks = mark.split()
mark_slot = [int(w) for w in marks]
label_list = label.split()
label_slot = [obj.label_dict.get(w) for w in label_list]
yield word_slot, predicate_slot, ctx_n1_slot, \
ctx_0_slot, ctx_p1_slot, mark_slot, label_slot
label_slot = [settings.label_dict.get(w) for w in label_list]
yield word_slot, ctx_n2_slot, ctx_n1_slot, \
ctx_0_slot, ctx_p1_slot, ctx_p2_slot, predicate_slot, mark_slot, label_slot
demo/semantic_role_labeling/db_lstm.py
浏览文件 @ 96615fe3
......@@ -18,8 +18,9 @@ import sys
from paddle.trainer_config_helpers import *
#file paths
word_dict_file = './data/src.dict'
label_dict_file = './data/tgt.dict'
word_dict_file = './data/wordDict.txt'
label_dict_file = './data/targetDict.txt'
predicate_file= './data/verbDict.txt'
train_list_file = './data/train.list'
test_list_file = './data/test.list'
......@@ -30,8 +31,10 @@ if not is_predict:
#load dictionaries
word_dict = dict()
label_dict = dict()
predicate_dict = dict()
with open(word_dict_file, 'r') as f_word, \
open(label_dict_file, 'r') as f_label:
open(label_dict_file, 'r') as f_label, \
open(predicate_file, 'r') as f_pre:
for i, line in enumerate(f_word):
w = line.strip()
word_dict[w] = i
......@@ -40,6 +43,11 @@ if not is_predict:
w = line.strip()
label_dict[w] = i
for i, line in enumerate(f_pre):
w = line.strip()
predicate_dict[w] = i
if is_test:
train_list_file = None
......@@ -50,91 +58,157 @@ if not is_predict:
module='dataprovider',
obj='process',
args={'word_dict': word_dict,
'label_dict': label_dict})
'label_dict': label_dict,
'predicate_dict': predicate_dict })
word_dict_len = len(word_dict)
label_dict_len = len(label_dict)
pred_len = len(predicate_dict)
else:
word_dict_len = get_config_arg('dict_len', int)
label_dict_len = get_config_arg('label_len', int)
pred_len = get_config_arg('pred_len', int)
############################## Hyper-parameters ##################################
mark_dict_len = 2
word_dim = 32
mark_dim = 5
hidden_dim = 128
hidden_dim = 512
depth = 8
emb_lr = 1e-2
fc_lr = 1e-2
lstm_lr = 2e-2
########################### Optimizer #######################################
settings(
batch_size=150,
learning_method=AdamOptimizer(),
learning_rate=1e-3,
learning_method=MomentumOptimizer(momentum=0),
learning_rate=2e-2,
regularization=L2Regularization(8e-4),
gradient_clipping_threshold=25)
is_async=False,
model_average=ModelAverage(average_window=0.5,
max_average_window=10000),
)
#6 features
####################################### network ##############################
#8 features and 1 target
word = data_layer(name='word_data', size=word_dict_len)
predicate = data_layer(name='verb_data', size=word_dict_len)
predicate = data_layer(name='verb_data', size=pred_len)
ctx_n2 = data_layer(name='ctx_n2_data', size=word_dict_len)
ctx_n1 = data_layer(name='ctx_n1_data', size=word_dict_len)
ctx_0 = data_layer(name='ctx_0_data', size=word_dict_len)
ctx_p1 = data_layer(name='ctx_p1_data', size=word_dict_len)
ctx_p2 = data_layer(name='ctx_p2_data', size=word_dict_len)
mark = data_layer(name='mark_data', size=mark_dict_len)
if not is_predict:
target = data_layer(name='target', size=label_dict_len)
ptt = ParameterAttribute(name='src_emb', learning_rate=emb_lr)
layer_attr = ExtraLayerAttribute(drop_rate=0.5)
fc_para_attr = ParameterAttribute(learning_rate=fc_lr)
lstm_para_attr = ParameterAttribute(initial_std=0., learning_rate=lstm_lr)
para_attr = [fc_para_attr, lstm_para_attr]
word_embedding = embedding_layer(size=word_dim, input=word, param_attr=ptt)
predicate_embedding = embedding_layer(
size=word_dim, input=predicate, param_attr=ptt)
ctx_n1_embedding = embedding_layer(size=word_dim, input=ctx_n1, param_attr=ptt)
ctx_0_embedding = embedding_layer(size=word_dim, input=ctx_0, param_attr=ptt)
ctx_p1_embedding = embedding_layer(size=word_dim, input=ctx_p1, param_attr=ptt)
mark_embedding = embedding_layer(size=mark_dim, input=mark)
default_std=1/math.sqrt(hidden_dim)/3.0
emb_para = ParameterAttribute(name='emb', initial_std=0., learning_rate=0.)
std_0 = ParameterAttribute(initial_std=0.)
std_default = ParameterAttribute(initial_std=default_std)
predicate_embedding = embedding_layer(size=word_dim, input=predicate, param_attr=ParameterAttribute(name='vemb',initial_std=default_std))
mark_embedding = embedding_layer(name='word_ctx-in_embedding', size=mark_dim, input=mark, param_attr=std_0)
word_input=[word, ctx_n2, ctx_n1, ctx_0, ctx_p1, ctx_p2]
emb_layers = [embedding_layer(size=word_dim, input=x, param_attr=emb_para) for x in word_input]
emb_layers.append(predicate_embedding)
emb_layers.append(mark_embedding)
hidden_0 = mixed_layer(
name='hidden0',
size=hidden_dim,
input=[
full_matrix_projection(input=word_embedding),
full_matrix_projection(input=predicate_embedding),
full_matrix_projection(input=ctx_n1_embedding),
full_matrix_projection(input=ctx_0_embedding),
full_matrix_projection(input=ctx_p1_embedding),
full_matrix_projection(input=mark_embedding),
])
bias_attr=std_default,
input=[ full_matrix_projection(input=emb, param_attr=std_default ) for emb in emb_layers ])
lstm_0 = lstmemory(input=hidden_0, layer_attr=layer_attr)
mix_hidden_lr = 1e-3
lstm_para_attr = ParameterAttribute(initial_std=0.0, learning_rate=1.0)
hidden_para_attr = ParameterAttribute(initial_std=default_std, learning_rate=mix_hidden_lr)
lstm_0 = lstmemory(name='lstm0',
input=hidden_0,
act=ReluActivation(),
gate_act=SigmoidActivation(),
state_act=SigmoidActivation(),
bias_attr=std_0,
param_attr=lstm_para_attr)
#stack L-LSTM and R-LSTM with direct edges
input_tmp = [hidden_0, lstm_0]
for i in range(1, depth):
fc = fc_layer(input=input_tmp, size=hidden_dim, param_attr=para_attr)
mix_hidden = mixed_layer(name='hidden'+str(i),
size=hidden_dim,
bias_attr=std_default,
input=[full_matrix_projection(input=input_tmp[0], param_attr=hidden_para_attr),
full_matrix_projection(input=input_tmp[1], param_attr=lstm_para_attr)
]
)
lstm = lstmemory(name='lstm'+str(i),
input=mix_hidden,
act=ReluActivation(),
gate_act=SigmoidActivation(),
state_act=SigmoidActivation(),
reverse=((i % 2)==1),
bias_attr=std_0,
param_attr=lstm_para_attr)
input_tmp = [mix_hidden, lstm]
feature_out = mixed_layer(name='output',
size=label_dict_len,
bias_attr=std_default,
input=[full_matrix_projection(input=input_tmp[0], param_attr=hidden_para_attr),
full_matrix_projection(input=input_tmp[1], param_attr=lstm_para_attr)
],
)
lstm = lstmemory(
input=fc,
act=ReluActivation(),
reverse=(i % 2) == 1,
layer_attr=layer_attr)
input_tmp = [fc, lstm]
prob = fc_layer(
input=input_tmp,
size=label_dict_len,
act=SoftmaxActivation(),
param_attr=para_attr)
if not is_predict:
cls = classification_cost(input=prob, label=target)
outputs(cls)
crf_l = crf_layer( name = 'crf',
size = label_dict_len,
input = feature_out,
label = target,
param_attr=ParameterAttribute(name='crfw',initial_std=default_std, learning_rate=mix_hidden_lr)
)
crf_dec_l = crf_decoding_layer(name = 'crf_dec_l',
size = label_dict_len,
input = feature_out,
label = target,
param_attr=ParameterAttribute(name='crfw')
)
eval = sum_evaluator(input=crf_dec_l)
outputs(crf_l)
else:
outputs(prob)
crf_dec_l = crf_decoding_layer(name = 'crf_dec_l',
size = label_dict_len,
input = feature_out,
param_attr=ParameterAttribute(name='crfw')
)
outputs(crf_dec_l)
demo/semantic_role_labeling/predict.py
浏览文件 @ 96615fe3
......@@ -26,7 +26,7 @@ UNK_IDX = 0
class Prediction():
def __init__(self, train_conf, dict_file, model_dir, label_file):
def __init__(self, train_conf, dict_file, model_dir, label_file, predicate_dict_file):
"""
train_conf: trainer configure.
dict_file: word dictionary file name.
......@@ -35,26 +35,37 @@ class Prediction():
self.dict = {}
self.labels = {}
self.predicate_dict={}
self.labels_reverse = {}
self.load_dict_label(dict_file, label_file)
self.load_dict_label(dict_file, label_file, predicate_dict_file)
len_dict = len(self.dict)
len_label = len(self.labels)
conf = parse_config(train_conf, 'dict_len=' + str(len_dict) +
',label_len=' + str(len_label) + ',is_predict=True')
len_pred = len(self.predicate_dict)
conf = parse_config(
train_conf,
'dict_len=' + str(len_dict) +
',label_len=' + str(len_label) +
',pred_len=' + str(len_pred) +
',is_predict=True')
self.network = swig_paddle.GradientMachine.createFromConfigProto(
conf.model_config)
self.network.loadParameters(model_dir)
slots = [
integer_value_sequence(len_dict), integer_value_sequence(len_dict),
integer_value_sequence(len_dict), integer_value_sequence(len_dict),
integer_value_sequence(len_dict), integer_value_sequence(2)
]
integer_value_sequence(len_dict),
integer_value_sequence(len_dict),
integer_value_sequence(len_dict),
integer_value_sequence(len_dict),
integer_value_sequence(len_dict),
integer_value_sequence(len_dict),
integer_value_sequence(len_pred),
integer_value_sequence(2)
]
self.converter = DataProviderConverter(slots)
def load_dict_label(self, dict_file, label_file):
def load_dict_label(self, dict_file, label_file, predicate_dict_file):
"""
Load dictionary from self.dict_file.
"""
......@@ -65,39 +76,42 @@ class Prediction():
self.labels[line.strip()] = line_count
self.labels_reverse[line_count] = line.strip()
for line_count, line in enumerate(open(predicate_dict_file, 'r')):
self.predicate_dict[line.strip()] = line_count
def get_data(self, data_file):
"""
Get input data of paddle format.
"""
with open(data_file, 'r') as fdata:
for line in fdata:
sentence, predicate, ctx_n1, ctx_0, ctx_p1, mark, label = line.strip(
sentence, predicate, ctx_n2, ctx_n1, ctx_0, ctx_p1, ctx_p2, mark, label = line.strip(
).split('\t')
words = sentence.split()
sen_len = len(words)
word_slot = [self.dict.get(w, UNK_IDX) for w in words]
predicate_slot = [self.dict.get(predicate, UNK_IDX)] * sen_len
predicate_slot = [self.predicate_dict.get(predicate, UNK_IDX)] * sen_len
ctx_n2_slot = [self.dict.get(ctx_n2, UNK_IDX)] * sen_len
ctx_n1_slot = [self.dict.get(ctx_n1, UNK_IDX)] * sen_len
ctx_0_slot = [self.dict.get(ctx_0, UNK_IDX)] * sen_len
ctx_p1_slot = [self.dict.get(ctx_p1, UNK_IDX)] * sen_len
ctx_p2_slot = [self.dict.get(ctx_p2, UNK_IDX)] * sen_len
marks = mark.split()
mark_slot = [int(w) for w in marks]
yield word_slot, ctx_n2_slot, ctx_n1_slot, \
ctx_0_slot, ctx_p1_slot, ctx_p2_slot, predicate_slot, mark_slot
yield word_slot, predicate_slot, ctx_n1_slot, \
ctx_0_slot, ctx_p1_slot, mark_slot
def predict(self, data_file):
def predict(self, data_file, output_file):
"""
data_file: file name of input data.
"""
input = self.converter(self.get_data(data_file))
output = self.network.forwardTest(input)
prob = output[0]["value"]
lab = list(np.argsort(-prob)[:, 0])
lab = output[0]["id"].tolist()
with open(data_file, 'r') as fin, open('predict.res', 'w') as fout:
with open(data_file, 'r') as fin, open(output_file, 'w') as fout:
index = 0
for line in fin:
sen = line.split('\t')[0]
......@@ -109,8 +123,8 @@ class Prediction():
def option_parser():
usage = ("python predict.py -c config -w model_dir "
"-d word dictionary -l label_file -i input_file")
usage = ("python predict.py -c config -w model_dir "
"-d word dictionary -l label_file -i input_file -p pred_dict_file")
parser = OptionParser(usage="usage: %s [options]" % usage)
parser.add_option(
"-c",
......@@ -131,6 +145,13 @@ def option_parser():
dest="label_file",
default=None,
help="label file")
parser.add_option(
"-p",
"--predict_dict_file",
action="store",
dest="predict_dict_file",
default=None,
help="predict_dict_file")
parser.add_option(
"-i",
"--data",
......@@ -144,6 +165,14 @@ def option_parser():
dest="model_path",
default=None,
help="model path")
parser.add_option(
"-o",
"--output_file",
action="store",
dest="output_file",
default=None,
help="output file")
return parser.parse_args()
......@@ -154,10 +183,12 @@ def main():
dict_file = options.dict_file
model_path = options.model_path
label_file = options.label_file
predict_dict_file = options.predict_dict_file
output_file = options.output_file
swig_paddle.initPaddle("--use_gpu=0")
predict = Prediction(train_conf, dict_file, model_path, label_file)
predict.predict(data_file)
predict = Prediction(train_conf, dict_file, model_path, label_file, predict_dict_file)
predict.predict(data_file,output_file)
if __name__ == '__main__':
......
demo/semantic_role_labeling/predict.sh
浏览文件 @ 96615fe3
......@@ -18,7 +18,7 @@ set -e
function get_best_pass() {
cat $1 | grep -Pzo 'Test .*\n.*pass-.*' | \
sed -r 'N;s/Test.* cost=([0-9]+\.[0-9]+).*\n.*pass-([0-9]+)/\1 \2/g' | \
sort | head -n 1
sort -n | head -n 1
}
log=train.log
......@@ -26,15 +26,18 @@ LOG=`get_best_pass $log`
LOG=(${LOG})
best_model_path="output/pass-${LOG[1]}"
config_file=db_lstm.py
dict_file=./data/src.dict
label_file=./data/tgt.dict
dict_file=./data/wordDict.txt
label_file=./data/targetDict.txt
predicate_dict_file=./data/verbDict.txt
input_file=./data/feature
output_file=predict.res
python predict.py \
-c $config_file \
-w $best_model_path \
-l $label_file \
-p $predicate_dict_file \
-d $dict_file \
-i $input_file
-i $input_file \
-o $output_file
demo/semantic_role_labeling/test.sh
浏览文件 @ 96615fe3
......@@ -18,7 +18,7 @@ set -e
function get_best_pass() {
cat $1 | grep -Pzo 'Test .*\n.*pass-.*' | \
sed -r 'N;s/Test.* cost=([0-9]+\.[0-9]+).*\n.*pass-([0-9]+)/\1 \2/g' |\
sort | head -n 1
sort -n | head -n 1
}
log=train.log
......@@ -36,4 +36,5 @@ paddle train \
--job=test \
--use_gpu=false \
--config_args=is_test=1 \
--test_all_data_in_one_period=1 \
2>&1 | tee 'test.log'
demo/semantic_role_labeling/train.sh
浏览文件 @ 96615fe3
......@@ -16,11 +16,14 @@
set -e
paddle train \
--config=./db_lstm.py \
--use_gpu=0 \
--log_period=5000 \
--trainer_count=1 \
--show_parameter_stats_period=5000 \
--save_dir=./output \
--trainer_count=4 \
--log_period=10 \
--num_passes=500 \
--use_gpu=false \
--show_parameter_stats_period=10 \
--num_passes=10000 \
--average_test_period=10000000 \
--init_model_path=./data \
--load_missing_parameter_strategy=rand \
--test_all_data_in_one_period=1 \
2>&1 | tee 'train.log'
2>&1 | tee 'train.log'
demo/sentiment/test.sh
浏览文件 @ 96615fe3
......@@ -17,7 +17,7 @@ set -e
function get_best_pass() {
cat $1 | grep -Pzo 'Test .*\n.*pass-.*' | \
sed -r 'N;s/Test.* classification_error_evaluator=([0-9]+\.[0-9]+).*\n.*pass-([0-9]+)/\1 \2/g' |\
sort | head -n 1
sort -n | head -n 1
}
log=train.log
......
demo/sentiment/trainer_config.py
浏览文件 @ 96615fe3
......@@ -29,6 +29,7 @@ settings(
batch_size=128,
learning_rate=2e-3,
learning_method=AdamOptimizer(),
average_window=0.5,
regularization=L2Regularization(8e-4),
gradient_clipping_threshold=25)
......
demo/seqToseq/data/paraphrase_data.sh
浏览文件 @ 96615fe3
......@@ -16,9 +16,7 @@ set -e
set -x
# download the in-house paraphrase dataset
# following is the google drive address
# you can also directly download from https://pan.baidu.com/s/1o8q577s
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/embedding/paraphrase.tar.gz --no-check-certificate
wget http://paddlepaddle.bj.bcebos.com/model_zoo/embedding/paraphrase.tar.gz
# untar the dataset
tar -zxvf paraphrase.tar.gz
......
demo/seqToseq/data/wmt14_model.sh
浏览文件 @ 96615fe3
......@@ -16,9 +16,7 @@ set -e
set -x
# download the pretrained model
# following is the google drive address
# you can also directly download from https://pan.baidu.com/s/1o8q577s
wget https://www.googledrive.com/host/0B7Q8d52jqeI9ejh6Q1RpMTFQT1k/wmt14_model.tar.gz --no-check-certificate
wget http://paddlepaddle.bj.bcebos.com/model_zoo/wmt14_model.tar.gz
# untar the model
tar -zxvf wmt14_model.tar.gz
......
doc/algorithm/rnn/rnn.rst
浏览文件 @ 96615fe3
......@@ -17,7 +17,7 @@ PaddlePaddle does not need any preprocessing to sequence data, such as padding.
.. code-block:: python
settings.slots = [
settings.input_types = [
integer_value_sequence(len(settings.src_dict)),
integer_value_sequence(len(settings.trg_dict)),
integer_value_sequence(len(settings.trg_dict))]
......
doc/build/build_from_source.md
浏览文件 @ 96615fe3
......@@ -6,10 +6,10 @@ Installing from Sources
* [3. Build on Ubuntu](#ubuntu)
## <span id="download">Download and Setup</span>
You can download PaddlePaddle from the [github source](https://github.com/gangliao/Paddle).
You can download PaddlePaddle from the [github source](https://github.com/PaddlePaddle/Paddle).
```bash
git clone https://github.com/baidu/Paddle paddle
git clone https://github.com/PaddlePaddle/Paddle paddle
cd paddle
```
......@@ -95,7 +95,7 @@ As a simple example, consider the following:
```bash
# necessary
sudo apt-get update
sudo apt-get install -y g++ make cmake build-essential libatlas-base-dev python python-pip libpython-dev m4 libprotobuf-dev protobuf-compiler python-protobuf python-numpy git
sudo apt-get install -y g++ make cmake swig build-essential libatlas-base-dev python python-pip libpython-dev m4 libprotobuf-dev protobuf-compiler python-protobuf python-numpy git
# optional
sudo apt-get install libgoogle-glog-dev
sudo apt-get install libgflags-dev
......@@ -149,15 +149,15 @@ If still not found, you can manually set it based on CMake error information fro
As a simple example, consider the following:
- **Only CPU**
- **Only CPU with swig**
```bash
cmake .. -DWITH_GPU=OFF
cmake .. -DWITH_GPU=OFF -DWITH_SWIG_PY=ON
```
- **GPU**
- **GPU with swig**
```bash
cmake .. -DWITH_GPU=ON
cmake .. -DWITH_GPU=ON -DWITH_SWIG_PY=ON
```
- **GPU with doc and swig**
......@@ -170,15 +170,13 @@ Finally, you can build PaddlePaddle:
```bash
# you can add build option here, such as:
cmake .. -DWITH_GPU=ON -DCMAKE_INSTALL_PREFIX=<path to install>
cmake .. -DWITH_GPU=ON -DCMAKE_INSTALL_PREFIX=<path to install> -DWITH_SWIG_PY=ON
# please use sudo make install, if you want to install PaddlePaddle into the system
make -j `nproc` && make install
# set PaddlePaddle installation path in ~/.bashrc
export PATH=<path to install>/bin:$PATH
```
**Note:**
If you set `WITH_SWIG_PY=ON`, related python dependencies also need to be installed.
Otherwise, PaddlePaddle will automatically install python dependencies
at first time when user run paddle commands, such as `paddle version`, `paddle train`.
......
doc/demo/quick_start/index_en.md
浏览文件 @ 96615fe3
......@@ -477,7 +477,7 @@ The scripts of data downloading, network configurations, and training scrips are
<td class="left">Word embedding</td>
<td class="left"> 15MB </td>
<td class="left"> 8.484%</td>
<td class="left">trainer_config.bow.py</td>
<td class="left">trainer_config.emb.py</td>
</tr>
<tr>
......
doc/demo/semantic_role_labeling/semantic_role_labeling.md
浏览文件 @ 96615fe3
# Semantic Role labeling Tutorial #
Semantic role labeling (SRL) is a form of shallow semantic parsing whose goal is to discover the predicate-argument structure of each predicate in a given input sentence. SRL is useful as an intermediate step in a wide range of natural language processing tasks, such as information extraction. automatic document categorization and question answering. An instance is as following [1]:
[ <sub>A0</sub> He ] [ <sub>AM-MOD</sub> would ][ <sub>AM-NEG</sub> n’t ] [ <sub>V</sub> accept] [ <sub>A1</sub> anything of value ] from [<sub>A2</sub> those he was writing about ].
- V: verb
- A0: acceptor
- A1: thing accepted
- A2: accepted-from
- A3: Attribute
- AM-MOD: modal
- AM-NEG: negation
Given the verb "accept", the chunks in sentence would play certain semantic roles. Here, the label scheme is from Penn Proposition Bank.
To this date, most of the successful SRL systems are built on top of some form of parsing results where pre-defined feature templates over the syntactic structure are used. This tutorial will present an end-to-end system using deep bidirectional long short-term memory (DB-LSTM)[2] for solving the SRL task, which largely outperforms the previous state-of-the-art systems. The system regards SRL task as the sequence labelling problem.
## Data Description
The relevant paper[2] takes the data set in CoNLL-2005&2012 Shared Task for training and testing. Accordingto data license, the demo adopts the test data set of CoNLL-2005, which can be reached on website.
To download and process the original data, user just need to execute the following command:
```bash
cd data
./get_data.sh
```
Several new files appear in the `data `directory as follows.
```bash
conll05st-release：the test data set of CoNll-2005 shared task
test.wsj.words：the Wall Street Journal data sentences
test.wsj.props: the propositional arguments
src.dict：the dictionary of words in sentences
tgt.dict：the labels dictionary
feature: the extracted features from data set
```
## Training
### DB-LSTM
Please refer to the Sentiment Analysis demo to learn more about the long short-term memory unit.
Unlike Bidirectional-LSTM that used in Sentiment Analysis demo, the DB-LSTM adopts another way to stack LSTM layer. First a standard LSTM processes the sequence in forward direction. The input and output of this LSTM layer are taken by the next LSTM layer as input, processed in reversed direction. These two standard LSTM layers compose a pair of LSTM. Then we stack LSTM layers pair after pair to obtain the deep LSTM model.
The following figure shows a temporal expanded 2-layer DB-LSTM network.
<center>
![pic](./network_arch.png)
</center>
### Features
Two input features play an essential role in this pipeline: predicate (pred) and argument (argu). Two other features: predicate context (ctx-p) and region mark (mr) are also adopted. Because a single predicate word can not exactly describe the predicate information, especially when the same words appear more than one times in a sentence. With the predicate context, the ambiguity can be largely eliminated. Similarly, we use region mark m<sub>r</sub> = 1 to denote the argument position if it locates in the predicate context region, or m<sub>r</sub> = 0 if does not. These four simple features are all we need for our SRL system. Features of one sample with context size set to 1 is showed as following[2]:
<center>
![pic](./feature.jpg)
</center>
In this sample, the coresponding labelled sentence is:
[ <sub>A1</sub> A record date ] has [ <sub>AM-NEG</sub> n't ] been [ <sub>V</sub> set ] .
In the demo, we adopt the feature template as above, consists of : `argument`, `predicate`, `ctx-p (p=-1,0,1)`, `mark` and use `B/I/O` scheme to label each argument. These features and labels are stored in `feature` file, and separated by `\t`.
### Data Provider
`dataprovider.py` is the python file to wrap data. `hook()` function is to define the data slots for network. The Six features and label are all IndexSlots.
```
def hook(settings, word_dict, label_dict, **kwargs):
settings.word_dict = word_dict
settings.label_dict = label_dict
#all inputs are integral and sequential type
settings.slots = [
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(2),
integer_value_sequence(len(label_dict))]
```
The corresponding data iterator is as following:
```
@provider(use_seq=True, init_hook=hook)
def process(obj, file_name):
with open(file_name, 'r') as fdata:
for line in fdata:
sentence, predicate, ctx_n1, ctx_0, ctx_p1, mark, label = line.strip().split('\t')
words = sentence.split()
sen_len = len(words)
word_slot = [obj.word_dict.get(w, UNK_IDX) for w in words]
predicate_slot = [obj.word_dict.get(predicate, UNK_IDX)] * sen_len
ctx_n1_slot = [obj.word_dict.get(ctx_n1, UNK_IDX) ] * sen_len
ctx_0_slot = [obj.word_dict.get(ctx_0, UNK_IDX) ] * sen_len
ctx_p1_slot = [obj.word_dict.get(ctx_p1, UNK_IDX) ] * sen_len
marks = mark.split()
mark_slot = [int(w) for w in marks]
label_list = label.split()
label_slot = [obj.label_dict.get(w) for w in label_list]
yield word_slot, predicate_slot, ctx_n1_slot, ctx_0_slot, ctx_p1_slot, mark_slot, label_slot
```
The `process`function yield 7 lists which are six features and labels.
### Neural Network Config
`db_lstm.py` is the neural network config file to load the dictionaries and define the data provider module and network architecture during the training procedure.
Seven `data_layer` load instances from data provider. Six features are transformed into embedddings respectively, and mixed by `mixed_layer` . Deep bidirectional LSTM layers extract features for the softmax layer. The objective function is cross entropy of labels.
### Run Training
The script for training is `train.sh`, user just need to execute:
```bash
./train.sh
```
The content in `train.sh`:
```
paddle train \
--config=./db_lstm.py \
--save_dir=./output \
--trainer_count=4 \
--log_period=10 \
--num_passes=500 \
--use_gpu=false \
--show_parameter_stats_period=10 \
--test_all_data_in_one_period=1 \
2>&1 | tee 'train.log'
```
- \--config=./db_lstm.py : network config file.
- \--save_di=./output: output path to save models.
- \--trainer_count=4 : set thread number (or GPU count).
- \--log_period=10 : print log every 20 batches.
- \--num_passes=500: set pass number, one pass in PaddlePaddle means training all samples in dataset one time.
- \--use_gpu=false: use CPU to train, set true, if you install GPU version of PaddlePaddle and want to use GPU to train.
- \--show_parameter_stats_period=10: show parameter statistic every 100 batches.
- \--test_all_data_in_one_period=1: test all data in every testing.
After training, the models will be saved in directory `output`.
### Run testing
The script for testing is `test.sh`, user just need to execute:
```bash
./test.sh
```
The main part in `tesh.sh`
```
paddle train \
--config=./db_lstm.py \
--model_list=$model_list \
--job=test \
--config_args=is_test=1 \
```
- \--config=./db_lstm.py: network config file
- \--model_list=$model_list.list: model list file
- \--job=test: indicate the test job
- \--config_args=is_test=1: flag to indicate test
### Run prediction
The script for prediction is `predict.sh`, user just need to execute:
```bash
./predict.sh
```
In `predict.sh`, user should offer the network config file, model path, label file, word dictionary file, feature file
```
python predict.py
-c $config_file
-w $model_path
-l $label_file
-d $dict_file
-i $input_file
```
`predict.py` is the main executable python script, which includes functions: load model, load data, data prediction. The network model will output the probability distribution of labels. In the demo, we take the label with maximum probability as result. User can also implement the beam search or viterbi decoding upon the probability distribution matrix.
After prediction, the result is saved in `predict.res`.
## Reference
[1] Martha Palmer, Dan Gildea, and Paul Kingsbury. The Proposition Bank: An Annotated Corpus of Semantic Roles , Computational Linguistics, 31(1), 2005.
[2] Zhou, Jie, and Wei Xu. "End-to-end learning of semantic role labeling using recurrent neural networks." Proceedings of the Annual Meeting of the Association for Computational Linguistics. 2015.
# Semantic Role labeling Tutorial #
Semantic role labeling (SRL) is a form of shallow semantic parsing whose goal is to discover the predicate-argument structure of each predicate in a given input sentence. SRL is useful as an intermediate step in a wide range of natural language processing tasks, such as information extraction. automatic document categorization and question answering. An instance is as following [1]:
[ <sub>A0</sub> He ] [ <sub>AM-MOD</sub> would ][ <sub>AM-NEG</sub> n’t ] [ <sub>V</sub> accept] [ <sub>A1</sub> anything of value ] from [<sub>A2</sub> those he was writing about ].
- V: verb
- A0: acceptor
- A1: thing accepted
- A2: accepted-from
- A3: Attribute
- AM-MOD: modal
- AM-NEG: negation
Given the verb "accept", the chunks in sentence would play certain semantic roles. Here, the label scheme is from Penn Proposition Bank.
To this date, most of the successful SRL systems are built on top of some form of parsing results where pre-defined feature templates over the syntactic structure are used. This tutorial will present an end-to-end system using deep bidirectional long short-term memory (DB-LSTM)[2] for solving the SRL task, which largely outperforms the previous state-of-the-art systems. The system regards SRL task as the sequence labelling problem.
## Data Description
The relevant paper[2] takes the data set in CoNLL-2005&2012 Shared Task for training and testing. Accordingto data license, the demo adopts the test data set of CoNLL-2005, which can be reached on website.
To download and process the original data, user just need to execute the following command:
```bash
cd data
./get_data.sh
```
Several new files appear in the `data `directory as follows.
```bash
conll05st-release：the test data set of CoNll-2005 shared task
test.wsj.words：the Wall Street Journal data sentences
test.wsj.props: the propositional arguments
feature: the extracted features from data set
```
## Training
### DB-LSTM
Please refer to the Sentiment Analysis demo to learn more about the long short-term memory unit.
Unlike Bidirectional-LSTM that used in Sentiment Analysis demo, the DB-LSTM adopts another way to stack LSTM layer. First a standard LSTM processes the sequence in forward direction. The input and output of this LSTM layer are taken by the next LSTM layer as input, processed in reversed direction. These two standard LSTM layers compose a pair of LSTM. Then we stack LSTM layers pair after pair to obtain the deep LSTM model.
The following figure shows a temporal expanded 2-layer DB-LSTM network.
<center>
![pic](./network_arch.png)
</center>
### Features
Two input features play an essential role in this pipeline: predicate (pred) and argument (argu). Two other features: predicate context (ctx-p) and region mark (mr) are also adopted. Because a single predicate word can not exactly describe the predicate information, especially when the same words appear more than one times in a sentence. With the predicate context, the ambiguity can be largely eliminated. Similarly, we use region mark m<sub>r</sub> = 1 to denote the argument position if it locates in the predicate context region, or m<sub>r</sub> = 0 if does not. These four simple features are all we need for our SRL system. Features of one sample with context size set to 1 is showed as following[2]:
<center>
![pic](./feature.jpg)
</center>
In this sample, the coresponding labelled sentence is:
[ <sub>A1</sub> A record date ] has [ <sub>AM-NEG</sub> n't ] been [ <sub>V</sub> set ] .
In the demo, we adopt the feature template as above, consists of : `argument`, `predicate`, `ctx-p (p=-1,0,1)`, `mark` and use `B/I/O` scheme to label each argument. These features and labels are stored in `feature` file, and separated by `\t`.
### Data Provider
`dataprovider.py` is the python file to wrap data. `hook()` function is to define the data slots for network. The Six features and label are all IndexSlots.
```
def hook(settings, word_dict, label_dict, **kwargs):
settings.word_dict = word_dict
settings.label_dict = label_dict
#all inputs are integral and sequential type
settings.slots = [
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(predicate_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(len(word_dict)),
integer_value_sequence(2),
integer_value_sequence(len(label_dict))]
```
The corresponding data iterator is as following:
```
@provider(init_hook=hook, should_shuffle=True, calc_batch_size=get_batch_size,
can_over_batch_size=False, cache=CacheType.CACHE_PASS_IN_MEM)
def process(settings, file_name):
with open(file_name, 'r') as fdata:
for line in fdata:
sentence, predicate, ctx_n2, ctx_n1, ctx_0, ctx_p1, ctx_p2, mark, label = \
line.strip().split('\t')
words = sentence.split()
sen_len = len(words)
word_slot = [settings.word_dict.get(w, UNK_IDX) for w in words]
predicate_slot = [settings.predicate_dict.get(predicate)] * sen_len
ctx_n2_slot = [settings.word_dict.get(ctx_n2, UNK_IDX)] * sen_len
ctx_n1_slot = [settings.word_dict.get(ctx_n1, UNK_IDX)] * sen_len
ctx_0_slot = [settings.word_dict.get(ctx_0, UNK_IDX)] * sen_len
ctx_p1_slot = [settings.word_dict.get(ctx_p1, UNK_IDX)] * sen_len
ctx_p2_slot = [settings.word_dict.get(ctx_p2, UNK_IDX)] * sen_len
marks = mark.split()
mark_slot = [int(w) for w in marks]
label_list = label.split()
label_slot = [settings.label_dict.get(w) for w in label_list]
yield word_slot, predicate_slot, ctx_n2_slot, ctx_n1_slot, \
ctx_0_slot, ctx_p1_slot, ctx_p2_slot, mark_slot, label_slot
```
The `process`function yield 9 lists which are 8 features and label.
### Neural Network Config
`db_lstm.py` is the neural network config file to load the dictionaries and define the data provider module and network architecture during the training procedure.
Nine `data_layer` load instances from data provider. Eight features are transformed into embedddings respectively, and mixed by `mixed_layer` . Deep bidirectional LSTM layers extract features for the softmax layer. The objective function is cross entropy of labels.
### Run Training
The script for training is `train.sh`, user just need to execute:
```bash
./train.sh
```
The content in `train.sh`:
```
paddle train \
--config=./db_lstm.py \
--use_gpu=0 \
--log_period=5000 \
--trainer_count=1 \
--show_parameter_stats_period=5000 \
--save_dir=./output \
--num_passes=10000 \
--average_test_period=10000000 \
--init_model_path=./data \
--load_missing_parameter_strategy=rand \
--test_all_data_in_one_period=1 \
2>&1 | tee 'train.log'
```
- \--config=./db_lstm.py : network config file.
- \--use_gpu=false: use CPU to train, set true, if you install GPU version of PaddlePaddle and want to use GPU to train, until now crf_layer do not support GPU
- \--log_period=500: print log every 20 batches.
- \--trainer_count=1: set thread number (or GPU count).
- \--show_parameter_stats_period=5000: show parameter statistic every 100 batches.
- \--save_dir=./output: output path to save models.
- \--num_passes=10000: set pass number, one pass in PaddlePaddle means training all samples in dataset one time.
- \--average_test_period=10000000: do test on average parameter every average_test_period batches
- \--init_model_path=./data: parameter initialization path
- \--load_missing_parameter_strategy=rand: random initialization unexisted parameters
- \--test_all_data_in_one_period=1: test all data in one period
After training, the models will be saved in directory `output`. Our training curve is as following:
<center>
![pic](./curve.jpg)
</center>
### Run testing
The script for testing is `test.sh`, user just need to execute:
```bash
./test.sh
```
The main part in `tesh.sh`
```
paddle train \
--config=./db_lstm.py \
--model_list=$model_list \
--job=test \
--config_args=is_test=1 \
```
- \--config=./db_lstm.py: network config file
- \--model_list=$model_list.list: model list file
- \--job=test: indicate the test job
- \--config_args=is_test=1: flag to indicate test
- \--test_all_data_in_one_period=1: test all data in 1 period
### Run prediction
The script for prediction is `predict.sh`, user just need to execute:
```bash
./predict.sh
```
In `predict.sh`, user should offer the network config file, model path, label file, word dictionary file, feature file
```
python predict.py
-c $config_file \
-w $best_model_path \
-l $label_file \
-p $predicate_dict_file \
-d $dict_file \
-i $input_file \
-o $output_file
```
`predict.py` is the main executable python script, which includes functions: load model, load data, data prediction. The network model will output the probability distribution of labels. In the demo, we take the label with maximum probability as result. User can also implement the beam search or viterbi decoding upon the probability distribution matrix.
After prediction, the result is saved in `predict.res`.
## Reference
[1] Martha Palmer, Dan Gildea, and Paul Kingsbury. The Proposition Bank: An Annotated Corpus of Semantic Roles , Computational Linguistics, 31(1), 2005.
[2] Zhou, Jie, and Wei Xu. "End-to-end learning of semantic role labeling using recurrent neural networks." Proceedings of the Annual Meeting of the Association for Computational Linguistics. 2015.
doc/demo/sentiment_analysis/sentiment_analysis.md
浏览文件 @ 96615fe3
......@@ -6,7 +6,7 @@ Sentiment analysis is also used to monitor social media based on large amount of
On the other hand, grabbing the user comments of products and analyzing their sentiment are useful to understand user preferences for companies, products, even competing products.
This tutorial will guide you through the process of training a Long Short Term Memory (LSTM) Network to classify the sentiment of sentences from [Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/), sometimes known as the [Internet Movie Database (IMDB)](http://ai.stanford.edu/~amaas/papers/wvSent_acl2011.pdf). This dataset contains movie reviews along with their associated binary sentiment polarity labels, namely positive and negative. So randomly guessing yields 50% accuracy.
This tutorial will guide you through the process of training a Long Short Term Memory (LSTM) Network to classify the sentiment of sentences from [Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/), sometimes known as the Internet Movie Database (IMDB). This dataset contains movie reviews along with their associated binary sentiment polarity labels, namely positive and negative. So randomly guessing yields 50% accuracy.
## Data Preparation
......@@ -39,7 +39,7 @@ imdbEr.txt imdb.vocab README test train
* imdbEr.txt: expected rating for each token in imdb.vocab.
* README: data documentation.
Both train and test set directory contains:
The file in train set directory is as follows. The test set also contains them except `unsup` and `urls_unsup.txt`.
```
labeledBow.feat neg pos unsup unsupBow.feat urls_neg.txt urls_pos.txt urls_unsup.txt
......@@ -151,6 +151,7 @@ settings(
batch_size=128,
learning_rate=2e-3,
learning_method=AdamOptimizer(),
average_window=0.5,
regularization=L2Regularization(8e-4),
gradient_clipping_threshold=25
)
......@@ -163,17 +164,18 @@ stacked_lstm_net(dict_dim, class_dim=class_dim,
* **Data Definition**:
* get\_config\_arg(): get arguments setted by `--config_args=xx` in commandline argument.
* Define TrainData and TestData provider, here using Python interface (PyDataProviderWrapper) of PaddlePaddle to load data. For details, you can refer to the document of PyDataProvider.
* Define data provider, here using Python interface to load data. For details, you can refer to the document of PyDataProvider2.
* **Algorithm Configuration**:
* use sgd algorithm.
* use adam optimization.
* set batch size of 128.
* set average sgd window.
* set global learning rate.
* use adam optimization.
* set average sgd window.
* set L2 regularization.
* set gradient clipping threshold.
* **Network Configuration**:
* dict_dim: get dictionary dimension.
* class_dim: set category number, IMDB has two label, namely positive and negative label.
* dict_dim: dictionary dimension.
* class_dim: category number, IMDB has two label, namely positive and negative label.
* `stacked_lstm_net`: predefined network as shown in Figure 3, use this network by default.
* `bidirectional_lstm_net`: predefined network as shown in Figure 2.
......
doc/dev/new_layer/new_layer.rst
浏览文件 @ 96615fe3
......@@ -60,7 +60,7 @@ Implement C++ Class
The C++ class of the layer implements the initialization, forward, and backward part of the layer. The fully connected layer is at :code:`paddle/gserver/layers/FullyConnectedLayer.h` and :code:`paddle/gserver/layers/FullyConnectedLayer.cpp`. We list simplified version of the code below.
It needs to derive the base class :code:`paddle::BaseLayer`, and it needs to override the following functions:
It needs to derive the base class :code:`paddle::Layer`, and it needs to override the following functions:
- constructor and destructor.
- :code:`init` function. It is used to initialize the parameters and settings.
......
doc/index.rst
浏览文件 @ 96615fe3
......@@ -8,3 +8,4 @@ PaddlePaddle Documentation
user_guide.rst
dev/index.rst
algorithm/index.rst
optimization/index.rst
doc/optimization/gpu_profiling.rst 0 → 100644
浏览文件 @ 96615fe3
Profiling on PaddlePaddle
=========================
This tutorial will guide you step-by-step through how to conduct profiling and performance tuning using built-in timer, **nvprof** and **nvvp**.
- What is profiling?
- Why we need profiling?
- How to do profiling?
- Profile tools
- Hands-on Tutorial
- Profiling tips
What's profiling?
=================
In software engineering, profiling is a form of dynamic program analysis that measures the space (memory) or time
complexity of a program, the usage of particular instructions, or the frequency and duration of function calls.
Most commonly, profiling information serves to aid program optimization.
Briefly, profiler is used to measure application performance. Program analysis tools are extremely important for
understanding program behavior. Simple profiling can tell you that how long does an operation take? For advanced
profiling, it can interpret why does an operation take a long time?
Why we need profiling?
======================
Since training deep neural network typically take a very long time to get over, performance is gradually becoming
the most important thing in deep learning field. The first step to improve performance is to understand what parts
are slow. There is no point in improving performance of a region which doesn’t take much time!
How to do profiling?
====================
To achieve maximum performance, there are five steps you can take to reach your goals.
- Profile the code
- Find the slow parts
- Work out why they’re slow
- Make them fast
- Profile the code again
Usually, processor has two key performance limits include float point throughput and
memory throughput. For GPU, it also need more parallelism to fulfill its potential.
This is why they can be so fast.
Profiler Tools
==============
For general GPU profiling, a bunch of tools are provided from both NVIDIA and third party.
**nvprof** is Nvidia profiler and **nvvp** is (GUI based) Nvidia visual profiler.
In this tutorial, we will focus on nvprof and nvvp.
:code:`test_GpuProfiler` from :code:`paddle/math/tests` directory will be used to evaluate
above profilers.
.. literalinclude:: ../../paddle/math/tests/test_GpuProfiler.cpp
:language: c++
:lines: 111-124
:linenos:
The above code snippet includes two methods, you can use any of them to profile the regions of interest.
1. :code:`REGISTER_TIMER_INFO` is a built-in timer wrapper which can calculate the time overhead of both cpu functions and cuda kernels.
2. :code:`REGISTER_GPU_PROFILER` is a general purpose wrapper object of :code:`cudaProfilerStart` and :code:`cudaProfilerStop` to avoid
program crashes when CPU version of PaddlePaddle invokes them.
You can find more details about how to use both of them in the next session.
Hands-on Approach
=================
Built-in Timer
--------------
To enable built-in timer in PaddlePaddle, first you have to add :code:`REGISTER_TIMER_INFO` into the regions of you interest.
Then, all information could be stamped in the console via :code:`printStatus` or :code:`printAllStatus` function.
As a simple example, consider the following:
1. Add :code:`REGISTER_TIMER_INFO` and :code:`printAllStatus` functions (see the emphasize-lines).
.. literalinclude:: ../../paddle/math/tests/test_GpuProfiler.cpp
:language: c++
:lines: 111-124
:emphasize-lines: 8-10,13
:linenos:
2. Configure cmake with **WITH_TIMER** and recompile PaddlePaddle.
.. code-block:: bash
cmake .. -DWITH_TIMER=ON
make
3. Execute your code and observe the results (see the emphasize-lines).
.. code-block:: bash
:emphasize-lines: 1,12-15
> ./paddle/math/tests/test_GpuProfiler
I1117 11:13:42.313065 2522362816 Util.cpp:155] commandline: ./paddle/math/tests/test_GpuProfiler
I1117 11:13:42.845065 2522362816 Util.cpp:130] Calling runInitFunctions
I1117 11:13:42.845208 2522362816 Util.cpp:143] Call runInitFunctions done.
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Profiler
[ RUN ] Profiler.BilinearFwdBwd
I1117 11:13:42.845310 2522362816 test_GpuProfiler.cpp:114] Enable GPU Profiler Stat: [testBilinearFwdBwd] "numSamples = 10, channels = 16, im
gSizeX = 64, imgSizeY = 64"
I1117 11:13:42.850154 2522362816 ThreadLocal.cpp:37] thread use undeterministic rand seed:20659751
I1117 11:13:42.981501 2522362816 Stat.cpp:130] ======= StatSet: [GlobalStatInfo] status ======
I1117 11:13:42.981539 2522362816 Stat.cpp:133] Stat=testBilinearFwdBwd total=136.141 avg=136.141 max=136.141 min=136.141 count=1
I1117 11:13:42.981572 2522362816 Stat.cpp:141] ======= BarrierStatSet status ======
I1117 11:13:42.981575 2522362816 Stat.cpp:154] --------------------------------------------------
[ OK ] Profiler.BilinearFwdBwd (136 ms)
[----------] 1 test from Profiler (136 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (136 ms total)
[ PASSED ] 1 test.
nvprof profiler
---------------
To use this command line profiler **nvprof**, you can simply issue the following command:
1. Add :code:`REGISTER_GPU_PROFILER` function (see the emphasize-lines).
.. literalinclude:: ../../paddle/math/tests/test_GpuProfiler.cpp
:language: c++
:lines: 111-124
:emphasize-lines: 6-7
:linenos:
2. Configure cmake with **WITH_PROFILER** and recompile PaddlePaddle.
.. code-block:: bash
cmake .. -DWITH_PROFILER=ON
make
3. Use Nvidia profiler **nvprof** to profile the binary.
.. code-block:: bash
nvprof ./paddle/math/tests/test_GpuProfiler
Then, you can get the following profiling result:
.. code-block:: bash
==78544== Profiling application: ./paddle/math/tests/test_GpuProfiler
==78544== Profiling result:
Time(%) Time Calls Avg Min Max Name
27.60% 9.6305ms 5 1.9261ms 3.4560us 6.4035ms [CUDA memcpy HtoD]
26.07% 9.0957ms 1 9.0957ms 9.0957ms 9.0957ms KeBilinearInterpBw
23.78% 8.2977ms 1 8.2977ms 8.2977ms 8.2977ms KeBilinearInterpFw
22.55% 7.8661ms 2 3.9330ms 1.5798ms 6.2863ms [CUDA memcpy DtoH]
==78544== API calls:
Time(%) Time Calls Avg Min Max Name
46.85% 682.28ms 8 85.285ms 12.639us 682.03ms cudaStreamCreateWithFlags
39.83% 580.00ms 4 145.00ms 302ns 550.27ms cudaFree
9.82% 143.03ms 9 15.892ms 8.7090us 142.78ms cudaStreamCreate
1.23% 17.983ms 7 2.5690ms 23.210us 6.4563ms cudaMemcpy
1.23% 17.849ms 2 8.9247ms 8.4726ms 9.3768ms cudaStreamSynchronize
0.66% 9.5969ms 7 1.3710ms 288.43us 2.4279ms cudaHostAlloc
0.13% 1.9530ms 11 177.54us 7.6810us 591.06us cudaMalloc
0.07% 1.0424ms 8 130.30us 1.6970us 453.72us cudaGetDevice
0.04% 527.90us 40 13.197us 525ns 253.99us cudaEventCreateWithFlags
0.03% 435.73us 348 1.2520us 124ns 42.704us cuDeviceGetAttribute
0.03% 419.36us 1 419.36us 419.36us 419.36us cudaGetDeviceCount
0.02% 260.75us 2 130.38us 129.32us 131.43us cudaGetDeviceProperties
0.02% 222.32us 2 111.16us 106.94us 115.39us cudaLaunch
0.01% 214.06us 4 53.514us 28.586us 77.655us cuDeviceGetName
0.01% 115.45us 4 28.861us 9.8250us 44.526us cuDeviceTotalMem
0.01% 83.988us 4 20.997us 578ns 77.760us cudaSetDevice
0.00% 38.918us 1 38.918us 38.918us 38.918us cudaEventCreate
0.00% 34.573us 31 1.1150us 279ns 12.784us cudaDeviceGetAttribute
0.00% 17.767us 1 17.767us 17.767us 17.767us cudaProfilerStart
0.00% 15.228us 2 7.6140us 3.5460us 11.682us cudaConfigureCall
0.00% 14.536us 2 7.2680us 1.1490us 13.387us cudaGetLastError
0.00% 8.6080us 26 331ns 173ns 783ns cudaSetupArgument
0.00% 5.5470us 6 924ns 215ns 2.6780us cuDeviceGet
0.00% 5.4090us 6 901ns 328ns 3.3320us cuDeviceGetCount
0.00% 4.1770us 3 1.3920us 1.0630us 1.8300us cuDriverGetVersion
0.00% 3.4650us 3 1.1550us 1.0810us 1.2680us cuInit
0.00% 830ns 1 830ns 830ns 830ns cudaRuntimeGetVersion
nvvp profiler
-------------
For visual profiler **nvvp**, you can either import the output of :code:`nvprof –o ...` or
run application through GUI.
**Note: nvvp also support CPU profiling** (Click the box in nvvp to enable profile execution on CPU).
.. image:: nvvp1.png
:align: center
:scale: 33%
From the perspective of kernel functions, **nvvp** can even illustrate why does an operation take a long time?
As shown in the following figure, kernel's block usage, register usage and shared memory usage from :code:`nvvp`
allow us to fully utilize all warps on the GPU.
.. image:: nvvp2.png
:align: center
:scale: 33%
From the perspective of application, **nvvp** can give you some suggestions to address performance bottleneck.
For instance, some advice in data movement and compute utilization from the below figure can guide you to tune performance.
.. image:: nvvp3.png
:align: center
:scale: 33%
.. image:: nvvp4.png
:align: center
:scale: 33%
Profiling tips
==============
- The **nvprof** and **nvvp** output is a very good place to start.
- The timeline is a good place to go next.
- Only dig deep into a kernel if it’s taking a significant amount of your time.
- Where possible, try to match profiler output with theory.
1) For example, if I know I’m moving 1GB, and my kernel takes 10ms, I expect the profiler to report 100GB/s.
2) Discrepancies are likely to mean your application isn’t doing what you thought it was.
- Know your hardware: If your GPU can do 6 TFLOPs, and you’re already doing 5.5 TFLOPs, you won’t go much faster!
Profiling is a key step in optimization. Sometimes quite simple changes can lead to big improvements in performance.
Your mileage may vary!
Reference
=========
Jeremy Appleyard, `GPU Profiling for Deep Learning <http://www.robots.ox.ac.uk/~seminars/seminars/Extra/2015_10_08_JeremyAppleyard.pdf>`_, 2015
doc/source/cuda/utils/index.rst → doc/optimization/index.rst
浏览文件 @ 96615fe3
Utils
====================
Performance Tuning
==================
.. toctree::
:maxdepth: 3
utils.rst
gpu_profiling.rst
doc/source/api/api.rst → doc/source/api.rst
浏览文件 @ 96615fe3
API
========
===
.. doxygenfile:: paddle/api/PaddleAPI.h
.. doxygenfile:: paddle/api/Internal.h
doc/source/cuda/cuda/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
CUDA
====================
.. toctree::
:maxdepth: 3
cuda.rst
doc/source/cuda/matrix/index.rst → doc/source/cuda/index.rst
浏览文件 @ 96615fe3
Matrix
====================
CUDA
====
.. toctree::
:maxdepth: 3
:maxdepth: 2
matrix.rst
nn.rst
utils.rst
doc/source/cuda/matrix/matrix.rst → doc/source/cuda/matrix.rst
浏览文件 @ 96615fe3
Matrix
=======
======
Base Matrix
-------------
Base
----
hl_matrix.h
``````````````````
```````````
.. doxygenfile:: paddle/cuda/include/hl_matrix.h
hl_matrix_base.h
``````````````````
````````````````
.. doxygenfile:: paddle/cuda/include/hl_matrix_base.cuh
hl_matrix_apply.cuh
``````````````````````
```````````````````
.. doxygenfile:: paddle/cuda/include/hl_matrix_apply.cuh
hl_matrix_ops.cuh
``````````````````````
`````````````````
.. doxygenfile:: paddle/cuda/include/hl_matrix_ops.cuh
hl_matrix_type.cuh
``````````````````````
``````````````````
.. doxygenfile:: paddle/cuda/include/hl_matrix_type.cuh
hl_sse_matrix_kernel.cuh
``````````````````````````
````````````````````````
.. doxygenfile:: paddle/cuda/include/hl_sse_matrix_kernel.cuh
Matrix Function
---------------
hl_batch_transpose.h
``````````````````````````
````````````````````
.. doxygenfile:: paddle/cuda/include/hl_batch_transpose.h
Sparse Matrix
--------------
hl_sparse.h
``````````````````
.. doxygenfile:: paddle/cuda/include/hl_sparse.h
hl_sparse.ph
``````````````````````
.. doxygenfile:: paddle/cuda/include/hl_sparse.ph
Others
---------------
hl_aggregate.h
``````````````````
``````````````
.. doxygenfile:: paddle/cuda/include/hl_aggregate.h
hl_top_k.h
``````````
.. doxygenfile:: paddle/cuda/include/hl_top_k.h
hl_table_apply.h
``````````````````
````````````````
.. doxygenfile:: paddle/cuda/include/hl_table_apply.h
hl_top_k.h
``````````````````
.. doxygenfile:: paddle/cuda/include/hl_top_k.h
Sparse Matrix
-------------
hl_sparse.h
```````````
.. doxygenfile:: paddle/cuda/include/hl_sparse.h
hl_sparse.ph
````````````
.. doxygenfile:: paddle/cuda/include/hl_sparse.ph
doc/source/cuda/rnn/rnn.rst → doc/source/cuda/nn.rst
浏览文件 @ 96615fe3
Neural Networks
==================
Neural Network
==============
Base
-------
----
.. doxygenfile:: paddle/cuda/include/hl_gpu.h
.. doxygenfile:: paddle/cuda/include/hl_cnn.h
.. doxygenfile:: paddle/cuda/include/hl_functions.h
.. doxygenfile:: paddle/cuda/include/hl_avx_functions.h
.. doxygenfile:: paddle/cuda/include/hl_device_functions.cuh
.. doxygenfile:: paddle/cuda/include/hl_gpu_functions.cuh
Activation Functions
-----------------------
.. doxygenfile:: paddle/cuda/include/hl_activation_functions.h
CNN Related APIs
----------------
.. doxygenfile:: paddle/cuda/include/hl_cnn.h
.. doxygenfile:: paddle/cuda/include/hl_cuda_cudnn.h
.. doxygenfile:: paddle/cuda/include/hl_cuda_cudnn.ph
RNN Related APIs
-----------------
----------------
.. doxygenfile:: paddle/cuda/include/hl_recurrent_apply.cuh
.. doxygenfile:: paddle/cuda/include/hl_sequence.h
LSTM Model
``````````````
``````````
.. doxygenfile:: paddle/cuda/include/hl_lstm.h
.. dpxygenfile:: paddle/cuda/include/hl_cpu_lstm.cuh
.. doxygenfile:: paddle/cuda/include/hl_gpu_lstm.cuh
.. doxygenfile:: paddle/cuda/include/hl_lstm_ops.cuh
GRU Model
````````````````
`````````
.. doxygenfile:: paddle/cuda/include/hl_gru_ops.cuh
.. doxygenfile:: paddle/cuda/include/hl_cpu_gru.cuh
.. doxygenfile:: paddle/cuda/include/hl_gpu_gru.cuh
doc/source/cuda/rnn/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
RNN
====================
.. toctree::
:maxdepth: 3
rnn.rst
doc/source/cuda/cuda/cuda.rst → doc/source/cuda/utils.rst
浏览文件 @ 96615fe3
Cuda
=============
Utils
=====
Dynamic Link Libs
--------------------------
hl_dso_loader.h
``````````````````
-----------------
.. doxygenfile:: paddle/cuda/include/hl_dso_loader.h
GPU Resources
----------------
-------------
hl_cuda.ph
``````````````
``````````
.. doxygenfile:: paddle/cuda/include/hl_cuda.ph
hl_cuda.h
``````````````
`````````
.. doxygenfile:: paddle/cuda/include/hl_cuda.h
CUDA Wrapper
--------------
HPPL Base
---------
.. doxygenfile:: paddle/cuda/include/hl_base.h
hl_cuda_cublas.h
``````````````````````
CUBLAS Wrapper
--------------
.. doxygenfile:: paddle/cuda/include/hl_cuda_cublas.h
hl_cuda_cudnn.h
``````````````````````
.. doxygenfile:: paddle/cuda/include/hl_cuda_cudnn.h
hl_cuda_cudnn.h
``````````````````````
.. doxygenfile:: paddle/cuda/include/hl_cuda_cudnn.ph
Timer
-----
.. doxygenfile:: paddle/cuda/include/hl_time.h
Thread Resource
---------------
.. doxygenfile:: paddle/cuda/include/hl_thread.ph
Device Function
---------------
.. doxygenfile:: paddle/cuda/include/hl_device_functions.cuh
doc/source/cuda/utils/utils.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Utilities
===========
HPPL Base
------------
hl_base.h
``````````````
.. doxygenfile:: paddle/cuda/include/hl_base.h
Timer
-----------
hl_time.h
``````````````
.. doxygenfile:: paddle/cuda/include/hl_time.h
Thread Resource
-----------
hl_thread.ph
``````````````
.. doxygenfile:: paddle/cuda/include/hl_thread.ph
doc/source/gserver/activations/index.rst → doc/source/gserver/activations.rst
浏览文件 @ 96615fe3
Activations
=============
===========
.. doxygenclass:: paddle::ActivationFunction
:members:
doc/source/gserver/dataprovider/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Data Providers Documents
==========================
.. toctree::
:maxdepth: 3
dataproviders.rst
doc/source/gserver/dataprovider/dataproviders.rst → doc/source/gserver/dataproviders.rst
浏览文件 @ 96615fe3
==============
Data Providers
================
==============
Base DataProvider
------------------
DataProviders
=============
Base
----
.. doxygenclass:: paddle::DataProvider
:members:
DataProviderGroup
-------------------
-----------------
.. doxygenclass:: paddle::DataProviderGroup
:members:
MultiDataProvider
-------------------
-----------------
.. doxygenclass:: paddle::MultiDataProvider
:members:
PyDataProvider
===================
==============
IFieldScanner
-------------
......@@ -45,7 +49,7 @@ SparseValueScanner
:members:
SequenceScanner
------------------
---------------
.. doxygenclass:: paddle::SparseValueScanner
:members:
......@@ -69,8 +73,8 @@ IPyDataProvider
.. doxygenclass:: paddle::PyDataProvider2
:members:
Proto Data Provider
===================
ProtoDataProvider
=================
ProtoDataProvider
----------------
......@@ -78,6 +82,6 @@ ProtoDataProvider
:members:
ProtoSequenceDataProvider
----------------
-------------------------
.. doxygenclass:: paddle::ProtoSequenceDataProvider
:members:
doc/source/gserver/evaluators/evaluators.rst → doc/source/gserver/evaluators.rst
浏览文件 @ 96615fe3
Base Evaluator
==============
==========
Evaluators
==========
Base
====
Evaluator
---------
.. doxygenclass:: paddle::Evaluator
:members:
Utils
=====
Sum
===
SumEvaluator
------------
......
doc/source/gserver/evaluators/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Evaluators
==========
.. toctree::
:maxdepth: 3
evaluators.rst
doc/source/gserver/gradientmachines/gradientmachines.rst → doc/source/gserver/gradientmachines.rst
浏览文件 @ 96615fe3
Gradient Machines
================
=================
GradientMachine
---------------------
---------------
.. doxygenclass:: paddle::GradientMachine
:members:
GradientMachineModel
--------------------
GradientMachineMode
-------------------
.. doxygenclass:: paddle::IGradientMachineMode
:members:
MultiGradientMachine
---------------------
--------------------
.. doxygenclass:: paddle::MultiGradientMachine
:members:
......@@ -21,20 +21,7 @@ TrainerThread
.. doxygenclass:: paddle::TrainerThread
:members:
Recurrent Gradient Machines
---------------------------
RecurrentGradientMachine
------------------------
.. doxygenclass:: paddle::RecurrentGradientMachine
:members:
Networks
========
NeuralNetwork
-------------
.. doxygenclass:: paddle::NeuralNetwork
:members:
ParallelNeuralNetwork
---------------------
.. doxygenclass:: paddle::ParallelNeuralNetwork
:members:
doc/source/gserver/gradientmachines/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Gradient Machines Documents
=============================
.. toctree::
:maxdepth: 3
gradientmachines.rst
doc/source/gserver/index.rst 0 → 100644
浏览文件 @ 96615fe3
GServer
=======
.. toctree::
:maxdepth: 2
activations.rst
dataproviders.rst
evaluators.rst
gradientmachines.rst
layers.rst
neworks.rst
doc/source/gserver/layers/layer.rst → doc/source/gserver/layers.rst
浏览文件 @ 96615fe3
Base
======
Layers
======
Base
====
Layer
-----
.. doxygenclass:: paddle::Layer
......@@ -17,7 +21,7 @@ Operator
:members:
Data Layer
===========
==========
.. doxygenclass:: paddle::DataLayer
:members:
......@@ -58,6 +62,11 @@ CudnnConvLayer
.. doxygenclass:: paddle::CudnnConvLayer
:members:
ExpandConvBaseLayer
-------------------
.. doxygenclass:: paddle::ExpandConvBaseLayer
:members:
ExpandConvLayer
---------------
.. doxygenclass:: paddle::ExpandConvLayer
......@@ -86,6 +95,16 @@ CudnnPoolLayer
.. doxygenclass:: paddle::CudnnPoolLayer
:members:
SpatialPyramidPoolLayer
-----------------------
.. doxygenclass:: paddle::SpatialPyramidPoolLayer
:members:
MaxOutLayer
-----------
.. doxygenclass:: paddle::MaxOutLayer
:members:
Norm Layers
===========
......@@ -402,6 +421,11 @@ TransLayer
Sampling Layers
===============
BilinearInterpLayer
-------------------
.. doxygenclass:: paddle::BilinearInterpLayer
:members:
MultinomialSampler
------------------
.. doxygenclass:: paddle::MultinomialSampler
......
doc/source/gserver/layers/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Layers Documents
====================
.. toctree::
:maxdepth: 3
layer.rst
doc/source/gserver/neworks.rst 0 → 100644
浏览文件 @ 96615fe3
Networks
========
NeuralNetwork
-------------
.. doxygenclass:: paddle::NeuralNetwork
:members:
ParallelNeuralNetwork
---------------------
.. doxygenclass:: paddle::ParallelNeuralNetwork
:members:
doc/source/index.md 已删除 100644 → 0
浏览文件 @ 496d64eb
# Source Code Documents
## cuda
- [CUDA](cuda/cuda/index.rst)
- [Matrix](cuda/matrix/index.rst)
- [RNN](cuda/rnn/index.rst)
- [Utils](cuda/utils/index.rst)
## gserver
- [Activations](gserver/activations/index.rst)
- [Data Providers](gserver/dataprovider/index.rst)
- [Evaluators](gserver/evaluators/index.rst)
- [Gradient Machines](gserver/gradientmachines/index.rst)
- [Layers](gserver/layers/index.rst)
## math
- [Matrix](math/matrix/index.rst)
- [Utils](math/utils/index.rst)
## parameter
- [Parameter](parameter/parameter/index.rst)
- [Update](parameter/update/index.rst)
- [Optimizer](parameter/optimizer/index.rst)
## pserver
- [Client](pserver/client/index.rst)
- [Network](pserver/network/index.rst)
- [Server](pserver/server/index.rst)
## trainer
- [Trainer](trainer/trainer.rst)
## api
- [API](api/api.rst)
## utils
- [CustomStackTrace](utils/customStackTrace.rst)
- [Enumeration wrapper](utils/enum.rst)
- [Lock](utils/lock.rst)
- [Queue](utils/queue.rst)
- [Thread](utils/thread.rst)
doc/source/index.rst 0 → 100644
浏览文件 @ 96615fe3
Source Code Documents
=====================
.. toctree::
:maxdepth: 1
gserver/index.rst
trainer.rst
parameter/index.rst
pserver/index.rst
api.rst
cuda/index.rst
math/index.rst
utils/index.rst
doc/source/math/functions.rst 0 → 100644
浏览文件 @ 96615fe3
Functions
=========
MathFunctions
-------------
.. doxygenfile:: paddle/math/MathFunctions.h
SIMDFunctions
-------------
.. doxygenfile:: paddle/math/SIMDFunctions.h
doc/source/math/index.rst 0 → 100644
浏览文件 @ 96615fe3
Math
====
.. toctree::
:maxdepth: 2
vector.rst
matrix.rst
functions.rst
utils.rst
doc/source/math/matrix.rst 0 → 100644
浏览文件 @ 96615fe3
Matrix
======
Base
----
BaseMatrix Template
```````````````````
.. doxygenclass:: paddle::BaseMatrixT
:members:
Matrix
``````
.. doxygenclass:: paddle::Matrix
:members:
MatrixOffset
````````````
.. doxygenclass:: paddle::MatrixOffset
:members:
CpuMatrix
---------
CpuMatrix
`````````
.. doxygenclass:: paddle::CpuMatrix
:members:
SharedCpuMatrix
```````````````
.. doxygenclass:: paddle::SharedCpuMatrix
:members:
GpuMatrix
---------
.. doxygenclass:: paddle::GpuMatrix
:members:
CpuSparseMatrix
---------------
CpuSparseMatrix
```````````````
.. doxygenclass:: paddle::CpuSparseMatrix
:members:
SparseRowCpuMatrix
``````````````````
.. doxygenclass:: paddle::SparseRowCpuMatrix
:members:
SparseAutoGrowRowCpuMatrix
``````````````````````````
.. doxygenclass:: paddle::SparseAutoGrowRowCpuMatrix
:members:
SparsePrefetchRowCpuMatrix
``````````````````````````
.. doxygenclass:: paddle::SparsePrefetchRowCpuMatrix
:members:
SparseRowIdsCpuMatrix
`````````````````````
.. doxygenclass:: paddle::SparseRowIdsCpuMatrix
:members:
CacheRowCpuMatrix
`````````````````
.. doxygenclass:: paddle::CacheRowCpuMatrix
:members:
GpuSparseMatrix
---------------
.. doxygenclass:: paddle::GpuSparseMatrix
:members:
doc/source/math/matrix/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Matrix Documents
====================
.. toctree::
:maxdepth: 3
matrix.rst
doc/source/math/matrix/matrix.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Matrix
=======
Base
--------
.. doxygenfile:: paddle/math/BaseMatrix.h
Sparse Matrix
----------------
.. doxygenfile:: paddle/math/Matrix.h
.. doxygenfile:: paddle/math/Vector.h
.. doxygenfile:: paddle/math/MathUtils.h
.. doxygenfile:: paddle/math/SparseMatrix.h
.. doxygenfile:: paddle/math/SparseRowMatrix.h
.. doxygenfile:: paddle/math/CpuSparseMatrix.h
Others
----------
.. doxygenfile:: paddle/math/MathFunctions.h
.. doxygenfile:: paddle/math/SIMDFunctions.h
doc/source/math/utils/utils.rst → doc/source/math/utils.rst
浏览文件 @ 96615fe3
Utils
=======
Memory Manager
==============
Memory Handle
--------------
-------------
.. doxygenfile:: paddle/math/MemoryHandle.h
Allocator
---------
.. doxygenfile:: paddle/math/Allocator.h
PoolAllocator
`````````````
.. doxygenfile:: paddle/math/PoolAllocator.h
Storage
-------
.. doxygenfile:: paddle/math/Storage.h
doc/source/math/utils/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Utils Documents
====================
.. toctree::
:maxdepth: 3
utils.rst
doc/source/math/vector.rst 0 → 100644
浏览文件 @ 96615fe3
Vector
======
BaseVector
``````````
.. doxygenclass:: paddle::BaseVector
:members:
Vector Template
```````````````
.. doxygenclass:: paddle::VectorT
:members:
CpuVector Template
``````````````````
.. doxygenclass:: paddle::CpuVectorT
:members:
GpuVector Template
``````````````````
.. doxygenclass:: paddle::GpuVectorT
:members:
ParallelCpuVector Template
``````````````````````````
.. doxygenclass:: paddle::ParallelCpuVectorT
:members:
ParallelGpuVector Template
``````````````````````````
.. doxygenclass:: paddle::ParallelGpuVectorT
:members:
CpuGpuVector Template
`````````````````````
.. doxygenclass:: paddle::CpuGpuVectorT
:members:
doc/source/parameter/optimizer/index.rst → doc/source/parameter/index.rst
浏览文件 @ 96615fe3
Parameter Documents
====================
Parameter
=========
.. toctree::
:maxdepth: 3
:maxdepth: 2
parameter.rst
optimizer.rst
updater.rst
doc/source/parameter/optimizer/optimizer.rst → doc/source/parameter/optimizer.rst
浏览文件 @ 96615fe3
Optimizer
============
=========
ParameterOptimizer
------------------
.. doxygenfile:: paddle/parameter/ParameterOptimizer.h
Regularizer
-----------
.. doxygenfile:: paddle/parameter/Regularizer.h
FirstOrderOptimizer
-------------------
.. doxygenfile:: paddle/parameter/FirstOrderOptimizer.h
AverageOptimizer
----------------
.. doxygenfile:: paddle/parameter/AverageOptimizer.h
.. doxygenfile:: paddle/parameter/ParameterOptimizer.h
OptimizerWithRegularizer
------------------------
.. doxygenfile:: paddle/parameter/OptimizerWithRegularizer.h
doc/source/parameter/parameter/parameter.rst → doc/source/parameter/parameter.rst
浏览文件 @ 96615fe3
Parameter
=============
Weight
--------
.. doxygenfile:: paddle/parameter/Weight.h
Regularizer
------------
.. doxygenfile:: paddle/parameter/Regularizer.h
=========
Parameter
-------------
---------
.. doxygenfile:: paddle/parameter/Argument.h
.. doxygenfile:: paddle/parameter/Parameter.h
.. doxygenfile:: paddle/parameter/ParallelParameter.h
Weight
------
.. doxygenfile:: paddle/parameter/Weight.h
doc/source/parameter/parameter/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Parameter Documents
====================
.. toctree::
:maxdepth: 3
parameter.rst
doc/source/parameter/update/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Parameter Documents
====================
.. toctree::
:maxdepth: 3
update.rst
doc/source/parameter/update/update.rst → doc/source/parameter/updater.rst
浏览文件 @ 96615fe3
Update
==========
Updater
=======
Base
----
.. doxygenfile:: paddle/parameter/ParameterUpdaterBase.h
Hook
----
.. doxygenfile:: paddle/parameter/ParameterUpdaterHook.h
.. doxygenfile:: paddle/parameter/ParameterUpdateFunctions.h
Functions
---------
.. doxygenfile:: paddle/parameter/ParameterUpdateFunctions.h
doc/source/pserver/client.rst 0 → 100644
浏览文件 @ 96615fe3
Client
======
BaseClient
----------
.. doxygenclass:: paddle::BaseClient
:members:
ParameterClient2
----------------
.. doxygenclass:: paddle::ParameterClient2
:members:
doc/source/pserver/client/client.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Client
=========
.. doxygenclass:: paddle::BaseClient
:members:
:protected-members:
:private-members:
:undoc-members:
.. doxygenclass:: paddle::ParameterClient2
:members:
:protected-members:
:private-members:
:undoc-members:
doc/source/pserver/client/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Client Documents
====================
.. toctree::
:maxdepth: 3
client.rst
doc/source/pserver/index.rst 0 → 100644
浏览文件 @ 96615fe3
PServer
=======
.. toctree::
:maxdepth: 2
client.rst
network.rst
server.rst
utils.rst
doc/source/pserver/network.rst 0 → 100644
浏览文件 @ 96615fe3
Network
=======
SocketServer
------------
.. doxygenclass:: paddle::SocketServer
:members:
SocketWorker
------------
.. doxygenclass:: paddle::SocketWorker
:members:
SocketClient
------------
.. doxygenclass:: paddle::SocketClient
:members:
SocketChannel
-------------
.. doxygenclass:: paddle::SocketChannel
:members:
MessageReader
-------------
.. doxygenclass:: paddle::MsgReader
:members:
doc/source/pserver/network/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Network Documents
====================
.. toctree::
:maxdepth: 3
network.rst
doc/source/pserver/network/network.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Network
==========
Socket Server
----------------
.. doxygenclass:: paddle::SocketServer
:members:
:protected-members:
:private-members:
:undoc-members:
Socket Worker
----------------
.. doxygenclass:: paddle::SocketWorker
:members:
:protected-members:
:private-members:
:undoc-members:
Socket Client
----------------
.. doxygenclass:: paddle::SocketClient
:members:
:protected-members:
:private-members:
:undoc-members:
Socket Channel
---------------
.. doxygenclass:: paddle::SocketChannel
:members:
:protected-members:
:private-members:
:undoc-members:
Message Reader
---------------
.. doxygenclass:: paddle::MsgReader
:members:
:protected-members:
:private-members:
:undoc-members:
doc/source/pserver/server.rst 0 → 100644
浏览文件 @ 96615fe3
Server
======
ProtoServer
-----------
.. doxygenclass:: paddle::ProtoServer
:members:
ParameterServer2
----------------
.. doxygenclass:: paddle::ParameterServer2
:members:
doc/source/pserver/server/index.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Server Documents
====================
.. toctree::
:maxdepth: 3
server.rst
doc/source/pserver/server/server.rst 已删除 100644 → 0
浏览文件 @ 496d64eb
Server
==========
.. doxygenclass:: paddle::ProtoServer
:members:
:protected-members:
:private-members:
:undoc-members:
.. doxygenclass:: paddle::ParameterServer2
:members:
:protected-members:
:private-members:
:undoc-members:
doc/source/trainer/trainer.rst → doc/source/trainer.rst
浏览文件 @ 96615fe3
......@@ -14,7 +14,7 @@ RemoteParameterUpdater
:members:
ConcurrentRemoteParameterUpdater
---------------------------------
--------------------------------
.. doxygenclass:: paddle::ConcurrentRemoteParameterUpdater
:members:
......
doc/source/utils/customStackTrace.rst
浏览文件 @ 96615fe3
CustomStackTrace
================
class CustomStackTrace
----------------------
.. doxygenclass:: paddle::CustomStackTrace
:members:
doc/source/utils/enum.rst
浏览文件 @ 96615fe3
enumeration_wrapper
Enumeration wrapper
===================
namespace paddle::enumeration_wrapper
-------------------------------------
.. doxygennamespace:: paddle::enumeration_wrapper
doc/source/utils/index.rst 0 → 100644
浏览文件 @ 96615fe3
Utils
=====
.. toctree::
:maxdepth: 2
lock.rst
queue.rst
thread.rst
customStackTrace.rst
enum.rst
doc/source/utils/lock.rst
浏览文件 @ 96615fe3
Thread
======
Lock
====
class Thread
------------
.. doxygenclass:: paddle::Thread
RWLock
------
.. doxygenclass:: paddle::RWLock
:members:
class ThreadWorker
------------------
.. doxygenclass:: paddle::ThreadWorker
ReadLockGuard
-------------
.. doxygenclass:: paddle::ReadLockGuard
:members:
class SyncThreadPool
--------------------
.. doxygenclass:: paddle::SyncThreadPool
SpinLock
--------
.. doxygenclass:: paddle::SpinLock
:members:
class MultiThreadWorker
-----------------------
.. doxygenclass:: paddle::MultiThreadWorker
Semaphore
---------
.. doxygenclass:: paddle::Semaphore
:members:
class AsyncThreadPool
---------------------
ThreadBarrier
-------------
.. doxygenclass:: paddle::ThreadBarrier
:members:
.. doxygenclass:: paddle::AsyncThreadPool
LockedCondition
---------------
.. doxygenclass:: paddle::LockedCondition
:members:
doc/source/utils/queue.rst
浏览文件 @ 96615fe3
Queue
=====
class Queue
------------
Queue
-----
.. doxygenclass:: paddle::Queue
:members:
class BlockingQueue
-------------------
BlockingQueue
-------------
.. doxygenclass:: paddle::BlockingQueue
:members:
doc/source/utils/thread.rst
浏览文件 @ 96615fe3
Lock
====
Thread
======
class RWLock
------------
.. doxygenclass:: paddle::RWLock
Thread
------
.. doxygenclass:: paddle::Thread
:members:
class ReadLockGuard
-------------------
.. doxygenclass:: paddle::ReadLockGuard
ThreadWorker
------------
.. doxygenclass:: paddle::ThreadWorker
:members:
class SpinLock
SyncThreadPool
--------------
.. doxygenclass:: paddle::SpinLock
.. doxygenclass:: paddle::SyncThreadPool
:members:
class Semaphore
---------------
.. doxygenclass:: paddle::Semaphore
:members:
class ThreadBarrier
-------------------
.. doxygenclass:: paddle::ThreadBarrier
MultiThreadWorker
-----------------
.. doxygenclass:: paddle::MultiThreadWorker
:members:
class LockedCondition
---------------------
.. doxygenclass:: paddle::LockedCondition
AsyncThreadPool
---------------
.. doxygenclass:: paddle::AsyncThreadPool
:members:
doc_cn/algorithm/rnn/hierarchical-layer.md → doc_cn/algorithm/rnn/hierarchical-layer.rst
浏览文件 @ 96615fe3
# 支持双层序列作为输入的Layer
###########################
支持双层序列作为输入的Layer
###########################
## 概述
.. contents::
概述
====
在自然语言处理任务中，序列是一种常见的数据类型。一个独立的词语，可以看作是一个非序列输入，或者，我们称之为一个0层的序列；由词语构成的句子，是一个单层序列；若干个句子构成一个段落，是一个双层的序列。
......@@ -12,55 +17,79 @@
+ 单层序列：排成一列的多个元素，每个元素是一个0层序列，元素之间的顺序是重要的输入信息
+ 双层序列：排成一列的多个元素，每个元素是一个单层序列，称之为双层序列的一个子序列（subseq），subseq的每个元素是一个0层序列
在 PaddlePaddle中，下面这些Layer能够接受双层序列作为输入，完成相应的计算。
## pooling_layer
pooling_layer的使用示例如下，详细见<a href = "../../../doc/ui/api/trainer_config_helpers/layers.html#pooling-layer">配置API</a>。
```python
seq_pool = pooling_layer(input=layer,
pooling_type=AvgPooling(),
agg_level=AggregateLevel.EACH_SEQUENCE)
```
pooling_layer
==============
pooling_layer 的使用示例如下，详细见 `pooling_layer`_ 配置API。
.. code-block:: bash
seq_pool = pooling_layer(input=layer,
pooling_type=AvgPooling(),
agg_level=AggregateLevel.EACH_SEQUENCE)
- `pooling_type` 目前支持两种，分别是：MaxPooling()和AvgPooling()。
- `agg_level=AggregateLevel.TIMESTEP`时（默认值）：
- `agg_level=AggregateLevel.TIMESTEP` 时（默认值）：
- 作用：双层序列经过运算变成一个0层序列，或单层序列经过运算变成一个0层序列
- 输入：一个双层序列，或一个单层序列
- 输出：一个0层序列，即整个输入序列（单层或双层）的平均值（或最大值）
- `agg_level=AggregateLevel.EACH_SEQUENCE`时：
- `agg_level=AggregateLevel.EACH_SEQUENCE` 时：
- 作用：一个双层序列经过运算变成一个单层序列
- 输入：必须是一个双层序列
- 输出：一个单层序列，序列的每个元素是原来双层序列每个subseq元素的平均值（或最大值）
## last_seq 和 first_seq
last_seq 和 first_seq
=====================
last_seq 的使用示例如下（ `first_seq`_ 类似），详细见 `last_seq`_ 配置API。
.. code-block:: bash
last = last_seq(input=layer,
agg_level=AggregateLevel.EACH_SEQUENCE)
- `agg_level=AggregateLevel.TIMESTEP` 时（默认值）：
last_seq的使用示例如下（first_seq类似），详细见<a href = "../../../doc/ui/api/trainer_config_helpers/layers.html#last-seq">配置API</a>。
```python
last = last_seq(input=layer,
agg_level=AggregateLevel.EACH_SEQUENCE)
```
- `agg_level=AggregateLevel.TIMESTEP`时（默认值）：
- 作用：一个双层序列经过运算变成一个0层序列，或一个单层序列经过运算变成一个0层序列
- 输入：一个双层序列或一个单层序列
- 输出：一个0层序列，即整个输入序列（双层或者单层）最后一个，或第一个元素。
- `agg_level=AggregateLevel.EACH_SEQUENCE`时：
- `agg_level=AggregateLevel.EACH_SEQUENCE` 时：
- 作用：一个双层序列经过运算变成一个单层序列
- 输入：必须是一个双层序列
- 输出：一个单层序列，其中每个元素是双层序列中每个subseq最后一个（或第一个）元素。
## expand_layer
expand_layer
============
expand_layer 的使用示例如下，详细见 `expand_layer`_ 配置API。
.. code-block:: bash
expand = expand_layer(input=layer1,
expand_as=layer2,
expand_level=ExpandLevel.FROM_TIMESTEP)
- `expand_level=ExpandLevel.FROM_TIMESTEP` 时（默认值）：
expand_layer的使用示例如下，详细见<a href = "../../../doc/ui/api/trainer_config_helpers/layers.html#expand-layer">配置API</a>。
```python
expand = expand_layer(input=layer1,
expand_as=layer2,
expand_level=ExpandLevel.FROM_TIMESTEP)
```
- `expand_level=ExpandLevel.FROM_TIMESTEP`时（默认值）：
- 作用：一个0层序列经过运算扩展成一个单层序列，或者一个双层序列
- 输入：layer1必须是一个0层序列，是待扩展的数据；layer2可以是一个单层序列，或者是一个双层序列，提供扩展的长度信息
- 输出：一个单层序列，或一个双层序列，输出序列的类型（双层序列，或单层序列）和序列中含有元素的数目同 layer2一致。若输出是单层序列，单层序列的每个元素（0层序列），都是对layer1元素的拷贝；若输出是双层序列，双层序列每个subseq中每个元素（0层序列），都是对layer1元素的拷贝
- `expand_level=ExpandLevel.FROM_SEQUENCE`时：
- 输入：layer1必须是一个0层序列，是待扩展的数据；layer2 可以是一个单层序列，或者是一个双层序列，提供扩展的长度信息
- 输出：一个单层序列或一个双层序列，输出序列的类型（双层序列或单层序列）和序列中含有元素的数目同 layer2 一致。若输出是单层序列，单层序列的每个元素（0层序列），都是对layer1元素的拷贝；若输出是双层序列，双层序列每个subseq中每个元素（0层序列），都是对layer1元素的拷贝
- `expand_level=ExpandLevel.FROM_SEQUENCE` 时：
- 作用：一个单层序列经过运算扩展成一个双层序列
- 输入：layer1必须是一个单层序列，是待扩展的数据；layer2必须是一个双层序列，提供扩展的长度信息
- 输出：一个双层序列，序列中含有元素的数目同layer2一致。要求单层序列含有元素的数目（0层序列），和双层序列含有subseq 的数目一致。单层序列第i个元素（0层序列），被扩展为一个单层序列，构成了输出双层序列的第i个subseq。
- 输入：layer1必须是一个单层序列，是待扩展的数据；layer2 必须是一个双层序列，提供扩展的长度信息
- 输出：一个双层序列，序列中含有元素的数目同 layer2 一致。要求单层序列含有元素的数目（0层序列）和双层序列含有subseq 的数目一致。单层序列第i个元素（0层序列），被扩展为一个单层序列，构成了输出双层序列的第i个 subseq 。
.. _pooling_layer: ../../../doc/ui/api/trainer_config_helpers/layers.html#pooling-layer
.. _last_seq: ../../../doc/ui/api/trainer_config_helpers/layers.html#last-seq
.. _first_seq: ../../../doc/ui/api/trainer_config_helpers/layers.html#first-seq
.. _expand_layer: ../../../doc/ui/api/trainer_config_helpers/layers.html#expand-layer
doc_cn/build_and_install/cmake/cblas_settings.csv
浏览文件 @ 96615fe3
MKL_ROOT,mkl的路径，在${MKL_ROOT}/include下需要包含mkl.h，在${MKL_ROOT}/lib目录下需要包含 mkl_core，mkl_sequential和mkl_intel_lp64三个库
ATLAS_ROOT,ATLAS库的路径，在${ATLAS_ROOT}/include下需要包含cblas.h，而在${ATLAS_ROOT}/lib下需要包含cblas和atlas两个库
OPENBLAS_ROOT,在${OPENBLAS_ROOT}/include下需要包含cblas.h，而在${OPENBLAS_ROOT}/lib下需要包含openblas库
REFERENCE_CBLAS_ROOT,在${REFERENCE_CBLAS_ROOT}/include下需要包含cblas.h，在${REFERENCE_CBLAS_ROOT}/lib下需要包含cblas库
\ No newline at end of file
编译选项,描述,注意
MKL_ROOT,MKL的路径,${MKL_ROOT}/include下需要包含mkl.h，${MKL_ROOT}/lib目录下需要包含mkl_core，mkl_sequential和mkl_intel_lp64三个库。
ATLAS_ROOT,ATLAS的路径,${ATLAS_ROOT}/include下需要包含cblas.h，${ATLAS_ROOT}/lib下需要包含cblas和atlas两个库。
OPENBLAS_ROOT,OpenBLAS的路径,${OPENBLAS_ROOT}/include下需要包含cblas.h，${OPENBLAS_ROOT}/lib下需要包含openblas库。
REFERENCE_CBLAS_ROOT,REFERENCE BLAS的路径,${REFERENCE_CBLAS_ROOT}/include下需要包含cblas.h，${REFERENCE_CBLAS_ROOT}/lib下需要包含cblas库。
\ No newline at end of file
doc_cn/build_and_install/cmake/compile_options.csv
浏览文件 @ 96615fe3
选项,说明,默认值
WITH_GPU,是否编译GPU支持。,是否寻找到cuda工具链
WITH_DOUBLE,是否使用双精度浮点数。,否
WITH_DSO,是否使用运行时动态加载cuda动态库，而非静态加载cuda动态库。,是
WITH_AVX,是否编译含有AVX指令集的PaddlePaddle二进制,是
WITH_PYTHON,是否内嵌python解释器。可以方便嵌入式工作。,是
WITH_STYLE_CHECK,是否编译时进行代码风格检查,是
WITH_RDMA,是否开启RDMA支持,否
WITH_GLOG,是否使用GLOG，如果不使用则会使用一个简化版的日志实现。可以方便嵌入式工作。,取决于是否寻找到GLOG
WITH_GFLAGS,是否使用GFLAGS，如果不使用则会使用一个简化版的命令行参数解析。可以方便嵌入式工作。,取决于是否寻找到GFLAGS
WITH_TIMER,是否开启计时功能开启计时功能会导致运行略慢，打印的日志变多。但是方便调试和benchmark,否
WITH_TESTING,是否开启单元测试,取决于是否寻找到gtest
WITH_DOC,是否编译英文文档,否
WITH_DOC_CN,是否编译中文文档,否
WITH_SWIG_PY,是否编译python的swig接口，python的swig接口可以方便进行预测和定制化训练,取决于是否找到swig
选项,说明,默认值
WITH_GPU,是否支持GPU。,取决于是否寻找到CUDA工具链
WITH_DOUBLE,是否使用双精度浮点数。,否
WITH_DSO,是否运行时动态加载CUDA动态库，而非静态加载CUDA动态库。,是
WITH_AVX,是否编译含有AVX指令集的PaddlePaddle二进制文件,是
WITH_PYTHON,是否内嵌PYTHON解释器。方便今后的嵌入式移植工作。,是
WITH_STYLE_CHECK,是否编译时进行代码风格检查,是
WITH_RDMA,是否开启RDMA,否
WITH_GLOG,是否开启GLOG。如果不开启，则会使用一个简化版的日志，同时方便今后的嵌入式移植工作。,取决于是否寻找到GLOG
WITH_GFLAGS,是否使用GFLAGS。如果不开启，则会使用一个简化版的命令行参数解析器，同时方便今后的嵌入式移植工作。,取决于是否寻找到GFLAGS
WITH_TIMER,是否开启计时功能。如果开启会导致运行略慢，打印的日志变多，但是方便调试和测Benchmark,否
WITH_TESTING,是否开启单元测试,取决于是否寻找到GTEST
WITH_DOC,是否编译中英文文档,否
WITH_SWIG_PY,是否编译PYTHON的SWIG接口，该接口可用于预测和定制化训练,取决于是否寻找到SWIG
\ No newline at end of file
doc_cn/build_and_install/cmake/compile_options.rst
浏览文件 @ 96615fe3
设置PaddlePaddle的编译选项
==========================
PaddlePaddle的编译选项可以在调用cmake的时候设置。cmake是一个跨平台的编译脚本，调用
cmake可以将cmake项目文件，生成各个平台的makefile。详细的cmake使用方法可以参考
`cmake的官方文档 <https://cmake.org/cmake-tutorial>`_ 。
PaddlePaddle的编译选项是可以控制PaddlePaddle生成CPU/GPU版本二进制，链接何种blas等等。所有的
编译选项列表如下
PaddlePaddle的编译选项
----------------------
bool型的编译选项
++++++++++++++++
设置下列编译选项时，可以在cmake的命令行设置。使用 -D命令即可。例如
:code:`cmake -D WITH_GPU=OFF`
.. csv-table:: PaddlePaddle的bool型编译选项
:widths: 1, 7, 2
:file: compile_options.csv
blas相关的编译选项
++++++++++++++++++
PaddlePaddle可以使用 `MKL <https://software.intel.com/en-us/intel-mkl>`_ ，
`Atlas <http://math-atlas.sourceforge.net/>`_ ,
`OpenBlas <http://www.openblas.net/>`_ 和
`refference Blas <http://www.netlib.org/blas/>`_ ，任意一种cblas实现。
通过编译时指定路径来实现引用各种blas。
cmake编译时会首先在系统路径(/usr/lib\:/usr/local/lib)中寻找这些blas的实现。同时
也会读取相关路径变量来进行搜索。路径变量为\:
.. csv-table:: PaddlePaddle的cblas编译选项
:widths: 1, 9
:header: "编译选项", "描述"
:file: cblas_settings.csv
这些变量均可以使用 -D命令指定。例如 :code:`cmake -D MKL_ROOT=/opt/mkl/`。这些变
量也可以通过调用cmake命令前通过环境变量指定。例如
.. code-block:: bash
export MKL_ROOT=/opt/mkl
cmake
需要注意的是，这些变量只在第一次cmake的时候有效。如果在第一次cmake之后想要重新设
置这些变量，推荐清理( :code:`rm -rf` )掉编译目录后，再指定。
cuda/cudnn相关的编译选项
++++++++++++++++++++++++
PaddlePaddle可以使用 cudnn v2之后的任何一个cudnn版本来编译运行。但需要注意的是编译和
运行使用的cudnn尽量是同一个版本。推荐使用最新版本的cudnn v5.1。
在cmake配置时可以使用 :code:`CUDNN_ROOT` 来配置CUDNN的安装路径。使用的命令也是
-D，例如 :code:`cmake -D CUDNN_ROOT=/opt/cudnnv5` 。
需要注意的是，这些变量只在第一次cmake的时候有效。如果在第一次cmake之后想要重新设
置这些变量，推荐清理( :code:`rm -rf` )掉编译目录后，再指定。
PaddlePaddle的编译选项
======================
PaddlePaddle的编译选项，包括生成CPU/GPU二进制文件、链接何种BLAS库等。用户可在调用cmake的时候设置它们，详细的cmake使用方法可以参考 `官方文档 <https://cmake.org/cmake-tutorial>`_ 。
Bool型的编译选项
----------------
用户可在cmake的命令行中，通过使用 ``-D`` 命令设置该类编译选项，例如
.. code-block:: bash
cmake .. -DWITH_GPU=OFF
.. csv-table:: Bool型的编译选项
:widths: 1, 7, 2
:file: compile_options.csv
BLAS/CUDA/Cudnn的编译选项
--------------------------
BLAS
+++++
PaddlePaddle支持以下任意一种BLAS库：`MKL <https://software.intel.com/en-us/intel-mkl>`_ ，`ATLAS <http://math-atlas.sourceforge.net/>`_ ，`OpenBlAS <http://www.openblas.net/>`_ 和 `REFERENCE BLAS <http://www.netlib.org/blas/>`_ 。
.. csv-table:: BLAS路径相关的编译选项
:widths: 1, 2, 7
:file: cblas_settings.csv
CUDA/Cudnn
+++++++++++
PaddlePaddle可以使用cudnn v2之后的任何一个版本来编译运行，但尽量请保持编译和运行使用的cudnn是同一个版本。 我们推荐使用最新版本的cudnn v5.1。
编译选项的设置
++++++++++++++
PaddePaddle通过编译时指定路径来实现引用各种BLAS/CUDA/Cudnn库。cmake编译时，首先在系统路径(/usr/lib\:/usr/local/lib)中搜索这几个库，同时也会读取相关路径变量来进行搜索。 通过使用 ``-D`` 命令可以设置，例如
.. code-block:: bash
cmake .. -DMKL_ROOT=/opt/mkl/ -DCUDNN_ROOT=/opt/cudnnv5
注意：这几个编译选项的设置，只在第一次cmake的时候有效。如果之后想要重新设置，推荐清理整个编译目录（``rm -rf``）后，再指定。
\ No newline at end of file
doc_cn/build_and_install/install/ubuntu_install.rst
浏览文件 @ 96615fe3
......@@ -11,7 +11,7 @@ PaddlePaddle的ubuntu安装包分为四个版本，他们是 cpu、gpu、cpu-noa
.. code-block:: shell
gdebi paddle-*-cpu.deb
gdebi paddle-*-cpu*.deb
如果 :code:`gdebi` 没有安装,则需要使用 :code:`sudo apt-get install gdebi`, 来安装 :code:`gdebi` 。
......@@ -20,7 +20,7 @@ PaddlePaddle的ubuntu安装包分为四个版本，他们是 cpu、gpu、cpu-noa
.. code-block:: shell
dpkg -i paddle-*-cpu.deb
dpkg -i paddle-*-cpu*.deb
apt-get install -f
在 :code:`dpkg -i` 的时候如果报一些依赖未找到的错误是正常的，
......
doc_cn/build_and_install/paddle_on_kubernetes.md 0 → 100644
浏览文件 @ 96615fe3
# Paddle On Kubernetes：单机训练
在这篇文档里，我们介绍如何在 Kubernetes 集群上启动一个单机使用CPU的Paddle训练作业。在下一篇中，我们将介绍如何启动分布式训练作业。
## 制作Docker镜像
在一个功能齐全的Kubernetes机群里，通常我们会安装Ceph等分布式文件系统来存储训练数据。这样的话，一个分布式Paddle训练任务中的每个进程都可以从Ceph读取数据。在这个例子里，我们只演示一个单机作业，所以可以简化对环境的要求，把训练数据直接放在
Paddle的Docker image里。为此，我们需要制作一个包含训练数据的Paddle镜像。
Paddle 的 [Quick Start Tutorial](http://www.paddlepaddle.org/doc/demo/quick_start/index_en.html)
里介绍了用Paddle源码中的脚本下载训练数据的过程。
而 `paddledev/paddle:cpu-demo-latest` 镜像里有 Paddle 源码与demo，（ 请注意，默认的
Paddle镜像 `paddledev/paddle:cpu-latest` 是不包括源码的, Paddle的各版本镜像可以参考 [Docker installation guide](http://www.paddlepaddle.org/doc/build/docker_install.html) ），所以我们使用这个镜像来下载训练数据到Docker container中，然后把这个包含了训练数据的container保存为一个新的镜像。
### 运行容器
```
$ docker run --name quick_start_data -it paddledev/paddle:cpu-demo-latest
```
### 下载数据
进入容器`/root/paddle/demo/quick_start/data`目录，使用`get_data.sh`下载数据
```
$ root@fbd1f2bb71f4:~/paddle/demo/quick_start/data# ./get_data.sh
Downloading Amazon Electronics reviews data...
--2016-10-31 01:33:43-- http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Electronics_5.json.gz
Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80
Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 495854086 (473M) [application/x-gzip]
Saving to: 'reviews_Electronics_5.json.gz'
10% [=======> ] 874,279 64.7KB/s eta 2h 13m
```
### 修改启动脚本
下载完数据后，修改`/root/paddle/demo/quick_start/train.sh`文件，内容如下（增加了一条cd命令）
```
set -e
cd /root/paddle/demo/quick_start
cfg=trainer_config.lr.py
#cfg=trainer_config.emb.py
#cfg=trainer_config.cnn.py
#cfg=trainer_config.lstm.py
#cfg=trainer_config.bidi-lstm.py
#cfg=trainer_config.db-lstm.py
paddle train \
--config=$cfg \
--save_dir=./output \
--trainer_count=4 \
--log_period=20 \
--num_passes=15 \
--use_gpu=false \
--show_parameter_stats_period=100 \
--test_all_data_in_one_period=1 \
2>&1 | tee 'train.log'
```
### 提交镜像
修改启动脚本后，退出容器，使用`docker commit`命令创建新镜像。
```
$ docker commit quick_start_data mypaddle/paddle:quickstart
```
## 使用 Kubernetes 进行训练
>针对任务运行完成后容器自动退出的场景，Kubernetes有Job类型的资源来支持。下文就是用Job类型的资源来进行训练。
### 编写yaml文件
在训练时，输出结果可能会随着容器的消耗而被删除，需要在创建容器前挂载卷以便我们保存训练结果。使用我们之前构造的镜像，可以创建一个 [Kubernetes Job](http://kubernetes.io/docs/user-guide/jobs/#what-is-a-job)，简单的yaml文件如下：
```
apiVersion: batch/v1
kind: Job
metadata:
name: quickstart
spec:
parallelism: 1
completions: 1
template:
metadata:
name: quickstart
spec:
volumes:
- name: output
hostPath:
path: /home/work/paddle_output
containers:
- name: pi
image: mypaddle/paddle:quickstart
command: ["bin/bash", "-c", "/root/paddle/demo/quick_start/train.sh"]
volumeMounts:
- name: output
mountPath: /root/paddle/demo/quick_start/output
restartPolicy: Never
```
### 创建Paddle Job
使用上文创建的yaml文件创建Kubernetes Job，命令为：
```
$ kubectl create -f paddle.yaml
```
查看job的详细情况：
```
$ kubectl get job
NAME DESIRED SUCCESSFUL AGE
quickstart 1 0 58s
$ kubectl describe job quickstart
Name: quickstart
Namespace: default
Image(s): registry.baidu.com/public/paddle:cpu-demo-latest
Selector: controller-uid=f120da72-9f18-11e6-b363-448a5b355b84
Parallelism: 1
Completions: 1
Start Time: Mon, 31 Oct 2016 11:20:16 +0800
Labels: controller-uid=f120da72-9f18-11e6-b363-448a5b355b84,job-name=quickstart
Pods Statuses: 0 Running / 1 Succeeded / 0 Failed
Volumes:
output:
Type: HostPath (bare host directory volume)
Path: /home/work/paddle_output
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: quickstart-fa0wx
```
### 查看训练结果
根据Job对应的Pod信息，可以查看此Pod运行的宿主机。
```
kubectl describe pod quickstart-fa0wx
Name: quickstart-fa0wx
Namespace: default
Node: paddle-demo-let02/10.206.202.44
Start Time: Mon, 31 Oct 2016 11:20:17 +0800
Labels: controller-uid=f120da72-9f18-11e6-b363-448a5b355b84,job-name=quickstart
Status: Succeeded
IP: 10.0.0.9
Controllers: Job/quickstart
Containers:
quickstart:
Container ID: docker://b8561f5c79193550d64fa47418a9e67ebdd71546186e840f88de5026b8097465
Image: registry.baidu.com/public/paddle:cpu-demo-latest
Image ID: docker://18e457ce3d362ff5f3febf8e7f85ffec852f70f3b629add10aed84f930a68750
Port:
Command:
bin/bash
-c
/root/paddle/demo/quick_start/train.sh
QoS Tier:
cpu: BestEffort
memory: BestEffort
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 31 Oct 2016 11:20:20 +0800
Finished: Mon, 31 Oct 2016 11:21:46 +0800
Ready: False
Restart Count: 0
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
output:
Type: HostPath (bare host directory volume)
Path: /home/work/paddle_output
```
我们还可以登录到宿主机上查看训练结果。
```
[root@paddle-demo-let02 paddle_output]# ll
total 60
drwxr-xr-x 2 root root 4096 Oct 31 11:20 pass-00000
drwxr-xr-x 2 root root 4096 Oct 31 11:20 pass-00001
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00002
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00003
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00004
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00005
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00006
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00007
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00008
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00009
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00010
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00011
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00012
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00013
drwxr-xr-x 2 root root 4096 Oct 31 11:21 pass-00014
```
doc_cn/cluster/k8s/Dockerfile 0 → 100644
浏览文件 @ 96615fe3
FROM paddledev/paddle:cpu-latest
MAINTAINER zjsxzong89@gmail.com
COPY start.sh /root/
COPY start_paddle.py /root/
CMD ["bash"," -c","/root/start.sh"]
\ No newline at end of file
doc_cn/cluster/k8s/distributed_training_on_kubernetes.md 0 → 100644
浏览文件 @ 96615fe3
# PaddlePaddle on Kubernetes：分布式训练
前一篇文章介绍了如何在Kubernetes集群上启动一个单机PaddlePaddle训练作业 (Job)。在这篇文章里，我们介绍如何在Kubernetes集群上进行分布式PaddlePaddle训练作业。关于PaddlePaddle的分布式训练，文章 [Cluster Training](https://github.com/baidu/Paddle/blob/develop/doc/cluster/opensource/cluster_train.md)介绍了一种通过SSH远程分发任务，进行分布式训练的方法，与此不同的是，本文将介绍在Kubernetes容器管理平台上快速构建PaddlePaddle容器集群，进行分布式训练的方案。
## Kubernetes 基本概念
[*Kubernetes*](http://kubernetes.io/)是Google开源的容器集群管理系统，其提供应用部署、维护、 扩展机制等功能，利用Kubernetes能方便地管理跨机器运行容器化的应用。Kubernetes可以在物理机或虚拟机上运行，且支持部署到[AWS](http://kubernetes.io/docs/getting-started-guides/aws)，[Azure](http://kubernetes.io/docs/getting-started-guides/azure/)，[GCE](http://kubernetes.io/docs/getting-started-guides/gce)等多种公有云环境。介绍分布式训练之前，需要对[Kubernetes](http://kubernetes.io/)有一个基本的认识，下面先简要介绍一下本文用到的几个Kubernetes概念。
- [*Node*](http://kubernetes.io/docs/admin/node/) 表示一个Kubernetes集群中的一个工作节点，这个节点可以是物理机或者虚拟机，Kubernetes集群就是由node节点与master节点组成的。
- [*Pod*](http://kubernetes.io/docs/user-guide/pods/) 是一组(一个或多个)容器，pod是Kubernetes的最小调度单元，一个pod中的所有容器会被调度到同一个node上。Pod中的容器共享NET，PID，IPC，UTS等Linux namespace。由于容器之间共享NET namespace，所以它们使用同一个IP地址，可以通过*localhost*互相通信。不同pod之间可以通过IP地址访问。
- [*Job*](http://kubernetes.io/docs/user-guide/jobs/) 是Kubernetes上运行的作业，一次作业称为一个job，通常每个job包括一个或者多个pods。
- [*Volume*](http://kubernetes.io/docs/user-guide/volumes/) 存储卷，是pod内的容器都可以访问的共享目录，也是容器与node之间共享文件的方式，因为容器内的文件都是暂时存在的，当容器因为各种原因被销毁时，其内部的文件也会随之消失。通过volume，就可以将这些文件持久化存储。Kubernetes支持多种volume，例如hostPath(宿主机目录)，gcePersistentDisk，awsElasticBlockStore等。
- [*Namespaces*](http://kubernetes.io/docs/user-guide/volumes/) 命名空间，在kubernetes中创建的所有资源对象(例如上文的pod，job)等都属于一个命名空间，在同一个命名空间中，资源对象的名字是唯一的，不同空间的资源名可以重复，命名空间主要为了对象进行逻辑上的分组便于管理。本文只使用了默认命名空间。
## 整体方案
### 部署Kubernetes集群
首先，我们需要拥有一个Kubernetes集群，在这个集群中所有node与pod都可以互相通信。关于Kubernetes集群搭建，可以参考[官方文档](http://kubernetes.io/docs/getting-started-guides/kubeadm/)，在以后的文章中我们也会介绍AWS上搭建的方案。本文假设大家能找到几台物理机，并且可以按照官方文档在上面部署Kubernetes。在本文的环境中，Kubernetes集群中所有node都挂载了一个[MFS](http://moosefs.org/)（Moose filesystem，一种分布式文件系统）共享目录，我们通过这个目录来存放训练文件与最终输出的模型。关于MFS的安装部署，可以参考[MooseFS documentation](https://moosefs.com/documentation.html)。在训练之前，用户将配置与训练数据切分好放在MFS目录中，训练时，程序从此目录拷贝文件到容器内进行训练，将结果保存到此目录里。整体的结构图如下：
![paddle on kubernetes结构图](k8s-paddle-arch.png)
上图描述了一个3节点的分布式训练场景，Kubernetes集群的每个node上都挂载了一个MFS目录，这个目录可以通过volume的形式挂载到容器中。Kubernetes为这次训练创建了3个pod并且调度到了3个node上运行，每个pod包含一个PaddlePaddle容器。在容器创建后，会启动pserver与trainer进程，读取volume中的数据进行这次分布式训练。
### 使用 Job
我们使用Kubernetes中的job这个概念来代表一次分布式训练。Job表示一次性作业，在作业完成后，Kubernetes会销毁job产生的容器并且释放相关资源。
在Kubernetes中，可以通过编写一个YAML文件，来描述这个job，在这个文件中，主要包含了一些配置信息，例如PaddlePaddle的节点个数，`paddle pserver`开放的端口个数与端口号，使用的网卡设备等，这些信息通过环境变量的形式传递给容器内的程序使用。
在一次分布式训练中，用户确定好本次训练需要的PaddlePaddle节点个数，将切分好的训练数据与配置文件上传到MFS共享目录中。然后编写这次训练的job YAML文件，提交给Kubernetes集群创建并开始作业。
### 创建PaddlePaddle节点
当Kubernetes master收到请求，解析完YAML文件后，会创建出多个pod(个数为PaddlePaddle节点数)，Kubernetes会把这些pod调度到集群的node上运行。一个pod就代表一个PaddlePaddle节点，当pod被成功分配到一台物理/虚拟机上后，Kubernetes会启动pod内的容器，这个容器会根据YAML文件中的环境变量，启动`paddle pserver`与`paddle train`进程。
### 启动训练
在容器启动后，会通过脚本来启动这次分布式训练，我们知道`paddle train`进程启动时需要知道其他节点的IP地址以及本节点的trainer_id，由于PaddlePaddle本身不提供类似服务发现的功能，所以在本文的启动脚本中，每个节点会根据job name向Kubernetes apiserver查询这个job对应的所有pod信息(Kubernetes默认会在每个容器的环境变量中写入apiserver的地址)。
根据这些pod信息，就可以通过某种方式，为每个pod分配一个唯一的trainer_id。本文把所有pod的IP地址进行排序，将顺序作为每个PaddlePaddle节点的trainer_id。启动脚本的工作流程大致如下：
1. 查询Kubernetes apiserver获取pod信息，根据IP分配trainer_id
1. 从MFS共享目录中拷贝训练文件到容器内
1. 根据环境变量，解析出`paddle pserver`与`paddle train`的启动参数，启动进程
1. 训练时，PaddlePaddle会自动将结果保存在trainer_id为0的节点上，将输出路径设置为MFS目录，保存输出的文件
## 搭建过程
根据前文的描述，要在已有的Kubernetes集群上进行PaddlePaddle的分布式训练，主要分为以下几个步骤：
1. 制作PaddlePaddle镜像
1. 将训练文件与切分好的数据上传到共享存储
1. 编写本次训练的YAML文件，创建一个Kubernetes job
1. 训练结束后查看输出结果
下面就根据这几个步骤分别介绍。
### 制作镜像
PaddlePaddle镜像需要提供`paddle pserver`与`paddle train`进程的运行环境，用这个镜像创建的容器需要有以下两个功能：
- 拷贝训练文件到容器内
- 生成`paddle pserver`与`paddle train`进程的启动参数，并且启动训练
因为官方镜像 `paddledev/paddle:cpu-latest` 内已经包含PaddlePaddle的执行程序但是还没上述功能，所以我们可以在这个基础上，添加启动脚本，制作新镜像来完成以上的工作。镜像的*Dockerfile*如下：
```Dockerfile
FROM paddledev/paddle:cpu-latest
MAINTAINER zjsxzong89@gmail.com
COPY start.sh /root/
COPY start_paddle.py /root/
CMD ["bash"," -c","/root/start.sh"]
```
[`start.sh`](start.sh)文件拷贝训练文件到容器内，然后执行[`start_paddle.py`](start_paddle.py)脚本启动训练，前文提到的获取其他节点IP地址，分配`trainer_id`等都在`start_paddle.py`脚本中完成。
`start_paddle.py`脚本开始时，会先进行参数的初始化与解析。
```python
parser = argparse.ArgumentParser(prog="start_paddle.py",
description='simple tool for k8s')
args, train_args_list = parser.parse_known_args()
train_args = refine_unknown_args(train_args_list)
train_args_dict = dict(zip(train_args[:-1:2], train_args[1::2]))
podlist = getPodList()
```
然后通过函数`getPodList()`访问Kubernetes的接口来查询此job对应的所有pod信息。当所有pod都处于running状态（容器运行都运行）时，再通过函数`getIdMap(podlist)`获取trainer_id。
```python
podlist = getPodList()
# need to wait until all pods are running
while not isPodAllRunning(podlist):
time.sleep(10)
podlist = getPodList()
idMap = getIdMap(podlist)
```
在函数`getIdMap(podlist)`内部，我们通过读取`podlist`中每个pod的IP地址，将IP排序生成的序号作为trainer_id。
```python
def getIdMap(podlist):
'''
generate tainer_id by ip
'''
ips = []
for pod in podlist["items"]:
ips.append(pod["status"]["podIP"])
ips.sort()
idMap = {}
for i in range(len(ips)):
idMap[ips[i]] = i
return idMap
```
在得到`idMap`后，通过函数`startPaddle(idMap, train_args_dict)`构造`paddle pserver`与`paddle train`的启动参数并执行进程。
在函数`startPaddle`中，最主要的工作就是解析出`paddle pserver`与`paddle train`的启动参数。例如`paddle train`参数的解析，解析环境变量得到`PADDLE_NIC`，`PADDLE_PORT`，`PADDLE_PORTS_NUM`等参数，然后通过自身的IP地址在`idMap`中获取`trainerId`。
```python
program = 'paddle train'
args = " --nics=" + PADDLE_NIC
args += " --port=" + str(PADDLE_PORT)
args += " --ports_num=" + str(PADDLE_PORTS_NUM)
args += " --comment=" + "paddle_process_by_paddle"
ip_string = ""
for ip in idMap.keys():
ip_string += (ip + ",")
ip_string = ip_string.rstrip(",")
args += " --pservers=" + ip_string
args_ext = ""
for key, value in train_args_dict.items():
args_ext += (' --' + key + '=' + value)
localIP = socket.gethostbyname(socket.gethostname())
trainerId = idMap[localIP]
args += " " + args_ext + " --trainer_id=" + \
str(trainerId) + " --save_dir=" + JOB_PATH_OUTPUT
```
使用 `docker build` 构建镜像：
```bash
docker build -t your_repo/paddle:mypaddle .
```
然后将构建成功的镜像上传到镜像仓库。
```bash
docker push your_repo/paddle:mypaddle
```
### 上传训练文件
本文使用PaddlePaddle官方的[recommendation demo](http://www.paddlepaddle.org/doc/demo/index.html#recommendation)作为这次训练的内容，我们将训练文件与数据放在一个job name命名的目录中，上传到MFS共享存储。完成后MFS上的文件内容大致如下：
```bash
[root@paddle-kubernetes-node0 mfs]# tree -d
.
└── paddle-cluster-job
├── data
│   ├── 0
│   │
│   ├── 1
│   │
│   └── 2
├── output
└── recommendation
```
目录中paddle-cluster-job是本次训练对应的job name，本次训练要求有3个PaddlePaddle节点，在paddle-cluster-job/data目录中存放切分好的数据，文件夹0，1，2分别代表3个节点的trainer_id。recommendation文件夹内存放训练文件，output文件夹存放训练结果与日志。
### 创建Job
Kubernetes可以通过YAML文件来创建相关对象，然后可以使用命令行工具创建job。
Job YAML文件描述了这次训练使用的Docker镜像，需要启动的节点个数以及 `paddle pserver`与 `paddle train`进程启动的必要参数，也描述了容器需要使用的存储卷挂载的情况。YAML文件中各个字段的具体含义，可以查看[Kubernetes Job API](http://kubernetes.io/docs/api-reference/batch/v1/definitions/#_v1_job)。例如，本次训练的YAML文件可以写成：
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: paddle-cluster-job
spec:
parallelism: 3
completions: 3
template:
metadata:
name: paddle-cluster-job
spec:
volumes:
- name: jobpath
hostPath:
path: /home/work/mfs
containers:
- name: trainer
image: your_repo/paddle:mypaddle
command: ["bin/bash", "-c", "/root/start.sh"]
env:
- name: JOB_NAME
value: paddle-cluster-job
- name: JOB_PATH
value: /home/jobpath
- name: JOB_NAMESPACE
value: default
- name: TRAIN_CONFIG_DIR
value: recommendation
- name: CONF_PADDLE_NIC
value: eth0
- name: CONF_PADDLE_PORT
value: "7164"
- name: CONF_PADDLE_PORTS_NUM
value: "2"
- name: CONF_PADDLE_PORTS_NUM_SPARSE
value: "2"
- name: CONF_PADDLE_GRADIENT_NUM
value: "3"
volumeMounts:
- name: jobpath
mountPath: /home/jobpath
restartPolicy: Never
```
文件中，`metadata`下的`name`表示这个job的名字。`parallelism，completions`字段表示这个job会同时开启3个PaddlePaddle节点，成功训练且退出的pod数目为3时，这个job才算成功结束。然后申明一个存储卷`jobpath`，代表宿主机目录`/home/work/mfs`，在对容器的描述`containers`字段中，将此目录挂载为容器的`/home/jobpath`目录，这样容器的`/home/jobpath`目录就成为了共享存储，放在这个目录里的文件其实是保存到了MFS上。
`env`字段表示容器的环境变量，我们将`paddle`运行的一些参数通过这种方式传递到容器内。
`JOB_PATH`表示共享存储挂载的路径，`JOB_NAME`表示job名字，`TRAIN_CONFIG_DIR`表示本次训练文件所在目录，这三个变量组合就可以找到本次训练需要的文件路径。
`CONF_PADDLE_NIC`表示`paddle pserver`进程需要的`--nics`参数，即网卡名
`CONF_PADDLE_PORT`表示`paddle pserver`的`--port`参数，`CONF_PADDLE_PORTS_NUM`则表示稠密更新的端口数量，也就是`--ports_num`参数。
`CONF_PADDLE_PORTS_NUM_SPARSE`表示稀疏更新的端口数量，也就是`--ports_num_for_sparse`参数。
`CONF_PADDLE_GRADIENT_NUM`表示训练节点数量，即`--num_gradient_servers`参数
编写完YAML文件后，可以使用Kubernetes的命令行工具创建job。
```bash
kubectl create -f job.yaml
```
创建成功后，Kubernetes就会创建3个pod作为PaddlePaddle节点然后拉取镜像，启动容器开始训练。
### 查看输出
在训练过程中，可以在共享存储上查看输出的日志和模型，例如output目录下就存放了输出结果。注意node_0，node_1，node_2这几个目录表示PaddlePaddle节点与trainer_id，并不是Kubernetes中的node概念。
```bash
[root@paddle-kubernetes-node0 output]# tree -d
.
├── node_0
│   ├── server.log
│   └── train.log
├── node_1
│   ├── server.log
│   └── train.log
├── node_2
......
├── pass-00002
│   ├── done
│   ├── ___embedding_0__.w0
│   ├── ___embedding_1__.w0
......
```
我们可以通过日志查看容器训练的情况，例如：
```bash
[root@paddle-kubernetes-node0 node_0]# cat train.log
I1116 09:10:17.123121 50 Util.cpp:155] commandline:
/usr/local/bin/../opt/paddle/bin/paddle_trainer
--nics=eth0 --port=7164
--ports_num=2 --comment=paddle_process_by_paddle
--pservers=192.168.129.66,192.168.223.143,192.168.129.71
--ports_num_for_sparse=2 --config=./trainer_config.py
--trainer_count=4 --num_passes=10 --use_gpu=0
--log_period=50 --dot_period=10 --saving_period=1
--local=0 --trainer_id=0
--save_dir=/home/jobpath/paddle-cluster-job/output
I1116 09:10:17.123440 50 Util.cpp:130] Calling runInitFunctions
I1116 09:10:17.123764 50 Util.cpp:143] Call runInitFunctions done.
[WARNING 2016-11-16 09:10:17,227 default_decorators.py:40] please use keyword arguments in paddle config.
[INFO 2016-11-16 09:10:17,239 networks.py:1282] The input order is [movie_id, title, genres, user_id, gender, age, occupation, rating]
[INFO 2016-11-16 09:10:17,239 networks.py:1289] The output order is [__regression_cost_0__]
I1116 09:10:17.392917 50 Trainer.cpp:170] trainer mode: Normal
I1116 09:10:17.613910 50 PyDataProvider2.cpp:257] loading dataprovider dataprovider::process
I1116 09:10:17.680917 50 PyDataProvider2.cpp:257] loading dataprovider dataprovider::process
I1116 09:10:17.681543 50 GradientMachine.cpp:134] Initing parameters..
I1116 09:10:18.012390 50 GradientMachine.cpp:141] Init parameters done.
I1116 09:10:18.018641 50 ParameterClient2.cpp:122] pserver 0 192.168.129.66:7164
I1116 09:10:18.018950 50 ParameterClient2.cpp:122] pserver 1 192.168.129.66:7165
I1116 09:10:18.019069 50 ParameterClient2.cpp:122] pserver 2 192.168.223.143:7164
I1116 09:10:18.019492 50 ParameterClient2.cpp:122] pserver 3 192.168.223.143:7165
I1116 09:10:18.019716 50 ParameterClient2.cpp:122] pserver 4 192.168.129.71:7164
I1116 09:10:18.019836 50 ParameterClient2.cpp:122] pserver 5 192.168.129.71:7165
```
\ No newline at end of file
doc_cn/cluster/k8s/job.yaml 0 → 100644
浏览文件 @ 96615fe3
apiVersion: batch/v1
kind: Job
metadata:
name: paddle-cluster-job
spec:
parallelism: 3
completions: 3
template:
metadata:
name: paddle-cluster-job
spec:
volumes:
- name: jobpath
hostPath:
path: /home/work/paddle_output
containers:
- name: trainer
image: registry.baidu.com/public/paddle:mypaddle
command: ["bin/bash", "-c", "/root/start.sh"]
env:
- name: JOB_NAME
value: paddle-cluster-job
- name: JOB_PATH
value: /home/jobpath
- name: JOB_NAMESPACE
value: default
- name: TRAIN_CONFIG_DIR
value: recommendation
- name: CONF_PADDLE_NIC
value: eth0
- name: CONF_PADDLE_PORT
value: "7164"
- name: CONF_PADDLE_PORTS_NUM
value: "2"
- name: CONF_PADDLE_PORTS_NUM_SPARSE
value: "2"
- name: CONF_PADDLE_GRADIENT_NUM
value: "3"
volumeMounts:
- name: jobpath
mountPath: /home/jobpath
restartPolicy: Never
\ No newline at end of file
doc_cn/cluster/k8s/start.sh 0 → 100755
浏览文件 @ 96615fe3
#!/bin/sh
set -eu
jobconfig=${JOB_PATH}"/"${JOB_NAME}"/"${TRAIN_CONFIG_DIR}
cd /root
cp -rf $jobconfig .
cd $TRAIN_CONFIG_DIR
python /root/start_paddle.py \
--dot_period=10 \
--ports_num_for_sparse=$CONF_PADDLE_PORTS_NUM \
--log_period=50 \
--num_passes=10 \
--trainer_count=4 \
--saving_period=1 \
--local=0 \
--config=./trainer_config.py \
--use_gpu=0
doc_cn/cluster/k8s/start_paddle.py 0 → 100755
浏览文件 @ 96615fe3
#!/usr/bin/python
# Copyright (c) 2016 Baidu, Inc. All Rights Reserved
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import requests
import time
import socket
import os
import argparse
# configuration for cluster
API = "/api/v1/namespaces/"
JOBSELECTOR = "labelSelector=job-name="
JOB_PATH = os.getenv("JOB_PATH") + "/" + os.getenv("JOB_NAME")
JOB_PATH_DATA = JOB_PATH + "/data"
JOB_PATH_OUTPUT = JOB_PATH + "/output"
JOBNAME = os.getenv("JOB_NAME")
NAMESPACE = os.getenv("JOB_NAMESPACE")
PADDLE_NIC = os.getenv("CONF_PADDLE_NIC")
PADDLE_PORT = os.getenv("CONF_PADDLE_PORT")
PADDLE_PORTS_NUM = os.getenv("CONF_PADDLE_PORTS_NUM")
PADDLE_PORTS_NUM_SPARSE = os.getenv("CONF_PADDLE_PORTS_NUM_SPARSE")
PADDLE_SERVER_NUM = os.getenv("CONF_PADDLE_GRADIENT_NUM")
def refine_unknown_args(cmd_args):
'''
refine unknown parameters to handle some special parameters
'''
new_args = []
for arg in cmd_args:
if arg.startswith("--") and arg.find("=") != -1:
equal_pos = arg.find("=") # find first = pos
arglist = list(arg)
arglist[equal_pos] = " "
arg = "".join(arglist)
arg = arg.lstrip("-")
new_args += arg.split(" ")
elif arg.startswith("--") and arg.find("=") == -1:
arg = arg.lstrip("-")
new_args.append(arg)
else:
new_args.append(arg)
return new_args
def isPodAllRunning(podlist):
'''
check all pod is running
'''
require = len(podlist["items"])
running = 0
for pod in podlist["items"]:
if pod["status"]["phase"] == "Running":
running += 1
if require == running:
return True
return False
def getPodList():
'''
get all container status of the job
'''
apiserver = "https://" + \
os.getenv("KUBERNETES_SERVICE_HOST") + ":" + \
os.getenv("KUBERNETES_SERVICE_PORT_HTTPS")
pod = API + NAMESPACE + "/pods?"
job = JOBNAME
return requests.get(apiserver + pod + JOBSELECTOR + job,
verify=False).json()
def getIdMap(podlist):
'''
generate tainer_id by ip
'''
ips = []
for pod in podlist["items"]:
ips.append(pod["status"]["podIP"])
ips.sort()
idMap = {}
for i in range(len(ips)):
idMap[ips[i]] = i
return idMap
def startPaddle(idMap={}, train_args_dict=None):
'''
start paddle pserver and trainer
'''
program = 'paddle train'
args = " --nics=" + PADDLE_NIC
args += " --port=" + str(PADDLE_PORT)
args += " --ports_num=" + str(PADDLE_PORTS_NUM)
args += " --comment=" + "paddle_process_by_paddle"
ip_string = ""
for ip in idMap.keys():
ip_string += (ip + ",")
ip_string = ip_string.rstrip(",")
args += " --pservers=" + ip_string
args_ext = ""
for key, value in train_args_dict.items():
args_ext += (' --' + key + '=' + value)
localIP = socket.gethostbyname(socket.gethostname())
trainerId = idMap[localIP]
args += " " + args_ext + " --trainer_id=" + \
str(trainerId) + " --save_dir=" + JOB_PATH_OUTPUT
logDir = JOB_PATH_OUTPUT + "/node_" + str(trainerId)
if not os.path.exists(JOB_PATH_OUTPUT):
os.makedirs(JOB_PATH_OUTPUT)
os.mkdir(logDir)
copyCommand = 'cp -rf ' + JOB_PATH_DATA + \
"/" + str(trainerId) + " ./data"
os.system(copyCommand)
startPserver = 'nohup paddle pserver' + \
" --port=" + str(PADDLE_PORT) + \
" --ports_num=" + str(PADDLE_PORTS_NUM) + \
" --ports_num_for_sparse=" + str(PADDLE_PORTS_NUM_SPARSE) + \
" --nics=" + PADDLE_NIC + \
" --comment=" + "paddle_process_by_paddle" + \
" --num_gradient_servers=" + str(PADDLE_SERVER_NUM) +\
" > " + logDir + "/server.log 2>&1 &"
print startPserver
os.system(startPserver)
# wait until pservers completely start
time.sleep(10)
startTrainer = program + args + " > " + \
logDir + "/train.log 2>&1 < /dev/null"
print startTrainer
os.system(startTrainer)
if __name__ == '__main__':
parser = argparse.ArgumentParser(prog="start_paddle.py",
description='simple tool for k8s')
args, train_args_list = parser.parse_known_args()
train_args = refine_unknown_args(train_args_list)
train_args_dict = dict(zip(train_args[:-1:2], train_args[1::2]))
podlist = getPodList()
# need to wait until all pods are running
while not isPodAllRunning(podlist):
time.sleep(10)
podlist = getPodList()
idMap = getIdMap(podlist)
startPaddle(idMap, train_args_dict)
doc_cn/howto/how_to_write_docs/index.rst
浏览文件 @ 96615fe3
......@@ -2,32 +2,19 @@
如何贡献/修改PaddlePaddle的文档
###############################
PaddlePaddle的文档使用 `cmake`_ 驱动 `sphinx`_ 生成。公有两个文档，:code:`doc` 和 :code:`doc_cn` 。这两者会在 `cmake`_ 中进行编译，生成后的文档会存储在服务器的 :code:`doc` 和 :code:`doc_cn` 两个目录下。
PaddlePaddle的文档包括英文文档 ``doc`` 和中文文档 ``doc_cn`` 两个部分。文档都是通过 `cmake`_ 驱动 `sphinx`_ 编译生成，生成后的文档分别存储在编译目录的 ``doc`` 和 ``doc_cn`` 两个子目录下。
下面分几个部分介绍一下PaddlePaddle文档的贡献方法。
如何书写PaddlePaddle的文档
==========================
TBD
如何构建PaddlePaddle的文档
==========================
构建PaddlePaddle文档，需要使用构建Paddle的全部环境。准备这个环境相对来说比较复杂，所以本文档提供两种方式构建PaddlePaddle的文档，即
* 使用Docker构建PaddlePaddle的文档
* 直接构建PaddlePaddle的文档。
并且，我们推荐使用Docker来构建PaddlePaddle的文档。
PaddlePaddle的文档构建有直接构建和基于Docker构建两种方式。构建PaddlePaddle文档需要准备的环境相对较复杂，所以我们推荐使用基于Docker来构建PaddlePaddle的文档。
使用Docker构建PaddlePaddle的文档
--------------------------------
使用Docker构建PaddlePaddle的文档，首先要求在系统里安装好Docker工具包。安装Docker请参考 `Docker的官网 <https://docs.docker.com/>`_ 。
安装好Docker之后可以使用源码目录下的脚本构建文档，即
使用Docker构建PaddlePaddle的文档，需要在系统里先安装好Docker工具包。Docker安装请参考 `Docker的官网 <https://docs.docker.com/>`_ 。安装好Docker之后可以使用源码目录下的脚本构建文档，即
.. code-block:: bash
......@@ -35,10 +22,10 @@ TBD
cd paddle/scripts/tools/build_docs
bash build_docs.sh
执行完这个脚本后，该目录下会生成两个目录，分别是\:
编译完成后，该目录下会生成如下两个子目录\:
* doc 目录，英文文档地址
* doc_cn 目录，中文文档地址
* doc 英文文档目录
* doc_cn 中文文档目录
打开浏览器访问对应目录下的index.html即可访问本地文档。
......@@ -52,6 +39,10 @@ TBD
TBD
如何书写PaddlePaddle的文档
==========================
TBD
如何更新www.paddlepaddle.org文档
================================
......
doc_cn/ui/cmd/index.rst
浏览文件 @ 96615fe3
此差异已折叠。
doc_cn/ui/cmd/paddle_pserver.rst
浏览文件 @ 96615fe3
此差异已折叠。
doc_cn/ui/cmd/paddle_train.rst
浏览文件 @ 96615fe3
此差异已折叠。
doc_cn/ui/cmd/paddle_version.rst
浏览文件 @ 96615fe3
此差异已折叠。
doc_cn/ui/index.rst
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/Arguments.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/ConfigParser.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/GradientMachine.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/Internal.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/Matrix.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/PaddleAPI.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/Parameter.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/ParameterOptimizer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/SequenceGenerator.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/Trainer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/Util.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/api/Vector.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/CMakeLists.txt
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_aggregate.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_base.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_cnn.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_cuda.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_cuda_cudnn.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_dso_loader.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_functions.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_gpu.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_lstm.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_matrix.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_sequence.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_sparse.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_time.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/include/hl_top_k.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/avx_mathfun.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_avx_functions.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_cpu_functions.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_cuda_cublas.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_cuda_cudnn.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_cuda_device.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_cudart_wrap.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_dso_loader.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_math.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/cuda/src/hl_time.cc
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/AddtoLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/AgentLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/CRFLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/CRFLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/CTCLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/CTCLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/CosSimLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/CostLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/CostLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/DataLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/DataLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/GruCompute.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/Layer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/Layer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/LstmCompute.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/LstmLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/LstmLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/MaxLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/MaxLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/MixedLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/NCELayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/NormLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/NormLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/Operator.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/Operator.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/PoolLayer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/PoolLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/Projection.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/TensorLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/layers/TransLayer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/tests/TestUtil.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/gserver/tests/TestUtil.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/Allocator.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/BaseMatrix.cu
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/BaseMatrix.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/CpuSparseMatrix.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/CpuSparseMatrix.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/ExecViaCpu.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/MathFunctions.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/MathFunctions.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/MathUtils.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/MathUtils.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/Matrix.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/Matrix.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/MatrixBitCode.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/MemoryHandle.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/MemoryHandle.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/PoolAllocator.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/PoolAllocator.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/SIMDFunctions.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/SIMDFunctions.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/SparseMatrix.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/SparseMatrix.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/SparseRowMatrix.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/SparseRowMatrix.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/Storage.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/Vector.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/Vector.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/tests/CMakeLists.txt
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/tests/test_GpuProfiler.cpp 0 → 100644
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/tests/test_matrix.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/math/tests/test_matrixUtil.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/Argument.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/Argument.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/AverageOptimizer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/Parameter.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/Parameter.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/Regularizer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/Regularizer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/parameter/Weight.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/BaseClient.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/BaseClient.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/LightNetwork.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/LightNetwork.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/ParameterClient2.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/ParameterClient2.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/ParameterServer2.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/ParameterServer2.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/ProtoServer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/ProtoServer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/RDMANetwork.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/SocketChannel.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/SocketChannel.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/pserver/test/SocketTest.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/scripts/docker/Dockerfile.m4
浏览文件 @ 96615fe3
此差异已折叠。
paddle/scripts/submit_local.sh.in
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/ParamUtil.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/ParamUtil.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/ParameterUpdater.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/ParameterUpdater.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/Tester.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/Tester.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/TesterConfig.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/Trainer.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/Trainer.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/TrainerInternal.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/TrainerInternal.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/TrainerMain.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/trainer/tests/picojson.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/BarrierStat.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/BarrierStat.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/ClassRegistrar.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/CommandLineParser.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/CommandLineParser.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/CompilerMacros.h 0 → 100644
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/CustomStackTrace.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/CustomStackTrace.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/DisableCopy.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Excepts.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Flags.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Flags.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/GlobalConstants.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/GlobalConstants.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Locks.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Logging.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Logging.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/PythonUtil.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/PythonUtil.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Queue.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Stat.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Stat.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/StringUtil.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Thread.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/ThreadLocal.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/ThreadLocal.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/TypeDefs.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Util.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Util.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Version.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/Version.h
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/arch/linux/Locks.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/arch/osx/Locks.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/tests/test_Logging.cpp
浏览文件 @ 96615fe3
此差异已折叠。
paddle/utils/tests/test_Thread.cpp
浏览文件 @ 96615fe3
此差异已折叠。
proto/ModelConfig.proto.m4
浏览文件 @ 96615fe3
此差异已折叠。
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或