Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
迷途归一
Paddle
提交
f648b3aa
P
Paddle
项目概览
迷途归一
/
Paddle
与 Fork 源项目一致
Fork自
PaddlePaddle / Paddle
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
P
Paddle
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
f648b3aa
编写于
2月 16, 2017
作者:
Y
Yu Yang
浏览文件
操作
浏览文件
下载
差异文件
Merge branch 'develop' of github.com:baidu/Paddle into feature/new_api
上级
8dc4c053
6ab6c356
变更
2
隐藏空白更改
内联
并排
Showing
2 changed file
with
166 addition
and
170 deletion
+166
-170
doc/design/api.md
doc/design/api.md
+155
-170
python/paddle/v2/event.py
python/paddle/v2/event.py
+11
-0
未找到文件。
doc/design/api.md
浏览文件 @
f648b3aa
...
@@ -2,140 +2,148 @@
...
@@ -2,140 +2,148 @@
## Ingredients
## Ingredients
As the first step of our design, we list important concepts in deep
As our design principle is starting from the essence: how could we
learning and try to figure their relationship, as shown below:
allow users to express and solve their problems at neural networks.
Some essential concepts that our API have to provide include:
```
1.
A
*topology*
is an expression of
*layers*
.
Model = {topology, parameters}
Evaluator = {Model*, activations}
1.
A layer could be any kind of computation, including
*cost*
.
- forward
- test(cost, ...)
GradientMachine = {Evaluator*, gradients}
1.
Some layers have parameters, some don't. Most costs don't have
- backward
parameters.
Optimizer = {GradientMachine*}
1.
In some topologies, layers share parameters. For
- train(cost, ...)
example,
- update
[
the network for training a ranking model
](
https://github.com/PaddlePaddle/Paddle/issues/1311#issuecomment-279121850
)
.
- checkpoint
```
where the pair of curly braces
`{`
and
`}`
indicate
*composition*
,
`*`
1.
At programming time, users specify topologies and possible sharing
indicates a
*reference*
, and
`-`
marks a "class method".
of parameters. PaddlePaddle can figure out and create parameters
required (and possibly shared) by one or more topologies.
##
# Model
##
Starting from Examples
We used to think that parameters are part of the topology (or layers).
As a summarization
But that is not true because multiple layers could share the same
of
parameter matrix. An example is a network that compares two text
[
our disucssion
](
https://github.com/PaddlePaddle/Paddle/issues/1315
)
,
segments in a semantic spac
e:
let us present two examples her
e:
```
semantic
text A -> projection ---\
layer A \
cosine
similarity -> output
layer
semantic /
text B -> projection ---/
layer B
```
In this network, the two semantic projection layers (A and B) share
### Example 1. Sharing Parameters between Layers
the same parameter matrix.
For more information about our API that specifies topology and
We use
parameter sharing, please refer to [TODO: API].
the
[
3-branch ranking
](
https://github.com/PaddlePaddle/Paddle/issues/1311#issuecomment-279121850
)
model
in this example. For your convenience, I copy-a-paste the model's
topology as follows:
```
A -> f -\
Q -> f --> cost
B -> f -/
```
### Evaluator
The following program trains the topology including the cost, and then
use the sub-network in the trained topology in inference:
Supposed that we have a trained ranking model, we should be able to
use it in our search engine. The search engine's Web server is a
concurrent program so to serve many HTTP requests simultaneously. It
doesn't make sense for each of these threads to have its own copy of the model because that would duplicate topologies and parameters.
However, each thread should be able to record layer outputs, i.e.,
activations, computed from an input, derived from the request. With
*Evaluator*
that saves activations, we can write the over-simplified
server program as:
```
python
```
python
m
=
paddle
.
model
.
load
(
"trained.model"
)
def
f
(
in
):
e
=
paddle
.
layer
.
embedding
(
in
,
parameter_name
=
"embedding"
)
http
.
handle
(
"/"
,
o
=
paddle
.
layer
.
softmax
(
e
,
parameter_name
=
"semantic"
)
lambda
req
:
return
o
e
=
paddle
.
evaluator
.
create
(
m
)
e
.
forward
(
req
)
# Create 3 topologies (subnets), they share parameters because all
e
.
activation
(
layer
=
"output"
))
# returns activations of layer "output"
# correspoinding layers have the same parameter names.
fA
=
f
(
paddle
.
layer
.
data
(
input_name
=
"A"
))
fB
=
f
(
paddle
.
layer
.
data
(
input_name
=
"B"
))
fQ
=
f
(
paddle
.
layer
.
data
(
input_name
=
"Q"
))
topology
=
paddle
.
layer
.
less_than
(
paddle
.
layer
.
cross_entropy
(
fA
,
fQ
),
paddle
.
layer
.
corss_entropy
(
fB
,
fQ
))
# Derive parameters required in topology and create them in model.
parameters
=
paddle
.
parameters
.
create
(
topology
)
# Estimate parameters used in topology from data.
paddle
.
train
(
topology
,
parameters
,
reader
=
read_ranking_model_data
)
# Inference using fA (or fB or fC, as they share their parameters).
[
testA
,
testB
,
testQ
]
=
read_ranking_model_data
()
print
"The sematic-vector of testA: "
,
paddle
.
infer
(
fA
,
parameters
,
testA
)
```
```
### GradientMachine
Similar to the evaluation, the training needs to compute gradients so
to update model parameters. Because an
[
optimizer
](
#optimizer
)
might
run multiple simultaneous threads to update the same model, gradients
should be separated from the model. Because gradients are only used
in training, but not serving, they should be separate from Evaluator.
Hence the
`GradientMachine`
.
### Optimizer
None of Model, Evaluator, nor GradientMachine implements the training
loop, hence Optimizer. We can define a concurrent optimizer that runs
multiple simultaneous threads to train a model -- just let each
thread has its own GradientMachine object.
Most models should be able to be trained using the
### Example 2. Sharing Parameters between "Models"
`paddle.optimizer.SGD`
by calling its
`train`
method. Many
customizations to the SGD algorithm happens with the update equation,
e.g., momentum and the Adam SGD algorithm. We make
`train`
calls
`update`
to do an update, so that we can derive a
`paddle.optimizer.Adam`
from
`paddle.optimizer.SGD`
by overrides only the
`update`
method.
We use
[
GAN
](
https://github.com/PaddlePaddle/book/tree/develop/gan
)
in
this example. In the following example program,
`d0`
and
`d1`
correspond to the two networks in the following figure:
## Programming Interface
<img
src=
"https://github.com/wangyang59/book/raw/00036f4b0da5225041a6824587c1a01cf20159b1/gan/image/gan_ig.png"
width=
400
/>
A fictive example of PaddlePaddle program looks like the following:
```
python
```
python
import
paddle
def
G
(
in
):
# over-simplified example as G has only one layers:
return
paddle
.
layer
.
fc
(
in
,
parameter_name
=
"G"
)
def
D
(
in
);
# again, over-simplified:
return
paddle
.
layer
.
fc
(
in
,
parameter_name
=
"D"
)
# Construct the first topology, which contains both D and G.
# By learning this topology, we update parameters of G.
d0
=
paddle
.
layer
.
should_be_false
(
D
(
G
(
paddle
.
layer
.
data
())))
# Construct a second topology d1, which contains only D. By
# training this topology, we update parameters of D. Note
# that d1 share parameters with d0.
d1
=
paddle
.
layer
.
should_be_true
(
D
(
paddle
.
layer
.
data
()))
# Create parameters from a list of multiple topologies (models) for
# the chance to share parameters between these topologies.
parameters
=
paddle
.
parameters
.
create
([
d0
,
d1
])
# Iterative training of GAN.
for
...:
train
(
d0
,
parameters
,
reader
=
read_from_rng
,
immutable_parameters
=
{
"D"
})
train
(
d1
,
parameters
,
reader
=
read_from_realistic_images
)
# Use d1 for inference:
print
"D thinks a batch of images are realistic "
,
infer
(
d1
,
parameters
,
read_mnist_images
)
```
def
read
(
args
):
f
=
open_file
(
args
[
"filename"
])
mb
=
read_a_minibatch
(
f
)
end_pass
=
eof
(
f
)
if
end_pass
:
f
=
open_file
(
args
[
"filename"
])
# rewind for reading again
yield
mb
,
end_pass
input
=
paddle
.
layer
.
data
(...)
### Summarization
intermediate
=
paddle
.
layers
.
fc
(
input
)
output
=
paddle
.
layer
.
softmax
(
intermediate
)
model
=
paddle
.
model
.
create
(
output
)
paddle
.
train
(
model
,
data_provider
=
read
)
Above two programs reveal some important design concerns:
```
This shows some important part of a program:
1.
Users describe a topology as an expression of layers. Every layer
has a
*parameter name*
. If the users don't specify it explicitly, it's automatically generated as a unique name. By
specifying the parameter name, users can specify the sharing of
parameters between layers and even between topologies.
1.
Define how to read (and augment) data by defining a function, in
1.
`paddle.parameters.create`
figures out parameters required by one
this example,
`read`
, that
`yields`
a minibatch and a boolean flag
or more topologies from parameter names of layers. It creates these
`eof_of_pass`
.
parameters and returns a
`ParameterSet`
object, which is in essence
a map from
*parameter names*
to
*parameters*
.
1.
Define the topology,
`input`
,
`intermediate`
, and
`output`
in this
1.
At training and inference time,
`paddle.train`
and
`paddle.infer`
example.
requires both a topology and the parameter set that holds the parameters of that topology. There are some reasons:
1.
Create parameters from the topology thus forms the model by calling
1.
This prevents users from forgetting to call
`paddel.model.create`
.
`paddle.parameters.create`
.
1.
`paddle.train`
needs to know which parameter set to update.
1.
Users could load another (pre-trained) parameter set and use it
with a topology in
`train.infer`
.
1.
Train the model by calling
`paddle.train`
.
1.
By specifying the
`immutable_parameters`
parameter of
`paddle.train`
, we can forbid the update of these parameters.
##
#
Reader
## Reader
Not all programming frameworks allow users to define I/O functions.
Not all programming frameworks allow users to define I/O functions.
An example is Google MapReduce, which can only read from text,
An example is Google MapReduce, which can only read from text,
...
@@ -145,91 +153,67 @@ readers and writers by deriving from base classes `Reader` and
...
@@ -145,91 +153,67 @@ readers and writers by deriving from base classes `Reader` and
decide to provide the flexibility to users to define their readers.
decide to provide the flexibility to users to define their readers.
#### A Synthetic Data Reader
There are some open questions here:
Sometimes we want to test a topology and/or a training algorithm using
1.
**Should a reader return a Python dictionary?**
synthetic data. We can do this by defining the reader a synthesizer:
```
python
1.
**How to map multiple outputs from a reader to multiple data layers?**
def
read
(
args
):
x
=
sample_from_uniform
(
0.0
,
1.0
)
y
=
sample_from_gauss
(
2
*
x
,
sigma
)
yield
{
x
,
y
},
False
# no end-of-file so no end-of-pass
```
#### A Reader for Online Learning
1.
**
How to easily compose some existing readers to read more data and
feed a topology with more data layers?
**
Readers can also read an infinite data stream, e.g., a log stream from
a search engine and collected by Kafka:
```
python
## Training
def
read
(
args
):
log_stream
=
kafka
.
open_channel
(
args
[
"kafka channel name"
])
yeild
log_stream
.
read
(),
False
# no end-of-pass in online learning
```
### Topology
The recommended way to training a model is to call
`paddle.train`
,
which simply calls
`paddle.trainer.Default`
, a global variable of
By default, layers don't have names. But if we want to refer to a
type
`paddle.trainer.SGD`
. Equivalently, we can do
layer later some time, for example, when we do serving using the model
and wants activations/outputs of a layer, we should give it a name.
```
python
```
python
input
=
paddle
.
layer
.
data
(...)
opt
=
paddle
.
trainer
.
SGD
(...,
paddle
.
updater
.
Adam
(...))
intermediate
=
paddle
.
layer
.
fc
(
input
,
name
=
"inter"
,
...)
opt
.
train
(
topology
,
parameters
,
reader
=
read
,
...)
output
=
paddle
.
layer
.
softmax
(
intermediate
,
name
=
"output"
,
...)
m
=
paddle
.
model
.
create
(
output
)
e
=
paddle
.
evaluator
.
create
(
model
)
e
.
forward
(
read_an_input
())
# compute activations of all layers.
print
e
.
activations
(
layer
=
"inter"
)
# retrieve the activations of layer "inter"
print
e
.
activations
(
layer
=
"output"
)
# retrieve the activations of layer "output"
```
```
###
# Sharing Parameters
###
Updater
In
[
above section
](
#model
)
we shows a network whose two layers share
Please be aware that a trainer can accept an updater as its data
the same parameter matrix. To specify such cases, we give "parameter
member, where an updater is a class derived from
names" to layers. If some layers have the same paraemter names,
`paddle.trainer.Updater`
. This is to make it easier to customize
`paddle.model.create`
creates a single parameter matrix for these
trainers, as discussed
layers:
[
here
](
https://github.com/PaddlePaddle/Paddle/issues/1319
)
.
```
python
### Event Handler
text1
=
paddle
.
layer
.
data
(...)
sematic1
=
paddle
.
layer
.
fc
(
text1
,
...,
parameter_name
=
"sematic_projection"
)
text2
=
paddle
.
layer
.
data
(...)
sematic2
=
paddle
.
layer
.
fc
(
text2
,
...,
parameter_name
=
"sematic_projection"
)
out
=
paddle
.
layer
.
cosine
(
semantic1
,
semantic2
)
```
We can also share parameter matrices between layers in different
`paddle.train`
and
`paddle.trainer.XXX.train`
take an optional
models. To do this, we need an additional parameter that refers to a
parameter
`event_handler`
, which should be either
`None`
or a function
model
:
that handle some events
:
```
python
1.
BeginTraining
model1_input
=
paddle
.
layer
.
data
(...)
1.
EndTraining
model1_output
=
paddle
.
layer
.
softmax
(
model1_input
,
...,
1.
BeginIteration
parameter_name
=
"a_parameter_matrix"
)
1.
EndIteration
model1
=
paddle
.
model
.
create
(
model1_output
)
1.
BeginPass
1.
EndPass
# Another model
model2_semantic
=
paddle
.
layer
.
fc
(
text2
,
...,
parameter_name
=
"a_parameter_matrix"
,
parameter_model
=
model1
)
```
### Training
where EndPass is sent if and only if the reader yields
`end_pass=True`
.
The recommended way to training a model is to call
`paddle.train`
,
An example as follows:
which simply calls
`paddle.optimizer.Default`
, a global variable of
type
`paddle.optimizer.SGD`
. Equivalently, we can do
```
python
```
python
opt
=
paddle
.
optimizer
.
SGD
(...)
def
event_handler
(
event
):
opt
.
train
(
model
,
reader
=
read
,
...)
if
ininstance
(
event
,
paddle
.
event
.
EndIteration
):
print
paddle
.
test
(...)
paddle
.
train
(
topology
,
parameters
,
reader
,
event_handler
)
```
```
#### Distributed Training
If we are writing a PaddlePaddle program in and for iPython/Jypyter,
we can use metaplotlib in the event handler to plot a curve of
cost/error versus iterations, as shown
[
here
](
https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
)
.
### Distributed Training
If users want to do distributed training on a cluster, s/he should
If users want to do distributed training on a cluster, s/he should
call
`paddle.dist_train`
and provides access tokens to the cluster as
call
`paddle.dist_train`
and provides access tokens to the cluster as
...
@@ -240,8 +224,9 @@ access a Kubernetes cluster, s/he should be able to call
...
@@ -240,8 +224,9 @@ access a Kubernetes cluster, s/he should be able to call
```
python
```
python
paddle
.
dist_train
(
model
,
paddle
.
dist_train
(
model
,
trainer
=
paddle
.
trainer
.
SGD
(...,
paddle
.
updater
.
Adam
(...)),
reader
=
read
,
reader
=
read
,
optimizer
=
paddle
.
optimizer
.
SGDOptimizer
(...),
k8s_user
=
"yi"
,
k8s_user
=
"yi"
,
k8s_token
=
"kube_cluster_tls.pem"
,
k8s_token
=
"kube_cluster_tls.pem"
,
k8s_job
=
"hello"
,
k8s_job
=
"hello"
,
...
@@ -251,7 +236,7 @@ paddle.dist_train(model,
...
@@ -251,7 +236,7 @@ paddle.dist_train(model,
The pseudo code if
`paddle.dist_train`
is as follows:
The pseudo code if
`paddle.dist_train`
is as follows:
```
python
```
python
def
dist_train
():
def
dist_train
(
topology
,
parameters
,
trainer
,
reader
,
...
):
if
os
.
getenv
(
"KUBERNETES_SERVICE_HOST"
)
==
None
:
if
os
.
getenv
(
"KUBERNETES_SERVICE_HOST"
)
==
None
:
image_name
=
k8s_user
+
'/'
+
k8s_job
image_name
=
k8s_user
+
'/'
+
k8s_job
docker_build
(
image_name
)
docker_build
(
image_name
)
...
@@ -264,13 +249,13 @@ def dist_train():
...
@@ -264,13 +249,13 @@ def dist_train():
elif
rank
<
15
:
elif
rank
<
15
:
parameter_server
()
parameter_server
()
else
:
else
:
optimiz
er
.
train
(
model
,
reader
=
read
)
train
er
.
train
(
model
,
reader
=
read
)
```
```
Please be aware that if a process is running on the Kubernetes
Please be aware that if a process is running on the Kubernetes
cluster, it will have some environment variables pre-defined.
cluster, it will have some environment variables pre-defined.
If
`dist_train`
doesn't see these environment variables, it know
n
s
If
`dist_train`
doesn't see these environment variables, it knows
that it's running on users' personal computer, and it should work as a
that it's running on users' personal computer, and it should work as a
*launcher*
. Otherwise, it knows that it's running on the cluster and
*launcher*
. Otherwise, it knows that it's running on the cluster and
need to figure out its role as either the master, or a trainer, or a
need to figure out its role as either the master, or a trainer, or a
...
...
python/paddle/v2/event.py
浏览文件 @
f648b3aa
"""
"""
All training events.
All training events.
There are:
* BeginTraining
* EndTraining
* BeginIteration
* EndIteration
* BeginPass
* EndPass
TODO(yuyang18): Complete it!
"""
"""
__all__
=
[
'EndIteration'
]
__all__
=
[
'EndIteration'
]
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录