Networks

The networks module contains pieces of neural network that combine multiple layers.

The pieces are:

  • NLP
    • Text Convolution component
  • Images
    • Conv_Pool
    • Conv_BatchNorm_Pool
  • Recurrent
    • simple_lstm
  • Miscs
    • dropout_layer
paddle.trainer_config_helpers.networks.sequence_conv_pool(*args, **kwargs)

Text convolution pooling layers helper.

Text input => Context Projection => FC Layer => Pooling => Output.

Parameters:
  • name (basestring) – name of output layer(pooling layer name)
  • input (LayerOutput) – name of input layer
  • context_len (int) – context projection length. See context_projection’s document.
  • hidden_size (int) – FC Layer size.
  • context_start (int or None) – context projection length. See context_projection’s context_start.
  • pool_type (BasePoolingType.) – pooling layer type. See pooling_layer’s document.
  • context_proj_layer_name (basestring) – context projection layer name. None if user don’t care.
  • context_proj_param_attr (ParameterAttribute or None.) – context projection parameter attribute. None if user don’t care.
  • fc_layer_name (basestring) – fc layer name. None if user don’t care.
  • fc_param_attr (ParameterAttribute or None) – fc layer parameter attribute. None if user don’t care.
  • fc_bias_attr (ParameterAttribute or None) – fc bias parameter attribute. False if no bias, None if user don’t care.
  • fc_act (BaseActivation) – fc layer activation type. None means tanh
  • pool_bias_attr (ParameterAttribute or None.) – pooling layer bias attr. None if don’t care. False if no bias.
  • fc_attr (ExtraLayerAttribute) – fc layer extra attribute.
  • context_attr (ExtraLayerAttribute) – context projection layer extra attribute.
  • pool_attr (ExtraLayerAttribute) – pooling layer extra attribute.
Returns:

output layer name.
Return type:

LayerOutput
paddle.trainer_config_helpers.networks.text_conv_pool(*args, **kwargs)

Text convolution pooling layers helper.

Text input => Context Projection => FC Layer => Pooling => Output.

Parameters:
  • name (basestring) – name of output layer(pooling layer name)
  • input (LayerOutput) – name of input layer
  • context_len (int) – context projection length. See context_projection’s document.
  • hidden_size (int) – FC Layer size.
  • context_start (int or None) – context projection length. See context_projection’s context_start.
  • pool_type (BasePoolingType.) – pooling layer type. See pooling_layer’s document.
  • context_proj_layer_name (basestring) – context projection layer name. None if user don’t care.
  • context_proj_param_attr (ParameterAttribute or None.) – context projection parameter attribute. None if user don’t care.
  • fc_layer_name (basestring) – fc layer name. None if user don’t care.
  • fc_param_attr (ParameterAttribute or None) – fc layer parameter attribute. None if user don’t care.
  • fc_bias_attr (ParameterAttribute or None) – fc bias parameter attribute. False if no bias, None if user don’t care.
  • fc_act (BaseActivation) – fc layer activation type. None means tanh
  • pool_bias_attr (ParameterAttribute or None.) – pooling layer bias attr. None if don’t care. False if no bias.
  • fc_attr (ExtraLayerAttribute) – fc layer extra attribute.
  • context_attr (ExtraLayerAttribute) – context projection layer extra attribute.
  • pool_attr (ExtraLayerAttribute) – pooling layer extra attribute.
Returns:

output layer name.
Return type:

LayerOutput
paddle.trainer_config_helpers.networks.simple_img_conv_pool(*args, **kwargs)

Simple image convolution and pooling group.

Input => conv => pooling

Parameters:
  • name (basestring) – group name
  • input (LayerOutput) – input layer name.
  • filter_size (int) – see img_conv_layer for details
  • num_filters (int) – see img_conv_layer for details
  • pool_size (int) – see img_pool_layer for details
  • pool_type (BasePoolingType) – see img_pool_layer for details
  • act (BaseActivation) – see img_conv_layer for details
  • groups (int) – see img_conv_layer for details
  • conv_stride (int) – see img_conv_layer for details
  • conv_padding (int) – see img_conv_layer for details
  • bias_attr (ParameterAttribute) – see img_conv_layer for details
  • num_channel (int) – see img_conv_layer for details
  • param_attr (ParameterAttribute) – see img_conv_layer for details
  • shared_bias (bool) – see img_conv_layer for details
  • conv_layer_attr (ExtraLayerAttribute) – see img_conv_layer for details
  • pool_stride (int) – see img_conv_layer for details
  • pool_start (int) – see img_conv_layer for details
  • pool_padding (int) – see img_conv_layer for details
  • pool_layer_attr (ExtraLayerAttribute) – see img_conv_layer for details
Returns:

Layer’s output
Return type:

LayerOutput
paddle.trainer_config_helpers.networks.img_conv_bn_pool(*args, **kwargs)

Convolution, batch normalization, pooling group.

Parameters:
  • name (basestring) – group name
  • input (LayerOutput) – layer’s input
  • filter_size (int) – see img_conv_layer’s document
  • num_filters (int) – see img_conv_layer’s document
  • pool_size (int) – see img_pool_layer’s document.
  • pool_type (BasePoolingType) – see img_pool_layer’s document.
  • act (BaseActivation) – see batch_norm_layer’s document.
  • groups (int) – see img_conv_layer’s document
  • conv_stride (int) – see img_conv_layer’s document.
  • conv_padding (int) – see img_conv_layer’s document.
  • conv_bias_attr (ParameterAttribute) – see img_conv_layer’s document.
  • num_channel (int) – see img_conv_layer’s document.
  • conv_param_attr (ParameterAttribute) – see img_conv_layer’s document.
  • shared_bias (bool) – see img_conv_layer’s document.
  • conv_layer_attr (ExtraLayerOutput) – see img_conv_layer’s document.
  • bn_param_attr (ParameterAttribute.) – see batch_norm_layer’s document.
  • bn_bias_attr – see batch_norm_layer’s document.
  • bn_layer_attr – ParameterAttribute.
  • pool_stride (int) – see img_pool_layer’s document.
  • pool_start (int) – see img_pool_layer’s document.
  • pool_padding (int) – see img_pool_layer’s document.
  • pool_layer_attr (ExtraLayerAttribute) – see img_pool_layer’s document.
Returns:

Layer groups output
Return type:

LayerOutput
paddle.trainer_config_helpers.networks.img_conv_group(*args, **kwargs)

Image Convolution Group, Used for vgg net.

TODO(yuyang18): Complete docs

Parameters:
  • conv_batchnorm_drop_rate –
  • input –
  • conv_num_filter –
  • pool_size –
  • num_channels –
  • conv_padding –
  • conv_filter_size –
  • conv_act –
  • conv_with_batchnorm –
  • pool_stride –
  • pool_type –
Returns:
paddle.trainer_config_helpers.networks.vgg_16_network(input_image, num_channels, num_classes=1000)

Same model from https://gist.github.com/ksimonyan/211839e770f7b538e2d8

Parameters:
  • num_classes –
  • input_image (LayerOutput) –
  • num_channels (int) –
Returns:
paddle.trainer_config_helpers.networks.simple_lstm(*args, **kwargs)

Simple LSTM Cell.

It just combine a mix_layer with fully_matrix_projection and a lstmemory layer. The simple lstm cell was implemented as follow equations.

\[i_t = \sigma(W_{xi}x_{t} + W_{hi}h_{t-1} + W_{ci}c_{t-1} + b_i)\]\[f_t = \sigma(W_{xf}x_{t} + W_{hf}h_{t-1} + W_{cf}c_{t-1} + b_f)\]\[c_t = f_tc_{t-1} + i_t tanh (W_{xc}x_t+W_{hc}h_{t-1} + b_c)\]\[o_t = \sigma(W_{xo}x_{t} + W_{ho}h_{t-1} + W_{co}c_t + b_o)\]\[h_t = o_t tanh(c_t)\]

Please refer Generating Sequences With Recurrent Neural Networks if you want to know what lstm is. Link is here.

Parameters:
  • name (basestring) – lstm layer name.
  • input (LayerOutput) – input layer name.
  • size (int) – lstm layer size.
  • reverse (bool) – is lstm reversed
  • mat_param_attr (ParameterAttribute) – mixed layer’s matrix projection parameter attribute.
  • bias_param_attr (ParameterAttribute|False) – bias parameter attribute. False means no bias, None means default bias.
  • inner_param_attr (ParameterAttribute) – lstm cell parameter attribute.
  • act (BaseActivation) – lstm final activate type
  • gate_act (BaseActivation) – lstm gate activate type
  • state_act (BaseActivation) – lstm state activate type.
  • mixed_layer_attr (ExtraLayerAttribute) – mixed layer’s extra attribute.
  • lstm_cell_attr (ExtraLayerAttribute) – lstm layer’s extra attribute.
Returns:

lstm layer name.
Return type:

LayerOutput
paddle.trainer_config_helpers.networks.lstmemory_unit(*args, **kwargs)

TODO(yuyang18): complete docs

@param input: @param name: @param size: @param mixed_bias_attr: @param mixed_layer_attr: @param param_attr: @param lstm_bias_attr: @param act: @param gate_act: @param state_act: @param lstm_layer_attr: @param get_output_layer_attr: @return:

paddle.trainer_config_helpers.networks.lstmemory_group(*args, **kwargs)

TODO(yuyang18): complete docs

@param input: @param size: @param name: @param reverse: @param param_attr: @param mix_bias_attr: @param lstm_bias_attr: @param act: @param gate_act: @param state_act: @param mixed_layer_attr: @param lstm_layer_attr: @param get_output_layer_attr: @return:

paddle.trainer_config_helpers.networks.gru_unit(*args, **kwargs)
Parameters:
  • input (LayerOutput) –
  • name –
  • size –
  • gru_bias_attr –
  • act –
  • gate_act –
  • gru_layer_attr –
Returns:
paddle.trainer_config_helpers.networks.bidirectional_lstm(*args, **kwargs)

TODO(yuyang18): Complete docs

Parameters:
  • name (basestring) – bidirectional lstm layer name.
  • input (LayerOutput) – input layer.
  • size (int) – lstm layer size.
  • return_seq (bool) – If False, concat word in last time step and return. If True, concat sequnce in all time step and return.
Returns:

lstm layer name.
Return type:

LayerOutput
paddle.trainer_config_helpers.networks.simple_attention(*args, **kwargs)

Calculate and then return a context vector by attention machanism. Size of the context vector equals to size of encoded_sequence.

\[a(s_{i-1},h_{j}) = v_{a}f(W_{a}s_{t-1} + U_{a}h_{j})\]
\[e_{i,j} = a(s_{i-1}, h_{j})\]
\[a_{i,j} = \frac{exp(e_{i,i})}{\sum_{k=1}^{T_{x}{exp(e_{i,k})}}}\]
\[c_{i} = \sum_{j=1}^{T_{x}}a_{i,j}h_{j}\]

where \(h_{j}\) is the jth element of encoded_sequence, \(U_{a}h_{j}\) is the jth element of encoded_proj \(s_{i-1}\) is decoder_state \(f\) is weight_act, and is set to tanh by default.

Please refer to Neural Machine Translation by Jointly Learning to Align and Translate for more details. The link is as follows: https://arxiv.org/abs/1409.0473.

The example usage is: .. code-block:: python

context = simple_attention(encoded_sequence=enc_seq,
encoded_proj=enc_proj, decoder_state=decoder_prev,)
Parameters:
  • name (basestring) – name of the attention model.
  • softmax_param_attr (ParameterAttribute) – parameter attribute of sequence softmax that is used to produce attention weight
  • weight_act (Activation) – activation of the attention model
  • encoded_sequence (LayerOutput) – output of the encoder
  • encoded_proj (LayerOutput) – attention weight is computed by a feed forward neural network which has two inputs : decoder’s hidden state of previous time step and encoder’s output. encoded_proj is output of the feed-forward network for encoder’s output. Here we pre-compute it outside simple_attention for speed consideration.
  • decoder_state (LayerOutput) – hidden state of decoder in previous time step
  • transform_param_attr (ParameterAttribute) – parameter attribute of the feed-forward network that takes decoder_state as inputs to compute attention weight.
Returns:

a context vector
paddle.trainer_config_helpers.networks.dropout_layer(*args, **kwargs)

@TODO(yuyang18): Add comments.

Parameters:
  • name –
  • input –
  • dropout_rate –
Returns:
paddle.trainer_config_helpers.networks.outputs(layers)

Declare the end of network. Currently it will only calculate the input/output order of network. It will calculate the predict network or train network’s output automatically.

Parameters:layers (list|tuple|LayerOutput) –
Returns: