未验证 提交 e2bb4052 编写于 作者: Q Qiao Longfei 提交者: GitHub

Merge branch 'develop' into improve-prefetch-on-server

...@@ -34,7 +34,7 @@ addons: ...@@ -34,7 +34,7 @@ addons:
- automake - automake
- libtool - libtool
- ccache - ccache
ssh_known_hosts: 52.76.173.135 ssh_known_hosts: 13.229.163.131
before_install: before_install:
- if [[ "$JOB" == "check_style" ]]; then sudo ln -s /usr/bin/clang-format-3.8 /usr/bin/clang-format; fi - if [[ "$JOB" == "check_style" ]]; then sudo ln -s /usr/bin/clang-format-3.8 /usr/bin/clang-format; fi
# Paddle is using protobuf 3.1 currently. Protobuf 3.2 breaks the compatibility. So we specify the python # Paddle is using protobuf 3.1 currently. Protobuf 3.2 breaks the compatibility. So we specify the python
......
# API Doc Standard
- [API Doc Structure](#API Doc Structure)
- [Format and Examples](#Format and Examples)
- [Complete Example](#Complete Example)
## API Doc Structure
API Doc should contain the following parts(please write them in order):
- Python API Definition
The definition of API
- Function Description
Description of API's function.
The description includes: meaning, purpose and operation on input of API, reference and corresponding link(if any), formula(if necessary) and explanations of key variables in the formula.
- Args Description
Description of API parameters.
Introduce parameters one by one according to the order in API definition.
The introduction includes: data type, default value(if any), meaning, etc.
- Returns
Introduction of API returned value.
Introduce meaning of returned value, provide correspoding format if necessary.
If returned value is a tuple containing multiple parameters, then introduce parameters one by one in order.
- Raises(if any)
Abnormality, error that may occur, and possible reasons. If there are more than one possible abnormity or error, they should be listed in order.
- Note(if any)
Matters needing attention. If there are more than one matters, they should be listed in order.
- Examples
Examples of how to use API.
## Format and Examples
API documentation must obey reStructuredText format, please refer to [here](http://sphinx-doc-zh.readthedocs.io/en/latest/rest.html).
Format and examples of each part of API documantation are as follows: (take fc for example)
- Python API Definition
- Format
[Python API Definition]
- Example
```
fc(input,
size,
num_flatten_dims=1,
param_attr=None,
bias_attr=None,
act=None,
name=None,
main_program=None,
startup_program=None)
```
- Function Description
- Format
This part contains (please write them in order):
[Function Description]
[Formula]
[Symbols' Descriptions if necessary]
[References if necessary]
- Example
[Function Description]
```
**Fully Connected Layer**
The fully connected layer can take multiple tensors as its inputs. It
creates a variable called weights for each input tensor, which represents
a fully connected weight matrix from each input unit to each output unit.
The fully connected layer multiplies each input tensor with its coresponding
weight to produce an output Tensor. If multiple input tensors are given,
the results of multiple multiplications will be sumed up. If bias_attr is
not None, a bias variable will be created and added to the output. Finally,
if activation is not None, it will be applied to the output as well.
```
[Formula]
```
This process can be formulated as follows:
.. math::
Out = Act({\sum_{i=0}^{N-1}X_iW_i + b})
```
[Symbols' Descriptions if necessary]
```
In the above equation:
* :math:`N`: Number of the input.
* :math:`X_i`: The input tensor.
* :math:`W`: The weights created by this layer.
* :math:`b`: The bias parameter created by this layer (if needed).
* :math:`Act`: The activation function.
* :math:`Out`: The output tensor.
```
[References if necessary]
Since there is no need for reference of fc, we omit them here. Under other circumstances, please provide explicit reference and link, take layer_norm for example:
```
Refer to `Layer Normalization <https://arxiv.org/pdf/1607.06450v1.pdf>`_ for more details.
```
- Args Description
- Format
\[Arg's Name\][(Data Type, Default Value)][Description]
- Example
part of fc parameters are as follows:
```
Args:
input (Variable|list of Variable): The input tensor(s) of this layer, and the dimension of
the input tensor(s) is at least 2.
param_attr (ParamAttr|list of ParamAttr, default None): The parameter attribute for learnable
parameters/weights of this layer.
name (str, default None): The name of this layer.
```
- Returns
- Format
[Name][Shape]
- Example
```
Returns:
A tensor variable storing the transformation result.
```
when returned value contain more than one tuple, please introduce every parameter in order, take dynamic_lstm for example:
```
Returns:
A tuple containing:
The hidden state of LSTM whose shape is (T X D).
The cell state of LSTM whose shape is (T X D).
```
- Raises
- Format
[Exception Type][Condition]
- Example
```
Raises:
ValueError: If the rank of the input is less than 2.
```
- Note
- Format
[Note]
- Example
there is no Note in fc, so we omit this part. If there is any note, please write clearly. If there are more than one notes, please list them in order. Take scaled\_dot\_product\_attention for example:
```
Note:
1. When num_heads > 1, three linear projections are learned respectively
to map input queries, keys and values into queries', keys' and values'.
queries', keys' and values' have the same shapes with queries, keys
and values.
2. When num_heads == 1, scaled_dot_product_attention has no learnable
parameters.
```
- Examples
- Format
\[Python Code Snipper]
- Example
```
Examples:
.. code-block:: python
data = fluid.layers.data(name="data", shape=[32, 32], dtype="float32")
fc = fluid.layers.fc(input=data, size=1000, act="tanh")
```
## Complete Example
Complete Example of fc please see [here](src/fc.py)
...@@ -104,7 +104,7 @@ cc_test(init_test SRCS init_test.cc DEPS init) ...@@ -104,7 +104,7 @@ cc_test(init_test SRCS init_test.cc DEPS init)
cc_test(op_kernel_type_test SRCS op_kernel_type_test.cc DEPS place device_context framework_proto) cc_test(op_kernel_type_test SRCS op_kernel_type_test.cc DEPS place device_context framework_proto)
cc_test(cow_ptr_tests SRCS details/cow_ptr_test.cc) cc_test(cow_ptr_tests SRCS details/cow_ptr_test.cc)
# cc_test(channel_test SRCS channel_test.cc) cc_test(channel_test SRCS channel_test.cc)
cc_test(tuple_test SRCS tuple_test.cc ) cc_test(tuple_test SRCS tuple_test.cc )
cc_test(concurrency_test SRCS concurrency_test.cc DEPS go_op channel_close_op channel_create_op cc_test(concurrency_test SRCS concurrency_test.cc DEPS go_op channel_close_op channel_create_op
channel_send_op channel_recv_op sum_op select_op elementwise_add_op compare_op channel_send_op channel_recv_op sum_op select_op elementwise_add_op compare_op
......
...@@ -138,8 +138,8 @@ void ChannelImpl<T>::Send(T *item) { ...@@ -138,8 +138,8 @@ void ChannelImpl<T>::Send(T *item) {
// If channel is closed, throw exception // If channel is closed, throw exception
if (closed_) { if (closed_) {
lock.unlock();
send_return(); send_return();
lock.unlock();
PADDLE_THROW("Cannot send on closed channel"); PADDLE_THROW("Cannot send on closed channel");
} }
...@@ -152,11 +152,9 @@ void ChannelImpl<T>::Send(T *item) { ...@@ -152,11 +152,9 @@ void ChannelImpl<T>::Send(T *item) {
if (m != nullptr) { if (m != nullptr) {
*(m->data) = std::move(*item); *(m->data) = std::move(*item);
m->Notify(); m->Notify();
lock.unlock();
send_return(); send_return();
return; return;
} else { } else {
lock.unlock();
Send(item); Send(item);
send_return(); send_return();
return; return;
...@@ -169,8 +167,6 @@ void ChannelImpl<T>::Send(T *item) { ...@@ -169,8 +167,6 @@ void ChannelImpl<T>::Send(T *item) {
if (buf_.size() < cap_) { if (buf_.size() < cap_) {
// Copy to buffer // Copy to buffer
buf_.push_back(std::move(*item)); buf_.push_back(std::move(*item));
// Release lock and return true
lock.unlock();
send_return(); send_return();
return; return;
} }
...@@ -181,8 +177,8 @@ void ChannelImpl<T>::Send(T *item) { ...@@ -181,8 +177,8 @@ void ChannelImpl<T>::Send(T *item) {
sendq.push_back(m); sendq.push_back(m);
m->Wait(lock); m->Wait(lock);
if (m->chan_closed) { if (m->chan_closed) {
lock.unlock();
send_return(); send_return();
lock.unlock();
PADDLE_THROW("Cannot send on closed channel"); PADDLE_THROW("Cannot send on closed channel");
} }
send_return(); send_return();
...@@ -195,10 +191,7 @@ bool ChannelImpl<T>::Receive(T *item) { ...@@ -195,10 +191,7 @@ bool ChannelImpl<T>::Receive(T *item) {
// If channel is closed and buffer is empty or // If channel is closed and buffer is empty or
// channel is unbuffered // channel is unbuffered
if (closed_ && buf_.empty()) { if (closed_ && buf_.empty()) return recv_return(false);
lock.unlock();
return recv_return(false);
}
// If there is a sender, directly receive the value we want // If there is a sender, directly receive the value we want
// from the sender. In case of a buffered channel, read from // from the sender. In case of a buffered channel, read from
...@@ -229,7 +222,6 @@ bool ChannelImpl<T>::Receive(T *item) { ...@@ -229,7 +222,6 @@ bool ChannelImpl<T>::Receive(T *item) {
} else } else
return recv_return(Receive(item)); return recv_return(Receive(item));
} }
lock.unlock();
return recv_return(true); return recv_return(true);
} }
...@@ -238,8 +230,7 @@ bool ChannelImpl<T>::Receive(T *item) { ...@@ -238,8 +230,7 @@ bool ChannelImpl<T>::Receive(T *item) {
// Directly read from buffer // Directly read from buffer
*item = std::move(buf_.front()); *item = std::move(buf_.front());
buf_.pop_front(); buf_.pop_front();
// Release lock and return true // return true
lock.unlock();
return recv_return(true); return recv_return(true);
} }
......
...@@ -2,8 +2,8 @@ if(WITH_DISTRIBUTE) ...@@ -2,8 +2,8 @@ if(WITH_DISTRIBUTE)
grpc_library(sendrecvop_grpc SRCS bytebuffer_stream.cc sendrecvop_utils.cc grpc_client.cc grpc_library(sendrecvop_grpc SRCS bytebuffer_stream.cc sendrecvop_utils.cc grpc_client.cc
grpc_server.cc variable_response.cc PROTO send_recv.proto DEPS lod_tensor selected_rows) grpc_server.cc variable_response.cc PROTO send_recv.proto DEPS lod_tensor selected_rows)
set(DISTRIBUTE_COMPILE_FLAGS "-Wno-non-virtual-dtor -Wno-error=non-virtual-dtor -Wno-error=delete-non-virtual-dtor") set(DISTRIBUTE_COMPILE_FLAGS "-Wno-non-virtual-dtor -Wno-error=non-virtual-dtor -Wno-error=delete-non-virtual-dtor")
set_source_files_properties(test_serde.cc grpc_server_test.cc PROPERTIES COMPILE_FLAGS ${DISTRIBUTE_COMPILE_FLAGS}) set_source_files_properties(serde_test.cc grpc_server_test.cc PROPERTIES COMPILE_FLAGS ${DISTRIBUTE_COMPILE_FLAGS})
cc_test(serde_test SRCS test_serde.cc variable_response.cc DEPS grpc++_unsecure grpc_unsecure gpr cc_test(serde_test SRCS serde_test.cc variable_response.cc DEPS grpc++_unsecure grpc_unsecure gpr
cares zlib protobuf sendrecvop_grpc) cares zlib protobuf sendrecvop_grpc)
cc_test(grpc_server_test SRCS grpc_server_test.cc DEPS sendrecvop_grpc grpc++_unsecure grpc_unsecure gpr cares zlib protobuf) cc_test(grpc_server_test SRCS grpc_server_test.cc DEPS sendrecvop_grpc grpc++_unsecure grpc_unsecure gpr cares zlib protobuf)
endif() endif()
...@@ -156,12 +156,12 @@ class RequestPrefetch final : public RequestBase { ...@@ -156,12 +156,12 @@ class RequestPrefetch final : public RequestBase {
virtual void Process() { virtual void Process() {
// prefetch process... // prefetch process...
::grpc::ByteBuffer relay; ::grpc::ByteBuffer reply;
// TODO(Yancey1989): execute the Block which containers prefetch ops // TODO(Yancey1989): execute the Block which containers prefetch ops
VLOG(3) << "RequestPrefetch Process in"; VLOG(3) << "RequestPrefetch Process in";
responder_.Finish(relay, ::grpc::Status::OK, this); responder_.Finish(reply, ::grpc::Status::OK, this);
status_ = FINISH; status_ = FINISH;
} }
......
...@@ -73,12 +73,13 @@ add_custom_target(paddle_python ALL DEPENDS ${paddle_python_deps}) ...@@ -73,12 +73,13 @@ add_custom_target(paddle_python ALL DEPENDS ${paddle_python_deps})
set(PADDLE_PYTHON_PACKAGE_DIR ${CMAKE_CURRENT_BINARY_DIR}/dist/) set(PADDLE_PYTHON_PACKAGE_DIR ${CMAKE_CURRENT_BINARY_DIR}/dist/)
if (WITH_TESTING) if (WITH_TESTING)
add_subdirectory(paddle/reader/tests)
add_subdirectory(paddle/dataset/tests)
if(NOT WITH_FLUID_ONLY) if(NOT WITH_FLUID_ONLY)
add_subdirectory(paddle/trainer_config_helpers/tests) add_subdirectory(paddle/trainer_config_helpers/tests)
if (WITH_SWIG_PY) if (WITH_SWIG_PY)
# enable v2 API unittest only when paddle swig api is compiled # enable v2 API unittest only when paddle swig api is compiled
add_subdirectory(paddle/v2/tests) add_subdirectory(paddle/v2/tests)
add_subdirectory(paddle/v2/reader/tests)
add_subdirectory(paddle/v2/plot/tests) add_subdirectory(paddle/v2/plot/tests)
endif() endif()
endif() endif()
......
...@@ -14,8 +14,14 @@ ...@@ -14,8 +14,14 @@
try: try:
from version import full_version as __version__ from version import full_version as __version__
from version import commit as __git_commit__ from version import commit as __git_commit__
except ImportError: except ImportError:
import sys import sys
sys.stderr.write('''Warning with import paddle: you should not sys.stderr.write('''Warning with import paddle: you should not
import paddle from the source directory; please install paddlepaddle*.whl firstly.''' import paddle from the source directory; please install paddlepaddle*.whl firstly.'''
) )
import reader
import dataset
import batch
batch = batch.batch
...@@ -28,6 +28,7 @@ import wmt16 ...@@ -28,6 +28,7 @@ import wmt16
import mq2007 import mq2007
import flowers import flowers
import voc2012 import voc2012
import image
__all__ = [ __all__ = [
'mnist', 'mnist',
...@@ -43,4 +44,5 @@ __all__ = [ ...@@ -43,4 +44,5 @@ __all__ = [
'mq2007', 'mq2007',
'flowers', 'flowers',
'voc2012', 'voc2012',
'image',
] ]
...@@ -31,7 +31,7 @@ images per class. ...@@ -31,7 +31,7 @@ images per class.
import cPickle import cPickle
import itertools import itertools
import numpy import numpy
import paddle.v2.dataset.common import paddle.dataset.common
import tarfile import tarfile
__all__ = ['train100', 'test100', 'train10', 'test10', 'convert'] __all__ = ['train100', 'test100', 'train10', 'test10', 'convert']
...@@ -75,7 +75,7 @@ def train100(): ...@@ -75,7 +75,7 @@ def train100():
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(CIFAR100_URL, 'cifar', CIFAR100_MD5), paddle.dataset.common.download(CIFAR100_URL, 'cifar', CIFAR100_MD5),
'train') 'train')
...@@ -90,7 +90,7 @@ def test100(): ...@@ -90,7 +90,7 @@ def test100():
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(CIFAR100_URL, 'cifar', CIFAR100_MD5), paddle.dataset.common.download(CIFAR100_URL, 'cifar', CIFAR100_MD5),
'test') 'test')
...@@ -105,7 +105,7 @@ def train10(): ...@@ -105,7 +105,7 @@ def train10():
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(CIFAR10_URL, 'cifar', CIFAR10_MD5), paddle.dataset.common.download(CIFAR10_URL, 'cifar', CIFAR10_MD5),
'data_batch') 'data_batch')
...@@ -120,20 +120,20 @@ def test10(): ...@@ -120,20 +120,20 @@ def test10():
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(CIFAR10_URL, 'cifar', CIFAR10_MD5), paddle.dataset.common.download(CIFAR10_URL, 'cifar', CIFAR10_MD5),
'test_batch') 'test_batch')
def fetch(): def fetch():
paddle.v2.dataset.common.download(CIFAR10_URL, 'cifar', CIFAR10_MD5) paddle.dataset.common.download(CIFAR10_URL, 'cifar', CIFAR10_MD5)
paddle.v2.dataset.common.download(CIFAR100_URL, 'cifar', CIFAR100_MD5) paddle.dataset.common.download(CIFAR100_URL, 'cifar', CIFAR100_MD5)
def convert(path): def convert(path):
""" """
Converts dataset to recordio format Converts dataset to recordio format
""" """
paddle.v2.dataset.common.convert(path, train100(), 1000, "cifar_train100") paddle.dataset.common.convert(path, train100(), 1000, "cifar_train100")
paddle.v2.dataset.common.convert(path, test100(), 1000, "cifar_test100") paddle.dataset.common.convert(path, test100(), 1000, "cifar_test100")
paddle.v2.dataset.common.convert(path, train10(), 1000, "cifar_train10") paddle.dataset.common.convert(path, train10(), 1000, "cifar_train10")
paddle.v2.dataset.common.convert(path, test10(), 1000, "cifar_test10") paddle.dataset.common.convert(path, test10(), 1000, "cifar_test10")
...@@ -19,7 +19,7 @@ import errno ...@@ -19,7 +19,7 @@ import errno
import shutil import shutil
import sys import sys
import importlib import importlib
import paddle.v2.dataset import paddle.dataset
import cPickle import cPickle
import glob import glob
import cPickle as pickle import cPickle as pickle
...@@ -105,24 +105,24 @@ def download(url, module_name, md5sum, save_name=None): ...@@ -105,24 +105,24 @@ def download(url, module_name, md5sum, save_name=None):
def fetch_all(): def fetch_all():
for module_name in filter(lambda x: not x.startswith("__"), for module_name in filter(lambda x: not x.startswith("__"),
dir(paddle.v2.dataset)): dir(paddle.dataset)):
if "fetch" in dir( if "fetch" in dir(
importlib.import_module("paddle.v2.dataset.%s" % module_name)): importlib.import_module("paddle.dataset.%s" % module_name)):
getattr( getattr(
importlib.import_module("paddle.v2.dataset.%s" % module_name), importlib.import_module("paddle.dataset.%s" % module_name),
"fetch")() "fetch")()
def fetch_all_recordio(path): def fetch_all_recordio(path):
for module_name in filter(lambda x: not x.startswith("__"), for module_name in filter(lambda x: not x.startswith("__"),
dir(paddle.v2.dataset)): dir(paddle.dataset)):
if "convert" in dir( if "convert" in dir(
importlib.import_module("paddle.v2.dataset.%s" % module_name)) and \ importlib.import_module("paddle.dataset.%s" % module_name)) and \
not module_name == "common": not module_name == "common":
ds_path = os.path.join(path, module_name) ds_path = os.path.join(path, module_name)
must_mkdirs(ds_path) must_mkdirs(ds_path)
getattr( getattr(
importlib.import_module("paddle.v2.dataset.%s" % module_name), importlib.import_module("paddle.dataset.%s" % module_name),
"convert")(ds_path) "convert")(ds_path)
...@@ -130,7 +130,7 @@ def split(reader, line_count, suffix="%05d.pickle", dumper=cPickle.dump): ...@@ -130,7 +130,7 @@ def split(reader, line_count, suffix="%05d.pickle", dumper=cPickle.dump):
""" """
you can call the function as: you can call the function as:
split(paddle.v2.dataset.cifar.train10(), line_count=1000, split(paddle.dataset.cifar.train10(), line_count=1000,
suffix="imikolov-train-%05d.pickle") suffix="imikolov-train-%05d.pickle")
the output files as: the output files as:
......
...@@ -23,7 +23,7 @@ to initialize SRL model. ...@@ -23,7 +23,7 @@ to initialize SRL model.
import tarfile import tarfile
import gzip import gzip
import itertools import itertools
import paddle.v2.dataset.common import paddle.dataset.common
__all__ = ['test, get_dict', 'get_embedding', 'convert'] __all__ = ['test, get_dict', 'get_embedding', 'convert']
...@@ -203,14 +203,11 @@ def get_dict(): ...@@ -203,14 +203,11 @@ def get_dict():
Get the word, verb and label dictionary of Wikipedia corpus. Get the word, verb and label dictionary of Wikipedia corpus.
""" """
word_dict = load_dict( word_dict = load_dict(
paddle.v2.dataset.common.download(WORDDICT_URL, 'conll05st', paddle.dataset.common.download(WORDDICT_URL, 'conll05st', WORDDICT_MD5))
WORDDICT_MD5))
verb_dict = load_dict( verb_dict = load_dict(
paddle.v2.dataset.common.download(VERBDICT_URL, 'conll05st', paddle.dataset.common.download(VERBDICT_URL, 'conll05st', VERBDICT_MD5))
VERBDICT_MD5))
label_dict = load_label_dict( label_dict = load_label_dict(
paddle.v2.dataset.common.download(TRGDICT_URL, 'conll05st', paddle.dataset.common.download(TRGDICT_URL, 'conll05st', TRGDICT_MD5))
TRGDICT_MD5))
return word_dict, verb_dict, label_dict return word_dict, verb_dict, label_dict
...@@ -218,7 +215,7 @@ def get_embedding(): ...@@ -218,7 +215,7 @@ def get_embedding():
""" """
Get the trained word vector based on Wikipedia corpus. Get the trained word vector based on Wikipedia corpus.
""" """
return paddle.v2.dataset.common.download(EMB_URL, 'conll05st', EMB_MD5) return paddle.dataset.common.download(EMB_URL, 'conll05st', EMB_MD5)
def test(): def test():
...@@ -235,23 +232,23 @@ def test(): ...@@ -235,23 +232,23 @@ def test():
""" """
word_dict, verb_dict, label_dict = get_dict() word_dict, verb_dict, label_dict = get_dict()
reader = corpus_reader( reader = corpus_reader(
paddle.v2.dataset.common.download(DATA_URL, 'conll05st', DATA_MD5), paddle.dataset.common.download(DATA_URL, 'conll05st', DATA_MD5),
words_name='conll05st-release/test.wsj/words/test.wsj.words.gz', words_name='conll05st-release/test.wsj/words/test.wsj.words.gz',
props_name='conll05st-release/test.wsj/props/test.wsj.props.gz') props_name='conll05st-release/test.wsj/props/test.wsj.props.gz')
return reader_creator(reader, word_dict, verb_dict, label_dict) return reader_creator(reader, word_dict, verb_dict, label_dict)
def fetch(): def fetch():
paddle.v2.dataset.common.download(WORDDICT_URL, 'conll05st', WORDDICT_MD5) paddle.dataset.common.download(WORDDICT_URL, 'conll05st', WORDDICT_MD5)
paddle.v2.dataset.common.download(VERBDICT_URL, 'conll05st', VERBDICT_MD5) paddle.dataset.common.download(VERBDICT_URL, 'conll05st', VERBDICT_MD5)
paddle.v2.dataset.common.download(TRGDICT_URL, 'conll05st', TRGDICT_MD5) paddle.dataset.common.download(TRGDICT_URL, 'conll05st', TRGDICT_MD5)
paddle.v2.dataset.common.download(EMB_URL, 'conll05st', EMB_MD5) paddle.dataset.common.download(EMB_URL, 'conll05st', EMB_MD5)
paddle.v2.dataset.common.download(DATA_URL, 'conll05st', DATA_MD5) paddle.dataset.common.download(DATA_URL, 'conll05st', DATA_MD5)
def convert(path): def convert(path):
""" """
Converts dataset to recordio format Converts dataset to recordio format
""" """
paddle.v2.dataset.common.convert(path, test(), 1000, "conl105_train") paddle.dataset.common.convert(path, test(), 1000, "conl105_train")
paddle.v2.dataset.common.convert(path, test(), 1000, "conl105_test") paddle.dataset.common.convert(path, test(), 1000, "conl105_test")
...@@ -34,8 +34,8 @@ import functools ...@@ -34,8 +34,8 @@ import functools
from common import download from common import download
import tarfile import tarfile
import scipy.io as scio import scipy.io as scio
from paddle.v2.image import * from paddle.dataset.image import *
from paddle.v2.reader import * from paddle.reader import *
import os import os
import numpy as np import numpy as np
from multiprocessing import cpu_count from multiprocessing import cpu_count
......
...@@ -20,7 +20,7 @@ of 25,000 highly polar movie reviews for training, and 25,000 for testing. ...@@ -20,7 +20,7 @@ of 25,000 highly polar movie reviews for training, and 25,000 for testing.
Besides, this module also provides API for building dictionary. Besides, this module also provides API for building dictionary.
""" """
import paddle.v2.dataset.common import paddle.dataset.common
import collections import collections
import tarfile import tarfile
import re import re
...@@ -37,8 +37,7 @@ def tokenize(pattern): ...@@ -37,8 +37,7 @@ def tokenize(pattern):
Read files that match the given pattern. Tokenize and yield each file. Read files that match the given pattern. Tokenize and yield each file.
""" """
with tarfile.open(paddle.v2.dataset.common.download(URL, 'imdb', with tarfile.open(paddle.dataset.common.download(URL, 'imdb', MD5)) as tarf:
MD5)) as tarf:
# Note that we should use tarfile.next(), which does # Note that we should use tarfile.next(), which does
# sequential access of member files, other than # sequential access of member files, other than
# tarfile.extractfile, which does random access and might # tarfile.extractfile, which does random access and might
...@@ -136,7 +135,7 @@ def word_dict(): ...@@ -136,7 +135,7 @@ def word_dict():
def fetch(): def fetch():
paddle.v2.dataset.common.download(URL, 'imdb', MD5) paddle.dataset.common.download(URL, 'imdb', MD5)
def convert(path): def convert(path):
...@@ -144,5 +143,5 @@ def convert(path): ...@@ -144,5 +143,5 @@ def convert(path):
Converts dataset to recordio format Converts dataset to recordio format
""" """
w = word_dict() w = word_dict()
paddle.v2.dataset.common.convert(path, lambda: train(w), 1000, "imdb_train") paddle.dataset.common.convert(path, lambda: train(w), 1000, "imdb_train")
paddle.v2.dataset.common.convert(path, lambda: test(w), 1000, "imdb_test") paddle.dataset.common.convert(path, lambda: test(w), 1000, "imdb_test")
...@@ -18,7 +18,7 @@ This module will download dataset from ...@@ -18,7 +18,7 @@ This module will download dataset from
http://www.fit.vutbr.cz/~imikolov/rnnlm/ and parse training set and test set http://www.fit.vutbr.cz/~imikolov/rnnlm/ and parse training set and test set
into paddle reader creators. into paddle reader creators.
""" """
import paddle.v2.dataset.common import paddle.dataset.common
import collections import collections
import tarfile import tarfile
...@@ -54,9 +54,9 @@ def build_dict(min_word_freq=50): ...@@ -54,9 +54,9 @@ def build_dict(min_word_freq=50):
train_filename = './simple-examples/data/ptb.train.txt' train_filename = './simple-examples/data/ptb.train.txt'
test_filename = './simple-examples/data/ptb.valid.txt' test_filename = './simple-examples/data/ptb.valid.txt'
with tarfile.open( with tarfile.open(
paddle.v2.dataset.common.download( paddle.dataset.common.download(paddle.dataset.imikolov.URL,
paddle.v2.dataset.imikolov.URL, 'imikolov', 'imikolov',
paddle.v2.dataset.imikolov.MD5)) as tf: paddle.dataset.imikolov.MD5)) as tf:
trainf = tf.extractfile(train_filename) trainf = tf.extractfile(train_filename)
testf = tf.extractfile(test_filename) testf = tf.extractfile(test_filename)
word_freq = word_count(testf, word_count(trainf)) word_freq = word_count(testf, word_count(trainf))
...@@ -77,9 +77,9 @@ def build_dict(min_word_freq=50): ...@@ -77,9 +77,9 @@ def build_dict(min_word_freq=50):
def reader_creator(filename, word_idx, n, data_type): def reader_creator(filename, word_idx, n, data_type):
def reader(): def reader():
with tarfile.open( with tarfile.open(
paddle.v2.dataset.common.download( paddle.dataset.common.download(
paddle.v2.dataset.imikolov.URL, 'imikolov', paddle.dataset.imikolov.URL, 'imikolov',
paddle.v2.dataset.imikolov.MD5)) as tf: paddle.dataset.imikolov.MD5)) as tf:
f = tf.extractfile(filename) f = tf.extractfile(filename)
UNK = word_idx['<unk>'] UNK = word_idx['<unk>']
...@@ -145,7 +145,7 @@ def test(word_idx, n, data_type=DataType.NGRAM): ...@@ -145,7 +145,7 @@ def test(word_idx, n, data_type=DataType.NGRAM):
def fetch(): def fetch():
paddle.v2.dataset.common.download(URL, "imikolov", MD5) paddle.dataset.common.download(URL, "imikolov", MD5)
def convert(path): def convert(path):
...@@ -154,8 +154,7 @@ def convert(path): ...@@ -154,8 +154,7 @@ def convert(path):
""" """
N = 5 N = 5
word_dict = build_dict() word_dict = build_dict()
paddle.v2.dataset.common.convert(path, paddle.dataset.common.convert(path,
train(word_dict, N), 1000, train(word_dict, N), 1000, "imikolov_train")
"imikolov_train") paddle.dataset.common.convert(path,
paddle.v2.dataset.common.convert(path, test(word_dict, N), 1000, "imikolov_test")
test(word_dict, N), 1000, "imikolov_test")
...@@ -17,7 +17,7 @@ MNIST dataset. ...@@ -17,7 +17,7 @@ MNIST dataset.
This module will download dataset from http://yann.lecun.com/exdb/mnist/ and This module will download dataset from http://yann.lecun.com/exdb/mnist/ and
parse training set and test set into paddle reader creators. parse training set and test set into paddle reader creators.
""" """
import paddle.v2.dataset.common import paddle.dataset.common
import subprocess import subprocess
import numpy import numpy
import platform import platform
...@@ -85,10 +85,10 @@ def train(): ...@@ -85,10 +85,10 @@ def train():
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(TRAIN_IMAGE_URL, 'mnist', paddle.dataset.common.download(TRAIN_IMAGE_URL, 'mnist',
TRAIN_IMAGE_MD5), TRAIN_IMAGE_MD5),
paddle.v2.dataset.common.download(TRAIN_LABEL_URL, 'mnist', paddle.dataset.common.download(TRAIN_LABEL_URL, 'mnist',
TRAIN_LABEL_MD5), 100) TRAIN_LABEL_MD5), 100)
def test(): def test():
...@@ -102,22 +102,21 @@ def test(): ...@@ -102,22 +102,21 @@ def test():
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(TEST_IMAGE_URL, 'mnist', paddle.dataset.common.download(TEST_IMAGE_URL, 'mnist', TEST_IMAGE_MD5),
TEST_IMAGE_MD5), paddle.dataset.common.download(TEST_LABEL_URL, 'mnist', TEST_LABEL_MD5),
paddle.v2.dataset.common.download(TEST_LABEL_URL, 'mnist', 100)
TEST_LABEL_MD5), 100)
def fetch(): def fetch():
paddle.v2.dataset.common.download(TRAIN_IMAGE_URL, 'mnist', TRAIN_IMAGE_MD5) paddle.dataset.common.download(TRAIN_IMAGE_URL, 'mnist', TRAIN_IMAGE_MD5)
paddle.v2.dataset.common.download(TRAIN_LABEL_URL, 'mnist', TRAIN_LABEL_MD5) paddle.dataset.common.download(TRAIN_LABEL_URL, 'mnist', TRAIN_LABEL_MD5)
paddle.v2.dataset.common.download(TEST_IMAGE_URL, 'mnist', TEST_IMAGE_MD5) paddle.dataset.common.download(TEST_IMAGE_URL, 'mnist', TEST_IMAGE_MD5)
paddle.v2.dataset.common.download(TEST_LABEL_URL, 'mnist', TRAIN_LABEL_MD5) paddle.dataset.common.download(TEST_LABEL_URL, 'mnist', TRAIN_LABEL_MD5)
def convert(path): def convert(path):
""" """
Converts dataset to recordio format Converts dataset to recordio format
""" """
paddle.v2.dataset.common.convert(path, train(), 1000, "minist_train") paddle.dataset.common.convert(path, train(), 1000, "minist_train")
paddle.v2.dataset.common.convert(path, test(), 1000, "minist_test") paddle.dataset.common.convert(path, test(), 1000, "minist_test")
...@@ -23,7 +23,7 @@ set and test set into paddle reader creators. ...@@ -23,7 +23,7 @@ set and test set into paddle reader creators.
""" """
import zipfile import zipfile
import paddle.v2.dataset.common import paddle.dataset.common
import re import re
import random import random
import functools import functools
...@@ -100,7 +100,7 @@ USER_INFO = None ...@@ -100,7 +100,7 @@ USER_INFO = None
def __initialize_meta_info__(): def __initialize_meta_info__():
fn = paddle.v2.dataset.common.download(URL, "movielens", MD5) fn = paddle.dataset.common.download(URL, "movielens", MD5)
global MOVIE_INFO global MOVIE_INFO
if MOVIE_INFO is None: if MOVIE_INFO is None:
pattern = re.compile(r'^(.*)\((\d+)\)$') pattern = re.compile(r'^(.*)\((\d+)\)$')
...@@ -247,15 +247,15 @@ def unittest(): ...@@ -247,15 +247,15 @@ def unittest():
def fetch(): def fetch():
paddle.v2.dataset.common.download(URL, "movielens", MD5) paddle.dataset.common.download(URL, "movielens", MD5)
def convert(path): def convert(path):
""" """
Converts dataset to recordio format Converts dataset to recordio format
""" """
paddle.v2.dataset.common.convert(path, train(), 1000, "movielens_train") paddle.dataset.common.convert(path, train(), 1000, "movielens_train")
paddle.v2.dataset.common.convert(path, test(), 1000, "movielens_test") paddle.dataset.common.convert(path, test(), 1000, "movielens_test")
if __name__ == '__main__': if __name__ == '__main__':
......
...@@ -26,7 +26,7 @@ from itertools import chain ...@@ -26,7 +26,7 @@ from itertools import chain
import nltk import nltk
from nltk.corpus import movie_reviews from nltk.corpus import movie_reviews
import paddle.v2.dataset.common import paddle.dataset.common
__all__ = ['train', 'test', 'get_word_dict', 'convert'] __all__ = ['train', 'test', 'get_word_dict', 'convert']
NUM_TRAINING_INSTANCES = 1600 NUM_TRAINING_INSTANCES = 1600
...@@ -39,13 +39,13 @@ def download_data_if_not_yet(): ...@@ -39,13 +39,13 @@ def download_data_if_not_yet():
""" """
try: try:
# make sure that nltk can find the data # make sure that nltk can find the data
if paddle.v2.dataset.common.DATA_HOME not in nltk.data.path: if paddle.dataset.common.DATA_HOME not in nltk.data.path:
nltk.data.path.append(paddle.v2.dataset.common.DATA_HOME) nltk.data.path.append(paddle.dataset.common.DATA_HOME)
movie_reviews.categories() movie_reviews.categories()
except LookupError: except LookupError:
print "Downloading movie_reviews data set, please wait....." print "Downloading movie_reviews data set, please wait....."
nltk.download( nltk.download(
'movie_reviews', download_dir=paddle.v2.dataset.common.DATA_HOME) 'movie_reviews', download_dir=paddle.dataset.common.DATA_HOME)
print "Download data set success....." print "Download data set success....."
print "Path is " + nltk.data.find('corpora/movie_reviews').path print "Path is " + nltk.data.find('corpora/movie_reviews').path
...@@ -129,13 +129,12 @@ def test(): ...@@ -129,13 +129,12 @@ def test():
def fetch(): def fetch():
nltk.download( nltk.download('movie_reviews', download_dir=paddle.dataset.common.DATA_HOME)
'movie_reviews', download_dir=paddle.v2.dataset.common.DATA_HOME)
def convert(path): def convert(path):
""" """
Converts dataset to recordio format Converts dataset to recordio format
""" """
paddle.v2.dataset.common.convert(path, train, 1000, "sentiment_train") paddle.dataset.common.convert(path, train, 1000, "sentiment_train")
paddle.v2.dataset.common.convert(path, test, 1000, "sentiment_test") paddle.dataset.common.convert(path, test, 1000, "sentiment_test")
py_test(test_image SRCS test_image.py)
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.cifar import paddle.dataset.cifar
import unittest import unittest
...@@ -29,25 +29,25 @@ class TestCIFAR(unittest.TestCase): ...@@ -29,25 +29,25 @@ class TestCIFAR(unittest.TestCase):
def test_test10(self): def test_test10(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.cifar.test10()) paddle.dataset.cifar.test10())
self.assertEqual(instances, 10000) self.assertEqual(instances, 10000)
self.assertEqual(max_label_value, 9) self.assertEqual(max_label_value, 9)
def test_train10(self): def test_train10(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.cifar.train10()) paddle.dataset.cifar.train10())
self.assertEqual(instances, 50000) self.assertEqual(instances, 50000)
self.assertEqual(max_label_value, 9) self.assertEqual(max_label_value, 9)
def test_test100(self): def test_test100(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.cifar.test100()) paddle.dataset.cifar.test100())
self.assertEqual(instances, 10000) self.assertEqual(instances, 10000)
self.assertEqual(max_label_value, 99) self.assertEqual(max_label_value, 99)
def test_train100(self): def test_train100(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.cifar.train100()) paddle.dataset.cifar.train100())
self.assertEqual(instances, 50000) self.assertEqual(instances, 50000)
self.assertEqual(max_label_value, 99) self.assertEqual(max_label_value, 99)
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.common import paddle.dataset.common
import unittest import unittest
import tempfile import tempfile
import glob import glob
...@@ -24,14 +24,14 @@ class TestCommon(unittest.TestCase): ...@@ -24,14 +24,14 @@ class TestCommon(unittest.TestCase):
with open(temp_path, 'w') as f: with open(temp_path, 'w') as f:
f.write("Hello\n") f.write("Hello\n")
self.assertEqual('09f7e02f1290be211da707a266f153b3', self.assertEqual('09f7e02f1290be211da707a266f153b3',
paddle.v2.dataset.common.md5file(temp_path)) paddle.dataset.common.md5file(temp_path))
def test_download(self): def test_download(self):
yi_avatar = 'https://avatars0.githubusercontent.com/u/1548775?v=3&s=460' yi_avatar = 'https://avatars0.githubusercontent.com/u/1548775?v=3&s=460'
self.assertEqual( self.assertEqual(
paddle.v2.dataset.common.DATA_HOME + '/test/1548775?v=3&s=460', paddle.dataset.common.DATA_HOME + '/test/1548775?v=3&s=460',
paddle.v2.dataset.common.download( paddle.dataset.common.download(yi_avatar, 'test',
yi_avatar, 'test', 'f75287202d6622414c706c36c16f8e0d')) 'f75287202d6622414c706c36c16f8e0d'))
def test_split(self): def test_split(self):
def test_reader(): def test_reader():
...@@ -42,7 +42,7 @@ class TestCommon(unittest.TestCase): ...@@ -42,7 +42,7 @@ class TestCommon(unittest.TestCase):
return reader return reader
_, temp_path = tempfile.mkstemp() _, temp_path = tempfile.mkstemp()
paddle.v2.dataset.common.split( paddle.dataset.common.split(
test_reader(), 4, suffix=temp_path + '/test-%05d.pickle') test_reader(), 4, suffix=temp_path + '/test-%05d.pickle')
files = glob.glob(temp_path + '/test-%05d.pickle') files = glob.glob(temp_path + '/test-%05d.pickle')
self.assertEqual(len(files), 3) self.assertEqual(len(files), 3)
...@@ -52,7 +52,7 @@ class TestCommon(unittest.TestCase): ...@@ -52,7 +52,7 @@ class TestCommon(unittest.TestCase):
for x in xrange(5): for x in xrange(5):
with open(temp_path + '/%05d.test' % x) as f: with open(temp_path + '/%05d.test' % x) as f:
f.write('%d\n' % x) f.write('%d\n' % x)
reader = paddle.v2.dataset.common.cluster_files_reader( reader = paddle.dataset.common.cluster_files_reader(
temp_path + '/*.test', 5, 0) temp_path + '/*.test', 5, 0)
for idx, e in enumerate(reader()): for idx, e in enumerate(reader()):
self.assertEqual(e, str("0")) self.assertEqual(e, str("0"))
...@@ -69,9 +69,9 @@ class TestCommon(unittest.TestCase): ...@@ -69,9 +69,9 @@ class TestCommon(unittest.TestCase):
return reader return reader
path = tempfile.mkdtemp() path = tempfile.mkdtemp()
paddle.v2.dataset.common.convert(path, paddle.dataset.common.convert(path,
test_reader(), num_shards, test_reader(), num_shards,
'random_images') 'random_images')
files = glob.glob(path + '/random_images-*') files = glob.glob(path + '/random_images-*')
self.assertEqual(len(files), num_shards) self.assertEqual(len(files), num_shards)
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.flowers import paddle.dataset.flowers
import unittest import unittest
...@@ -30,19 +30,19 @@ class TestFlowers(unittest.TestCase): ...@@ -30,19 +30,19 @@ class TestFlowers(unittest.TestCase):
def test_train(self): def test_train(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.flowers.train()) paddle.dataset.flowers.train())
self.assertEqual(instances, 6149) self.assertEqual(instances, 6149)
self.assertEqual(max_label_value, 102) self.assertEqual(max_label_value, 102)
def test_test(self): def test_test(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.flowers.test()) paddle.dataset.flowers.test())
self.assertEqual(instances, 1020) self.assertEqual(instances, 1020)
self.assertEqual(max_label_value, 102) self.assertEqual(max_label_value, 102)
def test_valid(self): def test_valid(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.flowers.valid()) paddle.dataset.flowers.valid())
self.assertEqual(instances, 1020) self.assertEqual(instances, 1020)
self.assertEqual(max_label_value, 102) self.assertEqual(max_label_value, 102)
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.imdb import paddle.dataset.imdb
import unittest import unittest
import re import re
...@@ -30,15 +30,13 @@ class TestIMDB(unittest.TestCase): ...@@ -30,15 +30,13 @@ class TestIMDB(unittest.TestCase):
def test_build_dict(self): def test_build_dict(self):
if self.word_idx == None: if self.word_idx == None:
self.word_idx = paddle.v2.dataset.imdb.build_dict(TRAIN_PATTERN, self.word_idx = paddle.dataset.imdb.build_dict(TRAIN_PATTERN, 150)
150)
self.assertEqual(len(self.word_idx), 7036) self.assertEqual(len(self.word_idx), 7036)
def check_dataset(self, dataset, expected_size): def check_dataset(self, dataset, expected_size):
if self.word_idx == None: if self.word_idx == None:
self.word_idx = paddle.v2.dataset.imdb.build_dict(TRAIN_PATTERN, self.word_idx = paddle.dataset.imdb.build_dict(TRAIN_PATTERN, 150)
150)
sum = 0 sum = 0
for l in dataset(self.word_idx): for l in dataset(self.word_idx):
...@@ -47,10 +45,10 @@ class TestIMDB(unittest.TestCase): ...@@ -47,10 +45,10 @@ class TestIMDB(unittest.TestCase):
self.assertEqual(sum, expected_size) self.assertEqual(sum, expected_size)
def test_train(self): def test_train(self):
self.check_dataset(paddle.v2.dataset.imdb.train, 25000) self.check_dataset(paddle.dataset.imdb.train, 25000)
def test_test(self): def test_test(self):
self.check_dataset(paddle.v2.dataset.imdb.test, 25000) self.check_dataset(paddle.dataset.imdb.test, 25000)
if __name__ == '__main__': if __name__ == '__main__':
......
...@@ -12,10 +12,10 @@ ...@@ -12,10 +12,10 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.imikolov import paddle.dataset.imikolov
import unittest import unittest
WORD_DICT = paddle.v2.dataset.imikolov.build_dict() WORD_DICT = paddle.dataset.imikolov.build_dict()
class TestMikolov(unittest.TestCase): class TestMikolov(unittest.TestCase):
...@@ -25,7 +25,7 @@ class TestMikolov(unittest.TestCase): ...@@ -25,7 +25,7 @@ class TestMikolov(unittest.TestCase):
def test_train(self): def test_train(self):
n = 5 n = 5
self.check_reader(paddle.v2.dataset.imikolov.train(WORD_DICT, n), n) self.check_reader(paddle.dataset.imikolov.train(WORD_DICT, n), n)
first_line = 'aer banknote berlitz calloway centrust cluett fromstein '\ first_line = 'aer banknote berlitz calloway centrust cluett fromstein '\
'gitano guterman hydro-quebec ipo kia memotec mlx nahb punts '\ 'gitano guterman hydro-quebec ipo kia memotec mlx nahb punts '\
...@@ -34,16 +34,16 @@ class TestMikolov(unittest.TestCase): ...@@ -34,16 +34,16 @@ class TestMikolov(unittest.TestCase):
WORD_DICT.get(ch, WORD_DICT['<unk>']) WORD_DICT.get(ch, WORD_DICT['<unk>'])
for ch in first_line.split(' ') for ch in first_line.split(' ')
] ]
for l in paddle.v2.dataset.imikolov.train( for l in paddle.dataset.imikolov.train(
WORD_DICT, n=-1, WORD_DICT, n=-1,
data_type=paddle.v2.dataset.imikolov.DataType.SEQ)(): data_type=paddle.dataset.imikolov.DataType.SEQ)():
read_line = l[0][1:] read_line = l[0][1:]
break break
self.assertEqual(first_line, read_line) self.assertEqual(first_line, read_line)
def test_test(self): def test_test(self):
n = 5 n = 5
self.check_reader(paddle.v2.dataset.imikolov.test(WORD_DICT, n), n) self.check_reader(paddle.dataset.imikolov.test(WORD_DICT, n), n)
first_line = 'consumers may want to move their telephones a little '\ first_line = 'consumers may want to move their telephones a little '\
'closer to the tv set' 'closer to the tv set'
...@@ -51,9 +51,9 @@ class TestMikolov(unittest.TestCase): ...@@ -51,9 +51,9 @@ class TestMikolov(unittest.TestCase):
WORD_DICT.get(ch, WORD_DICT['<unk>']) WORD_DICT.get(ch, WORD_DICT['<unk>'])
for ch in first_line.split(' ') for ch in first_line.split(' ')
] ]
for l in paddle.v2.dataset.imikolov.test( for l in paddle.dataset.imikolov.test(
WORD_DICT, n=-1, WORD_DICT, n=-1,
data_type=paddle.v2.dataset.imikolov.DataType.SEQ)(): data_type=paddle.dataset.imikolov.DataType.SEQ)():
read_line = l[0][1:] read_line = l[0][1:]
break break
self.assertEqual(first_line, read_line) self.assertEqual(first_line, read_line)
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.mnist import paddle.dataset.mnist
import unittest import unittest
...@@ -29,13 +29,13 @@ class TestMNIST(unittest.TestCase): ...@@ -29,13 +29,13 @@ class TestMNIST(unittest.TestCase):
def test_train(self): def test_train(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.mnist.train()) paddle.dataset.mnist.train())
self.assertEqual(instances, 60000) self.assertEqual(instances, 60000)
self.assertEqual(max_label_value, 9) self.assertEqual(max_label_value, 9)
def test_test(self): def test_test(self):
instances, max_label_value = self.check_reader( instances, max_label_value = self.check_reader(
paddle.v2.dataset.mnist.test()) paddle.dataset.mnist.test())
self.assertEqual(instances, 10000) self.assertEqual(instances, 10000)
self.assertEqual(max_label_value, 9) self.assertEqual(max_label_value, 9)
......
...@@ -12,19 +12,19 @@ ...@@ -12,19 +12,19 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.mq2007 import paddle.dataset.mq2007
import unittest import unittest
class TestMQ2007(unittest.TestCase): class TestMQ2007(unittest.TestCase):
def test_pairwise(self): def test_pairwise(self):
for label, query_left, query_right in paddle.v2.dataset.mq2007.test( for label, query_left, query_right in paddle.dataset.mq2007.test(
format="pairwise"): format="pairwise"):
self.assertEqual(query_left.shape(), (46, )) self.assertEqual(query_left.shape(), (46, ))
self.assertEqual(query_right.shape(), (46, )) self.assertEqual(query_right.shape(), (46, ))
def test_listwise(self): def test_listwise(self):
for label_array, query_array in paddle.v2.dataset.mq2007.test( for label_array, query_array in paddle.dataset.mq2007.test(
format="listwise"): format="listwise"):
self.assertEqual(len(label_array), len(query_array)) self.assertEqual(len(label_array), len(query_array))
......
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
import unittest import unittest
import numpy as np import numpy as np
import paddle.v2.image as image import paddle.dataset.image as image
class Image(unittest.TestCase): class Image(unittest.TestCase):
......
...@@ -17,7 +17,7 @@ ...@@ -17,7 +17,7 @@
import unittest import unittest
import nltk import nltk
import paddle.v2.dataset.sentiment as st import paddle.dataset.sentiment as st
from nltk.corpus import movie_reviews from nltk.corpus import movie_reviews
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.voc2012 import paddle.dataset.voc2012
import unittest import unittest
...@@ -26,15 +26,15 @@ class TestVOC(unittest.TestCase): ...@@ -26,15 +26,15 @@ class TestVOC(unittest.TestCase):
return sum return sum
def test_train(self): def test_train(self):
count = self.check_reader(paddle.v2.dataset.voc_seg.train()) count = self.check_reader(paddle.dataset.voc_seg.train())
self.assertEqual(count, 2913) self.assertEqual(count, 2913)
def test_test(self): def test_test(self):
count = self.check_reader(paddle.v2.dataset.voc_seg.test()) count = self.check_reader(paddle.dataset.voc_seg.test())
self.assertEqual(count, 1464) self.assertEqual(count, 1464)
def test_val(self): def test_val(self):
count = self.check_reader(paddle.v2.dataset.voc_seg.val()) count = self.check_reader(paddle.dataset.voc_seg.val())
self.assertEqual(count, 1449) self.assertEqual(count, 1449)
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2.dataset.wmt16 import paddle.dataset.wmt16
import unittest import unittest
...@@ -34,28 +34,28 @@ class TestWMT16(unittest.TestCase): ...@@ -34,28 +34,28 @@ class TestWMT16(unittest.TestCase):
def test_train(self): def test_train(self):
for idx, sample in enumerate( for idx, sample in enumerate(
paddle.v2.dataset.wmt16.train( paddle.dataset.wmt16.train(
src_dict_size=100000, trg_dict_size=100000)()): src_dict_size=100000, trg_dict_size=100000)()):
if idx >= 10: break if idx >= 10: break
self.checkout_one_sample(sample) self.checkout_one_sample(sample)
def test_test(self): def test_test(self):
for idx, sample in enumerate( for idx, sample in enumerate(
paddle.v2.dataset.wmt16.test( paddle.dataset.wmt16.test(
src_dict_size=1000, trg_dict_size=1000)()): src_dict_size=1000, trg_dict_size=1000)()):
if idx >= 10: break if idx >= 10: break
self.checkout_one_sample(sample) self.checkout_one_sample(sample)
def test_val(self): def test_val(self):
for idx, sample in enumerate( for idx, sample in enumerate(
paddle.v2.dataset.wmt16.validation( paddle.dataset.wmt16.validation(
src_dict_size=1000, trg_dict_size=1000)()): src_dict_size=1000, trg_dict_size=1000)()):
if idx >= 10: break if idx >= 10: break
self.checkout_one_sample(sample) self.checkout_one_sample(sample)
def test_get_dict(self): def test_get_dict(self):
dict_size = 1000 dict_size = 1000
word_dict = paddle.v2.dataset.wmt16.get_dict("en", dict_size, True) word_dict = paddle.dataset.wmt16.get_dict("en", dict_size, True)
self.assertEqual(len(word_dict), dict_size) self.assertEqual(len(word_dict), dict_size)
self.assertEqual(word_dict[0], "<s>") self.assertEqual(word_dict[0], "<s>")
self.assertEqual(word_dict[1], "<e>") self.assertEqual(word_dict[1], "<e>")
......
...@@ -21,8 +21,7 @@ parse training set and test set into paddle reader creators. ...@@ -21,8 +21,7 @@ parse training set and test set into paddle reader creators.
import numpy as np import numpy as np
import os import os
import paddle.v2.dataset.common import paddle.dataset.common
from paddle.v2.parameters import Parameters
__all__ = ['train', 'test'] __all__ = ['train', 'test']
...@@ -85,7 +84,7 @@ def train(): ...@@ -85,7 +84,7 @@ def train():
:rtype: callable :rtype: callable
""" """
global UCI_TRAIN_DATA global UCI_TRAIN_DATA
load_data(paddle.v2.dataset.common.download(URL, 'uci_housing', MD5)) load_data(paddle.dataset.common.download(URL, 'uci_housing', MD5))
def reader(): def reader():
for d in UCI_TRAIN_DATA: for d in UCI_TRAIN_DATA:
...@@ -105,7 +104,7 @@ def test(): ...@@ -105,7 +104,7 @@ def test():
:rtype: callable :rtype: callable
""" """
global UCI_TEST_DATA global UCI_TEST_DATA
load_data(paddle.v2.dataset.common.download(URL, 'uci_housing', MD5)) load_data(paddle.dataset.common.download(URL, 'uci_housing', MD5))
def reader(): def reader():
for d in UCI_TEST_DATA: for d in UCI_TEST_DATA:
...@@ -114,21 +113,13 @@ def test(): ...@@ -114,21 +113,13 @@ def test():
return reader return reader
def model():
tar_file = paddle.v2.dataset.common.download(URL_MODEL, 'fit_a_line.tar',
MD5_MODEL)
with open(tar_file, 'r') as f:
parameters = Parameters.from_tar(f)
return parameters
def fetch(): def fetch():
paddle.v2.dataset.common.download(URL, 'uci_housing', MD5) paddle.dataset.common.download(URL, 'uci_housing', MD5)
def convert(path): def convert(path):
""" """
Converts dataset to recordio format Converts dataset to recordio format
""" """
paddle.v2.dataset.common.convert(path, train(), 1000, "uci_housing_train") paddle.dataset.common.convert(path, train(), 1000, "uci_housing_train")
paddle.v2.dataset.common.convert(path, test(), 1000, "uci_houseing_test") paddle.dataset.common.convert(path, test(), 1000, "uci_houseing_test")
...@@ -22,8 +22,8 @@ with segmentation has been increased from 7,062 to 9,993. ...@@ -22,8 +22,8 @@ with segmentation has been increased from 7,062 to 9,993.
import tarfile import tarfile
import io import io
import numpy as np import numpy as np
from paddle.v2.dataset.common import download from paddle.dataset.common import download
from paddle.v2.image import * from paddle.dataset.image import *
from PIL import Image from PIL import Image
__all__ = ['train', 'test', 'val'] __all__ = ['train', 'test', 'val']
......
...@@ -22,8 +22,7 @@ parse training set and test set into paddle reader creators. ...@@ -22,8 +22,7 @@ parse training set and test set into paddle reader creators.
import tarfile import tarfile
import gzip import gzip
import paddle.v2.dataset.common import paddle.dataset.common
from paddle.v2.parameters import Parameters
__all__ = [ __all__ = [
'train', 'train',
...@@ -123,7 +122,7 @@ def train(dict_size): ...@@ -123,7 +122,7 @@ def train(dict_size):
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN), paddle.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN),
'train/train', dict_size) 'train/train', dict_size)
...@@ -139,27 +138,20 @@ def test(dict_size): ...@@ -139,27 +138,20 @@ def test(dict_size):
:rtype: callable :rtype: callable
""" """
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN), paddle.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN),
'test/test', dict_size) 'test/test', dict_size)
def gen(dict_size): def gen(dict_size):
return reader_creator( return reader_creator(
paddle.v2.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN), paddle.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN),
'gen/gen', dict_size) 'gen/gen', dict_size)
def model():
tar_file = paddle.v2.dataset.common.download(URL_MODEL, 'wmt14', MD5_MODEL)
with gzip.open(tar_file, 'r') as f:
parameters = Parameters.from_tar(f)
return parameters
def get_dict(dict_size, reverse=True): def get_dict(dict_size, reverse=True):
# if reverse = False, return dict = {'a':'001', 'b':'002', ...} # if reverse = False, return dict = {'a':'001', 'b':'002', ...}
# else reverse = true, return dict = {'001':'a', '002':'b', ...} # else reverse = true, return dict = {'001':'a', '002':'b', ...}
tar_file = paddle.v2.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN) tar_file = paddle.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN)
src_dict, trg_dict = __read_to_dict(tar_file, dict_size) src_dict, trg_dict = __read_to_dict(tar_file, dict_size)
if reverse: if reverse:
src_dict = {v: k for k, v in src_dict.items()} src_dict = {v: k for k, v in src_dict.items()}
...@@ -168,8 +160,8 @@ def get_dict(dict_size, reverse=True): ...@@ -168,8 +160,8 @@ def get_dict(dict_size, reverse=True):
def fetch(): def fetch():
paddle.v2.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN) paddle.dataset.common.download(URL_TRAIN, 'wmt14', MD5_TRAIN)
paddle.v2.dataset.common.download(URL_MODEL, 'wmt14', MD5_MODEL) paddle.dataset.common.download(URL_MODEL, 'wmt14', MD5_MODEL)
def convert(path): def convert(path):
...@@ -177,6 +169,5 @@ def convert(path): ...@@ -177,6 +169,5 @@ def convert(path):
Converts dataset to recordio format Converts dataset to recordio format
""" """
dict_size = 30000 dict_size = 30000
paddle.v2.dataset.common.convert(path, paddle.dataset.common.convert(path, train(dict_size), 1000, "wmt14_train")
train(dict_size), 1000, "wmt14_train") paddle.dataset.common.convert(path, test(dict_size), 1000, "wmt14_test")
paddle.v2.dataset.common.convert(path, test(dict_size), 1000, "wmt14_test")
...@@ -33,7 +33,7 @@ import tarfile ...@@ -33,7 +33,7 @@ import tarfile
import gzip import gzip
from collections import defaultdict from collections import defaultdict
import paddle.v2.dataset.common import paddle.dataset.common
__all__ = [ __all__ = [
"train", "train",
...@@ -76,7 +76,7 @@ def __build_dict(tar_file, dict_size, save_path, lang): ...@@ -76,7 +76,7 @@ def __build_dict(tar_file, dict_size, save_path, lang):
def __load_dict(tar_file, dict_size, lang, reverse=False): def __load_dict(tar_file, dict_size, lang, reverse=False):
dict_path = os.path.join(paddle.v2.dataset.common.DATA_HOME, dict_path = os.path.join(paddle.dataset.common.DATA_HOME,
"wmt16/%s_%d.dict" % (lang, dict_size)) "wmt16/%s_%d.dict" % (lang, dict_size))
if not os.path.exists(dict_path) or ( if not os.path.exists(dict_path) or (
len(open(dict_path, "r").readlines()) != dict_size): len(open(dict_path, "r").readlines()) != dict_size):
...@@ -178,8 +178,8 @@ def train(src_dict_size, trg_dict_size, src_lang="en"): ...@@ -178,8 +178,8 @@ def train(src_dict_size, trg_dict_size, src_lang="en"):
src_lang) src_lang)
return reader_creator( return reader_creator(
tar_file=paddle.v2.dataset.common.download(DATA_URL, "wmt16", DATA_MD5, tar_file=paddle.dataset.common.download(DATA_URL, "wmt16", DATA_MD5,
"wmt16.tar.gz"), "wmt16.tar.gz"),
file_name="wmt16/train", file_name="wmt16/train",
src_dict_size=src_dict_size, src_dict_size=src_dict_size,
trg_dict_size=trg_dict_size, trg_dict_size=trg_dict_size,
...@@ -227,8 +227,8 @@ def test(src_dict_size, trg_dict_size, src_lang="en"): ...@@ -227,8 +227,8 @@ def test(src_dict_size, trg_dict_size, src_lang="en"):
src_lang) src_lang)
return reader_creator( return reader_creator(
tar_file=paddle.v2.dataset.common.download(DATA_URL, "wmt16", DATA_MD5, tar_file=paddle.dataset.common.download(DATA_URL, "wmt16", DATA_MD5,
"wmt16.tar.gz"), "wmt16.tar.gz"),
file_name="wmt16/test", file_name="wmt16/test",
src_dict_size=src_dict_size, src_dict_size=src_dict_size,
trg_dict_size=trg_dict_size, trg_dict_size=trg_dict_size,
...@@ -274,8 +274,8 @@ def validation(src_dict_size, trg_dict_size, src_lang="en"): ...@@ -274,8 +274,8 @@ def validation(src_dict_size, trg_dict_size, src_lang="en"):
src_lang) src_lang)
return reader_creator( return reader_creator(
tar_file=paddle.v2.dataset.common.download(DATA_URL, "wmt16", DATA_MD5, tar_file=paddle.dataset.common.download(DATA_URL, "wmt16", DATA_MD5,
"wmt16.tar.gz"), "wmt16.tar.gz"),
file_name="wmt16/val", file_name="wmt16/val",
src_dict_size=src_dict_size, src_dict_size=src_dict_size,
trg_dict_size=trg_dict_size, trg_dict_size=trg_dict_size,
...@@ -303,12 +303,12 @@ def get_dict(lang, dict_size, reverse=False): ...@@ -303,12 +303,12 @@ def get_dict(lang, dict_size, reverse=False):
if lang == "en": dict_size = min(dict_size, TOTAL_EN_WORDS) if lang == "en": dict_size = min(dict_size, TOTAL_EN_WORDS)
else: dict_size = min(dict_size, TOTAL_DE_WORDS) else: dict_size = min(dict_size, TOTAL_DE_WORDS)
dict_path = os.path.join(paddle.v2.dataset.common.DATA_HOME, dict_path = os.path.join(paddle.dataset.common.DATA_HOME,
"wmt16/%s_%d.dict" % (lang, dict_size)) "wmt16/%s_%d.dict" % (lang, dict_size))
assert os.path.exists(dict_path), "Word dictionary does not exist. " assert os.path.exists(dict_path), "Word dictionary does not exist. "
"Please invoke paddle.dataset.wmt16.train/test/validation first " "Please invoke paddle.dataset.wmt16.train/test/validation first "
"to build the dictionary." "to build the dictionary."
tar_file = os.path.join(paddle.v2.dataset.common.DATA_HOME, "wmt16.tar.gz") tar_file = os.path.join(paddle.dataset.common.DATA_HOME, "wmt16.tar.gz")
return __load_dict(tar_file, dict_size, lang, reverse) return __load_dict(tar_file, dict_size, lang, reverse)
...@@ -323,7 +323,7 @@ def convert(path, src_dict_size, trg_dict_size, src_lang): ...@@ -323,7 +323,7 @@ def convert(path, src_dict_size, trg_dict_size, src_lang):
"""Converts dataset to recordio format. """Converts dataset to recordio format.
""" """
paddle.v2.dataset.common.convert( paddle.dataset.common.convert(
path, path,
train( train(
src_dict_size=src_dict_size, src_dict_size=src_dict_size,
...@@ -331,7 +331,7 @@ def convert(path, src_dict_size, trg_dict_size, src_lang): ...@@ -331,7 +331,7 @@ def convert(path, src_dict_size, trg_dict_size, src_lang):
src_lang=src_lang), src_lang=src_lang),
1000, 1000,
"wmt16_train") "wmt16_train")
paddle.v2.dataset.common.convert( paddle.dataset.common.convert(
path, path,
test( test(
src_dict_size=src_dict_size, src_dict_size=src_dict_size,
...@@ -339,7 +339,7 @@ def convert(path, src_dict_size, trg_dict_size, src_lang): ...@@ -339,7 +339,7 @@ def convert(path, src_dict_size, trg_dict_size, src_lang):
src_lang=src_lang), src_lang=src_lang),
1000, 1000,
"wmt16_test") "wmt16_test")
paddle.v2.dataset.common.convert( paddle.dataset.common.convert(
path, path,
validation( validation(
src_dict_size=src_dict_size, src_dict_size=src_dict_size,
......
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
# limitations under the License. # limitations under the License.
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.fluid.core as core import paddle.fluid.core as core
import paddle.fluid.framework as framework import paddle.fluid.framework as framework
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import contextlib import contextlib
import numpy import numpy
......
...@@ -14,7 +14,7 @@ ...@@ -14,7 +14,7 @@
from __future__ import print_function from __future__ import print_function
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import contextlib import contextlib
import math import math
......
...@@ -15,8 +15,8 @@ ...@@ -15,8 +15,8 @@
import math import math
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.v2.dataset.conll05 as conll05 import paddle.dataset.conll05 as conll05
import paddle.fluid as fluid import paddle.fluid as fluid
from paddle.fluid.initializer import init_on_cpu from paddle.fluid.initializer import init_on_cpu
import contextlib import contextlib
......
...@@ -14,7 +14,7 @@ ...@@ -14,7 +14,7 @@
import contextlib import contextlib
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.fluid.framework as framework import paddle.fluid.framework as framework
import paddle.fluid.layers as pd import paddle.fluid.layers as pd
......
...@@ -14,7 +14,7 @@ ...@@ -14,7 +14,7 @@
from __future__ import print_function from __future__ import print_function
import argparse import argparse
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.v2 as paddle import paddle
import sys import sys
import numpy import numpy
import unittest import unittest
......
...@@ -16,7 +16,7 @@ import math ...@@ -16,7 +16,7 @@ import math
import sys import sys
import os import os
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.fluid.framework as framework import paddle.fluid.framework as framework
import paddle.fluid.layers as layers import paddle.fluid.layers as layers
......
...@@ -15,7 +15,7 @@ from __future__ import print_function ...@@ -15,7 +15,7 @@ from __future__ import print_function
import unittest import unittest
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.v2 as paddle import paddle
import contextlib import contextlib
import math import math
import numpy as np import numpy as np
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import unittest import unittest
import os import os
......
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
# limitations under the License. # limitations under the License.
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import math import math
import sys import sys
......
...@@ -16,7 +16,7 @@ from __future__ import print_function ...@@ -16,7 +16,7 @@ from __future__ import print_function
import sys import sys
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import math import math
import sys import sys
......
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
# limitations under the License. # limitations under the License.
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.fluid.core as core import paddle.fluid.core as core
import paddle.fluid.framework as framework import paddle.fluid.framework as framework
......
...@@ -19,7 +19,7 @@ import os ...@@ -19,7 +19,7 @@ import os
import matplotlib import matplotlib
import numpy import numpy
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
matplotlib.use('Agg') matplotlib.use('Agg')
......
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
import numpy as np import numpy as np
import sys import sys
......
...@@ -14,7 +14,7 @@ ...@@ -14,7 +14,7 @@
from __future__ import print_function from __future__ import print_function
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
BATCH_SIZE = 128 BATCH_SIZE = 128
......
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
# limitations under the License. # limitations under the License.
import numpy as np import numpy as np
import paddle.v2 as paddle import paddle
import paddle.fluid as fluid import paddle.fluid as fluid
BATCH_SIZE = 128 BATCH_SIZE = 128
......
...@@ -12,12 +12,12 @@ ...@@ -12,12 +12,12 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import paddle
import paddle.fluid.layers as layers import paddle.fluid.layers as layers
from paddle.fluid.framework import Program, program_guard, default_main_program, default_startup_program from paddle.fluid.framework import Program, program_guard, default_main_program, default_startup_program
from paddle.fluid.executor import Executor from paddle.fluid.executor import Executor
from paddle.fluid.optimizer import MomentumOptimizer from paddle.fluid.optimizer import MomentumOptimizer
import paddle.fluid.core as core import paddle.fluid.core as core
import paddle.v2 as paddle
import unittest import unittest
import numpy as np import numpy as np
......
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
# limitations under the License. # limitations under the License.
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.v2 as paddle import paddle
import unittest import unittest
import numpy import numpy
......
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
# limitations under the License. # limitations under the License.
import unittest import unittest
import paddle.v2 as paddle import paddle
import paddle.fluid.core as core import paddle.fluid.core as core
import paddle.fluid as fluid import paddle.fluid as fluid
from paddle.fluid.backward import append_backward from paddle.fluid.backward import append_backward
......
...@@ -15,8 +15,8 @@ ...@@ -15,8 +15,8 @@
import unittest import unittest
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.v2 as paddle import paddle
import paddle.v2.dataset.mnist as mnist import paddle.dataset.mnist as mnist
class TestMultipleReader(unittest.TestCase): class TestMultipleReader(unittest.TestCase):
......
...@@ -15,8 +15,8 @@ ...@@ -15,8 +15,8 @@
import unittest import unittest
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.v2 as paddle import paddle
import paddle.v2.dataset.mnist as mnist import paddle.dataset.mnist as mnist
from shutil import copyfile from shutil import copyfile
......
...@@ -16,9 +16,9 @@ import numpy ...@@ -16,9 +16,9 @@ import numpy
import unittest import unittest
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.v2 as paddle import paddle
import paddle.v2.dataset.mnist as mnist import paddle.dataset.mnist as mnist
import paddle.v2.dataset.wmt16 as wmt16 import paddle.dataset.wmt16 as wmt16
def simple_fc_net(): def simple_fc_net():
......
...@@ -15,8 +15,8 @@ ...@@ -15,8 +15,8 @@
import unittest import unittest
import paddle.fluid as fluid import paddle.fluid as fluid
import paddle.v2 as paddle import paddle
import paddle.v2.dataset.mnist as mnist import paddle.dataset.mnist as mnist
class TestRecordIO(unittest.TestCase): class TestRecordIO(unittest.TestCase):
......
...@@ -16,7 +16,7 @@ Creator package contains some simple reader creator, which could ...@@ -16,7 +16,7 @@ Creator package contains some simple reader creator, which could
be used in user program. be used in user program.
""" """
__all__ = ['np_array', 'text_file', 'recordio', 'cloud_reader'] __all__ = ['np_array', 'text_file', 'recordio']
def np_array(x): def np_array(x):
...@@ -66,7 +66,7 @@ def recordio(paths, buf_size=100): ...@@ -66,7 +66,7 @@ def recordio(paths, buf_size=100):
""" """
import recordio as rec import recordio as rec
import paddle.v2.reader.decorator as dec import paddle.reader.decorator as dec
import cPickle as pickle import cPickle as pickle
def reader(): def reader():
...@@ -83,48 +83,3 @@ def recordio(paths, buf_size=100): ...@@ -83,48 +83,3 @@ def recordio(paths, buf_size=100):
f.close() f.close()
return dec.buffered(reader, buf_size) return dec.buffered(reader, buf_size)
pass_num = 0
def cloud_reader(paths, etcd_endpoints, timeout_sec=5, buf_size=64):
"""
Create a data reader that yield a record one by one from
the paths:
:paths: path of recordio files, can be a string or a string list.
:etcd_endpoints: the endpoints for etcd cluster
:returns: data reader of recordio files.
.. code-block:: python
from paddle.v2.reader.creator import cloud_reader
etcd_endpoints = "http://127.0.0.1:2379"
trainer.train.(
reader=cloud_reader(["/work/dataset/uci_housing/uci_housing*"], etcd_endpoints),
)
"""
import os
import cPickle as pickle
import paddle.v2.master as master
c = master.client(etcd_endpoints, timeout_sec, buf_size)
if isinstance(paths, basestring):
path = [paths]
else:
path = paths
c.set_dataset(path)
def reader():
global pass_num
c.paddle_start_get_records(pass_num)
pass_num += 1
while True:
r, e = c.next_record()
if not r:
if e != -2:
print "get record error: ", e
break
yield pickle.loads(r)
return reader
...@@ -28,14 +28,14 @@ ...@@ -28,14 +28,14 @@
import os import os
import unittest import unittest
import numpy as np import numpy as np
import paddle.v2.reader.creator import paddle.reader.creator
class TestNumpyArray(unittest.TestCase): class TestNumpyArray(unittest.TestCase):
def test_numpy_array(self): def test_numpy_array(self):
l = [[1, 2, 3], [4, 5, 6]] l = [[1, 2, 3], [4, 5, 6]]
x = np.array(l, np.int32) x = np.array(l, np.int32)
reader = paddle.v2.reader.creator.np_array(x) reader = paddle.reader.creator.np_array(x)
for idx, e in enumerate(reader()): for idx, e in enumerate(reader()):
self.assertItemsEqual(e, l[idx]) self.assertItemsEqual(e, l[idx])
...@@ -43,14 +43,14 @@ class TestNumpyArray(unittest.TestCase): ...@@ -43,14 +43,14 @@ class TestNumpyArray(unittest.TestCase):
class TestTextFile(unittest.TestCase): class TestTextFile(unittest.TestCase):
def test_text_file(self): def test_text_file(self):
path = os.path.join(os.path.dirname(__file__), "test_data_creator.txt") path = os.path.join(os.path.dirname(__file__), "test_data_creator.txt")
reader = paddle.v2.reader.creator.text_file(path) reader = paddle.reader.creator.text_file(path)
for idx, e in enumerate(reader()): for idx, e in enumerate(reader()):
self.assertEqual(e, str(idx * 2) + " " + str(idx * 2 + 1)) self.assertEqual(e, str(idx * 2) + " " + str(idx * 2 + 1))
class TestRecordIO(unittest.TestCase): class TestRecordIO(unittest.TestCase):
def do_test(self, path): def do_test(self, path):
reader = paddle.v2.reader.creator.recordio(path) reader = paddle.reader.creator.recordio(path)
idx = 0 idx = 0
for e in reader(): for e in reader():
if idx == 0: if idx == 0:
......
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
import time import time
import unittest import unittest
import paddle.v2.reader import paddle.reader
def reader_creator_10(dur): def reader_creator_10(dur):
...@@ -39,7 +39,7 @@ class TestMap(unittest.TestCase): ...@@ -39,7 +39,7 @@ class TestMap(unittest.TestCase):
yield "h" yield "h"
yield "i" yield "i"
r = paddle.v2.reader.map_readers(tokenize, read) r = paddle.reader.map_readers(tokenize, read)
for i, e in enumerate(r()): for i, e in enumerate(r()):
self.assertEqual(e, i) self.assertEqual(e, i)
...@@ -47,7 +47,7 @@ class TestMap(unittest.TestCase): ...@@ -47,7 +47,7 @@ class TestMap(unittest.TestCase):
class TestBuffered(unittest.TestCase): class TestBuffered(unittest.TestCase):
def test_read(self): def test_read(self):
for size in range(20): for size in range(20):
b = paddle.v2.reader.buffered(reader_creator_10(0), size) b = paddle.reader.buffered(reader_creator_10(0), size)
c = 0 c = 0
for i in b(): for i in b():
self.assertEqual(i, c) self.assertEqual(i, c)
...@@ -56,7 +56,7 @@ class TestBuffered(unittest.TestCase): ...@@ -56,7 +56,7 @@ class TestBuffered(unittest.TestCase):
def test_buffering(self): def test_buffering(self):
# read have 30ms delay. # read have 30ms delay.
b = paddle.v2.reader.buffered(reader_creator_10(0.03), 10) b = paddle.reader.buffered(reader_creator_10(0.03), 10)
last_time = time.time() last_time = time.time()
for idx, i in enumerate(b()): for idx, i in enumerate(b()):
elapsed_time = time.time() - last_time elapsed_time = time.time() - last_time
...@@ -70,17 +70,17 @@ class TestBuffered(unittest.TestCase): ...@@ -70,17 +70,17 @@ class TestBuffered(unittest.TestCase):
class TestCompose(unittest.TestCase): class TestCompose(unittest.TestCase):
def test_compse(self): def test_compse(self):
reader = paddle.v2.reader.compose( reader = paddle.reader.compose(
reader_creator_10(0), reader_creator_10(0)) reader_creator_10(0), reader_creator_10(0))
for idx, e in enumerate(reader()): for idx, e in enumerate(reader()):
self.assertEqual(e, (idx, idx)) self.assertEqual(e, (idx, idx))
def test_compose_not_aligned(self): def test_compose_not_aligned(self):
total = 0 total = 0
reader = paddle.v2.reader.compose( reader = paddle.reader.compose(
paddle.v2.reader.chain(reader_creator_10(0), reader_creator_10(0)), paddle.reader.chain(reader_creator_10(0), reader_creator_10(0)),
reader_creator_10(0)) reader_creator_10(0))
with self.assertRaises(paddle.v2.reader.ComposeNotAligned): with self.assertRaises(paddle.reader.ComposeNotAligned):
for e in reader(): for e in reader():
total += 1 total += 1
# expecting 10, not 20 # expecting 10, not 20
...@@ -88,8 +88,8 @@ class TestCompose(unittest.TestCase): ...@@ -88,8 +88,8 @@ class TestCompose(unittest.TestCase):
def test_compose_not_aligned_no_check(self): def test_compose_not_aligned_no_check(self):
total = 0 total = 0
reader = paddle.v2.reader.compose( reader = paddle.reader.compose(
paddle.v2.reader.chain(reader_creator_10(0), reader_creator_10(0)), paddle.reader.chain(reader_creator_10(0), reader_creator_10(0)),
reader_creator_10(0), reader_creator_10(0),
check_alignment=False) check_alignment=False)
for e in reader(): for e in reader():
...@@ -100,7 +100,7 @@ class TestCompose(unittest.TestCase): ...@@ -100,7 +100,7 @@ class TestCompose(unittest.TestCase):
class TestChain(unittest.TestCase): class TestChain(unittest.TestCase):
def test_chain(self): def test_chain(self):
c = paddle.v2.reader.chain(reader_creator_10(0), reader_creator_10(0)) c = paddle.reader.chain(reader_creator_10(0), reader_creator_10(0))
idx = 0 idx = 0
for e in c(): for e in c():
self.assertEqual(e, idx % 10) self.assertEqual(e, idx % 10)
...@@ -113,7 +113,7 @@ class TestShuffle(unittest.TestCase): ...@@ -113,7 +113,7 @@ class TestShuffle(unittest.TestCase):
case = [(0, True), (1, True), (10, False), (100, False)] case = [(0, True), (1, True), (10, False), (100, False)]
a = reader_creator_10(0) a = reader_creator_10(0)
for size, checkEq in case: for size, checkEq in case:
s = paddle.v2.reader.shuffle(a, size) s = paddle.reader.shuffle(a, size)
total = 0 total = 0
for idx, e in enumerate(s()): for idx, e in enumerate(s()):
if checkEq: if checkEq:
...@@ -133,9 +133,9 @@ class TestXmap(unittest.TestCase): ...@@ -133,9 +133,9 @@ class TestXmap(unittest.TestCase):
for order in orders: for order in orders:
for tNum in thread_nums: for tNum in thread_nums:
for size in buffered_size: for size in buffered_size:
reader = paddle.v2.reader.xmap_readers(mapper, reader = paddle.reader.xmap_readers(mapper,
reader_creator_10(0), reader_creator_10(0),
tNum, size, order) tNum, size, order)
for n in xrange(3): for n in xrange(3):
result = [] result = []
for i in reader(): for i in reader():
...@@ -150,7 +150,7 @@ class TestPipeReader(unittest.TestCase): ...@@ -150,7 +150,7 @@ class TestPipeReader(unittest.TestCase):
def test_pipe_reader(self): def test_pipe_reader(self):
def example_reader(myfiles): def example_reader(myfiles):
for f in myfiles: for f in myfiles:
pr = paddle.v2.reader.PipeReader("cat %s" % f, bufsize=128) pr = paddle.reader.PipeReader("cat %s" % f, bufsize=128)
for l in pr.get_line(): for l in pr.get_line():
yield l yield l
......
...@@ -22,17 +22,13 @@ import data_type ...@@ -22,17 +22,13 @@ import data_type
import topology import topology
import networks import networks
import evaluator import evaluator
from . import dataset
from . import reader
from . import plot from . import plot
import attr import attr
import op import op
import pooling import pooling
import inference import inference
import networks import networks
import minibatch
import plot import plot
import image
import paddle.trainer.config_parser as cp import paddle.trainer.config_parser as cp
__all__ = [ __all__ = [
...@@ -48,14 +44,11 @@ __all__ = [ ...@@ -48,14 +44,11 @@ __all__ = [
'data_type', 'data_type',
'attr', 'attr',
'pooling', 'pooling',
'dataset',
'reader',
'topology', 'topology',
'networks', 'networks',
'infer', 'infer',
'plot', 'plot',
'evaluator', 'evaluator',
'image',
'master', 'master',
] ]
...@@ -153,4 +146,3 @@ def init(**kwargs): ...@@ -153,4 +146,3 @@ def init(**kwargs):
infer = inference.infer infer = inference.infer
batch = minibatch.batch
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
import numpy import numpy
import collections import collections
import topology import topology
import minibatch import paddle
import cPickle import cPickle
__all__ = ['infer', 'Inference'] __all__ = ['infer', 'Inference']
...@@ -80,7 +80,7 @@ class Inference(object): ...@@ -80,7 +80,7 @@ class Inference(object):
for each_sample in input: for each_sample in input:
yield each_sample yield each_sample
reader = minibatch.batch(__reader_impl__, batch_size=batch_size) reader = paddle.batch(__reader_impl__, batch_size=batch_size)
self.__gradient_machine__.start() self.__gradient_machine__.start()
for data_batch in reader(): for data_batch in reader():
......
...@@ -20,7 +20,7 @@ The primary usage shows below. ...@@ -20,7 +20,7 @@ The primary usage shows below.
.. code-block:: python .. code-block:: python
import paddle.v2 as paddle import paddle
img = paddle.layer.data(name='img', type=paddle.data_type.dense_vector(784)) img = paddle.layer.data(name='img', type=paddle.data_type.dense_vector(784))
hidden = paddle.layer.fc(input=img, size=200) hidden = paddle.layer.fc(input=img, size=200)
......
py_test(test_op SRCS test_op.py) py_test(test_op SRCS test_op.py)
py_test(test_image SRCS test_image.py)
py_test(test_layer SRCS test_layer.py) py_test(test_layer SRCS test_layer.py)
py_test(test_topology SRCS test_topology.py) py_test(test_topology SRCS test_topology.py)
py_test(test_rnn_layer SRCS test_rnn_layer.py) py_test(test_rnn_layer SRCS test_rnn_layer.py)
......
...@@ -27,6 +27,7 @@ ...@@ -27,6 +27,7 @@
# limitations under the License. # limitations under the License.
import unittest import unittest
import math import math
import paddle.dataset as dataset
import paddle.v2 as paddle import paddle.v2 as paddle
...@@ -40,7 +41,7 @@ def wordemb(inlayer): ...@@ -40,7 +41,7 @@ def wordemb(inlayer):
def train(): def train():
word_dict = paddle.dataset.imikolov.build_dict() word_dict = dataset.imikolov.build_dict()
dict_size = len(word_dict) dict_size = len(word_dict)
# Every layer takes integer value of range [0, dict_size) # Every layer takes integer value of range [0, dict_size)
firstword = paddle.layer.data( firstword = paddle.layer.data(
......
...@@ -63,6 +63,8 @@ write_version_py(filename='@PADDLE_SOURCE_DIR@/python/paddle/version.py') ...@@ -63,6 +63,8 @@ write_version_py(filename='@PADDLE_SOURCE_DIR@/python/paddle/version.py')
packages=['paddle', packages=['paddle',
'paddle.utils', 'paddle.utils',
'paddle.dataset',
'paddle.reader',
'paddle.fluid', 'paddle.fluid',
'paddle.fluid.proto', 'paddle.fluid.proto',
'paddle.fluid.proto.profiler', 'paddle.fluid.proto.profiler',
...@@ -73,8 +75,6 @@ if '${WITH_FLUID_ONLY}'== 'OFF': ...@@ -73,8 +75,6 @@ if '${WITH_FLUID_ONLY}'== 'OFF':
'paddle.trainer', 'paddle.trainer',
'paddle.trainer_config_helpers', 'paddle.trainer_config_helpers',
'paddle.v2', 'paddle.v2',
'paddle.v2.dataset',
'paddle.v2.reader',
'paddle.v2.master', 'paddle.v2.master',
'paddle.v2.plot', 'paddle.v2.plot',
'py_paddle'] 'py_paddle']
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册