提交 cf0a0579 编写于 作者: Q Qiao Longfei

add document for ctr reader

test=develop
上级 45578c1b
## CTR READER
An multi-thread cpp reader that has the same interface with py_reader. It
uses cpp multi-thread to read file and is much more faster then the Python read
thread in py_reader.
Currently, it support two types of file:
- gzip
- plain text file
and two types of data format:
- cvs data format is :
* label dense_fea,dense_fea sparse_fea,sparse_fea
- the svm data format is :
* label slot1:fea_sign slot2:fea_sign slot1:fea_sign
...@@ -54,8 +54,8 @@ def ctr_reader( ...@@ -54,8 +54,8 @@ def ctr_reader(
feed_dict, feed_dict,
file_type, # gzip or plain file_type, # gzip or plain
file_format, # csv or svm file_format, # csv or svm
dense_slot_indexs, dense_slot_index,
sparse_slot_indexs, sparse_slot_index,
capacity, capacity,
thread_num, thread_num,
batch_size, batch_size,
...@@ -78,11 +78,20 @@ def ctr_reader( ...@@ -78,11 +78,20 @@ def ctr_reader(
Note that :code:`Program.clone()` method cannot clone :code:`py_reader`. Note that :code:`Program.clone()` method cannot clone :code:`py_reader`.
Args: Args:
feed_dict(list(variable)): a list of data variable.
file_type('gzip'|'plain'): the type of the data file
file_format('csv'|'svm'): csv data or svm data format.
cvs data format is :
label dense_fea,dense_fea sparse_fea,sparse_fea
the svm data format is :
label slot1:fea_sign slot2:fea_sign slot1:fea_sign
dense_slot_index(list(int)): the index of dense slots
sparse_slot_index(list(int)): the index of sparse slots
capacity(int): The buffer capacity maintained by :code:`py_reader`. capacity(int): The buffer capacity maintained by :code:`py_reader`.
thread_num(list|tuple): List of tuples which declaring data shapes. thread_num(list|tuple): List of tuples which declaring data shapes.
batch_size(list|tuple): List of strs which declaring data type. batch_size(list|tuple): List of strs which declaring data type.
file_list(list|tuple): List of ints which declaring data lod_level. file_list(list|tuple): List of ints which declaring data lod_level.
slots(bool): Whether use double buffer or not. slots(bool): slot id of all sparse feature
name(basestring): The prefix Python queue name and Reader name. None will name(basestring): The prefix Python queue name and Reader name. None will
be generated automatically. be generated automatically.
...@@ -116,8 +125,8 @@ def ctr_reader( ...@@ -116,8 +125,8 @@ def ctr_reader(
'file_list': file_list, 'file_list': file_list,
'file_type': file_type, 'file_type': file_type,
'file_format': file_format, 'file_format': file_format,
'dense_slot_index': dense_slot_indexs, 'dense_slot_index': dense_slot_index,
'sparse_slot_index': sparse_slot_indexs, 'sparse_slot_index': sparse_slot_index,
'sparse_slots': slots, 'sparse_slots': slots,
'ranks': [], 'ranks': [],
'lod_levels': [], 'lod_levels': [],
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册