Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
OpenDILab开源决策智能平台
DI-engine
提交
9caf963b
D
DI-engine
项目概览
OpenDILab开源决策智能平台
/
DI-engine
上一次同步 接近 3 年
通知
66
Star
322
Fork
1
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
1
Wiki
分析
仓库
DevOps
项目成员
Pages
D
DI-engine
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
1
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
9caf963b
编写于
11月 09, 2021
作者:
X
Xu Jingxin
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Padding after sampling from buffer
上级
d053cfcc
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
56 addition
and
4 deletion
+56
-4
ding/worker/buffer/middleware/__init__.py
ding/worker/buffer/middleware/__init__.py
+1
-0
ding/worker/buffer/middleware/clone_object.py
ding/worker/buffer/middleware/clone_object.py
+4
-4
ding/worker/buffer/middleware/padding.py
ding/worker/buffer/middleware/padding.py
+38
-0
ding/worker/buffer/tests/test_middleware.py
ding/worker/buffer/tests/test_middleware.py
+13
-0
未找到文件。
ding/worker/buffer/middleware/__init__.py
浏览文件 @
9caf963b
...
...
@@ -2,3 +2,4 @@ from .clone_object import clone_object
from
.use_time_check
import
use_time_check
from
.staleness_check
import
staleness_check
from
.priority
import
priority
from
.padding
import
padding
ding/worker/buffer/middleware/clone_object.py
浏览文件 @
9caf963b
from
typing
import
Callable
,
Any
,
List
from
typing
import
Callable
,
Any
,
List
,
Union
from
ding.worker.buffer
import
BufferedData
from
ding.worker.buffer.utils
import
fastcopy
...
...
@@ -14,15 +14,15 @@ def clone_object():
data
=
fastcopy
.
copy
(
data
)
return
chain
(
data
,
*
args
,
**
kwargs
)
def
sample
(
chain
:
Callable
,
*
args
,
**
kwargs
)
->
List
[
BufferedData
]:
def
sample
(
chain
:
Callable
,
*
args
,
**
kwargs
)
->
Union
[
List
[
BufferedData
],
List
[
List
[
BufferedData
]]
]:
data
=
chain
(
*
args
,
**
kwargs
)
return
fastcopy
.
copy
(
data
)
def
_
immutabl
e_object
(
action
:
str
,
chain
:
Callable
,
*
args
,
**
kwargs
):
def
_
clon
e_object
(
action
:
str
,
chain
:
Callable
,
*
args
,
**
kwargs
):
if
action
==
"push"
:
return
push
(
chain
,
*
args
,
**
kwargs
)
elif
action
==
"sample"
:
return
sample
(
chain
,
*
args
,
**
kwargs
)
return
chain
(
*
args
,
**
kwargs
)
return
_
immutabl
e_object
return
_
clon
e_object
ding/worker/buffer/middleware/padding.py
0 → 100644
浏览文件 @
9caf963b
import
random
from
typing
import
Callable
,
Union
,
List
from
ding.worker.buffer
import
BufferedData
from
ding.worker.buffer.utils
import
fastcopy
def
padding
(
method
=
"group"
):
"""
Overview:
Fill the nested buffer list to the same size as the largest list.
The default method `group` will randomly select data from each group
and fill it into the current group list.
Arguments:
- method (:obj:`str`): Padding method, currently only supports `group`.
"""
def
sample
(
chain
:
Callable
,
*
args
,
**
kwargs
)
->
Union
[
List
[
BufferedData
],
List
[
List
[
BufferedData
]]]:
sampled_data
=
chain
(
*
args
,
**
kwargs
)
if
len
(
sampled_data
)
==
0
or
isinstance
(
sampled_data
[
0
],
BufferedData
):
return
sampled_data
if
method
==
"group"
:
max_len
=
len
(
max
(
sampled_data
,
key
=
len
))
for
i
,
grouped_data
in
enumerate
(
sampled_data
):
group_len
=
len
(
grouped_data
)
if
group_len
==
max_len
:
continue
for
_
in
range
(
max_len
-
group_len
):
sampled_data
[
i
].
append
(
fastcopy
.
copy
(
random
.
choice
(
grouped_data
)))
return
sampled_data
def
_padding
(
action
:
str
,
chain
:
Callable
,
*
args
,
**
kwargs
):
if
action
==
"sample"
:
return
sample
(
chain
,
*
args
,
**
kwargs
)
return
chain
(
*
args
,
**
kwargs
)
return
_padding
ding/worker/buffer/tests/test_middleware.py
浏览文件 @
9caf963b
...
...
@@ -2,6 +2,7 @@ import pytest
import
torch
from
ding.worker.buffer
import
DequeBuffer
from
ding.worker.buffer.middleware
import
clone_object
,
use_time_check
,
staleness_check
,
priority
from
ding.worker.buffer.middleware.padding
import
padding
@
pytest
.
mark
.
unittest
...
...
@@ -106,3 +107,15 @@ def test_priority():
assert
buffer
.
count
()
==
N
+
N
-
1
buffer
.
clear
()
assert
buffer
.
count
()
==
0
@
pytest
.
mark
.
unittest
def
test_padding
():
buffer
=
DequeBuffer
(
size
=
10
)
buffer
.
use
(
padding
(
method
=
"group"
))
for
i
in
range
(
10
):
buffer
.
push
(
i
,
{
"group"
:
i
&
5
})
# [3,3,2,2]
sampled_data
=
buffer
.
sample
(
4
,
groupby
=
"group"
)
assert
len
(
sampled_data
)
==
4
for
grouped_data
in
sampled_data
:
assert
len
(
grouped_data
)
==
3
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录