Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
OpenDILab开源决策智能平台
DI-engine
提交
f764de31
D
DI-engine
项目概览
OpenDILab开源决策智能平台
/
DI-engine
上一次同步 接近 3 年
通知
65
Star
322
Fork
1
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
1
Wiki
分析
仓库
DevOps
项目成员
Pages
D
DI-engine
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
1
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
f764de31
编写于
11月 03, 2021
作者:
X
Xu Jingxin
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Add ignore_insufficient
上级
6e93ff58
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
34 addition
and
4 deletion
+34
-4
ding/worker/buffer/buffer.py
ding/worker/buffer/buffer.py
+4
-1
ding/worker/buffer/deque_buffer.py
ding/worker/buffer/deque_buffer.py
+18
-3
ding/worker/buffer/tests/test_buffer.py
ding/worker/buffer/tests/test_buffer.py
+12
-0
未找到文件。
ding/worker/buffer/buffer.py
浏览文件 @
f764de31
...
...
@@ -68,7 +68,8 @@ class Buffer:
size
:
Optional
[
int
]
=
None
,
indices
:
Optional
[
List
[
str
]]
=
None
,
replace
:
bool
=
False
,
range
:
Optional
[
slice
]
=
None
range
:
Optional
[
slice
]
=
None
,
ignore_insufficient
:
bool
=
False
)
->
List
[
BufferedData
]:
"""
Overview:
...
...
@@ -78,6 +79,8 @@ class Buffer:
- indices (:obj:`Optional[List[str]]`): Sample with multiple indices.
- replace (:obj:`bool`): If use replace is true, you may receive duplicated data from the buffer.
- range (:obj:`slice`): Range slice.
- ignore_insufficient (:obj:`bool`): If ignore_insufficient is true, sampling more than buffer size
with no repetition will not cause an exception.
Returns:
- sample_data (:obj:`List[BufferedData]`):
A list of data with length ``size``.
...
...
ding/worker/buffer/deque_buffer.py
浏览文件 @
f764de31
...
...
@@ -24,7 +24,8 @@ class DequeBuffer(Buffer):
size
:
Optional
[
int
]
=
None
,
indices
:
Optional
[
List
[
str
]]
=
None
,
replace
:
bool
=
False
,
range
:
Optional
[
slice
]
=
None
range
:
Optional
[
slice
]
=
None
,
ignore_insufficient
:
bool
=
False
)
->
List
[
BufferedData
]:
storage
=
self
.
storage
if
range
:
...
...
@@ -32,11 +33,25 @@ class DequeBuffer(Buffer):
assert
size
or
indices
,
"One of size and indices must not be empty."
if
(
size
and
indices
)
and
(
size
!=
len
(
indices
)):
raise
AssertionError
(
"Size and indices length must be equal."
)
if
not
size
:
size
=
len
(
indices
)
value_error
=
None
sampled_data
=
[]
if
indices
:
sampled_data
=
filter
(
lambda
item
:
item
.
index
in
indices
,
self
.
storage
)
sampled_data
=
list
(
filter
(
lambda
item
:
item
.
index
in
indices
,
self
.
storage
)
)
else
:
sampled_data
=
random
.
choices
(
storage
,
k
=
size
)
if
replace
else
random
.
sample
(
storage
,
k
=
size
)
if
replace
:
sampled_data
=
random
.
choices
(
storage
,
k
=
size
)
else
:
try
:
sampled_data
=
random
.
sample
(
storage
,
k
=
size
)
except
ValueError
as
e
:
value_error
=
e
if
not
ignore_insufficient
and
(
value_error
or
len
(
sampled_data
)
!=
size
):
raise
ValueError
(
"There are less than {} data in buffer"
.
format
(
size
))
return
sampled_data
@
apply_middleware
(
"update"
)
...
...
ding/worker/buffer/tests/test_buffer.py
浏览文件 @
f764de31
...
...
@@ -153,3 +153,15 @@ def test_update_delete():
[
item
]
=
buf
.
sample
(
1
)
buf
.
delete
(
item
.
index
)
assert
buf
.
count
()
==
0
@
pytest
.
mark
.
unittest
def
test_ignore_insufficient
():
buffer
=
DequeBuffer
(
size
=
10
)
for
i
in
range
(
2
):
buffer
.
push
(
i
)
with
pytest
.
raises
(
ValueError
):
buffer
.
sample
(
3
)
data
=
buffer
.
sample
(
3
,
ignore_insufficient
=
True
)
assert
len
(
data
)
==
0
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录