Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
08e2b94e
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
08e2b94e
编写于
8月 12, 2013
作者:
F
fjy
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #216 from metamx/announce-fix
Make batch data segment announcer thread safe
上级
bb18b162
5cb1475a
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
57 addition
and
38 deletion
+57
-38
client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
.../metamx/druid/coordination/BatchDataSegmentAnnouncer.java
+57
-38
未找到文件。
client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
浏览文件 @
08e2b94e
...
...
@@ -36,6 +36,8 @@ import java.io.IOException;
import
java.util.Iterator
;
import
java.util.Map
;
import
java.util.Set
;
import
java.util.concurrent.ConcurrentSkipListSet
;
import
java.util.concurrent.atomic.AtomicLong
;
/**
*/
...
...
@@ -48,8 +50,11 @@ public class BatchDataSegmentAnnouncer extends AbstractDataSegmentAnnouncer
private
final
ObjectMapper
jsonMapper
;
private
final
String
liveSegmentLocation
;
private
final
Set
<
SegmentZNode
>
availableZNodes
=
Sets
.
newHashSet
();
private
final
Map
<
DataSegment
,
SegmentZNode
>
segmentLookup
=
Maps
.
newHashMap
();
private
final
Object
lock
=
new
Object
();
private
final
AtomicLong
counter
=
new
AtomicLong
(
0
);
private
final
Set
<
SegmentZNode
>
availableZNodes
=
new
ConcurrentSkipListSet
<
SegmentZNode
>();
private
final
Map
<
DataSegment
,
SegmentZNode
>
segmentLookup
=
Maps
.
newConcurrentMap
();
public
BatchDataSegmentAnnouncer
(
DruidServerMetadata
server
,
...
...
@@ -74,32 +79,34 @@ public class BatchDataSegmentAnnouncer extends AbstractDataSegmentAnnouncer
throw
new
ISE
(
"byte size %,d exceeds %,d"
,
newBytesLen
,
config
.
getMaxNumBytes
());
}
// create new batch
if
(
availableZNodes
.
isEmpty
())
{
SegmentZNode
availableZNode
=
new
SegmentZNode
(
makeServedSegmentPath
(
new
DateTime
().
toString
()));
availableZNode
.
addSegment
(
segment
);
log
.
info
(
"Announcing segment[%s] at path[%s]"
,
segment
.
getIdentifier
(),
availableZNode
.
getPath
());
announcer
.
announce
(
availableZNode
.
getPath
(),
availableZNode
.
getBytes
());
segmentLookup
.
put
(
segment
,
availableZNode
);
availableZNodes
.
add
(
availableZNode
);
}
else
{
// update existing batch
Iterator
<
SegmentZNode
>
iter
=
availableZNodes
.
iterator
();
boolean
done
=
false
;
while
(
iter
.
hasNext
()
&&
!
done
)
{
SegmentZNode
availableZNode
=
iter
.
next
();
if
(
availableZNode
.
getBytes
().
length
+
newBytesLen
<
config
.
getMaxNumBytes
())
{
availableZNode
.
addSegment
(
segment
);
log
.
info
(
"Announcing segment[%s] at path[%s]"
,
segment
.
getIdentifier
(),
availableZNode
.
getPath
());
announcer
.
update
(
availableZNode
.
getPath
(),
availableZNode
.
getBytes
());
segmentLookup
.
put
(
segment
,
availableZNode
);
if
(
availableZNode
.
getCount
()
>=
config
.
getSegmentsPerNode
())
{
availableZNodes
.
remove
(
availableZNode
);
synchronized
(
lock
)
{
// create new batch
if
(
availableZNodes
.
isEmpty
())
{
SegmentZNode
availableZNode
=
new
SegmentZNode
(
makeServedSegmentPath
(
new
DateTime
().
toString
()));
availableZNode
.
addSegment
(
segment
);
log
.
info
(
"Announcing segment[%s] at path[%s]"
,
segment
.
getIdentifier
(),
availableZNode
.
getPath
());
announcer
.
announce
(
availableZNode
.
getPath
(),
availableZNode
.
getBytes
());
segmentLookup
.
put
(
segment
,
availableZNode
);
availableZNodes
.
add
(
availableZNode
);
}
else
{
// update existing batch
Iterator
<
SegmentZNode
>
iter
=
availableZNodes
.
iterator
();
boolean
done
=
false
;
while
(
iter
.
hasNext
()
&&
!
done
)
{
SegmentZNode
availableZNode
=
iter
.
next
();
if
(
availableZNode
.
getBytes
().
length
+
newBytesLen
<
config
.
getMaxNumBytes
())
{
availableZNode
.
addSegment
(
segment
);
log
.
info
(
"Announcing segment[%s] at path[%s]"
,
segment
.
getIdentifier
(),
availableZNode
.
getPath
());
announcer
.
update
(
availableZNode
.
getPath
(),
availableZNode
.
getBytes
());
segmentLookup
.
put
(
segment
,
availableZNode
);
if
(
availableZNode
.
getCount
()
>=
config
.
getSegmentsPerNode
())
{
availableZNodes
.
remove
(
availableZNode
);
}
done
=
true
;
}
done
=
true
;
}
}
}
...
...
@@ -109,15 +116,21 @@ public class BatchDataSegmentAnnouncer extends AbstractDataSegmentAnnouncer
public
void
unannounceSegment
(
DataSegment
segment
)
throws
IOException
{
final
SegmentZNode
segmentZNode
=
segmentLookup
.
remove
(
segment
);
segmentZNode
.
removeSegment
(
segment
);
log
.
info
(
"Unannouncing segment[%s] at path[%s]"
,
segment
.
getIdentifier
(),
segmentZNode
.
getPath
());
if
(
segmentZNode
.
getCount
()
==
0
)
{
availableZNodes
.
remove
(
segmentZNode
);
announcer
.
unannounce
(
segmentZNode
.
getPath
());
}
else
{
announcer
.
update
(
segmentZNode
.
getPath
(),
segmentZNode
.
getBytes
());
availableZNodes
.
add
(
segmentZNode
);
if
(
segmentZNode
==
null
)
{
return
;
}
synchronized
(
lock
)
{
segmentZNode
.
removeSegment
(
segment
);
log
.
info
(
"Unannouncing segment[%s] at path[%s]"
,
segment
.
getIdentifier
(),
segmentZNode
.
getPath
());
if
(
segmentZNode
.
getCount
()
==
0
)
{
availableZNodes
.
remove
(
segmentZNode
);
announcer
.
unannounce
(
segmentZNode
.
getPath
());
}
else
{
announcer
.
update
(
segmentZNode
.
getPath
(),
segmentZNode
.
getBytes
());
availableZNodes
.
add
(
segmentZNode
);
}
}
}
...
...
@@ -167,10 +180,10 @@ public class BatchDataSegmentAnnouncer extends AbstractDataSegmentAnnouncer
private
String
makeServedSegmentPath
(
String
zNode
)
{
return
ZKPaths
.
makePath
(
liveSegmentLocation
,
zNode
);
return
ZKPaths
.
makePath
(
liveSegmentLocation
,
String
.
format
(
"%s%s"
,
zNode
,
counter
.
getAndIncrement
())
);
}
private
class
SegmentZNode
private
class
SegmentZNode
implements
Comparable
<
SegmentZNode
>
{
private
final
String
path
;
...
...
@@ -286,5 +299,11 @@ public class BatchDataSegmentAnnouncer extends AbstractDataSegmentAnnouncer
{
return
path
.
hashCode
();
}
@Override
public
int
compareTo
(
SegmentZNode
segmentZNode
)
{
return
path
.
compareTo
(
segmentZNode
.
getPath
());
}
}
}
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录