Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Forever310
druid
提交
c3448af6
D
druid
项目概览
Forever310
/
druid
与 Fork 源项目一致
从无法访问的项目Fork
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
druid
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
c3448af6
编写于
3月 30, 2014
作者:
X
Xavier Léauté
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
ZooKeeper capitalization + minor wording changes for layout
上级
c7c91202
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
20 addition
and
20 deletion
+20
-20
publications/whitepaper/druid.tex
publications/whitepaper/druid.tex
+20
-20
未找到文件。
publications/whitepaper/druid.tex
浏览文件 @
c3448af6
...
...
@@ -238,10 +238,10 @@ streams. Events indexed via these nodes are immediately available for querying.
The nodes are only concerned with events for some small time range and
periodically hand off immutable batches of events they have collected over this
small time range to other nodes in the Druid cluster that are specialized in
dealing with batches of immutable events. Real-time nodes leverage Zoo
k
eeper
dealing with batches of immutable events. Real-time nodes leverage Zoo
K
eeper
\cite
{
hunt2010zookeeper
}
for coordination with the rest of the Druid cluster.
The nodes announce their online state and the data they serve in
Zoo
k
eeper.
Zoo
K
eeper.
Real-time nodes maintain an in-memory index buffer for all incoming events.
These indexes are incrementally populated as events are ingested and the
...
...
@@ -354,15 +354,15 @@ operationally simple; they only know how to load, drop, and serve immutable
segments.
Similar to real-time nodes, historical nodes announce their online state and
the data they are serving in Zoo
k
eeper. Instructions to load and drop segments
are sent over Zoo
k
eeper and contain information about where the segment is
the data they are serving in Zoo
K
eeper. Instructions to load and drop segments
are sent over Zoo
K
eeper and contain information about where the segment is
located in deep storage and how to decompress and process the segment. Before
a historical node downloads a particular segment from deep storage, it first
checks a local cache that maintains information about what segments already
exist on the node. If information about a segment is not present in the cache,
the historical node will proceed to download the segment from deep storage.
This process is shown in Figure~
\ref
{
fig:historical
_
download
}
. Once processing
is complete, the segment is announced in Zoo
k
eeper. At this point, the segment
is complete, the segment is announced in Zoo
K
eeper. At this point, the segment
is queryable. The local cache also allows for historical nodes to be quickly
updated and restarted. On startup, the node examines its cache and immediately
serves whatever data it finds.
...
...
@@ -393,16 +393,16 @@ can also be created with much less powerful backing hardware. The
“cold” cluster would only contain less frequently accessed segments.
\subsubsection
{
Availability
}
Historical nodes depend on Zoo
k
eeper for segment load and unload instructions.
Should Zoo
k
eeper become unavailable, historical nodes are no longer able to serve
Historical nodes depend on Zoo
K
eeper for segment load and unload instructions.
Should Zoo
K
eeper become unavailable, historical nodes are no longer able to serve
new data or drop outdated data, however, because the queries are served over
HTTP, historical nodes are still able to respond to query requests for
the data they are currently serving. This means that Zoo
k
eeper outages do not
the data they are currently serving. This means that Zoo
K
eeper outages do not
impact current data availability on historical nodes.
\subsection
{
Broker Nodes
}
Broker nodes act as query routers to historical and real-time nodes.
Broker
nodes understand the metadata published in Zook
eeper about what segments are
Broker nodes act as query routers to historical and real-time nodes.
They
understand the metadata published in ZooK
eeper about what segments are
queryable and where those segments are located. Broker nodes route incoming queries
such that the queries hit the right historical or real-time nodes. Broker nodes
also merge partial results from historical and real-time nodes before returning
...
...
@@ -436,13 +436,13 @@ that all historical nodes fail, it is still possible to query results if those
results already exist in the cache.
\subsubsection
{
Availability
}
In the event of a total Zoo
k
eeper outage, data is still queryable. If broker
nodes are unable to communicate to Zoo
k
eeper, they use their last known view of
In the event of a total Zoo
K
eeper outage, data is still queryable. If broker
nodes are unable to communicate to Zoo
K
eeper, they use their last known view of
the cluster and continue to forward queries to real-time and historical nodes.
Broker nodes make the assumption that the structure of the cluster is the same
as it was before the outage. In practice, this availability model has allowed
our Druid cluster to continue serving queries for a significant period of time while we
diagnosed Zoo
k
eeper outages.
diagnosed Zoo
K
eeper outages.
\subsection
{
Coordinator Nodes
}
Druid coordinator nodes are primarily in charge of data management and
...
...
@@ -458,7 +458,7 @@ functionality. The remaining coordinator nodes act as redundant backups.
A coordinator node runs periodically to determine the current state of the
cluster. It makes decisions by comparing the expected state of the cluster with
the actual state of the cluster at the time of the run. As with all Druid
nodes, coordinator nodes maintain a Zoo
k
eeper connection for current cluster
nodes, coordinator nodes maintain a Zoo
K
eeper connection for current cluster
information. Coordinator nodes also maintain a connection to a MySQL
database that contains additional operational parameters and configurations.
One of the key pieces of information located in the MySQL database is a table
...
...
@@ -513,13 +513,13 @@ cluster. Over the last two years, we have never taken downtime in our Druid
cluster for software upgrades.
\subsubsection
{
Availability
}
Druid coordinator nodes have Zoo
k
eeper and MySQL as external dependencies.
Coordinator nodes rely on Zoo
k
eeper to determine what historical nodes already
exist in the cluster. If Zoo
k
eeper becomes unavailable, the coordinator will no
Druid coordinator nodes have Zoo
K
eeper and MySQL as external dependencies.
Coordinator nodes rely on Zoo
K
eeper to determine what historical nodes already
exist in the cluster. If Zoo
K
eeper becomes unavailable, the coordinator will no
longer be able to send instructions to assign, balance, and drop segments.
However, these operations do not affect data availability at all.
The design principle for responding to MySQL and Zoo
k
eeper failures is the
The design principle for responding to MySQL and Zoo
K
eeper failures is the
same: if an external dependency responsible for coordination fails, the cluster
maintains the status quo. Druid uses MySQL to store operational management
information and segment metadata information about what segments should exist
...
...
@@ -1063,7 +1063,7 @@ be desired if one data center is situated much closer to users.
\label
{
sec:related
}
Cattell
\cite
{
cattell
2011
scalable
}
maintains a great summary about existing
Scalable SQL and NoSQL data stores. Hu
\cite
{
hu
2011
stream
}
contributed another
great summary for streaming databases. Druid
feature
-
wise
sits somewhere
great summary for streaming databases. Druid
, feature
-
wise,
sits somewhere
between Google’s Dremel
\cite
{
melnik
2010
dremel
}
and PowerDrill
\cite
{
hall
2012
processing
}
. Druid has most of the features implemented in Dremel
(
Dremel handles arbitrary nested data structures while Druid only allows for a
...
...
@@ -1083,7 +1083,7 @@ Druid allows system wide rolling software updates with no downtime.
Druid is similiar to C
-
Store
\cite
{
stonebraker
2005
c
}
and LazyBase
\cite
{
cipar
2012
lazybase
}
in that it has
two subsystems, a read
-
optimized subsystem in the historical nodes and a
write
-
optimized subsystem in real
-
time nodes. Real
-
time nodes are designed to
write
-
optimized subsystem in
the
real
-
time nodes. Real
-
time nodes are designed to
ingest a high volume of append heavy data, and do not support data updates.
Unlike the two aforementioned systems, Druid is meant for OLAP transactions and
not OLTP transactions.
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录