Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
DiDi
kafka-manager
提交
d8aa3d64
K
kafka-manager
项目概览
DiDi
/
kafka-manager
9 个月 前同步成功
通知
58
Star
6372
Fork
1229
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
K
kafka-manager
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
提交
d8aa3d64
编写于
10月 14, 2022
作者:
D
dianyang12138
提交者:
EricZeng
10月 21, 2022
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix:修复es模版错误
上级
13d8fd55
变更
5
隐藏空白更改
内联
并排
Showing
5 changed file
with
312 addition
and
103 deletion
+312
-103
bin/init_es_template.sh
bin/init_es_template.sh
+90
-27
km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/constant/ESIndexConstant.java
...ji/know/streaming/km/common/constant/ESIndexConstant.java
+1
-24
km-dist/docker/manager/es_template_create.sh
km-dist/docker/manager/es_template_create.sh
+87
-24
km-dist/init/template/ks_kafka_replication_metric
km-dist/init/template/ks_kafka_replication_metric
+44
-1
km-dist/init/template/template.sh
km-dist/init/template/template.sh
+90
-27
未找到文件。
bin/init_es_template.sh
浏览文件 @
d8aa3d64
...
...
@@ -439,7 +439,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
esaddr
}
:
${
port
}
/_template/ks_kafka_replication_metric
-d
'{
"order" : 10,
"index_patterns" : [
"ks_kafka_
parti
tion_metric*"
"ks_kafka_
replica
tion_metric*"
],
"settings" : {
"index" : {
...
...
@@ -500,30 +500,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
}
},
"aliases" : { }
}[root@10-255-0-23 template]# cat ks_kafka_replication_metric
PUT _template/ks_kafka_replication_metric
{
"order" : 10,
"index_patterns" : [
"ks_kafka_replication_metric*"
],
"settings" : {
"index" : {
"number_of_shards" : "10"
}
},
"mappings" : {
"properties" : {
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
"index" : true,
"type" : "date",
"doc_values" : true
}
}
},
"aliases" : { }
}'
}'
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
esaddr
}
:
${
port
}
/_template/ks_kafka_topic_metric
-d
'{
"order" : 10,
...
...
@@ -640,7 +617,92 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
}
},
"aliases" : { }
}'
}'
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
SERVER_ES_ADDRESS
}
/_template/ks_kafka_zookeeper_metric
-d
'{
"order" : 10,
"index_patterns" : [
"ks_kafka_zookeeper_metric*"
],
"settings" : {
"index" : {
"number_of_shards" : "10"
}
},
"mappings" : {
"properties" : {
"routingValue" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"clusterPhyId" : {
"type" : "long"
},
"metrics" : {
"properties" : {
"AvgRequestLatency" : {
"type" : "double"
},
"MinRequestLatency" : {
"type" : "double"
},
"MaxRequestLatency" : {
"type" : "double"
},
"OutstandingRequests" : {
"type" : "double"
},
"NodeCount" : {
"type" : "double"
},
"WatchCount" : {
"type" : "double"
},
"NumAliveConnections" : {
"type" : "double"
},
"PacketsReceived" : {
"type" : "double"
},
"PacketsSent" : {
"type" : "double"
},
"EphemeralsCount" : {
"type" : "double"
},
"ApproximateDataSize" : {
"type" : "double"
},
"OpenFileDescriptorCount" : {
"type" : "double"
},
"MaxFileDescriptorCount" : {
"type" : "double"
}
}
},
"key" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
"type" : "date"
}
}
},
"aliases" : { }
}'
for
i
in
{
0..6
}
;
do
...
...
@@ -650,6 +712,7 @@ do
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_group_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_partition_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_replication_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_zookeeper_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_topic_metric
${
logdate
}
||
\
exit
2
done
\ No newline at end of file
done
km-common/src/main/java/com/xiaojukeji/know/streaming/km/common/constant/ESIndexConstant.java
浏览文件 @
d8aa3d64
...
...
@@ -558,7 +558,7 @@ public class ESIndexConstant {
public
final
static
String
REPLICATION_TEMPLATE
=
"{\n"
+
" \"order\" : 10,\n"
+
" \"index_patterns\" : [\n"
+
" \"ks_kafka_
parti
tion_metric*\"\n"
+
" \"ks_kafka_
replica
tion_metric*\"\n"
+
" ],\n"
+
" \"settings\" : {\n"
+
" \"index\" : {\n"
+
...
...
@@ -619,29 +619,6 @@ public class ESIndexConstant {
" }\n"
+
" },\n"
+
" \"aliases\" : { }\n"
+
" }[root@10-255-0-23 template]# cat ks_kafka_replication_metric\n"
+
"PUT _template/ks_kafka_replication_metric\n"
+
"{\n"
+
" \"order\" : 10,\n"
+
" \"index_patterns\" : [\n"
+
" \"ks_kafka_replication_metric*\"\n"
+
" ],\n"
+
" \"settings\" : {\n"
+
" \"index\" : {\n"
+
" \"number_of_shards\" : \"10\"\n"
+
" }\n"
+
" },\n"
+
" \"mappings\" : {\n"
+
" \"properties\" : {\n"
+
" \"timestamp\" : {\n"
+
" \"format\" : \"yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis\",\n"
+
" \"index\" : true,\n"
+
" \"type\" : \"date\",\n"
+
" \"doc_values\" : true\n"
+
" }\n"
+
" }\n"
+
" },\n"
+
" \"aliases\" : { }\n"
+
" }"
;
public
final
static
String
ZOOKEEPER_INDEX
=
"ks_kafka_zookeeper_metric"
;
...
...
km-dist/docker/manager/es_template_create.sh
浏览文件 @
d8aa3d64
...
...
@@ -443,7 +443,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
SERVER_ES_ADDRESS
}
/_template/ks_kafka_replication_metric
-d
'{
"order" : 10,
"index_patterns" : [
"ks_kafka_
parti
tion_metric*"
"ks_kafka_
replica
tion_metric*"
],
"settings" : {
"index" : {
...
...
@@ -504,29 +504,6 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
}
},
"aliases" : { }
}[root@10-255-0-23 template]# cat ks_kafka_replication_metric
PUT _template/ks_kafka_replication_metric
{
"order" : 10,
"index_patterns" : [
"ks_kafka_replication_metric*"
],
"settings" : {
"index" : {
"number_of_shards" : "10"
}
},
"mappings" : {
"properties" : {
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
"index" : true,
"type" : "date",
"doc_values" : true
}
}
},
"aliases" : { }
}'
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
SERVER_ES_ADDRESS
}
/_template/ks_kafka_topic_metric
-d
'{
...
...
@@ -646,6 +623,91 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
"aliases" : { }
}'
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
SERVER_ES_ADDRESS
}
/_template/ks_kafka_zookeeper_metric
-d
'{
"order" : 10,
"index_patterns" : [
"ks_kafka_zookeeper_metric*"
],
"settings" : {
"index" : {
"number_of_shards" : "10"
}
},
"mappings" : {
"properties" : {
"routingValue" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"clusterPhyId" : {
"type" : "long"
},
"metrics" : {
"properties" : {
"AvgRequestLatency" : {
"type" : "double"
},
"MinRequestLatency" : {
"type" : "double"
},
"MaxRequestLatency" : {
"type" : "double"
},
"OutstandingRequests" : {
"type" : "double"
},
"NodeCount" : {
"type" : "double"
},
"WatchCount" : {
"type" : "double"
},
"NumAliveConnections" : {
"type" : "double"
},
"PacketsReceived" : {
"type" : "double"
},
"PacketsSent" : {
"type" : "double"
},
"EphemeralsCount" : {
"type" : "double"
},
"ApproximateDataSize" : {
"type" : "double"
},
"OpenFileDescriptorCount" : {
"type" : "double"
},
"MaxFileDescriptorCount" : {
"type" : "double"
}
}
},
"key" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
"type" : "date"
}
}
},
"aliases" : { }
}'
for
i
in
{
0..6
}
;
do
logdate
=
_
$(
date
-d
"
${
i
}
day ago"
+%Y-%m-%d
)
...
...
@@ -654,6 +716,7 @@ do
curl
-s
-o
/dev/null
-X
PUT http://
${
SERVER_ES_ADDRESS
}
/ks_kafka_group_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
SERVER_ES_ADDRESS
}
/ks_kafka_partition_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
SERVER_ES_ADDRESS
}
/ks_kafka_replication_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
SERVER_ES_ADDRESS
}
/ks_kafka_zookeeper_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
SERVER_ES_ADDRESS
}
/ks_kafka_topic_metric
${
logdate
}
||
\
exit
2
done
...
...
km-dist/init/template/ks_kafka_replication_metric
浏览文件 @
d8aa3d64
...
...
@@ -11,6 +11,49 @@ PUT _template/ks_kafka_replication_metric
},
"mappings" : {
"properties" : {
"brokerId" : {
"type" : "long"
},
"partitionId" : {
"type" : "long"
},
"routingValue" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"clusterPhyId" : {
"type" : "long"
},
"topic" : {
"type" : "keyword"
},
"metrics" : {
"properties" : {
"LogStartOffset" : {
"type" : "float"
},
"Messages" : {
"type" : "float"
},
"LogEndOffset" : {
"type" : "float"
}
}
},
"key" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
"index" : true,
...
...
@@ -20,4 +63,4 @@ PUT _template/ks_kafka_replication_metric
}
},
"aliases" : { }
}
\ No newline at end of file
}
km-dist/init/template/template.sh
浏览文件 @
d8aa3d64
...
...
@@ -439,7 +439,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
esaddr
}
:
${
port
}
/_template/ks_kafka_replication_metric
-d
'{
"order" : 10,
"index_patterns" : [
"ks_kafka_
parti
tion_metric*"
"ks_kafka_
replica
tion_metric*"
],
"settings" : {
"index" : {
...
...
@@ -500,30 +500,7 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
}
},
"aliases" : { }
}[root@10-255-0-23 template]# cat ks_kafka_replication_metric
PUT _template/ks_kafka_replication_metric
{
"order" : 10,
"index_patterns" : [
"ks_kafka_replication_metric*"
],
"settings" : {
"index" : {
"number_of_shards" : "10"
}
},
"mappings" : {
"properties" : {
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
"index" : true,
"type" : "date",
"doc_values" : true
}
}
},
"aliases" : { }
}'
}'
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
esaddr
}
:
${
port
}
/_template/ks_kafka_topic_metric
-d
'{
"order" : 10,
...
...
@@ -640,7 +617,92 @@ curl -s -o /dev/null -X POST -H 'cache-control: no-cache' -H 'content-type: appl
}
},
"aliases" : { }
}'
}'
curl
-s
-o
/dev/null
-X
POST
-H
'cache-control: no-cache'
-H
'content-type: application/json'
http://
${
SERVER_ES_ADDRESS
}
/_template/ks_kafka_zookeeper_metric
-d
'{
"order" : 10,
"index_patterns" : [
"ks_kafka_zookeeper_metric*"
],
"settings" : {
"index" : {
"number_of_shards" : "10"
}
},
"mappings" : {
"properties" : {
"routingValue" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"clusterPhyId" : {
"type" : "long"
},
"metrics" : {
"properties" : {
"AvgRequestLatency" : {
"type" : "double"
},
"MinRequestLatency" : {
"type" : "double"
},
"MaxRequestLatency" : {
"type" : "double"
},
"OutstandingRequests" : {
"type" : "double"
},
"NodeCount" : {
"type" : "double"
},
"WatchCount" : {
"type" : "double"
},
"NumAliveConnections" : {
"type" : "double"
},
"PacketsReceived" : {
"type" : "double"
},
"PacketsSent" : {
"type" : "double"
},
"EphemeralsCount" : {
"type" : "double"
},
"ApproximateDataSize" : {
"type" : "double"
},
"OpenFileDescriptorCount" : {
"type" : "double"
},
"MaxFileDescriptorCount" : {
"type" : "double"
}
}
},
"key" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"timestamp" : {
"format" : "yyyy-MM-dd HH:mm:ss Z||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd HH:mm:ss.SSS Z||yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd HH:mm:ss,SSS Z||yyyy/MM/dd HH:mm:ss,SSS Z||epoch_millis",
"type" : "date"
}
}
},
"aliases" : { }
}'
for
i
in
{
0..6
}
;
do
...
...
@@ -650,6 +712,7 @@ do
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_group_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_partition_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_replication_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_zookeeper_metric
${
logdate
}
&&
\
curl
-s
-o
/dev/null
-X
PUT http://
${
esaddr
}
:
${
port
}
/ks_kafka_topic_metric
${
logdate
}
||
\
exit
2
done
\ No newline at end of file
done
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录