Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
疯人忠
Cvat
提交
7720a8f6
C
Cvat
项目概览
疯人忠
/
Cvat
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
C
Cvat
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
7720a8f6
编写于
2月 15, 2021
作者:
P
PMazarovich
提交者:
GitHub
2月 15, 2021
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Logstash persistent queue improvement (#2744)
Co-authored-by:
N
Andrey Zhavoronkov
<
andrey.zhavoronkov@intel.com
>
上级
e43707d7
变更
7
隐藏空白更改
内联
并排
Showing
7 changed file
with
43 addition
and
8 deletion
+43
-8
CHANGELOG.md
CHANGELOG.md
+1
-0
components/analytics/docker-compose.analytics.yml
components/analytics/docker-compose.analytics.yml
+1
-1
components/analytics/logstash/Dockerfile
components/analytics/logstash/Dockerfile
+3
-2
components/analytics/logstash/logstash.conf
components/analytics/logstash/logstash.conf
+19
-2
components/analytics/logstash/logstash.yml
components/analytics/logstash/logstash.yml
+3
-0
cvat/requirements/base.txt
cvat/requirements/base.txt
+1
-1
cvat/settings/base.py
cvat/settings/base.py
+15
-2
未找到文件。
CHANGELOG.md
浏览文件 @
7720a8f6
...
...
@@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
-
Logstash is improved for using with configurable elasticsearch outputs (
<https://github.com/openvinotoolkit/cvat/pull/2531>
)
-
Bumped nuclio version to 1.5.16
-
All methods for interative segmentation accept negative points as well
-
Persistent queue added to logstash (
<https://github.com/openvinotoolkit/cvat/pull/2744>
)
### Deprecated
...
...
components/analytics/docker-compose.analytics.yml
浏览文件 @
7720a8f6
...
...
@@ -79,7 +79,7 @@ services:
cvat
:
environment
:
DJANGO_LOG_SERVER_HOST
:
logstash
DJANGO_LOG_SERVER_PORT
:
500
0
DJANGO_LOG_SERVER_PORT
:
808
0
DJANGO_LOG_VIEWER_HOST
:
kibana
DJANGO_LOG_VIEWER_PORT
:
5601
CVAT_ANALYTICS
:
1
...
...
components/analytics/logstash/Dockerfile
浏览文件 @
7720a8f6
...
...
@@ -3,5 +3,6 @@ FROM docker.elastic.co/logstash/logstash-oss:${ELK_VERSION}
RUN
logstash-plugin
install
logstash-input-http logstash-filter-aggregate
\
logstash-filter-prune logstash-output-email
COPY
logstash.conf /usr/share/logstash/pipeline/
EXPOSE
5000
COPY
logstash.yml /usr/share/logstash/config/
COPY
logstash.conf /usr/share/logstash/pipeline/
EXPOSE
8080
components/analytics/logstash/logstash.conf
浏览文件 @
7720a8f6
input
{
tc
p
{
port
=>
500
0
htt
p
{
port
=>
808
0
codec
=>
json
}
}
filter
{
mutate
{
add_field
=> {
"logger_name"
=>
""
}
add_field
=> {
"path"
=>
""
}
}
mutate
{
copy
=> {
"[extra][logger_name]"
=>
"logger_name"
}
copy
=> {
"[extra][path]"
=>
"path"
}
}
prune
{
blacklist_names
=> [
"type"
,
"logsource"
,
"extra"
,
"program"
,
"pid"
,
"headers"
]
}
if
[
logger_name
] =~ /
cvat
.
client
/ {
# 1. Decode the event from json in 'message' field
# 2. Remove unnecessary field from it
...
...
@@ -14,6 +25,9 @@ filter {
mutate
{
rename
=> {
"message"
=>
"source_message"
}
}
mutate
{
add_field
=> {
"[@metadata][target_index_client]"
=>
"cvat.client.%{+YYYY}.%{+MM}"
}
}
json
{
source
=>
"source_message"
...
...
@@ -77,6 +91,9 @@ filter {
# 2. Remove unnecessary field from it
# 3. Type it as server
if
[
logger_name
] =~ /
cvat
\.
server
\.
task_
[
0
-
9
]+/ {
mutate
{
add_field
=> {
"[@metadata][target_index_server]"
=>
"cvat.server.%{+YYYY}.%{+MM}"
}
}
mutate
{
rename
=> {
"logger_name"
=>
"task_id"
}
gsub
=> [
"task_id"
,
"cvat.server.task_"
,
""
]
...
...
components/analytics/logstash/logstash.yml
0 → 100644
浏览文件 @
7720a8f6
queue.type
:
persisted
queue.max_bytes
:
1gb
queue.checkpoint.writes
:
20
cvat/requirements/base.txt
浏览文件 @
7720a8f6
...
...
@@ -20,7 +20,7 @@ rq-scheduler==0.10.0
sqlparse==0.3.1
django-sendfile==0.3.11
dj-pagination==2.5.0
python-logstash
==0.4.6
python-logstash
-async==2.2.0
django-revproxy==0.10.0
rules==2.2
GitPython==3.1.8
...
...
cvat/settings/base.py
浏览文件 @
7720a8f6
...
...
@@ -336,7 +336,10 @@ STATIC_ROOT = os.path.join(BASE_DIR, 'static')
os
.
makedirs
(
STATIC_ROOT
,
exist_ok
=
True
)
DATA_ROOT
=
os
.
path
.
join
(
BASE_DIR
,
'data'
)
LOGSTASH_DB
=
os
.
path
.
join
(
DATA_ROOT
,
'logstash.db'
)
os
.
makedirs
(
DATA_ROOT
,
exist_ok
=
True
)
if
not
os
.
path
.
exists
(
LOGSTASH_DB
):
os
.
mknod
(
LOGSTASH_DB
)
MEDIA_DATA_ROOT
=
os
.
path
.
join
(
DATA_ROOT
,
'data'
)
os
.
makedirs
(
MEDIA_DATA_ROOT
,
exist_ok
=
True
)
...
...
@@ -366,6 +369,11 @@ LOGGING = {
'version'
:
1
,
'disable_existing_loggers'
:
False
,
'formatters'
:
{
'logstash'
:
{
'()'
:
'logstash_async.formatter.DjangoLogstashFormatter'
,
'message_type'
:
'python-logstash'
,
'fqdn'
:
False
,
# Fully qualified domain name. Default value: false.
},
'standard'
:
{
'format'
:
'[%(asctime)s] %(levelname)s %(name)s: %(message)s'
}
...
...
@@ -386,11 +394,16 @@ LOGGING = {
},
'logstash'
:
{
'level'
:
'INFO'
,
'class'
:
'logstash.TCPLogstashHandler'
,
'class'
:
'logstash_async.handler.AsynchronousLogstashHandler'
,
'formatter'
:
'logstash'
,
'transport'
:
'logstash_async.transport.HttpTransport'
,
'ssl_enable'
:
False
,
'ssl_verify'
:
False
,
'host'
:
os
.
getenv
(
'DJANGO_LOG_SERVER_HOST'
,
'localhost'
),
'port'
:
os
.
getenv
(
'DJANGO_LOG_SERVER_PORT'
,
500
0
),
'port'
:
os
.
getenv
(
'DJANGO_LOG_SERVER_PORT'
,
808
0
),
'version'
:
1
,
'message_type'
:
'django'
,
'database_path'
:
LOGSTASH_DB
,
}
},
'loggers'
:
{
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录