Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
2dot5
ClickHouse
提交
02b12cd3
C
ClickHouse
项目概览
2dot5
/
ClickHouse
通知
3
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
C
ClickHouse
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
前往新版Gitcode,体验更适合开发者的 AI 搜索 >>
未验证
提交
02b12cd3
编写于
2月 26, 2021
作者:
K
Kseniia Sumarokova
提交者:
GitHub
2月 26, 2021
浏览文件
操作
浏览文件
下载
差异文件
Merge pull request #21138 from kssenii/rabbit-fix
rabbitmq: fix bug when late onReady callback causes duplicates
上级
6a5ab409
1247c0f3
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
33 addition
and
28 deletion
+33
-28
src/Storages/RabbitMQ/StorageRabbitMQ.cpp
src/Storages/RabbitMQ/StorageRabbitMQ.cpp
+1
-1
src/Storages/RabbitMQ/WriteBufferToRabbitMQProducer.cpp
src/Storages/RabbitMQ/WriteBufferToRabbitMQProducer.cpp
+31
-27
src/Storages/RabbitMQ/WriteBufferToRabbitMQProducer.h
src/Storages/RabbitMQ/WriteBufferToRabbitMQProducer.h
+1
-0
未找到文件。
src/Storages/RabbitMQ/StorageRabbitMQ.cpp
浏览文件 @
02b12cd3
...
...
@@ -482,7 +482,7 @@ bool StorageRabbitMQ::restoreConnection(bool reconnecting)
/* Connection is not closed immediately (firstly, all pending operations are completed, and then
* an AMQP closing-handshake is performed). But cannot open a new connection until previous one is properly closed
*/
while
(
!
connection
->
closed
()
&&
++
cnt_retries
!=
RETRIES_MAX
)
while
(
!
connection
->
closed
()
&&
cnt_retries
++
!=
RETRIES_MAX
)
event_handler
->
iterateLoop
();
/// This will force immediate closure if not yet closed
...
...
src/Storages/RabbitMQ/WriteBufferToRabbitMQProducer.cpp
浏览文件 @
02b12cd3
...
...
@@ -97,7 +97,7 @@ WriteBufferToRabbitMQProducer::~WriteBufferToRabbitMQProducer()
connection
->
close
();
size_t
cnt_retries
=
0
;
while
(
!
connection
->
closed
()
&&
++
cnt_retries
!=
RETRIES_MAX
)
while
(
!
connection
->
closed
()
&&
cnt_retries
++
!=
RETRIES_MAX
)
{
event_handler
->
iterateLoop
();
std
::
this_thread
::
sleep_for
(
std
::
chrono
::
milliseconds
(
CONNECT_SLEEP
));
...
...
@@ -189,11 +189,12 @@ void WriteBufferToRabbitMQProducer::setupChannel()
/// Delivery tags are scoped per channel.
delivery_record
.
clear
();
delivery_tag
=
0
;
producer_ready
=
false
;
});
producer_channel
->
onReady
([
&
]()
{
channel_id
=
channel_id_base
+
std
::
to_string
(
channel_id_counter
++
);
channel_id
=
channel_id_base
+
"_"
+
std
::
to_string
(
channel_id_counter
++
);
LOG_DEBUG
(
log
,
"Producer's channel {} is ready"
,
channel_id
);
/* if persistent == true, onAck is received when message is persisted to disk or when it is consumed on every queue. If fails,
...
...
@@ -211,6 +212,7 @@ void WriteBufferToRabbitMQProducer::setupChannel()
{
removeRecord
(
nacked_delivery_tag
,
multiple
,
true
);
});
producer_ready
=
true
;
});
}
...
...
@@ -218,30 +220,27 @@ void WriteBufferToRabbitMQProducer::setupChannel()
void
WriteBufferToRabbitMQProducer
::
removeRecord
(
UInt64
received_delivery_tag
,
bool
multiple
,
bool
republish
)
{
auto
record_iter
=
delivery_record
.
find
(
received_delivery_tag
);
assert
(
record_iter
!=
delivery_record
.
end
());
if
(
record_iter
!=
delivery_record
.
end
()
)
if
(
multiple
)
{
if
(
multiple
)
{
/// If multiple is true, then all delivery tags up to and including current are confirmed (with ack or nack).
++
record_iter
;
/// If multiple is true, then all delivery tags up to and including current are confirmed (with ack or nack).
++
record_iter
;
if
(
republish
)
for
(
auto
record
=
delivery_record
.
begin
();
record
!=
record_iter
;
++
record
)
returned
.
tryPush
(
record
->
second
);
if
(
republish
)
for
(
auto
record
=
delivery_record
.
begin
();
record
!=
record_iter
;
++
record
)
returned
.
tryPush
(
record
->
second
);
/// Delete the records even in case when republished because new delivery tags will be assigned by the server.
delivery_record
.
erase
(
delivery_record
.
begin
(),
record_iter
);
}
else
{
if
(
republish
)
returned
.
tryPush
(
record_iter
->
second
);
/// Delete the records even in case when republished because new delivery tags will be assigned by the server.
delivery_record
.
erase
(
delivery_record
.
begin
(),
record_iter
);
}
else
{
if
(
republish
)
returned
.
tryPush
(
record_iter
->
second
);
delivery_record
.
erase
(
record_iter
);
}
delivery_record
.
erase
(
record_iter
);
}
/// else is theoretically not possible
}
...
...
@@ -308,13 +307,18 @@ void WriteBufferToRabbitMQProducer::writingFunc()
{
while
((
!
payloads
.
empty
()
||
wait_all
)
&&
wait_confirm
.
load
())
{
/* Publish main paylods only when there are no returned messages. This way it is ensured that returned messages are republished
* as fast as possible and no new publishes are made before returned messages are handled
*/
if
(
!
returned
.
empty
()
&&
producer_channel
->
usable
())
publish
(
returned
,
true
);
else
if
(
!
payloads
.
empty
()
&&
producer_channel
->
usable
())
publish
(
payloads
,
false
);
/// If onReady callback is not received, producer->usable() will anyway return true,
/// but must publish only after onReady callback.
if
(
producer_ready
)
{
/* Publish main paylods only when there are no returned messages. This way it is ensured that returned messages are republished
* as fast as possible and no new publishes are made before returned messages are handled
*/
if
(
!
returned
.
empty
()
&&
producer_channel
->
usable
())
publish
(
returned
,
true
);
else
if
(
!
payloads
.
empty
()
&&
producer_channel
->
usable
())
publish
(
payloads
,
false
);
}
iterateEventLoop
();
...
...
src/Storages/RabbitMQ/WriteBufferToRabbitMQProducer.h
浏览文件 @
02b12cd3
...
...
@@ -73,6 +73,7 @@ private:
std
::
unique_ptr
<
RabbitMQHandler
>
event_handler
;
std
::
unique_ptr
<
AMQP
::
TcpConnection
>
connection
;
std
::
unique_ptr
<
AMQP
::
TcpChannel
>
producer_channel
;
bool
producer_ready
=
false
;
/// Channel errors lead to channel closure, need to count number of recreated channels to update channel id
UInt64
channel_id_counter
=
0
;
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录