Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
李少辉-开发者
gitlab-foss
提交
3acf7ba9
G
gitlab-foss
项目概览
李少辉-开发者
/
gitlab-foss
通知
15
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
G
gitlab-foss
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
3acf7ba9
编写于
7月 03, 2018
作者:
S
Sean McGivern
提交者:
Oswaldo Ferreira
7月 10, 2018
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Revert "Temporarily remove MR diffs removal migration"
This reverts commit
15ec6a13
.
上级
3d4873fa
变更
3
隐藏空白更改
内联
并排
Showing
3 changed file
with
119 addition
and
1 deletion
+119
-1
db/post_migrate/20180619121030_enqueue_delete_diff_files_workers.rb
...grate/20180619121030_enqueue_delete_diff_files_workers.rb
+70
-0
spec/lib/gitlab/background_migration/delete_diff_files_spec.rb
...lib/gitlab/background_migration/delete_diff_files_spec.rb
+1
-1
spec/migrations/enqueue_delete_diff_files_workers_spec.rb
spec/migrations/enqueue_delete_diff_files_workers_spec.rb
+48
-0
未找到文件。
db/post_migrate/20180619121030_enqueue_delete_diff_files_workers.rb
0 → 100644
浏览文件 @
3acf7ba9
class
EnqueueDeleteDiffFilesWorkers
<
ActiveRecord
::
Migration
include
Gitlab
::
Database
::
MigrationHelpers
class
MergeRequestDiff
<
ActiveRecord
::
Base
self
.
table_name
=
'merge_request_diffs'
belongs_to
:merge_request
include
EachBatch
end
DOWNTIME
=
false
BATCH_SIZE
=
1000
MIGRATION
=
'DeleteDiffFiles'
DELAY_INTERVAL
=
8
.
minutes
TMP_INDEX
=
'tmp_partial_diff_id_with_files_index'
.
freeze
disable_ddl_transaction!
def
up
# We add temporary index, to make iteration over batches more performant.
# Conditional here is to avoid the need of doing that in a separate
# migration file to make this operation idempotent.
#
unless
index_exists_by_name?
(
:merge_request_diffs
,
TMP_INDEX
)
add_concurrent_index
(
:merge_request_diffs
,
:id
,
where:
"(state NOT IN ('without_files', 'empty'))"
,
name:
TMP_INDEX
)
end
diffs_with_files
=
MergeRequestDiff
.
where
.
not
(
state:
[
'without_files'
,
'empty'
])
# explain (analyze, buffers) example for the iteration:
#
# Index Only Scan using tmp_index_20013 on merge_request_diffs (cost=0.43..1630.19 rows=60567 width=4) (actual time=0.047..9.572 rows=56976 loops=1)
# Index Cond: ((id >= 764586) AND (id < 835298))
# Heap Fetches: 8
# Buffers: shared hit=18188
# Planning time: 0.752 ms
# Execution time: 12.430 ms
#
diffs_with_files
.
each_batch
(
of:
BATCH_SIZE
)
do
|
relation
,
outer_index
|
ids
=
relation
.
pluck
(
:id
)
ids
.
each_with_index
do
|
diff_id
,
inner_index
|
# This will give some space between batches of workers.
interval
=
DELAY_INTERVAL
*
outer_index
+
inner_index
.
minutes
# A single `merge_request_diff` can be associated with way too many
# `merge_request_diff_files`. It's better to avoid batching these and
# schedule one at a time.
#
# Considering roughly 6M jobs, this should take ~30 days to process all
# of them.
#
BackgroundMigrationWorker
.
perform_in
(
interval
,
MIGRATION
,
[
diff_id
])
end
end
# We remove temporary index, because it is not required during standard
# operations and runtime.
#
remove_concurrent_index_by_name
(
:merge_request_diffs
,
TMP_INDEX
)
end
def
down
if
index_exists_by_name?
(
:merge_request_diffs
,
TMP_INDEX
)
remove_concurrent_index_by_name
(
:merge_request_diffs
,
TMP_INDEX
)
end
end
end
spec/lib/gitlab/background_migration/delete_diff_files_spec.rb
浏览文件 @
3acf7ba9
require
'spec_helper'
describe
Gitlab
::
BackgroundMigration
::
DeleteDiffFiles
,
:migration
,
schema:
201806
26125654
do
describe
Gitlab
::
BackgroundMigration
::
DeleteDiffFiles
,
:migration
,
schema:
201806
19121030
do
describe
'#perform'
do
context
'when diff files can be deleted'
do
let
(
:merge_request
)
{
create
(
:merge_request
,
:merged
)
}
...
...
spec/migrations/enqueue_delete_diff_files_workers_spec.rb
0 → 100644
浏览文件 @
3acf7ba9
require
'spec_helper'
require
Rails
.
root
.
join
(
'db'
,
'post_migrate'
,
'20180619121030_enqueue_delete_diff_files_workers.rb'
)
describe
EnqueueDeleteDiffFilesWorkers
,
:migration
,
:sidekiq
do
let
(
:merge_request_diffs
)
{
table
(
:merge_request_diffs
)
}
let
(
:merge_requests
)
{
table
(
:merge_requests
)
}
let
(
:namespaces
)
{
table
(
:namespaces
)
}
let
(
:projects
)
{
table
(
:projects
)
}
before
do
stub_const
(
"
#{
described_class
.
name
}
::BATCH_SIZE"
,
2
)
namespaces
.
create!
(
id:
1
,
name:
'gitlab'
,
path:
'gitlab'
)
projects
.
create!
(
id:
1
,
namespace_id:
1
,
name:
'gitlab'
,
path:
'gitlab'
)
merge_requests
.
create!
(
id:
1
,
target_project_id:
1
,
source_project_id:
1
,
target_branch:
'feature'
,
source_branch:
'master'
,
state:
'merged'
)
merge_request_diffs
.
create!
(
id:
1
,
merge_request_id:
1
,
state:
'collected'
)
merge_request_diffs
.
create!
(
id:
2
,
merge_request_id:
1
,
state:
'without_files'
)
merge_request_diffs
.
create!
(
id:
3
,
merge_request_id:
1
,
state:
'collected'
)
merge_request_diffs
.
create!
(
id:
4
,
merge_request_id:
1
,
state:
'collected'
)
merge_request_diffs
.
create!
(
id:
5
,
merge_request_id:
1
,
state:
'empty'
)
merge_request_diffs
.
create!
(
id:
6
,
merge_request_id:
1
,
state:
'collected'
)
merge_requests
.
update
(
1
,
latest_merge_request_diff_id:
6
)
end
it
'correctly schedules diff file deletion workers'
do
Sidekiq
::
Testing
.
fake!
do
Timecop
.
freeze
do
migrate!
# 1st batch
expect
(
described_class
::
MIGRATION
).
to
be_scheduled_delayed_migration
(
8
.
minutes
,
1
)
expect
(
described_class
::
MIGRATION
).
to
be_scheduled_delayed_migration
(
9
.
minutes
,
3
)
# 2nd batch
expect
(
described_class
::
MIGRATION
).
to
be_scheduled_delayed_migration
(
16
.
minutes
,
4
)
expect
(
described_class
::
MIGRATION
).
to
be_scheduled_delayed_migration
(
17
.
minutes
,
6
)
expect
(
BackgroundMigrationWorker
.
jobs
.
size
).
to
eq
(
4
)
end
end
end
it
'migrates the data'
do
expect
{
migrate!
}.
to
change
{
merge_request_diffs
.
where
(
state:
'without_files'
).
count
}
.
from
(
1
).
to
(
4
)
end
end
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录