Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
milvus
milvus
提交
23208432
M
milvus
项目概览
milvus
/
milvus
12 个月 前同步成功
通知
261
Star
22476
Fork
2472
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
M
milvus
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
23208432
编写于
4月 20, 2022
作者:
Z
zhuwenxing
提交者:
GitHub
4月 20, 2022
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
[skip e2e]Add chaos test for kafka version milvus (#16522)
Signed-off-by:
N
zhuwenxing
<
wenxing.zhu@zilliz.com
>
上级
9769426e
变更
4
隐藏空白更改
内联
并排
Showing
4 changed file
with
267 addition
and
0 deletion
+267
-0
.github/workflows/pod-kill-chaos-test-kafka-version.yaml
.github/workflows/pod-kill-chaos-test-kafka-version.yaml
+226
-0
tests/python_client/chaos/chaos_objects/pod_kill/chaos_kafka_pod_kill.yaml
...nt/chaos/chaos_objects/pod_kill/chaos_kafka_pod_kill.yaml
+22
-0
tests/python_client/chaos/chaos_objects/pod_kill/testcases.yaml
...python_client/chaos/chaos_objects/pod_kill/testcases.yaml
+13
-0
tests/python_client/chaos/cluster-values.yaml
tests/python_client/chaos/cluster-values.yaml
+6
-0
未找到文件。
.github/workflows/pod-kill-chaos-test-kafka-version.yaml
0 → 100644
浏览文件 @
23208432
name
:
Pod Kill Chaos Test For Kafka Version
on
:
workflow_dispatch
:
inputs
:
image_tag
:
description
:
The image tag to use for the chaos test
required
:
true
default
:
'
master-latest'
schedule
:
-
cron
:
"
30
17
*
*
*"
jobs
:
test-pod-kill-chaos
:
runs-on
:
ubuntu-latest
timeout-minutes
:
40
strategy
:
fail-fast
:
false
matrix
:
pod
:
[
allcluster
,
datacoord
,
datanode
,
indexcoord
,
indexnode
,
proxy
,
kafka
,
querycoord
,
querynode
,
rootcoord
,
etcd
,
minio
]
steps
:
-
name
:
Set env param
env
:
DEFAULT_IMAGE_TAG
:
master-latest
run
:
|
echo "RELEASE=test-${{ matrix.pod }}-pod-kill" >> $GITHUB_ENV
echo "IMAGE_TAG=${{ github.event.inputs.image_tag || env.DEFAULT_IMAGE_TAG}}" >> $GITHUB_ENV
-
name
:
Creating kind cluster
uses
:
helm/kind-action@v1.2.0
-
name
:
Print cluster information
run
:
|
kubectl config view
kubectl cluster-info
kubectl get nodes
kubectl get pods -n kube-system
helm version
kubectl version
-
uses
:
actions/checkout@v2
-
name
:
Set up Python
uses
:
actions/setup-python@v2
with
:
python-version
:
3.8
-
name
:
Install dependency
uses
:
nick-invision/retry@v2
with
:
timeout_minutes
:
5
max_attempts
:
3
retry_on
:
error
shell
:
bash
command
:
|
pip install -r tests/python_client/requirements.txt --trusted-host https://test.pypi.org
pip install --upgrade protobuf
-
name
:
Deploy Chaos Mesh
timeout-minutes
:
2
shell
:
bash
run
:
|
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm search repo chaos-mesh
kubectl create ns chaos-testing
helm install --wait --timeout 360s chaos-mesh chaos-mesh/chaos-mesh --namespace=chaos-testing --version v2.0.3 --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock
kubectl get po -n chaos-testing
-
name
:
Deploy Milvus
timeout-minutes
:
15
shell
:
bash
working-directory
:
tests/python_client/chaos
run
:
|
echo "latest tag:"
bash ../../../scripts/docker_image_find_tag.sh -n milvusdb/milvus-dev -t master-latest -f master- -F -L -q
helm repo add milvus https://milvus-io.github.io/milvus-helm
helm repo update
if [[ ${{ matrix.pod }} != *"standalone"* ]]; then helm install --wait --timeout 720s ${{ env.RELEASE }} milvus/milvus --set pulsar.enabled=false --set kafka.enabled=true --set image.all.tag=${{ env.IMAGE_TAG }} -f cluster-values.yaml -n=chaos-testing; fi
if [[ ${{ matrix.pod }} == *"standalone"* ]]; then helm install --wait --timeout 720s ${{ env.RELEASE }} milvus/milvus --set image.all.tag=${{ env.IMAGE_TAG }} -f standalone-values.yaml -n=chaos-testing; fi
kubectl get pods -n chaos-testing
sleep 20s
kubectl get pods -n chaos-testing
kubectl port-forward service/${{ env.RELEASE }}-milvus 19530 -n chaos-testing >/dev/null 2>&1 &
sleep 20s
# check whether port-forward success
nc -vz 127.0.0.1 19530
# check whether milvus server is healthy
python scripts/hello_milvus.py
-
name
:
Chaos Test
timeout-minutes
:
15
shell
:
bash
working-directory
:
tests/python_client/chaos
run
:
|
# replace chaos object
sed -i "s/TESTS_CONFIG_LOCATION =.*/TESTS_CONFIG_LOCATION = \'chaos_objects\/pod_kill\/'/g" constants.py
sed -i "s/ALL_CHAOS_YAMLS =.*/ALL_CHAOS_YAMLS = \'chaos_${{ matrix.pod }}_pod_kill.yaml\'/g" constants.py
sed -i "s/RELEASE_NAME =.*/RELEASE_NAME = \'${{ env.RELEASE }}\'/g" constants.py
cat constants.py
timeout 14m pytest -s -v test_chaos.py --host 127.0.0.1 --log-cli-level=INFO --capture=no || echo "chaos test failed"
-
name
:
Result Analysis
timeout-minutes
:
1
shell
:
bash
working-directory
:
tests/python_client/chaos/reports
run
:
|
echo "result analysis"
cat ${{ env.RELEASE }}.log || echo "no log file"
-
name
:
Wait all pods ready
timeout-minutes
:
5
shell
:
bash
working-directory
:
tests/python_client
run
:
|
kubectl get pod -n chaos-testing
# wait all pod to be ready
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/instance=${{ env.RELEASE }} -n chaos-testing --timeout=360s
kubectl wait --for=condition=Ready pod -l release=${{ env.RELEASE }} -n chaos-testing --timeout=360s
kubectl get pod -n chaos-testing
ps aux|grep forward|grep -v grep|awk '{print $2}'|xargs kill -9
kubectl port-forward service/${{ env.RELEASE }}-milvus 19530 -n chaos-testing >/dev/null 2>&1 &
sleep 20s
nc -vz 127.0.0.1 19530
-
name
:
Run e2e test after chaos
timeout-minutes
:
5
shell
:
bash
working-directory
:
tests/python_client
run
:
|
pytest -s -v testcases/test_e2e.py --host 127.0.0.1 --log-cli-level=INFO --capture=no
-
name
:
Run hello_milvus after chaos
timeout-minutes
:
5
shell
:
bash
working-directory
:
tests/python_client
run
:
|
python chaos/scripts/hello_milvus.py --host 127.0.0.1
-
name
:
Verify all collections after chaos
timeout-minutes
:
15
shell
:
bash
working-directory
:
tests/python_client
run
:
|
python chaos/scripts/verify_all_collections.py --host 127.0.0.1
-
name
:
Export logs
if
:
${{ always() }}
shell
:
bash
working-directory
:
tests/python_client/chaos
run
:
|
#in this step, verify whether pod has been killed by pod's age
kubectl get po -n chaos-testing
# export k8s log for chaos mesh and milvus
bash ../../scripts/export_log_k8s.sh chaos-testing ${{ env.RELEASE }} k8s_logs/chaos-test
-
name
:
Deploy Milvus Again If Previous E2E Test Failed
timeout-minutes
:
15
if
:
${{ failure() }}
shell
:
bash
working-directory
:
tests/python_client/chaos
run
:
|
kubectl config set-context --current --namespace=chaos-testing
bash scripts/uninstall_milvus.sh ${{ env.RELEASE }}
if [ ${{ matrix.pod }} != "standalone" ]; then helm install --wait --timeout 720s ${{ env.RELEASE }} milvus/milvus -f cluster-values.yaml -n=chaos-testing; fi
if [ ${{ matrix.pod }} == "standalone" ]; then helm install --wait --timeout 720s ${{ env.RELEASE }} milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false -n=chaos-testing; fi
kubectl get pods -n chaos-testing
sleep 20s
kubectl get pods -n chaos-testing
ps aux|grep forward|grep -v grep|awk '{print $2}'|xargs kill -9
kubectl port-forward service/${{ env.RELEASE }}-milvus 19530 -n chaos-testing >/dev/null 2>&1 &
sleep 20s
# check whether port-forward success
nc -vz 127.0.0.1 19530
# check whether milvus server is healthy
python scripts/hello_milvus.py
-
name
:
Data Consist Test
timeout-minutes
:
5
if
:
${{ always() }}
shell
:
bash
working-directory
:
tests/python_client/chaos
run
:
|
pytest -s -v test_chaos_data_consist.py --host 127.0.0.1 --log-cli-level=INFO --capture=no
-
name
:
Milvus E2E Test
timeout-minutes
:
10
if
:
${{ always() }}
shell
:
bash
working-directory
:
tests/python_client
run
:
|
kubectl get pod -n chaos-testing
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/instance=${{ env.RELEASE }} -n chaos-testing --timeout=360s
kubectl wait --for=condition=Ready pod -l release=${{ env.RELEASE }} -n chaos-testing --timeout=360s
kubectl get pod -n chaos-testing
ps aux|grep forward|grep -v grep|awk '{print $2}'|xargs kill -9
kubectl port-forward service/${{ env.RELEASE }}-milvus 19530 -n chaos-testing >/dev/null 2>&1 &
sleep 20s
nc -vz 127.0.0.1 19530
pytest -s -v testcases/test_e2e.py --host 127.0.0.1 --log-cli-level=INFO --capture=no
python chaos/scripts/hello_milvus.py --host 127.0.0.1
-
name
:
Export logs
if
:
${{ always() }}
shell
:
bash
working-directory
:
tests/python_client/chaos
run
:
|
#in this step, verify whether pod has been killed by pod's age
kubectl get po -n chaos-testing
# export k8s log for chaos mesh and milvus
bash ../../scripts/export_log_k8s.sh chaos-testing ${{ env.RELEASE }} k8s_logs/data-consist-test
bash ../../scripts/export_log_k8s.sh chaos-testing chaos-daemon k8s_logs/chaos-mesh-daemon
-
name
:
Upload logs
if
:
${{ always() }}
uses
:
actions/upload-artifact@v2
with
:
name
:
logs-${{ matrix.pod }}
path
:
|
tests/python_client/chaos/k8s_logs
tests/python_client/chaos/reports
tests/python_client/chaos/chaos_objects/pod_kill/chaos_kafka_pod_kill.yaml
0 → 100644
浏览文件 @
23208432
kind
:
Schedule
apiVersion
:
chaos-mesh.org/v1alpha1
metadata
:
name
:
test-kafka-pod-kill
namespace
:
chaos-testing
spec
:
schedule
:
'
*/5
*
*
*
*
*'
startingDeadlineSeconds
:
60
concurrencyPolicy
:
Forbid
historyLimit
:
1
type
:
PodChaos
podChaos
:
selector
:
namespaces
:
-
chaos-testing
labelSelectors
:
app.kubernetes.io/instance
:
milvus-chaos
app.kubernetes.io/name
:
kafka
mode
:
fixed
value
:
"
1"
action
:
pod-kill
gracePeriod
:
0
\ No newline at end of file
tests/python_client/chaos/chaos_objects/pod_kill/testcases.yaml
浏览文件 @
23208432
...
...
@@ -161,6 +161,19 @@ Collections:
testcase
:
name
:
test_allcluster_pod_kill
chaos
:
chaos_allcluster_pod_kill.yaml
expectation
:
cluster_1_node
:
create
:
fail
insert
:
fail
flush
:
fail
index
:
fail
search
:
fail
query
:
fail
-
testcase
:
name
:
test_kafka_pod_kill
chaos
:
chaos_kafka_pod_kill.yaml
expectation
:
cluster_1_node
:
create
:
fail
...
...
tests/python_client/chaos/cluster-values.yaml
浏览文件 @
23208432
...
...
@@ -12,6 +12,12 @@ minio:
resources
:
requests
:
memory
:
256Mi
kafka
:
enabled
:
false
name
:
kafka
replicaCount
:
3
pulsar
:
enabled
:
true
extra
:
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录