Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Greenplum
DeepSpeed
提交
ccaa9901
D
DeepSpeed
项目概览
Greenplum
/
DeepSpeed
上一次同步 大约 1 年
通知
10
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
DeepSpeed
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
ccaa9901
编写于
5月 14, 2020
作者:
S
Shaden Smith
提交者:
GitHub
5月 14, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
reverting test matrix to debug node (#215)
上级
57678bed
变更
1
隐藏空白更改
内联
并排
Showing
1 changed file
with
53 addition
and
67 deletion
+53
-67
azure-pipelines.yml
azure-pipelines.yml
+53
-67
未找到文件。
azure-pipelines.yml
浏览文件 @
ccaa9901
jobs
:
-
job
:
De
epSpeed_Tests
-
job
:
De
fault
timeoutInMinutes
:
360
pool
:
name
:
'
DS
_testing'
name
:
'
GPU
_testing'
strategy
:
matrix
:
Py
Torch12
:
Py
thon36
:
python.version
:
'
3.6'
cuda.version
:
'
10.0'
pytorch.version
:
'
1.2'
runmodeltests
:
false
PyTorch15
:
python.version
:
'
3.7'
cuda.version
:
'
10.1'
pytorch.version
:
'
1.5'
runmodeltests
:
true
#Python35:
# python.version: '3.5'
#Python37:
# python.version: '3.7'
#Python38:
# python.version: '3.8'
variables
:
conda_root
:
'
/home/deepspeed/miniconda3'
conda_env
:
'
ds_test_py$(python.version)_cuda$(cuda.version)_pytorch$(pytorch.version)'
conda_env_path
:
'
$(conda_root)/envs/$(conda_env)'
steps
:
# Unfortunately nvidia's nvcc_linux-64=<version> seems to install 10.1 regardless?
# Most of this complexity is a workaround to get the compiler toolchain to match the
# cudatoolkit runtime
-
task
:
UsePythonVersion@0
inputs
:
versionSpec
:
'
$(python.version)'
addToPath
:
true
architecture
:
'
x64'
displayName
:
'
Use
Python
$(python.version)'
-
script
:
|
conda create --force --yes -n $(conda_env) python=$(python.version) cudatoolkit=$(cuda.version)
source $(conda_root)/bin/activate $(conda_env_path)
conda install --yes -c pytorch pytorch=$(pytorch.version) cudatoolkit=$(cuda.version)
conda install --yes conda
conda install --yes gxx_linux-64
conda install --yes -c conda-forge cudatoolkit-dev=$(cuda.version)
displayName
:
'
Setup
environment
python=$(python.version)
pytorch=$(pytorch.version)
cuda=$(cuda.version)'
python -m pip install --upgrade pip
pip install --user -r requirements.txt
./install.sh --pip_sudo
displayName
:
'
Install
dependencies'
-
script
:
|
source $(conda_root)/bin/activate $(conda_env_path)
python --version
which nvcc
nvcc --version
python -c "import torch; print('torch:', torch.__version__)"
displayName
:
'
Show
environment'
pre-commit run --all-files
displayName
:
'
Formatting
checks'
-
script
:
|
source $(conda_root)/bin/activate $(conda_env_path)
rm -rf third_party/apex/build/
rm -rf third_party/apex/dist/
rm -rf build/
rm -rf dist/
./install.sh
displayName
:
'
Install
DeepSpeed'
pip install --user pylint
pylint --exit-zero deepspeed/
displayName
:
'
Code
linter'
-
script
:
|
source $(conda_root)/bin/activate $(conda_env_path)
pytest --forked --verbose tests/unit/
displayName
:
'
Unit
tests'
-
script
:
|
source $(conda_root)/bin/activate $(conda_env_path)
ln -s /data/Megatron-LM/data DeepSpeedExamples/Megatron-LM/
pip install -r DeepSpeedExamples/Megatron-LM/requirements.txt
pip install -
-user -
r DeepSpeedExamples/Megatron-LM/requirements.txt
cd tests/model/
rm -rf BingBertSquad/baseline
rm -rf Megatron_GPT2/baseline
pytest -s run_sanity_check.py
condition
:
eq(variables['runmodeltests'],
true
)
displayName
:
'
Model
tests'
#BingBertSquad logs
...
...
@@ -75,31 +57,35 @@ jobs:
targetPath
:
'
$(Build.SourcesDirectory)/tests/model/BingBertSquad/test/'
artifactName
:
BingBertSquad_logs
displayName
:
'
BingBertSquad
log
uploads'
condition
:
eq(variables['runmodeltests'],
true
)
condition
:
always(
)
# Megatron test logs
#- task: PublishPipelineArtifact@1
# inputs:
# targetPath: '$(Build.SourcesDirectory)/tests/model/Megatron_GPT2/test/'
# artifactName: Megatron_GPT2_logs
# displayName: 'Megatron GPT2 log uploads'
# condition: always()
-
job
:
Code_Quality_Checks
pool
:
name
:
'
DS_testing'
variables
:
conda_root
:
'
/home/deepspeed/miniconda3'
conda_env
:
'
ds_codetest'
conda_env_path
:
'
$(conda_root)/envs/$(conda_env)'
#- task: PublishPipelineArtifact@1
# inputs:
# targetPath: '$(Build.SourcesDirectory)/tests/model/Megatron_GPT2/checkpoint_test_logs/'
# artifactName: Megatron_GPT2_checkpoint_logs
# displayName: 'Megatron GPT2 checkpoint log uploads'
# condition: always()
steps
:
-
script
:
|
conda create --force --yes -n $(conda_env) python=3.7
source $(conda_root)/bin/activate $(conda_env_path)
displayName
:
'
Create
code
test
environment'
-
script
:
|
source $(conda_root)/bin/activate $(conda_env_path)
pip install pre-commit
pre-commit run --all-files
displayName
:
'
Formatting
checks'
#BingBert logs
#- task: PublishPipelineArtifact@1
# inputs:
# targetPath: '$(Build.SourcesDirectory)/tests/model/bing_bert/pretrain_test/'
# artifactName: BingBert_pretrain_logs
# displayName: 'BingBert pretrain logs'
# condition: always()
-
script
:
|
source $(conda_root)/bin/activate $(conda_env_path)
pip install pylint
pylint --exit-zero deepspeed/
displayName
:
'
Code
linter'
#- task: PublishPipelineArtifact@1
# inputs:
# targetPath: '$(Build.SourcesDirectory)/tests/model/bing_bert/checkpoint_test_logs/'
# artifactName: BingBert_checkpoint_logs
# displayName: 'BingBert checkpoint logs'
# condition: always()
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录