Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
Greenplum
DeepSpeed
提交
16be6de6
D
DeepSpeed
项目概览
Greenplum
/
DeepSpeed
上一次同步 大约 1 年
通知
10
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
D
DeepSpeed
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
未验证
提交
16be6de6
编写于
1月 31, 2020
作者:
J
Jeff Rasley
提交者:
GitHub
1月 31, 2020
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
Install script
上级
26e5ecab
变更
1
显示空白变更内容
内联
并排
Showing
1 changed file
with
137 addition
and
0 deletion
+137
-0
install.sh
install.sh
+137
-0
未找到文件。
install.sh
0 → 100644
浏览文件 @
16be6de6
#!/bin/bash
set
-e
err_report
()
{
echo
"Error on line
$1
"
echo
"Fail to install deepspeed"
}
trap
'err_report $LINENO'
ERR
usage
()
{
echo
"""
Usage: install.sh [options...]
By default will install deepspeed and all third party dependecies accross all machines listed in
hostfile (hostfile: /job/hostfile). If no hostfile exists, will only install locally
[optional]
-d, --deepspeed_only Install only deepspeed and no third party dependencies
-t, --third_party_only Install only third party dependencies and not deepspeed
-l, --local_only Installs only on local machine
-h, --help This help text
"""
}
ds_only
=
0
tp_only
=
0
deepspeed_install
=
1
third_party_install
=
1
local_only
=
0
entire_dlts_job
=
1
while
[[
$#
-gt
0
]]
do
key
=
"
$1
"
case
$key
in
-d
|
--deepspeed_only
)
deepspeed_install
=
1
;
third_party_install
=
0
;
ds_only
=
1
;
shift
;;
-t
|
--third_party_only
)
deepspeed_install
=
0
;
third_party_install
=
1
;
tp_only
=
1
;
shift
;;
-l
|
--local_only
)
local_only
=
1
;
shift
;;
-h
|
--help
)
usage
exit
0
;;
*
)
echo
"Unkown argument(s)"
usage
exit
1
shift
;;
esac
done
if
[
"
$ds_only
"
==
"1"
]
&&
[
"
$tp_only
"
==
"1"
]
;
then
echo
"-d and -t are mutually exclusive, only choose one or none"
usage
exit
1
fi
echo
"Updating git hash/branch info"
echo
"git_hash = '
$(
git rev-parse
--short
HEAD
)
'"
>
deepspeed/version_info.py
echo
"git_branch = '
$(
git rev-parse
--abbrev-ref
HEAD
)
'"
>>
deepspeed/version_info.py
cat
deepspeed/version_info.py
install_apex
=
'sudo -H pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" third_party/apex'
if
[
!
-f
/job/hostfile
]
;
then
echo
"No hostfile exists at /job/hostfile, installing locally"
local_only
=
1
fi
if
[
"
$local_only
"
==
"1"
]
;
then
if
[
"
$third_party_install
"
==
"1"
]
;
then
echo
"Checking out sub-module(s)"
git submodule update
--init
--recursive
echo
"Building apex wheel"
cd
third_party/apex
python setup.py
--cpp_ext
--cuda_ext
bdist_wheel
cd
-
echo
"Installing apex"
sudo
-H
pip uninstall
-y
apex
sudo
-H
pip
install
third_party/apex/dist/apex
*
.whl
fi
if
[
"
$deepspeed_install
"
==
"1"
]
;
then
echo
"Installing deepspeed"
python setup.py bdist_wheel
sudo
-H
pip uninstall
-y
deepspeed
sudo
-H
pip
install
dist/deepspeed
*
.whl
python
-c
'import deepspeed; print("deepspeed info:", deepspeed.__version__, deepspeed.__git_branch__, deepspeed.__git_hash__)'
echo
"Installation is successful"
fi
else
local_path
=
`
pwd
`
hostfile
=
/job/hostfile
if
[
-f
$hostfile
]
;
then
hosts
=
`
cat
$hostfile
|
awk
'{print $1}'
|
paste
-sd
","
-
`
;
else
echo
"hostfile not found, cannot proceed"
exit
1
fi
export
PDSH_RCMD_TYPE
=
ssh
;
if
[
"
$third_party_install
"
==
"1"
]
;
then
echo
"Checking out sub-module(s)"
git submodule update
--init
--recursive
echo
"Installing apex"
cd
third_party/apex
python setup.py
--cpp_ext
--cuda_ext
bdist_wheel
cd
-
pdsh
-w
$hosts
"sudo -H pip uninstall -y apex"
pdsh
-w
$hosts
"cd
$local_path
; sudo -H pip install third_party/apex/dist/apex*.whl"
pdsh
-w
$hosts
'python -c "import apex"'
fi
if
[
"
$deepspeed_install
"
==
"1"
]
;
then
echo
"Installing deepspeed"
python setup.py bdist_wheel
pdsh
-w
$hosts
"sudo -H pip uninstall -y deepspeed"
pdsh
-w
$hosts
"cd
$local_path
; sudo -H pip install dist/deepspeed*.whl"
pdsh
-w
$hosts
"python -c 'import deepspeed; print(
\"
deepspeed info:
\"
, deepspeed.__version__, deepspeed.__git_branch__, deepspeed.__git_hash__)'"
echo
"Installation is successful"
fi
fi
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录