Skip to content

P
PaddleDetection

PaddlePaddle / PaddleDetection
大约 1 年 前同步成功

通知 695
Star 11112
Fork 2696
P
PaddleDetection
提交 60f69c2b 编写于 作者: G gongweibao
浏览文件
操作

fix by comments

上级 327727d4
隐藏空白更改
内联 并排
Showing with 88 addition and 120 deletion
doc/design/file_manager/README.md
浏览文件 @ 60f69c2b
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
在本文档中，我们设计说明了名为FileManager系统，方便用户管理存放到PaddlePaddle Cloud上的文件。 在本文档中，我们设计说明了名为FileManager系统，方便用户管理存放到PaddlePaddle Cloud上的文件。
主要功能包括： 主要功能包括：
- 提供常用的命令行文件管理命令管理文件 - 提供常用的命令行管理命令管理文件和目录
- 支持的命令在[Here](./pfs/pfs.md) - 支持的命令在[Here](./pfs/pfs.md)
- 支持大文件的断点上传、下载 - 支持大文件的断点上传、下载
...@@ -27,7 +27,7 @@ ...@@ -27,7 +27,7 @@
- 用Golang写，可以跨平台执行 - 用Golang写，可以跨平台执行
- 双向验证 - 双向验证
PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>，所有用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的Key、CRT、CA下载到本地，然后才能使用PFSClient。 PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>，所以用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的Key、CRT、CA下载到本地，然后才能使用PFSClient。
### Ingress ### Ingress
- 功能： - 功能：
...@@ -43,7 +43,7 @@ FileServer是一个用GoRPC写的HTTPServer，提供[RESTful API](./RESTAPI.md) ...@@ -43,7 +43,7 @@ FileServer是一个用GoRPC写的HTTPServer，提供[RESTful API](./RESTAPI.md)
## 文件传输优化 ## 文件传输优化
### 分块文件传输 ### 分块文件传输
用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了Chunk的概念，一个Chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过Chunk的操作来实现的。由于Chunk比较小（默认256K），完成一个传输动作完成的时间也比较短，不容易出错。PFSClient在传输完毕最后一个Chunk的时候检查desttination文件的MD5值是否和source文件一致。 用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了Chunk的概念，一个Chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过Chunk的操作来实现的。由于Chunk比较小（默认256K），完成一个传输动作完成的时间也比较短，不容易出错。PFSClient在传输完毕最后一个Chunk的时候检查destination文件的MD5值是否和source文件一致。
一个典型的Chunk如下所示： 一个典型的Chunk如下所示：
......
doc/design/file_manager/pfs/cp.md
浏览文件 @ 60f69c2b
# Name # Name
cp - copy files and directories cp - copy files
# Synopsis # Synopsis
` cp [OPTION]... ```
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>` cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> ... <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <PFSPath>
```
# Description # Description
```
The following options are available:
-r
Copy directories recursively
-f
Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
-n
Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause cp to be verbose, showing files after they are copied.
```
--preserve--links --preserve--links
Reserve links when copy links Reserve links when copy links
-R, -r, --recursive
Copy directories recursively
``` ```
# Examples # Examples
- The following command copies a single file to pfs - The following command copies a single file to pfs
``` ```
paddle pfs cp ./text1.txt /pfs/mydir/text1.txt paddle pfs cp ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt
```
Output
```
upload ./text1.txt to /pfs/mydir/text1.txt
``` ```
- The following command copies pfs file to a local file - The following command copies pfs file to a local file
``` ```
paddle pfs cp /pfs/mydir/text1.txt ./text1.txt paddle pfs cp /pfs/$DATACENTER/home/$USER/text1.txt ./text1.txt
```
Output
```
download /pfs/mydir/text1.txt to ./text1.txt
``` ```
doc/design/file_manager/pfs/ls.md
浏览文件 @ 60f69c2b
# Name # Name
ls - list directory contents or a file attributes ls - list directory(ies)'s contents or file(s)'s attributes
# Synopsis # Synopsis
` ls [OPTION]... <PFSPath>` `ls [-r] <PFSPath> ...`
# Description # Description
``` ```
-R, -r, --recursive The following options are available:
Copy directories recursively
-r
--page-size (integer) List directory(ies) recursively
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if operation time out.
``` ```
# Examples # Examples
- The following command lists a single file - The following command lists a single file
``` ```
paddle pfs ls /pfs/mydir/text1.txt paddle pfs ls /pfs/$DATACENTER/home/$USER/text1.txt
```
Output
```
2017-05-5 17:04:30 text1.txt
``` ```
- The following command lists directory contents - The following command lists directory contents
``` ```
paddle pfs ls /pfs/mydir paddle pfs ls / /pfs/$DATACENTER/home/$USER/folder
```
Output
```
2017-05-5 17:04:30 text1.txt
2017-05-5 17:04:30 text2.txt
...
``` ```
doc/design/file_manager/pfs/mkdir.md
浏览文件 @ 60f69c2b
...@@ -2,18 +2,12 @@ ...@@ -2,18 +2,12 @@
mkdir - mkdir directory(ies) mkdir - mkdir directory(ies)
# Synopsis # Synopsis
`mkdir [OPTION]... <PFSPath> ...` `mkdir <PFSPath> ...`
# Description # Description
Create the pfs directory(ies), if they do not already exist. And create intermediate directories as required Create the pfs directory(ies), if it(they) does(do) not already exist. And create intermediate directories as required.
# Examples # Examples
``` ```
paddle pfs mkdir /pfs/mydir1 paddle pfs mkdir /pfs/$DATACENTER/home/$USER/folder
```
Output
```
make directory /pfs/mydir1
``` ```
doc/design/file_manager/pfs/mv.md
浏览文件 @ 60f69c2b
# Name # Name
mv - move (rename) files or directories mv - move (rename) files
# Synopsis # Synopsis
If destination already exist, please [rm](rm.md) it first.
``` ```
mv [OPTION]... mv [-f | -n] [-v] <LocalPath> <PFSPath>
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath> mv [-f | -n] [-v] <LocalPath> ... <PFSPath>
mv [-f | -n] [-v] <PFSPath> <LocalPath>
mv [-f | -n] [-v] <PFSPath> ... <LocalPath>
mv [-f | -n] [-v] <PFSPath> <PFSPath>
mv [-f | -n] [-v] <PFSPath> ... <PFSPath>
``` ```
# Description # Description
```
``` ```
The following options are available:
# Examples -f
- The following command move a single file to pfs Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
``` -n
paddle pfs mv ./text1.txt /pfs/mydirectory/test1.txt Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause mv to be verbose, showing files after they are moved.
``` ```
Output # Examples
- The following command moves a single file to pfs
``` ```
move ./text1.txt /pfs/mydirectory/test1.txt paddle pfs mv ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt
``` ```
doc/design/file_manager/pfs/pfs.md
浏览文件 @ 60f69c2b
# PFS Client # PFS Client
## Description ## Description
The `pfs` command is a Command Line Interface to manage your files on Paddle Cloud The `pfs` command is a Command Line Interface to manage your files on PaddlePaddle Cloud
## Synopsis ## Synopsis
``` ```
...@@ -28,10 +28,10 @@ paddle [options] pfs <subcommand> [parameters] ...@@ -28,10 +28,10 @@ paddle [options] pfs <subcommand> [parameters]
## Path Arguments ## Path Arguments
When using a command, we need to specify path arguments. There are two path argument type: `localpath` and `pfspath`. When using a command, we need to specify path arguments. There are two path argument type: `localpath` and `pfspath`.
A `pfspath` begin with `/pfs`, eg: `/pfs/mydir/text1.txt`. A `pfspath` begin with `/pfs`, eg: `/pfs/$DATACENTER/home/$USER/folder`.
## order of Path Arguments ## order of Path Arguments
Commonly, if there are two path arguments. The first is the source, and the second is the destination. Commonly, if there are two path arguments, the first is the source, and the second is the destination.
## Subcommonds ## Subcommonds
- [rm](rm.md) - [rm](rm.md)
......
doc/design/file_manager/pfs/rm.md
浏览文件 @ 60f69c2b
...@@ -3,43 +3,30 @@ rm - remove files or directories ...@@ -3,43 +3,30 @@ rm - remove files or directories
# Synopsis # Synopsis
``` ```
rm [OPTION]... <PFSPath>... rm [-r] [-v] <PFSPath> ...
``` ```
# Description # Description
``` ```
-r, -R, --recursive The following options are available:
remove directories and their contents recursively
--page-size (integer) -r
The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out remove directories and their contents recursively
-v
Cause rm to be verbose, showing files after they are removed.
``` ```
# Examples # Examples
- The following command deletes a single files - The following command deletes a single file:
```
paddle pfs rm /pfs/mydirectory/test1.txt
```
Output
```
delete /pfs/mydirectory/test1.txt
```
- The following command deletes a directory recursively.
``` ```
paddle pfs rm -r /pfs/mydirectory paddle pfs rm /pfs/$DATACENTER/home/$USER/test1.txt
``` ```
Output - The following command deletes a directory recursively:
``` ```
delete /pfs/mydirectory/1.txt paddle pfs rm -r /pfs/$DATACENTER/home/$USER/folder
delete /pfs/mydirectory/2.txt
...
``` ```
doc/design/file_manager/pfs/sync.md
浏览文件 @ 60f69c2b
# Name # Name
sync - sync directories. Recursively copies new and updated files from the source directory to the destination sync - sync directories. Recursively copies new and updated files from the source directory to the destination.
# Synopsis # Synopsis
` sync [OPTIONS]... ```
<LocalPath> <PFSPath> or <PFSPath> <LocalPath> or <PFSPath> <PFSPath>` sync [--preserve--links] [-v] <LocalPath> <PFSPath>
sync [--preserve--links] [-v] <PFSPath> <LocalPath>
sync [--preserve--links] [-v] <PFSPath> <PFSPath>`
```
# Description # Description
``` ```
The following options are available:
--preserve--links --preserve--links
Reserve links when copy links Reserve links when copy links.
-v
Cause sync to be verbose, showing files after their's synchronization is complete.
``` ```
# Examples # Examples
- The following command sync locally directory to pfs - The following command sync locally directory to pfs.
```
paddle pfs sync ./dir1 /pfs/mydir1
```
Output
```
upload ./dir1/text1.txt to /pfs/mydir1/text2.txt
upload ./dir1/text2.txt to /pfs/mydir1/text2.txt
...
```
- The following command sync pfs directory to local
``` ```
paddle pfs sync /pfs/mydir1 . paddle pfs sync ./dir1 /pfs/$DATACENTER/home/$USER/mydir1
``` ```
Output - The following command sync pfs directory to local.
``` ```
download /pfs/mydir1/text1.txt to ./text1.txt paddle pfs sync /pfs/$DATACENTER/home/$USER/mydir1 .
download /pfs/mydir1/text2.txt to ./text2.txt
...
``` ```
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或