diff --git a/doc/design/file_manager/README.md b/doc/design/file_manager/README.md index 9d75089257f7489111d45345cbf4b6ce09edcd3b..2fe874581e7a3eebfe145aeb09476261ade1e135 100644 --- a/doc/design/file_manager/README.md +++ b/doc/design/file_manager/README.md @@ -3,7 +3,7 @@ 在本文档中，我们设计说明了名为FileManager系统，方便用户管理存放到PaddlePaddle Cloud上的文件。 主要功能包括： -- 提供常用的命令行文件管理命令管理文件 +- 提供常用的命令行管理命令管理文件和目录 - 支持的命令在[Here](./pfs/pfs.md) - 支持大文件的断点上传、下载 @@ -27,7 +27,7 @@ - 用Golang写，可以跨平台执行 - 双向验证 - PFSClient需要和Ingress之间做双向验证[tls](#tls)，所有用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的Key、CRT、CA下载到本地，然后才能使用PFSClient。 + PFSClient需要和Ingress之间做双向验证[tls](#tls)，所以用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的Key、CRT、CA下载到本地，然后才能使用PFSClient。 ### Ingress - 功能： @@ -43,7 +43,7 @@ FileServer是一个用GoRPC写的HTTPServer，提供[RESTful API](./RESTAPI.md) ## 文件传输优化 ### 分块文件传输 -用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了Chunk的概念，一个Chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过Chunk的操作来实现的。由于Chunk比较小（默认256K），完成一个传输动作完成的时间也比较短，不容易出错。PFSClient在传输完毕最后一个Chunk的时候检查desttination文件的MD5值是否和source文件一致。 +用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了Chunk的概念，一个Chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过Chunk的操作来实现的。由于Chunk比较小（默认256K），完成一个传输动作完成的时间也比较短，不容易出错。PFSClient在传输完毕最后一个Chunk的时候检查destination文件的MD5值是否和source文件一致。 一个典型的Chunk如下所示： diff --git a/doc/design/file_manager/pfs/cp.md b/doc/design/file_manager/pfs/cp.md index 605c0b7c8bfcf39a5236ffeb614b32ac33b92881..1f1670e882748042be0e0810a997eec38069e707 100644 --- a/doc/design/file_manager/pfs/cp.md +++ b/doc/design/file_manager/pfs/cp.md @@ -1,41 +1,45 @@ # Name -cp - copy files and directories +cp - copy files # Synopsis -` cp [OPTION]... - or or ` +``` +cp [-r] [-f | -n] [-v] [--preserve--links] +cp [-r] [-f | -n] [-v] [--preserve--links] ... +cp [-r] [-f | -n] [-v] [--preserve--links] +cp [-r] [-f | -n] [-v] [--preserve--links] ... +cp [-r] [-f | -n] [-v] [--preserve--links] +cp [-r] [-f | -n] [-v] [--preserve--links] ... +``` # Description +``` +The following options are available: + +-r + Copy directories recursively + +-f + Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.) + +-n + Do not overwrite an existing file. (The -n option overrides previous -f options.) + +-v + Cause cp to be verbose, showing files after they are copied. -``` --preserve--links Reserve links when copy links - --R, -r, --recursive - Copy directories recursively ``` # Examples - The following command copies a single file to pfs ``` -paddle pfs cp ./text1.txt /pfs/mydir/text1.txt -``` - -Output - -``` -upload ./text1.txt to /pfs/mydir/text1.txt +paddle pfs cp ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt ``` - The following command copies pfs file to a local file ``` -paddle pfs cp /pfs/mydir/text1.txt ./text1.txt -``` - -Output - -``` -download /pfs/mydir/text1.txt to ./text1.txt +paddle pfs cp /pfs/$DATACENTER/home/$USER/text1.txt ./text1.txt ``` diff --git a/doc/design/file_manager/pfs/ls.md b/doc/design/file_manager/pfs/ls.md index ab254cfc5d9d362fcb3282dda84f175e1c4cdfe5..0db163e08bdc98879f7a76f83e54491e90918bc8 100644 --- a/doc/design/file_manager/pfs/ls.md +++ b/doc/design/file_manager/pfs/ls.md @@ -1,42 +1,27 @@ # Name -ls - list directory contents or a file attributes +ls - list directory(ies)'s contents or file(s)'s attributes # Synopsis -` ls [OPTION]... ` +`ls [-r] ...` # Description -``` --R, -r, --recursive - Copy directories recursively - ---page-size (integer) - The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if operation time out. +``` +The following options are available: + +-r + List directory(ies) recursively ``` # Examples - The following command lists a single file ``` -paddle pfs ls /pfs/mydir/text1.txt -``` - -Output - -``` -2017-05-5 17:04:30 text1.txt +paddle pfs ls /pfs/$DATACENTER/home/$USER/text1.txt ``` - The following command lists directory contents ``` -paddle pfs ls /pfs/mydir -``` - -Output - -``` -2017-05-5 17:04:30 text1.txt -2017-05-5 17:04:30 text2.txt -... +paddle pfs ls / /pfs/$DATACENTER/home/$USER/folder ``` diff --git a/doc/design/file_manager/pfs/mkdir.md b/doc/design/file_manager/pfs/mkdir.md index c11dadd61f5cf6e0b2911fc0790a7ed443b7df18..3ca51704027ae6bfd6fe2bed89db3f9c7190c12e 100644 --- a/doc/design/file_manager/pfs/mkdir.md +++ b/doc/design/file_manager/pfs/mkdir.md @@ -2,18 +2,12 @@ mkdir - mkdir directory(ies) # Synopsis -`mkdir [OPTION]... ...` +`mkdir ...` # Description -Create the pfs directory(ies), if they do not already exist. And create intermediate directories as required +Create the pfs directory(ies), if it(they) does(do) not already exist. And create intermediate directories as required. # Examples ``` -paddle pfs mkdir /pfs/mydir1 -``` - -Output - -``` -make directory /pfs/mydir1 +paddle pfs mkdir /pfs/$DATACENTER/home/$USER/folder ``` diff --git a/doc/design/file_manager/pfs/mv.md b/doc/design/file_manager/pfs/mv.md index 3929d43394adfa956194b24ce5b8b2d3a39fc730..01b795be02cf81a0776745bd339f2c0d9569e549 100644 --- a/doc/design/file_manager/pfs/mv.md +++ b/doc/design/file_manager/pfs/mv.md @@ -1,28 +1,34 @@ # Name -mv - move (rename) files or directories +mv - move (rename) files # Synopsis -If destination already exist, please [rm](rm.md) it first. - ``` -mv [OPTION]... - or or +mv [-f | -n] [-v] +mv [-f | -n] [-v] ... +mv [-f | -n] [-v] +mv [-f | -n] [-v] ... +mv [-f | -n] [-v] +mv [-f | -n] [-v] ... ``` # Description -``` ``` +The following options are available: -# Examples -- The following command move a single file to pfs +-f + Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.) -``` -paddle pfs mv ./text1.txt /pfs/mydirectory/test1.txt +-n + Do not overwrite an existing file. (The -n option overrides previous -f options.) + +-v + Cause mv to be verbose, showing files after they are moved. ``` -Output +# Examples +- The following command moves a single file to pfs ``` -move ./text1.txt /pfs/mydirectory/test1.txt +paddle pfs mv ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt ``` diff --git a/doc/design/file_manager/pfs/pfs.md b/doc/design/file_manager/pfs/pfs.md index c23ffc7479192bc426e1cf25f73e86a67de45988..e26fc1095d23cd3722ca426dd5305d0915f9097d 100644 --- a/doc/design/file_manager/pfs/pfs.md +++ b/doc/design/file_manager/pfs/pfs.md @@ -1,7 +1,7 @@ # PFS Client ## Description -The `pfs` command is a Command Line Interface to manage your files on Paddle Cloud +The `pfs` command is a Command Line Interface to manage your files on PaddlePaddle Cloud ## Synopsis ``` @@ -28,10 +28,10 @@ paddle [options] pfs [parameters] ## Path Arguments When using a command, we need to specify path arguments. There are two path argument type: `localpath` and `pfspath`. -A `pfspath` begin with `/pfs`, eg: `/pfs/mydir/text1.txt`. +A `pfspath` begin with `/pfs`, eg: `/pfs/$DATACENTER/home/$USER/folder`. ## order of Path Arguments -Commonly, if there are two path arguments. The first is the source, and the second is the destination. +Commonly, if there are two path arguments, the first is the source, and the second is the destination. ## Subcommonds - [rm](rm.md) diff --git a/doc/design/file_manager/pfs/rm.md b/doc/design/file_manager/pfs/rm.md index 64774c7ad9610de4868ec52ad61435cdf2df57b9..8bcbabdfb106db9acbd150f2fa97dbc8980ffae3 100644 --- a/doc/design/file_manager/pfs/rm.md +++ b/doc/design/file_manager/pfs/rm.md @@ -3,43 +3,30 @@ rm - remove files or directories # Synopsis ``` -rm [OPTION]... ... +rm [-r] [-v] ... ``` # Description ``` --r, -R, --recursive - remove directories and their contents recursively +The following options are available: ---page-size (integer) - The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out +-r + remove directories and their contents recursively + +-v + Cause rm to be verbose, showing files after they are removed. ``` # Examples -- The following command deletes a single files - -``` -paddle pfs rm /pfs/mydirectory/test1.txt -``` - -Output - -``` -delete /pfs/mydirectory/test1.txt -``` - - -- The following command deletes a directory recursively. +- The following command deletes a single file: ``` -paddle pfs rm -r /pfs/mydirectory +paddle pfs rm /pfs/$DATACENTER/home/$USER/test1.txt ``` -Output +- The following command deletes a directory recursively: ``` -delete /pfs/mydirectory/1.txt -delete /pfs/mydirectory/2.txt -... +paddle pfs rm -r /pfs/$DATACENTER/home/$USER/folder ``` diff --git a/doc/design/file_manager/pfs/sync.md b/doc/design/file_manager/pfs/sync.md index c921e8ac81a5b74179a2e6bb09250e0063650812..4801ceedf9262d9180824c30451d932a540d4f89 100644 --- a/doc/design/file_manager/pfs/sync.md +++ b/doc/design/file_manager/pfs/sync.md @@ -1,42 +1,34 @@ # Name -sync - sync directories. Recursively copies new and updated files from the source directory to the destination +sync - sync directories. Recursively copies new and updated files from the source directory to the destination. # Synopsis -` sync [OPTIONS]... - or or ` +``` +sync [--preserve--links] [-v] +sync [--preserve--links] [-v] +sync [--preserve--links] [-v] ` +``` # Description ``` +The following options are available: + --preserve--links - Reserve links when copy links + Reserve links when copy links. + +-v + Cause sync to be verbose, showing files after their's synchronization is complete. ``` # Examples -- The following command sync locally directory to pfs - -``` -paddle pfs sync ./dir1 /pfs/mydir1 -``` - -Output - -``` -upload ./dir1/text1.txt to /pfs/mydir1/text2.txt -upload ./dir1/text2.txt to /pfs/mydir1/text2.txt -... -``` - -- The following command sync pfs directory to local +- The following command sync locally directory to pfs. ``` -paddle pfs sync /pfs/mydir1 . +paddle pfs sync ./dir1 /pfs/$DATACENTER/home/$USER/mydir1 ``` -Output +- The following command sync pfs directory to local. ``` -download /pfs/mydir1/text1.txt to ./text1.txt -download /pfs/mydir1/text2.txt to ./text2.txt -... +paddle pfs sync /pfs/$DATACENTER/home/$USER/mydir1 . ```