diff --git a/doc/design/file_mananger/README.md b/doc/design/file_mananger/README.md index 4a9d0214d94ad943ba7b9c19549dc8caae18cfc2..6078696e6e2ff3c7000b5a2d0d306e4593a7c3ea 100644 --- a/doc/design/file_mananger/README.md +++ b/doc/design/file_mananger/README.md @@ -5,25 +5,11 @@ ## Module -### Client -Client提供用户管理本地或者远程文件的命令行程序。 +### PFS Client +- 提供用户管理Cloud文件的命令 +- 用Golang写，可以跨平台执行 -- 路径参数: -当用户输入一个命令的时候，一般需要指定路径参数。这里有两种路径参数：LocalPath 或者 PFSPath。 - - LocalPath：代表本地的一个路径 - PFSPath：代表PaddlePaddle Cloud上的一个路径。它需要满足类似这样的格式：`pfs://dir1/dir2`。路径必须要以`pds://`开始。 - -- 路径参数的顺序 -如果命令都有一个或者多个路径参数，那么一般第一个路径参数代表source，第二个路径参数代表destination。 - -- 支持的操作命令 - - [rm](cmd_rm.md) - - [mv](cmd_mv.md) - - [cp](cmd_cp.md) - - [ls](cmd_ls.md) - - [mkdir](cmd_mkdir.md) - - [sync](cmd_sync.md) +命令的详细内容看[这里](./pfs/pfs.md) ### Ingress @@ -37,26 +23,14 @@ Client提供用户管理本地或者远程文件的命令行程序。 - gorpc写的HttpServer - 响应外部的REST API的请求 - 在kubernets中运行 +- [RESTAPI](./RESTAPI.md)接口 -REST API说明: - -- file - -``` -GET /file: Get attribue of files -POST /file: Touch a file -DELETE /file: Delete a File -``` - -- chunk - -``` -GET /file/chunk: Get a chunk info -POST /file/chunk: Update a chunk -``` -为什么有chunk的抽象： +## 流程 +### 为什么有chunk的抽象： 用户文件可能是比较大的，上传到Cloud或者下载到本地的时间可能比较长，而且在传输的过程中也可能出现网络不稳定的情况。为了应对以上的问题，我们提出了chunk的概念，一个chunk由所在的文件偏移、数据、数据长度及校验值组成。文件数据内容的上传和下载都是都过chunk的操作来实现的。由于chunk比较小（默认256K），完成一个传输动作的transaction的时间也比较短，不容易出错。 +一个典型的chunk如下所示： + ``` type Chunk struct { filePos int64 @@ -66,16 +40,6 @@ type Chunk struct { } ``` -- dir - -``` -GET /dir: List all files in a directory -POST /dir: Touch a directory -DELETE /dir: Delete a directory -``` - - -## 流程 ### 关于文件权限 - 每一个用户在Cloud注册后可以申请分配用户空间，系统默认会在CephFS上分配一个固定大小（比如初始10G）的、有所有权限的volume，对用户而言就是自己的`home`目录。用户彼此之间的数据是隔离的、无法访问的。用户的空间大小第一期也不允许扩大。 - 公共数据集合放到一个单独的volume下，对所有外部用户只读。由于其被读取的可能比较频繁，需要提高其备份数，防止成为热点文件。 @@ -112,3 +76,4 @@ DELETE /dir: Delete a directory ## 参考文档 - [Do you see tls?](https://github.com/k8sp/tls/blob/master/README.md) - [s3](http://docs.aws.amazon.com/cli/latest/reference/s3/) +- linux man document diff --git a/doc/design/file_mananger/RESTAPI.md b/doc/design/file_mananger/RESTAPI.md new file mode 100644 index 0000000000000000000000000000000000000000..4286195970bbbd1f8e4d2d9328f3dba597e8fdea --- /dev/null +++ b/doc/design/file_mananger/RESTAPI.md @@ -0,0 +1,23 @@ +# REST API +- file + +``` +GET /file: Get attribue of files +POST /file: Touch a file +DELETE /file: Delete a File +``` + +- chunk + +``` +GET /file/chunk: Get a chunk info +POST /file/chunk: Update a chunk +``` + +- dir + +``` +GET /dir: List all files in a directory +POST /dir: Touch a directory +DELETE /dir: Delete a directory +``` diff --git a/doc/design/file_mananger/pfs/cp.md b/doc/design/file_mananger/pfs/cp.md new file mode 100644 index 0000000000000000000000000000000000000000..fcc57258a1763849f051b95f9946b85e0ebaf842 --- /dev/null +++ b/doc/design/file_mananger/pfs/cp.md @@ -0,0 +1,47 @@ +# Name +pfs_cp - copy files and directories + +# Synopsis +` pfs_cp + or or ` + +# Description + +``` + --only-show-errors (boolean) + Only errors and warnings are displayed. All other output is suppressed. + + --page-size (integer) + The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out. + + --preserve--links + Reserve links when copy files + + -R, -r, --recursive + Copy directories recursively +``` + +# Examples +- The following command cp a single file to pfs + +``` +pfs_cp ./text1.txt pfs://mydir/text1.txt +``` + +Output + +``` +upload ./text1.txt to pfs://mydir/text1.txt +``` + +- The following command cp pfs file to a local file + +``` +pfs_cp pfs://mydir/text1.txt ./text1.txt +``` + +Output + +``` +download pfs://mydir/text1.txt to ./text1.txt +``` diff --git a/doc/design/file_mananger/pfs/ls.md b/doc/design/file_mananger/pfs/ls.md new file mode 100644 index 0000000000000000000000000000000000000000..41cf6fdf77f21138ba2ccc5c3b81f920e35af3a4 --- /dev/null +++ b/doc/design/file_mananger/pfs/ls.md @@ -0,0 +1,49 @@ +# Name +pfs_ls - list directory contents o + +# Synopsis +` pfs_ls [OPTION]... ` + +# Description + +``` + -h, --help + Display this help and exit + + --version + Output version information and exit + + --only-show-errors (boolean) + Only errors and warnings are displayed. All other output is suppressed. + + --page-size (integer) + The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out. + + -R, -r, --recursive + Copy directories recursively +``` + +# Examples +- The following command cp a single file to pfs + +``` +pfs_cp ./text1.txt pfs://mydir/text1.txt +``` + +Output + +``` +upload ./text1.txt to pfs://mydir/text1.txt +``` + +- The following command cp pfs file to a local file + +``` +pfs_cp pfs://mydir/text1.txt ./text1.txt +``` + +Output + +``` +download pfs://mydir/text1.txt to ./text1.txt +``` diff --git a/doc/design/file_mananger/pfs/mkdir.md b/doc/design/file_mananger/pfs/mkdir.md new file mode 100644 index 0000000000000000000000000000000000000000..e3b0d0cae78a6d1897da98f2f91dd2b93fb278b0 --- /dev/null +++ b/doc/design/file_mananger/pfs/mkdir.md @@ -0,0 +1,33 @@ +# Name +pdf_mkdir - mkdir directory(ies) + +# Synopsis +`pfs_mkdir [OPTION]... PFS_DIRECTORY...` + +# Description +Create the pfs directory(ies), if they do not already exist. + +``` + -h, --help + display this help and exit + + --version + output version information and exit + + --only-show-errors (boolean) + Only errors and warnings are displayed. All other output is suppressed. + + --page-size (integer) + The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out. +``` + +# Examples +``` +pfs_mkdir pfs://mydir1 +``` + +Output + +``` +make directory pfs://mydir1 +``` diff --git a/doc/design/file_mananger/pfs/mv.md b/doc/design/file_mananger/pfs/mv.md new file mode 100644 index 0000000000000000000000000000000000000000..3f4c1ac0761dae4db9c007416bd2ca1c353a5f7e --- /dev/null +++ b/doc/design/file_mananger/pfs/mv.md @@ -0,0 +1,38 @@ +# Name +mv - move (rename) files or directories + + +# Synopsis +If destination already exist, please rm it first. +``` +mv [OPTION]... + or or +``` + +# Description +``` + -h, --help + display this help and exit + + -v, --version + output version information and exit + + --only-show-errors (boolean) + Only errors and warnings are displayed. All other output is suppressed. + + --page-size (integer) + The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out. +``` + +# Examples +- The following command move a single file to pfs + +``` +mv ./text1.txt pfs://mydirectory/test1.txt +``` + +Output + +``` +move text1.txt pfs://mydirectory/test1.txt +``` diff --git a/doc/design/file_mananger/pfs/pfs.md b/doc/design/file_mananger/pfs/pfs.md new file mode 100644 index 0000000000000000000000000000000000000000..a17de7d8140fe6cd42f957b9c4f21a22973a538f --- /dev/null +++ b/doc/design/file_mananger/pfs/pfs.md @@ -0,0 +1,36 @@ +# PFS Client + +## Description +The `pfs` command is a Command Line Interface to manage your files on Paddle Cloud + +## Synopsis +``` +paddle [options] pfs [parameters] +``` + +## Options +``` +--profile (string) + Use a specific profile from your credential file. +--help pfs + Display more information about pfs +--version + Output version information and exit +--debug + Show detailed debugging log +``` + +## Path Arguments +When using a commnd, we need to specify path arguments. There are two path argument type: `localpath` and `pfspath`. +A pfspath begin with `pfs://`, eg: `pfs://mydir/text1.txt`. + +## Path Arguments‘s order +Commonly, there maybe two path arguments. The first is source, and the second is destination. + +## Subcommonds +- [rm](rm.md) +- [mv](mv.md) +- [cp](cp.md) +- [ls](ls.md) +- [mkdir](mkdir.md) +- [sync](sync.md) diff --git a/doc/design/file_mananger/pfs/rm.md b/doc/design/file_mananger/pfs/rm.md new file mode 100644 index 0000000000000000000000000000000000000000..6056c401f956c040165becdcbc34709a74ae13b7 --- /dev/null +++ b/doc/design/file_mananger/pfs/rm.md @@ -0,0 +1,50 @@ +# Name +rm - remove files or directories + +# Synopsis +`rm [OPTION]... PFS_DIRECTORY...` + +# Description +Create the directory, if it does not already exits + +``` + -r, -R, --recursive + remove directories and their contents recursively + + -v, --version + output version information and exit + + --only-show-errors (boolean) + Only errors and warnings are displayed. All other output is suppressed. + + --page-size (integer) + The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out. +``` + +# Examples +- The following command delete a single file + +``` +pfs_rm pfs://mydirectory/test1.txt +``` + +Output + +``` +delete pfs://mydirectory/test1.txt +``` + + +- The following command delete a directory recursively. + +``` +pfs_rm -r pfs://mydirectory +``` + +Output + +``` +delete pfs://mydirectory/1.txt +delete pfs://mydirectory/2.txt +... +``` diff --git a/doc/design/file_mananger/pfs/sync.md b/doc/design/file_mananger/pfs/sync.md new file mode 100644 index 0000000000000000000000000000000000000000..99248082743559c5e51866079e3bc0b199895a8a --- /dev/null +++ b/doc/design/file_mananger/pfs/sync.md @@ -0,0 +1,53 @@ +# Name +pfs_cp - copy files and directories + +# Synopsis +` pfs_cp + or or ` + +# Description + +``` + -h, --help + Display this help and exit + + --version + Output version information and exit + + --only-show-errors (boolean) + Only errors and warnings are displayed. All other output is suppressed. + + --page-size (integer) + The number of results to return in each response to a list operation. The default value is 1000 (the maximum allowed). Using a lower value may help if an operation times out. + + --preserve--links + Reserve links when copy files + + -R, -r, --recursive + Copy directories recursively +``` + +# Examples +- The following command cp a single file to pfs + +``` +pfs_cp ./text1.txt pfs://mydir/text1.txt +``` + +Output + +``` +upload ./text1.txt to pfs://mydir/text1.txt +``` + +- The following command cp pfs file to a local file + +``` +pfs_cp pfs://mydir/text1.txt ./text1.txt +``` + +Output + +``` +download pfs://mydir/text1.txt to ./text1.txt +```