diff --git a/doc/design/file_manager/README.md b/doc/design/file_manager/README.md index 2fe874581e7a3eebfe145aeb09476261ade1e135..8d9c7487c52a270f88f661dda4a17b27c31cf6a3 100644 --- a/doc/design/file_manager/README.md +++ b/doc/design/file_manager/README.md @@ -8,13 +8,10 @@ - 支持大文件的断点上传、下载 ## 名词解释 -- PFS：是Paddlepaddle cloud File System的简称，是对用户文件存储空间的抽象，与之相对的是Local File System。目前我们用CephFS来搭建。 +- PFS：是Paddlepaddle cloud File System的简称，是对用户文件存储空间的抽象，与之相对的是local filesystem。目前我们用CephFS来搭建。 - [CephFS](http://docs.ceph.com/docs/master/cephfs/)：一个POSIX兼容的文件系统。 - Chunk：逻辑划上文件分块的单位。 - [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/)：提供七层协议的反向代理、基于粘性会话的负载均衡。 -- CA：certificate authority[tls](#tls) -- CRT：CA signed certificate[tls](#tls) -- Key：用户私钥[tls](#tls) ## 模块 @@ -22,24 +19,49 @@ ### PFSClient -- 功能： 详细的内容看[Here](./pfs/pfs.md) +- 功能： 详细的内容看 - 提供用户管理文件的命令 - - 用Golang写，可以跨平台执行 + - 用Go写，可以跨平台执行 - 双向验证 - PFSClient需要和Ingress之间做双向验证[tls](#tls)，所以用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的Key、CRT、CA下载到本地，然后才能使用PFSClient。 + PFSClient需要和Ingress之间做双向验证[tls](#tls)，所以用户需要首先在`cloud.paddlepaddle.org`上注册一下，申请用户空间，并且把系统生成的CA(certificate authority)、Key、CRT(CA signed certificate)下载到本地，然后才能使用PFSClient。 + +- 命令格式 + +``` +paddle [options] pfs [parameters] + +options: + +``` ### Ingress - 功能： 提供七层协议的反向代理、基于粘性会话的负载均衡功能。 - 透传用户身份的办法 - Ingress需要把PFSClient的身份头传给FileServer，配置的方法参考[Here](http://www.integralist.co.uk/posts/clientcertauth.html#3) + Ingress需要把PFSClient的身份信息传给FileServer，配置的方法参考[Here](http://www.integralist.co.uk/posts/clientcertauth.html#3) -### FileServer -FileServer是一个用GoRPC写的HTTPServer，提供[RESTful API](./RESTAPI.md)接口，接收处理PFSClient端的文件管理请求，并且把结果返回PFSClient端。 +### PFSServer +PFSServer提供RESTful API接口，接收处理PFSClient端的文件管理请求，并且把结果返回PFSClient端。 +``` +RESTful API + +- /api/v1/files + - `GET /api/v1/files`: Get attributes of files or directories. + - `POST /api/v1/files`: Create files or directories. + - `DELETE /api/v1/files`: Delete files or directories. + +- /api/v1/storage/files + - `GET /api/v1/storage/files`: Download files or directories to local. + - `POST /api/v1/storage/files`: Upload files or directories to server. + +- /api/v1/storage/file/chunks + - `GET /api/v1/storage/file/chunks`: Get chunks's attributes in a file. + - `POST /api/v1/storage/file/chunks`: Upload chunks to a file. +``` ## 文件传输优化 ### 分块文件传输 @@ -57,7 +79,7 @@ type Chunk struct { ``` ### 生成sparse文件 -当destination文件不存在或者大小和source文件不一致时，可以用[Fallocate](https://golang.org/pkg/syscall/#Fallocate)生成sparse文件，然后就可以并发写入多个Chunk。 +当destination文件不存在或者大小和source文件不一致时，可以用[Fallocate](https://Go.org/pkg/syscall/#Fallocate)生成sparse文件，然后就可以并发写入多个Chunk。 ### 覆盖不一致的部分 文件传输的的关键在于需要PFSClient端对比source和destination的文件Chunks的checksum是否保持一致，不一致的由PFSClient下载或者传输Chunk完成。这样已经传输成功的部分就不用重新传输了。 diff --git a/doc/design/file_manager/RESTAPI.md b/doc/design/file_manager/RESTAPI.md deleted file mode 100644 index 9c94804d6b17976875981ef4d7175c66d1eba037..0000000000000000000000000000000000000000 --- a/doc/design/file_manager/RESTAPI.md +++ /dev/null @@ -1,23 +0,0 @@ -# REST API Interface -- file - -``` -GET /file: Get attribue of files -POST /file: Create a file -DELETE /file: Delete a File -``` - -- chunk - -``` -GET /file/chunk: Get a chunk info -POST /file/chunk: Update a chunk -``` - -- dir - -``` -GET /dir: List all files in a directory -POST /dir: Create a directory -DELETE /dir: Delete a directory -``` diff --git a/doc/design/file_manager/pfs/pfs.md b/doc/design/file_manager/pfs/pfs.md index e26fc1095d23cd3722ca426dd5305d0915f9097d..dd4578e3696305109228d402a8ac4617dd157115 100644 --- a/doc/design/file_manager/pfs/pfs.md +++ b/doc/design/file_manager/pfs/pfs.md @@ -34,9 +34,89 @@ A `pfspath` begin with `/pfs`, eg: `/pfs/$DATACENTER/home/$USER/folder`. Commonly, if there are two path arguments, the first is the source, and the second is the destination. ## Subcommonds -- [rm](rm.md) -- [mv](mv.md) -- [cp](cp.md) -- [ls](ls.md) -- [mkdir](mkdir.md) -- [sync](sync.md) +- rm - remove files or directories + +``` +Synopsis: + rm [-r] [-v] ... + +Options: + -r + remove directories and their contents recursively + -v + Cause rm to be verbose, showing files after they are removed. + +Examples: + paddle pfs rm /pfs/$DATACENTER/home/$USER/file + paddle pfs rm -r /pfs/$DATACENTER/home/$USER/folder +``` +- mv - move (rename) files + +``` +Synopsis: + mv [-f | -n] [-v] + mv [-f | -n] [-v] ... + mv [-f | -n] [-v] + mv [-f | -n] [-v] ... + mv [-f | -n] [-v] + mv [-f | -n] [-v] ... + +Options: + -f + Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.) + -n + Do not overwrite an existing file. (The -n option overrides previous -f options.) + -v + Cause mv to be verbose, showing files after they are moved. + +Examples: + paddle pfs mv ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt +``` +- cp - copy files or directories + +``` +Synopsis: + cp [-r] [-f | -n] [-v] [--preserve--links] + cp [-r] [-f | -n] [-v] [--preserve--links] ... + cp [-r] [-f | -n] [-v] [--preserve--links] + cp [-r] [-f | -n] [-v] [--preserve--links] ... + cp [-r] [-f | -n] [-v] [--preserve--links] + cp [-r] [-f | -n] [-v] [--preserve--links] ... + +Options: + -r + Copy directories recursively + -f + Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.) + -n + Do not overwrite an existing file. (The -n option overrides previous -f options.) + -v + Cause cp to be verbose, showing files after they are copied. + --preserve--links + Reserve links when copy links +``` +- ls- list files + +``` +Synopsis: + ls [-r] ... + +Options: + -r + List directory(ies) recursively + +Examples: + paddle pfs ls /pfs/$DATACENTER/home/$USER/file + paddle pfs ls /pfs/$DATACENTER/home/$USER/folder +``` + +- mkdir - mkdir directory(ies) +Create intermediate directory(ies) as required. + +``` +Synopsis: + mkdir ... + +Examples: + paddle pfs mkdir /pfs/$DATACENTER/home/$USER/folder +``` diff --git a/doc/design/file_manager/src/filemanager.graffle b/doc/design/file_manager/src/filemanager.graffle index beb57cdd786e24a57e63c06ef4d6cfa0df3d8c5a..7861a33072bc1908f69d12b37c20491dd8663103 100644 Binary files a/doc/design/file_manager/src/filemanager.graffle and b/doc/design/file_manager/src/filemanager.graffle differ diff --git a/doc/design/file_manager/src/filemanager.png b/doc/design/file_manager/src/filemanager.png index d6b25b7bf2766e78fd638ab9103af7b68e44b90a..8139a19f5722f56d3c211f3ab0d3982f751134b9 100644 Binary files a/doc/design/file_manager/src/filemanager.png and b/doc/design/file_manager/src/filemanager.png differ