提交 b2ccc477 编写于 作者: G gongweibao

fix by comments

上级 bd69fa31
...@@ -8,13 +8,10 @@ ...@@ -8,13 +8,10 @@
- 支持大文件的断点上传、下载 - 支持大文件的断点上传、下载
## 名词解释 ## 名词解释
- PFS:是Paddlepaddle cloud File System的简称,是对用户文件存储空间的抽象,与之相对的是Local File System。目前我们用CephFS来搭建。 - PFS:是Paddlepaddle cloud File System的简称,是对用户文件存储空间的抽象,与之相对的是local filesystem。目前我们用CephFS来搭建。
- [CephFS](http://docs.ceph.com/docs/master/cephfs/):一个POSIX兼容的文件系统。 - [CephFS](http://docs.ceph.com/docs/master/cephfs/):一个POSIX兼容的文件系统。
- Chunk:逻辑划上文件分块的单位。 - Chunk:逻辑划上文件分块的单位。
- [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/):提供七层协议的反向代理、基于粘性会话的负载均衡。 - [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/):提供七层协议的反向代理、基于粘性会话的负载均衡。
- CA:certificate authority<sup>[tls](#tls)</sup>
- CRT:CA signed certificate<sup>[tls](#tls)</sup>
- Key:用户私钥<sup>[tls](#tls)</sup>
## 模块 ## 模块
...@@ -22,24 +19,49 @@ ...@@ -22,24 +19,49 @@
<image src=./src/filemanager.png width=900> <image src=./src/filemanager.png width=900>
### PFSClient ### PFSClient
- 功能: 详细的内容看[Here](./pfs/pfs.md) - 功能: 详细的内容看
- 提供用户管理文件的命令 - 提供用户管理文件的命令
- 用Golang写,可以跨平台执行 - 用Go写,可以跨平台执行
- 双向验证 - 双向验证
PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>,所以用户需要首先在`cloud.paddlepaddle.org`上注册一下,申请用户空间,并且把系统生成的Key、CRT、CA下载到本地,然后才能使用PFSClient。 PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>,所以用户需要首先在`cloud.paddlepaddle.org`上注册一下,申请用户空间,并且把系统生成的CA(certificate authority)、Key、CRT(CA signed certificate)下载到本地,然后才能使用PFSClient。
- 命令格式
```
paddle [options] pfs <subcommand> [parameters]
options:
```
### Ingress ### Ingress
- 功能: - 功能:
提供七层协议的反向代理、基于粘性会话的负载均衡功能。 提供七层协议的反向代理、基于粘性会话的负载均衡功能。
- 透传用户身份的办法 - 透传用户身份的办法
Ingress需要把PFSClient的身份传给FileServer,配置的方法参考[Here](http://www.integralist.co.uk/posts/clientcertauth.html#3) Ingress需要把PFSClient的身份信息传给FileServer,配置的方法参考[Here](http://www.integralist.co.uk/posts/clientcertauth.html#3)
### FileServer ### PFSServer
FileServer是一个用GoRPC写的HTTPServer,提供[RESTful API](./RESTAPI.md)接口,接收处理PFSClient端的文件管理请求,并且把结果返回PFSClient端。 PFSServer提供RESTful API接口,接收处理PFSClient端的文件管理请求,并且把结果返回PFSClient端。
```
RESTful API
- /api/v1/files
- `GET /api/v1/files`: Get attributes of files or directories.
- `POST /api/v1/files`: Create files or directories.
- `DELETE /api/v1/files`: Delete files or directories.
- /api/v1/storage/files
- `GET /api/v1/storage/files`: Download files or directories to local.
- `POST /api/v1/storage/files`: Upload files or directories to server.
- /api/v1/storage/file/chunks
- `GET /api/v1/storage/file/chunks`: Get chunks's attributes in a file.
- `POST /api/v1/storage/file/chunks`: Upload chunks to a file.
```
## 文件传输优化 ## 文件传输优化
### 分块文件传输 ### 分块文件传输
...@@ -57,7 +79,7 @@ type Chunk struct { ...@@ -57,7 +79,7 @@ type Chunk struct {
``` ```
### 生成sparse文件 ### 生成sparse文件
当destination文件不存在或者大小和source文件不一致时,可以用[Fallocate](https://golang.org/pkg/syscall/#Fallocate)生成sparse文件,然后就可以并发写入多个Chunk。 当destination文件不存在或者大小和source文件不一致时,可以用[Fallocate](https://Go.org/pkg/syscall/#Fallocate)生成sparse文件,然后就可以并发写入多个Chunk。
### 覆盖不一致的部分 ### 覆盖不一致的部分
文件传输的的关键在于需要PFSClient端对比source和destination的文件Chunks的checksum是否保持一致,不一致的由PFSClient下载或者传输Chunk完成。这样已经传输成功的部分就不用重新传输了。 文件传输的的关键在于需要PFSClient端对比source和destination的文件Chunks的checksum是否保持一致,不一致的由PFSClient下载或者传输Chunk完成。这样已经传输成功的部分就不用重新传输了。
......
# REST API Interface
- file
```
GET /file: Get attribue of files
POST /file: Create a file
DELETE /file: Delete a File
```
- chunk
```
GET /file/chunk: Get a chunk info
POST /file/chunk: Update a chunk
```
- dir
```
GET /dir: List all files in a directory
POST /dir: Create a directory
DELETE /dir: Delete a directory
```
...@@ -34,9 +34,89 @@ A `pfspath` begin with `/pfs`, eg: `/pfs/$DATACENTER/home/$USER/folder`. ...@@ -34,9 +34,89 @@ A `pfspath` begin with `/pfs`, eg: `/pfs/$DATACENTER/home/$USER/folder`.
Commonly, if there are two path arguments, the first is the source, and the second is the destination. Commonly, if there are two path arguments, the first is the source, and the second is the destination.
## Subcommonds ## Subcommonds
- [rm](rm.md) - rm - remove files or directories
- [mv](mv.md)
- [cp](cp.md) ```
- [ls](ls.md) Synopsis:
- [mkdir](mkdir.md) rm [-r] [-v] <PFSPath> ...
- [sync](sync.md)
Options:
-r
remove directories and their contents recursively
-v
Cause rm to be verbose, showing files after they are removed.
Examples:
paddle pfs rm /pfs/$DATACENTER/home/$USER/file
paddle pfs rm -r /pfs/$DATACENTER/home/$USER/folder
```
- mv - move (rename) files
```
Synopsis:
mv [-f | -n] [-v] <LocalPath> <PFSPath>
mv [-f | -n] [-v] <LocalPath> ... <PFSPath>
mv [-f | -n] [-v] <PFSPath> <LocalPath>
mv [-f | -n] [-v] <PFSPath> ... <LocalPath>
mv [-f | -n] [-v] <PFSPath> <PFSPath>
mv [-f | -n] [-v] <PFSPath> ... <PFSPath>
Options:
-f
Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
-n
Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause mv to be verbose, showing files after they are moved.
Examples:
paddle pfs mv ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt
```
- cp - copy files or directories
```
Synopsis:
cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> ... <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <PFSPath>
Options:
-r
Copy directories recursively
-f
Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
-n
Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause cp to be verbose, showing files after they are copied.
--preserve--links
Reserve links when copy links
```
- ls- list files
```
Synopsis:
ls [-r] <PFSPath> ...
Options:
-r
List directory(ies) recursively
Examples:
paddle pfs ls /pfs/$DATACENTER/home/$USER/file
paddle pfs ls /pfs/$DATACENTER/home/$USER/folder
```
- mkdir - mkdir directory(ies)
Create intermediate directory(ies) as required.
```
Synopsis:
mkdir <PFSPath> ...
Examples:
paddle pfs mkdir /pfs/$DATACENTER/home/$USER/folder
```
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册