提交 b2ccc477 编写于 作者: G gongweibao

fix by comments

上级 bd69fa31
......@@ -8,13 +8,10 @@
- 支持大文件的断点上传、下载
## 名词解释
- PFS:是Paddlepaddle cloud File System的简称,是对用户文件存储空间的抽象,与之相对的是Local File System。目前我们用CephFS来搭建。
- PFS:是Paddlepaddle cloud File System的简称,是对用户文件存储空间的抽象,与之相对的是local filesystem。目前我们用CephFS来搭建。
- [CephFS](http://docs.ceph.com/docs/master/cephfs/):一个POSIX兼容的文件系统。
- Chunk:逻辑划上文件分块的单位。
- [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/):提供七层协议的反向代理、基于粘性会话的负载均衡。
- CA:certificate authority<sup>[tls](#tls)</sup>
- CRT:CA signed certificate<sup>[tls](#tls)</sup>
- Key:用户私钥<sup>[tls](#tls)</sup>
## 模块
......@@ -22,24 +19,49 @@
<image src=./src/filemanager.png width=900>
### PFSClient
- 功能: 详细的内容看[Here](./pfs/pfs.md)
- 功能: 详细的内容看
- 提供用户管理文件的命令
- 用Golang写,可以跨平台执行
- 用Go写,可以跨平台执行
- 双向验证
PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>,所以用户需要首先在`cloud.paddlepaddle.org`上注册一下,申请用户空间,并且把系统生成的Key、CRT、CA下载到本地,然后才能使用PFSClient。
PFSClient需要和Ingress之间做双向验证<sup>[tls](#tls)</sup>,所以用户需要首先在`cloud.paddlepaddle.org`上注册一下,申请用户空间,并且把系统生成的CA(certificate authority)、Key、CRT(CA signed certificate)下载到本地,然后才能使用PFSClient。
- 命令格式
```
paddle [options] pfs <subcommand> [parameters]
options:
```
### Ingress
- 功能:
提供七层协议的反向代理、基于粘性会话的负载均衡功能。
- 透传用户身份的办法
Ingress需要把PFSClient的身份传给FileServer,配置的方法参考[Here](http://www.integralist.co.uk/posts/clientcertauth.html#3)
Ingress需要把PFSClient的身份信息传给FileServer,配置的方法参考[Here](http://www.integralist.co.uk/posts/clientcertauth.html#3)
### FileServer
FileServer是一个用GoRPC写的HTTPServer,提供[RESTful API](./RESTAPI.md)接口,接收处理PFSClient端的文件管理请求,并且把结果返回PFSClient端。
### PFSServer
PFSServer提供RESTful API接口,接收处理PFSClient端的文件管理请求,并且把结果返回PFSClient端。
```
RESTful API
- /api/v1/files
- `GET /api/v1/files`: Get attributes of files or directories.
- `POST /api/v1/files`: Create files or directories.
- `DELETE /api/v1/files`: Delete files or directories.
- /api/v1/storage/files
- `GET /api/v1/storage/files`: Download files or directories to local.
- `POST /api/v1/storage/files`: Upload files or directories to server.
- /api/v1/storage/file/chunks
- `GET /api/v1/storage/file/chunks`: Get chunks's attributes in a file.
- `POST /api/v1/storage/file/chunks`: Upload chunks to a file.
```
## 文件传输优化
### 分块文件传输
......@@ -57,7 +79,7 @@ type Chunk struct {
```
### 生成sparse文件
当destination文件不存在或者大小和source文件不一致时,可以用[Fallocate](https://golang.org/pkg/syscall/#Fallocate)生成sparse文件,然后就可以并发写入多个Chunk。
当destination文件不存在或者大小和source文件不一致时,可以用[Fallocate](https://Go.org/pkg/syscall/#Fallocate)生成sparse文件,然后就可以并发写入多个Chunk。
### 覆盖不一致的部分
文件传输的的关键在于需要PFSClient端对比source和destination的文件Chunks的checksum是否保持一致,不一致的由PFSClient下载或者传输Chunk完成。这样已经传输成功的部分就不用重新传输了。
......
# REST API Interface
- file
```
GET /file: Get attribue of files
POST /file: Create a file
DELETE /file: Delete a File
```
- chunk
```
GET /file/chunk: Get a chunk info
POST /file/chunk: Update a chunk
```
- dir
```
GET /dir: List all files in a directory
POST /dir: Create a directory
DELETE /dir: Delete a directory
```
......@@ -34,9 +34,89 @@ A `pfspath` begin with `/pfs`, eg: `/pfs/$DATACENTER/home/$USER/folder`.
Commonly, if there are two path arguments, the first is the source, and the second is the destination.
## Subcommonds
- [rm](rm.md)
- [mv](mv.md)
- [cp](cp.md)
- [ls](ls.md)
- [mkdir](mkdir.md)
- [sync](sync.md)
- rm - remove files or directories
```
Synopsis:
rm [-r] [-v] <PFSPath> ...
Options:
-r
remove directories and their contents recursively
-v
Cause rm to be verbose, showing files after they are removed.
Examples:
paddle pfs rm /pfs/$DATACENTER/home/$USER/file
paddle pfs rm -r /pfs/$DATACENTER/home/$USER/folder
```
- mv - move (rename) files
```
Synopsis:
mv [-f | -n] [-v] <LocalPath> <PFSPath>
mv [-f | -n] [-v] <LocalPath> ... <PFSPath>
mv [-f | -n] [-v] <PFSPath> <LocalPath>
mv [-f | -n] [-v] <PFSPath> ... <LocalPath>
mv [-f | -n] [-v] <PFSPath> <PFSPath>
mv [-f | -n] [-v] <PFSPath> ... <PFSPath>
Options:
-f
Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
-n
Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause mv to be verbose, showing files after they are moved.
Examples:
paddle pfs mv ./text1.txt /pfs/$DATACENTER/home/$USER/text1.txt
```
- cp - copy files or directories
```
Synopsis:
cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <LocalPath> ... <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <LocalPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> <PFSPath>
cp [-r] [-f | -n] [-v] [--preserve--links] <PFSPath> ... <PFSPath>
Options:
-r
Copy directories recursively
-f
Do not prompt for confirmation before overwriting the destination path. (The -f option overrides previous -n options.)
-n
Do not overwrite an existing file. (The -n option overrides previous -f options.)
-v
Cause cp to be verbose, showing files after they are copied.
--preserve--links
Reserve links when copy links
```
- ls- list files
```
Synopsis:
ls [-r] <PFSPath> ...
Options:
-r
List directory(ies) recursively
Examples:
paddle pfs ls /pfs/$DATACENTER/home/$USER/file
paddle pfs ls /pfs/$DATACENTER/home/$USER/folder
```
- mkdir - mkdir directory(ies)
Create intermediate directory(ies) as required.
```
Synopsis:
mkdir <PFSPath> ...
Examples:
paddle pfs mkdir /pfs/$DATACENTER/home/$USER/folder
```
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册