未验证 提交 92376c3b 编写于 作者: H Herald Yu 提交者: GitHub

Docs: translate redis_best_practices.md (#582)

* Docs: translate redis_best_practices.md

* Docs: other docs that need to be translated.

* Docs: update fio.md translate

* Update mdtest.md

* Update redis_best_practices.md

* Update s3_gateway.md

* Update sync_accounts_between_multiple_hosts.md
Co-authored-by: NChangjian Gao <gcj@juicedata.io>
上级 ed852088
# fio 基准测试
## 测试方法
使用 [fio](https://github.com/axboe/fio) 在 JuiceFS、[EFS](https://aws.amazon.com/efs)[S3FS](https://github.com/s3fs-fuse/s3fs-fuse) 上执行顺序读、顺序写基准测试。
## 测试工具
以下测试使用的工具为 fio 3.1。
顺序读测试 (任务数: 1):
```
fio --name=sequential-read --directory=/s3fs --rw=read --refill_buffers --bs=4M --size=4G
fio --name=sequential-read --directory=/efs --rw=read --refill_buffers --bs=4M --size=4G
fio --name=sequential-read --directory=/jfs --rw=read --refill_buffers --bs=4M --size=4G
```
顺序写测试 (任务数: 1):
```
fio --name=sequential-write --directory=/s3fs --rw=write --refill_buffers --bs=4M --size=4G --end_fsync=1
fio --name=sequential-write --directory=/efs --rw=write --refill_buffers --bs=4M --size=4G --end_fsync=1
fio --name=sequential-write --directory=/jfs --rw=write --refill_buffers --bs=4M --size=4G --end_fsync=1
```
顺序读测试 (任务数: 16):
```
fio --name=big-file-multi-read --directory=/s3fs --rw=read --refill_buffers --bs=4M --size=4G --numjobs=16
fio --name=big-file-multi-read --directory=/efs --rw=read --refill_buffers --bs=4M --size=4G --numjobs=16
fio --name=big-file-multi-read --directory=/jfs --rw=read --refill_buffers --bs=4M --size=4G --numjobs=16
```
顺序写测试 (任务数: 16):
```
fio --name=big-file-multi-write --directory=/s3fs --rw=write --refill_buffers --bs=4M --size=4G --numjobs=16 --end_fsync=1
fio --name=big-file-multi-write --directory=/efs --rw=write --refill_buffers --bs=4M --size=4G --numjobs=16 --end_fsync=1
fio --name=big-file-multi-write --directory=/jfs --rw=write --refill_buffers --bs=4M --size=4G --numjobs=16 --end_fsync=1
```
## 测试环境
以下测试结果均使用 fio 在亚马逊云 c5d.18xlarge EC2 (72 CPU, 144G RAM) 实例得出,操作系统采用 Ubuntu 18.04 LTS (Kernel 5.4.0) ,JuiceFS 使用同主机的本地 Redis (version 4.0.9) 实例存储元数据。
JuiceFS 挂载命令:
```
./juicefs format --storage=s3 --bucket=https://<BUCKET>.s3.<REGION>.amazonaws.com localhost benchmark
./juicefs mount --maxUpload=150 --ioretries=20 localhost /jfs
```
EFS 挂载命令 (与配置说明中一致):
```
mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport, <EFS-ID>.efs.<REGION>.amazonaws.com:/ /efs
```
S3FS (version 1.82) 挂载命令:
```
s3fs <BUCKET>:/s3fs /s3fs -o host=https://s3.<REGION>.amazonaws.com,endpoint=<REGION>,passwd_file=${HOME}/.passwd-s3fs
```
## 测试结果
![Sequential Read Write Benchmark](../images/sequential-read-write-benchmark.svg)
# mdtest 基准测试
## 测试方法
使用 [mdtest](https://github.com/hpc/ior),分别在 JuiceFS、[EFS](https://aws.amazon.com/efs)[S3FS](https://github.com/s3fs-fuse/s3fs-fuse) 上执行元数据性能测试。
## 测试工具
以下测试使用 mdtest 3.4。
调整 mdtest 的参数以确保命令可以在 5 分钟内完成。
```
./mdtest -d /s3fs/mdtest -b 6 -I 8 -z 2
./mdtest -d /efs/mdtest -b 6 -I 8 -z 4
./mdtest -d /jfs/mdtest -b 6 -I 8 -z 4
```
## 测试环境
在下面的测试结果中,所有 mdtest 均在亚马逊云 c5.large EC2 实例(2 CPU,4G RAM),Ubuntu 18.04 LTS(Kernel 5.4.0)系统上进行,JuiceFS 使用的 Redis(4.0.9版本)实例运行在相同区域的 c5.large EC2 实例上。
JuiceFS 挂载命令:
```
./juicefs format --storage=s3 --bucket=https://<BUCKET>.s3.<REGION>.amazonaws.com localhost benchmark
nohup ./juicefs mount localhost /jfs &
```
EFS 挂载命令 (与配置说明保持一致):
```
mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport, <EFS-ID>.efs.<REGION>.amazonaws.com:/ /efs
```
S3FS (version 1.82) 挂载命令:
```
s3fs <BUCKET>:/s3fs /s3fs -o host=https://s3.<REGION>.amazonaws.com,endpoint=<REGION>,passwd_file=${HOME}/.passwd-s3fs
```
## 测试结果
![Metadata Benchmark](../images/metadata-benchmark.svg)
### S3FS
```
mdtest-3.4.0+dev was launched with 1 total task(s) on 1 node(s)
Command line used: ./mdtest '-d' '/s3fs/mdtest' '-b' '6' '-I' '8' '-z' '2'
WARNING: Read bytes is 0, thus, a read test will actually just open/close.
Path : /s3fs/mdtest
FS : 256.0 TiB Used FS: 0.0% Inodes: 0.0 Mi Used Inodes: -nan%
Nodemap: 1
1 tasks, 344 files/directories
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation : 5.977 5.977 5.977 0.000
Directory stat : 435.898 435.898 435.898 0.000
Directory removal : 8.969 8.969 8.969 0.000
File creation : 5.696 5.696 5.696 0.000
File stat : 68.692 68.692 68.692 0.000
File read : 33.931 33.931 33.931 0.000
File removal : 23.658 23.658 23.658 0.000
Tree creation : 5.951 5.951 5.951 0.000
Tree removal : 9.889 9.889 9.889 0.000
```
### EFS
```
mdtest-3.4.0+dev was launched with 1 total task(s) on 1 node(s)
Command line used: ./mdtest '-d' '/efs/mdtest' '-b' '6' '-I' '8' '-z' '4'
WARNING: Read bytes is 0, thus, a read test will actually just open/close.
Path : /efs/mdtest
FS : 8388608.0 TiB Used FS: 0.0% Inodes: 0.0 Mi Used Inodes: -nan%
Nodemap: 1
1 tasks, 12440 files/directories
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation : 192.301 192.301 192.301 0.000
Directory stat : 1311.166 1311.166 1311.166 0.000
Directory removal : 213.132 213.132 213.132 0.000
File creation : 179.293 179.293 179.293 0.000
File stat : 915.230 915.230 915.230 0.000
File read : 371.012 371.012 371.012 0.000
File removal : 217.498 217.498 217.498 0.000
Tree creation : 187.906 187.906 187.906 0.000
Tree removal : 218.357 218.357 218.357 0.000
```
### JuiceFS
```
mdtest-3.4.0+dev was launched with 1 total task(s) on 1 node(s)
Command line used: ./mdtest '-d' '/jfs/mdtest' '-b' '6' '-I' '8' '-z' '4'
WARNING: Read bytes is 0, thus, a read test will actually just open/close.
Path : /jfs/mdtest
FS : 1024.0 TiB Used FS: 0.0% Inodes: 10.0 Mi Used Inodes: 0.0%
Nodemap: 1
1 tasks, 12440 files/directories
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
Directory creation : 1416.582 1416.582 1416.582 0.000
Directory stat : 3810.083 3810.083 3810.083 0.000
Directory removal : 1115.108 1115.108 1115.108 0.000
File creation : 1410.288 1410.288 1410.288 0.000
File stat : 5023.227 5023.227 5023.227 0.000
File read : 3487.947 3487.947 3487.947 0.000
File removal : 1163.371 1163.371 1163.371 0.000
Tree creation : 1503.004 1503.004 1503.004 0.000
Tree removal : 1119.806 1119.806 1119.806 0.000
```
# Redis 最佳实践
This is a guide about Redis best practices. Redis is a critical component in JuiceFS architecture. It stores all the file system metadata and serve metadata operation from client. If Redis has any problem (either service unavailable or lose data), it will affect the user experience.
这是一份关于 Redis 的最佳实践指南。Redis 是 JuiceFS 架构中的关键组件,它负责存储所有元数据并响应客户端对元数据的操作。Redis 出现任何问题(服务不可用或数据丢失),都会对用户体验造成影响。
**It's highly recommended use Redis service managed by public cloud provider if possible.** See ["Recommended Managed Redis Service"](#recommended-managed-redis-service) for more information. If you still need operate Redis by yourself in production environment, continue read following contents.
**强烈建议使用公有云上的托管 Redis 服务。** 更多信息,请参阅[「推荐的 Redis 托管服务」](#推荐的redis-托管服务)。如果您需要在生产环境中自主维护 Redis,请继续阅读以下内容。
---
> **Note**: The following paragraphs are extracted from Redis official documentation. It may outdated, subject to latest version of the official documentation.
> **注**:以下内容摘自 Redis 官方文档。它可能已经过时,请以官方文档的最新版本为准。
## High Availability
## 高可用性
[Redis Sentinel](https://redis.io/topics/sentinel) is the official high availability solution for Redis. It provides following capabilities:
[Redis Sentinel](https://redis.io/topics/sentinel) 是 Redis 官方的高可用解决方案。它提供以下功能:
- **Monitoring**. Sentinel constantly checks if your master and replica instances are working as expected.
- **Notification**. Sentinel can notify the system administrator, or other computer programs, via an API, that something is wrong with one of the monitored Redis instances.
- **Automatic failover**. If a master is not working as expected, Sentinel can start a failover process where a replica is promoted to master, the other additional replicas are reconfigured to use the new master, and the applications using the Redis server are informed about the new address to use when connecting.
- **Configuration provider**. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.
- **监控**,Sentinel 会不断检查您的 master 实例和 replica 实例是否按预期工作。
- **通知**,当受监控的 Redis 实例出现问题时,Sentinel 可以通过 API 通知系统管理员或其他计算机程序。
- **自动故障转移**,如果 master 没有按预期工作,Sentinel 可以启动一个故障转移过程,其中一个 replica 被提升为 master,其他的副本被重新配置为使用新的 master,应用程序在连接 Redis 服务器时会被告知新的地址。
- **配置提供程序**,Sentinel 会充当客户端服务发现的权威来源:客户端连接到 Sentinel 以获取当前 Redis 主节点的地址。如果发生故障转移,Sentinel 会报告新地址。
**A stable release of Redis Sentinel is shipped since Redis 2.8**. Redis Sentinel version 1, shipped with Redis 2.6, is deprecated and should not be used.
**Redis 2.8 开始提供稳定版本的 Redis Sentinel**。Redis 2.6 提供的第一版 Redis Sentinel 已被弃用,不建议使用。
There're some [fundamental things](https://redis.io/topics/sentinel#fundamental-things-to-know-about-sentinel-before-deploying) you need to know about Redis Sentinel before using it:
在使用 Redis Sentinel 之前,您需要了解一些[基础知识](https://redis.io/topics/sentinel#fundamental-things-to-know-about-sentinel-before-deploying)
1. You need at least three Sentinel instances for a robust deployment.
2. The three Sentinel instances should be placed into computers or virtual machines that are believed to fail in an independent way. So for example different physical servers or Virtual Machines executed on different availability zones.
3. **Sentinel + Redis distributed system does not guarantee that acknowledged writes are retained during failures, since Redis uses asynchronous replication.** However there are ways to deploy Sentinel that make the window to lose writes limited to certain moments, while there are other less secure ways to deploy it.
4. There is no HA setup which is safe if you don't test from time to time in development environments, or even better if you can, in production environments, if they work. You may have a misconfiguration that will become apparent only when it's too late (at 3am when your master stops working).
5. **Sentinel, Docker, or other forms of Network Address Translation or Port Mapping should be mixed with care**: Docker performs port remapping, breaking Sentinel auto discovery of other Sentinel processes and the list of replicas for a master. Check the section about Sentinel and Docker later in this document for more information.
1. 您至少需要三个 Sentinel 实例才能进行稳健的部署。
2. 这三个 Sentinel 实例应放置在彼此独立的计算机或虚拟机中。例如,分别位于不同的可用区域上的不同物理服务器或虚拟机上。
3. **由于 Redis 使用异步复制,Sentinel + Redis 分布式系统无法保证在发生故障时能够保留已确认的写入。** 然而,有一些部署 Sentinel 的方法,可以使丢失写入的窗口限于某些时刻,当然还有其他不太安全的部署方法。
4. 如果您不在开发环境中经常进行测试,就无法确保 HA 的设置是安全的。在条件允许的情况,如果能够在生产环境中进行验证则更好。错误的配置往往都是在你难以预期和响应的时间出现(比如,凌晨 3 点你的 master 节点悄然罢工)。
5. **Sentinel、Docker 或其他形式的网络地址转换或端口映射应谨慎混用**:Docker 执行端口重映射,会破坏其他 Sentinel 进程的 Sentinel 自动发现以及 master 的 replicas 列表。 查看本文档后面有关 Sentinel 和 Docker 的部分以获取更多信息。
Please read the [official documentation](https://redis.io/topics/sentinel) for more information.
更多信息请阅读[官方文档](https://redis.io/topics/sentinel)
Once Redis servers and Sentinels are deployed, the `META-URL` can be specified as `[redis[s]://][USER:PASSWORD@]MASTERNAME,SENTINEL_ADDRS:SENTINEL_PORT[/DB]`, for example:
部署了 Redis 服务器和 Sentinels 以后,`META-URL` 可以指定为 `[redis[s]://][USER:PASSWORD@]MASTERNAME,SENTINEL_ADDRS:SENTINEL_PORT[/DB]`,例如:
```bash
$ ./juicefs mount rediss://:sentinelPass@masterName,1.2.3.4,1.2.5.6:5000/2 ~/jfs
```
> **Note**: The default port for Sentinel is 26379, but the above URL use 6379 (default port for Redis server) as the default, so the port for Sentinel is not optional.
> **注意**:Sentinel 的默认端口是 26379,但是上面的 URL 使用 6379(Redis 的默认端口)作为默认端口,所以 Sentinel 的端口不是可选的。
> **Note**: When the password is provided in the URL, it will also be used to connect Redis server. If they have different passwords, the passwords should be specified by enviroment viarables (`SENTINEL_PASSWORD` and `REDIS_PASSWORD`) separately.
> **注意**:当 URL 中提供密码时,也会用于连接 Redis 服务器。 如果他们有不同的密码,密码应该由环境变量(`SENTINEL_PASSWORD` 和`REDIS_PASSWORD`)分别指定。
## Data Durability
## 数据持久性
Redis provides a different range of [persistence](https://redis.io/topics/persistence) options:
Redis 提供了不同范围的[持久性](https://redis.io/topics/persistence)选项:
- The RDB persistence performs point-in-time snapshots of your dataset at specified intervals.
- The AOF persistence logs every write operation received by the server, that will be played again at server startup, reconstructing the original dataset. Commands are logged using the same format as the Redis protocol itself, in an append-only fashion. Redis is able to rewrite the log in the background when it gets too big.
- It is possible to combine both AOF and RDB in the same instance. Notice that, in this case, when Redis restarts the AOF file will be used to reconstruct the original dataset since it is guaranteed to be the most complete.
- RDB:以指定的时间间隔生成当前数据集的快照。
- AOF:记录服务器收到的每一个写操作,在服务器启动时会重放,重建原始数据集。命令使用与 Redis 协议本身相同的格式以追加写(append-only)的方式记录。当日志变得太大时,Redis 能够在后台重写日志。
- 可以在同一个实例中组合使用 AOF 和 RDB。请注意,在这种情况下,当 Redis 重新启动时,AOF 文件将用于重建原始数据集,因为它保证是最完整的。
**It's recommended enable RDB and AOF simultaneously.** Beware that when use AOF you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress.) **but you can only lose one second worth of writes**.
**建议同时启用 RDB 和 AOF。** 请注意,当使用 AOF 时,您可以有不同的 fsync 策略:根本没有 fsync,每秒 fsync,每次查询 fsync。使用默认的 fsync 每秒写入策略性能仍然很棒(fsync 是使用后台线程执行的,当没有 fsync 正在进行时,主线程会努力执行写入),**但你可能丢失最近一秒钟的写入**
**Remember backup is also required** (disk may break, VM may disappear). Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using `rename` only when the new snapshot is complete. You can also copy the AOF file in order to create backups.
**请记住备份依然是需要的**(磁盘可能会损坏,虚拟机可能会消失),Redis 对数据备份非常友好,因为您可以在数据库运行时复制 RDB 文件:RDB 一旦生成就永远不会被修改,当它被生成时,它使用一个临时名称,并且只有在新快照完成时才使用 `rename` 原子地重命名到其最终目的地。您还可以复制 AOF 文件以创建备份。
Please read the [official documentation](https://redis.io/topics/persistence) for more information.
更多信息请阅读[官方文档](https://redis.io/topics/persistence)
## Backing up Redis Data
## 备份 Redis 数据
**Make Sure to Backup Your Database.** Disks break, instances in the cloud disappear, and so forth.
磁盘会损坏、云中的实例会消失,**请务必备份数据库!**
By default Redis saves snapshots of the dataset on disk, in a binary file called `dump.rdb`. You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the [`SAVE`](https://redis.io/commands/save) or [`BGSAVE`](https://redis.io/commands/bgsave) commands.
默认情况下,Redis 将数据集的快照保存在磁盘上,名为 `dump.rdb` 的二进制文件中。你可以根据需要,将 Redis 配置为当数据集至少发生 M 次变化时,每 N 秒保存一次,也可以手动调用 [`SAVE`](https://redis.io/commands/save)[`BGSAVE`](https://redis.io/commands/bgsave) 命令。
Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using `rename(2)` only when the new snapshot is complete.
Redis 对数据备份非常友好,因为您可以在数据库运行时复制 RDB 文件:RDB 一旦生成就永远不会被修改,当它被生成时,它使用一个临时名称,并且只有在新快照完成时才使用 `rename(2)` 原子地重命名到其最终目的地。
This means that copying the RDB file is completely safe while the server is running. This is what we suggest:
这意味着在服务器运行时复制 RDB 文件是完全安全的。以下是我们的建议:
- Create a cron job in your server creating hourly snapshots of the RDB file in one directory, and daily snapshots in a different directory.
- Every time the cron script runs, make sure to call the `find` command to make sure too old snapshots are deleted: for instance you can take hourly snapshots for the latest 48 hours, and daily snapshots for one or two months. Make sure to name the snapshots with data and time information.
- At least one time every day make sure to transfer an RDB snapshot _outside your data center_ or at least _outside the physical machine_ running your Redis instance.
- 在您的服务器中创建一个 cron 任务,在一个目录中创建 RDB 文件的每小时快照,并在另一个目录中创建每日快照。
- 每次 cron 脚本运行时,请务必调用 `find` 命令以确保删除太旧的快照:例如,您可以保留最近 48 小时的每小时快照,以及一至两个月的每日快照。要确保使用数据和时间信息来命名快照。
- 确保每天至少一次将 RDB 快照从运行 Redis 的实例传输至 _数据中心以外_ 或至少传输至 _物理机以外_ 。
Please read the [official documentation](https://redis.io/topics/persistence) for more information.
更多信息请阅读[官方文档](https://redis.io/topics/persistence)
---
## Recommended Managed Redis Service
## 推荐的 Redis 托管服务
### Amazon ElastiCache for Redis
[Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis) is a fully managed, Redis-compatible in-memory data store built for the cloud. It provides [automatic failover](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoFailover.html), [automatic backup](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups-automatic.html) features to ensure availability and durability.
[Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis) 是为云构建的完全托管的、与 Redis 兼容的内存数据存储。它提供[自动故障切换](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoFailover.html)[自动备份](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups-automatic.html)等功能以确保可用性和持久性。
> **Note**: Amazon ElastiCache for Redis has two type: cluster mode disabled and cluster mode enabled. Because JuiceFS uses [transaction](https://redis.io/topics/transactions) to guarantee the atomicity of metadata operations, so you couldn't use "cluster mode enabled" type.
> **注意**:Amazon ElastiCache for Redis 有两种类型:禁用集群模式和启用集群模式。因为 JuiceFS 使用[事务](https://redis.io/topics/transactions)来保证元数据操作的原子性,所以不能使用「启用集群模式」类型。
### Google Cloud Memorystore for Redis
[Google Cloud Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis) is a fully managed Redis service for the Google Cloud. Applications running on Google Cloud can achieve extreme performance by leveraging the highly scalable, available, secure Redis service without the burden of managing complex Redis deployments.
[Google Cloud Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis) 是针对 Google Cloud 的完全托管的 Redis 服务。通过利用高度可扩展、可用且安全的 Redis 服务,在 Google Cloud 上运行的应用程序可以实现卓越的性能,而无需管理复杂的 Redis 部署。
### Azure Cache for Redis
[Azure Cache for Redis](https://azure.microsoft.com/en-us/services/cache) is a fully managed, in-memory cache that enables high-performance and scalable architectures. Use it to create cloud or hybrid deployments that handle millions of requests per second at sub-millisecond latency—all with the configuration, security, and availability benefits of a managed service.
[Azure Cache for Redis](https://azure.microsoft.com/en-us/services/cache) 是一个完全托管的内存缓存,支持高性能和可扩展的架构。使用它来创建云或混合部署,以亚毫秒级延迟处理每秒数百万个请求——所有这些都具有托管服务的配置、安全性和可用性优势。
### Alibaba Cloud ApsaraDB for Redis
### 阿里云 ApsaraDB for Redis
[Alibaba Cloud ApsaraDB for Redis](https://www.alibabacloud.com/product/apsaradb-for-redis) is a database service that is compatible with native Redis protocols. It supports a hybrid of memory and hard disks for data persistence. ApsaraDB for Redis provides a highly available hot standby architecture and can scale to meet requirements for high-performance and low-latency read/write operations.
[阿里云 ApsaraDB for Redis](https://www.alibabacloud.com/product/apsaradb-for-redis) 是一种兼容原生 Redis 协议的数据库服务。它支持混合内存和硬盘以实现数据持久性。云数据库 Redis 版提供高可用的热备架构,可扩展以满足高性能、低延迟的读写操作需求。
> **Note**: ApsaraDB for Redis supports 3 type [architectures](https://www.alibabacloud.com/help/doc-detail/86132.htm): standard, cluster and read/write splitting. Because JuiceFS uses [transaction](https://redis.io/topics/transactions) to guarantee the atomicity of metadata operations, so you couldn't use cluster type architecture.
> **注意**:ApsaraDB for Redis 支持 3 种类型的[架构](https://www.alibabacloud.com/help/doc-detail/86132.htm):标准、集群和读写分离。因为 JuiceFS 使用[事务](https://redis.io/topics/transactions)来保证元数据操作的原子性,所以不能使用集群类型架构。
### Tencent Cloud TencentDB for Redis
### 腾讯云 TencentDB for Redis
[Tencent Cloud TencentDB for Redis](https://intl.cloud.tencent.com/product/crs) is a caching and storage service compatible with the Redis protocol. It features a rich variety of data structure options to help you develop different types of business scenarios, and offers a complete set of database services such as primary-secondary hot backup, automatic switchover for disaster recovery, data backup, failover, instance monitoring, online scaling and data rollback.
[腾讯云 TencentDB for Redis](https://intl.cloud.tencent.com/product/crs) 是一种兼容 Redis 协议的缓存和存储服务。丰富多样的数据结构选项,帮助您开发不同类型的业务场景,提供主从热备份、容灾自动切换、数据备份、故障转移、实例监控、在线等一整套数据库服务缩放和数据回滚。
> **Note**: TencentDB for Redis supports 2 type [architectures](https://intl.cloud.tencent.com/document/product/239/3205): standard and cluster. Because JuiceFS uses [transaction](https://redis.io/topics/transactions) to guarantee the atomicity of metadata operations, so you couldn't use cluster type architecture.
> **注意**:TencentDB for Redis 支持两种类型的[架构](https://intl.cloud.tencent.com/document/product/239/3205):标准和集群。因为 JuiceFS 使用[事务](https://redis.io/topics/transactions)来保证元数据操作的原子性,所以不能使用集群类型架构。
# S3 Gateway
# S3 网关
JuiceFS S3 Gateway is a service which provides S3-compatible interface. It means you could interact with JuiceFS by existing tools, e.g. AWS CLI, s3cmd, MinIO client (`mc`). The JuiceFS S3 Gateway is based on [MinIO S3 Gateway](https://docs.min.io/docs/minio-gateway-for-s3.html).
JuiceFS S3 网关是一个提供 S3 兼容接口的服务。这意味着您可以通过现有工具与 JuiceFS 进行交互,例如 AWS CLI、s3cmd、MinIO 客户端 (`mc`)。JuiceFS S3 网关基于 [MinIO S3 网关](https://docs.min.io/docs/minio-gateway-for-s3.html)
## Prerequisites
## 先决条件
Before running the gateway, you need first formatting a volume. Please [follow the steps](../../README.md#getting-started) in README.
在运行网关之前,您需要先参考 [快速上手指南](quick_start_guide.md) 创建一个文件系统。
JuiceFS S3 Gateway is a feature introduced in v0.11.0, please ensure you have latest version of JuiceFS.
JuiceFS S3 网关是 v0.11.0 中引入的功能,请确保您拥有最新版本的 JuiceFS。
## Quickstart
## 快速开始
Use `juicefs gateway` command to run the gateway, [most options](command_reference.md#juicefs-gateway) of this command are same as `juicefs mount`, except following options:
使用 `juicefs gateway` 命令运行网关,该命令的 [大多数选项](command_reference.md#juicefs-gateway)`juicefs mount` 相同,除了以下选项:
```
--access-log value path for JuiceFS access log
--no-banner disable MinIO startup information (default: false)
--access-log value JuiceFS 访问日志的路径
--no-banner 禁用 MinIO 启动信息(默认:false)
```
The `--access-log` option controls where to store [access log](fault_diagnosis_and_analysis.md#access-log) of JuiceFS. By default access log will not be stored. The `--no-banner` option controls if disable logs from MinIO.
`--access-log` 选项控制 JuiceFS 的 [访问日志](fault_diagnosis_and_analysis.md#访问日志) 的存储位置。默认情况下,不会存储访问日志。`--no-banner` 选项控制是否禁用 MinIO 的日志。
MinIO S3 Gateway requires two environment variables been configured before startup: `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD`. You can set them with any value, but must meet the length requirements. `MINIO_ROOT_USER` length should be at least 3, and `MINIO_ROOT_PASSWORD` length at least 8 characters.
MinIO S3 网关需要在启动前配置两个环境变量:`MINIO_ROOT_USER``MINIO_ROOT_PASSWORD`。您可以将它们设置为任何值,但必须满足长度要求。`MINIO_ROOT_USER` 长度至少应为 3 个字符,而 `MINIO_ROOT_PASSWORD` 长度至少应为 8 个字符。
The following command shows how to run a gateway. The Redis address is `localhost:6379`, and the gateway is listening on `localhost:9000`.
以下命令展示了如何运行网关,Redis 地址是 `localhost:6379`,网关监听在 `localhost:9000`
```bash
$ export MINIO_ROOT_USER=admin
......@@ -29,13 +29,13 @@ $ export MINIO_ROOT_PASSWORD=12345678
$ juicefs gateway redis://localhost:6379 localhost:9000
```
If the gateway is running successfully, you could visit [http://localhost:9000](http://localhost:9000) in the browser:
如果网关运行成功,您可以在浏览器中访问 [http://localhost:9000](http://localhost:9000)
![MinIO browser](../images/minio-browser.png)
## Use AWS CLI
## 使用 AWS CLI
Install AWS CLI from [https://aws.amazon.com/cli](https://aws.amazon.com/cli). Then you need configure it:
[https://aws.amazon.com/cli](https://aws.amazon.com/cli) 安装 AWS CLI,然后进行配置:
```bash
$ aws configure
......@@ -45,9 +45,9 @@ Default region name [None]:
Default output format [None]:
```
The Access Key ID is same as `MINIO_ROOT_USER`, and Secret Access Key is same as `MINIO_ROOT_PASSWORD`. Region name and output format could be empty.
`Access Key ID``MINIO_ROOT_USER` 相同,`Secret Access Key``MINIO_ROOT_PASSWORD` 相同。区域名称和输出格式可以为空。
After that, you could use `aws s3` command to access the gateway, for example:
之后,您可以使用 `aws s3` 命令访问网关,例如:
```bash
# List buckets
......@@ -57,15 +57,15 @@ $ aws --endpoint-url http://localhost:9000 s3 ls
$ aws --endpoint-url http://localhost:9000 s3 ls s3://<bucket>
```
## Use MinIO Client
## 使用 MinIO 客户端
Install MinIO client from [https://docs.min.io/docs/minio-client-complete-guide.html](https://docs.min.io/docs/minio-client-complete-guide.html). Then add a new host called `juicefs`:
[https://docs.min.io/docs/minio-client-complete-guide.html](https://docs.min.io/docs/minio-client-complete-guide.html) 安装 MinIO 客户端,然后添加一个名为 `juicefs` 的新主机:
```bash
$ mc alias set juicefs http://localhost:9000 admin 12345678 --api S3v4
```
After that, you could use `mc` command to access the gateway, for example:
之后,您可以使用 `mc` 命令访问网关,例如:
```bash
# List buckets
......@@ -73,4 +73,4 @@ $ mc ls juicefs
# List objects in bucket
$ mc ls juicefs/<bucket>
```
\ No newline at end of file
```
# JuiceFS 多主机间同步账户
JuiceFS supports POSIX compatible ACL to manage permissions in the granularity of directory or file. The behavior is the same as a local file system.
JuiceFS 支持 POSIX 兼容的 ACL,以目录或文件的粒度管理权限。该行为与本地文件系统相同。
In order to make the permission experience intuitive to user (e.g. the files accessible by user A in host X should be accessible in host Y with the same user), the same user who want to access JuiceFS should have the same UID and GID on all hosts.
为了让用户获得直观的权限管理体验(例如,用户 A 在主机 X 中访问的文件,在主机 Y 中也应该可以用相同的用户身份访问),想要访问 JuiceFS 存储的同一个用户,应该在所有主机上具有相同的 UID 和 GID。
Here we provide a simple [Ansible](https://www.ansible.com/community) playbook to demonstrate how to ensure an account with same UID and GID on multiple hosts.
在这里,我们提供了一个简单的 [Ansible](https://www.ansible.com/community) playbook 来演示如何确保一个帐户在多个主机上具有相同的 UID 和 GID。
**Note: You can use two files to specify the id for all users and groups.**
**注意:您可以使用两个文件来指定所有用户和组的 ID。**
## 安装 Ansible
选择一个主机作为 [控制节点](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#managed-node-requirements),它可以使用 `ssh``root` 或其他在 sudo 用户组的身份,访问所有。在此主机上安装 Ansible。阅读 [安装 Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible) 了解更多安装细节。
## Install ansible
## 确保所有主机上的帐户相同
Select a host as a [control node](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#managed-node-requirements) which can access all hosts using `ssh` with the same privileged account like `root` or other sudo account. Install ansible on this host. Read [Installing Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible) for more installation details.
## Ensure the same account on all hosts
Create an empty directory `account-sync` , save below content in `play.yaml` under this directory.
创建一个空目录 `account-sync` ,将下面的内容保存在该目录下的 `play.yaml` 中。
```yaml
---
......@@ -38,11 +34,9 @@ Create an empty directory `account-sync` , save below content in `play.yaml` und
state: present
```
在该目录下创建一个名为 `hosts` 的文件,将所有需要创建账号的主机的 IP 地址放置在该文件中,每行一个 IP。
Create a file named `hosts` in this directory, place IP addresses of all hosts need to create account in this file, each line with a host's IP.
Here we ensure an account `alice` with UID 1200 and group `staff` with GID 500 on 2 hosts:
在这里,我们确保在 2 台主机上使用 UID 1200 的帐户 `alice` 和 GID 500 的 `staff` 组:
```
~/account-sync$ cat hosts
......@@ -70,9 +64,9 @@ PLAY RECAP *********************************************************************
172.16.255.180 : ok=3 changed=1 unreachable=0 failed=0
```
Now the new account `alice:staff` has been created on these 2 hosts.
现在已经在这 2 台主机上创建了新帐户 `alice:staff`。
If the UID or GID specified has been allocated to another user or group on some hosts, the creation would failed.
如果指定的 UID 或 GID 已分配给某些主机上的另一个用户或组,则创建将失败。
```
~/account-sync$ ansible-playbook -i hosts -u root --ssh-extra-args "-o StrictHostKeyChecking=no" \
......@@ -97,13 +91,13 @@ PLAY RECAP *********************************************************************
172.16.255.180 : ok=1 changed=0 unreachable=0 failed=1
```
In above example, the group ID 1000 has been allocated to another group on host `172.16.255.180` , we should **change the GID** or **delete the group with GID 1000** on host `172.16.255.180` , then run the playbook again.
在上面的示例中,组 ID 1000 已分配给主机 `172.16.255.180` 上的另一个组,我们应该 **更改 GID** 或 **删除主机 `172.16.255.180` 上 GID 为 1000** 的组,然后再次运行 playbook。
> **CAUTION**
> **小心**
>
> If the user account has already existed on the host and we change it to another UID or GID value, the user may loss permissions to the files and directories which they previously have. For example:
> 如果用户帐户已经存在于主机上,并且我们将其更改为另一个 UID 或 GID 值,则用户可能会失去对他们以前拥有的文件和目录的权限。例如:
>
> ```
> $ ls -l /tmp/hello.txt
......@@ -112,14 +106,14 @@ In above example, the group ID 1000 has been allocated to another group on host
> uid=1200(alice) gid=500(staff) groups=500(staff)
> ```
>
> We change the UID of alice from 1200 to 1201
> 我们将 alice 的 UID 从 1200 改为 1201
>
> ```
> ~/account-sync$ ansible-playbook -i hosts -u root --ssh-extra-args "-o StrictHostKeyChecking=no" \
> --extra-vars "group=staff gid=500 user=alice uid=1201" play.yaml
> ```
>
> Now we have no permission to remove this file as its owner is not alice:
> 现在我们没有权限删除这个文件,因为它的所有者不是 alice:
>
> ```
> $ ls -l /tmp/hello.txt
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册