Skip to content

JuiceFS
JuiceFS

Juicedata / JuiceFS

通知 10
Star 1
Fork 0
JuiceFS
JuiceFS

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

未验证 提交 3a3435b5 编写于 作者: H Herald Yu 提交者： GitHub
浏览文件
操作

Docs: A new README.md (#411) 

* Docs: A new README.md

* Docs: Add a link to README.md

* Docs: Fix some mistake and updated README_CN.md

* Docs: Update

* Docs: Fixed #411#discussion_r632334766

* Docs: fix #411 #discussion_r632341241_

* Docs: fixed https://github.com/juicedata/juicefs/pull/411#pullrequestreview-659700088
上级 317efa07
隐藏空白更改
内联 并排
Showing with 110 addition and 169 deletion
README.md
浏览文件 @ 3a3435b5
......@@ -3,25 +3,24 @@
<a href="https://travis-ci.com/juicedata/juicefs"><img alt="Build Status" src="https://travis-ci.com/juicedata/juicefs.svg?token=jKSPwswpc2ph4uMtwpHa&branch=main" /></a>
<a href="https://join.slack.com/t/juicefs/shared_invite/zt-n9h5qdxh-0bJojPaql8cfFgwerDQJgA"><img alt="Join Slack" src="https://badgen.net/badge/Slack/Join%20JuiceFS/0abd59?icon=slack" /></a>
<a href="https://goreportcard.com/report/github.com/juicedata/juicefs"><img alt="Go Report" src="https://goreportcard.com/badge/github.com/juicedata/juicefs" /></a>
<a href="README_CN.md"><img alt="Chinese Docs" src="https://img.shields.io/badge/docs-%E4%B8%AD%E6%96%87-informational" /></a>
<a href="README_CN.md"><img alt="中文手册" src="https://img.shields.io/badge/docs-%E4%B8%AD%E6%96%87%E6%89%8B%E5%86%8C-brightgreen" /></a>
</p>
**JuiceFS** is a high-performance [POSIX](https://en.wikipedia.org/wiki/POSIX) file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native environment. Using the JuiceFS to store data, the data itself will be persisted in object storage (e.g. Amazon S3), and the metadata corresponding to the data can be persisted in various database engines such as Redis, MySQL, and SQLite according to the needs of the scene.
**JuiceFS** is an open-source POSIX file system built on top of [Redis](https://redis.io) and object storage (e.g. Amazon S3), designed and optimized for cloud native environment. By using the widely adopted Redis and S3 as the persistent storage, JuiceFS serves as a stateless middleware to enable many applications to share data easily.
JuiceFS can simply and conveniently connect massive cloud storage directly to big data, machine learning, artificial intelligence, and various application platforms that have been put into production environment, without modifying the code, you can use massive cloud storage as efficiently as using local storage.
The highlighted features are:
## Highlighted Features
- **Fully POSIX-compatible**: JuiceFS is a fully POSIX-compatible file system. Existing applications can work with it without any change. See [pjdfstest result](#posix-compatibility) below.
- **Fully Hadoop-compatible**: JuiceFS [Hadoop Java SDK](docs/en/hadoop_java_sdk.md) is compatible with Hadoop 2.x and Hadoop 3.x. As well as variety of components in Hadoop ecosystem.
- **S3-compatible**: JuiceFS [S3 Gateway](docs/en/s3_gateway.md) provides S3-compatible interface.
- **Outstanding Performance**: The latency can be as low as a few milliseconds and the throughput can be expanded to nearly unlimited. See [benchmark result](#performance-benchmark) below.
- **Cloud Native**: JuiceFS provides [Kubernetes CSI driver](docs/en/how_to_use_on_kubernetes.md) to help people who want to use JuiceFS in Kubernetes.
- **Sharing**: JuiceFS is a shared file storage that can be read and written by thousands clients.
In addition, there are some other features:
- **Global File Locks**: JuiceFS supports both BSD locks (flock) and POSIX record locks (fcntl).
- **Data Compression**: JuiceFS supports use [LZ4](https://lz4.github.io/lz4) or [Zstandard](https://facebook.github.io/zstd) to compress all your data.
- **Data Encryption**: JuiceFS supports data encryption in transit and at rest, read [the guide](docs/en/encrypt.md) for more information.
1. **Fully POSIX-compatible**: Use like a local file system, seamlessly docking with existing applications, no business intrusion.
2. **Fully Hadoop-compatible**: JuiceFS [Hadoop Java SDK](docs/en/hadoop_java_sdk.md) is compatible with Hadoop 2.x and Hadoop 3.x. As well as variety of components in Hadoop ecosystem.
3. **S3-compatible**: JuiceFS [S3 Gateway](docs/en/s3_gateway.md) provides S3-compatible interface.
4. **Cloud Native**: JuiceFS provides [Kubernetes CSI driver](docs/en/how_to_use_on_kubernetes.md) to help people who want to use JuiceFS in Kubernetes.
5. **Sharing**: JuiceFS is a shared file storage that can be read and written by thousands clients.
6. **Strong Consistency**: The confirmed modification will be immediately visible on all servers mounted with the same file system .
7. **Outstanding Performance**: The latency can be as low as a few milliseconds and the throughput can be expanded to nearly unlimited. [Test results](docs/en/benchmark.md)
8. **Data Encryption**: Supports data encryption in transit and at rest, read [the guide](docs/en/encrypt.md) for more information.
9. **Global File Locks**: JuiceFS supports both BSD locks (flock) and POSIX record locks (fcntl).
10. **Data Compression**: JuiceFS supports use [LZ4](https://lz4.github.io/lz4) or [Zstandard](https://facebook.github.io/zstd) to compress all your data.
---
......@@ -31,69 +30,33 @@ In addition, there are some other features:
## Architecture
![JuiceFS Architecture](docs/images/juicefs-arch.png)
JuiceFS relies on Redis to store file system metadata. Redis is a fast, open-source, in-memory key-value data store and very suitable for storing the metadata. All the data will store into object storage through JuiceFS client.
![JuiceFS Storage Format](docs/images/juicefs-storage-format.png)
The storage format of one file in JuiceFS consists of three levels. The first level called **"Chunk"**. Each chunk has fixed size, currently it is 64MiB and cannot be changed. The second level called **"Slice"**. The slice size is variable. A chunk may have multiple slices. The third level called **"Block"**. Like chunk, its size is fixed. By default one block is 4MiB and you could modify it when formatting a volume (see following section). At last, the block will be compressed and encrypted (optional) store into object storage.
## Getting Started
### Precompiled binaries
You can download precompiled binaries from [releases page](https://github.com/juicedata/juicefs/releases).
### Building from source
You need first installing [Go](https://golang.org) 1.14+ and GCC 5.4+, then run following commands:
```bash
$ git clone https://github.com/juicedata/juicefs.git
$ cd juicefs
$ make
```
For users in China, it's recommended to set `GOPROXY` to speed up compilation, e.g. [Goproxy China](https://github.com/goproxy/goproxy.cn).
### Dependency
A Redis server (>= 2.8) is needed for metadata, please follow [Redis Quick Start](https://redis.io/topics/quickstart).
JuiceFS consists of three parts:
[macFUSE](https://osxfuse.github.io) is also needed for macOS.
1. **JuiceFS Client**: Coordinate the implementation of object storage and metadata storage engines, as well as file system interfaces such as POSIX, Hadoop, Kubernetes, and S3 gateway.
2. **Data Storage**: Store the data itself, support local disk and object storage.
3. **Metadata Engine**: Metadata corresponding to the stored data, supporting multiple engines such as Redis, MySQL, and SQLite;
The last one you need is object storage. There are many options for object storage, local disk is the easiest one to get started.
![JuiceFS Architecture](docs/images/juicefs-arch-new.png)
### Format a volume
JuiceFS relies on Redis to store file system metadata. Redis is a fast, open-source, in-memory key-value data store and very suitable for storing the metadata. All the data will store into object storage through JuiceFS client. [Learn more](docs/en/architecture.md)
Assume you have a Redis server running locally, we can create a volume called `test` using it to store metadata:
```bash
$ ./juicefs format localhost test
```
![JuiceFS Storage Format](docs/images/juicefs-storage-format-new.png)
It will create a volume with default settings. If there Redis server is not running locally, the address could be specified using URL, for example, `redis://username:password@host:6379/1`, the password can also be specified by environment variable `REDIS_PASSWORD` to hide it from command line options.
Any file stored in JuiceFS will be split into fixed-size **"Chunk"**, and the default upper limit is 64 MiB. Each Chunk is composed of one or more **"Slice"**. The length of the slice is not fixed, depending on the way the file is written. Each slice will be further split into fixed-size **"Block"**, which is 4 MiB by default. Finally, these blocks will be stored in the object storage. At the same time, JuiceFS will store each file and its Chunks, Slices, Blocks and other metadata information in metadata engines. [Learn more](docs/en/how_juicefs_store_files)
**Note: After Redis 6.0.0, [`AUTH`](https://redis.io/commands/auth) command was extended with two arguments, i.e. username and password. If you use Redis < 6.0.0, just omit the username parameter in the URL, e.g. `redis://:password@host:6379/1`.**
![How JuiceFS stores your files](docs/images/how-juicefs-stores-files-new.png)
As JuiceFS relies on object storage to store data, you can specify a object storage using `--storage`, `--bucket`, `--access-key` and `--secret-key`. By default, it uses a local directory to serve as an object store, for all the options, please see `./juicefs format -h`.
Using JuiceFS, files will eventually be split into Chunks, Slices and Blocks and stored in object storage. Therefore, you will find that the source files stored in JuiceFS cannot be found in the file browser of the object storage platform. There is a chunks directory and a bunch of digitally numbered directories and files in the bucket. Don't panic, this is the secret of the high-performance operation of the JuiceFS!
For the details about how to setup different object storage, please read [the guide](docs/en/how_to_setup_object_storage.md).
### Mount a volume
Once a volume is formatted, you can mount it to a directory, which is called *mount point*.
```bash
$ ./juicefs mount -d localhost ~/jfs
```
## Getting Started
After that you can access the volume just like a local directory.
To create a JuiceFS, you need the following 3 preparations:
To get all options, just run `./juicefs mount -h`.
1. Redis database for metadata storage
2. Object storage is used to store data blocks
3. JuiceFS Client
If you wanna mount JuiceFS automatically at boot, please read [the guide](docs/en/mount_at_boot.md).
Please refer to [Quick Start Guide](docs/en/quick_start_guide.md) to start using JuiceFS immediately!
### Command Reference
......@@ -117,7 +80,7 @@ If you wanna use JuiceFS in Hadoop, check [Hadoop Java SDK](docs/en/hadoop_java_
- [FUSE Mount Options](docs/en/fuse_mount_options.md)
- [Sync Account on Multiple Hosts](docs/en/how_to_sync_the_same_account.md)
- [Using JuiceFS on Kubernetes](docs/en/how_to_use_on_kubernetes.md)
- [Using JuiceFS on Windows](docs/en/windows.md)
- [Using JuiceFS on Windows](docs/en/juicefs_on_windows.md)
- [S3 Gateway](docs/en/s3_gateway.md)
## POSIX Compatibility
......@@ -209,11 +172,11 @@ The last number on each line is the time (in seconds) current operation takes. Y
- Local disk
- Redis
For the detailed list, see [this document](docs/en/how_to_setup_object_storage.md#supported-object-storage).
JuiceFS supports almost all object storage services. [Learn more](docs/en/how_to_setup_object_storage.md#supported-object-storage).
## Status
It's considered as beta quality, the storage format is not stabilized yet. It's not recommended to deploy it into production environment. Please test it with your use cases and give us feedback.
It's considered as beta quality, the storage format is not stabilized yet. It's not recommended to deploy it into production environment. Please test it with your use cases and give us [feedback](https://github.com/juicedata/juicefs/discussions).
## Roadmap
......@@ -266,4 +229,4 @@ See ["Redis Best Practices"](docs/en/redis_best_practices.md) for more informati
See ["Comparison with Others"](docs/en/comparison_with_others.md) for more information.
For more FAQs, please see the [full list](docs/en/faq.md).
For more FAQs, please see the [full list](docs/en/faq.md).
\ No newline at end of file
README_CN.md
浏览文件 @ 3a3435b5
......@@ -6,22 +6,21 @@
<a href="README.md"><img alt="English Docs" src="https://img.shields.io/badge/docs-English-informational" /></a>
</p>
**JuiceFS** 是一个建立在 [Redis](https://redis.io) 和 S3 等对象存储之上的开源 POSIX 文件系统。它是为云原生环境设计，通过把元数据和数据分别持久化到 Redis 和对象存储中，它相当于一个无状态的中间件，帮助各种应用通过标准的文件系统接口来共享数据。
主要特性有：
JuiceFS 是一款高性能 [POSIX](https://en.wikipedia.org/wiki/POSIX) 文件系统，针对云原生环境特别优化设计，在 GNU Affero General Public License v3.0 开源协议下发布。使用 JuiceFS 存储数据，数据本身会被持久化在对象存储（例如，Amazon S3），而数据所对应的元数据可以根据场景需求被持久化在 Redis、MySQL、SQLite 等多种数据库引擎中。JuiceFS 可以简单便捷的将海量云存储直接接入已投入生产环境的大数据、机器学习、人工智能以及各种应用平台，无需修改代码即可像使用本地存储一样高效使用海量云端存储。
- **完整 POSIX 兼容**：已有应用可以无缝对接，参考 [pjdfstest 测试结果](#posix-兼容性测试)；
- **完整 Hadoop 兼容**：JuiceFS [Hadoop Java SDK](docs/zh_cn/hadoop_java_sdk.md) 同时兼容 Hadoop 2.x 以及 Hadoop 3.x 环境，以及 Hadoop 生态中的各种主流组件。
- **S3 兼容**：JuiceFS [S3 网关](docs/en/s3_gateway.md)提供与 S3 协议兼容的访问接口；
- **极致的性能**：毫秒级的延迟，近乎无限的吞吐量（取决于对象存储规模），查看[性能测试结果](#性能测试)；
- **云原生**：通过 [Kubernetes CSI driver](docs/en/how_to_use_on_kubernetes.md) 可以很便捷地在 Kubernetes 中使用 JuiceFS；
- **共享**：可以被上千客户端同时读写，方便数据共享。
## 核心特性
除此之外，JuiceFS 还具有其它一些特性：
- **文件锁**：支持 BSD 锁（flock）及 POSIX 锁（fcntl）；
- **数据压缩**：支持使用 [LZ4](https://lz4.github.io/lz4) 或者 [Zstandard](https://facebook.github.io/zstd) 压缩数据，节省存储空间；
- **数据加密**: 支持传输中加密（encryption in transit）以及静态加密（encryption at rest），请查看[这个文档](docs/en/encrypt.md)了解更多信息。
1. **POSIX 兼容**：像本地文件系统一样使用，无缝对接已有应用，无业务侵入性；
2. **HDFS 兼容**：完整兼容 [HDFS API](docs/zh_cn/hadoop_java_sdk.md)，提供更强的元数据性能；
3. **S3 兼容**：提供 [S3 Gateway](docs/zh_cn/s3_gateway.md) 实现 S3 协议兼容的访问接口；
4. **云原生**：通过 [Kubernetes CSI driver](docs/zh_cn/how_to_use_on_kubernetes.md) 可以很便捷地在 Kubernetes 中使用 JuiceFS；
5. **多端共享**：同一文件系统可在上千台服务器同时挂载，高性能并发读写，共享数据；
6. **强一致性**：确认的修改会在所有挂载了同一文件系统的服务器上立即可见，保证强一致性；
7. **强悍性能**：毫秒级的延迟，近乎无限的吞吐量（取决于对象存储规模），查看[性能测试结果](docs/zh_cn/benchmark.md)；
8. **数据安全**：支持传输中加密（encryption in transit）以及静态加密（encryption at rest），[查看详情](docs/en/encrypt.md)；
9. **文件锁**：支持 BSD 锁（flock）及 POSIX 锁（fcntl）；
10. **数据压缩**：支持使用 [LZ4](https://lz4.github.io/lz4) 或 [Zstandard](https://facebook.github.io/zstd) 压缩数据，节省存储空间；
---
......@@ -31,76 +30,41 @@
## 架构
JuiceFS 使用 Redis 来存储文件系统的元数据。Redis 是一个开源的内存数据库，可以保障元数据的高性能访问。所有文件的数据会通过客户端存储到对象存储中，以下是它的架构图：
![JuiceFS Architecture](docs/images/juicefs-arch.png)
JuiceFS 中的文件格式，如下图所示。一个文件首先被拆分成固定大小的 **"Chunk"**，默认 64 MiB。每个 Chunk 可以由一个或者多个 **"Slice"** 组成，它们是变长的。对于每一个 Slice，又会被拆分成固定大小的 **"Block"**，默认为 4 MiB（格式化后就不可以修改）。最后，这些 Block 会被压缩和加密保存到对象存储中。压缩和加密都是可选的。
![JuiceFS Storage Format](docs/images/juicefs-storage-format.png)
## 开始使用
JuiceFS 由三个部分组成：
### 预编译版本
你可以直接下载预编译的版本：[二进制版本](https://github.com/juicedata/juicefs/releases)。
### 从源代码编译
你需要先安装 [Go](https://golang.org) 1.14+ 以及 GCC 5.4+，然后通过下面的方式来编译：
```bash
$ git clone https://github.com/juicedata/juicefs.git
$ cd juicefs
$ make
```
1. **JuiceFS 客户端**：协调对象存储和元数据存储引擎，以及 POSIX、Hadoop、Kubernetes、S3 Gateway 等文件系统接口的实现；
2. **数据存储**：存储数据本身，支持本地磁盘、对象存储；
3. **元数据引擎**：存储数据对应的元数据，支持 Redis、MySQL、SQLite 等多种引擎；
对于中国用户，建议设置 `GOPROXY` 到更快的镜像以加速编译，比如 [Goproxy 中国](https://github.com/goproxy/goproxy.cn)。
![JuiceFS Architecture](docs/images/juicefs-arch-new.png)
### 依赖
JuiceFS 依靠 Redis 来存储文件的元数据。Redis 是基于内存的高性能的键值数据存储，非常适合存储元数据。与此同时，所有数据将通过 JuiceFS 客户端存储到对象存储中。[了解详情](docs/zh_cn/architecture.md)
需要 Redis（2.8 及以上）服务器来存储元数据，请参考 [Redis Quick Start](https://redis.io/topics/quickstart)。
![JuiceFS Storage Format](docs/images/juicefs-storage-format-new.png)
如果是 macOS 系统，还需要 [macFUSE](https://osxfuse.github.io)。
任何存入 JuiceFS 的文件都会被拆分成固定大小的 **"Chunk"**，默认的容量上限是 64 MiB。每个 Chunk 由一个或多个 **"Slice"** 组成，Slice 的长度不固定，取决于文件写入的方式。每个 Slice 又会被进一步拆分成固定大小的 **"Block"**，默认为 4 MiB。最后，这些 Block 会被存储到对象存储。与此同时，JuiceFS 会将每个文件以及它的 Chunks、Slices、Blocks 等元数据信息存储在元数据引擎中。[了解详情](docs/zh_cn/how_juicefs_store_files.md)
还需要一个对象存储，测试时可以用本地目录代替。
![How JuiceFS stores your files](docs/images/how-juicefs-stores-files-new.png)
### 格式化
使用 JuiceFS，文件最终会被拆分成 Chunks、Slices 和 Blocks 存储在对象存储。因此，你会发现在对象存储平台的文件浏览器中找不到存入 JuiceFS 的源文件，存储桶中只有一个 chunks 目录和一堆数字编号的目录和文件。不要惊慌，这正是 JuiceFS 高性能运作的秘诀！
假定你已经有一个本地运行的 Redis 服务，下面用它来格式化一个叫做 `test` 的文件系统：
```bash
$ ./juicefs format localhost test
```
它会使用默认参数来格式化。如果 Redis 服务不在本地，你可以像这样完整填写它的地址：`redis://username:password@host:6379/1`。Redis 密码可以通过环境变量 `REDIS_PASSWORD` 来指定，避免暴露在命令行选项中。
**注意：Redis 6.0.0 版本以后 [`AUTH`](https://redis.io/commands/auth) 命令支持两个参数，也就是 username 和 password。如果你使用的 Redis 版本小于 6.0.0，省略 URL 中的 username 参数即可，例如 `redis://:password@host:6379/1`。**
JuiceFS 还需要一个对象存储，可以通过参数 `--storage`、`--bucket`、`--access-key` 和 `--secret-key` 来指定。它默认会使用本地目录来模拟一个对象存储用于测试，详细的参数请看 `./juicefs format -h`。
关于各种对象存储如何设置的详细介绍，请查看[这个文档](docs/en/how_to_setup_object_storage.md)。
### 挂载
一旦文件系统格式化好了，你可以把它挂载成一个目录，这个目录叫做 *挂载点*。
## 开始使用
```bash
$ ./juicefs mount -d localhost ~/jfs
```
创建 JuiceFS ，需要以下 3 个方面的准备：
挂载之后你可以像使用本地盘一样使用它，详细的挂载参数，请运行 `./juicefs mount -h`。
1. 准备 Redis 数据库
2. 准备对象存储
3. 下载安装 JuiceFS 客户端
如果你希望开机自动挂载 JuiceFS，请查看[这个文档](docs/en/mount_at_boot.md)。
请参照 [快速上手指南](docs/zh_cn/quick_start_guide.md) 立即开始使用 JuiceFS！
### 命令索引
请点击[这里](docs/en/command_reference.md)查看所有子命令以及命令行参数。
请点击 [这里](docs/zh_cn/command_reference.md) 查看所有子命令以及命令行参数。
### Kubernetes
在 Kubernetes 中使用 JuiceFS 非常便捷，请查看[这个文档](docs/en/how_to_use_on_kubernetes.md)了解更多信息。
在 Kubernetes 中使用 JuiceFS 非常便捷，请查看 [这个文档](docs/zh_cn/how_to_use_on_kubernetes.md) 了解更多信息。
### Hadoop Java SDK
......@@ -108,16 +72,16 @@ JuiceFS 使用 [Hadoop Java SDK](docs/zh_cn/hadoop_java_sdk.md) 与 Hadoop 生
## 运维管理
- [Redis 最佳实践](docs/en/redis_best_practices.md)
- [开机自动挂载 JuiceFS](docs/en/mount_at_boot.md)
- [如何设置对象存储](docs/en/how_to_setup_object_storage.md)
- [缓存管理](docs/en/cache_management.md)
- [故障诊断和分析](docs/en/fault_diagnosis_and_analysis.md)
- [FUSE 挂载选项](docs/en/fuse_mount_options.md)
- [同步账号到多个主机](docs/en/how_to_sync_the_same_account.md)
- [在 Kubernetes 中使用 JuiceFS](docs/en/how_to_use_on_kubernetes.md)
- [在 Windows 中使用 JuiceFS](docs/en/windows.md)
- [S3 网关](docs/en/s3_gateway.md)
- [Redis 最佳实践](docs/zh_cn/redis_best_practices.md)
- [开机自动挂载 JuiceFS](docs/zh_cn/mount_at_boot.md)
- [如何设置对象存储](docs/zh_cn/how_to_setup_object_storage.md)
- [缓存管理](docs/zh_cn/cache_management.md)
- [故障诊断和分析](docs/zh_cn/fault_diagnosis_and_analysis.md)
- [FUSE 挂载选项](docs/zh_cn/fuse_mount_options.md)
- [同步账号到多个主机](docs/zh_cn/how_to_sync_the_same_account.md)
- [在 Kubernetes 中使用 JuiceFS](docs/zh_cn/how_to_use_on_kubernetes.md)
- [在 Windows 中使用 JuiceFS](docs/zh_cn/windows.md)
- [S3 网关](docs/zh_cn/s3_gateway.md)
## POSIX 兼容性测试
......@@ -193,7 +157,7 @@ $ cat /jfs/.accesslog
2021.01.15 08:26:11.003616 [uid:0,gid:0,pid:4403] write (17666,390,951582): OK <0.000006>
```
每一行的最后一个数字是该操作所消耗的时间，单位是秒。你可以直接利用它来分析各种性能问题，或者尝试 `./juicefs profile /jfs` 命令实时监控统计信息。欲进一步了解此子命令请运行 `./juicefs profile -h` 或参阅[这里](docs/en/operations_profiling.md)。
每一行的最后一个数字是该操作所消耗的时间，单位是秒。你可以直接利用它来分析各种性能问题，或者尝试 `./juicefs profile /jfs` 命令实时监控统计信息。欲进一步了解此子命令请运行 `./juicefs profile -h` 或参阅[这里](docs/zh_cn/operations_profiling.md)。
## 支持的对象存储
......@@ -208,11 +172,11 @@ $ cat /jfs/.accesslog
- 本地目录
- Redis
完整的支持列表，请参照[这个文档](docs/en/how_to_setup_object_storage.md#supported-object-storage)。
JuiceFS 支持几乎所有主流的对象存储服务，[查看详情](docs/zh_cn/how_to_setup_object_storage.md)。
## 状态
JuiceFS 目前是 beta 状态，核心的存储格式还没有完全确定，还不建议使用到生产环境中。如果你对它有兴趣，请尽早测试，并给我们反馈。
JuiceFS 目前是 beta 状态，核心的存储格式还没有完全确定，还不建议使用到生产环境中。如果你对它有兴趣，请尽早测试，并给我们[反馈](https://github.com/juicedata/juicefs/discussions)。
## 产品路线图
......@@ -229,11 +193,11 @@ JuiceFS 目前是 beta 状态，核心的存储格式还没有完全确定，还
## 社区
欢迎加入 [Discussions](https://github.com/juicedata/juicefs/discussions) 和 [Slack 频道](https://join.slack.com/t/juicefs/shared_invite/zt-n9h5qdxh-0bJojPaql8cfFgwerDQJgA)跟我们的团队和其他社区成员交流。
欢迎加入 [Discussions](https://github.com/juicedata/juicefs/discussions) 和 [Slack 频道](https://join.slack.com/t/juicefs/shared_invite/zt-n9h5qdxh-0bJojPaql8cfFgwerDQJgA) 跟我们的团队和其他社区成员交流。
## 使用量收集
JuiceFS 的客户端会收集 **匿名** 使用数据来帮助我们更好地了解大家如何使用它，它只上报诸如版本号等使用量数据，不包含任何用户信息，完整的代码在[这里](pkg/usage/usage.go)。
JuiceFS 的客户端会收集 **匿名** 使用数据来帮助我们更好地了解大家如何使用它，它只上报诸如版本号等使用量数据，不包含任何用户信息，完整的代码在 [这里](pkg/usage/usage.go)。
你也可以通过下面的方式禁用它：
......@@ -265,4 +229,4 @@ JuiceFS 的设计参考了 [Google File System](https://research.google/pubs/pub
请查看[「与其它项目比较」](docs/en/comparison_with_others.md)文档了解更多信息。
更多 FAQ 请查看[完整列表](docs/zh_cn/faq.md)。
更多 FAQ 请查看[完整列表](docs/zh_cn/faq.md)。
\ No newline at end of file
docs/en/README.md
浏览文件 @ 3a3435b5
......@@ -4,7 +4,7 @@
![JuiceFS LOGO](../images/juicefs-logo.png)
JuiceFS is a high-performance [POSIX](https://en.wikipedia.org/wiki/POSIX) file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native environment. Using the JuiceFS file system to store data, the data itself will be persisted in object storage (e.g. AWS S3), and the metadata corresponding to the data can be persisted in various database engines such as Redis, MySQL, and SQLite according to the needs of the scene.
JuiceFS is a high-performance [POSIX](https://en.wikipedia.org/wiki/POSIX) file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native environment. Using the JuiceFS to store data, the data itself will be persisted in object storage (e.g. Amazon S3), and the metadata corresponding to the data can be persisted in various database engines such as Redis, MySQL, and SQLite according to the needs of the scene.
JuiceFS can simply and conveniently connect massive cloud storage directly to big data, machine learning, artificial intelligence, and various application platforms that have been put into production environment, without modifying the code, you can use massive cloud storage as efficiently as using local storage.
......@@ -16,7 +16,7 @@ JuiceFS can simply and conveniently connect massive cloud storage directly to bi
4. **Cloud Native**: JuiceFS provides [Kubernetes CSI driver](how_to_use_on_kubernetes.md) to help people who want to use JuiceFS in Kubernetes.
5. **Sharing**: JuiceFS is a shared file storage that can be read and written by thousands clients.
6. **Strong Consistency**: The confirmed modification will be immediately visible on all servers mounted with the same file system .
7. **Outstanding Performance**: The latency can be as low as a few milliseconds and the throughput can be expanded to nearly unlimited.
7. **Outstanding Performance**: The latency can be as low as a few milliseconds and the throughput can be expanded to nearly unlimited. [Test results](benchmark.md)
8. **Data Encryption**: Supports data encryption in transit and at rest, read [the guide](encrypt.md) for more information.
9. **Global File Locks**: JuiceFS supports both BSD locks (flock) and POSIX record locks (fcntl).
10. **Data Compression**: JuiceFS supports use [LZ4](https://lz4.github.io/lz4) or [Zstandard](https://facebook.github.io/zstd) to compress all your data.
......
docs/en/databases_for_metadata.md
浏览文件 @ 3a3435b5
......@@ -28,6 +28,8 @@ $ sudo docker run -d --name redis \
> **Note**: The above command persists Redis data in Docker's `redis-data` data volume. You can modify the storage location of data persistence as needed.
> **Note**: After Redis 6.0.0, [AUTH](https://redis.io/commands/auth) command was extended with two arguments, i.e. username and password. If you use Redis < 6.0.0, just omit the username parameter in the URL, e.g. `redis://:password@host:6379/1`.
> **Security Tips**: The Redis database instance created by the above command does not enable authentication and exposes the host's `6379` port. If you want to access this database instance through the Internet, please refer to [Redis Security](https:// recommendations in redis.io/topics/security).
### Create a file system
......@@ -38,6 +40,8 @@ When using Redis as the metadata storage engine, the following format is usually
redis://<IP or Domain name>:6379
```
If there Redis server is not running locally, the address could be specified using URL, for example, `redis://username:password@host:6379/1`, the password can also be specified by environment variable `REDIS_PASSWORD` to hide it from command line options.
For example, the following command will create a JuiceFS file system named `pics`, using the database No. `1` in Redis to store metadata:
```shell
......
docs/en/how_juicefs_store_files.md
浏览文件 @ 3a3435b5
......@@ -8,7 +8,7 @@ Unlike the traditional file system that can only use local disks to store data a
Any file stored in JuiceFS will be split into fixed-size **"Chunk"**, and the default upper limit is 64 MiB. Each Chunk is composed of one or more **"Slice"**. The length of the slice is not fixed, depending on the way the file is written. Each slice will be further split into fixed-size **"Block"**, which is 4 MiB by default. Finally, these blocks will be stored in the object storage. At the same time, JuiceFS will store each file and its Chunks, Slices, Blocks and other metadata information in metadata engines.
![JuiceFS storage format](../images/juicefs-storage-format-new.png)
![ ](../images/juicefs-storage-format-new.png)
Using JuiceFS, files will eventually be split into Chunks, Slices and Blocks and stored in object storage. Therefore, you will find that the source files stored in JuiceFS cannot be found in the file browser of the object storage platform. There is a chunks directory and a bunch of digitally numbered directories and files in the bucket. Don't panic, this is the secret of the high-performance operation of the JuiceFS file system!
......
docs/en/introduction.md
浏览文件 @ 3a3435b5
......@@ -4,7 +4,7 @@
![JuiceFS LOGO](../images/juicefs-logo.png)
JuiceFS is a high-performance [POSIX](https://en.wikipedia.org/wiki/POSIX) file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native environment. Using the JuiceFS file system to store data, the data itself will be persisted in object storage (e.g. AWS S3), and the metadata corresponding to the data can be persisted in various database engines such as Redis, MySQL, and SQLite according to the needs of the scene.
JuiceFS is a high-performance [POSIX](https://en.wikipedia.org/wiki/POSIX) file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native environment. Using the JuiceFS to store data, the data itself will be persisted in object storage (e.g. Amazon S3), and the metadata corresponding to the data can be persisted in various database engines such as Redis, MySQL, and SQLite according to the needs of the scene.
JuiceFS can simply and conveniently connect massive cloud storage directly to big data, machine learning, artificial intelligence, and various application platforms that have been put into production environment, without modifying the code, you can use massive cloud storage as efficiently as using local storage.
......@@ -16,7 +16,7 @@ JuiceFS can simply and conveniently connect massive cloud storage directly to bi
4. **Cloud Native**: JuiceFS provides [Kubernetes CSI driver](how_to_use_on_kubernetes.md) to help people who want to use JuiceFS in Kubernetes.
5. **Sharing**: JuiceFS is a shared file storage that can be read and written by thousands clients.
6. **Strong Consistency**: The confirmed modification will be immediately visible on all servers mounted with the same file system .
7. **Outstanding Performance**: The latency can be as low as a few milliseconds and the throughput can be expanded to nearly unlimited.
7. **Outstanding Performance**: The latency can be as low as a few milliseconds and the throughput can be expanded to nearly unlimited. [Test results](benchmark.md)
8. **Data Encryption**: Supports data encryption in transit and at rest, read [the guide](encrypt.md) for more information.
9. **Global File Locks**: JuiceFS supports both BSD locks (flock) and POSIX record locks (fcntl).
10. **Data Compression**: JuiceFS supports use [LZ4](https://lz4.github.io/lz4) or [Zstandard](https://facebook.github.io/zstd) to compress all your data.
......
docs/en/quick_start_guide.md
浏览文件 @ 3a3435b5
......@@ -59,6 +59,8 @@ $ tar -zxf juicefs-0.12.1-linux-amd64.tar.gz
$ sudo install juicefs /usr/local/bin
```
> **Note**: You can also build the JuiceFS client manually from the source code. [Learn more](client_compile_and_upgrade.md)
## 4. Create JuiceFS file system
When creating a JuiceFS file system, you need to specify both the Redis database used to store metadata and the object storage used to store actual data.
......@@ -181,7 +183,7 @@ This can happen because some programs are reading and writing files in the file
> **Risk Tips**: The commands contained in the following content may cause files damage or loss, please be cautious!
Of course, you can also add the `--force` or `-f` parameter to the unmount command to force the file system to be uninstalled, but you have to bear the possible catastrophic consequences:
Of course, you can also add the `--force` or `-f` parameter to the unmount command to force the file system to be unmounted, but you have to bear the possible catastrophic consequences:
```shell
$ sudo juicefs umount --force /mnt/jfs
......
docs/zh_cn/README.md
浏览文件 @ 3a3435b5
......@@ -4,18 +4,18 @@
![JuiceFS LOGO](../images/juicefs-logo.png)
JuiceFS 是一款高性能 [POSIX](https://en.wikipedia.org/wiki/POSIX) 文件系统，针对云原生环境特别优化设计，在 GNU Affero General Public License v3.0 开源协议下发布。使用 JuiceFS 文件系统存储数据，数据本身会被持久化在对象存储（例如，AWS S3），而数据所对应的元数据可以根据场景需求被持久化在 Redis、MySQL、SQLite 等多种数据库引擎中。JuiceFS 可以简单便捷的将海量云存储直接接入已投入生产环境的大数据、机器学习、人工智能以及各种应用平台，无需修改代码即可像使用本地存储一样高效使用海量云端存储。
JuiceFS 是一款高性能 [POSIX](https://en.wikipedia.org/wiki/POSIX) 文件系统，针对云原生环境特别优化设计，在 GNU Affero General Public License v3.0 开源协议下发布。使用 JuiceFS 存储数据，数据本身会被持久化在对象存储（例如，Amazon S3），而数据所对应的元数据可以根据场景需求被持久化在 Redis、MySQL、SQLite 等多种数据库引擎中。JuiceFS 可以简单便捷的将海量云存储直接接入已投入生产环境的大数据、机器学习、人工智能以及各种应用平台，无需修改代码即可像使用本地存储一样高效使用海量云端存储。
## 核心特性
1. **POSIX 兼容**：像本地文件系统一样使用，无缝对接已有应用，无业务侵入性；
2. **HDFS 兼容**：完整兼容 HDFS API，提供更强的元数据性能；
3. **S3 兼容**：提供与 S3 协议兼容的访问接口；
2. **HDFS 兼容**：完整兼容 [HDFS API](hadoop_java_sdk.md)，提供更强的元数据性能；
3. **S3 兼容**：提供 [S3 Gateway](s3_gateway.md) 实现 S3 协议兼容的访问接口；
4. **云原生**：通过 [Kubernetes CSI driver](juicefs_on_kubernetes.md) 可以很便捷地在 Kubernetes 中使用 JuiceFS；
5. **多端共享**：同一文件系统可在上千台服务器同时挂载，高性能并发读写，共享数据；
6. **强一致性**：确认的修改会在所有挂载了同一文件系统的服务器上立即可见，保证强一致性；
7. **强悍性能**：毫秒级的延迟，近乎无限的吞吐量（取决于对象存储规模），查看 [性能测试结果](benchmark.md)；
8. **数据安全**：支持传输中加密（encryption in transit）以及静态加密（encryption at rest）；
7. **强悍性能**：毫秒级的延迟，近乎无限的吞吐量（取决于对象存储规模），查看[性能测试结果](benchmark.md)；
8. **数据安全**：支持传输中加密（encryption in transit）以及静态加密（encryption at rest），[查看详情](../en/encrypt.md)；
9. **文件锁**：支持 BSD 锁（flock）及 POSIX 锁（fcntl）；
10. **数据压缩**：支持使用 [LZ4](https://lz4.github.io/lz4) 或 [Zstandard](https://facebook.github.io/zstd) 压缩数据，节省存储空间；
......
docs/zh_cn/databases_for_metadata.md
浏览文件 @ 3a3435b5
......@@ -28,6 +28,8 @@ $ sudo docker run -d --name redis \
> **注意**：以上命令将 Redis 的数据持久化在 Docker 的 redis-data 数据卷当中，你可以按需修改数据持久化的存储位置。
> **注意**：在 Redis 6.0.0 之后，[AUTH](https://redis.io/commands/auth) 命令扩展了两个参数，即用户名和密码。 如果您使用 Redis < 6.0.0，则只需在 URL 中省略 username 参数，例如 `redis://:password@host:6379/1`。
> **安全提示**：以上命令创建的 Redis 数据库实例没有启用身份认证，且暴露了主机的 `6379` 端口，如果你要通过互联网访问这个数据库实例，请参考 [Redis Security](https://redis.io/topics/security) 中的建议。
### 创建文件系统
......@@ -38,6 +40,8 @@ $ sudo docker run -d --name redis \
redis://<IP or Domain name>:6379
```
如果该 Redis 服务器不在本地运行，则可以使用 URL 指定地址，例如，`redis://username:password@host:6379/1`，还可以通过环境变量 `REDIS_PASSWORD` 设置密码，避免在命令行选项中显式设置。
例如，以下命令将创建一个名为 `pics` 的 JuiceFS 文件系统，使用 Redis 中的 `1` 号数据库存储元数据：
```shell
......
docs/zh_cn/introduction.md
浏览文件 @ 3a3435b5
......@@ -2,18 +2,18 @@
![JuiceFS LOGO](../images/juicefs-logo.png)
JuiceFS 是一款高性能 [POSIX](https://en.wikipedia.org/wiki/POSIX) 文件系统，针对云原生环境特别优化设计，在 GNU Affero General Public License v3.0 开源协议下发布。使用 JuiceFS 文件系统存储数据，数据本身会被持久化在对象存储（例如，AWS S3），而数据所对应的元数据可以根据场景需求被持久化在 Redis、MySQL、SQLite 等多种数据库引擎中。JuiceFS 可以简单便捷的将海量云存储直接接入已投入生产环境的大数据、机器学习、人工智能以及各种应用平台，无需修改代码即可像使用本地存储一样高效使用海量云端存储。
JuiceFS 是一款高性能 [POSIX](https://en.wikipedia.org/wiki/POSIX) 文件系统，针对云原生环境特别优化设计，在 GNU Affero General Public License v3.0 开源协议下发布。使用 JuiceFS 存储数据，数据本身会被持久化在对象存储（例如，Amazon S3），而数据所对应的元数据可以根据场景需求被持久化在 Redis、MySQL、SQLite 等多种数据库引擎中。JuiceFS 可以简单便捷的将海量云存储直接接入已投入生产环境的大数据、机器学习、人工智能以及各种应用平台，无需修改代码即可像使用本地存储一样高效使用海量云端存储。
## 核心特性
1. **POSIX 兼容**：像本地文件系统一样使用，无缝对接已有应用，无业务侵入性；
2. **HDFS 兼容**：完整兼容 HDFS API，提供更强的元数据性能；
3. **S3 兼容**：提供与 S3 协议兼容的访问接口；
4. **云原生**：通过 [Kubernetes CSI driver](how_to_use_on_kubernetes.md) 可以很便捷地在 Kubernetes 中使用 JuiceFS；
2. **HDFS 兼容**：完整兼容 [HDFS API](hadoop_java_sdk.md)，提供更强的元数据性能；
3. **S3 兼容**：提供 [S3 Gateway](s3_gateway.md) 实现 S3 协议兼容的访问接口；
4. **云原生**：通过 [Kubernetes CSI driver](juicefs_on_kubernetes.md) 可以很便捷地在 Kubernetes 中使用 JuiceFS；
5. **多端共享**：同一文件系统可在上千台服务器同时挂载，高性能并发读写，共享数据；
6. **强一致性**：确认的修改会在所有挂载了同一文件系统的服务器上立即可见，保证强一致性；
7. **强悍性能**：毫秒级的延迟，近乎无限的吞吐量（取决于对象存储规模），查看[性能测试结果](benchmark.md)；
8. **数据安全**：支持传输中加密（encryption in transit）以及静态加密（encryption at rest）；
8. **数据安全**：支持传输中加密（encryption in transit）以及静态加密（encryption at rest），[查看详情](../en/encrypt.md)；
9. **文件锁**：支持 BSD 锁（flock）及 POSIX 锁（fcntl）；
10. **数据压缩**：支持使用 [LZ4](https://lz4.github.io/lz4) 或 [Zstandard](https://facebook.github.io/zstd) 压缩数据，节省存储空间；
......
docs/zh_cn/quick_start_guide.md
浏览文件 @ 3a3435b5
......@@ -26,6 +26,8 @@ $ sudo docker run -d --name redis \
> **安全提示**：以上命令创建的 Redis 数据库实例没有启用身份认证，且暴露了主机的 `6379` 端口，如果你要通过互联网访问这个数据库实例，请参考 [Redis Security](https://redis.io/topics/security) 中的建议。
有关 Redis 数据库相关的更多内容，[点此查看](databases_for_metadata.md#Redis)。
## 二、准备对象存储
和 Redis 数据库一样，几乎所有的公有云计算平台都提供对象存储服务。因为 JuiceFS 支持几乎所有主流平台的对象存储服务，因此你可以根据个人偏好自由选择。你可以查看我们的 [对象存储支持列表和设置指南](how_to_setup_object_storage.md)，其中列出了 JuiceFS 目前支持的所有对象存储服务，以及具体的使用方法。
......@@ -61,6 +63,8 @@ $ tar -zxf juicefs-0.12.1-linux-amd64.tar.gz
$ sudo install juicefs /usr/local/bin
```
> **提示**: 你也可以从源代码手动编译 JuiceFS 客户端。[查看详情](client_compile_and_upgrade.md)
## 四、创建 JuiceFS 文件系统
创建 JuiceFS 文件系统要使用 `format` 子命令，需要同时指定用来存储元数据的 Redis 数据库和用来存储实际数据的对象存储。
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或