README.md 7.5 KB
Newer Older
D
dailidong 已提交
1 2
Dolphin Scheduler Official Website
[dolphinscheduler.apache.org](https://dolphinscheduler.apache.org)
E
easyscheduler 已提交
3
============
4

E
easyscheduler 已提交
5
[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)
K
Kirs 已提交
6
[![codecov](https://codecov.io/gh/apache/dolphinscheduler/branch/dev/graph/badge.svg)](https://codecov.io/gh/apache/dolphinscheduler/branch/dev)
7
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=apache-dolphinscheduler&metric=alert_status)](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler)
追求进步的阶梯's avatar
追求进步的阶梯 已提交
8
[![Twitter Follow](https://img.shields.io/twitter/follow/dolphinschedule.svg?style=social&label=Follow)](https://twitter.com/dolphinschedule)
K
Kirs 已提交
9
[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://s.apache.org/dolphinscheduler-slack)
D
David 已提交
10

追求进步的阶梯's avatar
追求进步的阶梯 已提交
11

12

G
Guo Wei (William) 已提交
13

K
Kirs 已提交
14
[![Stargazers over time](https://starchart.cc/apache/dolphinscheduler.svg)](https://starchart.cc/apache/dolphinscheduler)
D
dailidong 已提交
15

D
dailidong 已提交
16 17
[![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)
[![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)
G
Guo Wei (William) 已提交
18

19
## Design Features
E
easyscheduler 已提交
20

J
John Bampton 已提交
21
DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available `out of the box`.
P
ptyp 已提交
22

E
easyscheduler 已提交
23
Its main objectives are as follows:
24
 -  Highly Reliable,
L
lidongdai 已提交
25 26 27 28 29 30 31
DolphinScheduler adopts a decentralized multi-master and multi-worker architecture design, which naturally supports easy expansion and high availability (not restricted by a single point of bottleneck), and its performance increases linearly with the increase of machines
 - High performance, supporting tens of millions of tasks every day
 - Support multi-tenant.
 - Cloud Native, DolphinScheduler supports multi-cloud/data center workflow management, also
supports Kubernetes, Docker deployment and custom task types, distributed
scheduling, with overall scheduling capability increased linearly with the
scale of the cluster
T
Tq 已提交
32 33
 - Support various task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Sub_Process, Procedure, etc.
 - Support scheduling of workflows and dependencies, manual scheduling to pause/stop/recover task, support failure task retry/alarm, recover specified nodes from failure, kill task, etc.
L
lidongdai 已提交
34 35
 - Associate the tasks according to the dependencies of the tasks in a DAG graph, which can visualize the running state of the task in real-time.
 - WYSIWYG online editing tasks
T
Tq 已提交
36 37 38 39
 - Support the priority of workflows & tasks, task failover, and task timeout alarm or failure.
 - Support workflow global parameters and node customized parameter settings.
 - Support online upload/download/management of resource files, etc. Support online file creation and editing.
 - Support task log online viewing and scrolling and downloading, etc.
P
ptyp 已提交
40
 - Support the viewing of Master/Worker CPU load, memory, and CPU usage metrics.
T
Tq 已提交
41 42
 - Support displaying workflow history in tree/Gantt chart, as well as statistical analysis on the task status & process status in each workflow.
 - Support back-filling data.
P
ptyp 已提交
43
 - Support internationalization.
T
Tq 已提交
44
 - More features waiting for partners to explore...
E
easyscheduler 已提交
45

46
## What's in DolphinScheduler
47

48
 Stability | Accessibility | Features | Scalability |
49
 --------- | ------------- | -------- | ------------|
T
Tq 已提交
50 51 52
Decentralized multi-master and multi-worker | Visualization of workflow key information, such as task status, task type, retry times, task operation machine information, visual variables, and so on at a glance.  |  Support pause, recover operation | Support customized task types
support HA | Visualization of all workflow operations, dragging tasks to draw DAGs, configuring data sources and resources. At the same time, for third-party systems, provide API mode operations. | Users on DolphinScheduler can achieve many-to-one or one-to-one mapping relationship through tenants and Hadoop users, which is very important for scheduling large data jobs.  | The scheduler supports distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster. Master and Worker support dynamic adjustment.
Overload processing: By using the task queue mechanism, the number of schedulable tasks on a single machine can be flexibly configured. Machine jam can be avoided with high tolerance to numbers of tasks cached in task queue. | One-click deployment | Support traditional shell tasks, and big data platform task scheduling: MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Procedure, Sub_Process |  |
E
easyscheduler 已提交
53

54
## User Interface Screenshots
E
easyscheduler 已提交
55

56 57 58 59 60
![dag](./images/en_US/dag.png)
![data-source](./images/en_US/data-source.png)
![home](./images/en_US/home.png)
![master](./images/en_US/master.png)
![workflow-tree](./images/en_US/workflow-tree.png)
E
easyscheduler 已提交
61

62
## QuickStart in Docker
E
easyscheduler 已提交
63

S
songjianet 已提交
64
Please refer the official website document: [QuickStart in Docker](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/start/docker.html)
B
bao liang 已提交
65

66 67
## QuickStart in Kubernetes

68
Please refer to the official website document: [QuickStart in Kubernetes](https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/kubernetes.html)
69 70

## How to Build
B
bao liang 已提交
71 72

```bash
73
./mvnw clean install -Prelease
B
bao liang 已提交
74 75 76 77 78
```

Artifact:

```
79 80
dolphinscheduler-dist/target/apache-dolphinscheduler-${latest.release.version}-bin.tar.gz: Binary package of DolphinScheduler
dolphinscheduler-dist/target/apache-dolphinscheduler-${latest.release.version}-src.tar.gz: Source code package of DolphinScheduler
B
bao liang 已提交
81
```
E
easyscheduler 已提交
82

83 84
## Thanks

85
DolphinScheduler is based on a lot of excellent open-source projects, such as Google guava, grpc, netty, quartz, and many open-source projects of Apache and so on.
T
Tq 已提交
86
We would like to express our deep gratitude to all the open-source projects used in Dolphin Scheduler. We hope that we are not only the beneficiaries of open-source, but also give back to the community. Besides, we hope everyone who have the same enthusiasm and passion for open source could join in and contribute to the open-source community!
E
easyscheduler 已提交
87

88 89
## Get Help

K
Kirs 已提交
90
1. Submit an [issue](https://github.com/apache/dolphinscheduler/issues/new/choose)
K
Kirs 已提交
91
2. [Join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
92

93
## Community
94

F
fredster33 已提交
95
You are very welcome to communicate with the developers and users of Dolphin Scheduler. There are two ways to find them:
L
lidongdai 已提交
96
1. Join the Slack channel [Slack](https://asf-dolphinscheduler.slack.com/).
R
Roy 已提交
97
2. Follow the [Twitter account of DolphinScheduler](https://twitter.com/dolphinschedule) and get the latest news on time.
98

99
### Contributor over time
100 101

[![Contributor over time](https://contributor-graph-api.apiseven.com/contributors-svg?chart=contributorOverTime&repo=apache/dolphinscheduler)](https://www.apiseven.com/en/contributor-graph?chart=contributorOverTime&repo=apache/dolphinscheduler)
102

103
## How to Contribute
P
ptyp 已提交
104

F
fredster33 已提交
105
The community welcomes everyone to contribute, please refer to this page to find out more: [How to contribute](https://dolphinscheduler.apache.org/en-us/community/development/contribute.html).
T
Tq 已提交
106

L
lifeng 已提交
107 108 109 110 111 112
# Landscapes

<p align="center">
<br/><br/>
<img src="https://landscape.cncf.io/images/left-logo.svg" width="150"/>&nbsp;&nbsp;<img src="https://landscape.cncf.io/images/right-logo.svg" width="200"/>
<br/><br/>
113
DolphinScheduler enriches the <a href="https://landscape.cncf.io/?landscape=observability-and-analysis&license=apache-license-2-0">CNCF CLOUD NATIVE Landscape.</a >
L
lifeng 已提交
114 115 116

</p >

117
## License
E
easyscheduler 已提交
118

K
Kirs 已提交
119
Please refer to the [LICENSE](https://github.com/apache/dolphinscheduler/blob/dev/LICENSE) file.