[Feature-6586][Server]add some ds process definition demo when init (#11759)

* [Feature-6586][Server]add some ds process definition demo when init
1.add some ds process definition demo when init, to display what task type can run and make user easy to
use ds.
2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service
3.need modify the tenant information in it

docs/configs/docsdev.js

@@ -393,6 +393,10 @@ export default {
                         title: 'Expansion and Reduction',
                         link: '/en-us/docs/dev/user_doc/guide/expansion-reduction.html',
                     },
+                    {
+                        title: 'Demo',
+                        link: '/en-us/docs/dev/user_doc/guide/demo.html',
+                    },
                 ],
             },
             {
@@ -1013,6 +1017,10 @@ export default {
                         title: '扩/缩容',
                         link: '/zh-cn/docs/dev/user_doc/guide/expansion-reduction.html',
                     },
+                    {
+                        title: 'Demo',
+                        link: '/zh-cn/docs/dev/user_doc/guide/demo.html',
+                    },
                 ],
             },
             {

docs/docs/en/guide/demo.md

+# DolphinScheduler Initialize The Workflow Demo
+
+## Prepare
+
+### Backup Previous Version's Files and Database
+
+To prevent data loss by some miss-operation, it is recommended to back up data before initializing the workflow demo. The backup way according to your environment.
+
+### Download the Latest Version Installation Package
+
+Download the latest binary distribute package from [download](/en-us/download/download.html) and then put it in the different
+directory where current service running. And all below command is running in this directory.
+
+## Start
+
+### Start Services of DolphinScheduler
+
+Start all services of dolphinscheduler according to your deployment method. If you deploy your dolphinscheduler according to [cluster deployment](installation/cluster.md), you can start all services by command `sh ./script/start-all.sh`.
+
+### Database Configuration
+
+Initializing the workflow demo needs to store metabase in other database like MySQL or PostgreSQL, they have to change some configuration. Follow the instructions in [datasource-setting](howto/datasource-setting.md) `Standalone Switching Metadata Database Configuration` section to create and initialize database.
+
+### Tenant Configuration
+
+#### Change `dolphinscheduler-tools/resources/application.yaml` Placement Details
+
+```
+demo:
+  tenant-code: default
+  domain-name: localhost
+  api-server-port: 5173
+```
+
+Mentioned above, tenant-code is the default tenant, users can modify the user name according to their operating system, this replaces the manual tenant creation action, api-server-port is the port number of the service.
+
+Then execute the startup script that initializes the workflow demo service: `sh ./tools/bin/create-demo-processes.sh` to start the service.
+
+To create a demo, you can refer to [Quick Start](start/quick-start.md)

docs/docs/zh/guide/demo.md

+# DolphinScheduler 初始化工作流 demo
+
+## 准备工作
+
+### 备份上一版本文件和数据库
+
+为了防止操作错误导致数据丢失，建议初始化工作流 demo 服务之前备份数据，备份方法请结合你数据库的情况来定
+
+### 下载新版本的安装包
+
+在[下载](/zh-cn/download/download.html)页面下载最新版本的二进制安装包，并将二进制包放到与当前 dolphinscheduler 服务不一样的路径中，以下服务启动操作都需要在新版本的目录进行。
+
+## 服务启动步骤
+
+### 开启 dolphinscheduler 服务
+
+根据你部署方式开启 dolphinscheduler 的所有服务，如果你是通过 [集群部署](installation/cluster.md) 来部署你的 dolphinscheduler 的话，可以通过 `sh ./script/start-all.sh` 开启全部服务。
+
+### 数据库配置
+
+初始化工作流 demo 服务需要使用 MySQL 或 PostgreSQL 等其他数据库作为其元数据存储数据，因此必须更改一些配置。
+请参考[数据源配置](howto/datasource-setting.md) `Standalone 切换元数据库`创建并初始化数据库 ，然后运行 demo 服务启动脚本。
+
+### 租户配置
+
+#### 修改 `dolphinscheduler-tools/resources/application.yaml` 配置内容
+
+```
+demo:
+  tenant-code: default
+  domain-name: localhost
+  api-server-port: 5173
+```
+
+其中 tenant-code 是默认租户 default ，用户可以根据自己操作系统用户名修改，从而代替手动创建租户操作。api-server-port 是 dolphinscheduler 服务的端口号
+
+然后执行初始化工作流 demo 服务的启动脚本：`sh ./tools/bin/create-demo-processes.sh` 来启动服务。
+
+创建 demo 可以参考[快速上手](start/quick-start.md)
+

dolphinscheduler-common/pom.xml

@@ -107,6 +107,11 @@
             <artifactId>netty-all</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>com.squareup.okhttp3</groupId>
+            <artifactId>okhttp</artifactId>
+        </dependency>
+
         <dependency>
             <groupId>javax.servlet</groupId>
             <artifactId>servlet-api</artifactId>

dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/OkHttpUtils.java → dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/OkHttpUtils.java

@@ -15,14 +15,13 @@
  * limitations under the License.
  */
 
-package org.apache.dolphinscheduler.plugin.task.api.utils;
-
-import org.apache.dolphinscheduler.common.utils.JSONUtils;
+package org.apache.dolphinscheduler.common.utils;
 
 import org.apache.http.HttpStatus;
 
 import java.io.IOException;
 import java.util.Map;
+import java.util.concurrent.TimeUnit;
 
 import javax.annotation.Nullable;
 
@@ -36,7 +35,11 @@ import okhttp3.Response;
 
 public class OkHttpUtils {
 
-    private static final OkHttpClient CLIENT = new OkHttpClient();
+    private static final OkHttpClient CLIENT = new OkHttpClient.Builder()
+            .connectTimeout(5, TimeUnit.MINUTES) // connect timeout
+            .writeTimeout(5, TimeUnit.MINUTES) // write timeout
+            .readTimeout(5, TimeUnit.MINUTES) // read timeout
+            .build();
 
     public static @NonNull String get(@NonNull String url,
                                       @Nullable Map<String, String> httpHeaders,
@@ -65,6 +68,32 @@ public class OkHttpUtils {
         }
     }
 
+    public static @NonNull String demoPost(@NonNull String url,
+                                           @Nullable String token,
+                                           @Nullable Map<String, Object> requestBodyMap) throws IOException {
+
+        StringBuffer stringBuffer = new StringBuffer();
+        if (requestBodyMap != null) {
+            for (String key : requestBodyMap.keySet()) {
+                stringBuffer.append(key + "=" + requestBodyMap.get(key) + "&");
+            }
+        }
+
+        RequestBody body =
+                RequestBody.create(MediaType.parse("application/x-www-form-urlencoded"), stringBuffer.toString());
+
+        Request request = new Request.Builder()
+                .url(url)
+                .header("token", token)
+                .addHeader("accpect", "application/json")
+                .post(body)
+                .build();
+
+        try (Response response = CLIENT.newCall(request).execute()) {
+            return response.body().string();
+        }
+
+    }
     private static String addUrlParams(@Nullable Map<String, Object> requestParams, @NonNull String url) {
         if (requestParams == null) {
             return url;

dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/loop/template/http/method/HttpLoopTaskCancelTaskMethodDefinition.java

@@ -17,10 +17,10 @@
 
 package org.apache.dolphinscheduler.plugin.task.api.loop.template.http.method;
 
+import org.apache.dolphinscheduler.common.utils.OkHttpUtils;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskCancelMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskMethodDefinition;
-import org.apache.dolphinscheduler.plugin.task.api.utils.OkHttpUtils;
 
 import org.apache.commons.lang3.StringUtils;
 

dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/loop/template/http/method/HttpLoopTaskQueryStatusMethodDefinition.java

@@ -17,13 +17,13 @@
 
 package org.apache.dolphinscheduler.plugin.task.api.loop.template.http.method;
 
+import org.apache.dolphinscheduler.common.utils.OkHttpUtils;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceStatus;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskQueryStatusMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskInstanceStatus;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.utils.JsonPathUtils;
-import org.apache.dolphinscheduler.plugin.task.api.utils.OkHttpUtils;
 
 import org.apache.commons.lang3.StringUtils;
 

dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/loop/template/http/method/HttpLoopTaskSubmitTaskMethodDefinition.java

@@ -17,12 +17,12 @@
 
 package org.apache.dolphinscheduler.plugin.task.api.loop.template.http.method;
 
+import org.apache.dolphinscheduler.common.utils.OkHttpUtils;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskSubmitTaskMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.utils.JsonPathUtils;
-import org.apache.dolphinscheduler.plugin.task.api.utils.OkHttpUtils;
 
 import org.apache.commons.lang3.StringUtils;
 

dolphinscheduler-tools/src/main/bin/create-demo-processes.sh

+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BIN_DIR=$(dirname $0)
+DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/../..; pwd)}
+
+if [ "$DOCKER" != "true" ]; then
+  source "$DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh"
+fi
+
+JAVA_OPTS=${JAVA_OPTS:-"-server -Duser.timezone=${SPRING_JACKSON_TIME_ZONE} -Xms1g -Xmx1g -Xmn512m -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof"}
+
+$JAVA_HOME/bin/java $JAVA_OPTS \
+  -cp "$DOLPHINSCHEDULER_HOME/tools/conf":"$DOLPHINSCHEDULER_HOME/tools/libs/*":"$DOLPHINSCHEDULER_HOME/tools/sql" \
+  -Dspring.profiles.active=demo,${DATABASE} \
+  org.apache.dolphinscheduler.tools.demo.CreateProcessDemo

dolphinscheduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/CreateDemoTenant.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.tools.demo;
+
+import org.apache.dolphinscheduler.dao.entity.Tenant;
+import org.apache.dolphinscheduler.dao.mapper.TenantMapper;
+
+import java.util.Date;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+
+public class CreateDemoTenant {
+
+    private static final Logger logger = LoggerFactory.getLogger(CreateDemoTenant.class);
+    @Autowired
+    private TenantMapper tenantMapper;
+
+    public void createTenantCode(String tenantCode) {
+        Date now = new Date();
+
+        if (!tenantCode.equals("default")) {
+            Boolean existTenant = tenantMapper.existTenant(tenantCode);
+            if (!Boolean.TRUE.equals(existTenant)) {
+                Tenant tenant = new Tenant();
+                tenant.setTenantCode(tenantCode);
+                tenant.setQueueId(1);
+                tenant.setDescription("");
+                tenant.setCreateTime(now);
+                tenant.setUpdateTime(now);
+                // save
+                tenantMapper.insert(tenant);
+                logger.info("create tenant success");
+            } else {
+                logger.warn("os tenant code already exists");
+            }
+        }
+    }
+}

dolphinscheduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/CreateProcessDemo.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.tools.demo;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.CommandLineRunner;
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.context.annotation.ComponentScan;
+import org.springframework.context.annotation.FilterType;
+import org.springframework.context.annotation.Profile;
+import org.springframework.stereotype.Component;
+
+@SpringBootApplication
+@ComponentScan(value = "org.apache.dolphinscheduler", excludeFilters = {
+        @ComponentScan.Filter(type = FilterType.REGEX, pattern = {
+                "org.apache.dolphinscheduler.tools.datasource.*",
+        })
+})
+public class CreateProcessDemo {
+
+    public static void main(String[] args) {
+        SpringApplication.run(CreateProcessDemo.class, args);
+    }
+
+    @Component
+    @Profile("demo")
+    static class DemoRunner implements CommandLineRunner {
+
+        private static final Logger logger = LoggerFactory.getLogger(DemoRunner.class);
+
+        private final ProcessDefinitionDemo processDefinitionDemo;
+
+        DemoRunner(ProcessDefinitionDemo processDefinitionDemo) {
+            this.processDefinitionDemo = processDefinitionDemo;
+        }
+
+        @Override
+        public void run(String... args) throws Exception {
+            processDefinitionDemo.createProcessDefinitionDemo();
+            logger.info("create process definition demo success");
+        }
+    }
+}

dolphinscheduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/DemoContants.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.tools.demo;
+
+public class DemoContants {
+
+    public static final String PARAMETER_CONTEXT_PARAMS =
+            "[{\"prop\":\"output\",\"value\":\"100\",\"direct\":\"IN\",\"type\":\"VARCHAR\"},{\"prop\":\"value\",\"value\":\"99\",\"direct\":\"IN\",\"type\":\"VARCHAR\"}]";
+
+    public static final String SHELL_GLOBAL_PARAMS =
+            "[{\"prop\":\"resources\",\"value\":\"Processing information\",\"direct\":\"IN\",\"type\":\"VARCHAR\"}]";
+
+    public static final String SWITCH_GLOBAL_PARAMS =
+            "[{\"prop\":\"switchValue\",\"value\":\"A\",\"direct\":\"IN\",\"type\":\"VARCHAR\"}]";
+
+    public static final String Expire_Time = "2050-09-30 15:59:23";
+}

dolphinscheduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/ProxyProcessDefinitionController.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.tools.demo;
+
+import org.apache.dolphinscheduler.common.enums.ProcessExecutionTypeEnum;
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+import org.apache.dolphinscheduler.common.utils.OkHttpUtils;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.stereotype.Service;
+
+@Service
+public class ProxyProcessDefinitionController {
+
+    @Value("${demo.api-server-port}")
+    private String ServerPort;
+
+    @Value("${demo.domain-name}")
+    private String DomainName;
+
+    public ProxyResult createProcessDefinition(String token,
+                                               long projectCode,
+                                               String name,
+                                               String description,
+                                               String globalParams,
+                                               String locations,
+                                               int timeout,
+                                               String tenantCode,
+                                               String taskRelationJson,
+                                               String taskDefinitionJson,
+                                               ProcessExecutionTypeEnum executionType) {
+        ProxyResult proxyResult = new ProxyResult();
+        String url =
+                "http://" + DomainName + ":" + ServerPort + "/dolphinscheduler/projects/" + projectCode
+                        + "/process-definition";
+        String responseBody;
+        Map<String, Object> requestBodyMap = new HashMap<>();
+
+        requestBodyMap.put("name", name);
+        requestBodyMap.put("description", description);
+        requestBodyMap.put("globalParams", globalParams);
+        requestBodyMap.put("locations", locations);
+        requestBodyMap.put("timeout", timeout);
+        requestBodyMap.put("tenantCode", tenantCode);
+        requestBodyMap.put("taskRelationJson", taskRelationJson);
+        requestBodyMap.put("taskDefinitionJson", taskDefinitionJson);
+        requestBodyMap.put("otherParamsJson", null);
+        requestBodyMap.put("executionType", executionType);
+
+        try {
+            responseBody = OkHttpUtils.demoPost(url, token, requestBodyMap);
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+        proxyResult = JSONUtils.parseObject(responseBody, ProxyResult.class);
+
+        return proxyResult;
+
+    }
+}

dolphinscheduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/ProxyResult.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.tools.demo;
+
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+public class ProxyResult<T> {
+
+    private Integer code;
+
+    private String msg;
+
+    private T data;
+
+    public static <T> ProxyResult<T> success(T data) {
+        return new ProxyResult<>(0, "success", data);
+    }
+
+    public static ProxyResult success() {
+        return success(null);
+    }
+
+    public boolean isSuccess() {
+        if (code == 0) {
+            return true;
+        }
+        return false;
+    }
+
+    public boolean isFailed() {
+        return !this.isSuccess();
+    }
+
+}

dolphinscheduler-tools/src/main/resources/application.yaml

@@ -34,6 +34,11 @@ spring:
       leak-detection-threshold: 0
       initialization-fail-timeout: 1
 
+demo:
+  tenant-code: default
+  domain-name: localhost
+  api-server-port: 5173
+
 # Override by profile
 
 ---