提交 · 68cb81fdf5e62c95444f22837545c3c3b8604ba3 · DolphinScheduler社区 / DolphinScheduler

28 3月, 2020 1 次提交

1，add UT in pom 2，refactor TaskUpdateQueue (#2326) · 68cb81fd

由 qiaozhanwei 提交于 3月 28, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency，MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1，get workergroup from zk modify
2，SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

* master fault tolerant bug modify

* UT add pom.xml

* timing online  modify

* when taskResponse is faster than taskAck to db，task state will error
add async queue and new a thread reslove this problem

* TaskExecutionContext set host

* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify

* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify

* add UT in pom.xml

* revert dolphinscheduler-daemon.sh

* ZookeeperRegister use common.properties zookeeperRoot path

* api start exclude org.apache.dolphinscheduler.server.*

* ZookeeperRegister use common.properties zookeeperRoot path

* 1，api start load server filter
2，SHELL task exitStatusCode modify

* java doc error modify

* java doc error modify

* remove todo

* add UT in pom

* 1，add UT in pom
2，refactor TaskUpdateQueue
Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>

68cb81fd

25 3月, 2020 1 次提交

1，ZookeeperRegister use common.properties zookeeperRoot path 2，api start... · 03e29b93

由 qiaozhanwei 提交于 3月 25, 2020

1，ZookeeperRegister use common.properties zookeeperRoot path 2，api start exclude org.apache.dolphinscheduler.server.* (#2307)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization

* AbstractTask modify

* ProcedureTask optimization

* MasterSchedulerService modify

* TaskUpdateQueueConsumer modify

* test

* DataxTask process run debug

* DataxTask process run debug

* add protobuf dependency，MR、Spark task etc need this

* TaskUpdateQueueConsumer modify

* TaskExecutionContextBuilder set TaskInstance workgroup

* WorkerGroupService queryAllGroup modify
query available work group

* 1，get workergroup from zk modify
2，SpringConnectionFactory repeat load modify

* master and worker register ip  use OSUtils.getHost()

* ProcessInstance host set ip:port format

* worker fault tolerance modify

* Constants and .env modify

* master fault tolerant bug modify

* UT add pom.xml

* timing online  modify

* when taskResponse is faster than taskAck to db，task state will error
add async queue and new a thread reslove this problem

* TaskExecutionContext set host

* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify

* 1，TaskManager refactor
2， api start load server dolphinschedule-daemon.sh modify

* add UT in pom.xml

* revert dolphinscheduler-daemon.sh

* ZookeeperRegister use common.properties zookeeperRoot path

* api start exclude org.apache.dolphinscheduler.server.*

* ZookeeperRegister use common.properties zookeeperRoot path

* 1，api start load server filter
2，SHELL task exitStatusCode modify

* java doc error modify

* java doc error modify

* remove todo
Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>

03e29b93

10 3月, 2020 1 次提交

1，encapsulate the parameters required by sqltask 2，SQLTask optimization (#2135) · fd2c2eeb

由 qiaozhanwei 提交于 3月 10, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs

* WorkerServer refactor

* MasterSchedulerService modify

* WorkerConfig listen port modify

* modify master and worker listen port

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* cancelTaskInstance set TaskExecutionContext host,logPath,executePath

* Encapsulate the parameters required by sqltask

* 1，Encapsulate the parameters required by sqltask
2，SQLTask optimization

* AbstractTask modify
Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>

fd2c2eeb

06 3月, 2020 1 次提交

TaskPriority refactor (#2097) · eb45ff9e

由 qiaozhanwei 提交于 3月 06, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

* task prioriry refator

* remove ITaskQueue

* TaskPriority refactor

* remove logs
Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>

eb45ff9e

05 3月, 2020 1 次提交

task prioriry refator (#2094) · 8aaee39a

由 qiaozhanwei 提交于 3月 05, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

* remove chinese log

* executeDirectly method add Override

* remote module modify

* TaskKillResponseProcessor command type modify

* create buildKillCommand

* host add host:port format

* host add host:port format

* TaskAckProcessor modify

* TaskAckProcessor modify

* task prioriry refator

* remove ITaskQueue

8aaee39a

04 3月, 2020 2 次提交
- T
  Refactor worker (#2081) · 15bf7402
  由 Tboy 提交于 3月 04, 2020
```
* refactor kill logic

* refactor ExecutionContext
```
  15bf7402
- T
  
  refactor ExecutionContext · 09173e8d
  由 Technoboy- 提交于 3月 04, 2020
  
  09173e8d
02 3月, 2020 1 次提交

master add kill task logic (#2058) · bb3885cf

由 qiaozhanwei 提交于 3月 02, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify

* worker remove db

* ShellTask modify

* master persistence processId and appIds

* master persistence processId and appIds

* master add kill task logic

* master add kill task logic

* master add kill task logic

* javadoc error modify

bb3885cf

28 2月, 2020 1 次提交

1，worker TaskPros use TaskExecutionContext replase TaskPros 2，Master kill Task... · 7f94122f

由 qiaozhanwei 提交于 2月 28, 2020

1，worker TaskPros use TaskExecutionContext replase TaskPros 2，Master kill Task , KillTaskProcessor modify (#2039)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* taskInstanceCache is null ,need load from db

* 1，worker TaskPros use TaskExecutionContext replase
2，Master kill Task , KillTaskProcessor modify

7f94122f

26 2月, 2020 2 次提交

Refactor worker (#10) · 72670683

由 Tboy 提交于 2月 26, 2020

* Refactor worker (#2000)

* Refactor worker (#2)

* Refactor worker (#1993)

* Refactor worker (#1)

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract master and worker communication model (#1992)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* TaskExecutionContext create modify (#1994)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates

* add- register processor
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* buildAckCommand taskInstanceId not set modify (#2002)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify (#2004)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment (#2006)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type (#2012)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* Refactor worker (#2018)

* Refactor worker (#7)

* Refactor worker (#2000)

* Refactor worker (#2)

* Refactor worker (#1993)

* Refactor worker (#1)

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract master and worker communication model (#1992)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* TaskExecutionContext create modify (#1994)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates

* add- register processor
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* buildAckCommand taskInstanceId not set modify (#2002)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify (#2004)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment (#2006)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type (#2012)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* Refactor worker (#8)

* Refactor worker (#2000)

* Refactor worker (#2)

* Refactor worker (#1993)

* Refactor worker (#1)

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract master and worker communication model (#1992)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* TaskExecutionContext create modify (#1994)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates

* add- register processor
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* buildAckCommand taskInstanceId not set modify (#2002)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify (#2004)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment (#2006)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type (#2012)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* add kill command
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* add TaskInstanceCacheManager receive Worker report result，modify master polling db transfrom to cache (#2021)

* 1, master persistent task
2. extract master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

72670683

add TaskInstanceCacheManager receive Worker report result，modify master... · 2bb5ebaf

由 qiaozhanwei 提交于 2月 26, 2020

add TaskInstanceCacheManager receive Worker report result，modify master polling db transfrom to cache (#2021)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

* add TaskInstanceCacheManager receive Worker report result

* TaskInstance setExecutePath

* add TaskInstanceCacheManager to receive Worker Task result report

* TaskInstanceCacheManager add remove method

* add license

* add dispatcht task method

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

* AbstractCommandExecutor remove db access

2bb5ebaf

25 2月, 2020 2 次提交

Refactor worker (#7) · f0f86447

由 Tboy 提交于 2月 25, 2020

* Refactor worker (#2000)

* Refactor worker (#2)

* Refactor worker (#1993)

* Refactor worker (#1)

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract  master and worker communication model (#1992)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* TaskExecutionContext create modify (#1994)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates

* add- register processor
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* buildAckCommand taskInstanceId not set modify (#2002)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify (#2004)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment (#2006)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type (#2012)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

f0f86447

ExecutorManager interface add generic type (#2012) · 33bf550a

由 qiaozhanwei 提交于 2月 25, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

* buildAckCommand taskInstanceId not set modify

* java doc error modify

* add comment

* ExecutorManager interface add generic type

33bf550a

22 2月, 2020 3 次提交

Refactor worker (#1997) · e4679109

由 Tboy 提交于 2月 22, 2020

* Refactor worker (#2)

* Refactor worker (#1993)

* Refactor worker (#1)

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract  master and worker communication model (#1992)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* TaskExecutionContext create modify (#1994)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

e4679109

T

updates · 00c9b801
由 Technoboy- 提交于 2月 22, 2020

00c9b801

Refactor worker (#2) · 65892205

由 Tboy 提交于 2月 22, 2020

* Refactor worker (#1993)

* Refactor worker (#1)

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract  master and worker communication model (#1992)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* TaskExecutionContext create modify (#1994)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

65892205

21 2月, 2020 6 次提交

TaskExecutionContext create modify (#1994) · 9d89eb03

由 qiaozhanwei 提交于 2月 21, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

* TaskExecutionContext create modify

9d89eb03

Refactor worker (#1993) · 3800a2a9

由 Tboy 提交于 2月 21, 2020

* Refactor worker (#1)

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract  master and worker communication model (#1992)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

* updates
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

3800a2a9

T

updates · 64d538e1
由 Technoboy- 提交于 2月 21, 2020

64d538e1

Refactor worker (#1) · d6ea202e

由 Tboy 提交于 2月 21, 2020

* add TaskResponseProcessor (#1983)

* 1, master persistent task 2. extract  master and worker communication model (#1992)

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error
Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>

d6ea202e

1, master persistent task 2. extract master and worker communication model (#1992) · ff86dc7d

由 qiaozhanwei 提交于 2月 21, 2020

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* 1, master persistent task
2. extract  master and worker communication model

* add license

* modify javadoc error

ff86dc7d

T

add TaskResponseProcessor (#1983) · 158322bd
由 Tboy 提交于 2月 21, 2020

158322bd

20 2月, 2020 6 次提交
- T
  
  add TaskResponseProcessor · 63b76d71
  由 Technoboy- 提交于 2月 20, 2020
  
  63b76d71
- Q
  
  master/worker basic communication · 2bb2d0ce
  由 qiaozhanwei 提交于 2月 20, 2020
  
  2bb2d0ce
- Q
  
  master/worker basic communication · 06927c2c
  由 qiaozhanwei 提交于 2月 20, 2020
  
  06927c2c
- Q
  
  master/worker basic communication · f6a2130b
  由 qiaozhanwei 提交于 2月 20, 2020
  
  f6a2130b
- Q
  
  master/worker basic communication · 910401fb
  由 qiaozhanwei 提交于 2月 20, 2020
  
  910401fb
- Q
  
  master/worker basic communication · e7c32aff
  由 qiaozhanwei 提交于 2月 20, 2020
  
  e7c32aff
14 2月, 2020 1 次提交

Refactor Architecture Basic modification #1658 (#1946) · c1ee1333

由 qiaozhanwei 提交于 2月 14, 2020

* 1，remove dolphinscheduler-rpc module  2，add dolphinscheduler-remote module 3，add dolphinscheduler-service module 4，refactor LoggerServer module (#1925)

* 1，remove dolphinscheduler-rpc module
2，add dolphinscheduler-remote module
3，add dolphinscheduler-service module
4，refactor LoggerServer module

* ProcessUtils modify

* Refactor architecture (#1926)

* move version to parent pom

* move version properties to parent pom for easy management

* remove freemarker dependency

* delete CombinedApplicationServer

* #1871 correct spelling

* #1873 some updates for TaskQueueZkImpl

* #1875 remove unused properties in pom

* #1878
1. remove tomcat dependency
2. remove combined_logback.xml in api module
3. format pom.xml for not aligning

* #1885 fix api server startup failure
1. add jsp-2.1 dependency
2. remove jasper-runtime dependency

* add stringutils ut (#1921)

* add stringutils ut

* Newfeature for #1675. (#1908)

Continue to finish the rest works, add the cache feature for dependence,mr,python,sub_process,procedure and shell.

* Add modify user name for process definition (#1919)

* class overrides equals() and should therefore also override hashCode()

* #1862 add modify user in process difinition list

* #1862 add pg-1.2.2 ddl.sql

* modify ScriptRunnerTest

* add updateProessDifinition UT

* modify updateProcessDifinition UT

* modify updateProcessDifinition UT

* modify mysql 1.2.2 ddl.sql&dml.sql

* add scope test to mysql in pom

* modify pg-1.2.2 ddl.sql

* refactor module

* updates
Co-authored-by: Nkhadgarmage <khadgar.mage@outlook.com>
Co-authored-by: Nzhukai <boness@qq.com>
Co-authored-by: NYelli <amarantine@my.com>

* dolphinscheduler-common remove spring (#1931)

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* SpringApplicationContext class title add license (#1932)

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* add license (#1934)

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* Refactor architecture (#1936)

* move datasource classes to dao module

* fix send4LetterWord bug

* LoggerServiceTest remove ProcessDao (#1944)

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* dolphinscheduler-common remove spring

* LoggerServiceTest remove ProcessDao

* exclude jasper-compiler in case of runtime conflict (#1938)

* move datasource classes to dao module

* fix send4LetterWord bug

* exclude jasper-compiler in case of runtime conflict

* DataAnaylysisServiceTest and ProcessDefinitionService modify

* remote module add comment

* OSUtilsTest modify

* add finally block to close channel (#1951)

* move datasource classes to dao module

* fix send4LetterWord bug

* exclude jasper-compiler in case of runtime conflict

* add finally block to close channel
Co-authored-by: NTboy <technoboy@yeah.net>
Co-authored-by: Nkhadgarmage <khadgar.mage@outlook.com>
Co-authored-by: Nzhukai <boness@qq.com>
Co-authored-by: NYelli <amarantine@my.com>

c1ee1333

07 1月, 2020 1 次提交

ds.server regularize log print (#1751) · 8125974c

由 Yelli 提交于 1月 07, 2020

* ds.server regularize log print

* log print modify

* log print modify

* modify lang3 to lang

8125974c

27 12月, 2019 1 次提交

refactor TaskQueueZkImpl (#1591) · 8b2b5ba6

由 Tboy 提交于 12月 27, 2019

* fix #1515

* sleep when resource in not satisfy. fix #1522

* add sleep 1s for no command

* fix MasterBaseTaskExecThread submit method bug

* updates

* add log

* delete lombok

* remove duplicate code

* refactor TaskQueueZkImpl

* ignore First , we have to rewrite

* updates

8b2b5ba6

26 12月, 2019 1 次提交
- D
  
  refactor TaskQueueZkImpl · f59bb10c
  由 dk.technoboy 提交于 12月 26, 2019
  
  f59bb10c
24 12月, 2019 2 次提交
- T
  fix MasterBaseTaskExecThread submit method bug (#1532) · 26d98931
  由 Tboy 提交于 12月 24, 2019
```
* fix #1515

* sleep when resource in not satisfy. fix #1522

* add sleep 1s for no command

* fix MasterBaseTaskExecThread submit method bug

* updates

* add log
```
  26d98931
- T
  
  add log · 16c29a00
  由 Technoboy- 提交于 12月 24, 2019
  
  16c29a00
20 12月, 2019 2 次提交
- T
  
  updates · d14cefdf
  由 Technoboy- 提交于 12月 20, 2019
  
  d14cefdf
- D
  
  fix MasterBaseTaskExecThread submit method bug · 7bb0bebf
  由 dk.technoboy 提交于 12月 20, 2019
  
  7bb0bebf
18 12月, 2019 2 次提交

merge db (#2) · 09540843

由 Tboy 提交于 12月 18, 2019

* Add FileUtilsTest.java , the unit test for FileUtils (#1493)

* #839 enhancement : add Spark Task Component can switch Spark Version (#1494)

* add Spark Version in Spark Component

add Spark Version in Spark Component

* add license for SparkVersion.class

add license

* 1 add spark task UT
2 add spark version param check

* add assert check for sparkTaskTest

* fix AbstractTask's handle method exception (#1490)

* fix AbstractTask's handle method exception

* update ut

* add ZkServer for UT (#1499)

* add ZkServer for UT

* Add FileUtilsTest.java , the unit test for FileUtils (#1493) (#1)

* updates for reference ZkServer

* DAG automatic layout (#1497)

* Password verification and v-for add key

* DAG automatic layout

* Add common utils CollectionUtils.java DateUtils.java unit test (#1496)

* dateutil test

* pom.xml

* Update README_zh_CN.md

* Update README.md

* Add ExcelUtilsTest.java , the unit test for ExcelUtils (#1500)

* fix issue:1477 some tasks would be running all the time when db delayed(#1477) (#1501)

* fix issue:1477 some tasks would be running all the time when db delayed

* fix issue:1477 some tasks would be running all the time when db delayed

* fix issue:1477 some tasks would be running all the time when db delayed

* change npm run build:combined to npm burn build:release (#1504)

09540843

fix issue:1477 some tasks would be running all the time when db delayed(#1477) (#1501) · 21cb38d1

由 bao liang 提交于 12月 18, 2019

* fix issue:1477 some tasks would be running all the time when db delayed

* fix issue:1477 some tasks would be running all the time when db delayed

* fix issue:1477 some tasks would be running all the time when db delayed

21cb38d1

10 12月, 2019 1 次提交

merge dev-db to dev (#1426) · a5277827

由 bao liang 提交于 12月 10, 2019

* [dolphinscheduler-1345] [newfeature] Add DB2 Datasource (#1391)

* Fix the problem that the 'queueId' is not present when creating a tenant based on the default queue. (#1409)

* dolphinscheduler-#1403][bug]improve the check rules (#1408)

1. When check failed, we don’t know whitch parameter has is wrong, Because username、password、email and phone were checks together. I refactored the check method ,Now it will return failed msg by each field.
2. The email check regex support [_|\-|\.]?) in createUser.vue, But it do not support in backend server , I fix it, Now they have the same check regex both in frontend and backend

* jcip-annotations define version information and maven-assembly-plugin add groupId (#1413)

* "v-for" add key (#1419)

* [dolphinscheduler-#1397] [bug]Resources can not be previewed or updated  (#1406)

When create an resource the name will add the suffix, But When rename the resource there is no suffix add to the name, So When update resource name without suffix just like "test.sh" => "test" , Then the bug reproduced.

To fix this bug i add the logic bellow:
When rename, if the name without suffix then add it ,else use the origin name

* simply server module configs (#1424)

* move updateTaskState into try/catch block in case of exception

* fix NPE

* using conf.getInt instead of getString

* for AbstractZKClient, remove the log, for it will print the same log message in createZNodePath.
for AlertDao, correct the spelling.

* duplicate

* refactor getTaskWorkerGroupId

* add friendly log

* update hearbeat thread num = 1

* fix the bug when worker execute task using queue. and remove checking Tenant user anymore in TaskScheduleThread

* 1. move verifyTaskInstanceIsNull after taskInstance
2. keep verifyTenantIsNull/verifyTaskInstanceIsNull clean and readable

* fix the message

* delete before check to avoid KeeperException$NoNodeException

* fix the message

* check processInstance state before delete tenant

* check processInstance state before delete worker group

* refactor

* merge api constants into common constatns

* update the resource perm

* update the dataSource perm

* fix CheckUtils.checkUserParams method

* update AlertGroupService, extends from BaseService, remove duplicate methods

* refactor

* modify method name

* add hasProjectAndPerm method

* using checkProject instead of getResultStatus

* delete checkAuth method, using hasProjectAndPerm instead.

* correct spelling

* add transactional for deleteWorkerGroupById

* add Transactional for deleteProcessInstanceById method

* change sqlSessionTemplate singleton

* change sqlSessionTemplate singleton and reformat code

* fix unsuitable error message

* update shutdownhook methods

* fix worker log bug

* fix api server debug mode bug

* upgrade zk version

* delete this line ,for zkClient.close() will do the whole thing

* fix master server shutdown error

* degrade zk version and add FourLetterWordMain class

* fix PathChildrenCache not close

* add Transactional for createSession method

* add more message for java-doc

* delete App, let spring manage connectionFactory

* add license

* add class Application for test support

* refactor masterServer and workerServer

* add args

* fix the spring transaction not work bug

* remove author

* delete @Bean annotation

* delete master/worker properties

* updates

* rename application.properties to application-dao.properties

* delete this class

* delete master/worker properties and  refactory master/worker

* delete unused imports

* merge

* delete unused config

a5277827

05 12月, 2019 1 次提交
- Q
  Revert "merge from dev-db (#1386)" (#1404) · c25a9b0b
  由 qiaozhanwei 提交于 12月 05, 2019
```
This reverts commit 1baa1f42.
```
  c25a9b0b

DolphinScheduler社区 / DolphinScheduler 与 Fork 源项目一致

DolphinScheduler社区 / DolphinScheduler
与 Fork 源项目一致