1. 30 3月, 2020 3 次提交
    • Q
      test · c2cf8fb7
      qiaozhanwei 提交于
      c2cf8fb7
    • D
      remove zookeeper as task queue relevant config (#2330) · 195dfb50
      dailidong 提交于
      * update logback
      
      * update log
      
      * refactor worker registry (#2107)
      
      * Refactor worker (#2115)
      
      * refactor worker registry
      
      * refactor master server
      
      * Modify workgroupid parameter name (#2105)
      
      * Delete worker group management page
      
      * Modify workgroupid parameter name
      
      * Refactor worker (#2121)
      
      * refactor worker registry
      
      * refactor master server
      
      * refactor MasterSchedulerService
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      
      * not exist in openjdk,just delete
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * fix cpu 100% bug
      
      * simplify master、 worker、alert、dao properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * Optimize code , reduce cost time from 60ms to 0.5ms
      
      * merge code form remote
      
      * remove zookeeper as task queue relevant config
      Co-authored-by: NTboy <guo.jiwei@immomo.com>
      Co-authored-by: Nbreak60 <790061044@qq.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      195dfb50
    • T
      Refactor worker (#2331) · 3bafa97f
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      
      * add NettyExecutorManagerTest
      
      * refactor TaskKillProcessor
      
      * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
      
      * add RoundRobinHostManagerTest, ExecutorDispatcherTest
      
      * refactor task response service
      
      * add TaskResponseServiceTest
      
      * modify close method for MasterSchedulerService
      3bafa97f
  2. 28 3月, 2020 1 次提交
    • Q
      1,add UT in pom 2,refactor TaskUpdateQueue (#2326) · 68cb81fd
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      
      * worker fault tolerance modify
      
      * Constants and .env modify
      
      * master fault tolerant bug modify
      
      * UT add pom.xml
      
      * timing online  modify
      
      * when taskResponse is faster than taskAck to db,task state will error
      add async queue and new a thread reslove this problem
      
      * TaskExecutionContext set host
      
      * 1,TaskManager refactor
      2, api start load server dolphinschedule-daemon.sh modify
      
      * 1,TaskManager refactor
      2, api start load server dolphinschedule-daemon.sh modify
      
      * add UT in pom.xml
      
      * revert dolphinscheduler-daemon.sh
      
      * ZookeeperRegister use common.properties zookeeperRoot path
      
      * api start exclude org.apache.dolphinscheduler.server.*
      
      * ZookeeperRegister use common.properties zookeeperRoot path
      
      * 1,api start load server filter
      2,SHELL task exitStatusCode modify
      
      * java doc error modify
      
      * java doc error modify
      
      * remove todo
      
      * add UT in pom
      
      * 1,add UT in pom
      2,refactor TaskUpdateQueue
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      68cb81fd
  3. 27 3月, 2020 4 次提交
    • T
      add TaskResponseServiceTest (#2325) · 23ba0351
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      
      * add NettyExecutorManagerTest
      
      * refactor TaskKillProcessor
      
      * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
      
      * add RoundRobinHostManagerTest, ExecutorDispatcherTest
      
      * refactor task response service
      
      * add TaskResponseServiceTest
      23ba0351
    • L
      af8efe51
    • D
      [Refactor worker] Optimize code , reduce cost time from 60ms to 0.5ms (#2318) · 7e9f45e5
      dailidong 提交于
      * update logback
      
      * update log
      
      * refactor worker registry (#2107)
      
      * Refactor worker (#2115)
      
      * refactor worker registry
      
      * refactor master server
      
      * Modify workgroupid parameter name (#2105)
      
      * Delete worker group management page
      
      * Modify workgroupid parameter name
      
      * Refactor worker (#2121)
      
      * refactor worker registry
      
      * refactor master server
      
      * refactor MasterSchedulerService
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      
      * not exist in openjdk,just delete
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * fix cpu 100% bug
      
      * simplify master、 worker、alert、dao properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * Optimize code , reduce cost time from 60ms to 0.5ms
      
      * merge code form remote
      Co-authored-by: NTboy <guo.jiwei@immomo.com>
      Co-authored-by: Nbreak60 <790061044@qq.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      7e9f45e5
    • T
      Refactor worker (#2319) · 5730bfe2
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      
      * add NettyExecutorManagerTest
      
      * refactor TaskKillProcessor
      
      * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
      
      * add RoundRobinHostManagerTest, ExecutorDispatcherTest
      
      * refactor task response service
      5730bfe2
  4. 26 3月, 2020 1 次提交
    • T
      Refactor worker (#2312) · e720ca54
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      
      * add NettyExecutorManagerTest
      
      * refactor TaskKillProcessor
      
      * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
      
      * add RoundRobinHostManagerTest, ExecutorDispatcherTest
      e720ca54
  5. 25 3月, 2020 2 次提交
    • Q
      1,ZookeeperRegister use common.properties zookeeperRoot path 2,api start... · 03e29b93
      qiaozhanwei 提交于
      1,ZookeeperRegister use common.properties zookeeperRoot path 2,api start exclude org.apache.dolphinscheduler.server.* (#2307)
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      
      * worker fault tolerance modify
      
      * Constants and .env modify
      
      * master fault tolerant bug modify
      
      * UT add pom.xml
      
      * timing online  modify
      
      * when taskResponse is faster than taskAck to db,task state will error
      add async queue and new a thread reslove this problem
      
      * TaskExecutionContext set host
      
      * 1,TaskManager refactor
      2, api start load server dolphinschedule-daemon.sh modify
      
      * 1,TaskManager refactor
      2, api start load server dolphinschedule-daemon.sh modify
      
      * add UT in pom.xml
      
      * revert dolphinscheduler-daemon.sh
      
      * ZookeeperRegister use common.properties zookeeperRoot path
      
      * api start exclude org.apache.dolphinscheduler.server.*
      
      * ZookeeperRegister use common.properties zookeeperRoot path
      
      * 1,api start load server filter
      2,SHELL task exitStatusCode modify
      
      * java doc error modify
      
      * java doc error modify
      
      * remove todo
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      03e29b93
    • D
      [refactor-worker] simplify master、 worker、alert、dao properties (#2304) · ace907e7
      dailidong 提交于
      * update logback
      
      * update log
      
      * refactor worker registry (#2107)
      
      * Refactor worker (#2115)
      
      * refactor worker registry
      
      * refactor master server
      
      * Modify workgroupid parameter name (#2105)
      
      * Delete worker group management page
      
      * Modify workgroupid parameter name
      
      * Refactor worker (#2121)
      
      * refactor worker registry
      
      * refactor master server
      
      * refactor MasterSchedulerService
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      
      * not exist in openjdk,just delete
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * fix cpu 100% bug
      
      * simplify master、 worker、alert、dao properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      Co-authored-by: NTboy <guo.jiwei@immomo.com>
      Co-authored-by: Nbreak60 <790061044@qq.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      ace907e7
  6. 24 3月, 2020 5 次提交
    • Q
      TaskManager refactor (#2302) · f112415b
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      
      * worker fault tolerance modify
      
      * Constants and .env modify
      
      * master fault tolerant bug modify
      
      * UT add pom.xml
      
      * timing online  modify
      
      * when taskResponse is faster than taskAck to db,task state will error
      add async queue and new a thread reslove this problem
      
      * TaskExecutionContext set host
      
      * 1,TaskManager refactor
      2, api start load server dolphinschedule-daemon.sh modify
      
      * 1,TaskManager refactor
      2, api start load server dolphinschedule-daemon.sh modify
      
      * add UT in pom.xml
      
      * revert dolphinscheduler-daemon.sh
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      f112415b
    • Q
      add async queue and new a thread reslove taskResponse is faster than taskAck to db (#2297) · 62f7d21b
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      
      * worker fault tolerance modify
      
      * Constants and .env modify
      
      * master fault tolerant bug modify
      
      * UT add pom.xml
      
      * timing online  modify
      
      * when taskResponse is faster than taskAck to db,task state will error
      add async queue and new a thread reslove this problem
      
      * TaskExecutionContext set host
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      62f7d21b
    • D
      [Refactor worker] fix cpu pressure may be too high bug (#2290) · 7da35238
      dailidong 提交于
      * update logback
      
      * update log
      
      * refactor worker registry (#2107)
      
      * Refactor worker (#2115)
      
      * refactor worker registry
      
      * refactor master server
      
      * Modify workgroupid parameter name (#2105)
      
      * Delete worker group management page
      
      * Modify workgroupid parameter name
      
      * Refactor worker (#2121)
      
      * refactor worker registry
      
      * refactor master server
      
      * refactor MasterSchedulerService
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      
      * not exist in openjdk,just delete
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      
      * fix cpu 100% bug
      Co-authored-by: NTboy <guo.jiwei@immomo.com>
      Co-authored-by: Nbreak60 <790061044@qq.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      7da35238
    • Q
      timing online modify (#2296) · 11e3bff6
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      
      * worker fault tolerance modify
      
      * Constants and .env modify
      
      * master fault tolerant bug modify
      
      * UT add pom.xml
      
      * timing online  modify
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      11e3bff6
    • T
      add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest (#2291) · 1e5a1140
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      
      * add NettyExecutorManagerTest
      
      * refactor TaskKillProcessor
      
      * add RandomSelectorTest, RoundRobinSelectorTest, TaskCallbackServiceTest
      1e5a1140
  7. 23 3月, 2020 3 次提交
    • T
      Refactor worker (#2279) · 92fd6479
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      
      * add NettyExecutorManagerTest
      
      * refactor TaskKillProcessor
      92fd6479
    • Q
      UT Coverage rate test (#2276) · 052f9d10
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      
      * worker fault tolerance modify
      
      * Constants and .env modify
      
      * master fault tolerant bug modify
      
      * UT add pom.xml
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      052f9d10
    • K
      worker ci ut yml (#2278) · ede3776c
      khadgarmage 提交于
      ede3776c
  8. 22 3月, 2020 1 次提交
    • T
      Refactor worker (#2266) · d678447e
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      
      * add NettyExecutorManagerTest
      d678447e
  9. 20 3月, 2020 1 次提交
    • T
      Refactor worker (#2248) · 1fcecc57
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      
      * add ZookeeperNodeManagerTest
      1fcecc57
  10. 19 3月, 2020 2 次提交
    • T
      Refactor worker (#2241) · 0dab5cb0
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      
      * refactor TaskCallbackService
      0dab5cb0
    • D
      add master and worker properties and remove data quality config (#2211) · ce2853ad
      dailidong 提交于
      * update logback
      
      * update log
      
      * refactor worker registry (#2107)
      
      * Refactor worker (#2115)
      
      * refactor worker registry
      
      * refactor master server
      
      * Modify workgroupid parameter name (#2105)
      
      * Delete worker group management page
      
      * Modify workgroupid parameter name
      
      * Refactor worker (#2121)
      
      * refactor worker registry
      
      * refactor master server
      
      * refactor MasterSchedulerService
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126)
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      
      * not exist in openjdk,just delete
      
      * add master and worker properties
      
      * add master and worker properties
      
      * add master and worker properties
      Co-authored-by: NTboy <guo.jiwei@immomo.com>
      Co-authored-by: Nbreak60 <790061044@qq.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@outlook.com>
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      ce2853ad
  11. 18 3月, 2020 1 次提交
    • B
      modify parameters (#2203) · 99c28c74
      break60 提交于
      * Delete worker group management page
      
      * Modify workgroupid parameter name
      
      * Worker group parameter modification
      
      * modify parameters
      99c28c74
  12. 17 3月, 2020 3 次提交
    • Q
      worker fault tolerance modify (#2212) · 055bb28d
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      
      * worker fault tolerance modify
      
      * Constants and .env modify
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      055bb28d
    • Q
      1,master and worker register ip use OSUtils.getHost() 2,ProcessInstance host... · cc878505
      qiaozhanwei 提交于
      1,master and worker register ip use OSUtils.getHost() 2,ProcessInstance host set ip:port format (#2209)
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      
      * master and worker register ip  use OSUtils.getHost()
      
      * ProcessInstance host set ip:port format
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      cc878505
    • Q
      1,get workergroup from zk modify 2,SpringConnectionFactory repeat load modify (#2201) · e31b48fd
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      
      * TaskExecutionContextBuilder set TaskInstance workgroup
      
      * WorkerGroupService queryAllGroup modify
      query available work group
      
      * 1,get workergroup from zk modify
      2,SpringConnectionFactory repeat load modify
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      e31b48fd
  13. 16 3月, 2020 2 次提交
    • T
      Refactor worker (#2189) · 11dde203
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      
      * move mybatis config from application.properties to SpringConnectionFactory
      
      * move mybatis-plus config from application.properties to SpringConnectionFactory
      11dde203
    • B
      Worker group parameter modification (#2188) · aa650b44
      break60 提交于
      * Delete worker group management page
      
      * Modify workgroupid parameter name
      
      * Worker group parameter modification
      aa650b44
  14. 15 3月, 2020 1 次提交
    • T
      Refactor worker (#2186) · 75efbb54
      Tboy 提交于
      * let quartz use the same datasource
      
      * move master/worker config from dao.properties to each config
      add master/worker registry test
      75efbb54
  15. 13 3月, 2020 2 次提交
    • Q
      DataxTask process test modify (#2162) · 71b11e2c
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      
      * DataxTask process run debug
      
      * DataxTask process run debug
      
      * add protobuf dependency,MR、Spark task etc need this
      
      * TaskUpdateQueueConsumer modify
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      71b11e2c
    • Q
      ProcedureTask process test modify (#2159) · 607ec3d1
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      
      * ProcedureTask optimization
      
      * MasterSchedulerService modify
      
      * TaskUpdateQueueConsumer modify
      
      * test
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      607ec3d1
  16. 12 3月, 2020 1 次提交
    • T
      Fix javadoc and DataxTaskTest UT error (#2147) · 16eeaf50
      t1mon 提交于
      * Fix javadoc and DataxTaskTest UT error
      
      * Fix UT error
      * LoggerServiceTest UT error
      * WorkerGroupServiceTest UT error
      Modify Host.of()
      * Add address NPE check
      
      * Trigger actions rerun
      
      * Try to solve maven VM crash
      
      * Try to solve maven VM crash 1
      
      * Try to solve maven VM crash 2
      
      * Try to solve maven VM crash 3
      16eeaf50
  17. 10 3月, 2020 4 次提交
    • T
      let quartz use the same datasource (#2137) · 7f72df4c
      Tboy 提交于
      7f72df4c
    • Q
      1,encapsulate the parameters required by sqltask 2,SQLTask optimization (#2135) · fd2c2eeb
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * Encapsulate the parameters required by sqltask
      
      * 1,Encapsulate the parameters required by sqltask
      2,SQLTask optimization
      
      * AbstractTask modify
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      fd2c2eeb
    • L
      refactor worker group (#2132) · bbe2cd48
      lgcareer 提交于
      * replace worker group id with worker group
      
      * add worker group field in ddl
      
      * remove worker group id
      bbe2cd48
    • T
      Merge branch 'refactor-worker' into refactor-worker · 621ebf60
      Tboy 提交于
      621ebf60
  18. 09 3月, 2020 3 次提交
    • D
      split logback config into seperate child module logback (#2120) · f5519132
      dailidong 提交于
      * update logback
      
      * update log
      f5519132
    • Q
      cancelTaskInstance set TaskExecutionContext host,logPath,executePath (#2126) · 20af8ff9
      qiaozhanwei 提交于
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * 1, master persistent task
      2. extract  master and worker communication model
      
      * add license
      
      * modify javadoc error
      
      * TaskExecutionContext create modify
      
      * buildAckCommand taskInstanceId not set modify
      
      * java doc error modify
      
      * add comment
      
      * ExecutorManager interface add generic type
      
      * add TaskInstanceCacheManager receive Worker report result
      
      * TaskInstance setExecutePath
      
      * add TaskInstanceCacheManager to receive Worker Task result report
      
      * TaskInstanceCacheManager add remove method
      
      * add license
      
      * add dispatcht task method
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * AbstractCommandExecutor remove db access
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * taskInstanceCache is null ,need load from db
      
      * 1,worker TaskPros use TaskExecutionContext replase
      2,Master kill Task , KillTaskProcessor modify
      
      * worker remove db
      
      * ShellTask modify
      
      * master persistence processId and appIds
      
      * master persistence processId and appIds
      
      * master add kill task logic
      
      * master add kill task logic
      
      * master add kill task logic
      
      * javadoc error modify
      
      * remove chinese log
      
      * executeDirectly method add Override
      
      * remote module modify
      
      * TaskKillResponseProcessor command type modify
      
      * create buildKillCommand
      
      * host add host:port format
      
      * host add host:port format
      
      * TaskAckProcessor modify
      
      * TaskAckProcessor modify
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * task prioriry refator
      
      * remove ITaskQueue
      
      * TaskPriority refactor
      
      * remove logs
      
      * WorkerServer refactor
      
      * MasterSchedulerService modify
      
      * WorkerConfig listen port modify
      
      * modify master and worker listen port
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      
      * cancelTaskInstance set TaskExecutionContext host,logPath,executePath
      Co-authored-by: Nqiaozhanwei <qiaozhanwei@analysys.com.cn>
      20af8ff9
    • T
      Refactor worker (#2121) · fd6f13ff
      Tboy 提交于
      * refactor worker registry
      
      * refactor master server
      
      * refactor MasterSchedulerService
      fd6f13ff