Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

* Remove redundant session startup
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Register datanode after start success
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* fix meta snap shot
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel
Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: Ndragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node
Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream
Signed-off-by: Ndragondriver <jiquan.long@zilliz.com>

* fix proxynode
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* add check for time tick
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* fix check
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream
Signed-off-by: Ndragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan
Signed-off-by: Ndragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream
Signed-off-by: Ndragondriver <jiquan.long@zilliz.com>

* fix bugs
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* fix describe collection
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics
Signed-off-by: Ndragondriver <jiquan.long@zilliz.com>

* fix data node
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* add log for data node
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Add more log in datanode
Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Rename policy
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection
Signed-off-by: Ndragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* fix master ut
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* add log for debug
Signed-off-by: Nyefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp
Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows
Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>
Co-authored-by: Nyefu.chen <yefu.chen@zilliz.com>
Co-authored-by: Nyangxuan <xuan.yang@zilliz.com>
Co-authored-by: Ndragondriver <jiquan.long@zilliz.com>
Co-authored-by: NCongqi Xia <congqi.xia@zilliz.com>

When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

Signed-off-by: Nyangxuan <xuan.yang@zilliz.com>

We save channels registered in datanode in etcd and restore the cluster info after
restarting.
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Pick up grpc calls to grpc_handler.go and refactor code format
Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Dataservice use lastest grpc design and saves ddl/dml pos pair

* Update SaveBinlogPaths grpc definition
* Saves segment ddl & dml flush pos pair(start, end)

See also #5220 
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

To ensure segments being flushed correctly, we need to load 
the segment allocation status from meta after starting the service.

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

* Add SaveBinlogPaths grpc in dataservice

* fix merged compile error
Signed-off-by: NCongqi Xia <congqi.xia@zilliz.com>

* Rename module
Signed-off-by: NXiangyu Wang <xiangyu.wang@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: NXuanYang-cn <xuan.yang@zilliz.com>

Signed-off-by: Ngodchen <qingxiang.chen@zilliz.com>

Signed-off-by: NThreadDao <yufen.zong@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: NXuanYang-cn <xuan.yang@zilliz.com>

Signed-off-by: Nzhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: Nzhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: Nneza2017 <yefu.chen@zilliz.com>

Signed-off-by: Nneza2017 <yefu.chen@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: NXuanYang-cn <xuan.yang@zilliz.com>

Signed-off-by: NXuanYang-cn <xuan.yang@zilliz.com>

Signed-off-by: Nneza2017 <yefu.chen@zilliz.com>

Signed-off-by: Nsunby <bingyi.sun@zilliz.com>

Signed-off-by: NXuanYang-cn <xuan.yang@zilliz.com>

Signed-off-by: Nzhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: Ncai.zhang <cai.zhang@zilliz.com>