This closes #7436

This closes #7436

This closes #7188.

Logs from failures show that even though the request never finishes,
kafka topics are actually deleted. Therefore this PR adds timeout on
delete request. If the request times out we check if the topic actually
exists. If not then we continue as if the delete request succeeded.

During the release of Flink 1.7, the value of serialVersionUID was uptick to 2L (was 1L before)
And although the AvroSerializer (along with it's snapshot class) were migrated to the new serialization
abstraction (hence free from Java serialization), there were composite serializers that were not migrated
and were serialized with Java serialization.
This commit manually Java-Deserializes the AvroSerializer to support backwards compatability.

This closes #7580.

The lifecycle methods should not have a RPC timeout. Instead the implementations
now use the RpcUtils#INF_TIMEOUT as the timeout.

For a better separation of concerns in the JobManagerRunner, this commit introduces
a JobMasterService which only exposes the JobMaster's lifecycle methods to the
JobManagerRunner. This allows for an easier substitution when testing the JobManagerRunner.

This closes #7563.

This closes #7544.

We add this with the hope of finding clues about FLINK-9920.

This closes #7486.

Test non concurrent access of StateSnapshotTransformer

Refactor out testNonConcurrentSnapshotTransformerAccess to separte StateSnapshotTransformerTest

use element serializer from new meta info, duplicate it in rocksdb transformer factory, test concurrent access for element serializer

This closes #7320.

Port YARNSessionCapacitySchedulerITCase#testTaskManagerFailure to flip6 codebase:
* Remove assertions that rely on log messages only
* Move part where TMs are killed to YARNHighAvailabilityITCase
* Rename test to a proper name that describes what it does
* Add Javadoc explaning what this test does

[FLINK-11390][tests] Move comment to right position

[FLINK-11390][tests] Reuse YarnClient from super class

Move waitUntilCondition to YarnTestBase

[FLINK-11390][tests] Extract method parse hostname

Extract method getOnlyApplicationReport

Extract method submitJob

Extract method getNumberOfSlotsPerTaskManager

Extract method getFlinkConfigFromRestApi

Delete useless comment

Rename: runner -> yarnSessionClusterRunner

Delete useless sleep & refactor

Reorder methods and add static keyword where possible

This closes #7546.

Summary:
This patch add the ability about uploading state files using multithread when snapshoting.

Differential Revision: https://aone.alibaba-inc.com/code/D818724

This closes #7351.

- Moved JobManagerHARecoveryTest#testJobRecoveryWhenLosingLeadership to
DispatcherHATest#testJobRecoveryWhenChangingLeadership

- Moved JobManagerHARecoveryTest#testFailingJobRecovery to
DispatcherHATest#testFailingRecoveryIsAFatalError

This closes #7539.

The builder creates a TestingHighAvailabilityServices with the standalone
implementation of the individual services.

This commit slightly refactors and cleans up the handling of over sized messages.

This closes #6876.

Removed testClientNonDetachedListeningBehaviour because the test was only relevant for
the JobClientActor. Moved testJobPersistencyWhenJobManagerShutdown to
DispatcherHATest#testPersistedJobGraphWhenDispatcherIsShutDown.

This closes #7526.

- Moved JobManagerStartupTest#testStartupWithPortInUse to
BootstrapToolsTest#testActorSystemInstantiationFailureWhenPortOccupied

- Moved JobManagerStartupTest#testJobManagerStartupFails to
BlobServerTest#testFailureIfStorageDirectoryCannotBeCreated

This closes #7541.

[FLINK-11282][network] Merge StreamRecordWriter into RecordWriter

This closes #7569

Port YARNSessionCapacitySchedulerITCase#testClientStartup to new flip6 codebase.
Rename test to testStartYarnSessionClusterInQaTeamQueue.
Remove -n command line parameter & argument.

The JobManagerProcessReapingTest is no longer relevant since we don't start a
ProcessReaper as it used to be the case in the legacy mode.

This commit renames JobManagerCleanupITCase into BlobsCleanupITCase and uses
the MiniCluster instead of the TestingCluster to execute the tests.

This closes #7524.

This closes #7553.