*Motivation*

Currently KeyValue schema doesn't support using AUTO_CONSUME.

This PR is to add this support.

This PR is based on #4836

*Changes*

- refactor a bit on Schema interface to support fetching schema info for both AutoConsumeSchema and KeyValueSchema before subscribing
- add AUTO_CONSUME support to KeyValueSchema
- add tests

*Motivation*

Currently fetching schema information is done synchronously.
It is called in netty callback threads and will potentially block
async operations.

*Modifications*

Make most of the operations asynchronously in SchemaInfoProvider.

* [Pulsar-Client] Add Producer Numeric Properties Validation

* Aligned deprecated and new Producer API validations

* Deprecated and new Producer API validations are being aligned

* batchingMaxMessages C++ API is being aligned with Java API

* batchingMaxMessages Java API Validation is being aligned with C++ API

* Review comments are addressed

* Fix broken UTs

### Motivation

After the changes in #3118, there has a been a sharp increase of memory utilization for the UnackedMessageTracker due to the time buckets being created. 

This is especially true when the acktimeout is set to a larger value (eg: 1h) where 3600 time-buckets are being created. This lead to use 20MB per partition even when no message is tracked.

Allowing to configure the tick time so that application can tune it based on needs.

Additionally, fixed the logic that keeps creating hash maps and throwing them away at each tick time iteration, since that creates a lot of garbage and doesn't take care of the fact that the hash maps are expanding based on the required capacity (so next time they are already of the "right" size). 

On a final note: the current default of 1sec seems very wasteful. Something like 10s should be more appropriate as default.

Upgrading to Mockito 2.28 and PowerMock 2.0. This a pre-step to be able to run CI with Java 11 / 12

* Allows consumer retrieve the sequence id that the producer set.

* fix comments.

[Client] Consumer for a single partition of a PartitionedTopic should be set with correct partitionIndex (#4591)

This PR fixes #4586 by getting `partitionIndex` from topic name.

*Motivation*

Currently producers uses `System.currentTimeMillis()` as publish timestamp by default.
However at some use cases, producers would like to a different way for generating publish timestamp.
E.g. in a database use case, a producer might be use HLC (Hybrid Logic Clock) as publish timestamp;
in integration tests, it might require the producer to use a deterministic way to generate publish timestamp.

*Changes*

This PR introduces a `clock` in building the client. This allows applications to override the system clock
with its own implementation.

*Verify the change*

Add unit test to test customized clock in both batch and non-batch cases.

*Motivation*

- The code for encoding and decoding key/value schema is spreading over multiple places.
- Make code changes to prepare supporting key/value schema in AUTO consumers
- Make schema tools display key/value schema in a pretty format

*Modifications*

- Move the common logic of encoding and decoding key/value schema to a common class KeyValueSchemaInfo
- Expose the common class in DefaultImplementation so that it can be available for public usage
- Fix the display problem on displaying key/value schema

*Verify this change*

- Add bunch of the unit tests for key/value schemas

### Motivation

Introduce batch message container framework to support multiple ways to do message batch. 
Currently, pulsar support a most basic batch message container, use the batch message container framework can quickly implement other types batch message container, even users can customize their own batch message container.

Add a new batch message container named BatchMessageKeyBasedContainer to support batching message in key_shared subscription mode.

* Do not strip ExecutionException from the stack trace

### Motivation

Make sure we record charset in the properties. So that we know which charset to be used for encoding and decoding strings.

### Modifications

Add charset to the properties 

* Feature - reset cursor on Reader to current position

*Motivation*

There are some cases in which is it useful to be able to include current
position of message when reset of cursor was made.

This was reported by a `vvy` on slack, no issue has been created to track this.

*Modifications*

  - Add startMessageIdInclusive() to support include current position of
    reset on ReaderBuilder.
  - Add resetIncludeHead field for Reader and Consumer Configuration Data
  - Fix position of cursor for non durable consumer.
  - Improve discard if statement for batch enable mode.
  - Add discard if statement for batch disable mode.
  - Improve test case for latest Reader seek.
  - Add test case to assert the start of specific message id at the expected
    position with data provider scenarios:
      A. Batch enable and start inclusive enable.
      B. Batch enable and start inclusive disable.
      C. Batch disable and start inclusive enable.
      D. Batch disable and start inclusive disable.

* Fix ProducerInterceptors NPE issue.

* Add UT test for ProducerInterceptors while interceptor cause exceptions.

* expose replicated info in message

* add negative test case

Fix issue for incorrect log parameters
* When calling Logger's method, set the appropriate parameters
* Fix #4408.

* o.a.p.c.api -> o.a.p.c.protocol

Classes in pulsar-common module were moved to avoid jpms split-package
conflict between the pulsar-client-api and pulsar-common modules.

* o.a.p.c.schema -> o.a.p.c.protocol.schema

Classes in pulsar-common module were moved to avoid jpms split-package
conflict between the pulsar-client-api and pulsar-common modules.

* Flink client to accept all pulsar client conf

In this patch, we provide handles for flink connecotr to accept ClientConfigurationData, ProducerConfigurationData, ConsumerConfigurationData so flink client can:
1. accept all params of client, producer and consumer
2. Keep pace with pulsar-client

* Flink client to accept all pulsar client conf

Added test cases

* Removing commented code

* flink: construct auth when building pulsarsource

* fixed failing tests

* removed unused import

* Added builder defaults for lombok builder
Set Auth from class and params (if set) in PulsarClientImpl.java

* Remove @BUilder.default from attributes where no defaults exist

* Added tests for ClientConfiguration Data builders

* cosmetic changes in code
Co-Authored-By: NSijie Guo <guosijie@gmail.com>

* fixing typo

* Removed test, not true anymore

* Removed lombok builders

* fixed the failing tests

* Because the authentication field is transient, it is not serialized. On desirialization then its null and desirialization crashes with NPE

* Deduce tls from protocol in service url

* fixed failing test

* fixed NPE
Co-Authored-By: NMatteo Merli <mmerli@apache.org>

Fix log format for module pulsar-client. 

Part of #4304

If we send a message to the dead letter topic, we shouldn't also
request that the same message be redelivered.

The process to send to DLQ does do an acknowledge(), but acknowledge
is async and not guaranteed, so it can race with the redelivery
request.

* Initialize TLS context only when attempting connection

* Fixed eviction time check

* Delayed message delivery implementation

* Fixed compilation

* Allow to configure the delayed tracker implementation

* Use int64 for timestamp

* Address comments

* More tests for TripleLongPriorityQueue

* Removing useless sync block that causes deadlock with consumer close

* Fixed merge conflict

* Avoid new list when passing entries to consumer

* Fixed test. Since entries are evicted from cache, they might be resent in diff order

* Fixed context message builder

* Fixed triggering writePromise when last entry was nullified

* Moved entries filtering from consumer to dispatcher

* Added Javadocs

* Reduced synchronized scope to minimum

* Configure static PulsarByteBufAllocator to handle OOM errors

* Always specify `pulsar.allocator.exit_on_oom` when starting pulsar services

* Reverted metrics back

* Fixed compression tests

* Explicitely set the underlying allocator to netty default

* Fixed shading

* resolve a nullPointer ex

*  renaming sinks and sources api to be consistent with the rest of Pulsar (#4363)

* Rename sources and sinks CLI to be consistent with rest of Pulsar

* renaming sinks and sources api to be consistent with the rest of Pulsar

* use new interfaces in cmd

* Fix install pip (#4387)

### Motivation
There is a problem with the current installation of pip when running the integration tests.
Therefore, install pip using the officially recommended method.

### Modifications

Modify the pip installation method, refer to https://pip.pypa.io/en/stable/installing/

Fix bug of Key_Shared ordering and add check for order by key, key distribution, message redelivery in Key_Shared subscription mode.

Don't leak resources when a message is being discarded.

*Modifications*

  - Fix missing release() and recycle() for discarded message on
    receiveIndividualMessagesFromBatch method.
  - Fix argument missing of debug logging {}-placeholder.
  - Fix unnecessary variable reference `payload` on messageReceived().

[schema] AutoConsume should use the schema associated with messages as both writer and reader schema (#4325)

* [schema] AutoConsume should use the schema associated with messages for both writer and reader schema

*Motivation*

AutoConsume should use the schema associated with the messages for decoding the schemas.

*Modifications*

- provide a flag enable or disable using the provided schema as the reader schema
- for AUTO_CONSUME schema, disable usnig the provided schema as the reader schema. so it can use the right
  schema version for decoding messages into right generic records
- provide a few util methods for displaying schema data

* Handle 64 bytes schema version

* Addressed review comments

Fixes #4101 

### Motivation

fix issue

### Modifications

fix keepAliveInterval method java doc