*Motivation*

Currently we don't specify charset. It will be causing troubles when client
and broker are using different system charsets.

*Modifications*

Use UTF_8 for the schema information for AVRO/PROTOBUF/JSON

*Motivation*

In order to introduce `GenericRecordBuilder`, we need to know the fields in a `GenericSchema`.
Otherwise, there is no way for us to build a GenericRecord.

*Modifications*

This proposal refactors current generic schema by introducing a `GenericSchema`. This generic schema
provides interfaces to retrieve the fields of a `GenericRecordSchema`.

*Additionally*

This proposal adding the primitive schemas into `Schema` class. So people can program primitive schemas
using Schema interface rather than specific implementations.

*Motivation*

Currently we are supporting POJO based schema in java clients.
POJO schema is only useful when the POJO is predefined. However
in applications like a CDC pipeline, POJO is no predefined, there
is no other way to define a schema.

Since we are using avro schema for schema management, this PR
is proposing a simple schema builder wrapper on avro schema builder.

*Modifications*

Introduce schema builder to build a record schema.

*NOTES*

Currently we only support primitives in defining fields in a record schema in this PR.
We will add nested types in future PRs.

*Motivation*

Currently AUTO_CONSUME only supports decoding records from latest schema.
All the schema versions are lost. It makes AUTO_CONSUME less useful in some use cases,
such as CDC. Because there is no way for the applications to know which version of schema
that a message is using.

In order to support multi-version schema, we need to propagate schema version from
message header through schema#decode method to the decoded record.

*Modifications*

- Introduce a new decode method `decode(byte[] data, byte[] schemaVersion)`. This allows the implementation
  to leverage the schema version.
- Introduce a method `supportSchemaVersioning` to tell which decode methods to use. Because most of the schema
  implementations such as primitive schemas and POJO based schema doesn't make any sense to use schema version.
- Introduce a SchemaProvider which returns a specific schema instance for a given schema version
- Implement a MultiVersionGenericRecordSchema which decode the messages based on schema version. All the records
  decoded by this schema will have schema version and its corresponding schema definitions.

*NOTES

This implementation only introduce the mechanism. But it doesn't wire the multi-version schema
with auto_consume schema. There will be a subsequent pull request on implementing a schema provider
that fetches and caches schemas from brokers.

*Motivation*

The script doesn't update the parent version for protobuf-shaded module

*Modifications*

use `versions:update-parent` to update parent version

*Motivation*

Topics count is a namespace level metric. Currently if `exposeTopicLevelMetricsInPrometheus=true`
we don't expose `pulsar_topics_count`. However `pulsar_topics_count` is a very useful metric for
monitoring the healthy of the cluster.

*Modifications*

if `exposeTopicLevelMetricsInPrometheus=true`, we expose namespace level metrics that are not
exposed at topic level. e.g. `pulsar_topics_count`.

*Motivation*

Currently integration tests are broken due to #3827

`PULSAR_` is used almost everywhere in our scripts. If we are using
`PULSAR_` for appending new keys, then we are potentially appending
a wrong key into the config file.

Currently this happens on presto. We are appending `PULSAR_ROOT_LOGGER`
into presto's config file and cause presto fail to load the config.

*Modifications*

Use `PULSAR_PREFIX_` instead of `PULSAR_`

* Add config variables if absent

* Took feedback into account

Same fix as #3746 but for cpp client.

  - Filter consumers for the same topic name.
  - Add test to verify that same subscription names with different topics are
    allowed to be different consumers subscription instead of reused.

* fix message_id_serialize to empty slice

* test for serialize and deserialize

* In Java allow messages that compress to <5mb to be sent with batching enabled

* Test is not anymore accurate since behavior changed and we're already testing in ProducerConsumerTest

*Motivation*

Fixes #3803

Hardcoding is a very bad practice. It means we have no way to alter system behavior
when production issues occur.

*Modifications*

introduce a few read batch related settings to make them configurable

* Feature - implement reference count for ConsumerImpl

Add reference count for ConsumerImpl in order to track reused instances of a
consumer instance returned by `subscribe()` method call.
Having the reference of subscribed consumer instances offers the ability to not
close a consumer until the last corresponding `close()` is being called.

Modifications:

  - Add field on ConsumerBase to track references of consumer instances
    subscribed by the user.
  - Add checks on ConsumerImpl to know whether close() action should be
    performed regarding of reference count being zero value.
  - Increment reference count when a previous built consumer instance is being
    used by caller.

Future considerations:

When optimization #3312 is going to be made for other consumers implementation
aside from ConsumerImpl it should add refCount checks on close() method call.

* Add tests for reference count on ConsumerImpl

  - Add test to verify ConsumerImpl reference count on close() method.
  - Fix test from dup consumers feature with refcount.

* [pulsar-webscoket] avoid creating temp list of metrics on every-metrics generator

* fix non-sync getMetrics

*Motivation*

Fixes #3686

Fixes #3734

*Motivation*

Exception occurred when using `BytesSchema.of()`

```
Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.apache.pulsar.examples.simple.ProducerExample.main(ProducerExample.java:32)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.pulsar.client.internal.ReflectionUtils.catchExceptions(ReflectionUtils.java:36)
	at org.apache.pulsar.client.internal.DefaultImplementation.newKeyValueSchema(DefaultImplementation.java:158)
	at org.apache.pulsar.client.api.Schema.<clinit>(Schema.java:123)
	... 1 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.pulsar.client.internal.DefaultImplementation.lambda$newKeyValueSchema$16(DefaultImplementation.java:160)
	at org.apache.pulsar.client.internal.ReflectionUtils.catchExceptions(ReflectionUtils.java:34)
	... 3 more
Caused by: java.lang.NullPointerException
	at org.apache.pulsar.client.impl.schema.KeyValueSchema.<init>(KeyValueSchema.java:68)
	... 9 more
```

The problem introduced because the weird class loading and reflection sequence.

When accessing `BytesSchema`, `BytesSchema` will try to initialize `Schema`. When initializing Schema, it will attempts
to initialize `KV_BYTES` using reflection, and initializing KV_BYTES requires `BytesSchema`. Hence it causes KV_BYTES not being
initialized correctly.

The change is to avoid this recursive class loading.

### Motivation

Fixes #3779 

The behavior of acknowledgment has always been to throw exception if consumer is not connected.

In reality it doesn't make much sense to bubble back exception to user in this case because: 
 * There's no ack on the ack anyway
 * The only reason for ack to fail is if consumer is disconnected
 * If consumer is disconnected, messages are going to be replayed in any case
 * Therefore, there's no action to do when acking while disconnected

This is already the case in the Java and Go clients (additionally Java client performs client side grouping of acks based on time). 

Changing the Python client behavior to be the same.

* fix: function config cleanupSubscription update bug

* add and fix unit tests

* propagate default resource values in Pulsar Functions

* fix unit tests

* Fixed Reader.HasNext() in Go client

* Fixed formatting

* Removed commented code

Fixes #3743 issue.

Return previous instance of a consumer in the subscription processed should only
be considered with the scope of the same topic.

Modifications:

  - Remove optimization of duplicated consumers for multi topics subscribe and
    pattern topics subscribe, this should be handled with a different approach.
  - Filter consumers for the same topic name.
  - Filter consumers which are connected to broker, this is not necessary to fix
    this issue but is a good thing to do.
  - Add test that verifies that same subscription will allow different consumers
    instance for different topics.

* Expose Secrets in Source/SinkContext

* Fix unittest

* Allow users to update everything in inputspecs except for isregexpattern

* Added more tests and fixed a bug caught by them

### Motivation

Use at least 8 threads to avoid having Jetty go into threads starving and
having the possibility of getting into a deadlock where a Jetty thread is
waiting for another HTTP call to complete in same thread.

This solve the issues of requests timing out when the broker is making REST
calls to itself. Such examples are when running in standalone mode and 
creating a function.

* Retry creation of assignment topic a few times before giving up

* Use Action based retry mechanism

* Fix build

* Catch interrupted exception

* Fix unittest

* Added header

* fix function termination cleanup

* cleaning up

* Simplified the workflow of functionruntime manager

* Fix unittest

* Took feedback into account

* added missing imports

* On publish failures, log error and count them as sys exceptions

* Took feedback

* When the loadmanager leader is not available, fall through regular least loaded selection

* Handle exceptions coming from mock zk in tests