*Motivation*

Currently KeyValue schema doesn't support using AUTO_CONSUME.

This PR is to add this support.

This PR is based on #4836

*Changes*

- refactor a bit on Schema interface to support fetching schema info for both AutoConsumeSchema and KeyValueSchema before subscribing
- add AUTO_CONSUME support to KeyValueSchema
- add tests

### Motivation

The current implementation of KeyValue schema stores key and value together as part of message payload. Ideally the key should be stored as part of message key.

It can be done by introducing a property in KeyValue schema to indicate whether store key in payload or as message key.

### Modifications

* Add keyIsStoredToMessage for encode and decode of KeyValueSchema

### Verifying this change
Unit test pass

### Motivation

Fix #3742

In order to decode the message correctly by AVRO schema, we need to know the schema what the message is.

### Modification

- Introduced Schema Reader and Schema Writer for StructSchema.
   - Reader is used to decode message
   - Writer is used to encode message
- The implementations of StructSchema, provides their schema reader and writer implementations. 
- Introduced a schema reader cache for caching the readers for different schema versions. 

Fixes #3831

### Motivation
Fix #3741 

### Modifications
Support define not not allow null field in schema

### Verifying this change
Add not allow null field schema verify

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (yes)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

* Support passing schema definition for JSON and AVRO schemas

*Motivation*

Currently AVRO and Schema generated schemas from POJO directly.
Sometime people would like to use pre-generated/defined schemas,
so allow passing in schema definitions would clear the confusions
on parsing schemas from POJO.

*Modifications*

- Abstract a common base class `StructSchema` for AVRO/PROTOBUF/JSON
- Standarize on using avro schema for defining schema (we already did that. this change only makes it clearer)
- Add methods to pass schema definition for JSON and AVRO schemas

*NOTES*

We don't support passing schema definition for PROTOBUF. since we only supported generated messages as POJO
class for protobuf schema, and we generate schema definition from the generated messages. it doesn't make sense
to pass in a different schema definition.

* Add missing license header

Fixes #3734

*Motivation*

Exception occurred when using `BytesSchema.of()`

```
Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.apache.pulsar.examples.simple.ProducerExample.main(ProducerExample.java:32)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.apache.pulsar.client.internal.ReflectionUtils.catchExceptions(ReflectionUtils.java:36)
	at org.apache.pulsar.client.internal.DefaultImplementation.newKeyValueSchema(DefaultImplementation.java:158)
	at org.apache.pulsar.client.api.Schema.<clinit>(Schema.java:123)
	... 1 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.pulsar.client.internal.DefaultImplementation.lambda$newKeyValueSchema$16(DefaultImplementation.java:160)
	at org.apache.pulsar.client.internal.ReflectionUtils.catchExceptions(ReflectionUtils.java:34)
	... 3 more
Caused by: java.lang.NullPointerException
	at org.apache.pulsar.client.impl.schema.KeyValueSchema.<init>(KeyValueSchema.java:68)
	... 9 more
```

The problem introduced because the weird class loading and reflection sequence.

When accessing `BytesSchema`, `BytesSchema` will try to initialize `Schema`. When initializing Schema, it will attempts
to initialize `KV_BYTES` using reflection, and initializing KV_BYTES requires `BytesSchema`. Hence it causes KV_BYTES not being
initialized correctly.

The change is to avoid this recursive class loading.

*Motivation*

In order to introduce `GenericRecordBuilder`, we need to know the fields in a `GenericSchema`.
Otherwise, there is no way for us to build a GenericRecord.

*Modifications*

This proposal refactors current generic schema by introducing a `GenericSchema`. This generic schema
provides interfaces to retrieve the fields of a `GenericRecordSchema`.

*Additionally*

This proposal adding the primitive schemas into `Schema` class. So people can program primitive schemas
using Schema interface rather than specific implementations.

*Motivation*

Currently we are supporting POJO based schema in java clients.
POJO schema is only useful when the POJO is predefined. However
in applications like a CDC pipeline, POJO is no predefined, there
is no other way to define a schema.

Since we are using avro schema for schema management, this PR
is proposing a simple schema builder wrapper on avro schema builder.

*Modifications*

Introduce schema builder to build a record schema.

*NOTES*

Currently we only support primitives in defining fields in a record schema in this PR.
We will add nested types in future PRs.

*Motivation*

Currently AUTO_CONSUME only supports decoding records from latest schema.
All the schema versions are lost. It makes AUTO_CONSUME less useful in some use cases,
such as CDC. Because there is no way for the applications to know which version of schema
that a message is using.

In order to support multi-version schema, we need to propagate schema version from
message header through schema#decode method to the decoded record.

*Modifications*

- Introduce a new decode method `decode(byte[] data, byte[] schemaVersion)`. This allows the implementation
  to leverage the schema version.
- Introduce a method `supportSchemaVersioning` to tell which decode methods to use. Because most of the schema
  implementations such as primitive schemas and POJO based schema doesn't make any sense to use schema version.
- Introduce a SchemaProvider which returns a specific schema instance for a given schema version
- Implement a MultiVersionGenericRecordSchema which decode the messages based on schema version. All the records
  decoded by this schema will have schema version and its corresponding schema definitions.

*NOTES

This implementation only introduce the mechanism. But it doesn't wire the multi-version schema
with auto_consume schema. There will be a subsequent pull request on implementing a schema provider
that fetches and caches schemas from brokers.

* Separate pulsar client API interfaces in different module

* Added provided protobuf dep

* Fixed build and tests

* Fixed import name

* Fixed AuthenticationTls class name

* Fixed AutoProduceSchema name

* remove pulsar-client-schema

* shade avro dependencies in pulsar-client-kafka-shaded

* refactor api to separate module

* adding schema back

* cleaning up

* fix license issue

* fixing licenses

* cleaning up

* adding back lombok

* cleaning up

change kafkaConnectSource to return KeyValue type record.

Kafka org.apache.kafka.connect.connector.ConnectRecord has both key and value schema.
Currently we need a KeyValueSchema to achieve it and support debezium.
There will be another PR to do the convert between KeyValueSchema and org.apache.kafka.connect.data.Schema.

*Motivation*

We need an upgrade/backward compatibility story for schema enforcement. Especially around:

- `Producers cannot connect without a schema to topics with a schema`

*Changes*

- provide a flag on brokers to enable schema validation (and disabled it by default). this allows a smooth upgrade on brokers,
  otherwise, it will break all non-java producers on topics with schema immediately when upgrade to the new version

*Motivation*

Currently trigger function is broken due to the schema enforcement we added recently:
A producer without schema can't produce messages into a topic with schema.

*Changes*

- Rename `AUTO` to `AUTO_CONSUME`
- Introduce `AUTO_PRODUCE` schema. The schema produces `byte[]`, but it will validate
  the bytes are compatible with the schema associated with the topic before producing.
- Change trigger function to use `AUTO_PRODUCE` schema.

### Motivation

Sometimes consumers and readers don't have the pojo for the message schemas. so we can use `Schema.Auto` to detect
the schema and reflect the messages into generic records.

 ### Changes

- Introduce AutoSchema and `SchemaType.AUTO`
- fetch schema information first when `AutoSchema` is used

* fixing protobuf schema shading issue

* Fixed NPE and added test for Schema.STRING

* Added license exclusion for generated file SchemaRegistryFormat.java

* Add JSON schema support

* Use different json schema library

* Add license headers to new files

* Revert "Use different json schema library"

This reverts commit 405b918ed9ccebe947916b7c87192b0cef3e910a.

* Changes related to review input

* Upgrade jackson schema module

* Check schema compatibility on subscribe and publish

* Add missing license headers:

* Ensure schema is stored

* Add negative tests on publish/consume with wrong schema

* Add missing license header

* Release ALL protobuf references

* Remove SecureRandom, it's unneeded and appears to timeout

* GRRRRR license headers

* Schema Registry proto changes

* Infrastructure to store schemas

* A default schema registry implementation

* Add admin api for the schema registry

* Renumber schema fields

* Update Pulsar API with schema changes

* Revert field number change

* Fix merge conflict

* Fix broken merge

* DestinationName has been renamed to TopicName

* Address issues in review

* Add schema type back to proto definition

* Address comments regarding lombok usage

* Remove reserved future enum fields

* regenerate code from protobuf

* Remove unused code

* Add schema version to producer success message

* plumb schema through to producer

* Revert "Add schema version to producer success message"

This reverts commit e7e72f468cf46f1605524a7399520c22763583c9.

* Revert "Revert "Add schema version to producer success message""

This reverts commit 7b902f6bdb1cb054e26577747ff4dd8c326a6248.

* Persist schema on producer connect

* Add principal to schema on publish

* Reformat function for readability

* Remove unused protoc profile

* Rename put on schema registry to putIfAbsent

* Reformat function for readability

* Remove unused protoc profile

* Rename put on schema registry to putIfAbsent

* fix compile errors from parent branch changes

* fix lombok tomfoolery on builder

* plumb hash through and allow lookup by data

* wip

* run tests

* wip: address review comments

* switch underscore to slash in schema name

* blah

* Get duplicate schema detection to work

* Fix protobuf version incompatibility

* fix merge issues

* Fix license headers

* Fix license headers

* Address review

* Fix webservice

* plumb schema from producer to server and back

* Plumb schema through subscriber

* Create and return schema via rest endpoint

* Make DELETE great again

* Clean up imports

* Move resource objects to common package

* Fix licenses

* Update error message for schema registry service

* Remove cruft

* Address review comments

- rename props to properties in GetSchemaResponse
- Use config for ledger parameters

* Address review comments

* Fix license headers

* deal with lombock stuff causing issues

* Resolve conflict

* Add type information to Consumer

* Typed Producer and compiling tests

* Typed consumers

* Typed Client with typed consumers and producers

* Wrap reader and writer configurations in typed variants

* remove type annotation from publisher impl

* Add license headers to new files.

* Remove type from PulsarClient

* Type the consumer on the content not the message

* Fix generic type for Reader config

* Remove type information from PartitionedProducerImpl

* Rename Codec to Schema and add javadoc for client interface

* Make the correct api calls

* Fix incorrectly merged test

* WIP: BROKEN REPLICATOR

* unwrap producer

* Passing tests

* Fix broken merge

* wip

* wip

* Updates headers with ASF source files header

* change license to additional files