Fixes #7481

Motivation
Default bookkeeper conf that ships with apache pulsar indicates that the unit of openLedgerRereplicationGracePeriod is in seconds, but official bookkeeper doc(and code) mentions that the unit is in milliseconds.

Pulsar should configure default as 30000 instead of 30 and bookkeeper.conf should mention that the unit of this period is milliseconds instead of seconds.

Modifications
Update the bookkeep.config file
Update the following Pulsar docs for releases 2.5.2, 2.6.0, 2.6.1, and master
Reference > Pulsar Config > BookKeeper

Update AWS deployment in reference-configuration.

Motivation
This doc PR is updated for configurations for PRs:
#6716
#6853
#6074

1: The broker configuration (for #6716) is updated by Jia Zhai.

2: Add other supported configurations to the client, standlone and proxy configuration docs based on the client.config, standlone.config and proxy.config files.

Modifications
1: Add TLS with keystore type config in standlone and proxy configuration file.
2: update reference > pulsar configuration > client for PIP-55: Refresh Authentication Credentials
Add other supported configurations to the standlone and proxy configuration files based on the standlone.config and proxy.config files.

* add proxy auth data forwarding and refine pulsar proxy content

* fix PR- 5845

* add a note

* Update site2/docs/administration-proxy.md
Co-authored-by: NYu Liu <50226895+Anonymitaet@users.noreply.github.com>

* Update site2/docs/administration-proxy.md
Co-authored-by: NHuanliMeng <48120384+Huanli-Meng@users.noreply.github.com>

* Update site2/docs/administration-proxy.md
Co-authored-by: NHuanliMeng <48120384+Huanli-Meng@users.noreply.github.com>
Co-authored-by: NYu Liu <50226895+Anonymitaet@users.noreply.github.com>
Co-authored-by: NHuanliMeng <48120384+Huanli-Meng@users.noreply.github.com>

Add advertised multiple listeners configurations #7274

### Motivation
This doc PR is for PIP 28: Pulsar Proxy Gateway Improvement released on 2.4.0. 

Doc contents have been approved by the PR: https://github.com/apache/pulsar/pull/7142

because that PR fails to pass CI tests. So i close that PR and create a new PR.


### Modifications

add a proxyLogLevel parameter in the proxy config file.

# Motivation

The [PR](https://github.com/apache/pulsar/pull/7078) introduce precise topic publish rate limiting on broker. But the doc is not updated accordingly.


### Modifications

Update the Pulsar doc: Reference > Pulsar configuration > broker

Update the docs for PR: https://github.com/apache/pulsar/pull/7116/files

Add a maxMessageSize configure item.

Motivation
Some users may be confused by bookkeeperMetadataServiceUri in multi-hosts case (e.g. comma list like zk+hierarchical://zk1:2181,zk2:2181/ledgers) since bookkeeper only support semicolon separated values currently ( my colleague is tracing this in bookkeeper master, we can overcome this in future).

Modifications
Add multi-hosts usage example in broker.conf.

Co-authored-by: Anonymitaet <anonymitaet_hotmail.com>

### Motivation

Fixes #6761

### Modifications

Correct tokenSecretKey base64 inline description in documents (based master & version 2.5.1) and examples.

* [Doc] Update for acknowledgment at batch index level

* Update site2/docs/concepts-messaging.md

Add advertisedAddress and subscriptionExpirationTimeMinutes conf for broker and proxy

Update the broker configuration file in the Reference > Pulsar configuration for the issue apache/pulsar#6087 (#6990)

Fixes #6640

### Motivation

Add Tls with keystore type config.

### Modifications

Add Tls with keystore type config.

### Verifying this change

- Unit test passed

Fixes #6793

### Motivation
Now broker, can't set max number of partitions per partitioned topic. So we can't prevent to create partitioned topic by many partitions from server side.
In this PR, introduce limit of max partitions to broker and to be able to control it.

### Modifications
Add `maxNumPartitionsPerPartitionedTopic` config to broker and compare with numPartitions at create or update partitioned topic.
If the config is set to `0`(is default), then disable the check.

Bookkeeper has settings to allow for periodic calls to refresh the
bookie info. In dynamic environments (like when running on k8s) fetching
the bookie info more often can be helpful to ensure the brokers stay up
to date on the list of bookies even in the event of zookeeper watch
issues.
Co-authored-by: NAddison Higham <ahigham@instructure.com>

### Motivation

Fixes #5935

Support multi pulsar clusters to use the specified bookkeeper cluster by pointing `BookKeeper Client` to the zookeeper connection string of bookkeeper cluster.

### Modifications

- Add a config `bookkeeperServiceUri` to discover bookkeeper cluster metadata store.
- Use metadata service uri to init bookkeeper client.

Fixes #6711

### Motivation
User like to be able to configure the JWT authentication provider to verify the audience on incoming tokens.  I believe this will improve security because it would prevent a spoofer from reusing a token that was intended for another purpose (yet signed by the same issuer).  [RFC 6749 section 4.1.3](https://tools.ietf.org/html/rfc7519#section-4.1.3) has some guidance on this.  In my scenario, the token is an OAuth 2.0 token, and OAuth 2.0 makes extensive use of the audience claim ([ref](https://auth0.com/docs/tokens/guides/validate-access-tokens#check-additional-standard-claims)).

1. a configurable audience claim name (e.g. `aud`).
2. if audience isn't configured, do not validate the audience (for back-compatibility).
3. if audience is configured, validate that the value is present in the token.

### Modifications
- Add the logic in AuthenticationProviderToken.
- Add related tests.

### Verifying this change
- Ut passed

### Motivation

Some time due to high dispatch rate on one of the topic can temporarily cause broker to go OOM and it will be transient error and broker can recover within a few seconds as soon as some memory gets released. However, 2.4 release has change #4196 which restarts broker on OOM which can cause huge instability in cluster where that topic moves from one broker to another and restarts multiple brokers and cause disruption for other topics as well. we have seen similar kind of issue mentioned at  #5896 . This could be transient error and we need a way to ignore this error. So, we need a dynamic flag to skip broker shutdown on OOM to avoid instability in a cluster.
```
01:48:49.549 [pulsar-io-22-37] ERROR org.apache.pulsar.PulsarBrokerStarter - -- Shutting down - Received OOM exception: Direct buffer memory
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:769) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:745) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164) ~[bookkeeper-common-allocator-4.9.4.2-yahoo.jar:4.9.4.2-yahoo]
        at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158) ~[bookkeeper-common-allocator-4.9.4.2-yahoo.jar:4.9.4.2-yahoo]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1912) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1923) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:826) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:797) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:778) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at org.apache.pulsar.broker.service.Consumer.lambda$sendMessages$51(Consumer.java:265) ~[pulsar-broker-2.4.6-yahoo.jar:2.4.6-yahoo]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.32.Final.jar:4.1.32.
:
: 
01:48:49.549 [pulsar-io-22-39] ERROR org.apache.pulsar.PulsarBrokerStarter - -- Shutting down - Received OOM exception: Direct buffer memory
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:769) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:745) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
        at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164) [bookkeeper-common-allocator-4.9.4.2-ya
hoo.jar:4.9.4.2-yahoo]
        at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158) [bookkeeper-common-allocator-4.9.4.2-yahoo.jar:4.9.4.2-yahoo]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) [netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) [netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) [netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) [netty-all-4.1.32.Final.jar:4.1.32.Final]
```

### Modification
Add dynamic flag to avoid broker shutdown on OOM.

### Motivation

Fixes #6394

### Modifications

- provide a flag `allowAutoSubscriptionCreation` in `ServiceConfiguration`, defaults to `true`
- when `allowAutoSubscriptionCreation` is disabled and the specified subscription (`Durable`) on the topic does not exist when trying to subscribe via a consumer, the server should reject the request directly by `handleSubscribe` in `ServerCnx`
- create the subscription on the coordination topic if it does not exist when init `WorkerService`

Fixes #6045 #6281 

### Motivation

Enable get precise backlog and backlog without delayed messages.

### Verifying this change

Added new unit tests for the change.

### Motivation

Currently, bundle split splits the bundle into two parts of the same size. When there are fewer topics, bundle split does not work well. The topic assigned to the broker according to the topic name hash value, hashing is not effective in a small number of topics bundle split.

So, this PR introduces an option(-balance-topic-count) for bundle split.  When setting it to true, the given bundle splits to 2 parts, each part has the same amount of topics.

And introduce a new Load Manager implementation named `org.apache.pulsar.broker.loadbalance.impl.BalanceTopicCountModularLoadManager`.  The new Load Manager implementation splits bundle with balance topics count, others are not different from ModularLoadManagerImpl.

### Motivation
Some parameters are added in the `broker.conf` and `standalone.conf` files. However, those parameters are not updated in the docs.
See the following PRs for details: #4150, #4066, #4197, #3819, #4261, #4273, #4320.

### Modifications
Add those parameter info, and sync docs with the code.

Does not update the description quite much, there are two reasons for this:
1. Keep doc content consistent with code. We need to update the description for those parameters in the code first, and then sync them in docs.
2. Will adopt a generator to generate those content automatically in the near future.

Most namespace level configurations have corresponding cluster
configuration that set a namespace default.

The offload threshold does not, which makes it more difficult to ensure
that namespaces have the cluster wide namespace defaults.

There is one small wrinkle with this commit in that `-1` is used as a
sentinel value to indicate to use the cluster default, this means that
if the cluster default is to have offloading on and it is desired to
  disable a specific namespace, the namespace needs to set this value to
  some negative number other than `-1`!

* Fix Issue 5614, Deprecate brokerServicePurgeInactiveFrequencyInSeconds

* [Issue 5614] Deprecate brokerServicePurgeInactiveFrequencyInSeconds

* update

* Update

Master Issue:  #4926

### Motivation


Curently the partitioned-topic and non-partitioned topic is a little confuse for users. in PR #3450 we add config for auto-topic-creation.
We could leverage this config to provide some more config for auto-topic-creation.

### Modifications

- Add `allowAutoTopicCreationType` and `allowAutoTopicCreationNumPartitions` to configuration.
- Users can use both configurations when they decide to create a topic automatically.
- Add test.
- Update doc.

This commit adds a new option optionally disable authentication for the
`/metrics` endpoint in the pulsar-proxy.

Currently, authentication is required for the metrics endpoint when
authentication is enabled, which makes monitoring more difficult.
However, rather than just disable it completely and allow for metrics to
be exposed to any unknown user, this makes it opt in.

It could be argued that it should default to false, but as it is likely
that the proxy is the only component potentially exposed to the public internet, we
default to not exposing data.

Fixes #4920

Before this patch, all keys are read as RSA, which meant that only RSA
compatible JWT signing algorithms could be used, specifically, this
limited the use of ECDSA family of JWT keys.

This changes this by changing the signature we use to parse keys to also
take a SignatureAlgorithm and also adds a new config option
`tokenPublicAlg` which can be used to signify what algorithm the
broker/proxy should use when reading public keys. However, these all
default to RS256, which, should indicate to decode as RSA (even if
another RS/PS algoritm is used).

This also adds some new options to the Token CLI tool for those commands
that weren't respecting the algorithm, but these are defaulted to RS256
as well.

* Add support for AWS instance and role creds

This commit makes changes to the tiered storage support for S3
to allow for support of ec2 metadata instance credentials as well as
additional config options for assuming a role to get credentials.

This works by changing the way we provide credentials to use the
funtional `Supplier` interface and for using the AWS specific
`SessionCredentials` object for when we detect that the
`CredentialProvider` is providing credentials that have a session token.

* [tiered_storage] Tweak s3 credential handling to check on boot

This changes the s3 handling slightly, instead of falling back to static
credentials, we instead now fail if no s3 credentials can be found and
change the unit tests to start a broker with s3 credentials.

With the new Supplier API, we now fetch credentials on every request.
Because of this, the failure and subsequent try/catch is costly and the
integration tests were using this, which caused them to be significantly
slower.

Instead, we just check to see if we can fetch creds, and if we can't
consider it an error condition to exit the app as it is unlikely in a
production scenario to not have some credentials.

* fix s3 test for missing creds

* [pulsar-broker] add support of secondary bookie-isolation-group

* fix docs

* add negative test

* Fix tests

* By default, auto configure the size of Bookie read/write cache

* Updated ref docs

* Allow to configure the managed ledger cache eviction frequency

* Fixed test

* Simplified the cache eviction to make it predictable at the configured frequency

* Address comments

* Apply eviction on slowest active reader by preference

* Re-introduced backlogged subscriptions test

* Addressed comments

* Use config option

* Fixed active/inactive logic and read position

* Use dedicated thread for cache evictions

* Added config options in docs

* Fixed tests

* Added time triggered eviction test

* Fixed flaky test

* Fixed tests