(cherry picked from commit 40d6248e)

* Use classloading to load use code for functions
(cherry picked from commit 6ff1bbae)

* Added an explicit field in the function details for componenttype

* Fixed unittests

* Updated the defn of python pb file

* Added licence

* Took feedback into account

* Added unittest

* Consolidating utils classes

* further refactoring

* adding missing class

* add license header

* Cleanup logic in JavaInstanceRunnable close method

* Added comments

1) Collect input topics from the function details spec
2) Catch all errors during source/sink close since its user code

* fix: NPE when stats manager not initialized

* remove unnecessary imports

* add test

### Motivation
   when I study and read this module code , I fix some error like:
1. words error, such as JaveInstance -> JavaInstance, Recieved -> Received,  Unknwon -> Unknown, alterted -> alterted, Updare -> Update, etc.
2. remove not use import java class.
3. duplicated code in this  java class I extract method. 
4. javadoc  styles are consistent (not required)

### Modifications

minor fix up in pulsar-functions module

* On publish failures, log error and count them as sys exceptions

* Took feedback

* report sys errors when failed to produce sucessfully

* enhancing error message

* adding metrics to context publish

* enhancing error logging and adding java error handling

* fixing tests

* cleaning up

* addressing comments and fix bug

* cleaning up

* fix bug

* fix test

* check is none

* fix unit test

* cleaning up

* Added ability to specify consumer queue size for function input topics

* fix and update PR

* Cleanup consumer subscriptions and fix graceful shutdown for functions

* cleaning up

* removing testing files

* add unit tests

* adding integration testing

* refactoring

* refactoring and adding tests

* cleaning up

Clean up and correct properties to producer and consumers created by Functions/Sinks/Sources (#3315)

* clean up and correct properties to producer and consumers created by functions

* fix test

* cleaning up comment

* refactoring

* add sink and source prometheus stats

* fixing stuff

### Motivation

Optimized the stats collection code for java functions to reduce stats collection foot print both memory and cpu.  Reduced the number of objects allocated by 20% in java functions

* Introduce Source and Sink exceptions in Status

* Added the counter at the right place

* Use instanceCache schedulerExecutorService for timer

* fixing process runtime

* addressing comments

### Motivation

Allow exceptions in functions to be reported via prometheus

example output:
```
# TYPE pulsar_function_user_exception gauge
pulsar_function_user_exception{cluster="standalone",error="error val: 0",fqfn="public/default/py-test",function="py-test",instance_id="0",namespace="public/default",tenant="public",ts="1543786615954"} 1.0
pulsar_function_user_exception{tenant="public",namespace="public/default",function="test",instance_id="0",cluster="standalone",fqfn="public/default/test",error="error val: 5c53460e-03cf-4368-88d4-f23aeb3adf84",ts="1543787031371",} 1.0
```

* prometheus metrics for functions served via brokers or instances themselves should match

* add additional testing

* Make sure to properly count number of processed messages in python

* Removed total processed

* Fixed build

* Fixed buil

* Address feedback

* Fixed unittest

* Removed unused value

* Added licence headers

* Removed unnecessary changes

* Fix integration tests

* Added numReceived as part of function status

* Unnecessary change revert

* Enhance test

* adding windowed metrics for functions

* adding license headers and cleaning up

* remove unnecessary import

* add RestException

* fixing bugs and refactoring code

* fix bug in instanceCache

* fix bug

* add test for stats and fix minor bug

[state][standalone] When running with more than one function instance, Function instances will crash because of table service throws INTERNAL_SERVER_ERROR (#3017)

*Motivation*

When running with more than one function instance, Function instances will crash because of table service throws INTERNAL_SERVER_ERROR

```
org.apache.pulsar.functions.runtime.shaded.org.apache.bookkeeper.clients.exceptions.ClientException: fail to access its root range : code = INTERNAL_SERVER_ERROR
        at org.apache.pulsar.functions.runtime.shaded.org.apache.bookkeeper.clients.impl.internal.ProtocolInternalUtils.createRootRangeException(ProtocolInternalUtils.java:105) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.org.apache.bookkeeper.clients.impl.internal.RootRangeClientImpl.processCreateNamespaceResponse(RootRangeClientImpl.java:129) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.org.apache.bookkeeper.clients.impl.internal.RootRangeClientImpl.lambda$createNamespace$4(RootRangeClientImpl.java:118) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.org.apache.bookkeeper.clients.utils.RpcUtils$1.onSuccess(RpcUtils.java:78) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.com.google.common.util.concurrent.Futures$4.run(Futures.java:1135) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:655) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.stub.ClientCalls$GrpcFuture.set(ClientCalls.java:487) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:465) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:391) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:471) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:553) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:474) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:591) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at org.apache.pulsar.functions.runtime.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[java-instance.jar:2.3.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144]

```

in current version of bk, `INTERNAL_SEVER_ERROR` was thrown a client attempts to create a table while the table already exists. In a newer version, the client handles that better.

*Changes*

A temp fix before upgrading BK to 4.9 is to handle creating state tables in a more reliable way.

* removing metrics module from functions

* removing other references ot Prometheus metrics server

* removing references to pulsar-functions-metrics

* fix unit tests

* revert test change

* cleaning up and improving function metrics

* slight refactoring

* fix unittest

* moving pulsar-spark tests into its own module

* fix bugs in python metrics

* instance expose metrics

* remove commented out code

* fix unit tests

* remove commented out code

* fixing test

* fix python instance test

* removing old code

* fix bug

* refactoring java metrics

* refactoring python metrics

* cleaning up code

* removing unneccessary code

* improving metrics format

* fixing test

* fix bugs and revising format

* fix bug

* fix for python3

* change user defined metric to summary

* renaming labels

* change back python

* Beginging to refactor function metrics to use prometheus

* fix unit test

* removing test code

* fix test

* minor refactoring

* addressing comments

* adding python instance

* remove test code

* adding prometheus client as instance dependency

* fix bug

* adding prometheus to license file

[state] Initialize bookkeeper table service metadata on initializing pulsar cluster metadata (#2706)


*Motivation*

This is the first set of changes to better integration with state storage.

*Changes*

- PulsarClusterMetadataSetup should initialize the metadata for bookkeeper table service
- Move the common util functions to `StateUtils`
- Delete state table on deleting functions

* Hooked up secrets function api with secret function implementations

* Fixed unittest

* Added more docs

* Took feedback into account

### Motivation

There are use cases that the destination topic for a message cannot be determined at source submission time.  This requires the ability for sources to to set which topic a record should be written to.

### Modifications

1. add an interface Record that allows users to set the destination topic for a record
2. Refactored the Pulsar sink code to support this

* [functions] change instance id from string to int and expose number of instances in context

 ### Motivation

When writing a connector reading from a list of sources, it is hard for the connector implementation to decide how
to distribute the list of sources across the function instances. because there is no way to tell how many function
instances is running.

 ### Changes

- change instance id from string to integer (since the implementation is already assuming instance id is an integer)
- add getNumInstances in the context
- expose both interfaces in source and sink connector context

* Fix compilation

* Added ability to specify dead letter topic to functions

* Fix bug

* Added an example function that fails on a particular message consistently

* Revert change

*Motivation*

with more and more connector are added, it becomes expensive to start all external services at the begin.

*Changes*

- refactor the connector testing framework to start external service before methods
- fix kafka, cassandra and mysql connectors

* Fix handling of nar archives

* Adress review comments

* Added a integration test to test archive

* Fixed build

* Fix build

* Fix integratin test

Attach Producer/Consumer property tags so its easier to identify topics being produced/consumed by functions (#2490)

* Integrate functions and io with schema registry

* Added missing license headers

* Renamed topicSchema to inputSpecs

* Fixed comments

* Fixed cli arg docs

* Fixed schema arguments

* Fixed errors after merge

* Fixed instance parameters

* Fixed tests and addressed comments

* Fixed PulsarSourceTest after merge

* Took feedback and made changes backwards compatible

* Fixed compilation issues

* Fixed bug

* fixed test compilation

* Fixed bug

* Fixed bug

* Fix pythn instance

* Fixed the way cmdsink populates sinkconfig

* Addressed feedback