1. 28 8月, 2019 1 次提交
  2. 23 5月, 2019 1 次提交
    • S
      [schema] AutoConsume should use the schema associated with messages as both... · bf06ef3e
      Sijie Guo 提交于
      [schema] AutoConsume should use the schema associated with messages as both writer and reader schema (#4325)
      
      * [schema] AutoConsume should use the schema associated with messages for both writer and reader schema
      
      *Motivation*
      
      AutoConsume should use the schema associated with the messages for decoding the schemas.
      
      *Modifications*
      
      - provide a flag enable or disable using the provided schema as the reader schema
      - for AUTO_CONSUME schema, disable usnig the provided schema as the reader schema. so it can use the right
        schema version for decoding messages into right generic records
      - provide a few util methods for displaying schema data
      
      * Handle 64 bytes schema version
      
      * Addressed review comments
      bf06ef3e
  3. 28 4月, 2019 1 次提交
    • C
      Add the multi version schema support (#3876) · d5ff0828
      congbo 提交于
      ### Motivation
      
      Fix #3742
      
      In order to decode the message correctly by AVRO schema, we need to know the schema what the message is.
      
      ### Modification
      
      - Introduced Schema Reader and Schema Writer for StructSchema.
         - Reader is used to decode message
         - Writer is used to encode message
      - The implementations of StructSchema, provides their schema reader and writer implementations. 
      - Introduced a schema reader cache for caching the readers for different schema versions. 
      d5ff0828
  4. 08 3月, 2019 1 次提交
    • S
      [schema] Introduce GenericRecordBuilder and the avro implementation (#3690) · 0e48029b
      Sijie Guo 提交于
      *Motivation*
      
      In order to introduce `GenericRecordBuilder`, we need to know the fields in a `GenericSchema`.
      Otherwise, there is no way for us to build a GenericRecord.
      
      *Modifications*
      
      This proposal refactors current generic schema by introducing a `GenericSchema`. This generic schema
      provides interfaces to retrieve the fields of a `GenericRecordSchema`.
      
      *Additionally*
      
      This proposal adding the primitive schemas into `Schema` class. So people can program primitive schemas
      using Schema interface rather than specific implementations.
      0e48029b
  5. 28 2月, 2019 1 次提交
    • S
      [schema] Introduce `GenericSchema` interface (#3683) · f4d56624
      Sijie Guo 提交于
      *Motivation*
      
      In order to introduce `GenericRecordBuilder`, we need to know the fields in a `GenericSchema`.
      Otherwise, there is no way for us to build a GenericRecord.
      
      *Modifications*
      
      This proposal refactors current generic schema by introducing a `GenericSchema`. This generic schema
      provides interfaces to retrieve the fields of a `GenericRecordSchema`.
      
      *Additionally*
      
      This proposal adding the primitive schemas into `Schema` class. So people can program primitive schemas
      using Schema interface rather than specific implementations.
      f4d56624
  6. 26 2月, 2019 1 次提交
    • S
      [schema] Introduce multi version generic record schema (#3670) · 3c36705f
      Sijie Guo 提交于
      *Motivation*
      
      Currently AUTO_CONSUME only supports decoding records from latest schema.
      All the schema versions are lost. It makes AUTO_CONSUME less useful in some use cases,
      such as CDC. Because there is no way for the applications to know which version of schema
      that a message is using.
      
      In order to support multi-version schema, we need to propagate schema version from
      message header through schema#decode method to the decoded record.
      
      *Modifications*
      
      - Introduce a new decode method `decode(byte[] data, byte[] schemaVersion)`. This allows the implementation
        to leverage the schema version.
      - Introduce a method `supportSchemaVersioning` to tell which decode methods to use. Because most of the schema
        implementations such as primitive schemas and POJO based schema doesn't make any sense to use schema version.
      - Introduce a SchemaProvider which returns a specific schema instance for a given schema version
      - Implement a MultiVersionGenericRecordSchema which decode the messages based on schema version. All the records
        decoded by this schema will have schema version and its corresponding schema definitions.
      
      *NOTES
      
      This implementation only introduce the mechanism. But it doesn't wire the multi-version schema
      with auto_consume schema. There will be a subsequent pull request on implementing a schema provider
      that fetches and caches schemas from brokers.
      3c36705f
  7. 07 1月, 2019 1 次提交
    • E
      [Pulsar-Client] Add minor Schema fixes (#3313) · 9e68dda4
      Eren Avsarogullari 提交于
      ### Motivation
      This PR aims to cover some of minor `Schema` fixes.
      
      ### Modifications
      1- `SchemaSerializationException` already extends `RuntimeException` so `RuntimeException` wraps look redundant.
      2- Fixes variable name
      3- Fixes redundant line breaks
      
      ### Test
      1- Adding a new UT for `SchemaSerializationException` case on `JSONSchema.decode()` function
      9e68dda4
  8. 15 12月, 2018 1 次提交
    • B
      remove pulsar-client-schema (#3184) · 5dd4b6e7
      Boyang Jerry Peng 提交于
      * remove pulsar-client-schema
      
      * shade avro dependencies in pulsar-client-kafka-shaded
      
      * refactor api to separate module
      
      * adding schema back
      
      * cleaning up
      
      * fix license issue
      
      * fixing licenses
      
      * cleaning up
      
      * adding back lombok
      
      * cleaning up
      5dd4b6e7
  9. 05 9月, 2018 1 次提交