README.md

    BDS

    logo

    Introduction

    JD Cloud Blockchain Data Service (BDS) is a realtime data aggregating, analyzing and visualization service for chain-like unstructured data from all kinds of 3rd party Blockchains

    Splitter is the key module of Blockchain Data Service (BDS) and provides data analysis capability.

    Splitter is responsible for consuming blockchain data from message queue (kafka) and inserting data into persistent data storage services (relational database, data warehouse, etc.) for further processing

    Architecture

    Architecture

    Environment Deployment

    Install BDS

    Environment initialization

    Before compiling and running BDS, you must install go's compilation environment locally: go install

    Install Splitter steps

    1. Set the path of project : $GOPATH/src/github.com/jdcloud-bds/bds/
    2. Inputgo build -v github.com/jdcloud-bds/bds/cmd/bds-splitter,compile to get executable file bds-splitter
    3. Build new configuration file splitter.conf, see /config/splitter_example.conf configuration file template
    4. Run program ./bds-splitter -c splitter.conf

    Install confluent and kafka

    Install kafka

    See kafka

    Modify config/server.properties
    • message.max.bytes=1048576000

    Install confluent

    see confluent

    Unzip the confluent package and run Confluent REST Proxy

    Modify /etc/kafka-rest/kafka-rest.properties
    • max.request.size = 1048576000
    • buffer.memory = 1048576000
    • send.buffer.bytes = 1048576000

    Database

    Database we now support SQL Server, PostgreSQL, you can choose one as a data storage method.

    SQL Server

    Buy JCS For SQL Server

    PostgreSQL

    Buy JCS For PostgreSQL

    After you run the database, you need to manually create new database and use the database name initialization splitter.conf.

    Install Grafana

    See Grafana Official

    Source code

    Splitter Modules

    Development Steps

    1. Define the data structure of Kafka messages.
    2. Define table structure.
    3. Analyze Kafka message and store data in database.

    Contributing

    Contributing guide

    License

    Apache License 2.0

    Project Demonstration

    Blockchain Data Service

    项目简介

    当前项目暂无项目简介

    发行版本

    当前项目没有发行版本

    贡献者 6

    开发语言

    • Go 99.5 %
    • Makefile 0.4 %
    • Smarty 0.2 %