diff --git a/README.md b/README.md index 67eb2d7102a7702c8499e4bbdeb917a3a9225ea1..fa57512166da6462d21f28a36c88f831b15aaa5a 100644 --- a/README.md +++ b/README.md @@ -4,89 +4,92 @@

 

+ + QuestDB community Slack channel + - + Apache 2.0 licence - - + + QuestDB Docker images - + Azure pipeline status - - - - + QuestDB on Apache Maven

+# QuestDB + +QuestDB is a high-performance, open-source SQL database for applications in +financial services, IoT, machine learning, DevOps and observability. It includes +endpoints for PostgreSQL wire protocol, high-throughput schema-agnostic +ingestion using InfluxDB Line Protocol, and a REST API for queries, bulk +imports, and exports. + +QuestDB implements ANSI SQL with native extensions for time-oriented language +features. These extensions make it simple to correlate data from multiple +sources using relational and time series joins. QuestDB achieves high +performance from a column-oriented storage model, massively-parallelized vector +execution, SIMD instructions, and various low-latency techniques. The entire +codebase was built from the ground up in Java and C++, with no dependencies, and +is 100% free from garbage collection. +
- - - - - + + QuestDB Web Console showing multiple SQL statements and visualizing a query as a chart
-## What is QuestDB +## Try QuestDB -QuestDB is a high performance open source SQL database for time series data. +We provide a [live demo](https://demo.questdb.io/) provisioned with the latest +QuestDB release and a 1.6 billion row dataset with 10 years of NYC taxi trips to +query. -It uses a column-oriented approach, heavy parallelized vectorized execution, -SIMD instructions and a whole array of low-latency techniques. The whole code -base is built from scratch, without dependencies and 100% free from garbage -collection. +To run QuestDB, Docker can be used to get started quickly: -QuestDB implements SQL and augments it for time-series with native extensions. -It exposes a PostgreSQL wire protocol, high-performance REST API and supports -ingestion with InfluxDB Line Protocol. QuestDB uses a relational model with -maintenance-free schemas. Relational and time-series joins make it easy to -correlate data over time. Writes are durably committed to disk, meaning that the -data is safe - yet instantly accessible. +```bash +docker run -p 9000:9000 -p 8812:8812 questdb/questdb +``` -[QuestDB's documentation](https://questdb.io/docs/introduction/) +macOS users can use Homebrew: -## Live demo +```bash +brew install questdb +brew services start questdb +``` -Query [our demo](https://demo.questdb.io) dataset with 1.6 billion rows in -milliseconds. +The [QuestDB downloads page](https://questdb.io/get-questdb/) provides direct +downloads for binaries and has details for other installation and deployment +methods. -## Web Console +### Connecting to QuestDB -The interactive console to import data (drag and drop) and start querying right -away. Check our Web Console guide to get started: +You can interact with QuestDB using the following interfaces: -
- - Screenshot of the Web Console showing various SQL statements and the result of one as a chart - -
-
- - Web Console guide - -
+- [Web Console](https://questdb.io/docs/reference/web-console/) listening on + port `9000` +- [REST API](https://questdb.io/docs/reference/api/rest/) on port `9000` +- [PostgreSQL](https://questdb.io/docs/reference/api/postgres/) wire protocol on + port `8812` +- [InfluxDB](https://questdb.io/docs/reference/api/influxdb/) line protocol for + high-throughput ingestion on port `9009` ## Performance figures -### Raw figures - -Number operations per second **per thread**. Writes are durable and written to -disk. +On a CPU with 6 memory channels, QuestDB can scan through 117GB of data per +second. The following table shows the number operations per second, per thread: | Operation | 64-bit double | 32-bit int | | --------- | -------------- | -------------- | | Read | 120 Million /s | 240 Million /s | | Write | 240 Million /s | 480 Million /s | -On a CPU with 6 memory channels, QuestDB can scan through **117GB of data per -second**. - -### Queries - -Execution time on a c5.metal instance using 16 of the 96 threads available. +The following table shows query execution time on a c5.metal instance using 16 +of the 96 threads available: | Query | Runtime | | --------------------------------------------------------- | ---------- | @@ -94,150 +97,41 @@ Execution time on a c5.metal instance using 16 of the 96 threads available. | `SELECT tag, sum(double) FROM 1bn` | 0.179 secs | | `SELECT tag, sum(double) FROM 1bn WHERE timestamp='2019'` | 0.05 secs | -## Getting Started - -The easiest way to get started is with Docker: - -```script -docker run -p 9000:9000 -p 8812:8812 questdb/questdb -``` - -You can find more information about Docker usage on the -[dedicated page](https://questdb.io/docs/get-started/docker/). - -#### Alternative methods - -- [Start with Homebrew](https://questdb.io/docs/get-started/homebrew/) -- [Start with the binaries](https://questdb.io/docs/get-started/binaries/) - -### Connecting to QuestDB - -You can interact with QuestDB using: - -- [Web Console](https://questdb.io/docs/reference/web-console/) listening - on port `9000`: [localhost:9000](http://localhost:9000) -- [Postgres](https://questdb.io/docs/reference/api/postgres/) on port `8812` -- [REST API](https://questdb.io/docs/reference/api/rest/) on port `9000` - -Both the HTTP and PostgreSQL servers reference the database in -`/db`. - -You can connect to the Postgres server as follows. The default password is -`quest`: - -```script -psql -h localhost -p 8812 -U admin -W -d qdb -``` - -## Building from source - -#### (a) Prerequisites - -- Java 11 64-bit -- Maven 3 -- Node.js 12 / NPM 6 - -```script -java --version -mvn --version -node --version -``` - -#### (b) Clone the Repository - -```script -git clone git@github.com:questdb/questdb.git -``` - -#### (c) Build the Code - -Commands below will create JAR without assembling executable binaries nor -building web console. - -```script -cd questdb -mvn clean package -DskipTests -``` - -To package web console with the jar use the following command: - -```script -mvn clean package -DskipTests -P build-web-console -``` - -To build executable binaries use the following command: - -```script -mvn clean package -DskipTests -P build-web-console,build-binaries -``` - -To run tests it is not required to have binaries built nor web console. There -are over 4000 tests that should complete without 2-6 minutes depending on the -system. - -```script -mvn clean test -``` - -To release to Maven Central use the following command that activates deploy -profile. Ensure that your `~/.m2/settings.xml` contains username/password for -server `central` and `gnupg` is on hand to sign the artefacts. - -```script -mvn -pl !benchmarks clean deploy -DskipTests -P build-web-console,maven-central-release -``` - -#### (d) Run QuestDB - -```script -# Create a database root directory and run QuestDB -mkdir -java -p core/target/questdb-5.0.5-SNAPSHOT.jar -m io.questdb/io.questdb.ServerMain -d -``` - -## Resources - -Complete references are available in the -[Documentation](https://questdb.io/docs/introduction/). - -Get started: - -- [Docker](https://questdb.io/docs/get-started/docker/) -- [Binaries](https://questdb.io/docs/get-started/binaries/) -- [Homebrew](https://questdb.io/docs/get-started/homebrew/) - -Develop: - -- [Connect](https://questdb.io/docs/develop/connect/) -- [Insert data](https://questdb.io/docs/develop/insert-data/) -- [Query data](https://questdb.io/docs/develop/query-data/) -- [Authenticate](https://questdb.io/docs/develop/authenticate/) - -Concepts: - -- [SQL extensions](https://questdb.io/docs/concept/sql-extensions/) -- [Storage model](https://questdb.io/docs/concept/storage-model/) -- [Partitions](https://questdb.io/docs/concept/partitions/) -- [Designated timestamp](https://questdb.io/docs/concept/designated-timestamp/) - -## Support / Contact - -[Slack Channel](https://slack.questdb.io) - -## Roadmap - -[Our roadmap is here](https://github.com/questdb/questdb/projects/3) - -## Contribution - -Feel free to contribute to the project by forking the repository and submitting -pull requests. Please make sure you have read our -[contributing guide](https://github.com/questdb/questdb/blob/master/CONTRIBUTING.md). - -## Contributors ✨ - -Thanks to these wonderful people -([emoji key](https://allcontributors.org/docs/en/emoji-key)): +## Documentation & resources + +- The [QuestDB documentation](https://questdb.io/docs/introduction/) describes + how to run and configure QuestDB with technical references. +- [Our Slack workspace](https://slack.questdb.io) is a great place for technical + discussions and to meet other users. :wave: +- [Tutorials](https://questdb.io/tutorial/) written by our community members + show what's possible with QuestDB. +- [QuestDB on Stack Overflow](https://stackoverflow.com/questions/tagged/questdb) + has common troubleshooting solutions. +- [GitHub issues](https://github.com/questdb/questdb/issues) are used to track + bug reports and feature requests. +- [The product roadmap](https://github.com/questdb/questdb/projects/3) lists the + tasks and features we're currently working on. + +## Contribute + +We are always happy to have contributions to the project whether it is source +code, documentation, bug reports, feature requests or feedback. To get started +with contributing: + +- Have a look through GitHub issues labeled + "[Good first issue](https://github.com/questdb/questdb/issues?q=is%3Aissue+is%3Aopen+label%3A%22Good+first+issue%22)". +- Read the + [contribution guide](https://github.com/questdb/questdb/blob/master/CONTRIBUTING.md). +- For details on building QuestDB, see the + [build instructions](https://github.com/questdb/questdb/blob/master/core/README.md). +- [Create a fork](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) + of QuestDB and submit a pull request with your proposed changes. + +As a sign of our gratitude, we'll send contributors some of our QuestDB swag +such as stickers and t-shirts! :star2: + +A big thanks goes to the following wonderful people who have contributed to +QuestDB: ([emoji key](https://allcontributors.org/docs/en/emoji-key)): diff --git a/core/README.md b/core/README.md index 174e85232a8c9f7733c16733cfa418e56da1b7be..31137aa6f0c97edf0980184fd7b0fe940cbafb1c 100644 --- a/core/README.md +++ b/core/README.md @@ -1,9 +1,80 @@ -## Building Docker images +# Building QuestDB -To build docker images successfully please follow these instructions precisely. There could be a lot of varions on how these images can be built. This is what worked: +As a first step, clone the repository: + +```script +git clone git@github.com:questdb/questdb.git +``` + +## Building from source + +### Prerequisites + +- Java 11 64-bit +- Maven 3 +- Node.js 12 / NPM 6 + +```script +java --version +mvn --version +node --version +``` + +### Maven commands + +The commands below will create a JAR without assembling executable binaries or +building web console: + +```script +cd questdb +mvn clean package -DskipTests +``` + +To package the web console with the JAR, use the following command: + +```script +mvn clean package -DskipTests -P build-web-console +``` + +To build executable binaries use the following command: + +```script +mvn clean package -DskipTests -P build-web-console,build-binaries +``` + +To run tests, it is not required to have the binaries or web console built. +There are over 4000 tests that should complete without 2-6 minutes depending on +the system: + +```script +mvn clean test +``` + +To release to Maven Central use the following command that activates deploy +profile. Ensure that your `~/.m2/settings.xml` contains username/password for +server `central` and `gnupg` is on hand to sign the artefacts. + +```script +mvn -pl !benchmarks clean deploy -DskipTests -P build-web-console,maven-central-release +``` + +### Run QuestDB + +```script +# Create a database root directory and run QuestDB +mkdir +java -p core/target/questdb-5.0.5-SNAPSHOT.jar -m io.questdb/io.questdb.ServerMain -d +``` + +## Docker images + +To build docker images successfully please follow these instructions precisely. +There could be a lot of variations on how these images can be built. This is +what worked: - Use Windows OS. Docker Desktop might also work on Mac, but not on Linux. -- Download Docker Desktop _Edge_. At the time of writing (Nov 2019) only Edge version is able to build ARM63 images and help create multi-platform manifest. +- Download Docker Desktop _Edge_. At the time of writing (Nov 2019) only Edge + version is able to build ARM64 images and help create multi-platform manifest. - Ensure "experimental" features are enabled. To verify that your Docker Desktop is good to go try the following command: @@ -21,111 +92,132 @@ default * docker default default running linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6 ``` -If docker complains on `buildx` command - you either running _stable_ version or have not enabled experimental features. Look through Docker Settings. There is JSON config you need to edit __and__ toggle a switch in UI +If docker complains on `buildx` command - you either running _stable_ version or +have not enabled experimental features. Look through Docker Settings. There is +JSON config you need to edit **and** toggle a switch in UI Assuming all is good, lets start building. ### Login to Docker Hub -The following command will prompt you for everything thats required and will keep your logged in for some time. You don't have to run it if you know you are logged in. +The following command will prompt you for everything thats required and will +keep your logged in for some time. You don't have to run it if you know you are +logged in. + ``` docker login ``` ### Switch Docker Desktop to Linux -To do that right click on Docker Desktop tray icon (bottom right) and chose switch from pop-up menu. +To do that right click on Docker Desktop tray icon (bottom right) and chose +switch from pop-up menu. Create new builder + ``` docker buildx create --name multiarchbuilder docker buildx use multiarchbuilder docker buildx inspect --bootstrap ``` -Create AMD64 image. You can test this image locally and push it only when its ready. + +Create AMD64 image. You can test this image locally and push it only when it's +ready. + ``` docker build -t questdb/questdb:4.0.0-linux-amd64 --file Dockerfile-linux . ``` Push this image eventually: + ``` docker push questdb/questdb:4.0.0-linux-amd64 ``` Create ARM64 image. + ``` docker buildx build --platform linux/arm64 -t questdb/questdb:4.0.0-linux-arm64 --file Dockerfile-linux-arm64 . --load ``` Push that eventually as well: + ``` docker push questdb/questdb:4.0.0-linux-arm64 ``` ### Switch Docker Desktop to Windows -Build Windows image. Notice that this build does not use `buildx`. Also make sure that tag (:4.0.0-windows) reflect QuestDB version. +Build Windows image. Notice that this build does not use `buildx`. Also make +sure that tag (:4.0.0-windows) reflect QuestDB version. + ``` docker build -t questdb/questdb:4.0.0-windows-amd64 --file Dockerfile-windows . ``` Push to Docker Hub: + ``` docker push questdb/questdb:4.0.0-windows-amd64 ``` ### Create manifest -The purpose of the manifest is to simplify image usage by end user. They will just hopefully install `questdb/questdb` and Docker Hub sill figure out appropriate image for target platform. +The purpose of the manifest is to simplify image usage by end user. They will +just hopefully install `questdb/questdb` and Docker Hub sill figure out +appropriate image for target platform. ``` docker manifest create questdb/questdb:4.0.0 questdb/questdb:4.0.0-linux-arm64 questdb/questdb:4.0.0-linux-amd64 questdb/questdb:4.0.0-windows-amd64 ``` Push manifest: + ``` docker manifest push questdb/questdb:4.0.0 --purge ``` -The --purge option is there to delete manifest on local Docker. If you do not do it and find out that you added wrong image to manifest, it would be impossible to take that image out! -## Running QuestDB in Docker Container +The --purge option is there to delete manifest on local Docker. If you do not do +it and find out that you added wrong image to manifest, it would be impossible +to take that image out! -Pull image -``` -docker pull questdb/questdb:4.0.0 -``` +## Running QuestDB via Docker + +Pull the latest image -To run QuestDB as interactive sandbox: ``` -docker run --rm -it -p 9000:9000 -p 8812:8812 questdb/questdb:4.0.0 +docker pull questdb/questdb ``` -You can Ctrl+C QuestDB. Container and all the data is removed when container stops. -To run QuestDB sensibly, create container without running it: +To run QuestDB interactively: + ``` -docker create --name questdb -p 9000:9000 -p 8812:8812 questdb/questdb:4.0.0 +docker run --rm -it -p 9000:9000 -p 8812:8812 questdb/questdb ``` -Start QuestDB +You can stop this container using Ctrl+C. The container and all the data is +removed when stopped. A practical process for running QuestDB is to create a +container with a name: + ``` -docker start questdb +docker create --name questdb -p 9000:9000 -p 8812:8812 questdb/questdb ``` -Stop QuestDB +The container named `questdb` can be started, stopped and the logs can be viewed +using the following commands: + ``` +docker start questdb docker stop questdb -``` -Check logs -``` docker logs questdb ``` QuestDB supports the following volumes: -`/root/.questdb/db` for Linux containers -`c:\questdb\db` for Windows containers +`/root/.questdb/db` for Linux containers `c:\questdb\db` for Windows containers You can mount host directories using -v option, e.g. + ``` -v /local/dir:/root/.questdb/db ```