Overview

Data deduplication

This release includes the highly anticipated data deduplication logic, which is now supported across all ingestion protocols. Its primary objective is to simplify the process of restarting data streams. If a data stream is interrupted, clients can easily resolve the issue by re-publishing an arbitrarily long window of data. The server will then handle the task of retaining only the missing data.

Data deduplication functions as an upsert operation by preserving the newer version of a record when duplicates are encountered.

To control data deduplication the following SQL syntax is available:

-- deduplicate on 2 columns
CREATE TABLE no_ts_dups_table (timestamp ts, l long, symbol sym, uuid id, string s) timestamps(ts) 
PARTITION BY DAY WAL 
DEDUPLICATE UPSERT KEYS(ts, id)

-- deduplication on 3 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, id, l);

-- deduplication on 4 columns
ALTER TABLE no_ts_dups_table DEDUP UPSERT KEYS(ts, sym, id, l);

We are thrilled to assure you that data deduplication has a negligible impact on ingestion performance, allowing it to be enabled at all times.

New IPV4 data type

This release introduces a new data type called ipv4. The primary objective behind this addition is to assist users working with IPv4 addresses in alleviating storage constraints and enhancing the overall SQL execution performance. Please see this the original pull request for more details.

What is new

CORE

• data deduplication by @ideoma in https://github.com/questdb/questdb/pull/3524, https://github.com/questdb/questdb/pull/3566
• add ipv4 as supported column type by @allegraharris in https://github.com/questdb/questdb/pull/3523
• enable prepending jvm flags from docker run by @yifanstefanwang in https://github.com/questdb/questdb/pull/3567
• make wal apply more fair in case of skewed transaction frequences by @ideoma in https://github.com/questdb/questdb/pull/3592

SQL

• implement function to convert two longs to UUID by @jerrinot in https://github.com/questdb/questdb/pull/3559

What is fixed

CORE

• fix wal to non-wal table conversion leaving table non-writable by @ideoma in https://github.com/questdb/questdb/pull/3572
• fix physical rows written metric on partition split by @ideoma in https://github.com/questdb/questdb/pull/3577
• fix vacuum command deleting live table columns by @ideoma in https://github.com/questdb/questdb/pull/3589

SQL

• fix exception updating symbol column with char value by @ideoma in https://github.com/questdb/questdb/pull/3578
• unexpected NullPointerException on COUNT(SELECT ...) query by @puzpuzpuz in https://github.com/questdb/questdb/pull/3565
• fix sql bugs by @bziobrowski in https://github.com/questdb/questdb/pull/3479
• fix data loss causes by concurrent column drop transactions by @bluestreak01 in https://github.com/questdb/questdb/pull/3583
• fix race condition in ILP authentication by @jerrinot in https://github.com/questdb/questdb/pull/3599
• fix Invalid Column error with multiple JOIN by @bziobrowski in https://github.com/questdb/questdb/pull/3604
• limit reordering of left join-ed tables by @bziobrowski in https://github.com/questdb/questdb/pull/3606
• fix string and char comparison by @bziobrowski in https://github.com/questdb/questdb/pull/3607

UI

• fix incorrect rendering of last few columns in Web Console grid

New Contributors

• @auula made their first contribution in https://github.com/questdb/questdb/pull/3593
• @yifanstefanwang made their first contribution in https://github.com/questdb/questdb/pull/3567
• @allegraharris made their first contribution in https://github.com/questdb/questdb/pull/3523

Full Changelog: https://github.com/questdb/questdb/compare/7.2.1...7.3