Unsupported Avro data types: `record` (non-root), `map`
...
...
@@ -1095,7 +1097,7 @@ SELECT * FROM topic1_stream;
## Parquet {#data-format-parquet}
[Apache Parquet](http://parquet.apache.org/) is a columnar storage format widespread in the Hadoop ecosystem. ClickHouse supports read and write operations for this format.
[Apache Parquet](https://parquet.apache.org/) is a columnar storage format widespread in the Hadoop ecosystem. ClickHouse supports read and write operations for this format.
### Data Types Matching {#data_types-matching-2}
...
...
@@ -1141,6 +1143,16 @@ $ clickhouse-client --query="SELECT * FROM {some_table} FORMAT Parquet" > {some_
To exchange data with Hadoop, you can use [HDFS table engine](../engines/table-engines/integrations/hdfs.md).
## Arrow {#data-format-arrow}
[Apache Arrow](https://arrow.apache.org/) comes with two built-in columnar storage formats. ClickHouse supports read and write operations for these formats.
`Arrow` is Apache Arrow's "file mode" format. It is designed for in-memory random access.
## ArrowStream {#data-format-arrow-stream}
`ArrowStream` is Apache Arrow's "stream mode" format. It is designed for in-memory stream processing.
## ORC {#data-format-orc}
[Apache ORC](https://orc.apache.org/) is a columnar storage format widespread in the Hadoop ecosystem. You can only insert data in this format to ClickHouse.
ContextLock Number of times the lock of Context was acquired or tried to acquire. This is global lock.
Query Number of queries to be interpreted and potentially executed. Does not include queries that failed to parse or were rejected due to AST size limits, quota limits or limits on the number of simultaneously running queries. May include internal queries initiated by ClickHouse itself. Does not count subqueries.
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types5 ORDER BY int8 FORMAT Arrow">${CLICKHOUSE_TMP}/arrow_all_types_5.arrow
#${CLICKHOUSE_CLIENT} --query="SELECT * FROM arrow_types5 FORMAT Arrow" | ${CLICKHOUSE_CLIENT} --query="INSERT INTO arrow_types6 FORMAT Arrow"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types5 ORDER BY int8 FORMAT Arrow" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types6 FORMAT Arrow"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types1 ORDER BY int8 FORMAT Arrow" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types6 FORMAT Arrow"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM system.contributors ORDER BY name DESC FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO contributors FORMAT ArrowStream"
# random results
${CLICKHOUSE_CLIENT}--query="SELECT * FROM contributors LIMIT 10"> /dev/null
${CLICKHOUSE_CLIENT}--query="SELECT * FROM system.numbers LIMIT 10000 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_numbers FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_numbers ORDER BY number DESC LIMIT 10"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM system.numbers LIMIT 100000 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_numbers FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_numbers ORDER BY number DESC LIMIT 10"
${CLICKHOUSE_CLIENT}--max_block_size=2 --query="SELECT * FROM system.numbers LIMIT 3 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_numbers FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_numbers ORDER BY number DESC LIMIT 10"
${CLICKHOUSE_CLIENT}--max_block_size=1 --query="SELECT * FROM system.numbers LIMIT 1000 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_numbers FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_numbers ORDER BY number DESC LIMIT 10"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types1 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types2 FORMAT ArrowStream"
echo original:
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types1 ORDER BY int8" | tee${CLICKHOUSE_TMP}/arrow_all_types_1.dump
echo converted:
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types2 ORDER BY int8" | tee${CLICKHOUSE_TMP}/arrow_all_types_2.dump
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types1 ORDER BY int8 FORMAT ArrowStream">${CLICKHOUSE_TMP}/arrow_all_types_1.arrow
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types2 ORDER BY int8 FORMAT ArrowStream">${CLICKHOUSE_TMP}/arrow_all_types_2.arrow
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types3 ORDER BY int8 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types2 FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types1 ORDER BY int8 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types3 FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types4 ORDER BY int8 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types2 FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types1 ORDER BY int8 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types4 FORMAT ArrowStream"
echo dest:
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types2 ORDER BY int8"
echo min:
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types3 ORDER BY int8"
echo max:
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types4 ORDER BY int8"
${CLICKHOUSE_CLIENT}--query="DROP TABLE IF EXISTS arrow_types5"
${CLICKHOUSE_CLIENT}--query="DROP TABLE IF EXISTS arrow_types6"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types5 ORDER BY int8 FORMAT ArrowStream">${CLICKHOUSE_TMP}/arrow_all_types_5.arrow
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types5 ORDER BY int8 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types6 FORMAT ArrowStream"
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types1 ORDER BY int8 FORMAT ArrowStream" | ${CLICKHOUSE_CLIENT}--query="INSERT INTO arrow_types6 FORMAT ArrowStream"
echo dest from null:
${CLICKHOUSE_CLIENT}--query="SELECT * FROM arrow_types6 ORDER BY int8"
