Array of T-type items. The T type can be any type, including an array.
We don't recommend using multidimensional arrays, because they are not well supported (for example, you can't store multidimensional arrays in tables with engines from MergeTree family).
Outputs data in JSON format. Besides data tables, it also outputs column names and types, along with some additional information - the total number of output rows, and the number of rows that could have been output if there weren't a LIMIT. Example:
@@ -35,10 +35,12 @@ The TabSeparated format is convenient for processing data using custom programs
The TabSeparated format supports outputting total values (when using WITH TOTALS) and extreme values (when 'extremes' is set to 1). In these cases, the total values and extremes are output after the main data. The main result, total values, and extremes are separated from each other by an empty line. Example:
``SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT TabSeparated``
.. code-block:: sql
SELECT EventDate, count() AS c FROM test.hits GROUP BY EventDate WITH TOTALS ORDER BY EventDate FORMAT TabSeparated
Similar to TabSeparated, but displays data in name=value format. Names are displayed just as in TabSeparated. Additionally, a ``=`` symbol is displayed.
For all arithmetic functions, the result type is calculated as the smallest number type that the result fits in, if there is such a type. The minimum is taken simultaneously based on the number of bits, whether it is signed, and whether it floats. If there are not enough bits, the highest bit type is taken.
@@ -4,16 +4,20 @@ Functions for working with dates and times
Time Zone Support
All functions for working with the date and time for which this makes sense, can take a second, optional argument - the time zone name. Example: Asia / Yekaterinburg. In this case, they do not use the local time zone (the default), but the specified one.
Only time zones are supported, different from UTC for an integer number of hours.
toYear
...
...
@@ -137,5 +141,5 @@ This function is specific to Yandex.Metrica, since half an hour is the minimum a
timeSlots(StartTime, Duration)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For a time interval starting at 'StartTime' and continuing for 'Duration' seconds, it returns an array of moments in time, consisting of points from this interval rounded down to the half hour.
For example, timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600)) = [toDateTime('2012-01-01 12:00:00'), toDateTime('2012-01-01 12:30:00')].
This is necessary for searching for pageviews in the corresponding session.
For example, ``timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600)) = [toDateTime('2012-01-01 12:00:00'), toDateTime('2012-01-01 12:30:00')]``.
This is necessary for searching for pageviews in the corresponding session.
- Gets the value of the 'attr_name' attribute from the 'dict_name' dictionary by the 'id' key.
Gets the value of the 'attr_name' attribute from the 'dict_name' dictionary by the 'id' key.
'dict_name' and 'attr_name' are constant strings.
'id' must be UInt64.
If the 'id' key is not in the dictionary, it returns the default value set in the dictionary definition.
...
...
@@ -30,14 +30,14 @@ Similar to the functions dictGetT, but the default value is taken from the last
dictIsIn
~~~~~~~~
``dictIsIn('dict_name', child_id, ancestor_id)``
- For the 'dict_name' hierarchical dictionary, finds out whether the 'child_id' key is located inside 'ancestor_id' (or matches 'ancestor_id'). Returns UInt8.
For the 'dict_name' hierarchical dictionary, finds out whether the 'child_id' key is located inside 'ancestor_id' (or matches 'ancestor_id'). Returns UInt8.
dictGetHierarchy
~~~~~~~~~~~~~~~~
``dictGetHierarchy('dict_name', id)``
- For the 'dict_name' hierarchical dictionary, returns an array of dictionary keys starting from 'id' and continuing along the chain of parent elements. Returns Array(UInt64).
For the 'dict_name' hierarchical dictionary, returns an array of dictionary keys starting from 'id' and continuing along the chain of parent elements. Returns Array(UInt64).
dictHas
~~~~~~~
``dictHas('dict_name', id)``
- check the presence of a key in the dictionary. Returns a value of type UInt8, equal to 0, if there is no key and 1 if there is a key.
check the presence of a key in the dictionary. Returns a value of type UInt8, equal to 0, if there is no key and 1 if there is a key.
@@ -45,14 +45,16 @@ Accepts a numeric argument and returns a Float64 number close to the cubic root
erf(x)
~~~~~~
If 'x' is non-negative, then erf(x / σ√2) - is the probability that a random variable having a normal distribution with standard deviation 'σ' takes the value that is separated from the expected value by more than 'x'.
If 'x' is non-negative, then ``erf(x / σ√2)`` - is the probability that a random variable having a normal distribution with standard deviation 'σ' takes the value that is separated from the expected value by more than 'x'.
All these functions don't follow the RFC. They are maximally simplified for improved performance.
Функции, извлекающие часть URL-а.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Functions tat extract part of the URL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If there isn't anything similar in a URL, an empty string is returned.
protocol
""""""""
- Selects the protocol. Examples: http, ftp, mailto, magnet...
Selects the protocol. Examples: http, ftp, mailto, magnet...
domain
""""""
- Selects the domain.
Selects the domain.
domainWithoutWWW
""""""""""""""""
- Selects the domain and removes no more than one 'www.' from the beginning of it, if present.
Selects the domain and removes no more than one 'www.' from the beginning of it, if present.
topLevelDomain
""""""""""""""
- Selects the top-level domain. Example: .ru.
Selects the top-level domain. Example: .ru.
firstSignificantSubdomain
"""""""""""""""""""""""""
- Selects the "first significant subdomain". This is a non-standard concept specific to Yandex.Metrica. The first significant subdomain is a second-level domain if it is 'com', 'net', 'org', or 'co'. Otherwise, it is a third-level domain. For example, firstSignificantSubdomain('https://news.yandex.ru/') = 'yandex', firstSignificantSubdomain('https://news.yandex.com.tr/') = 'yandex'. The list of "insignificant" second-level domains and other implementation details may change in the future.
Selects the "first significant subdomain". This is a non-standard concept specific to Yandex.Metrica. The first significant subdomain is a second-level domain if it is 'com', 'net', 'org', or 'co'. Otherwise, it is a third-level domain. For example, firstSignificantSubdomain('https://news.yandex.ru/') = 'yandex', firstSignificantSubdomain('https://news.yandex.com.tr/') = 'yandex'. The list of "insignificant" second-level domains and other implementation details may change in the future.
cutToFirstSignificantSubdomain
""""""""""""""""""""""""""""""
- Selects the part of the domain that includes top-level subdomains up to the "first significant subdomain" (see the explanation above).
Selects the part of the domain that includes top-level subdomains up to the "first significant subdomain" (see the explanation above).
For example, ``cutToFirstSignificantSubdomain('https://news.yandex.com.tr/') = 'yandex.com.tr'``.
path
""""
- Selects the path. Example: /top/news.html The path does not include the query-string.
Selects the path. Example: /top/news.html The path does not include the query-string.
pathFull
""""""""
- The same as above, but including query-string and fragment. Example: /top/news.html?page=2#comments
The same as above, but including query-string and fragment. Example: /top/news.html?page=2#comments
queryString
"""""""""""
- Selects the query-string. Example: page=1&lr=213. query-string does not include the first question mark, or # and everything that comes after #.
Selects the query-string. Example: page=1&lr=213. query-string does not include the first question mark, or # and everything that comes after #.
fragment
""""""""
- Selects the fragment identifier. fragment does not include the first number sign (#).
Selects the fragment identifier. fragment does not include the first number sign (#).
queryStringAndFragment
""""""""""""""""""""""
- Selects the query-string and fragment identifier. Example: page=1#29390.
Selects the query-string and fragment identifier. Example: page=1#29390.
extractURLParameter(URL, name)
""""""""""""""""""""""""""""""
- Selects the value of the 'name' parameter in the URL, if present. Otherwise, selects an empty string. If there are many parameters with this name, it returns the first occurrence. This function works under the assumption that the parameter name is encoded in the URL in exactly the same way as in the argument passed.
Selects the value of the 'name' parameter in the URL, if present. Otherwise, selects an empty string. If there are many parameters with this name, it returns the first occurrence. This function works under the assumption that the parameter name is encoded in the URL in exactly the same way as in the argument passed.
extractURLParameters(URL)
"""""""""""""""""""""""""
- Gets an array of name=value strings corresponding to the URL parameters. The values are not decoded in any way.
Gets an array of name=value strings corresponding to the URL parameters. The values are not decoded in any way.
extractURLParameterNames(URL)
"""""""""""""""""""""""""""""
- Gets an array of name=value strings corresponding to the names of URL parameters. The values are not decoded in any way.
Gets an array of name=value strings corresponding to the names of URL parameters. The values are not decoded in any way.
URLHierarchy(URL)
"""""""""""""""""
- Gets an array containing the URL trimmed to the ``/``, ``?`` characters in the path and query-string. Consecutive separator characters are counted as one. The cut is made in the position after all the consecutive separator characters. Example:
Gets an array containing the URL trimmed to the ``/``, ``?`` characters in the path and query-string. Consecutive separator characters are counted as one. The cut is made in the position after all the consecutive separator characters. Example:
URLPathHierarchy(URL)
"""""""""""""""""""""
- The same thing, but without the protocol and host in the result. The / element (root) is not included. Example:
The same thing, but without the protocol and host in the result. The / element (root) is not included. Example:
This function is used for implementing tree-view reports by URL in Yandex.Metrica.
In this case, the log will be printed to the console, which is convenient during development. If the configuration file is in the current directory, you don't need to specify the '--config-file' parameter. By default, it uses './config.xml'.
You can use the command-line client to connect to the server:
::
.. code-block:: bash
clickhouse-client
The default parameters indicate connecting with localhost:9000 on behalf of the user 'default' without a password.
The client can be used for connecting to a remote server. For example:
::
.. code-block:: bash
clickhouse-client --host=example.com
For more information, see the section "Command-line client".
To work for command line you can use ``clickhouse-client``:
::
.. code-block:: bash
$ clickhouse-client
ClickHouse client version 0.0.26176.
Connecting to localhost:9000.
...
...
@@ -37,9 +38,12 @@ Only works in non-interactive mode.
``--stacktrace`` - If specified, also prints the stack trace if an exception occurs.
``--config-file`` - Name of the configuration file that has additional settings or changed defaults for the settings listed above.
By default, files are searched for in this order:
./clickhouse-client.xml
~/./clickhouse-client/config.xml
/etc/clickhouse-client/config.xml
..
./clickhouse-client.xml
~/./clickhouse-client/config.xml
/etc/clickhouse-client/config.xml
Settings are only taken from the first file found.
You can also specify any settings that will be used for processing queries. For example, ``clickhouse-client --max_threads=1``. For more information, see the section "Settings".
...
...
@@ -49,7 +53,8 @@ To use batch mode, specify the 'query' parameter, or send data to 'stdin' (it ve
Similar to the HTTP interface, when using the 'query' parameter and sending data to 'stdin', the request is a concatenation of the 'query' parameter, a line break, and the data in 'stdin'. This is convenient for large INSERT queries.
Examples for insert data via clickhouse-client:
::
.. code-block:: bash
echo -ne "1, 'some text', '2016-08-14 00:00:00'\n2, 'some more text', '2016-08-14 00:00:01'" | clickhouse-client --database=test --query="INSERT INTO test FORMAT CSV";
cat <<_EOF | clickhouse-client --database=test --query="INSERT INTO test FORMAT CSV";
Uses asynchronous multimaster replication. After being written to any available replica, data is distributed to all the remaining replicas. The system maintains identical data on different replicas. Data is restored automatically after a failure, or using a "button" for complex cases.
Uses asynchronous multi-master replication. After being written to any available replica, data is distributed to all the remaining replicas. The system maintains identical data on different replicas. Data is restored automatically after a failure, or using a "button" for complex cases.
For more information, see the section "Data replication".
In other words, all the values related to a row are stored next to each other. Examples of a row-oriented DBMS are MySQL, Postgres, MS SQL Server, and others.
In a column-oriented DBMS, data is stored like this:
Title: Yandex Announcements - Investor Relations - Yandex Yandex — Contact us — Moscow Yandex — Mission Ru Yandex — History — History of Yandex Yandex Financial Releases - Investor Relations - Yandex Yandex — Locations Yandex Board of Directors - Corporate Governance - Yandex Yandex — Technologies
...
...
@@ -56,7 +59,8 @@ Columnar-oriented databases are better suited to OLAP scenarios (at least 100 ti
For example, the query "count the number of records for each advertising platform" requires reading one "advertising platform ID" column, which takes up 1 byte uncompressed. If most of the traffic was not from advertising platforms, you can expect at least 10-fold compression of this column. When using a quick compression algorithm, data decompression is possible at a speed of at least several gigabytes of uncompressed data per second. In other words, this query can be processed at a speed of approximately several billion rows per second on a single server. This speed is actually achieved in practice.
@@ -13,7 +14,8 @@ If "IF NOT EXISTS" is included, the query won't return an error if the database
CREATE TABLE
~~~~~~~~~~~~
The ``CREATE TABLE`` query can have several forms.
::
.. code-block:: sql
CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db.]name
(
name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1],
...
...
@@ -25,11 +27,13 @@ Creates a table named 'name' in the 'db' database or the current database if 'db
A column description is ``name type`` in the simplest case. For example: ``RegionID UInt32``.
Expressions can also be defined for default values (see below).
::
.. code-block:: sql
CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db.]name AS [db2.]name2 [ENGINE = engine]
Creates a table with the same structure as another table. You can specify a different engine for the table. If the engine is not specified, the same engine will be used as for the 'db2.name2' table.
::
.. code-block:: sql
CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db.]name ENGINE = engine AS SELECT ...
Creates a table with a structure like the result of the ``SELECT`` query, with the 'engine' engine, and fills it with data from SELECT.
...
...
@@ -92,14 +96,17 @@ Creates a view. There are two types of views: normal and MATERIALIZED.
Normal views don't store any data, but just perform a read from another table. In other words, a normal view is nothing more than a saved query. When reading from a view, this saved query is used as a subquery in the FROM clause.
As an example, assume you've created a view:
::
.. code-block:: sql
CREATE VIEW view AS SELECT ...
and written a query:
::
.. code-block:: sql
SELECT a, b, c FROM view
This query is fully equivalent to using the subquery:
::
.. code-block:: sql
SELECT a, b, c FROM (SELECT ...)
Materialized views store data transformed by the corresponding SELECT query.
...
...
@@ -130,21 +137,24 @@ This query is used when starting the server. The server stores table metadata as
DROP
~~~~
This query has two types: ``DROP DATABASE`` and ``DROP TABLE``.
::
.. code-block:: sql
DROP DATABASE [IF EXISTS] db
Deletes all tables inside the 'db' database, then deletes the 'db' database itself.
If IF EXISTS is specified, it doesn't return an error if the database doesn't exist.
::
.. code-block:: sql
DROP TABLE [IF EXISTS] [db.]name
Deletes the table.
If IF EXISTS is specified, it doesn't return an error if the table doesn't exist or the database doesn't exist.
If ``IF EXISTS`` is specified, it doesn't return an error if the table doesn't exist or the database doesn't exist.
DETACH
~~~~~~
Deletes information about the table from the server. The server stops knowing about the table's existence.
::
.. code-block:: sql
DETACH TABLE [IF EXISTS] [db.]name
This does not delete the table's data or metadata. On the next server launch, the server will read the metadata and find out about the table again. Similarly, a "detached" table can be re-attached using the ATTACH query (with the exception of system tables, which do not have metadata stored for them).
...
...
@@ -154,7 +164,8 @@ There is no DETACH DATABASE query.
RENAME
~~~~~~
Renames one or more tables.
::
.. code-block:: sql
RENAME TABLE [db11.]name11 TO [db12.]name12, [db21.]name21 TO [db22.]name22, ...
All tables are renamed under global locking. Renaming tables is a light operation. If you indicated another database after TO, the table will be moved to this database. However, the directories with databases must reside in the same file system (otherwise, an error is returned).
...
...
@@ -166,13 +177,15 @@ The ALTER query is only supported for *MergeTree type tables, as well as for Mer
Column manipulations
""""""""""""""""""""
Lets you change the table structure.
::
.. code-block:: sql
ALTER TABLE [db].name ADD|DROP|MODIFY COLUMN ...
In the query, specify a list of one or more comma-separated actions. Each action is an operation on a column.
The following actions are supported:
::
.. code-block:: sql
ADD COLUMN name [type] [default_expr] [AFTER name_after]
Adds a new column to the table with the specified name, type, and default expression (see the section "Default expressions"). If you specify 'AFTER name_after' (the name of another column), the column is added after the specified one in the list of table columns. Otherwise, the column is added to the end of the table. Note that there is no way to add a column to the beginning of a table. For a chain of actions, 'name_after' can be the name of a column that is added in one of the previous actions.
...
...
@@ -248,7 +261,8 @@ Another way to view a set of parts and partitions is to go into the directory wi
The directory with data is
/var/lib/clickhouse/data/database/table/,
where /var/lib/clickhouse/ is the path to ClickHouse data, 'database' is the database name, and 'table' is the table name. Example:
::
.. code-block:: bash
$ ls -l /var/lib/clickhouse/data/test/visits/
total 48
drwxrwxrwx 2 clickhouse clickhouse 20480 мая 13 02:58 20140317_20140323_2_2_0
...
...
@@ -271,8 +285,9 @@ Each part corresponds to a single partition and contains data for a single month
On an operating server, you can't manually change the set of parts or their data on the file system, since the server won't know about it. For non-replicated tables, you can do this when the server is stopped, but we don't recommended it. For replicated tables, the set of parts can't be changed in any case.
The 'detached' directory contains parts that are not used by the server - detached from the table using the ALTER ... DETACH query. Parts that are damaged are also moved to this directory, instead of deleting them. You can add, delete, or modify the data in the 'detached' directory at any time - the server won't know about this until you make the ALTER TABLE ... ATTACH query.
::
ALTER TABLE [db.]table DETACH PARTITION 'name'
.. code-block:: sql
ALTER TABLE [db.]table DETACH PARTITION 'name'
Move all data for partitions named 'name' to the 'detached' directory and forget about them.
The partition name is specified in YYYYMM format. It can be indicated in single quotes or without them.
...
...
@@ -280,11 +295,13 @@ The partition name is specified in YYYYMM format. It can be indicated in single
After the query is executed, you can do whatever you want with the data in the 'detached' directory — delete it from the file system, or just leave it.
The query is replicated - data will be moved to the 'detached' directory and forgotten on all replicas. The query can only be sent to a leader replica. To find out if a replica is a leader, perform SELECT to the 'system.replicas' system table. Alternatively, it is easier to make a query on all replicas, and all except one will throw an exception.
::
.. code-block:: sql
ALTER TABLE [db.]table DROP PARTITION 'name'
Similar to the DETACH operation. Deletes data from the table. Data parts will be tagged as inactive and will be completely deleted in approximately 10 minutes. The query is replicated - data will be deleted on all replicas.
::
.. code-block:: sql
ALTER TABLE [db.]table ATTACH PARTITION|PART 'name'
Adds data to the table from the 'detached' directory.
...
...
@@ -294,7 +311,8 @@ It is possible to add data for an entire partition or a separate part. For a par
The query is replicated. Each replica checks whether there is data in the 'detached' directory. If there is data, it checks the integrity, verifies that it matches the data on the server that initiated the query, and then adds it if everything is correct. If not, it downloads data from the query requestor replica, or from another replica where the data has already been added.
So you can put data in the 'detached' directory on one replica, and use the ALTER ... ATTACH query to add it to the table on all replicas.
::
.. code-block:: sql
ALTER TABLE [db.]table FREEZE PARTITION 'name'
Creates a local backup of one or multiple partitions. The name can be the full name of the partition (for example, 201403), or its prefix (for example, 2014) - then the backup will be created for all the corresponding partitions.
...
...
@@ -334,7 +352,8 @@ Replication provides protection from device failures. If all data disappeared on
For protection from device failures, you must use replication. For more information about replication, see the section "Data replication".
Backups protect against human error (accidentally deleting data, deleting the wrong data or in the wrong cluster, or corrupting data). For high-volume databases, it can be difficult to copy backups to remote servers. In such cases, to protect from human error, you can keep a backup on the same server (it will reside in /var/lib/clickhouse/shadow/).
::
.. code-block:: sql
ALTER TABLE [db.]table FETCH PARTITION 'name' FROM 'path-in-zookeeper'
This query only works for replicatable tables.
...
...
@@ -623,7 +642,7 @@ Allows executing JOIN with an array or nested data structure. The intent is simi
ARRAY JOIN is essentially INNER JOIN with an array. Example:
@@ -684,6 +703,8 @@ An alias can be specified for an array in the ARRAY JOIN clause. In this case, a
FROM arrays_test
ARRAY JOIN arr AS a
..
┌─s─────┬─arr─────┬─a─┐
│ Hello │ [1,2] │ 1 │
│ Hello │ [1,2] │ 2 │
...
...
@@ -697,7 +718,7 @@ An alias can be specified for an array in the ARRAY JOIN clause. In this case, a
Multiple arrays of the same size can be comma-separated in the ARRAY JOIN clause. In this case, JOIN is performed with them simultaneously (the direct sum, not the direct product).
Example:
.. code-block:: sql
..
:) SELECT s, arr, a, num, mapped FROM arrays_test ARRAY JOIN arr AS a, arrayEnumerate(arr) AS num, arrayMap(x -> x + 1, arr) AS mapped
...
...
@@ -733,7 +754,7 @@ Example:
ARRAY JOIN also works with nested data structures. Example:
.. code-block:: sql
..
:) CREATE TABLE nested_test (s String, nest Nested(x UInt8, y UInt32)) ENGINE = Memory
...
...
@@ -788,7 +809,7 @@ ARRAY JOIN also works with nested data structures. Example:
When specifying names of nested data structures in ARRAY JOIN, the meaning is the same as ARRAY JOIN with all the array elements that it consists of. Example:
There are two types of parsers in the system: a full SQL parser (a recursive descent parser), and a data format parser (a fast stream parser). In all cases except the INSERT query, only the full SQL parser is used.
The INSERT query uses both parsers:
::
.. code-block:: sql
INSERT INTO t VALUES (1, 'Hello, world'), (2, 'abc'), (3, 'def')
The ``INSERT INTO t VALUES`` fragment is parsed by the full parser, and the data ``(1, 'Hello, world'), (2, 'abc'), (3, 'def')`` is parsed by the fast stream parser.
...
...
@@ -82,7 +83,8 @@ Data types and table engines in the ``CREATE`` query are written the same way as
Synonyms
~~~~~~~~
In the SELECT query, expressions can specify synonyms using the AS keyword. Any expression is placed to the left of AS. The identifier name for the synonym is placed to the right of AS. As opposed to standard SQL, synonyms are not only declared on the top level of expressions:
::
.. code-block:: sql
SELECT (1 AS n) + 2, n
In contrast to standard SQL, synonyms can be used in all parts of a query, not just ``SELECT``.
@@ -26,7 +26,7 @@ The maximum number of query processing threads
This parameter applies to threads that perform the same stages of the query execution pipeline in parallel.
For example, if reading from a table, evaluating expressions with functions, filtering with WHERE and pre-aggregating for GROUP BY can all be done in parallel using at least ``max_threads`` number of threads, then 'max_threads' are used.
By default, ``8``.
By default, 8.
If less than one SELECT query is normally run on a server at a time, set this parameter to a value slightly less than the actual number of processor cores.
This system table is used for implementing the ``SHOW PROCESSLIST`` query.
Columns:
::
..
user String - Name of the user who made the request. For distributed query processing, this is the user who helped the requestor server send the query to this server, not the user who made the distributed request on the requestor server.
address String - The IP address the request was made from. The same for distributed processing.
Contains information about settings that are currently in use (i.e. used for executing the query you are using to read from the system.settings table).
Columns:
::
..
name String - Setting name.
value String - Setting value.
changed UInt8 - Whether the setting was explicitly defined in the config or explicitly changed.
Buffers the data to write in RAM, periodically flushing it to another table. During the read operation, data is read from the buffer and the other table simultaneously.
@@ -20,7 +21,8 @@ During the write operation, data is inserted to a 'num_layers' number of random
The conditions for flushing the data are calculated separately for each of the 'num_layers' buffers. For example, if num_layers = 16 and max_bytes = 100000000, the maximum RAM consumption is 1.6 GB.
Creating a 'merge.hits_buffer' table with the same structure as 'merge.hits' and using the Buffer engine. When writing to this table, data is buffered in RAM and later written to the 'merge.hits' table. 16 buffers are created. The data in each of them is flushed if either 100 seconds have passed, or one million rows have been written, or 100 MB of data have been written; or if simultaneously 10 seconds have passed and 10,000 rows and 10 MB of data have been written. For example, if just one row has been written, after 100 seconds it will be flushed, no matter what. But if many rows have been written, the data will be flushed sooner.
**The Distributed engine does not store data itself**, but allows distributed query processing on multiple servers.
**The Distributed engine by itself does not store data**, but allows distributed query processing on multiple servers.
Reading is automatically parallelized. During a read, the table indexes on remote servers are used, if there are any.
The Distributed engine accepts parameters: the cluster name in the server's config file, the name of a remote database, the name of a remote table, and (optionally) a sharding key.
Example:
::
..
Distributed(logs, default, hits[, sharding_key])
- Data will be read from all servers in the 'logs' cluster, from the 'default.hits' table located on every server in the cluster.
A prepared data structure for JOIN that is always located in RAM.
::
..
Join(ANY|ALL, LEFT|INNER, k1[, k2, ...])
Engine parameters: ``ANY``|``ALL`` - strictness, and ``LEFT``|``INNER`` - the type. These parameters are set without quotes and must match the JOIN that the table will be used for. k1, k2, ... are the key columns from the USING clause that the join will be made on.
The Merge engine (not to be confused with MergeTree) does not store data itself, but allows reading from any number of other tables simultaneously.
Reading is automatically parallelized. Writing to a table is not supported. When reading, the indexes of tables that are actually being read are used, if they exist.
The Merge engine accepts parameters: the database name and a regular expression for tables. Example:
::
..
Merge(hits, '^WatchLog')
- Data will be read from the tables in the 'hits' database with names that match the regex ``'^WatchLog'``.
...
...
@@ -19,7 +20,7 @@ It is possible to create two Merge tables that will endlessly try to read each o
The typical way to use the Merge engine is for working with a large number of TinyLog tables as if with a single table.
Virtual columns
~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~
Virtual columns are columns that are provided by the table engine, regardless of the table definition. In other words, these columns are not specified in CREATE TABLE, but they are accessible for SELECT.
A MergeTree type table must have a separate column containing the date. In this example, it is the 'EventDate' column. The type of the date column must be 'Date' (not 'DateTime').
As the example shows, these parameters can contain substitutions in curly brackets. The substituted values are taken from the 'macros' section of the config file. Example:
...
...
@@ -145,7 +146,7 @@ An alternative recovery option is to delete information about the lost replica f
There is no restriction on network bandwidth during recovery. Keep this in mind if you are restoring many replicas at once.
Converting from MergeTree to ReplicatedMergeTree
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From here on, we use ``MergeTree`` to refer to all the table engines in the ``MergeTree`` family, including ``ReplicatedMergeTree``.
...
...
@@ -160,7 +161,7 @@ Then run ALTER TABLE ATTACH PART on one of the replicas to add these data parts
If exactly the same parts exist on the other replicas, they are added to the working set on them. If not, the parts are downloaded from the replica that has them.
Converting from ReplicatedMergeTree to MergeTree
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create a MergeTree table with a different name. Move all the data from the directory with the ReplicatedMergeTree table data to the new table's data directory. Then delete the ReplicatedMergeTree table and restart the server.
...
...
@@ -170,6 +171,6 @@ If you want to get rid of a ReplicatedMergeTree table without launching the serv
After this, you can launch the server, create a MergeTree table, move the data to its directory, and then restart the server.
Recovery when metadata in the ZooKeeper cluster is lost or damaged
@@ -23,13 +23,13 @@ Summation is not performed for a read operation. If it is necessary, write the a
In addition, a table can have nested data structures that are processed in a special way.
If the name of a nested table ends in 'Map' and it contains at least two columns that meet the following criteria:
* for the first table, numeric ((U)IntN, Date, DateTime), we'll refer to it as 'key'
* for other tables, arithmetic ((U)IntN, Float32/64), we'll refer to it as '(values...)'
then this nested table is interpreted as a mapping of key => (values...), and when merging its rows, the elements of two data sets are merged by 'key' with a summation of the corresponding (values...).
Then this nested table is interpreted as a mapping of key => (values...), and when merging its rows, the elements of two data sets are merged by 'key' with a summation of the corresponding (values...).
