@@ -33,42 +33,41 @@ For the SQL INSERT Grammar, please refer to [Taos SQL insert](https://www.taosd
## <a class="anchor" id="sql"></a> Data Writing via Schemaless
**Introduction**
<br/> In many IoT applications, data collection are often used in intelligent control, business analysis and device monitoring etc. As fast application upgrade and iteration, or hardware adjustment, data collection metrics can change rapidly over time. To provide solutions to such use cases, from version 2.2.0.0, TDengine supports writting data via Schemaless. When using Schemaless, action of pre-creating table before inserting data is no longer needed anymore. Tables, data columns and tags can be created automatically. Schemaless can also add additonal data columns to tables if necessary, to make sure data can be properly storaged into TDengine.
<br/> In many IoT applications, data collection is often used in intelligent control, business analysis and device monitoring etc. As fast application upgrade and iteration, or hardware adjustment, data collection metrics can change rapidly over time. To provide solutions to such use cases, from version 2.2.0.0, TDengine supports writing data via Schemaless. When using Schemaless, action of pre-creating table before inserting data is no longer needed anymore. Tables, data columns and tags can be created automatically. Schemaless can also add additional data columns to tables if necessary, to make sure data can be properly stored into TDengine.
<br/> TDengine C/C++ Connector provides Schemaless API. Please see [Schemaless data writting API](https://www.taosdata.com/en/documentation/connector#schemaless) for detailed data writting format.
<br/> Super table and corresponding child talbes created via Schemaless are identical to the ones created via SQL, so inserting data into these tables via SQL is also supported. Note that child table names are generated via schemaless are following special rules through tags mapping. Therefore, child table names are usually not meaningful with respect to readability.
<br/> TDengine C/C++ Connector provides Schemaless API. Please see [Schemaless data writting API](https://www.taosdata.com/en/documentation/connector#schemaless) for detailed data writing format.
<br/> Super table and corresponding child tables created via Schemaless are identical to the ones created via SQL, so inserting data into these tables via SQL is also supported. Note that child table names are generated via Schemaless are following special rules through tags mapping. Therefore, child table names are usually not meaningful in terms of readability.
**Schemaless writting protocol**
<br/>TDengine schemaless writting protocol is compatible with InfluxDB's Line Protocol, OpenTSDB's telnet and JSON format protocols. Users need to specify which protocol to use as parameter when writting data using Schemaless API.
**Schemaless writing protocols**
<br/>TDengine Schemaless writing protocol is compatible with InfluxDB's Line Protocol, OpenTSDB's telnet and JSON format protocols. Users need to specify which protocol to use as parameter when writing data using Schemaless API!
For InfluxDB, OpenTSDB data writting protocol format, users can refer to corresponding official documentation for details. Following will give examples of introducing protocol extension from TDengine based on InfluxDB's Line Protocol, allowing users to use Schemaless with more precision.
For InfluxDB, OpenTSDB data writing protocol format, users can refer to corresponding official documentation for details. Following will give examples of introducing protocol extension from TDengine based on InfluxDB's Line Protocol, allowing users to use Schemaless with more precision.
Schemaless use one line of string literals to represent one data record. (Users can also pass multiple lines to the Schemaless API for batch insertion), the format is as follows:
```json
measurement,tag_setfield_settimestamp
```
* measurement is used as the table name。Comma delimeter is used to seperate measurement and tag_set.
* tag_set represent tag data in key-value pairs. The format is: `<tag_key>=<tag_value>,<tag_key>=<tag_value>`. Comma delimeter is used to seperate multiple tag key-value pairs。Space delimiter is used to seperate tag_set and field_set.
* field_set represent column data in key-value pairs. The format is similar to tag_set: `<field_key>=<field_value>,<field_key>=<field_value>`，Comma delimeter is used to seperate multiple tag key-value pairs。Space delimiter is used to seperate field set and timestamp.
*timestamp is the primary key of one data row.
* measurement is used as the table name。Comma delimiter is used to separate measurement and tag_set.
* tag_set represent tag data in key-value pairs. The format is: `<tag_key>=<tag_value>,<tag_key>=<tag_value>`. Comma delimiter is used to separate multiple tag key-value pairs. Space delimiter is used to separate tag_set and field_set.
* field_set represent column data in key-value pairs. The format is similar to tag_set: `<field_key>=<field_value>,<field_key>=<field_value>`，Comma delimiter is used to separate multiple tag key-value pairs。Space delimiter is used to separate field set and timestamp.
*Timestamp is the primary key of one data row.
All tag values in tag_set are automatically converted and stored as NCHAR data type in TDengine and no need to be surrounded by double quote("）
<br/> In Schemaless Line Protocol, data format in field_set need to be self-descriptive in order to convert data to different TDengine data types. For example:
* Field value surrounded by double quote indicate data is BINARY(32) data types. For example `"abc"`.
* Field value surrounded by double quote indicate data is BINARY(32) data types. For example,`"abc"`.
* Field value surrounded by double quote and L letter prefix indicate data is NCHAR(32) data type. For example `L"报错信息"`.
* Space, equal sign(=), comma(,), double quote(") need to use backslash(\) to escape.
* Numerical values will convert to different data types according to its suffix:
tag_key1, tag_key2 are not following the orginal order of user input, but sorted according to tag names.
tag_key1, tag_key2 are not following the original order of user input, but sorted according to tag names.
After MD5 value "md5_val" calculated using the above string, prefix "t_" is added to "md5_val" to form the child table name.
<br/>2. If super table does not exist, a new super table will be created.
<br/>3. If child table does not exist, a new child table will be created with its name generated in 1 and 2.
<br/>4. If columns/tags does not exist, new columns/tags will be created.(Columns/tags can only be added, existing columns/tags can not be deleted with Schemaless)
<br/>5. If columns/tags are not specified in a line, values of such columns/tags will be set to NULL.
<br/>6. For BINARY/NCHAR type columns. If value length exceeds max length of the column, max lengh will be automatically extented to make sure data integrity.
<br/>4. If columns/tags do not exist, new columns/tags will be created. (Columns/tags can only be added, existing columns/tags cannot be deleted with Schemaless)
<br/>5. If columns/tags are not specified in a line, values of such columns/tags will be set to NULL.
<br/>6. For BINARY/NCHAR type columns. If value length exceeds max length of the column, max length will be automatically extended to make sure data integrity.
<br/>7. If child table is already created and tag value is different than previous stored value，old value will be overwritten by new value
<br/>8. If any error occurs during processing, error code will be return.
**Note：**
<br/>Schemaless will follow TDengine data structure limitations. For example each table row cannot exceeds 16KB. For detailed TDengine limitations please refer to (https://www.taosdata.com/en/documentation/taos-sql#limitation).
**Note**
<br/>Schemaless will follow TDengine data structure limitations. For example, each table row cannot exceed 16KB. For detailed TDengine limitations please refer to (https://www.taosdata.com/en/documentation/taos-sql#limitation).
**Timestamp precisions**
<br/>Following protocols are supported in Schemaless:
...
...
@@ -116,7 +115,7 @@ After MD5 value "md5_val" calculated using the above string, prefix "t_" is adde
When SML_TELNET_PROTOCOL or SML_JSON_PROTOCOL used，timestamp precision is determined by how many digits used in timestamp（following OpenTSDB convention），precision from user input will be ignored。
**Schemaless data mapping rules**
<br/>This section describes how schemaless data are mapped to TDengine structured data。Measurement is mapped to super table name。keys in tag_set/field_set are mapped to tag/column names。For example:
<br/>This section describes how Schemaless data are mapped to TDengine structured data。Measurement is mapped to super table name。keys in tag_set/field_set are mapped to tag/column names。For example:
For first line of data, c4 column type is declared as DOUBLE with no suffix. However, the second line declared the column type to be BIGINT with suffix "i". Schemaless parsing error will be occured.
For first line of data, c4 column type is declared as DOUBLE with no suffix. However, the second line declared the column type to be BIGINT with suffix "i". Schemaless parsing error will be occurred.
When data column is declared as BINARY type, but follow up data inserting requires longer BINARY length for this column, super table schema will be changed accordingly:
In above example in second line has one more column c6 with value "passit", compared to the first line. A new column c6 will be added with type BINARY(6).
**Data integrity**
<br/>TDengine ensure data wrtting through schemaless is idempotent, which means users can call the API multiple times for writting data with errors. However. atomicity is not guaranteed. When writting multiple lines of data as a batch, data might be partially inserted due to errors.
<br/>TDengine ensure data writing through Schemaless is idempotent, which means users can call the API multiple times for writing data with errors. However. atomicity is not guaranteed. When writing multiple lines of data as a batch, data might be partially inserted due to errors.
**Error code**
<br/>If users does not writting data according to protocol syntax, application will get TSDB_CODE_TSC_LINE_SYNTAX_ERROR error code, which indicates error is happened in text. Other generic error codes returned by TDengine can also be obtained through taos_errstr API to get detailed error message.
<br/>If users do not write data following corresponding protocol syntax, application will get TSDB_CODE_TSC_LINE_SYNTAX_ERROR error code, which indicates error is happened in text. Other generic error codes returned by TDengine can also be obtained through taos_errstr API to get detailed error message.
**Future enhancement**
<br/> Currently TDengine only provides clang API for Schemaless. In future versions, API/connectors with more language will be supported, e.g. Java/Go/Python/C# etc. From TDengine v2.3 and later versions, users can also use taosAdaptor to wrtting data via schemaless through REST interface.
<br/> Currently TDengine only provides clang API for Schemaless. In future versions, API/connectors with more language will be supported, e.g., Java/Go/Python/C# etc. From TDengine v2.3 and later versions, users can also use taosAdaptor to writing data via Schemaless through REST interface.
## <a class="anchor" id="prometheus"></a> Data Writing via Prometheus
