提交 df74c49b 编写于 作者: C Chait Diwadkar 提交者: gccgdb1234

docs:cdiwadkar16-patch-4-16 - several changes

Several changes:
-spelling
-reordering paragraphs for clarity
-rewrote some sentences
-added clarification for UDF being available across databases in an instance i.e. scope is not limited to a database when it is created.
上级 06a54081
--- ---
sidebar_label: UDF sidebar_label: UDF
title: User Defined Functions title: User Defined Functions
description: "Scalar functions and aggregate functions developed by users can be utilized by the query framework to expand the query capability" description: "Scalar functions and aggregate functions developed by users can be utilized by the query framework to expand query capability"
--- ---
In some use cases, the query capability required by application programs can't be achieved directly by builtin functions. With UDF, the functions developed by users can be utilized by query framework to meet some special requirements. UDF normally takes one column of data as input, but can also support the result of sub query as input. In some use cases, built-in functions are not adequate for the query capability required by application programs. With UDF, the functions developed by users can be utilized by the query framework to meet business and application requirements. UDF normally takes one column of data as input, but can also support the result of a sub-query as input.
From version 2.2.0.0, UDF programmed in C/C++ language can be supported by TDengine. From version 2.2.0.0, UDF written in C/C++ are supported by TDengine.
Two kinds of functions can be implemented by UDF: scalar function and aggregate function.
## Define UDF ## Types of UDF
Two kinds of functions can be implemented by UDF: scalar functions and aggregate functions.
Scalar functions return multiple rows and aggregate functions return either 0 or 1 row.
In the case of a scalar function you only have to implement the "normal" function template.
In the case of an aggregate function, in addition to the "normal" function, you also need to implement the "merge" and "finalize" function templates even if the implementation is empty. This will become clear in the sections below.
### Scalar Function ### Scalar Function
Below function template can be used to define your own scalar function. As mentioned earlier, a scalar UDF only has to implement the "normal" function template. The function template below can be used to define your own scalar function.
`void udfNormalFunc(char* data, short itype, short ibytes, int numOfRows, long long* ts, char* dataOutput, char* interBuf, char* tsOutput, int* numOfOutput, short otype, short obytes, SUdfInit* buf)` `void udfNormalFunc(char* data, short itype, short ibytes, int numOfRows, long long* ts, char* dataOutput, char* interBuf, char* tsOutput, int* numOfOutput, short otype, short obytes, SUdfInit* buf)`
`udfNormalFunc` is the place holder of function name, a function implemented based on the above template can be used to perform scalar computation on data rows. The parameters are fixed to control the data exchange between UDF and TDengine. `udfNormalFunc` is the place holder for a function name. A function implemented based on the above template can be used to perform scalar computation on data rows. The parameters are fixed to control the data exchange between UDF and TDengine.
- Definitions of the parameters: - Definitions of the parameters:
...@@ -30,20 +37,24 @@ Below function template can be used to define your own scalar function. ...@@ -30,20 +37,24 @@ Below function template can be used to define your own scalar function.
- numOfRows:the number of rows in the input data - numOfRows:the number of rows in the input data
- ts: the column of timestamp corresponding to the input data - ts: the column of timestamp corresponding to the input data
- dataOutput:the buffer for output data, total size is `oBytes * numberOfRows` - dataOutput:the buffer for output data, total size is `oBytes * numberOfRows`
- interBuf:the buffer for intermediate result, its size is specified by `BUFSIZE` parameter when creating a UDF. It's normally used when the intermediate result is not same as the final result, it's allocated and freed by TDengine. - interBuf:the buffer for an intermediate result. Its size is specified by the `BUFSIZE` parameter when creating a UDF. It's normally used when the intermediate result is not same as the final result. This buffer is allocated and freed by TDengine.
- tsOutput:the column of timestamps corresponding to the output data; it can be used to output timestamp together with the output data if it's not NULL - tsOutput:the column of timestamps corresponding to the output data; it can be used to output timestamp together with the output data if it's not NULL
- numOfOutput:the number of rows in output data - numOfOutput:the number of rows in output data
- buf:for the state exchange between UDF and TDengine - buf:for the state exchange between UDF and TDengine
[add_one.c](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/add_one.c) is one example of the simplest UDF implementations, i.e. one instance of the above `udfNormalFunc` template. It adds one to each value of a column passed in which can be filtered using `where` clause and outputs the result. [add_one.c](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/add_one.c) is one example of a very simple UDF implementation, i.e. one instance of the above `udfNormalFunc` template. It adds one to each value of a passed in column, which can be filtered using the `where` clause, and outputs the result.
### Aggregate Function ### Aggregate Function
Below function template can be used to define your own aggregate function. For aggregate UDF, as mentioned earlier you must implement a "normal" function template (described above) and also implement the "merge" and "finalize" templates.
`void abs_max_merge(char* data, int32_t numOfRows, char* dataOutput, int32_t* numOfOutput, SUdfInit* buf)` #### Merge Function Template
`udfMergeFunc` is the place holder of function name, the function implemented with the above template is used to aggregate the intermediate result, only can be used in the aggregate query for STable. The function template below can be used to define your own merge function for an aggregate UDF.
`void udfMergeFunc(char* data, int32_t numOfRows, char* dataOutput, int32_t* numOfOutput, SUdfInit* buf)`
`udfMergeFunc` is the place holder for a function name. The function implemented with the above template is used to aggregate intermediate results and can only be used in the aggregate query for STable.
Definitions of the parameters: Definitions of the parameters:
...@@ -53,17 +64,11 @@ Definitions of the parameters: ...@@ -53,17 +64,11 @@ Definitions of the parameters:
- numOfOutput:number of rows in the output data - numOfOutput:number of rows in the output data
- buf:for the state exchange between UDF and TDengine - buf:for the state exchange between UDF and TDengine
[abs_max.c](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/abs_max.c) is an user defined aggregate function to get the maximum from the absolute value of a column. #### Finalize Function Template
The internal processing is that the data affected by the select statement will be divided into multiple row blocks and `udfNormalFunc`, i.e. `abs_max` in this case, is performed on each row block to generate the intermediate of each sub table, then `udfMergeFunc`, i.e. `abs_max_merge` in this case, is performed on the intermediate result of sub tables to aggregate to generate the final or intermediate result of STable. The intermediate result of STable is finally processed by `udfFinalizeFunc` to generate the final result, which contain either 0 or 1 row.
Other typical scenarios, like covariance, can also be achieved by aggregate UDF.
### Finalize The function template below can be used to finalize the result of your own UDF, normally used when interBuf is used.
Below function template can be used to finalize the result of your own UDF, normally used when interBuf is used. `void udfFinalizeFunc(char* dataOutput, char* interBuf, int* numOfOutput, SUdfInit* buf)`
`void abs_max_finalize(char* dataOutput, char* interBuf, int* numOfOutput, SUdfInit* buf)`
`udfFinalizeFunc` is the place holder of function name, definitions of the parameter are as below: `udfFinalizeFunc` is the place holder of function name, definitions of the parameter are as below:
...@@ -72,47 +77,64 @@ Below function template can be used to finalize the result of your own UDF, norm ...@@ -72,47 +77,64 @@ Below function template can be used to finalize the result of your own UDF, norm
- numOfOutput:number of output data, can only be 0 or 1 for aggregate function - numOfOutput:number of output data, can only be 0 or 1 for aggregate function
- buf:for state exchange between UDF and TDengine - buf:for state exchange between UDF and TDengine
## UDF Conventions ### Example abs_max.c
[abs_max.c](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/abs_max.c) is an example of a user defined aggregate function to get the maximum from the absolute values of a column.
The internal processing happens as follows. The results of the select statement are divided into multiple row blocks and `udfNormalFunc`, i.e. `abs_max` in this case, is performed on each row block to generate the intermediate results for each sub table. Then `udfMergeFunc`, i.e. `abs_max_merge` in this case, is performed on the intermediate result of sub tables to aggregate and generate the final or intermediate result of STable. The intermediate result of STable is finally processed by `udfFinalizeFunc`, i.e. `abs_max_finalize` in this example, to generate the final result, which contains either 0 or 1 row.
Other typical aggregation functions such as covariance, can also be implemented using aggregate UDF.
The naming of 3 kinds of UDF, i.e. udfNormalFunc, udfMergeFunc, and udfFinalizeFunc is required to have same prefix, i.e. the actual name of udfNormalFunc, which means udfNormalFunc doesn't need a suffix following the function name. While udfMergeFunc should be udfNormalFunc followed by `_merge`, udfFinalizeFunc should be udfNormalFunc followed by `_finalize`. The naming convention is part of UDF framework, TDengine follows this convention to invoke corresponding actual functions.\ ## UDF Naming Conventions
According to the kind of UDF to implement, the functions that need to be implemented are different. The naming convention for the 3 kinds of function templates required by UDF is as follows:
- udfNormalFunc, udfMergeFunc, and udfFinalizeFunc are required to have same prefix, i.e. the actual name of udfNormalFunc. The udfNormalFunc doesn't need a suffix following the function name.
- udfMergeFunc should be udfNormalFunc followed by `_merge`
- udfFinalizeFunc should be udfNormalFunc followed by `_finalize`.
- Scalar function:udfNormalFunc is required The naming convention is part of TDengine's UDF framework. TDengine follows this convention to invoke the corresponding actual functions.
- Aggregate function:udfNormalFunc, udfMergeFunc (if query on STable) and udfFinalizeFunc are required
To be more accurate, assuming we want to implement a UDF named "foo". If the function is a scalar function, what we really need to implement is `foo`; if the function is aggregate function, we need to implement `foo`, `foo_merge`, and `foo_finalize`. For aggregate UDF, even though one of the three functions is not necessary, there must be an empty implementation. Depending on whether you are creating a scalar UDF or aggregate UDF, the functions that you need to implement are different.
- Scalar function:udfNormalFunc is required.
- Aggregate function:udfNormalFunc, udfMergeFunc (if query on STable) and udfFinalizeFunc are required.
For clarity, assuming we want to implement a UDF named "foo":
- If the function is a scalar function, we only need to implement the "normal" function template and it should be named simply `foo`.
- If the function is an aggregate function, we need to implement `foo`, `foo_merge`, and `foo_finalize`. Note that for aggregate UDF, even though one of the three functions is not necessary, there must be an empty implementation.
## Compile UDF ## Compile UDF
The source code of UDF in C can't be utilized by TDengine directly. UDF can only be loaded into TDengine after compiling to dynamically linked library. The source code of UDF in C can't be utilized by TDengine directly. UDF can only be loaded into TDengine after compiling to dynamically linked library (DLL).
For example, the example UDF `add_one.c` mentioned in previous sections need to be compiled into DLL using below command on Linux Shell. For example, the example UDF `add_one.c` mentioned earlier, can be compiled into DLL using the command below, in a Linux Shell.
```bash ```bash
gcc -g -O0 -fPIC -shared add_one.c -o add_one.so gcc -g -O0 -fPIC -shared add_one.c -o add_one.so
``` ```
The generated DLL file `dd_one.so` can be used later when creating UDF. It's recommended to use GCC not older than 7.5. The generated DLL file `add_one.so` can be used later when creating a UDF. It's recommended to use GCC not older than 7.5.
## Create and Use UDF ## Create and Use UDF
When a UDF is created in a TDengine instance, it is available across the databases in that instance.
### Create UDF ### Create UDF
SQL command can be executed on the same hos where the generated UDF DLL resides to load the UDF DLL into TDengine, this operation can't be done through REST interface or web console. Once created, all the clients of the current TDengine can use these UDF functions in their SQL commands. UDF are stored in the management node of TDengine. The UDFs loaded in TDengine would be still available after TDengine is restarted. SQL command can be executed on the host where the generated UDF DLL resides to load the UDF DLL into TDengine. This operation cannot be done through REST interface or web console. Once created, any client of the current TDengine can use these UDF functions in their SQL commands. UDF are stored in the management node of TDengine. The UDFs loaded in TDengine would be still available after TDengine is restarted.
When creating UDF, it needs to be clarified as either scalar function or aggregate function. If the specified type is wrong, the SQL statements using the function would fail with error. Besides, the input type and output type don't need to be same in UDF, but the input data type and output data type need to be consistent with the UDF definition. When creating UDF, the type of UDF, i.e. a scalar function or aggregate function must be specified. If the specified type is wrong, the SQL statements using the function would fail with errors. The input type and output type don't need to be the same in UDF, but the input data type and output data type must be consistent with the UDF definition.
- Create Scalar Function - Create Scalar Function
```sql ```sql
CREATE FUNCTION ids(X) AS ids(Y) OUTPUTTYPE typename(Z) [ BUFSIZE B ]; CREATE FUNCTION userDefinedFunctionName AS "/absolute/path/to/userDefinedFunctionName.so" OUTPUTTYPE <supported TDengine type> [BUFSIZE B];
``` ```
- ids(X):the function name to be sued in SQL statement, must be consistent with the function name defined by `udfNormalFunc` - userDefinedFunctionName:The function name to be used in SQL statement which must be consistent with the function name defined by `udfNormalFunc` and is also the name of the compiled DLL (.so file).
- ids(Y):the absolute path of the DLL file including the implementation of the UDF, the path needs to be quoted by single or double quotes - path:The absolute path of the DLL file including the name of the shared object file (.so). The path must be quoted with single or double quotes.
- typename(Z):the output data type, the value is the literal string of the type - outputtype:The output data type, the value is the literal string of the supported TDengine data type.
- B:the size of intermediate buffer, in bytes; it's an optional parameter and the range is [0,512] - B:the size of intermediate buffer, in bytes; it is an optional parameter and the range is [0,512].
For example, below SQL statement can be used to create a UDF from `add_one.so`. For example, below SQL statement can be used to create a UDF from `add_one.so`.
...@@ -123,17 +145,17 @@ CREATE FUNCTION add_one AS "/home/taos/udf_example/add_one.so" OUTPUTTYPE INT; ...@@ -123,17 +145,17 @@ CREATE FUNCTION add_one AS "/home/taos/udf_example/add_one.so" OUTPUTTYPE INT;
- Create Aggregate Function - Create Aggregate Function
```sql ```sql
CREATE AGGREGATE FUNCTION ids(X) AS ids(Y) OUTPUTTYPE typename(Z) [ BUFSIZE B ]; CREATE AGGREGATE FUNCTION userDefinedFunctionName AS "/absolute/path/to/userDefinedFunctionName.so" OUTPUTTYPE <supported TDengine data type> [ BUFSIZE B ];
``` ```
- ids(X):the function name to be sued in SQL statement, must be consistent with the function name defined by `udfNormalFunc` - userDefinedFunctionName:the function name to be used in SQL statement which must be consistent with the function name defined by `udfNormalFunc` and is also the name of the compiled DLL (.so file).
- ids(Y):the absolute path of the DLL file including the implementation of the UDF, the path needs to be quoted by single or double quotes - path:the absolute path of the DLL file including the name of the shared object file (.so). The path needs to be quoted by single or double quotes.
- typename(Z):the output data type, the value is the literal string of the type - OUTPUTTYPE:the output data type, the value is the literal string of the type
- B:the size of intermediate buffer, in bytes; it's an optional parameter and the range is [0,512] - B:the size of intermediate buffer, in bytes; it's an optional parameter and the range is [0,512]
For details about how to use intermediate result, please refer to example program [demo.c](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/demo.c). For details about how to use intermediate result, please refer to example program [demo.c](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/demo.c).
For example, below SQL statement can be used to create a UDF rom `demo.so`. For example, below SQL statement can be used to create a UDF from `demo.so`.
```sql ```sql
CREATE AGGREGATE FUNCTION demo AS "/home/taos/udf_example/demo.so" OUTPUTTYPE DOUBLE bufsize 14; CREATE AGGREGATE FUNCTION demo AS "/home/taos/udf_example/demo.so" OUTPUTTYPE DOUBLE bufsize 14;
...@@ -176,11 +198,11 @@ In current version there are some restrictions for UDF ...@@ -176,11 +198,11 @@ In current version there are some restrictions for UDF
1. Only Linux is supported when creating and invoking UDF for both client side and server side 1. Only Linux is supported when creating and invoking UDF for both client side and server side
2. UDF can't be mixed with builtin functions 2. UDF can't be mixed with builtin functions
3. Only one UDF can be used in a SQL statement 3. Only one UDF can be used in a SQL statement
4. Single column is supported as input for UDF 4. Only a single column is supported as input for UDF
5. Once created successfully, UDF is persisted in MNode of TDengineUDF 5. Once created successfully, UDF is persisted in MNode of TDengineUDF
6. UDF can't be created through REST interface 6. UDF can't be created through REST interface
7. The function name used when creating UDF in SQL must be consistent with the function name defined in the DLL, i.e. the name defined by `udfNormalFunc` 7. The function name used when creating UDF in SQL must be consistent with the function name defined in the DLL, i.e. the name defined by `udfNormalFunc`
8. The name name of UDF name should not conflict with any of builtin functions 8. The name of a UDF should not conflict with any of TDengine's built-in functions
## Examples ## Examples
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册