diff --git a/README.md b/README.md index 0babfe77763812ae1251e6ac0773091e29724a01..d75680cb22204011b628f45990355982a097ef64 100644 --- a/README.md +++ b/README.md @@ -115,251 +115,6 @@ TDengine provides abundant developing tools for users to develop on TDengine. Fo - [RESTful API](https://www.taosdata.com/en/documentation/connector/#RESTful-Connector) - [Node.js](https://www.taosdata.com/en/documentation/connector/#Node.js-Connector) -# How to run the test cases and how to add a new test case? - -### Prepare development environment - -1. sudo apt install - build-essential cmake net-tools python-pip python-setuptools python3-pip - python3-setuptools valgrind psmisc curl - -2. git clone ; cd TDengine - -3. mkdir debug; cd debug; cmake ..; make ; sudo make install - -4. pip install src/connector/python/linux/python2 ; pip3 install - src/connector/python/linux/python3 - -### How to run TSIM test suite - -1. cd \/tests/script - -2. sudo ./test.sh - -### How to run Python test suite - -1. cd \/tests/pytest - -2. ./smoketest.sh \# for smoke test - -3. ./smoketest.sh -g \# for memory leak detection test with valgrind - -4. ./fulltest.sh \# for full test - -> Note1: TDengine daemon's configuration and data files are stored in -> \/sim directory. As a historical design, it's same place with -> TSIM script. So after the TSIM script ran with sudo privilege, the directory -> has been used by TSIM then the python script cannot write it by a normal -> user. You need to remove the directory completely first before running the -> Python test case. We should consider using two different locations to store -> for TSIM and Python script. - -> Note2: if you need to debug crash problem with a core dump, you need -> manually edit smoketest.sh or fulltest.sh to add "ulimit -c unlimited" -> before the script line. Then you can look for the core file in -> \/tests/pytest after the program crash. - -### How to add a new test case - -**1. add a new TSIM test cases:** - -TSIM test cases are now included in the new development branch and can be -added to the TDengine/tests/script/test.sh script based on the manual test -methods necessary to add test cases as described above. - -**2. add a new Python test cases:** - -**2.1 Please refer to \/tests/pytest/insert/basic.py to add a new -test case.** The new test case must implement 3 functions, where self.init() -and self.stop() simply copy the contents of insert/basic.py and the test -logic is implemented in self.run(). You can refer to the code in the util -directory for more information. - -**2.2 Edit smoketest.sh to add the path and filename of the new test case** - -Note: The Python test framework may continue to be improved in the future, -hopefully, to provide more functionality and ease of writing test cases. The -method of writing the test case above does not exclude that it will also be -affected. - -**2.3 What test.py does in detail:** - -test.py is the entry program for test case execution and monitoring. - -test.py has the following functions. - -\-f --file, Specifies the test case file name to be executed --p --path, Specifies deployment path - -\-m --master, Specifies the master server IP for cluster deployment --c--cluster, test cluster function --s--stop, terminates all running nodes - -\-g--valgrind, load valgrind for memory leak detection test - -\-h--help, display help - -**2.4 What util/log.py does in detail:** - -log.py is quite simple, the main thing is that you can print the output in -different colors as needed. The success() should be called for successful -test case execution and the success() will print green text. The exit() will -print red text and exit the program, exit() should be called for test -failure. - -**util/log.py** - -... - -    def info(self, info): - -        printf("%s %s" % (datetime.datetime.now(), info)) - -  - -    def sleep(self, sec): - -        printf("%s sleep %d seconds" % (datetime.datetime.now(), sec)) - -        time.sleep(sec) - -  - -    def debug(self, err): - -        printf("\\033[1;36m%s %s\\033[0m" % (datetime.datetime.now(), err)) - -  - -    def success(self, info): - -        printf("\\033[1;32m%s %s\\033[0m" % (datetime.datetime.now(), info)) - -  - -    def notice(self, err): - -        printf("\\033[1;33m%s %s\\033[0m" % (datetime.datetime.now(), err)) - -  - -    def exit(self, err): - -        printf("\\033[1;31m%s %s\\033[0m" % (datetime.datetime.now(), err)) - -        sys.exit(1) - -  - -    def printNoPrefix(self, info): - -        printf("\\033[1;36m%s\\033[0m" % (info) - -... - -**2.5 What util/sql.py does in detail:** - -SQL.py is mainly used to execute SQL statements to manipulate the database, -and the code is extracted and commented as follows: - -**util/sql.py** - -\# prepare() is mainly used to set up the environment for testing table and -data, and to set up the database db for testing. do not call prepare() if you -need to test the database operation command. - -def prepare(self): - -tdLog.info("prepare database:db") - -self.cursor.execute('reset query cache') - -self.cursor.execute('drop database if exists db') - -self.cursor.execute('create database db') - -self.cursor.execute('use db') - -... - -\# query() is mainly used to execute select statements for normal syntax input - -def query(self, sql): - -... - -\# error() is mainly used to execute the select statement with the wrong syntax -input, the error will be caught as a reasonable behavior, if not caught it will -prove that the test failed - -def error() - -... - -\# checkRows() is used to check the number of returned lines after calling -query(select ...) after calling the query(select ...) to check the number of -rows of returned results. - -def checkRows(self, expectRows): - -... - -\# checkData() is used to check the returned result data after calling -query(select ...) after the query(select ...) is called, failure to meet -expectation is - -def checkData(self, row, col, data): - -... - -\# getData() returns the result data after calling query(select ...) to return -the resulting data after calling query(select ...) - -def getData(self, row, col): - -... - -\# execute() used to execute sql and return the number of affected rows - -def execute(self, sql): - -... - -\# executeTimes() Multiple executions of the same sql statement - -def executeTimes(self, sql, times): - -... - -\# CheckAffectedRows() Check if the number of affected rows is as expected - -def checkAffectedRows(self, expectAffectedRows): - -... - -> Note: Both Python2 and Python3 are currently supported by the Python test -> case. Since Python2 is no longer officially supported by January 1, 2020, it -> is recommended that subsequent test case development be guaranteed to run -> correctly on Python3. For Python2, please consider being compatible if -> appropriate without additional -> burden.   - -### CI Covenant submission adoption principle. - -- Every commit / PR compilation must pass. Currently, the warning is treated - as an error, so the warning must also be resolved. - -- Test cases that already exist must pass. - -- Because CI is very important to support build and automatically test - procedure, it is necessary to manually test the test case before adding it - and do as many iterations as possible to ensure that the test case provides - stable and reliable test results when added. - -> Note: In the future, according to the requirements and test development -> progress will add stress testing, performance testing, code style, -> and other features based on functional testing. - ### Third Party Connectors The TDengine community has also kindly built some of their own connectors! Follow the links below to find the source code for them. @@ -367,6 +122,10 @@ The TDengine community has also kindly built some of their own connectors! Follo - [Rust Connector](https://github.com/taosdata/TDengine/tree/master/tests/examples/rust) - [.Net Core Connector](https://github.com/maikebing/Maikebing.EntityFrameworkCore.Taos) +# How to run the test cases and how to add a new test case? + TDengine's test framework and all test cases are fully open source. + Please refer to [this document](tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md) for how to run test and develop new test case. + # TDengine Roadmap - Support event-driven stream computing - Support user defined functions diff --git a/tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md b/tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md new file mode 100644 index 0000000000000000000000000000000000000000..7c42d47d1b9c95a863539e3dbe1f4b94abf6c753 --- /dev/null +++ b/tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md @@ -0,0 +1,235 @@ +### Prepare development environment + +1. sudo apt install + build-essential cmake net-tools python-pip python-setuptools python3-pip + python3-setuptools valgrind psmisc curl + +2. git clone ; cd TDengine + +3. mkdir debug; cd debug; cmake ..; make ; sudo make install + +4. pip install src/connector/python/linux/python2 ; pip3 install + src/connector/python/linux/python3 + +### How to run Python test suite + +1. cd \/tests/pytest + +2. ./smoketest.sh \# for smoke test + +3. ./smoketest.sh -g \# for memory leak detection test with valgrind + +4. ./fulltest.sh \# for full test + +> Note1: TDengine daemon's configuration and data files are stored in +> \/sim directory. As a historical design, it's same place with +> TSIM script. So after the TSIM script ran with sudo privilege, the directory +> has been used by TSIM then the python script cannot write it by a normal +> user. You need to remove the directory completely first before running the +> Python test case. We should consider using two different locations to store +> for TSIM and Python script. + +> Note2: if you need to debug crash problem with a core dump, you need +> manually edit smoketest.sh or fulltest.sh to add "ulimit -c unlimited" +> before the script line. Then you can look for the core file in +> \/tests/pytest after the program crash. + + +### How to add a new test case + +**1. TSIM test cases:** + +TSIM was the testing framework has been used internally. Now it still be used to run the test cases we develop in the past as a legacy system. We are turning to use Python to develop new test case and are abandoning TSIM gradually. + +**2. Python test cases:** + +**2.1 Please refer to \/tests/pytest/insert/basic.py to add a new +test case.** The new test case must implement 3 functions, where self.init() +and self.stop() simply copy the contents of insert/basic.py and the test +logic is implemented in self.run(). You can refer to the code in the util +directory for more information. + +**2.2 Edit smoketest.sh to add the path and filename of the new test case** + +Note: The Python test framework may continue to be improved in the future, +hopefully, to provide more functionality and ease of writing test cases. The +method of writing the test case above does not exclude that it will also be +affected. + +**2.3 What test.py does in detail:** + +test.py is the entry program for test case execution and monitoring. + +test.py has the following functions. + +\-f --file, Specifies the test case file name to be executed +-p --path, Specifies deployment path + +\-m --master, Specifies the master server IP for cluster deployment +-c--cluster, test cluster function +-s--stop, terminates all running nodes + +\-g--valgrind, load valgrind for memory leak detection test + +\-h--help, display help + +**2.4 What util/log.py does in detail:** + +log.py is quite simple, the main thing is that you can print the output in +different colors as needed. The success() should be called for successful +test case execution and the success() will print green text. The exit() will +print red text and exit the program, exit() should be called for test +failure. + +**util/log.py** + +... + +    def info(self, info): + +        printf("%s %s" % (datetime.datetime.now(), info)) + +  + +    def sleep(self, sec): + +        printf("%s sleep %d seconds" % (datetime.datetime.now(), sec)) + +        time.sleep(sec) + +  + +    def debug(self, err): + +        printf("\\033[1;36m%s %s\\033[0m" % (datetime.datetime.now(), err)) + +  + +    def success(self, info): + +        printf("\\033[1;32m%s %s\\033[0m" % (datetime.datetime.now(), info)) + +  + +    def notice(self, err): + +        printf("\\033[1;33m%s %s\\033[0m" % (datetime.datetime.now(), err)) + +  + +    def exit(self, err): + +        printf("\\033[1;31m%s %s\\033[0m" % (datetime.datetime.now(), err)) + +        sys.exit(1) + +  + +    def printNoPrefix(self, info): + +        printf("\\033[1;36m%s\\033[0m" % (info) + +... + +**2.5 What util/sql.py does in detail:** + +SQL.py is mainly used to execute SQL statements to manipulate the database, +and the code is extracted and commented as follows: + +**util/sql.py** + +\# prepare() is mainly used to set up the environment for testing table and +data, and to set up the database db for testing. do not call prepare() if you +need to test the database operation command. + +def prepare(self): + +tdLog.info("prepare database:db") + +self.cursor.execute('reset query cache') + +self.cursor.execute('drop database if exists db') + +self.cursor.execute('create database db') + +self.cursor.execute('use db') + +... + +\# query() is mainly used to execute select statements for normal syntax input + +def query(self, sql): + +... + +\# error() is mainly used to execute the select statement with the wrong syntax +input, the error will be caught as a reasonable behavior, if not caught it will +prove that the test failed + +def error() + +... + +\# checkRows() is used to check the number of returned lines after calling +query(select ...) after calling the query(select ...) to check the number of +rows of returned results. + +def checkRows(self, expectRows): + +... + +\# checkData() is used to check the returned result data after calling +query(select ...) after the query(select ...) is called, failure to meet +expectation is + +def checkData(self, row, col, data): + +... + +\# getData() returns the result data after calling query(select ...) to return +the resulting data after calling query(select ...) + +def getData(self, row, col): + +... + +\# execute() used to execute sql and return the number of affected rows + +def execute(self, sql): + +... + +\# executeTimes() Multiple executions of the same sql statement + +def executeTimes(self, sql, times): + +... + +\# CheckAffectedRows() Check if the number of affected rows is as expected + +def checkAffectedRows(self, expectAffectedRows): + +... + +> Note: Both Python2 and Python3 are currently supported by the Python test +> case. Since Python2 is no longer officially supported by January 1, 2020, it +> is recommended that subsequent test case development be guaranteed to run +> correctly on Python3. For Python2, please consider being compatible if +> appropriate without additional +> burden.   + +### CI submission adoption principle. + +- Every commit / PR compilation must pass. Currently, the warning is treated + as an error, so the warning must also be resolved. + +- Test cases that already exist must pass. + +- Because CI is very important to support build and automatically test + procedure, it is necessary to manually test the test case before adding it + and do as many iterations as possible to ensure that the test case provides + stable and reliable test results when added. + +> Note: In the future, according to the requirements and test development +> progress will add stress testing, performance testing, code style, +> and other features based on functional testing.