Merge pull request #19564 from taosdata/docs-cloud-ug

Docs cloud ug

jemalloc @ ea6b3e97

-Subproject commit ea6b3e973b477b8061e0076bb257dbd7f3faa756

docs/en/01-index.md

@@ -17,17 +17,19 @@ This is the documentation structure for TDengine Cloud.
 
 5. The [Visualization](./visual) section shows you how you can visualize the data that you store in TDengine, as well as how you can visualize and monitor the status of your TDengine Cloud instance(s) and databases.
 
-6. Data [Subscription](./tmq) is an advanced and useful feature of TDengine. It is similar to asynchronous publish/subscribe where a message published to a topic is immediately received by all subscribers to that topic. TDengine Subscriptions allow you to create event driven applications without having to install an external pub/sub messaging system.
+6. [Data Sharing](./data-sharing) is an advanced and useful feature of TDengine Cloud. In this section, we provide an easy way to share your data in the TDengine Cloud to others through only simple click operations.
 
-7. [Stream Processing](./stream) is another extremely useful feature of TDengine Cloud that obviates the need to install external stream processing systems like Kafka or Flink. TDengine's Stream Processing feature allows you to process incoming data streams in real time and push data to tables based on rules that you can define easily.  
+7. Data [Subscription](./tmq) is an advanced and useful feature of TDengine. It is similar to asynchronous publish/subscribe where a message published to a topic is immediately received by all subscribers to that topic. TDengine Subscriptions allow you to create event driven applications without having to install an external pub/sub messaging system.
 
-8. TDengine provides sophisticated [Data Replication](./replication) features. You can replicate from Cloud to a private instance and vice versa. You can replicate between Cloud providers regardless of region and you can also replicate between edge instances and Cloud or edge instances and private centralized instances.
+8. [Stream Processing](./stream) is another extremely useful feature of TDengine Cloud that obviates the need to install external stream processing systems like Kafka or Flink. TDengine's Stream Processing feature allows you to process incoming data streams in real time and push data to tables based on rules that you can define easily.  
 
-9. The [Developer Guide](./programming) is a must read if you are developing IoT or Big Data applications for time series data. In this section we introduce the database connection, data modeling, data ingestion, query, stream processing, cache, data subscription, user-defined functions (coming soon), and other functionality in detail. Sample code is provided for a variety of programming languages. In most cases, you can just copy and paste the sample code, make a few changes to accommodate your application, and it will work.
+9. TDengine provides sophisticated [Data Replication](./replication) features. You can replicate from Cloud to a private instance and vice versa. You can replicate between Cloud providers regardless of region and you can also replicate between edge instances and Cloud or edge instances and private centralized instances.
 
-10. The [TDengine SQL](./taos-sql) section provides comprehensive information about both standard SQL as well as TDengine's extensions for easy time series analysis.
+10. The [Developer Guide](./programming) is a must read if you are developing IoT or Big Data applications for time series data. In this section we introduce the database connection, data modeling, data ingestion, query, stream processing, cache, data subscription, user-defined functions (coming soon), and other functionality in detail. Sample code is provided for a variety of programming languages. In most cases, you can just copy and paste the sample code, make a few changes to accommodate your application, and it will work.
 
-11. The [Tools](./tools) section introduces the Taos CLI which gives you shell access to easily perform ad hoc queries on your instances and databases. Additionally, taosBenchmark is introduced. It is a tool that can help you generate large amounts of data very easily with simple configurations and test the performance of TDengine Cloud.
+11. The [TDengine SQL](./taos-sql) section provides comprehensive information about both standard SQL as well as TDengine's extensions for easy time series analysis.
+
+12. The [Tools](./tools) section introduces the Taos CLI which gives you shell access to easily perform ad hoc queries on your instances and databases. Additionally, taosBenchmark is introduced. It is a tool that can help you generate large amounts of data very easily with simple configurations and test the performance of TDengine Cloud.
 
 <!-- 10. Finally, in the [FAQ](./faq) section, we try to preemptively answer questions that we anticipate. Of course, we will continue to add to this section all the time. -->
 

docs/en/02-intro.md

@@ -3,7 +3,7 @@ sidebar_label: Introduction
 title: Introduction to TDengine Cloud Service
 ---
 
-TDengine Cloud, is the fast, elastic, serverless and cost effective time-series data processing service based on the popular open source time-series database, TDengine. With TDengine Cloud you get the highly optimized and purpose-built for IoT time-series platform, for which TDengine is known. 
+TDengine Cloud, is the fast, elastic, serverless and cost effective time-series data processing service based on the popular open source time-series database, TDengine. With TDengine Cloud you get the highly optimized and purpose-built for IoT time-series platform, for which TDengine is known.
 
 This section introduces the major features, competitive advantages and typical use-cases to help you get a high level overview of TDengine cloud service.
 
@@ -11,7 +11,7 @@ This section introduces the major features, competitive advantages and typical u
 
 The major features are listed below:
 
-1. Data In 
+1. Data In
    - Supports [using SQL to insert](../data-in/insert-data).
    - Supports [Telegraf](../data-in/telegraf/).
    - Supports [Prometheus](../data-in/prometheus/).
@@ -21,13 +21,13 @@ The major features are listed below:
    - Supports writing data to [Prometheus](../data-out/prometheus/).
    - Supports exporting data via [data subscription](../tmq/).
 3. Data Explorer: browse through databases and even run SQL queryies once you login.
-4. Visualization: 
+4. Visualization:
    - Supports [Grafana](../visual/grafana/)
    - Supports Google data studio (to be released soon)
-   - Supports Grafana cloud (to be released soon) 
+   - Supports Grafana cloud (to be released soon)
 6. [Stream Processing](../stream/): Not only is the continuous query is supported, but TDengine also supports event driven stream processing, so Flink or Spark is not needed for time-series data processing.
 7. [Data Subscription](../tmq/): Application can subscribe a table or a set of tables. API is the same as Kafka, but you can specify filter conditions.
-8. Enterprise 
+8. Enterprise
    - Supports backuping data everyday.
    - Supports replicating a database to another region or cloud.
    - Supports VPC peering.
@@ -45,7 +45,7 @@ For more details on features, please read through the entire documentation.
 
 By making full use of [characteristics of time series data](https://tdengine.com/tsdb/characteristics-of-time-series-data/) and its cloud native design, TDengine Cloud differentiates itself from other time series data cloud services, with the following advantages.
 
-- **Worry Free**: TDengine Cloud is a fast, elastic, serverless purpose built cloud platform for time-series data. It provides worry-free operations with a fully managed cloud service. You pay as you go. 
+- **Worry Free**: TDengine Cloud is a fast, elastic, serverless purpose built cloud platform for time-series data. It provides worry-free operations with a fully managed cloud service. You pay as you go.
 
 - **[Simplified Solution](https://tdengine.com/tdengine/simplified-time-series-data-solution/)**: Through built-in caching, stream processing and data subscription features, TDengine provides a simplified solution for time-series data processing. It reduces system design complexity and operation costs significantly.
 
@@ -78,4 +78,4 @@ On the left-hand side, there are data collection agents like OPC-UA, MQTT, Teleg
 
 ## Typical Use Cases
 
-As a high-performance and cloud native time-series database, TDengine's typical use case include but are not limited to IoT, Industrial Internet, Connected Vehicles, IT operation and maintenance, energy, financial markets and other fields. TDengine is a purpose-built database optimized for the characteristics of time series data. As such, it cannot be used to process data from web crawlers, social media, e-commerce, ERP, CRM and so on. More generally TDengine is not a suitable storage engine for non-time-series data. 
+As a high-performance and cloud native time-series database, TDengine's typical use case include but are not limited to IoT, Industrial Internet, Connected Vehicles, IT operation and maintenance, energy, financial markets and other fields. TDengine is a purpose-built database optimized for the characteristics of time series data. As such, it cannot be used to process data from web crawlers, social media, e-commerce, ERP, CRM and so on. More generally TDengine is not a suitable storage engine for non-time-series data.

docs/en/11-data-sharing/index.md

+---
+sidebar_label: Data Sharing
+title: Data Sharing
+description: Using topics to share data from TDengine.
+---
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+The topic introduces how to share data from TDengine instance through the access control management of TDengine Cloud and the subscription interfaces of each supported connectors. The data owner first creates the topic through the topic wizard. Then adds the users or user groups which he wants to share the data with to the subscribers of the topic. The subscriber of the topic can get the details about how to access the shared data from TDengine in the data subscription way. In this document we will briefly explain these main steps of data sharing.
+
+## Create Topic
+
+You can create the topic in Topics of TDengine Cloud. In the Create Topic dialog, you can choose wizard or SQL way to create the topic. In the wizard way, you need to input the topic name and select the database of the current TDengine instance. Then select the super table or specify the subquery with the super table or sub table. Also you can add fields selections or add result set and condition set for each field. In the following, you can get the detail of how to create the topic in three levels through wizard way.
+
+### To Database
+
+The default selection in the Add New Topic dialog is database type. After select a database in the selection, you can click Confirm button to create a topic to a database.
+
+![Create a new topic](./topic/add-topic-db.webp)
+
+### To Super Table
+
+In the opened Add New Topic dialog, you can click STable type and select a specified super table from the selections. Then click Confirm button to create a topic to a super table.
+
+![Create a new topic to stable](./topic/add-topic-stable.webp)
+
+### With Subquery
+
+In the opened Add New Topic dialog, you can click Subquery type to show all subquery form items. The first item is Table Type and the default selection is STable. After you select or input a super table name, the following will show you all fields from the super table. You can check or uncheck each field for the sub query and also you can set the result set or condition set for each field. If you want to preview the SQL based on your chooses, click SQL Preiview to open a SQL dialog to view.
+
+![Create a new topic with subquery for stable](./topic/add-topic-sub-stable.webp)
+
+You can select another Table Table Table and then select a table from the selections or input an existed table name. You can get all fields of the selected table. You can check or uncheck each field for the sub query and also you can set the result set or condition set for each field. If you want to preview the SQL based on your chooses, click SQL Preiview to open a SQL dialog to view.
+
+![Create a new topic with subquery for table](./topic/add-topic-sub-table.webp)
+
+## Share Topic
+
+In each row of the topic list in the Topics page, you can click Share Topic action icon to the Share Topic page. Also you can directly click Share Topic tab to switch to the right location. In the Share Topic tab, you can get only one row for yourself in the Users page.
+
+### Users
+
+In the default tab Users of the Share Topic page, you can click **Add Users** button to add more users who are active in the current organization. In the opened Add New Users dialog, you can select the new users who you want to share the topic with. Then you can set the expired time for the sharing to these users.
+
+![Share topic](./share/share-topic-users.webp)
+
+![Share topic users](./share/share-topic-adduser.webp)
+
+### User Groups
+
+You can click User Groups tab to switch to the User Groups page of the Share Topic. Then you can click **Add User Groups** button to add more user groups which are active in the current organization. In the opened Add New User Groups dialog, you can select the new user groups which you want to share the topic with. Then you can set the expired time for the sharing to these user groups.
+
+![Share topic user groups](./share/share-topic-usergroup.webp)
+
+![Share topic add user groups](./share/share-topic-addusergroup.webp)
+
+## Consume Shared Topic
+
+The shared user can get all topics which the creator shared with him, when he goes to the Topic page of Data Subscription. The user can click **Sample Code** icon of each topic **Action** area to the **Sample Code** page. Then he can follow the steps of the sample code how to consume the shared topic from TDengine instance.
+
+![Share topic example](./example/share-topic-example.webp)
+
+
+### Data Schema and API
+
+The related schemas and APIs in various languages are described as follows:
+
+<Tabs defaultValue="Go" groupId="lang">
+
+<TabItem label="Go" value="Go">
+
+```go
+func NewConsumer(conf *Config) (*Consumer, error)
+
+func (c *Consumer) Close() error
+
+func (c *Consumer) Commit(ctx context.Context, message unsafe.Pointer) error
+
+func (c *Consumer) FreeMessage(message unsafe.Pointer)
+
+func (c *Consumer) Poll(timeout time.Duration) (*Result, error)
+
+func (c *Consumer) Subscribe(topics []string) error
+
+func (c *Consumer) Unsubscribe() error
+```
+
+</TabItem>
+
+<TabItem label="Rust" value="Rust">
+
+```rust
+impl TBuilder for TmqBuilder
+  fn from_dsn<D: IntoDsn>(dsn: D) -> Result<Self, Self::Error>
+  fn build(&self) -> Result<Self::Target, Self::Error>
+
+impl AsAsyncConsumer for Consumer
+  async fn subscribe<T: Into<String>, I: IntoIterator<Item = T> + Send>(
+        &mut self,
+        topics: I,
+    ) -> Result<(), Self::Error>;
+  fn stream(
+        &self,
+    ) -> Pin<
+        Box<
+            dyn '_
+                + Send
+                + futures::Stream<
+                    Item = Result<(Self::Offset, MessageSet<Self::Meta, Self::Data>), Self::Error>,
+                >,
+        >,
+    >;
+  async fn commit(&self, offset: Self::Offset) -> Result<(), Self::Error>;
+
+  async fn unsubscribe(self);
+```
+
+For more information, see [Crate taos](https://docs.rs/taos).
+
+</TabItem>
+
+<TabItem value="Python" label="Python">
+
+```python
+class TaosConsumer():
+    def __init__(self, *topics, **configs)
+
+    def __iter__(self)
+
+    def __next__(self)
+
+    def sync_next(self)
+    
+    def subscription(self)
+
+    def unsubscribe(self)
+
+    def close(self)
+    
+    def __del__(self)
+```
+
+</TabItem>
+</Tabs>
+
+### Configure TDengine DSN
+
+You can set the following for Go and Rust:
+
+<Tabs defaultValue="Bash" groupId="config">
+<TabItem value="Bash" label="Bash">
+
+```shell
+export TDENGINE_CLOUD_TMQ="<TDENGINE_CLOUD_TMQ>"
+```
+
+</TabItem>
+<TabItem value="CMD" label="CMD">
+
+```shell
+set TDENGINE_CLOUD_TMQ="<TDENGINE_CLOUD_TMQ>"
+```
+
+</TabItem>
+<TabItem value="Powershell" label="Powershell">
+
+```shell
+$env:TDENGINE_CLOUD_TMQ="<TDENGINE_CLOUD_TMQ>"
+```
+
+</TabItem>
+</Tabs>
+
+:::note
+Replace <TDENGINE_CLOUD_TMQ> with the real value, the format should be `wss://<cloud_endpoint>)/rest/tmq?token=<token>`.
+To obtain the value of `TDENGINE_CLOUD_TMQ`, please log in [TDengine Cloud](https://cloud.tdengine.com) and click **Topcis** on the left menu, then click **Sample Code** action of the each topic to **Example** part.
+:::
+
+Especially, for Python, you need to set the following variables:
+
+<Tabs defaultValue="Bash" groupId="config">
+<TabItem value="Bash" label="Bash">
+
+```shell
+export TDENGINE_CLOUD_ENDPOINT="<TDENGINE_CLOUD_ENDPOINT>"
+export TDENGINE_CLOUD_TOKEN="<TDENGINE_CLOUD_TOKEN>"
+```
+
+</TabItem>
+<TabItem value="CMD" label="CMD">
+
+```shell
+set TDENGINE_CLOUD_ENDPOINT="<TDENGINE_CLOUD_ENDPOINT>"
+set TDENGINE_CLOUD_TOKEN="<TDENGINE_CLOUD_TOKEN>"
+```
+
+</TabItem>
+<TabItem value="Powershell" label="Powershell">
+
+```shell
+$env:TDENGINE_CLOUD_ENDPOINT="<TDENGINE_CLOUD_ENDPOINT>"
+$env:TDENGINE_CLOUD_TOKEN="<TDENGINE_CLOUD_TOKEN>"
+```
+
+</TabItem>
+</Tabs>
+
+:::note
+Replace <TDENGINE_CLOUD_ENDPOINT> and <TDENGINE_CLOUD_TOKEN> with the real values. To obtain the value of these, please log in [TDengine Cloud](https://cloud.tdengine.com) and click **Topcis** on the left menu, then click **Sample Code** action of the each topic to the **Python** tab of the **Example** part.
+:::
+### Create a Consumer from Instance
+
+You configure the following parameters when creating a consumer:
+
+|            Parameter            |  Type   | Description                                                 | Remarks                                        |
+| :----------------------------: | :-----: | -------------------------------------------------------- | ------------------------------------------- |
+|        `td.connect.ip`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|        `td.connect.user`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|        `td.connect.pass`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|        `td.connect.port`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|           `group.id`           | string  | Consumer group ID; consumers with the same ID are in the same group                        | **Required**. Maximum length: 192.                 |
+|          `client.id`           | string  | Client ID                                                | Maximum length: 192.                             |
+|      `auto.offset.reset`       |  enum   | Initial offset for the consumer group                                     | Specify `earliest`, `latest`, or `none`(default) |
+|      `enable.auto.commit`      | boolean | Commit automatically                                             | Specify `true` or `false`.                   |
+|   `auto.commit.interval.ms`    | integer | Interval for automatic commits, in milliseconds                           |
+| `enable.heartbeat.background`  | boolean | Backend heartbeat; if enabled, the consumer does not go offline even if it has not polled for a long time |                                             |
+| `experimental.snapshot.enable` | boolean | Specify whether to consume messages from the WAL or from TSBS                    |                                             |
+|     `msg.with.table.name`      | boolean | Specify whether to deserialize table names from messages                                 |
+
+The method of specifying these parameters depends on the language used:
+
+<Tabs defaultValue="Go" groupId="lang">
+
+<TabItem label="Go" value="Go">
+
+```go
+import (
+  "github.com/taosdata/driver-go/v3/common"
+	tmqcommon "github.com/taosdata/driver-go/v3/common/tmq"
+	"github.com/taosdata/driver-go/v3/ws/tmq"
+)
+tmqStr := os.Getenv("TDENGINE_CLOUD_TMQ")
+consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
+  "ws.url":                tmqStr,
+  "ws.message.channelLen": uint(0),
+  "ws.message.timeout":    common.DefaultMessageTimeout,
+  "ws.message.writeWait":  common.DefaultWriteWait,
+  "group.id":              "test_group",
+  "client.id":             "test_consumer_ws",
+  "auto.offset.reset":     "earliest",
+})
+if err != nil {
+  panic(err)
+}
+```
+
+</TabItem>
+
+<TabItem label="Rust" value="Rust">
+
+```rust
+let tmq_str = std::env::var("TDENGINE_CLOUD_TMQ")?;
+let tmq_uri = format!( "{}&group.id=test_group_rs&client.id=test_consumer_ws", tmq_str);
+println!("request tmq URI is {tmq_uri}\n");
+let tmq = TmqBuilder::from_dsn(tmq_uri,)?;
+let mut consumer = tmq.build()?;
+```
+
+</TabItem>
+
+<TabItem value="Python" label="Python">
+
+Python programs use the following parameters:
+
+|            Parameter            |  Type   | Description                                                 | Remarks                                        |
+| :----------------------------: | :----: | -------------------------------------------------------- | ------------------------------------------- |
+|        `td_connect_ip`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|        `td_connect_user`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|        `td_connect_pass`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|        `td_connect_port`         | string  | Used in establishing a connection; same as `taos_connect`                          |                                             |
+|           `group_id`           | string  | Consumer group ID; consumers with the same ID are in the same group                        | **Required**. Maximum length: 192.                 |
+|          `client_id`           | string  | Client ID                                                | Maximum length: 192.                             |
+|      `auto_offset_reset`       |  string   | Initial offset for the consumer group                                     | Specify `earliest`, `latest`, or `none`(default) |
+|      `enable_auto_commit`      | string | Commit automatically                                             | Specify `true` or `false`.                   |
+|   `auto_commit_interval_ms`    | string | Interval for automatic commits, in milliseconds                           |
+| `enable_heartbeat_background`  | string | Backend heartbeat; if enabled, the consumer does not go offline even if it has not polled for a long time |  Specify `true` or `false`.                                           |
+| `experimental_snapshot_enable` | string | Specify whether to consume messages from the WAL or from TSBS                    | Specify `true` or `false`.                                            |
+|     `msg_with_table_name`      | string | Specify whether to deserialize table names from messages                                 | Specify `true` or `false`.
+|           `timeout`            |  int   | Consumer pull timeout                                     |                                      |
+
+```python
+endpoint = os.environ["TDENGINE_CLOUD_ENDPOINT"]
+token = os.environ["TDENGINE_CLOUD_TOKEN"]
+urlparts = endpoint.split(":", 1)
+
+conf = {
+    # auth options
+    "td.connect.websocket.scheme": "wss",
+    "td.connect.ip": urlparts[0],
+    "td.connect.port": urlparts[1],
+    "td.connect.token": token,
+    # consume options
+    "group.id": "test_group_py",
+    "client.id": "test_consumer_ws_py",
+}
+consumer = Consumer(conf)
+```
+
+</TabItem>
+</Tabs>
+
+A consumer group is automatically created when multiple consumers are configured with the same consumer group ID.
+
+### Subscribe to a Topic
+
+A single consumer can subscribe to multiple topics.
+
+<Tabs defaultValue="Go" groupId="lang">
+
+<TabItem value="Go" label="Go">
+
+```go
+err = consumer.Subscribe("<TDC_TOPIC>", nil)
+if err != nil {
+  panic(err)
+}
+```
+
+</TabItem>
+<TabItem value="Rust" label="Rust">
+
+```rust
+consumer.subscribe(["<TDC_TOPIC>"]).await?;
+```
+
+</TabItem>
+
+<TabItem value="Python" label="Python">
+
+```python
+consumer.subscribe(["<TDC_TOPIC>"])
+```
+
+</TabItem>
+</Tabs>
+
+:::note
+
+Replace <TDC_TOPIC\> with the real value. To obtain the value of `TDC_TOPIC`, please log in [TDengine Cloud](https://cloud.tdengine.com) and click **Topcis** on the left menu, then copy the topic name you want to consume.
+
+:::
+
+## Consume messages
+
+The following code demonstrates how to consume the messages in a queue.
+
+<Tabs defaultValue="Go" groupId="lang">
+
+<TabItem value="Go" label="Go">
+
+```go
+for {
+  ev := consumer.Poll(0)
+  if ev != nil {
+    switch e := ev.(type) {
+    case *tmqcommon.DataMessage:
+      fmt.Printf("get message:%v\n", e.String())
+      consumer.Commit()
+    case tmqcommon.Error:
+      fmt.Printf("%% Error: %v: %v\n", e.Code(), e)
+      return
+    default:
+      fmt.Printf("unexpected event:%v\n", e)
+      return
+    }
+  }
+}
+```
+
+</TabItem>
+
+<TabItem value="Rust" label="Rust">
+
+```rust
+// consume loop
+consumer
+    .stream()
+    .try_for_each_concurrent(10, |(offset, message)| async {
+      let topic = offset.topic();
+      // the vgroup id, like partition id in kafka.
+      let vgroup_id = offset.vgroup_id();
+      println!("* in vgroup id {vgroup_id} of topic {topic}\n");
+
+      if let Some(data) = message.into_data() {
+        while let Some(block) = data.fetch_raw_block().await? {
+          // A two-dimension matrix while each cell is a [taos::Value] object.
+          let values = block.to_values();
+          // Number of rows.
+          assert_eq!(values.len(), block.nrows());
+          // Number of columns
+          assert_eq!(values[0].len(), block.ncols());
+          println!("first row: {}", values[0].iter().join(", "));
+        }
+      }
+      consumer.commit(offset).await?;
+      Ok(())
+    })
+    .await?;
+```
+
+</TabItem>
+<TabItem value="Python" label="Python">
+
+```python
+while 1:
+  message = consumer.poll(timeout=1.0)
+  if message:
+    id = message.vgroup()
+    topic = message.topic()
+    database = message.database()
+
+    for block in message:
+      nrows = block.nrows()
+      ncols = block.ncols()
+      for row in block:
+          print(row)
+      values = block.fetchall()
+      print(nrows, ncols)
+  else:
+    break
+```
+
+</TabItem>
+</Tabs>
+
+## Close the consumer
+
+After message consumption is finished, the consumer is unsubscribed.
+
+<Tabs defaultValue="Go" groupId="lang">
+
+<TabItem value="Go" label="Go">
+
+```go
+consumer.Close()
+```
+
+</TabItem>
+
+<TabItem value="Rust" label="Rust">
+
+```rust
+consumer.unsubscribe().await;
+```
+
+</TabItem>
+
+<TabItem value="Python" label="Python">
+
+```py
+# Unsubscribe
+consumer.unsubscribe()
+# Close consumer
+consumer.close()
+```
+
+</TabItem>
+
+</Tabs>
+
+### Sample Code
+
+The following are full sample codes about how to consume the shared topic **test**:
+
+<Tabs defaultValue="Go" groupId="lang">
+
+<TabItem label="Go" value="Go">
+
+```go
+{{#include docs/examples/go/sub/cloud/main.go}}
+```
+
+</TabItem>
+
+<TabItem label="Rust" value="Rust">
+
+```rust
+{{#include docs/examples/rust/cloud-example/examples/sub.rs}}
+```
+
+</TabItem>
+
+<TabItem value="Python" label="Python">
+
+```python
+{{#include docs/examples/python/cloud/sub.py}}
+```
+
+</TabItem>
+
+</Tabs>

docs/en/11-tmq.md

----
-sidebar_label: Subscription
-title: Data Subscritpion
-description: Use data subscription to get data from TDengine.
----
-
-import Tabs from "@theme/Tabs";
-import TabItem from "@theme/TabItem";
-
-This topic introduces how to read out data from TDengine using data subscription, which is an advanced feature in TDengine. To access the data in TDengine in data subscription way, you need to create topic, create consumer, subscribe to a topic, and consume data. In this document we will briefly explain these main steps of data subscription.
-
-## Create Topic
-
-A topic can be created on a database, on some selected columns,or on a supertable. 
-
-### Topic on Columns
-
-The most common way to create a topic is to create a topic on some specifically selected columns. The Syntax is like below:
-
-```sql
-CREATE TOPIC topic_name as subquery;
-```
-
-You can subscribe to a topic through a SELECT statement. Statements that specify columns, such as `SELECT *` and `SELECT ts, cl` are supported, as are filtering conditions and scalar functions. Aggregate functions and time window aggregation are not supported. Note:
-
-- The schema of topics created in this manner is determined by the subscribed data.
-- You cannot modify (`ALTER <table> MODIFY`) or delete (`ALTER <table> DROP`) columns or tags that are used in a subscription or calculation.
-- Columns added to a table after the subscription is created are not displayed in the results. Deleting columns will cause an error.
-
-For example:
-
-```sql
-CREATE TOPIC topic_name AS SELECT ts, c1, c2, c3 FROM tmqdb.stb WHERE c1 > 1;
-```
-
-### Topic on SuperTable
-
-Syntax:
-
-```sql
-CREATE TOPIC topic_name AS STABLE stb_name;
-```
-
-Creating a topic in this manner differs from a `SELECT * from stbName` statement as follows:
-
-- The table schema can be modified.
-- Unstructured data is returned. The format of the data returned changes based on the supertable schema.
-- A different table schema may exist for every data block to be processed.
-- The data returned does not include tags.
-
-### Topic on Database
-
-Syntax:
-
-```sql
-CREATE TOPIC topic_name [WITH META] AS DATABASE db_name;
-```
-
-This SQL statement creates a subscription to all tables in the database. You can add the `WITH META` parameter to include schema changes in the subscription, including creating and deleting supertables; adding, deleting, and modifying columns; and creating, deleting, and modifying the tags of subtables. Consumers can determine the message type from the API. Note that this differs from Kafka.
-
-## Programming Model
-
-To subscribe the data from a created topic, the client program needs to follow the programming model described in this section.
-
-1. Create Consumer
-
-To create a consumer, you must use the APIs provided by TDengine connectors. Below is the sample code of using connectors of different languages.
-
-2. Subscribe to a Topic
-
-A single consumer can subscribe to multiple topics.
-
-3. Consume messages
-
-4. Subscribe to a Topic
-
-A single consumer can subscribe to multiple topics.
-
-5. Consume Data
-
-6. Close the consumer
-
-After message consumption is finished, the consumer is unsubscribed.
-
-## Sample Code
-
-<Tabs defaultValue="Rust" groupId="lang">
-
-<TabItem value="c" label="C">
-
-Will be available soon
-
-</TabItem>
-<TabItem value="java" label="Java">
-
-Will be available soon
-
-</TabItem>
-
-<TabItem label="Go" value="Go">
-
-Will be available soon
-
-</TabItem>
-
-<TabItem label="Rust" value="Rust">
-
-```rust
-{{#include docs/examples/rust/cloud-example/examples/subscribe_demo.rs}}
-```
-
-</TabItem>
-
-<TabItem value="Python" label="Python">
-
-Will be available soon
-
-</TabItem>
-
-<TabItem label="Node.JS" value="Node.JS">
-
-Will be available soon
-
-</TabItem>
-
-<TabItem value="C#" label="C#">
-
-Will be available soon
-
-</TabItem>
-
-</Tabs>
-
-## Delete Topic
-
-Once a topic becomes useless, it can be deleted.
-
-You can delete topics that are no longer useful. Note that you must unsubscribe all consumers from a topic before deleting it.
-
-```sql
-DROP TOPIC topic_name;
-```
-
-## Check Status
-
-At any time, you can check the status of existing topics and consumers.
-
-1. Query all existing topics.
-
-```sql
-SHOW TOPICS;
-```
-
-2. Query the status and subscribed topics of all consumers.
-
-```sql
-SHOW CONSUMERS;
-```

docs/en/15-programming/01-connect/03-go.md

@@ -54,8 +54,8 @@ $env:TDENGINE_GO_DSN="<goDSN>"
 
 <!-- exclude -->
 :::note
-Replace <goDSN\> with the real value, the format should be `https(<cloud_host>)/?token=<token>`.
-To obtain the value of `goDSN`, please log in [TDengine Cloud](https://cloud.tdengine.com) and click "Data In" on the lef menu.
+Replace <goDSN\> with the real value, the format should be `https(<cloud_endpoint>)/?token=<token>`.
+To obtain the value of `goDSN`, please log in [TDengine Cloud](https://cloud.tdengine.com) and click "Data In" on the left menu.
 
 :::
 <!-- exclude-end -->

docs/examples/csharp/subscribe/Program.cs

+using System;
+using TDengineTMQ;
+using TDengineDriver;
+using System.Runtime.InteropServices;
+
+namespace TMQExample
+{
+    internal class SubscribeDemo
+    {
+        static void Main(string[] args)
+        {
+            IntPtr conn = GetConnection();
+            string topic = "topic_example";
+            //create topic 
+            IntPtr res = TDengine.Query(conn, $"create topic if not exists {topic} as select * from meters");
+
+            if (TDengine.ErrorNo(res) != 0 )
+            {
+                throw new Exception($"create topic failed, reason:{TDengine.Error(res)}");
+            }
+
+            var cfg = new ConsumerConfig
+            {
+                GourpId = "group_1",
+                TDConnectUser = "root",
+                TDConnectPasswd = "taosdata",
+                MsgWithTableName = "true",
+                TDConnectIp = "127.0.0.1",
+            };
+
+            // create consumer 
+            var consumer = new ConsumerBuilder(cfg)
+                .Build();
+
+            // subscribe
+            consumer.Subscribe(topic);
+
+            // consume 
+            for (int i = 0; i < 5; i++)
+            {
+                var consumeRes = consumer.Consume(300);
+                // print consumeResult
+                foreach (KeyValuePair<TopicPartition, TaosResult> kv in consumeRes.Message)
+                {
+                    Console.WriteLine("topic partitions:\n{0}", kv.Key.ToString());
+
+                    kv.Value.Metas.ForEach(meta =>
+                    {
+                        Console.Write("{0} {1}({2}) \t|", meta.name, meta.TypeName(), meta.size);
+                    });
+                    Console.WriteLine("");
+                    kv.Value.Datas.ForEach(data =>
+                    {
+                        Console.WriteLine(data.ToString());
+                    });
+                }
+
+                consumer.Commit(consumeRes);
+                Console.WriteLine("\n================ {0} done ", i);
+
+            }
+
+            // retrieve topic list
+            List<string> topics = consumer.Subscription();
+            topics.ForEach(t => Console.WriteLine("topic name:{0}", t));
+
+            // unsubscribe
+            consumer.Unsubscribe();
+
+            // close consumer after use.Otherwise will lead memory leak.
+            consumer.Close();
+            TDengine.Close(conn);
+
+        }
+
+        static IntPtr GetConnection()
+        {
+            string host = "localhost";
+            short port = 6030;
+            string username = "root";
+            string password = "taosdata";
+            string dbname = "power";
+            var conn = TDengine.Connect(host, username, password, dbname, port);
+            if (conn == IntPtr.Zero)
+            {
+                throw new Exception("Connect to TDengine failed");
+            }
+            else
+            {
+                Console.WriteLine("Connect to TDengine success");
+            }
+            return conn;
+        }
+    }
+
+}

docs/examples/csharp/subscribe/subscribe.csproj

+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <TargetFramework>net6.0</TargetFramework>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <Nullable>enable</Nullable>
+    <StartupObject>TMQExample.SubscribeDemo</StartupObject>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="TDengine.Connector" Version="3.0.1" />
+  </ItemGroup>
+
+</Project>

docs/examples/go/sub/cloud/main.go

+package main
+
+import (
+	"fmt"
+	"github.com/taosdata/driver-go/v3/common"
+	tmqcommon "github.com/taosdata/driver-go/v3/common/tmq"
+	"github.com/taosdata/driver-go/v3/ws/tmq"
+	"os"
+)
+
+func main() {
+	tmqStr := os.Getenv("TDENGINE_CLOUD_TMQ")
+	consumer, err := tmq.NewConsumer(&tmqcommon.ConfigMap{
+		"ws.url":                tmqStr,
+		"ws.message.channelLen": uint(0),
+		"ws.message.timeout":    common.DefaultMessageTimeout,
+		"ws.message.writeWait":  common.DefaultWriteWait,
+		"group.id":              "test_group",
+		"client.id":             "test_consumer_ws",
+		"auto.offset.reset":     "earliest",
+	})
+	if err != nil {
+		panic(err)
+	}
+	err = consumer.Subscribe("test", nil)
+	if err != nil {
+		panic(err)
+	}
+	defer consumer.Close()
+	for {
+		ev := consumer.Poll(0)
+		if ev != nil {
+			switch e := ev.(type) {
+			case *tmqcommon.DataMessage:
+				fmt.Printf("get message:%v\n", e.String())
+				consumer.Commit()
+			case tmqcommon.Error:
+				fmt.Printf("%% Error: %v: %v\n", e.Code(), e)
+				return
+			default:
+				fmt.Printf("unexpected event:%v\n", e)
+				return
+			}
+		}
+	}
+}

docs/examples/python/cloud/sub.py

+#!/usr/bin/env python
+import os
+from taosws import Consumer
+
+endpoint = os.environ["TDENGINE_CLOUD_ENDPOINT"]
+token = os.environ["TDENGINE_CLOUD_TOKEN"]
+urlparts = endpoint.split(":", 1)
+
+conf = {
+    # auth options
+    "td.connect.websocket.scheme": "ws",
+    "td.connect.ip": urlparts[0],
+    "td.connect.port": urlparts[1],
+    "td.connect.token": token,
+    # consume options
+    "group.id": "test_group_py",
+    "client.id": "test_consumer_ws_py",
+}
+consumer = Consumer(conf)
+
+consumer.subscribe(["test"])
+
+while 1:
+  message = consumer.poll(timeout=1.0)
+  if message:
+      id = message.vgroup()
+      topic = message.topic()
+      database = message.database()
+
+      for block in message:
+          nrows = block.nrows()
+          ncols = block.ncols()
+          for row in block:
+              print(row)
+          values = block.fetchall()
+          print(nrows, ncols)
+  else:
+      break
+
+consumer.close()
\ No newline at end of file

docs/examples/python/tmq_example.py

 import taos
-from taos.tmq import TaosConsumer
-consumer = TaosConsumer('topic_ctb_column', group_id='vg2')
-for msg in consumer:
-    for row in msg:
-        print(row)
+from taos.tmq import *
+
+conn = taos.connect()
+
+print("init")
+conn.execute("drop topic if exists topic_ctb_column")
+conn.execute("drop database if exists py_tmq")
+conn.execute("create database if not exists py_tmq vgroups 2")
+conn.select_db("py_tmq")
+conn.execute(
+    "create stable if not exists stb1 (ts timestamp, c1 int, c2 float, c3 binary(10)) tags(t1 int)"
+)
+conn.execute("create table if not exists tb1 using stb1 tags(1)")
+conn.execute("create table if not exists tb2 using stb1 tags(2)")
+conn.execute("create table if not exists tb3 using stb1 tags(3)")
+
+print("create topic")
+conn.execute(
+    "create topic if not exists topic_ctb_column as select ts, c1, c2, c3 from stb1"
+)
+
+print("build consumer")
+conf = TaosTmqConf()
+conf.set("group.id", "tg2")
+conf.set("td.connect.user", "root")
+conf.set("td.connect.pass", "taosdata")
+conf.set("enable.auto.commit", "true")
+
+
+def tmq_commit_cb_print(tmq, resp, offset, param=None):
+    print(f"commit: {resp}, tmq: {tmq}, offset: {offset}, param: {param}")
+
+
+conf.set_auto_commit_cb(tmq_commit_cb_print, None)
+tmq = conf.new_consumer()
+
+print("build topic list")
+
+topic_list = TaosTmqList()
+topic_list.append("topic_ctb_column")
+
+print("basic consume loop")
+tmq.subscribe(topic_list)
+
+sub_list = tmq.subscription()
+
+print("subscribed topics: ", sub_list)
+
+while 1:
+    res = tmq.poll(1000)
+    if res:
+        topic = res.get_topic_name()
+        vg = res.get_vgroup_id()
+        db = res.get_db_name()
+        print(f"topic: {topic}\nvgroup id: {vg}\ndb: {db}")
+        for row in res:
+            print(row)

docs/examples/rust/cloud-example/examples/sub.rs

+use taos::*;
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+    // subscribe
+    let tmq_str = std::env::var("TDENGINE_CLOUD_TMQ")?;
+    let tmq_uri = format!( "{}&group.id=test_group_rs&client.id=test_consumer_ws", tmq_str);
+    println!("request tmq URI is {tmq_uri}\n");
+    let tmq = TmqBuilder::from_dsn(tmq_uri,)?;
+    let mut consumer = tmq.build()?;
+    consumer.subscribe(["test"]).await?;
+
+    // consume loop
+    consumer
+        .stream()
+        .try_for_each_concurrent(10, |(offset, message)| async {
+            let topic = offset.topic();
+            // the vgroup id, like partition id in kafka.
+            let vgroup_id = offset.vgroup_id();
+            println!("* in vgroup id {vgroup_id} of topic {topic}\n");
+
+            if let Some(data) = message.into_data() {
+                while let Some(block) = data.fetch_raw_block().await? {
+                    // A two-dimension matrix while each cell is a [taos::Value] object.
+                    let values = block.to_values();
+                    // Number of rows.
+                    assert_eq!(values.len(), block.nrows());
+                    // Number of columns
+                    assert_eq!(values[0].len(), block.ncols());
+                    println!("first row: {}", values[0].iter().join(", "));
+                }
+            }
+            consumer.commit(offset).await?;
+            Ok(())
+        })
+        .await?;
+
+    consumer.unsubscribe().await;
+
+    Ok(())
+}
\ No newline at end of file

docs/examples/rust/cloud-example/examples/subscribe_demo.rs

@@ -18,15 +18,14 @@ struct Record {
 
 async fn prepare(taos: Taos) -> anyhow::Result<()> {
     let inserted = taos.exec_many([
-        "use tmq",
         // create child table
-        "CREATE TABLE `d0` USING `meters` TAGS(0, 'Los Angles')",
+        "CREATE TABLE `d0` USING `meters` TAGS(0, 'California.LosAngles')",
         // insert into child table
         "INSERT INTO `d0` values(now - 10s, 10, 116, 0.32)",
         // insert with NULL values
         "INSERT INTO `d0` values(now - 8s, NULL, NULL, NULL)",
         // insert and automatically create table with tags if not exists
-        "INSERT INTO `d1` USING `meters` TAGS(1, 'San Francisco') values(now - 9s, 10.1, 119, 0.33)",
+        "INSERT INTO `d1` USING `meters` TAGS(1, 'California.SanFrancisco') values(now - 9s, 10.1, 119, 0.33)",
         // insert many records in a single sql
         "INSERT INTO `d1` values (now-8s, 10, 120, 0.33) (now - 6s, 10, 119, 0.34) (now - 4s, 11.2, 118, 0.322)",
     ]).await?;
@@ -36,20 +35,22 @@ async fn prepare(taos: Taos) -> anyhow::Result<()> {
 
 #[tokio::main]
 async fn main() -> anyhow::Result<()> {
-    std::env::set_var("RUST_LOG", "debug");
-    pretty_env_logger::init();
-    let dsn = std::env::var("TDENGINE_CLOUD_DSN")?;
-
-    let builder = TaosBuilder::from_dsn(&dsn)?;
+    let dsn = "taos://localhost:6030";
+    let builder = TaosBuilder::from_dsn(dsn)?;
 
     let taos = builder.build()?;
+    let db = "tmq";
 
     // prepare database
     taos.exec_many([
-        "DROP TOPIC IF EXISTS tmq_meters",
-        "USE tmq",
-        "CREATE STABLE IF NOT EXISTS `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) TAGS (`groupid` INT, `location` BINARY(16))",
-        "CREATE TOPIC tmq_meters with META AS DATABASE tmq"
+        format!("DROP TOPIC IF EXISTS tmq_meters"),
+        format!("DROP DATABASE IF EXISTS `{db}`"),
+        format!("CREATE DATABASE `{db}`"),
+        format!("USE `{db}`"),
+        // create super table
+        format!("CREATE TABLE `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) TAGS (`groupid` INT, `location` BINARY(24))"),
+        // create topic for subscription
+        format!("CREATE TOPIC tmq_meters AS SELECT * FROM `meters`")
     ])
     .await?;
 
@@ -58,21 +59,14 @@ async fn main() -> anyhow::Result<()> {
     tokio::time::sleep(Duration::from_secs(1)).await;
 
     // subscribe
-    let dsn2 = format!("{dsn}&group.id=test");
-    dbg!(&dsn2);
-    let tmq = TmqBuilder::from_dsn(dsn2)?;
+    let tmq = TmqBuilder::from_dsn("taos://localhost:6030/?group.id=test")?;
 
     let mut consumer = tmq.build()?;
     consumer.subscribe(["tmq_meters"]).await?;
-    println!("start subscription");
-
-    {
-        let mut stream = consumer.stream();
-
-        while let Some((offset, message)) = stream.try_next().await? {
-            // get information from offset
 
-            // the topic
+    consumer
+        .stream()
+        .try_for_each(|(offset, message)| async {
             let topic = offset.topic();
             // the vgroup id, like partition id in kafka.
             let vgroup_id = offset.vgroup_id();
@@ -80,20 +74,14 @@ async fn main() -> anyhow::Result<()> {
 
             if let Some(data) = message.into_data() {
                 while let Some(block) = data.fetch_raw_block().await? {
-                    // one block for one table, get table name if needed
-                    let name = block.table_name();
                     let records: Vec<Record> = block.deserialize().try_collect()?;
-                    println!(
-                        "** table: {}, got {} records: {:#?}\n",
-                        name.unwrap(),
-                        records.len(),
-                        records
-                    );
+                    println!("** read {} records: {:#?}\n", records.len(), records);
                 }
             }
             consumer.commit(offset).await?;
-        }
-    }
+            Ok(())
+        })
+        .await?;
 
     consumer.unsubscribe().await;
 

taosadapter @ 566540d4

+Subproject commit 566540d4a7f59d859378ff4ae7bb64ed4f0bc096

taosws-rs @ f406d516

+Subproject commit f406d516dfab06979ca02e4f0abfc4b924264a1d