mirror of
https://github.com/alibaba/DataX.git
synced 2025-05-02 17:09:36 +08:00
197 lines
6.8 KiB
Markdown
197 lines
6.8 KiB
Markdown
# DataX TDengineWriter
|
|
|
|
[简体中文](./tdengine30writer-CN.md) | English
|
|
|
|
## 1 Quick Introduction
|
|
|
|
The TDengineWriter plugin enables writing data to the target table of the TDengine database. At the bottom level, TDengineWriter connects TDengine through JDBC, executes insert statement /schemaless statement according to TDengine SQL syntax, and writes data to TDengine.
|
|
|
|
TDengineWriter can be used as a data migration tool for DBAs to import data from other databases into TDengine.
|
|
|
|
|
|
|
|
## 2 Implementation
|
|
|
|
TDengineWriter obtains the protocol data generated by Reader through DataX framework, connects to TDengine through JDBC Driver, executes insert statement /schemaless statement, and writes the data to TDengine.
|
|
|
|
In TDengine, table can be divided into super table, sub-table and ordinary table. Super table and sub-table include Colum and Tag. The value of tag column of sub-table is fixed value. (details please refer to: [data model](https://www.taosdata.com/docs/cn/v2.0/architecture#model))
|
|
|
|
The TDengineWriter can write data to super tables, sub-tables, and ordinary tables using the following methods based on the type of the table and whether the column parameter contains TBName:
|
|
|
|
1. Table is a super table and column specifies tbname: use the automatic insert statement to create the table and use tbname as the name of the sub-table.
|
|
|
|
2. Table is a super table and column does not contain tbname: use schemaless to write the table. TDengine will auto-create a tbname based on the super table name and tag value.
|
|
|
|
3. Table is a sub-table: Use insert statement to write, ignoreTagUnmatched parameter is true, ignore data in record whose tag value is inconsistent with that of table.
|
|
|
|
4. Table is a common table: use insert statement to write data.
|
|
|
|
|
|
## 3 Features Introduction
|
|
### 3.1 Sample
|
|
Configure a job to write to TDengine
|
|
|
|
Create a supertable on TDengine:
|
|
|
|
```sql
|
|
create database if not exists test;
|
|
create table test.weather (ts timestamp, temperature int, humidity double) tags(is_normal bool, device_id binary(100), address nchar(100));
|
|
```
|
|
|
|
Write data to TDengine using the following Job configuration:
|
|
|
|
```json
|
|
{
|
|
"job": {
|
|
"content": [
|
|
{
|
|
"reader": {
|
|
"name": "streamreader",
|
|
"parameter": {
|
|
"column": [
|
|
{
|
|
"type": "string",
|
|
"value": "tb1"
|
|
},
|
|
{
|
|
"type": "date",
|
|
"value": "2022-02-20 12:00:01"
|
|
},
|
|
{
|
|
"type": "long",
|
|
"random": "0, 10"
|
|
},
|
|
{
|
|
"type": "double",
|
|
"random": "0, 10"
|
|
},
|
|
{
|
|
"type": "bool",
|
|
"random": "0, 50"
|
|
},
|
|
{
|
|
"type": "bytes",
|
|
"value": "abcABC123"
|
|
},
|
|
{
|
|
"type": "string",
|
|
"value": "北京朝阳望京"
|
|
}
|
|
],
|
|
"sliceRecordCount": 1
|
|
}
|
|
},
|
|
"writer": {
|
|
"name": "tdengine30writer",
|
|
"parameter": {
|
|
"username": "root",
|
|
"password": "taosdata",
|
|
"column": [
|
|
"tbname",
|
|
"ts",
|
|
"temperature",
|
|
"humidity",
|
|
"is_normal",
|
|
"device_id",
|
|
"address"
|
|
],
|
|
"connection": [
|
|
{
|
|
"table": [
|
|
"weather"
|
|
],
|
|
"jdbcUrl": "jdbc:TAOS-RS://192.168.56.105:6041/test"
|
|
}
|
|
],
|
|
"batchSize": 100,
|
|
"ignoreTagsUnmatched": true
|
|
}
|
|
}
|
|
}
|
|
],
|
|
"setting": {
|
|
"speed": {
|
|
"channel": 1
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3.2 Configuration
|
|
|
|
* jdbcUrl
|
|
* Descrption: Data source JDBC connection information, TDengine JDBC information please refer to: [Java connector](https://www.taosdata.com/docs/cn/v2.0/connector/java#url)
|
|
* Required: yes
|
|
* Default: none
|
|
* username
|
|
* Descrption: username
|
|
* Required: yes
|
|
* Default: none
|
|
|
|
* password
|
|
* Descrption: password of username
|
|
* Required: yes
|
|
* Default: none
|
|
|
|
* table
|
|
* Descrption: A list of table names that should contain all of the columns in the column parameter (except tbname). Note that tbname in column is used as the TDengine sub-table name.
|
|
* Required: yes
|
|
* Default: none
|
|
|
|
* column
|
|
* Descrption: A list of field names, the order of the fields should be the column in the record
|
|
* Required: yes
|
|
* Default: none
|
|
|
|
* batchSize
|
|
* Descrption: Each batchSize record is written to a batch
|
|
* Required: no
|
|
* Default: 1
|
|
|
|
* ignoreTagsUnmatched
|
|
* Descrption: When table is a sub-table in TDengine, table has a tag value. If the tag value of the data and the tag value of the table are not equal, the data is not written to the table.
|
|
* Required: no
|
|
* Default: false
|
|
|
|
|
|
#### 3.3 Type Convert
|
|
|
|
Data types in datax that can be mapped to data types in TDengine
|
|
|
|
| DataX Type | TDengine Type |
|
|
| ---------- | ----------------------------------------- |
|
|
| INT | TINYINT, SMALLINT, INT |
|
|
| LONG | TIMESTAMP, TINYINT, SMALLINT, INT, BIGINT |
|
|
| DOUBLE | FLOAT, DOUBLE |
|
|
| STRING | TIMESTAMP, BINARY, NCHAR |
|
|
| BOOL | BOOL |
|
|
| DATE | TIMESTAMP |
|
|
| BYTES | BINARY |
|
|
|
|
### 3.2 From MongoDB to TDengine
|
|
|
|
Here are some examples of data sources migrating to TDengine
|
|
|
|
| Sample | Configuration |
|
|
| -------------------- | ------------------------------------------------------------ |
|
|
| TDengine to TDengine | [super table to super table with tbname](../src/test/resources/t2t-1.json) |
|
|
| TDengine to TDengine | [super table to super table without tbname](../src/test/resources/t2t-2.json) |
|
|
| TDengine to TDengine | [super table to sub-table](../src/test/resources/t2t-3.json) |
|
|
| TDengine to TDengine | [table to table](../src/test/resources/t2t-4.json) |
|
|
| RDBMS to TDengine | [table to super table with tbname](../src/test/resources/dm2t-1.json) |
|
|
| RDBMS to TDengine | [table to super table without tbname](../src/test/resources/dm2t-2.json) |
|
|
| RDBMS to TDengine | [table to sub-table](../src/test/resources/dm2t-3.json) |
|
|
| RDBMS to TDengine | [table to table](../src/test/resources/dm2t-4.json) |
|
|
| OpenTSDB to TDengine | [metric to table](../src/test/resources/o2t-1.json) |
|
|
|
|
## 4 Restriction
|
|
|
|
|
|
|
|
## FAQ
|
|
|
|
### Do columns in source table and columns in target table must be in the same order?
|
|
|
|
Yes, TDengineWriter parses the data from the Datax in the order of the fields in the column.
|