+ + * 默认值：无
+ +* **username** + + * 描述：数据源的用户名
+ + * 必选：是
+ + * 默认值：无
+ +* **password** + + * 描述：数据源指定用户名的密码
+ + * 必选：是
+ + * 默认值：无
+ +* **table** + + * 描述：所选取的需要同步的表。使用JSON的数组描述，因此支持多张表同时抽取。当配置为多张表时，用户自己需保证多张表是同一schema结构，ClickhouseReader不予检查表是否同一逻辑表。注意，table必须包含在connection配置单元中。
+ + * 必选：是
+ + * 默认值：无
+ +* **column** + + * 描述：所配置的表中需要同步的列名集合，使用JSON的数组描述字段信息。用户使用\*代表默认使用所有列配置，例如['\*']。 + + 支持列裁剪，即列可以挑选部分列进行导出。 + + 支持列换序，即列可以不按照表schema信息进行导出。 + + 支持常量配置，用户需要按照JSON格式: + ["id", "`table`", "1", "'bazhen.csy'", "null", "to_char(a + 1)", "2.3" , "true"] + id为普通列名，\`table\`为包含保留在的列名，1为整形数字常量，'bazhen.csy'为字符串常量，null为空指针，to_char(a + 1)为表达式，2.3为浮点数，true为布尔值。 + + Column必须显示填写，不允许为空！ + + * 必选：是
+ + * 默认值：无
+ +* **splitPk** + + * 描述：ClickhouseReader进行数据抽取时，如果指定splitPk，表示用户希望使用splitPk代表的字段进行数据分片，DataX因此会启动并发任务进行数据同步，这样可以大大提供数据同步的效能。 + + 推荐splitPk用户使用表主键，因为表主键通常情况下比较均匀，因此切分出来的分片也不容易出现数据热点。 + + 目前splitPk仅支持整形数据切分，`不支持浮点、日期等其他类型`。如果用户指定其他非支持类型，ClickhouseReader将报错！ + + splitPk如果不填写，将视作用户不对单表进行切分，ClickhouseReader使用单通道同步全量数据。 + + * 必选：否
+ + * 默认值：无
+ +* **where** + + * 描述：筛选条件，MysqlReader根据指定的column、table、where条件拼接SQL，并根据这个SQL进行数据抽取。在实际业务场景中，往往会选择当天的数据进行同步，可以将where条件指定为gmt_create > $bizdate 。注意：不可以将where条件指定为limit 10，limit不是SQL的合法where子句。
+ + where条件可以有效地进行业务增量同步。 + + * 必选：否
+ + * 默认值：无
+ +* **querySql** + + * 描述：在有些业务场景下，where这一配置项不足以描述所筛选的条件，用户可以通过该配置型来自定义筛选SQL。当用户配置了这一项之后，DataX系统就会忽略table，column这些配置型，直接使用这个配置项的内容对数据进行筛选，例如需要进行多表join后同步数据，使用select a,b from table_a join table_b on table_a.id = table_b.id
+ + `当用户配置querySql时，ClickhouseReader直接忽略table、column、where条件的配置`。 + + * 必选：否
+ + * 默认值：无
+ +* **fetchSize** + + * 描述：该配置项定义了插件和数据库服务器端每次批量数据获取条数，该值决定了DataX和服务器端的网络交互次数，能够较大的提升数据抽取性能。
+ + `注意，该值过大(>2048)可能造成DataX进程OOM。`。 + + * 必选：否
+ + * 默认值：1024
+ +* **session** + + * 描述：控制写入数据的时间格式，时区等的配置，如果表中有时间字段，配置该值以明确告知写入 clickhouse 的时间格式。通常配置的参数为：NLS_DATE_FORMAT,NLS_TIME_FORMAT。其配置的值为 json 格式，例如： +``` +"session": [ + "alter session set NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'", + "alter session set NLS_TIMESTAMP_FORMAT='yyyy-mm-dd hh24:mi:ss'", + "alter session set NLS_TIMESTAMP_TZ_FORMAT='yyyy-mm-dd hh24:mi:ss'", + "alter session set TIME_ZONE='US/Pacific'" + ] +``` + `(注意"是 " 的转义字符串)`。 + + * 必选：否
+ + * 默认值：无
+ + +### 3.3 类型转换 + +目前ClickhouseReader支持大部分Clickhouse类型，但也存在部分个别类型没有支持的情况，请注意检查你的类型。 + +下面列出ClickhouseReader针对Clickhouse类型转换列表: + + +| DataX 内部类型| Clickhouse 数据类型 | +| -------- |--------------------------------------------------------------------------------------------| +| Long | UInt8, UInt16, UInt32, UInt64, UInt128, UInt256, Int8, Int16, Int32, Int64, Int128, Int256 | +| Double | Float32, Float64, Decimal | +| String | String, FixedString | +| Date | DATE, Date32, DateTime, DateTime64 | +| Boolean | Boolean | +| Bytes | BLOB,BFILE,RAW,LONG RAW | + + + +请注意: + +* `除上述罗列字段类型外，其他类型均不支持`。 + + +## 4 性能报告 + +### 4.1 环境准备 + +#### 4.1.1 数据特征 + +为了模拟线上真实数据，我们设计两个Clickhouse数据表，分别为: + +#### 4.1.2 机器参数 + +* 执行DataX的机器参数为: + +* Clickhouse数据库机器参数为: + +### 4.2 测试报告 + +#### 4.2.1 表1测试报告 + + +| 并发任务数| DataX速度(Rec/s)|DataX流量|网卡流量|DataX运行负载|DB运行负载| +|--------| --------|--------|--------|--------|--------| +|1| DataX 统计速度(Rec/s)|DataX统计流量|网卡流量|DataX运行负载|DB运行负载| + +## 5 约束限制 + +### 5.1 主备同步数据恢复问题 + +主备同步问题指Clickhouse使用主从灾备，备库从主库不间断通过binlog恢复数据。由于主备数据同步存在一定的时间差，特别在于某些特定情况，例如网络延迟等问题，导致备库同步恢复的数据与主库有较大差别，导致从备库同步的数据不是一份当前时间的完整镜像。 + +针对这个问题，我们提供了preSql功能，该功能待补充。 + +### 5.2 一致性约束 + +Clickhouse在数据存储划分中属于RDBMS系统，对外可以提供强一致性数据查询接口。例如当一次同步任务启动运行过程中，当该库存在其他数据写入方写入数据时，ClickhouseReader完全不会获取到写入更新数据，这是由于数据库本身的快照特性决定的。关于数据库快照特性，请参看[MVCC Wikipedia](https://en.wikipedia.org/wiki/Multiversion_concurrency_control) + +上述是在ClickhouseReader单线程模型下数据同步一致性的特性，由于ClickhouseReader可以根据用户配置信息使用了并发数据抽取，因此不能严格保证数据一致性：当ClickhouseReader根据splitPk进行数据切分后，会先后启动多个并发任务完成数据同步。由于多个并发任务相互之间不属于同一个读事务，同时多个并发任务存在时间间隔。因此这份数据并不是`完整的`、`一致的`数据快照信息。 + +针对多线程的一致性快照需求，在技术上目前无法实现，只能从工程角度解决，工程化的方式存在取舍，我们提供几个解决思路给用户，用户可以自行选择： + +1. 使用单线程同步，即不再进行数据切片。缺点是速度比较慢，但是能够很好保证一致性。 + +2. 关闭其他数据写入方，保证当前数据为静态数据，例如，锁表、关闭备库同步等等。缺点是可能影响在线业务。 + +### 5.3 数据库编码问题 + + +ClickhouseReader底层使用JDBC进行数据抽取，JDBC天然适配各类编码，并在底层进行了编码转换。因此ClickhouseReader不需用户指定编码，可以自动获取编码并转码。 + +对于Clickhouse底层写入编码和其设定的编码不一致的混乱情况，ClickhouseReader对此无法识别，对此也无法提供解决方案，对于这类情况，`导出有可能为乱码`。 + +### 5.4 增量数据同步 + +ClickhouseReader使用JDBC SELECT语句完成数据抽取工作，因此可以使用SELECT...WHERE...进行增量数据抽取，方式有多种： + +* 数据库在线应用写入数据库时，填充modify字段为更改时间戳，包括新增、更新、删除(逻辑删)。对于这类应用，ClickhouseReader只需要WHERE条件跟上一同步阶段时间戳即可。 +* 对于新增流水型数据，ClickhouseReader可以WHERE条件后跟上一阶段最大自增ID即可。 + +对于业务上无字段区分新增、修改数据情况，ClickhouseReader也无法进行增量数据同步，只能同步全量数据。 + +### 5.5 Sql安全性 + +ClickhouseReader提供querySql语句交给用户自己实现SELECT抽取语句，ClickhouseReader本身对querySql不做任何安全性校验。这块交由DataX用户方自己保证。 + +## 6 FAQ + +*** + +**Q: ClickhouseReader同步报错，报错信息为XXX** + + A: 网络或者权限问题，请使用Clickhouse命令行测试 + + +如果上述命令也报错，那可以证实是环境问题，请联系你的DBA。 + + +**Q: ClickhouseReader抽取速度很慢怎么办？** + + A: 影响抽取时间的原因大概有如下几个：(来自专业 DBA 卫绾) + 1. 由于SQL的plan异常，导致的抽取时间长； 在抽取时，尽可能使用全表扫描代替索引扫描; + 2. 合理sql的并发度，减少抽取时间； + 3. 抽取sql要简单，尽量不用replace等函数，这个非常消耗cpu，会严重影响抽取速度; diff --git a/clickhousereader/pom.xml b/clickhousereader/pom.xml new file mode 100644 index 00000000..4b095796 --- /dev/null +++ b/clickhousereader/pom.xml @@ -0,0 +1,91 @@ + +
* a 库上有表：0, 1, 2
- * a 库上有表：3, 4
+ * b 库上有表：3, 4
* c 库上有表：5, 6, 7
*
* 如果有 4个 taskGroup
diff --git a/core/src/main/java/com/alibaba/datax/core/job/JobContainer.java b/core/src/main/java/com/alibaba/datax/core/job/JobContainer.java
index 26b2989f..49f5a0a1 100755
--- a/core/src/main/java/com/alibaba/datax/core/job/JobContainer.java
+++ b/core/src/main/java/com/alibaba/datax/core/job/JobContainer.java
@@ -27,7 +27,7 @@ import com.alibaba.datax.core.util.container.ClassLoaderSwapper;
import com.alibaba.datax.core.util.container.CoreConstant;
import com.alibaba.datax.core.util.container.LoadUtil;
import com.alibaba.datax.dataxservice.face.domain.enums.ExecuteMode;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import org.apache.commons.lang.StringUtils;
import org.apache.commons.lang.Validate;
import org.slf4j.Logger;
diff --git a/core/src/main/java/com/alibaba/datax/core/statistics/communication/CommunicationTool.java b/core/src/main/java/com/alibaba/datax/core/statistics/communication/CommunicationTool.java
index 51a601ae..1815ea02 100755
--- a/core/src/main/java/com/alibaba/datax/core/statistics/communication/CommunicationTool.java
+++ b/core/src/main/java/com/alibaba/datax/core/statistics/communication/CommunicationTool.java
@@ -2,7 +2,7 @@ package com.alibaba.datax.core.statistics.communication;
import com.alibaba.datax.common.statistics.PerfTrace;
import com.alibaba.datax.common.util.StrUtil;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import org.apache.commons.lang.Validate;
import java.text.DecimalFormat;
diff --git a/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/StdoutPluginCollector.java b/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/StdoutPluginCollector.java
index 8b2a8378..d88ad0a8 100755
--- a/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/StdoutPluginCollector.java
+++ b/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/StdoutPluginCollector.java
@@ -6,7 +6,7 @@ import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.core.statistics.communication.Communication;
import com.alibaba.datax.core.util.container.CoreConstant;
import com.alibaba.datax.core.statistics.plugin.task.util.DirtyRecord;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
diff --git a/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/util/DirtyRecord.java b/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/util/DirtyRecord.java
index 1b0d5238..caa4cb5b 100755
--- a/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/util/DirtyRecord.java
+++ b/core/src/main/java/com/alibaba/datax/core/statistics/plugin/task/util/DirtyRecord.java
@@ -4,7 +4,7 @@ import com.alibaba.datax.common.element.Column;
import com.alibaba.datax.common.element.Record;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.core.util.FrameworkErrorCode;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import java.math.BigDecimal;
import java.math.BigInteger;
diff --git a/core/src/main/java/com/alibaba/datax/core/taskgroup/TaskGroupContainer.java b/core/src/main/java/com/alibaba/datax/core/taskgroup/TaskGroupContainer.java
index c30c94d9..b4b45695 100755
--- a/core/src/main/java/com/alibaba/datax/core/taskgroup/TaskGroupContainer.java
+++ b/core/src/main/java/com/alibaba/datax/core/taskgroup/TaskGroupContainer.java
@@ -27,7 +27,7 @@ import com.alibaba.datax.core.util.TransformerUtil;
import com.alibaba.datax.core.util.container.CoreConstant;
import com.alibaba.datax.core.util.container.LoadUtil;
import com.alibaba.datax.dataxservice.face.domain.enums.State;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import org.apache.commons.lang3.Validate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java b/core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java
index e49c7878..5bce085f 100755
--- a/core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java
+++ b/core/src/main/java/com/alibaba/datax/core/transport/channel/memory/MemoryChannel.java
@@ -29,7 +29,7 @@ public class MemoryChannel extends Channel {
private ReentrantLock lock;
- private Condition notInsufficient, notEmpty;
+ private Condition notSufficient, notEmpty;
public MemoryChannel(final Configuration configuration) {
super(configuration);
@@ -37,7 +37,7 @@ public class MemoryChannel extends Channel {
this.bufferSize = configuration.getInt(CoreConstant.DATAX_CORE_TRANSPORT_EXCHANGER_BUFFERSIZE);
lock = new ReentrantLock();
- notInsufficient = lock.newCondition();
+ notSufficient = lock.newCondition();
notEmpty = lock.newCondition();
}
@@ -75,7 +75,7 @@ public class MemoryChannel extends Channel {
lock.lockInterruptibly();
int bytes = getRecordBytes(rs);
while (memoryBytes.get() + bytes > this.byteCapacity || rs.size() > this.queue.remainingCapacity()) {
- notInsufficient.await(200L, TimeUnit.MILLISECONDS);
+ notSufficient.await(200L, TimeUnit.MILLISECONDS);
}
this.queue.addAll(rs);
waitWriterTime += System.nanoTime() - startTime;
@@ -116,7 +116,7 @@ public class MemoryChannel extends Channel {
waitReaderTime += System.nanoTime() - startTime;
int bytes = getRecordBytes(rs);
memoryBytes.addAndGet(-bytes);
- notInsufficient.signalAll();
+ notSufficient.signalAll();
} catch (InterruptedException e) {
throw DataXException.asDataXException(
FrameworkErrorCode.RUNTIME_ERROR, e);
diff --git a/core/src/main/java/com/alibaba/datax/core/transport/record/DefaultRecord.java b/core/src/main/java/com/alibaba/datax/core/transport/record/DefaultRecord.java
index c78a2a87..1dfa02e8 100755
--- a/core/src/main/java/com/alibaba/datax/core/transport/record/DefaultRecord.java
+++ b/core/src/main/java/com/alibaba/datax/core/transport/record/DefaultRecord.java
@@ -5,7 +5,7 @@ import com.alibaba.datax.common.element.Record;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.core.util.ClassSize;
import com.alibaba.datax.core.util.FrameworkErrorCode;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import java.util.ArrayList;
import java.util.HashMap;
diff --git a/core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java b/core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java
index 20039864..24f43d55 100755
--- a/core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java
+++ b/core/src/main/java/com/alibaba/datax/core/util/ConfigParser.java
@@ -168,6 +168,7 @@ public final class ConfigParser {
boolean isDefaultPath = StringUtils.isBlank(pluginPath);
if (isDefaultPath) {
configuration.set("path", path);
+ configuration.set("loadType","jarLoader");
}
Configuration result = Configuration.newDefault();
diff --git a/core/src/main/java/com/alibaba/datax/core/util/container/CoreConstant.java b/core/src/main/java/com/alibaba/datax/core/util/container/CoreConstant.java
index 6a0b6205..a1ca164d 100755
--- a/core/src/main/java/com/alibaba/datax/core/util/container/CoreConstant.java
+++ b/core/src/main/java/com/alibaba/datax/core/util/container/CoreConstant.java
@@ -105,7 +105,7 @@ public class CoreConstant {
public static final String DATAX_JOB_POSTHANDLER_PLUGINNAME = "job.postHandler.pluginName";
// ----------------------------- 局部使用的变量
- public static final String JOB_WRITER = "reader";
+ public static final String JOB_WRITER = "writer";
public static final String JOB_READER = "reader";
diff --git a/core/src/main/java/com/alibaba/datax/core/util/container/JarLoader.java b/core/src/main/java/com/alibaba/datax/core/util/container/JarLoader.java
index 9fc113dc..ddf22bae 100755
--- a/core/src/main/java/com/alibaba/datax/core/util/container/JarLoader.java
+++ b/core/src/main/java/com/alibaba/datax/core/util/container/JarLoader.java
@@ -15,7 +15,7 @@ import java.util.List;
/**
* 提供Jar隔离的加载机制，会把传入的路径、及其子路径、以及路径中的jar文件加入到class path。
*/
-public class JarLoader extends URLClassLoader {
+public class JarLoader extends URLClassLoader{
public JarLoader(String[] paths) {
this(paths, JarLoader.class.getClassLoader());
}
diff --git a/core/src/main/java/com/alibaba/datax/core/util/container/LoadUtil.java b/core/src/main/java/com/alibaba/datax/core/util/container/LoadUtil.java
index 30e926c3..9a6a8302 100755
--- a/core/src/main/java/com/alibaba/datax/core/util/container/LoadUtil.java
+++ b/core/src/main/java/com/alibaba/datax/core/util/container/LoadUtil.java
@@ -49,7 +49,7 @@ public class LoadUtil {
/**
* jarLoader的缓冲
*/
- private static Map jarLoaderCenter = new HashMap();
+ private static Map jarLoaderCenter = new HashMap();
/**
* 设置pluginConfigs，方便后面插件来获取
diff --git a/databendwriter/doc/databendwriter-CN.md b/databendwriter/doc/databendwriter-CN.md
new file mode 100644
index 00000000..5b26ed7e
--- /dev/null
+++ b/databendwriter/doc/databendwriter-CN.md
@@ -0,0 +1,183 @@
+# DataX DatabendWriter
+[简体中文](./databendwriter-CN.md) | [English](./databendwriter.md)
+
+## 1 快速介绍
+
+Databend Writer 是一个 DataX 的插件，用于从 DataX 中写入数据到 Databend 表中。
+该插件基于[databend JDBC driver](https://github.com/databendcloud/databend-jdbc) ，它使用 [RESTful http protocol](https://databend.rs/doc/integrations/api/rest)
+在开源的 databend 和 [databend cloud](https://app.databend.com/) 上执行查询。
+
+在每个写入批次中，databend writer 将批量数据上传到内部的 S3 stage，然后执行相应的 insert SQL 将数据上传到 databend 表中。
+
+为了最佳的用户体验，如果您使用的是 databend 社区版本，您应该尝试采用 [S3](https://aws.amazon.com/s3/)/[minio](https://min.io/)/[OSS](https://www.alibabacloud.com/product/object-storage-service) 作为其底层存储层，因为
+它们支持预签名上传操作，否则您可能会在数据传输上浪费不必要的成本。
+
+您可以在[文档](https://databend.rs/doc/deploy/deploying-databend)中了解更多详细信息
+
+## 2 实现原理
+
+Databend Writer 将使用 DataX 从 DataX Reader 中获取生成的记录，并将记录批量插入到 databend 表中指定的列中。
+
+## 3 功能说明
+
+### 3.1 配置样例
+
+* 以下配置将从内存中读取一些生成的数据，并将数据上传到databend表中
+
+#### 准备工作
+```sql
+--- create table in databend
+drop table if exists datax.sample1;
+drop database if exists datax;
+create database if not exists datax;
+create table if not exsits datax.sample1(a string, b int64, c date, d timestamp, e bool, f string, g variant);
+```
+
+#### 配置样例
+```json
+{
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "streamreader",
+ "parameter": {
+ "column" : [
+ {
+ "value": "DataX",
+ "type": "string"
+ },
+ {
+ "value": 19880808,
+ "type": "long"
+ },
+ {
+ "value": "1926-08-08 08:08:08",
+ "type": "date"
+ },
+ {
+ "value": "1988-08-08 08:08:08",
+ "type": "date"
+ },
+ {
+ "value": true,
+ "type": "bool"
+ },
+ {
+ "value": "test",
+ "type": "bytes"
+ },
+ {
+ "value": "{\"type\": \"variant\", \"value\": \"test\"}",
+ "type": "string"
+ }
+
+ ],
+ "sliceRecordCount": 10000
+ }
+ },
+ "writer": {
+ "name": "databendwriter",
+ "parameter": {
+ "writeMode": "replace",
+ "onConflictColumn": ["id"],
+ "username": "databend",
+ "password": "databend",
+ "column": ["a", "b", "c", "d", "e", "f", "g"],
+ "batchSize": 1000,
+ "preSql": [
+ ],
+ "postSql": [
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:databend://localhost:8000/datax",
+ "table": [
+ "sample1"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ],
+ "setting": {
+ "speed": {
+ "channel": 1
+ }
+ }
+ }
+}
+```
+
+### 3.2 参数说明
+* jdbcUrl
+ * 描述: JDBC 数据源 url。请参阅仓库中的详细[文档](https://github.com/databendcloud/databend-jdbc)
+ * 必选: 是
+ * 默认值: 无
+ * 示例: jdbc:databend://localhost:8000/datax
+* username
+ * 描述: JDBC 数据源用户名
+ * 必选: 是
+ * 默认值: 无
+ * 示例: databend
+* password
+ * 描述: JDBC 数据源密码
+ * 必选: 是
+ * 默认值: 无
+ * 示例: databend
+* table
+ * 描述: 表名的集合，table应该包含column参数中的所有列。
+ * 必选: 是
+ * 默认值: 无
+ * 示例: ["sample1"]
+* column
+ * 描述: 表中的列名集合，字段顺序应该与reader的record中的column类型对应
+ * 必选: 是
+ * 默认值: 无
+ * 示例: ["a", "b", "c", "d", "e", "f", "g"]
+* batchSize
+ * 描述: 每个批次的记录数
+ * 必选: 否
+ * 默认值: 1000
+ * 示例: 1000
+* preSql
+ * 描述: 在写入数据之前执行的SQL语句
+ * 必选: 否
+ * 默认值: 无
+ * 示例: ["delete from datax.sample1"]
+* postSql
+ * 描述: 在写入数据之后执行的SQL语句
+ * 必选: 否
+ * 默认值: 无
+ * 示例: ["select count(*) from datax.sample1"]
+* writeMode
+ * 描述：写入模式，支持 insert 和 replace 两种模式，默认为 insert。若为 replace，务必填写 onConflictColumn 参数
+ * 必选：否
+ * 默认值：insert
+ * 示例："replace"
+* onConflictColumn
+ * 描述：on conflict 字段，指定 writeMode 为 replace 后，需要此参数
+ * 必选：否
+ * 默认值：无
+ * 示例：["id","user"]
+
+### 3.3 类型转化
+DataX中的数据类型可以转换为databend中的相应数据类型。下表显示了两种类型之间的对应关系。
+
+| DataX 内部类型 | Databend 数据类型 |
+|------------|-----------------------------------------------------------|
+| INT | TINYINT, INT8, SMALLINT, INT16, INT, INT32, BIGINT, INT64 |
+| LONG | TINYINT, INT8, SMALLINT, INT16, INT, INT32, BIGINT, INT64 |
+| STRING | STRING, VARCHAR |
+| DOUBLE | FLOAT, DOUBLE |
+| BOOL | BOOLEAN, BOOL |
+| DATE | DATE, TIMESTAMP |
+| BYTES | STRING, VARCHAR |
+
+## 4 性能测试
+
+## 5 约束限制
+目前，复杂数据类型支持不稳定，如果您想使用复杂数据类型，例如元组，数组，请检查databend和jdbc驱动程序的进一步版本。
+
+## FAQ
\ No newline at end of file
diff --git a/databendwriter/doc/databendwriter.md b/databendwriter/doc/databendwriter.md
new file mode 100644
index 00000000..c92d6387
--- /dev/null
+++ b/databendwriter/doc/databendwriter.md
@@ -0,0 +1,176 @@
+# DataX DatabendWriter
+[简体中文](./databendwriter-CN.md) | [English](./databendwriter.md)
+
+## 1 Introduction
+Databend Writer is a plugin for DataX to write data to Databend Table from dataX records.
+The plugin is based on [databend JDBC driver](https://github.com/databendcloud/databend-jdbc) which use [RESTful http protocol](https://databend.rs/doc/integrations/api/rest)
+to execute query on open source databend and [databend cloud](https://app.databend.com/).
+
+During each write batch, databend writer will upload batch data into internal S3 stage and execute corresponding insert SQL to upload data into databend table.
+
+For best user experience, if you are using databend community distribution, you should try to adopt [S3](https://aws.amazon.com/s3/)/[minio](https://min.io/)/[OSS](https://www.alibabacloud.com/product/object-storage-service) as its underlying storage layer since
+they support presign upload operation otherwise you may expend unneeded cost on data transfer.
+
+You could see more details on the [doc](https://databend.rs/doc/deploy/deploying-databend)
+
+## 2 Detailed Implementation
+Databend Writer would use DataX to fetch records generated by DataX Reader, and then batch insert records to the designated columns for your databend table.
+
+## 3 Features
+### 3.1 Example Configurations
+* the following configuration would read some generated data in memory and upload data into databend table
+
+#### Preparation
+```sql
+--- create table in databend
+drop table if exists datax.sample1;
+drop database if exists datax;
+create database if not exists datax;
+create table if not exsits datax.sample1(a string, b int64, c date, d timestamp, e bool, f string, g variant);
+```
+
+#### Configurations
+```json
+{
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "streamreader",
+ "parameter": {
+ "column" : [
+ {
+ "value": "DataX",
+ "type": "string"
+ },
+ {
+ "value": 19880808,
+ "type": "long"
+ },
+ {
+ "value": "1926-08-08 08:08:08",
+ "type": "date"
+ },
+ {
+ "value": "1988-08-08 08:08:08",
+ "type": "date"
+ },
+ {
+ "value": true,
+ "type": "bool"
+ },
+ {
+ "value": "test",
+ "type": "bytes"
+ },
+ {
+ "value": "{\"type\": \"variant\", \"value\": \"test\"}",
+ "type": "string"
+ }
+
+ ],
+ "sliceRecordCount": 10000
+ }
+ },
+ "writer": {
+ "name": "databendwriter",
+ "parameter": {
+ "username": "databend",
+ "password": "databend",
+ "column": ["a", "b", "c", "d", "e", "f", "g"],
+ "batchSize": 1000,
+ "preSql": [
+ ],
+ "postSql": [
+ ],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:databend://localhost:8000/datax",
+ "table": [
+ "sample1"
+ ]
+ }
+ ]
+ }
+ }
+ }
+ ],
+ "setting": {
+ "speed": {
+ "channel": 1
+ }
+ }
+ }
+}
+```
+
+### 3.2 Configuration Description
+* jdbcUrl
+ * Description: JDBC Data source url in Databend. Please take a look at repository for detailed [doc](https://github.com/databendcloud/databend-jdbc)
+ * Required: yes
+ * Default: none
+ * Example: jdbc:databend://localhost:8000/datax
+* username
+ * Description: Databend user name
+ * Required: yes
+ * Default: none
+ * Example: databend
+* password
+ * Description: Databend user password
+ * Required: yes
+ * Default: none
+ * Example: databend
+* table
+ * Description: A list of table names that should contain all of the columns in the column parameter.
+ * Required: yes
+ * Default: none
+ * Example: ["sample1"]
+* column
+ * Description: A list of column field names that should be inserted into the table. if you want to insert all column fields use `["*"]` instead.
+ * Required: yes
+ * Default: none
+ * Example: ["a", "b", "c", "d", "e", "f", "g"]
+* batchSize
+ * Description: The number of records to be inserted in each batch.
+ * Required: no
+ * Default: 1024
+* preSql
+ * Description: A list of SQL statements that will be executed before the write operation.
+ * Required: no
+ * Default: none
+* postSql
+ * Description: A list of SQL statements that will be executed after the write operation.
+ * Required: no
+ * Default: none
+* writeMode
+ * Description：The write mode, support `insert` and `replace` two mode.
+ * Required：no
+ * Default：insert
+ * Example："replace"
+* onConflictColumn
+ * Description：On conflict fields list.
+ * Required：no
+ * Default：none
+ * Example：["id","user"]
+
+### 3.3 Type Convert
+Data types in datax can be converted to the corresponding data types in databend. The following table shows the correspondence between the two types.
+
+| DataX Type | Databend Type |
+|------------|-----------------------------------------------------------|
+| INT | TINYINT, INT8, SMALLINT, INT16, INT, INT32, BIGINT, INT64 |
+| LONG | TINYINT, INT8, SMALLINT, INT16, INT, INT32, BIGINT, INT64 |
+| STRING | STRING, VARCHAR |
+| DOUBLE | FLOAT, DOUBLE |
+| BOOL | BOOLEAN, BOOL |
+| DATE | DATE, TIMESTAMP |
+| BYTES | STRING, VARCHAR |
+
+
+## 4 Performance Test
+
+
+## 5 Restrictions
+Currently, complex data type support is not stable, if you want to use complex data type such as tuple, array, please check further release version of databend and jdbc driver.
+
+## FAQ
diff --git a/databendwriter/pom.xml b/databendwriter/pom.xml
new file mode 100644
index 00000000..b99ca5d8
--- /dev/null
+++ b/databendwriter/pom.xml
@@ -0,0 +1,101 @@
+
+
+
+ datax-all
+ com.alibaba.datax
+ 0.0.1-SNAPSHOT
+
+
+ 4.0.0
+ databendwriter
+ databendwriter
+ jar
+
+
+
+ com.databend
+ databend-jdbc
+ 0.1.0
+
+
+ com.alibaba.datax
+ datax-core
+ ${datax-project-version}
+
+
+ com.alibaba.datax
+ datax-common
+ ${datax-project-version}
+
+
+ org.slf4j
+ slf4j-api
+
+
+
+ ch.qos.logback
+ logback-classic
+
+
+
+ com.alibaba.datax
+ plugin-rdbms-util
+ ${datax-project-version}
+
+
+ com.google.guava
+ guava
+
+
+
+
+
+
+ junit
+ junit
+ test
+
+
+
+
+
+ src/main/java
+
+ **/*.properties
+
+
+
+
+
+
+ maven-compiler-plugin
+
+ ${jdk-version}
+ ${jdk-version}
+ ${project-sourceEncoding}
+
+
+
+
+ maven-assembly-plugin
+
+
+ src/main/assembly/package.xml
+
+ datax
+
+
+
+ dwzip
+ package
+
+ single
+
+
+
+
+
+
+
diff --git a/databendwriter/src/main/assembly/package.xml b/databendwriter/src/main/assembly/package.xml
new file mode 100755
index 00000000..8a9ba1b2
--- /dev/null
+++ b/databendwriter/src/main/assembly/package.xml
@@ -0,0 +1,34 @@
+
+
+ dir
+
+ false
+
+
+ src/main/resources
+
+ plugin.json
+ plugin_job_template.json
+
+ plugin/writer/databendwriter
+
+
+ target/
+
+ databendwriter-0.0.1-SNAPSHOT.jar
+
+ plugin/writer/databendwriter
+
+
+
+
+
+ false
+ plugin/writer/databendwriter/libs
+
+
+
diff --git a/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/DatabendWriter.java b/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/DatabendWriter.java
new file mode 100644
index 00000000..ddb8fc9a
--- /dev/null
+++ b/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/DatabendWriter.java
@@ -0,0 +1,241 @@
+package com.alibaba.datax.plugin.writer.databendwriter;
+
+import com.alibaba.datax.common.element.Column;
+import com.alibaba.datax.common.element.StringColumn;
+import com.alibaba.datax.common.exception.CommonErrorCode;
+import com.alibaba.datax.common.exception.DataXException;
+import com.alibaba.datax.common.plugin.RecordReceiver;
+import com.alibaba.datax.common.spi.Writer;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
+import com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter;
+import com.alibaba.datax.plugin.writer.databendwriter.util.DatabendWriterUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.*;
+import java.util.List;
+import java.util.regex.Pattern;
+
+public class DatabendWriter extends Writer {
+ private static final DataBaseType DATABASE_TYPE = DataBaseType.Databend;
+
+ public static class Job
+ extends Writer.Job {
+ private static final Logger LOG = LoggerFactory.getLogger(Job.class);
+ private Configuration originalConfig;
+ private CommonRdbmsWriter.Job commonRdbmsWriterMaster;
+
+ @Override
+ public void init() throws DataXException {
+ this.originalConfig = super.getPluginJobConf();
+ this.commonRdbmsWriterMaster = new CommonRdbmsWriter.Job(DATABASE_TYPE);
+ this.commonRdbmsWriterMaster.init(this.originalConfig);
+ // placeholder currently not supported by databend driver, needs special treatment
+ DatabendWriterUtil.dealWriteMode(this.originalConfig);
+ }
+
+ @Override
+ public void preCheck() {
+ this.init();
+ this.commonRdbmsWriterMaster.writerPreCheck(this.originalConfig, DATABASE_TYPE);
+ }
+
+ @Override
+ public void prepare() {
+ this.commonRdbmsWriterMaster.prepare(this.originalConfig);
+ }
+
+ @Override
+ public List split(int mandatoryNumber) {
+ return this.commonRdbmsWriterMaster.split(this.originalConfig, mandatoryNumber);
+ }
+
+ @Override
+ public void post() {
+ this.commonRdbmsWriterMaster.post(this.originalConfig);
+ }
+
+ @Override
+ public void destroy() {
+ this.commonRdbmsWriterMaster.destroy(this.originalConfig);
+ }
+ }
+
+
+ public static class Task extends Writer.Task {
+ private static final Logger LOG = LoggerFactory.getLogger(Task.class);
+
+ private Configuration writerSliceConfig;
+
+ private CommonRdbmsWriter.Task commonRdbmsWriterSlave;
+
+ @Override
+ public void init() {
+ this.writerSliceConfig = super.getPluginJobConf();
+
+ this.commonRdbmsWriterSlave = new CommonRdbmsWriter.Task(DataBaseType.Databend) {
+ @Override
+ protected PreparedStatement fillPreparedStatementColumnType(PreparedStatement preparedStatement, int columnIndex, int columnSqltype, String typeName, Column column) throws SQLException {
+ try {
+ if (column.getRawData() == null) {
+ preparedStatement.setNull(columnIndex + 1, columnSqltype);
+ return preparedStatement;
+ }
+
+ java.util.Date utilDate;
+ switch (columnSqltype) {
+
+ case Types.TINYINT:
+ case Types.SMALLINT:
+ case Types.INTEGER:
+ preparedStatement.setInt(columnIndex + 1, column.asBigInteger().intValue());
+ break;
+ case Types.BIGINT:
+ preparedStatement.setLong(columnIndex + 1, column.asLong());
+ break;
+ case Types.DECIMAL:
+ preparedStatement.setBigDecimal(columnIndex + 1, column.asBigDecimal());
+ break;
+ case Types.FLOAT:
+ case Types.REAL:
+ preparedStatement.setFloat(columnIndex + 1, column.asDouble().floatValue());
+ break;
+ case Types.DOUBLE:
+ preparedStatement.setDouble(columnIndex + 1, column.asDouble());
+ break;
+ case Types.DATE:
+ java.sql.Date sqlDate = null;
+ try {
+ utilDate = column.asDate();
+ } catch (DataXException e) {
+ throw new SQLException(String.format(
+ "Date type conversion error: [%s]", column));
+ }
+
+ if (null != utilDate) {
+ sqlDate = new java.sql.Date(utilDate.getTime());
+ }
+ preparedStatement.setDate(columnIndex + 1, sqlDate);
+ break;
+
+ case Types.TIME:
+ java.sql.Time sqlTime = null;
+ try {
+ utilDate = column.asDate();
+ } catch (DataXException e) {
+ throw new SQLException(String.format(
+ "Date type conversion error: [%s]", column));
+ }
+
+ if (null != utilDate) {
+ sqlTime = new java.sql.Time(utilDate.getTime());
+ }
+ preparedStatement.setTime(columnIndex + 1, sqlTime);
+ break;
+
+ case Types.TIMESTAMP:
+ Timestamp sqlTimestamp = null;
+ if (column instanceof StringColumn && column.asString() != null) {
+ String timeStampStr = column.asString();
+ // JAVA TIMESTAMP 类型入参必须是 "2017-07-12 14:39:00.123566" 格式
+ String pattern = "^\\d+-\\d+-\\d+ \\d+:\\d+:\\d+.\\d+";
+ boolean isMatch = Pattern.matches(pattern, timeStampStr);
+ if (isMatch) {
+ sqlTimestamp = Timestamp.valueOf(timeStampStr);
+ preparedStatement.setTimestamp(columnIndex + 1, sqlTimestamp);
+ break;
+ }
+ }
+ try {
+ utilDate = column.asDate();
+ } catch (DataXException e) {
+ throw new SQLException(String.format(
+ "Date type conversion error: [%s]", column));
+ }
+
+ if (null != utilDate) {
+ sqlTimestamp = new Timestamp(
+ utilDate.getTime());
+ }
+ preparedStatement.setTimestamp(columnIndex + 1, sqlTimestamp);
+ break;
+
+ case Types.BINARY:
+ case Types.VARBINARY:
+ case Types.BLOB:
+ case Types.LONGVARBINARY:
+ preparedStatement.setBytes(columnIndex + 1, column
+ .asBytes());
+ break;
+
+ case Types.BOOLEAN:
+
+ // warn: bit(1) -> Types.BIT 可使用setBoolean
+ // warn: bit(>1) -> Types.VARBINARY 可使用setBytes
+ case Types.BIT:
+ if (this.dataBaseType == DataBaseType.MySql) {
+ Boolean asBoolean = column.asBoolean();
+ if (asBoolean != null) {
+ preparedStatement.setBoolean(columnIndex + 1, asBoolean);
+ } else {
+ preparedStatement.setNull(columnIndex + 1, Types.BIT);
+ }
+ } else {
+ preparedStatement.setString(columnIndex + 1, column.asString());
+ }
+ break;
+
+ default:
+ // cast variant / array into string is fine.
+ preparedStatement.setString(columnIndex + 1, column.asString());
+ break;
+ }
+ return preparedStatement;
+ } catch (DataXException e) {
+ // fix类型转换或者溢出失败时，将具体哪一列打印出来
+ if (e.getErrorCode() == CommonErrorCode.CONVERT_NOT_SUPPORT ||
+ e.getErrorCode() == CommonErrorCode.CONVERT_OVER_FLOW) {
+ throw DataXException
+ .asDataXException(
+ e.getErrorCode(),
+ String.format(
+ "type conversion error. columnName: [%s], columnType:[%d], columnJavaType: [%s]. please change the data type in given column field or do not sync on the column.",
+ this.resultSetMetaData.getLeft()
+ .get(columnIndex),
+ this.resultSetMetaData.getMiddle()
+ .get(columnIndex),
+ this.resultSetMetaData.getRight()
+ .get(columnIndex)));
+ } else {
+ throw e;
+ }
+ }
+ }
+
+ };
+ this.commonRdbmsWriterSlave.init(this.writerSliceConfig);
+ }
+
+ @Override
+ public void destroy() {
+ this.commonRdbmsWriterSlave.destroy(this.writerSliceConfig);
+ }
+
+ @Override
+ public void prepare() {
+ this.commonRdbmsWriterSlave.prepare(this.writerSliceConfig);
+ }
+
+ @Override
+ public void post() {
+ this.commonRdbmsWriterSlave.post(this.writerSliceConfig);
+ }
+
+ @Override
+ public void startWrite(RecordReceiver lineReceiver) {
+ this.commonRdbmsWriterSlave.startWrite(lineReceiver, this.writerSliceConfig, this.getTaskPluginCollector());
+ }
+
+ }
+}
diff --git a/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/DatabendWriterErrorCode.java b/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/DatabendWriterErrorCode.java
new file mode 100644
index 00000000..21cbf428
--- /dev/null
+++ b/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/DatabendWriterErrorCode.java
@@ -0,0 +1,33 @@
+package com.alibaba.datax.plugin.writer.databendwriter;
+
+import com.alibaba.datax.common.spi.ErrorCode;
+
+
+public enum DatabendWriterErrorCode implements ErrorCode {
+ CONF_ERROR("DatabendWriter-00", "配置错误."),
+ WRITE_DATA_ERROR("DatabendWriter-01", "写入数据时失败."),
+ ;
+
+ private final String code;
+ private final String description;
+
+ private DatabendWriterErrorCode(String code, String description) {
+ this.code = code;
+ this.description = description;
+ }
+
+ @Override
+ public String getCode() {
+ return this.code;
+ }
+
+ @Override
+ public String getDescription() {
+ return this.description;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("Code:[%s], Description:[%s].", this.code, this.description);
+ }
+}
\ No newline at end of file
diff --git a/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/util/DatabendWriterUtil.java b/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/util/DatabendWriterUtil.java
new file mode 100644
index 00000000..516a75eb
--- /dev/null
+++ b/databendwriter/src/main/java/com/alibaba/datax/plugin/writer/databendwriter/util/DatabendWriterUtil.java
@@ -0,0 +1,72 @@
+package com.alibaba.datax.plugin.writer.databendwriter.util;
+
+import com.alibaba.datax.common.exception.DataXException;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.rdbms.writer.Constant;
+import com.alibaba.datax.plugin.rdbms.writer.Key;
+
+import com.alibaba.datax.plugin.writer.databendwriter.DatabendWriterErrorCode;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.xml.crypto.Data;
+import java.util.List;
+import java.util.StringJoiner;
+
+public final class DatabendWriterUtil {
+ private static final Logger LOG = LoggerFactory.getLogger(DatabendWriterUtil.class);
+
+ private DatabendWriterUtil() {
+ }
+
+ public static void dealWriteMode(Configuration originalConfig) throws DataXException {
+ List columns = originalConfig.getList(Key.COLUMN, String.class);
+ List onConflictColumns = originalConfig.getList(Key.ONCONFLICT_COLUMN, String.class);
+ StringBuilder writeDataSqlTemplate = new StringBuilder();
+
+ String jdbcUrl = originalConfig.getString(String.format("%s[0].%s",
+ Constant.CONN_MARK, Key.JDBC_URL, String.class));
+
+ String writeMode = originalConfig.getString(Key.WRITE_MODE, "INSERT");
+ LOG.info("write mode is {}", writeMode);
+ if (writeMode.toLowerCase().contains("replace")) {
+ if (onConflictColumns == null || onConflictColumns.size() == 0) {
+ throw DataXException
+ .asDataXException(
+ DatabendWriterErrorCode.CONF_ERROR,
+ String.format(
+ "Replace mode must has onConflictColumn config."
+ ));
+ }
+
+ // for databend if you want to use replace mode, the writeMode should be: "writeMode": "replace"
+ writeDataSqlTemplate.append("REPLACE INTO %s (")
+ .append(StringUtils.join(columns, ",")).append(") ").append(onConFlictDoString(onConflictColumns))
+ .append(" VALUES");
+
+ LOG.info("Replace data [\n{}\n], which jdbcUrl like:[{}]", writeDataSqlTemplate, jdbcUrl);
+ originalConfig.set(Constant.INSERT_OR_REPLACE_TEMPLATE_MARK, writeDataSqlTemplate);
+ } else {
+ writeDataSqlTemplate.append("INSERT INTO %s");
+ StringJoiner columnString = new StringJoiner(",");
+
+ for (String column : columns) {
+ columnString.add(column);
+ }
+ writeDataSqlTemplate.append(String.format("(%s)", columnString));
+ writeDataSqlTemplate.append(" VALUES");
+
+ LOG.info("Insert data [\n{}\n], which jdbcUrl like:[{}]", writeDataSqlTemplate, jdbcUrl);
+
+ originalConfig.set(Constant.INSERT_OR_REPLACE_TEMPLATE_MARK, writeDataSqlTemplate);
+ }
+
+ }
+
+ public static String onConFlictDoString(List conflictColumns) {
+ return " ON " +
+ "(" +
+ StringUtils.join(conflictColumns, ",") + ") ";
+ }
+}
diff --git a/databendwriter/src/main/resources/plugin.json b/databendwriter/src/main/resources/plugin.json
new file mode 100644
index 00000000..bab0130d
--- /dev/null
+++ b/databendwriter/src/main/resources/plugin.json
@@ -0,0 +1,6 @@
+{
+ "name": "databendwriter",
+ "class": "com.alibaba.datax.plugin.writer.databendwriter.DatabendWriter",
+ "description": "execute batch insert sql to write dataX data into databend",
+ "developer": "databend"
+}
\ No newline at end of file
diff --git a/databendwriter/src/main/resources/plugin_job_template.json b/databendwriter/src/main/resources/plugin_job_template.json
new file mode 100644
index 00000000..34d4b251
--- /dev/null
+++ b/databendwriter/src/main/resources/plugin_job_template.json
@@ -0,0 +1,19 @@
+{
+ "name": "databendwriter",
+ "parameter": {
+ "username": "username",
+ "password": "password",
+ "column": ["col1", "col2", "col3"],
+ "connection": [
+ {
+ "jdbcUrl": "jdbc:databend://:[/]",
+ "table": "table1"
+ }
+ ],
+ "preSql": [],
+ "postSql": [],
+
+ "maxBatchRows": 65536,
+ "maxBatchSize": 134217728
+ }
+}
\ No newline at end of file
diff --git a/datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubClientHelper.java b/datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubClientHelper.java
index 6f601fb4..2b7bcec4 100644
--- a/datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubClientHelper.java
+++ b/datahubreader/src/main/java/com/alibaba/datax/plugin/reader/datahubreader/DatahubClientHelper.java
@@ -1,8 +1,8 @@
package com.alibaba.datax.plugin.reader.datahubreader;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import com.aliyun.datahub.client.DatahubClient;
import com.aliyun.datahub.client.DatahubClientBuilder;
import com.aliyun.datahub.client.auth.Account;
diff --git a/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubClientHelper.java b/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubClientHelper.java
index 2d94212c..c25d1210 100644
--- a/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubClientHelper.java
+++ b/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubClientHelper.java
@@ -3,8 +3,8 @@ package com.alibaba.datax.plugin.writer.datahubwriter;
import org.apache.commons.lang3.StringUtils;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import com.aliyun.datahub.client.DatahubClient;
import com.aliyun.datahub.client.DatahubClientBuilder;
import com.aliyun.datahub.client.auth.Account;
diff --git a/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubWriter.java b/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubWriter.java
index f6dc1105..cd414fc5 100644
--- a/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubWriter.java
+++ b/datahubwriter/src/main/java/com/alibaba/datax/plugin/writer/datahubwriter/DatahubWriter.java
@@ -8,7 +8,7 @@ import com.alibaba.datax.common.spi.Writer;
import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.common.util.DataXCaseEnvUtil;
import com.alibaba.datax.common.util.RetryUtil;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import com.aliyun.datahub.client.DatahubClient;
import com.aliyun.datahub.client.model.FieldType;
import com.aliyun.datahub.client.model.GetTopicResult;
diff --git a/datax-example/datax-example-core/pom.xml b/datax-example/datax-example-core/pom.xml
new file mode 100644
index 00000000..6a2e9e8e
--- /dev/null
+++ b/datax-example/datax-example-core/pom.xml
@@ -0,0 +1,20 @@
+
+
+ 4.0.0
+
+ com.alibaba.datax
+ datax-example
+ 0.0.1-SNAPSHOT
+
+
+ datax-example-core
+
+
+ 8
+ 8
+ UTF-8
+
+
+
\ No newline at end of file
diff --git a/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/ExampleContainer.java b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/ExampleContainer.java
new file mode 100644
index 00000000..a4229fd1
--- /dev/null
+++ b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/ExampleContainer.java
@@ -0,0 +1,26 @@
+package com.alibaba.datax.example;
+
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.core.Engine;
+import com.alibaba.datax.example.util.ExampleConfigParser;
+
+/**
+ * {@code Date} 2023/8/6 11:22
+ *
+ * @author fuyouj
+ */
+
+public class ExampleContainer {
+ /**
+ * example对外暴露的启动入口
+ * 使用前最好看下 datax-example/doc/README.MD
+ * @param jobPath 任务json绝对路径
+ */
+ public static void start(String jobPath) {
+
+ Configuration configuration = ExampleConfigParser.parse(jobPath);
+
+ Engine engine = new Engine();
+ engine.start(configuration);
+ }
+}
diff --git a/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/Main.java b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/Main.java
new file mode 100644
index 00000000..56bf9f0b
--- /dev/null
+++ b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/Main.java
@@ -0,0 +1,23 @@
+package com.alibaba.datax.example;
+
+
+import com.alibaba.datax.example.util.PathUtil;
+
+/**
+ * @author fuyouj
+ */
+public class Main {
+
+ /**
+ * 1.在example模块pom文件添加你依赖的的调试插件，
+ * 你可以直接打开本模块的pom文件,参考是如何引入streamreader，streamwriter
+ * 2. 在此处指定你的job文件
+ */
+ public static void main(String[] args) {
+
+ String classPathJobPath = "/job/stream2stream.json";
+ String absJobPath = PathUtil.getAbsolutePathFromClassPath(classPathJobPath);
+ ExampleContainer.start(absJobPath);
+ }
+
+}
diff --git a/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/util/ExampleConfigParser.java b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/util/ExampleConfigParser.java
new file mode 100644
index 00000000..6bbb4a23
--- /dev/null
+++ b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/util/ExampleConfigParser.java
@@ -0,0 +1,154 @@
+package com.alibaba.datax.example.util;
+
+import com.alibaba.datax.common.exception.DataXException;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.core.util.ConfigParser;
+import com.alibaba.datax.core.util.FrameworkErrorCode;
+import com.alibaba.datax.core.util.container.CoreConstant;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Paths;
+import java.util.*;
+
+/**
+ * @author fuyouj
+ */
+public class ExampleConfigParser {
+ private static final String CORE_CONF = "/example/conf/core.json";
+
+ private static final String PLUGIN_DESC_FILE = "plugin.json";
+
+ /**
+ * 指定Job配置路径，ConfigParser会解析Job、Plugin、Core全部信息，并以Configuration返回
+ * 不同于Core的ConfigParser,这里的core,plugin 不依赖于编译后的datax.home,而是扫描程序编译后的target目录
+ */
+ public static Configuration parse(final String jobPath) {
+
+ Configuration configuration = ConfigParser.parseJobConfig(jobPath);
+ configuration.merge(coreConfig(),
+ false);
+
+ Map pluginTypeMap = new HashMap<>();
+ String readerName = configuration.getString(CoreConstant.DATAX_JOB_CONTENT_READER_NAME);
+ String writerName = configuration.getString(CoreConstant.DATAX_JOB_CONTENT_WRITER_NAME);
+ pluginTypeMap.put(readerName, "reader");
+ pluginTypeMap.put(writerName, "writer");
+ Configuration pluginsDescConfig = parsePluginsConfig(pluginTypeMap);
+ configuration.merge(pluginsDescConfig, false);
+ return configuration;
+ }
+
+ private static Configuration parsePluginsConfig(Map pluginTypeMap) {
+
+ Configuration configuration = Configuration.newDefault();
+
+ //最初打算通过user.dir获取工作目录来扫描插件，
+ //但是user.dir在不同有一些不确定性，所以废弃了这个选择
+
+ for (File basePackage : runtimeBasePackages()) {
+ if (pluginTypeMap.isEmpty()) {
+ break;
+ }
+ scanPluginByPackage(basePackage, configuration, basePackage.listFiles(), pluginTypeMap);
+ }
+ if (!pluginTypeMap.isEmpty()) {
+ String failedPlugin = pluginTypeMap.keySet().toString();
+ String message = "\nplugin %s load failed ：ry to analyze the reasons from the following aspects.。\n" +
+ "1: Check if the name of the plugin is spelled correctly, and verify whether DataX supports this plugin\n" +
+ "2：Verify if the " +
+ " src/main/resources \n" +
+ " \n" +
+ " **/*.* \n" +
+ " \n" +
+ " true \n" +
+ " \n [Refer to the streamreader pom file] \n" +
+ "3: Check that the datax-yourPlugin-example module imported your test plugin";
+ message = String.format(message, failedPlugin);
+ throw DataXException.asDataXException(FrameworkErrorCode.PLUGIN_INIT_ERROR, message);
+ }
+ return configuration;
+ }
+
+ /**
+ * 通过classLoader获取程序编译的输出目录
+ *
+ * @return File[/datax-example/target/classes,xxReader/target/classes,xxWriter/target/classes]
+ */
+ private static File[] runtimeBasePackages() {
+ List basePackages = new ArrayList<>();
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ Enumeration resources = null;
+ try {
+ resources = classLoader.getResources("");
+ } catch (IOException e) {
+ throw DataXException.asDataXException(e.getMessage());
+ }
+
+ while (resources.hasMoreElements()) {
+ URL resource = resources.nextElement();
+ File file = new File(resource.getFile());
+ if (file.isDirectory()) {
+ basePackages.add(file);
+ }
+ }
+
+ return basePackages.toArray(new File[0]);
+ }
+
+ /**
+ * @param packageFile 编译出来的target/classes根目录 便于找到插件时设置插件的URL目录，设置根目录是最保险的方式
+ * @param configuration pluginConfig
+ * @param files 待扫描文件
+ * @param needPluginTypeMap 需要的插件
+ */
+ private static void scanPluginByPackage(File packageFile,
+ Configuration configuration,
+ File[] files,
+ Map needPluginTypeMap) {
+ if (files == null) {
+ return;
+ }
+ for (File file : files) {
+ if (file.isFile() && PLUGIN_DESC_FILE.equals(file.getName())) {
+ Configuration pluginDesc = Configuration.from(file);
+ String descPluginName = pluginDesc.getString("name", "");
+
+ if (needPluginTypeMap.containsKey(descPluginName)) {
+
+ String type = needPluginTypeMap.get(descPluginName);
+ configuration.merge(parseOnePlugin(packageFile.getAbsolutePath(), type, descPluginName, pluginDesc), false);
+ needPluginTypeMap.remove(descPluginName);
+
+ }
+ } else {
+ scanPluginByPackage(packageFile, configuration, file.listFiles(), needPluginTypeMap);
+ }
+ }
+ }
+
+
+ private static Configuration parseOnePlugin(String packagePath,
+ String pluginType,
+ String pluginName,
+ Configuration pluginDesc) {
+ //设置path 兼容jarLoader的加载方式URLClassLoader
+ pluginDesc.set("path", packagePath);
+ Configuration pluginConfInJob = Configuration.newDefault();
+ pluginConfInJob.set(
+ String.format("plugin.%s.%s", pluginType, pluginName),
+ pluginDesc.getInternal());
+ return pluginConfInJob;
+ }
+
+ private static Configuration coreConfig() {
+ try {
+ URL resource = ExampleConfigParser.class.getResource(CORE_CONF);
+ return Configuration.from(Paths.get(resource.toURI()).toFile());
+ } catch (Exception ignore) {
+ throw DataXException.asDataXException("Failed to load the configuration file core.json. " +
+ "Please check whether /example/conf/core.json exists!");
+ }
+ }
+}
diff --git a/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/util/PathUtil.java b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/util/PathUtil.java
new file mode 100644
index 00000000..e197fa73
--- /dev/null
+++ b/datax-example/datax-example-core/src/main/java/com/alibaba/datax/example/util/PathUtil.java
@@ -0,0 +1,26 @@
+package com.alibaba.datax.example.util;
+
+
+import com.alibaba.datax.common.exception.DataXException;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.nio.file.Paths;
+
+/**
+ * @author fuyouj
+ */
+public class PathUtil {
+ public static String getAbsolutePathFromClassPath(String path) {
+ URL resource = PathUtil.class.getResource(path);
+ try {
+ assert resource != null;
+ URI uri = resource.toURI();
+ return Paths.get(uri).toString();
+ } catch (NullPointerException | URISyntaxException e) {
+ throw DataXException.asDataXException("path error,please check whether the path is correct");
+ }
+
+ }
+}
diff --git a/datax-example/datax-example-core/src/main/resources/example/conf/core.json b/datax-example/datax-example-core/src/main/resources/example/conf/core.json
new file mode 100755
index 00000000..33281ac0
--- /dev/null
+++ b/datax-example/datax-example-core/src/main/resources/example/conf/core.json
@@ -0,0 +1,60 @@
+{
+ "entry": {
+ "jvm": "-Xms1G -Xmx1G",
+ "environment": {}
+ },
+ "common": {
+ "column": {
+ "datetimeFormat": "yyyy-MM-dd HH:mm:ss",
+ "timeFormat": "HH:mm:ss",
+ "dateFormat": "yyyy-MM-dd",
+ "extraFormats":["yyyyMMdd"],
+ "timeZone": "GMT+8",
+ "encoding": "utf-8"
+ }
+ },
+ "core": {
+ "dataXServer": {
+ "address": "http://localhost:7001/api",
+ "timeout": 10000,
+ "reportDataxLog": false,
+ "reportPerfLog": false
+ },
+ "transport": {
+ "channel": {
+ "class": "com.alibaba.datax.core.transport.channel.memory.MemoryChannel",
+ "speed": {
+ "byte": -1,
+ "record": -1
+ },
+ "flowControlInterval": 20,
+ "capacity": 512,
+ "byteCapacity": 67108864
+ },
+ "exchanger": {
+ "class": "com.alibaba.datax.core.plugin.BufferedRecordExchanger",
+ "bufferSize": 32
+ }
+ },
+ "container": {
+ "job": {
+ "reportInterval": 10000
+ },
+ "taskGroup": {
+ "channel": 5
+ },
+ "trace": {
+ "enable": "false"
+ }
+
+ },
+ "statistics": {
+ "collector": {
+ "plugin": {
+ "taskClass": "com.alibaba.datax.core.statistics.plugin.task.StdoutPluginCollector",
+ "maxDirtyNumber": 10
+ }
+ }
+ }
+ }
+}
diff --git a/datax-example/datax-example-core/src/test/java/com/alibaba/datax/example/util/PathUtilTest.java b/datax-example/datax-example-core/src/test/java/com/alibaba/datax/example/util/PathUtilTest.java
new file mode 100644
index 00000000..8985b54c
--- /dev/null
+++ b/datax-example/datax-example-core/src/test/java/com/alibaba/datax/example/util/PathUtilTest.java
@@ -0,0 +1,19 @@
+package com.alibaba.datax.example.util;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * {@code Author} FuYouJ
+ * {@code Date} 2023/8/19 21:38
+ */
+
+public class PathUtilTest {
+
+ @Test
+ public void testParseClassPathFile() {
+ String path = "/pathTest.json";
+ String absolutePathFromClassPath = PathUtil.getAbsolutePathFromClassPath(path);
+ Assert.assertNotNull(absolutePathFromClassPath);
+ }
+}
diff --git a/datax-example/datax-example-core/src/test/resources/pathTest.json b/datax-example/datax-example-core/src/test/resources/pathTest.json
new file mode 100644
index 00000000..9e26dfee
--- /dev/null
+++ b/datax-example/datax-example-core/src/test/resources/pathTest.json
@@ -0,0 +1 @@
+{}
\ No newline at end of file
diff --git a/datax-example/datax-example-neo4j/pom.xml b/datax-example/datax-example-neo4j/pom.xml
new file mode 100644
index 00000000..303b14a8
--- /dev/null
+++ b/datax-example/datax-example-neo4j/pom.xml
@@ -0,0 +1,43 @@
+
+
+ 4.0.0
+
+ com.alibaba.datax
+ datax-example
+ 0.0.1-SNAPSHOT
+
+
+ datax-example-neo4j
+
+
+ 8
+ 8
+ UTF-8
+ 1.17.6
+ 4.4.9
+
+
+
+ com.alibaba.datax
+ datax-example-core
+ 0.0.1-SNAPSHOT
+
+
+ org.testcontainers
+ testcontainers
+ ${test.container.version}
+
+
+ com.alibaba.datax
+ neo4jwriter
+ 0.0.1-SNAPSHOT
+
+
+ com.alibaba.datax
+ datax-example-streamreader
+ 0.0.1-SNAPSHOT
+
+
+
\ No newline at end of file
diff --git a/datax-example/datax-example-neo4j/src/test/java/com/alibaba/datax/example/neo4j/StreamReader2Neo4jWriterTest.java b/datax-example/datax-example-neo4j/src/test/java/com/alibaba/datax/example/neo4j/StreamReader2Neo4jWriterTest.java
new file mode 100644
index 00000000..9cf01253
--- /dev/null
+++ b/datax-example/datax-example-neo4j/src/test/java/com/alibaba/datax/example/neo4j/StreamReader2Neo4jWriterTest.java
@@ -0,0 +1,138 @@
+package com.alibaba.datax.example.neo4j;
+
+import com.alibaba.datax.example.ExampleContainer;
+import com.alibaba.datax.example.util.PathUtil;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.neo4j.driver.*;
+import org.neo4j.driver.types.Node;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.lifecycle.Startables;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
+import org.testcontainers.utility.DockerImageName;
+import org.testcontainers.utility.DockerLoggerFactory;
+
+import java.net.URI;
+import java.util.Arrays;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+
+/**
+ * {@code Author} FuYouJ
+ * {@code Date} 2023/8/19 21:48
+ */
+
+public class StreamReader2Neo4jWriterTest {
+ private static final Logger LOGGER = LoggerFactory.getLogger(StreamReader2Neo4jWriterTest.class);
+ private static final String CONTAINER_IMAGE = "neo4j:5.9.0";
+
+ private static final String CONTAINER_HOST = "neo4j-host";
+ private static final int HTTP_PORT = 7474;
+ private static final int BOLT_PORT = 7687;
+ private static final String CONTAINER_NEO4J_USERNAME = "neo4j";
+ private static final String CONTAINER_NEO4J_PASSWORD = "Test@12343";
+ private static final URI CONTAINER_URI = URI.create("neo4j://localhost:" + BOLT_PORT);
+
+ protected static final Network NETWORK = Network.newNetwork();
+
+ private GenericContainer container;
+ protected Driver neo4jDriver;
+ protected Session neo4jSession;
+ private static final int CHANNEL = 5;
+ private static final int READER_NUM = 10;
+
+ @Before
+ public void init() {
+ DockerImageName imageName = DockerImageName.parse(CONTAINER_IMAGE);
+ container =
+ new GenericContainer<>(imageName)
+ .withNetwork(NETWORK)
+ .withNetworkAliases(CONTAINER_HOST)
+ .withExposedPorts(HTTP_PORT, BOLT_PORT)
+ .withEnv(
+ "NEO4J_AUTH",
+ CONTAINER_NEO4J_USERNAME + "/" + CONTAINER_NEO4J_PASSWORD)
+ .withEnv("apoc.export.file.enabled", "true")
+ .withEnv("apoc.import.file.enabled", "true")
+ .withEnv("apoc.import.file.use_neo4j_config", "true")
+ .withEnv("NEO4J_PLUGINS", "[\"apoc\"]")
+ .withLogConsumer(
+ new Slf4jLogConsumer(
+ DockerLoggerFactory.getLogger(CONTAINER_IMAGE)));
+ container.setPortBindings(
+ Arrays.asList(
+ String.format("%s:%s", HTTP_PORT, HTTP_PORT),
+ String.format("%s:%s", BOLT_PORT, BOLT_PORT)));
+ Startables.deepStart(Stream.of(container)).join();
+ LOGGER.info("container started");
+ Awaitility.given()
+ .ignoreExceptions()
+ .await()
+ .atMost(30, TimeUnit.SECONDS)
+ .untilAsserted(this::initConnection);
+ }
+
+ //在neo4jWriter模块使用Example测试整个job,方便发现整个流程的代码问题
+ @Test
+ public void streamReader2Neo4j() {
+
+ deleteHistoryIfExist();
+
+ String path = "/streamreader2neo4j.json";
+ String jobPath = PathUtil.getAbsolutePathFromClassPath(path);
+
+ ExampleContainer.start(jobPath);
+
+ //根据channel和reader的mock数据，校验结果集是否符合预期
+ verifyWriteResult();
+ }
+
+ private void deleteHistoryIfExist() {
+ String query = "match (n:StreamReader) return n limit 1";
+ String delete = "match (n:StreamReader) delete n";
+ if (neo4jSession.run(query).hasNext()) {
+ neo4jSession.run(delete);
+ }
+ }
+
+ private void verifyWriteResult() {
+ int total = CHANNEL * READER_NUM;
+ String query = "match (n:StreamReader) return n";
+ Result run = neo4jSession.run(query);
+ int count = 0;
+ while (run.hasNext()) {
+ Record record = run.next();
+ Node node = record.get("n").asNode();
+ if (node.hasLabel("StreamReader")) {
+ count++;
+ }
+ }
+ Assert.assertEquals(count, total);
+ }
+ @After
+ public void destroy() {
+ if (neo4jSession != null) {
+ neo4jSession.close();
+ }
+ if (neo4jDriver != null) {
+ neo4jDriver.close();
+ }
+ if (container != null) {
+ container.close();
+ }
+ }
+
+ private void initConnection() {
+ neo4jDriver =
+ GraphDatabase.driver(
+ CONTAINER_URI,
+ AuthTokens.basic(CONTAINER_NEO4J_USERNAME, CONTAINER_NEO4J_PASSWORD));
+ neo4jSession = neo4jDriver.session(SessionConfig.forDatabase("neo4j"));
+ }
+}
diff --git a/datax-example/datax-example-neo4j/src/test/resources/streamreader2neo4j.json b/datax-example/datax-example-neo4j/src/test/resources/streamreader2neo4j.json
new file mode 100644
index 00000000..3d543ce3
--- /dev/null
+++ b/datax-example/datax-example-neo4j/src/test/resources/streamreader2neo4j.json
@@ -0,0 +1,51 @@
+{
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "streamreader",
+ "parameter": {
+ "sliceRecordCount": 10,
+ "column": [
+ {
+ "type": "string",
+ "value": "StreamReader"
+ },
+ {
+ "type": "string",
+ "value": "1997"
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "neo4jWriter",
+ "parameter": {
+ "uri": "bolt://localhost:7687",
+ "username":"neo4j",
+ "password":"Test@12343",
+ "database":"neo4j",
+ "cypher": "unwind $batch as row CALL apoc.cypher.doIt( 'create (n:`' + row.Label + '`{id:$id})' ,{id: row.id} ) YIELD value RETURN 1 ",
+ "batchDataVariableName": "batch",
+ "batchSize": "3",
+ "properties": [
+ {
+ "name": "Label",
+ "type": "string"
+ },
+ {
+ "name": "id",
+ "type": "STRING"
+ }
+ ]
+ }
+ }
+ }
+ ],
+ "setting": {
+ "speed": {
+ "channel": 5
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/datax-example/datax-example-streamreader/pom.xml b/datax-example/datax-example-streamreader/pom.xml
new file mode 100644
index 00000000..ea70de10
--- /dev/null
+++ b/datax-example/datax-example-streamreader/pom.xml
@@ -0,0 +1,37 @@
+
+
+ 4.0.0
+
+ com.alibaba.datax
+ datax-example
+ 0.0.1-SNAPSHOT
+
+
+ datax-example-streamreader
+
+
+ 8
+ 8
+ UTF-8
+
+
+
+ com.alibaba.datax
+ datax-example-core
+ 0.0.1-SNAPSHOT
+
+
+ com.alibaba.datax
+ streamreader
+ 0.0.1-SNAPSHOT
+
+
+ com.alibaba.datax
+ streamwriter
+ 0.0.1-SNAPSHOT
+
+
+
+
\ No newline at end of file
diff --git a/datax-example/datax-example-streamreader/src/test/java/com/alibaba/datax/example/streamreader/StreamReader2StreamWriterTest.java b/datax-example/datax-example-streamreader/src/test/java/com/alibaba/datax/example/streamreader/StreamReader2StreamWriterTest.java
new file mode 100644
index 00000000..71d083d0
--- /dev/null
+++ b/datax-example/datax-example-streamreader/src/test/java/com/alibaba/datax/example/streamreader/StreamReader2StreamWriterTest.java
@@ -0,0 +1,19 @@
+package com.alibaba.datax.example.streamreader;
+
+import com.alibaba.datax.example.ExampleContainer;
+import com.alibaba.datax.example.util.PathUtil;
+import org.junit.Test;
+
+/**
+ * {@code Author} FuYouJ
+ * {@code Date} 2023/8/14 20:16
+ */
+
+public class StreamReader2StreamWriterTest {
+ @Test
+ public void testStreamReader2StreamWriter() {
+ String path = "/stream2stream.json";
+ String jobPath = PathUtil.getAbsolutePathFromClassPath(path);
+ ExampleContainer.start(jobPath);
+ }
+}
diff --git a/datax-example/datax-example-streamreader/src/test/resources/stream2stream.json b/datax-example/datax-example-streamreader/src/test/resources/stream2stream.json
new file mode 100644
index 00000000..b2a57395
--- /dev/null
+++ b/datax-example/datax-example-streamreader/src/test/resources/stream2stream.json
@@ -0,0 +1,36 @@
+{
+ "job": {
+ "content": [
+ {
+ "reader": {
+ "name": "streamreader",
+ "parameter": {
+ "sliceRecordCount": 10,
+ "column": [
+ {
+ "type": "long",
+ "value": "10"
+ },
+ {
+ "type": "string",
+ "value": "hello，你好，世界-DataX"
+ }
+ ]
+ }
+ },
+ "writer": {
+ "name": "streamwriter",
+ "parameter": {
+ "encoding": "UTF-8",
+ "print": true
+ }
+ }
+ }
+ ],
+ "setting": {
+ "speed": {
+ "channel": 5
+ }
+ }
+ }
+}
\ No newline at end of file
diff --git a/datax-example/doc/README.md b/datax-example/doc/README.md
new file mode 100644
index 00000000..15f77e87
--- /dev/null
+++ b/datax-example/doc/README.md
@@ -0,0 +1,107 @@
+## [DataX-Example]调试datax插件的模块
+
+### 为什么要开发这个模块
+
+一般使用DataX启动数据同步任务是从datax.py 脚本开始，获取程序datax包目录设置到系统变量datax.home里，此后系统核心插件的加载，配置初始化均依赖于变量datax.home,这带来了一些麻烦，以一次本地 DeBug streamreader 插件为例。
+
+- maven 打包 datax 生成 datax 目录
+- 在 IDE 中 设置系统环境变量 datax.home，或者在Engine启动类中硬编码设置datax.home。
+- 修改插件 streamreader 代码
+- 再次 maven 打包，使JarLoader 能够加载到最新的 streamreader 代码。
+- 调试代码
+
+在以上步骤中，打包完全不必要且最耗时,等待打包也最煎熬。
+
+所以我编写一个新的模块(datax-example)，此模块特用于本地调试和复现 BUG。如果模块顺利编写完成，那么以上流程将被简化至两步。
+
+- 修改插件 streamreader 代码。
+- 调试代码
+
+
+
+### 目录结构
+该目录结构演示了如何使用datax-example-core编写测试用例，和校验代码流程。
+
+
+### 实现原理
+
+- 不修改原有的ConfigParer,使用新的ExampleConfigParser,仅用于example模块。他不依赖datax.home,而是依赖ide编译后的target目录
+- 将ide的target目录作为每个插件的目录类加载目录。
+
+
+
+### 如何使用
+1.修改插件的pom文件，做如下改动。以streamreader为例。
+改动前
+```xml
+
+
+
+
+ maven-compiler-plugin
+
+ ${jdk-version}
+ ${jdk-version}
+ ${project-sourceEncoding}
+
+
+
+
+```
+改动后
+```xml
+
+
+
+
+ src/main/resources
+
+ **/*.*
+
+ true
+
+
+
+
+
+ maven-compiler-plugin
+
+ ${jdk-version}
+ ${jdk-version}
+ ${project-sourceEncoding}
+
+
+
+
+```
+#### 在测试模块模块使用
+参考datax-example/datax-example-streamreader的StreamReader2StreamWriterTest.java
+```java
+public class StreamReader2StreamWriterTest {
+ @Test
+ public void testStreamReader2StreamWriter() {
+ String path = "/stream2stream.json";
+ String jobPath = PathUtil.getAbsolutePathFromClassPath(path);
+ ExampleContainer.start(jobPath);
+ }
+}
+
+```
+参考datax-example/datax-example-neo4j的StreamReader2Neo4jWriterTest
+```java
+public class StreamReader2Neo4jWriterTest{
+@Test
+ public void streamReader2Neo4j() {
+
+ deleteHistoryIfExist();
+
+ String path = "/streamreader2neo4j.json";
+ String jobPath = PathUtil.getAbsolutePathFromClassPath(path);
+
+ ExampleContainer.start(jobPath);
+
+ //根据channel和reader的mock数据，校验结果集是否符合预期
+ verifyWriteResult();
+ }
+}
+```
\ No newline at end of file
diff --git a/datax-example/doc/img/img01.png b/datax-example/doc/img/img01.png
new file mode 100644
index 00000000..d0431c1a
Binary files /dev/null and b/datax-example/doc/img/img01.png differ
diff --git a/datax-example/doc/img/img02.png b/datax-example/doc/img/img02.png
new file mode 100644
index 00000000..eec860d4
Binary files /dev/null and b/datax-example/doc/img/img02.png differ
diff --git a/datax-example/doc/img/img03.png b/datax-example/doc/img/img03.png
new file mode 100644
index 00000000..731f81bd
Binary files /dev/null and b/datax-example/doc/img/img03.png differ
diff --git a/datax-example/pom.xml b/datax-example/pom.xml
new file mode 100644
index 00000000..9c4c9200
--- /dev/null
+++ b/datax-example/pom.xml
@@ -0,0 +1,68 @@
+
+
+ 4.0.0
+
+ com.alibaba.datax
+ datax-all
+ 0.0.1-SNAPSHOT
+
+
+ datax-example
+ pom
+
+ datax-example-core
+ datax-example-streamreader
+ datax-example-neo4j
+
+
+
+ 8
+ 8
+ UTF-8
+ 4.13.2
+
+
+
+ com.alibaba.datax
+ datax-common
+ 0.0.1-SNAPSHOT
+
+
+ com.alibaba.datax
+ datax-core
+ 0.0.1-SNAPSHOT
+
+
+ junit
+ junit
+ ${junit4.version}
+ test
+
+
+
+
+
+
+ src/main/resources
+
+ **/*.*
+
+ true
+
+
+
+
+
+ maven-compiler-plugin
+
+ ${jdk-version}
+ ${jdk-version}
+ ${project-sourceEncoding}
+
+
+
+
+
+
\ No newline at end of file
diff --git a/dataxPluginDev.md b/dataxPluginDev.md
index 4483f270..098bf819 100644
--- a/dataxPluginDev.md
+++ b/dataxPluginDev.md
@@ -447,6 +447,9 @@ DataX的内部类型在实现上会选用不同的java类型：
3. 用户在插件中在`reader`/`writer`配置的`name`字段指定插件名字。框架根据插件的类型（`reader`/`writer`）和插件名称去插件的路径下扫描所有的jar，加入`classpath`。
4. 根据插件配置中定义的入口类，框架通过反射实例化对应的`Job`和`Task`对象。
+### 编写测试用例
+1. 在datax-example工程下新建新的插件测试模块,调用`ExampleContainer.start(jobPath)`方法来检测你的代码逻辑是否正确。[datax-example使用](https://github.com/alibaba/DataX/datax-example/doc/README.md)
+
## 三、Last but not Least
diff --git a/doriswriter/doc/mysql2doris.json b/doriswriter/doc/mysql2doris.json
index 6992a2be..5810d6db 100644
--- a/doriswriter/doc/mysql2doris.json
+++ b/doriswriter/doc/mysql2doris.json
@@ -29,9 +29,11 @@
"postSql": [],
"preSql": [],
"connection": [
+ {
"jdbcUrl":"jdbc:mysql://192.168.1.1:9030/",
"table":["xxx"],
"selectedDatabase":"xxxx"
+ }
]
}
}
diff --git a/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisJsonCodec.java b/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisJsonCodec.java
index e6c05733..68abd9eb 100644
--- a/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisJsonCodec.java
+++ b/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisJsonCodec.java
@@ -1,7 +1,7 @@
package com.alibaba.datax.plugin.writer.doriswriter;
import com.alibaba.datax.common.element.Record;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import java.util.HashMap;
import java.util.List;
diff --git a/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisStreamLoadObserver.java b/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisStreamLoadObserver.java
index efb3d9db..6f7e9a5a 100644
--- a/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisStreamLoadObserver.java
+++ b/doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisStreamLoadObserver.java
@@ -1,6 +1,6 @@
package com.alibaba.datax.plugin.writer.doriswriter;
-import com.alibaba.fastjson.JSON;
+import com.alibaba.fastjson2.JSON;
import org.apache.commons.codec.binary.Base64;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHeaders;
diff --git a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchClient.java b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchClient.java
index 12ac3dd9..08486e1f 100644
--- a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchClient.java
+++ b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchClient.java
@@ -5,8 +5,8 @@ import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.plugin.writer.elasticsearchwriter.jest.ClusterInfo;
import com.alibaba.datax.plugin.writer.elasticsearchwriter.jest.ClusterInfoResult;
import com.alibaba.datax.plugin.writer.elasticsearchwriter.jest.PutMapping7;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.JSONObject;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
@@ -53,6 +53,8 @@ public class ElasticSearchClient {
public ElasticSearchClient(Configuration conf) {
this.conf = conf;
String endpoint = Key.getEndpoint(conf);
+ //es是支持集群写入的
+ String[] endpoints = endpoint.split(",");
String user = Key.getUsername(conf);
String passwd = Key.getPassword(conf);
boolean multiThread = Key.isMultiThread(conf);
@@ -63,7 +65,7 @@ public class ElasticSearchClient {
int totalConnection = this.conf.getInt("maxTotalConnection", 200);
JestClientFactory factory = new JestClientFactory();
Builder httpClientConfig = new HttpClientConfig
- .Builder(endpoint)
+ .Builder(Arrays.asList(endpoints))
// .setPreemptiveAuth(new HttpHost(endpoint))
.multiThreaded(multiThread)
.connTimeout(readTimeout)
diff --git a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchWriter.java b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchWriter.java
index 6236e333..2c8ed2d0 100644
--- a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchWriter.java
+++ b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/ElasticSearchWriter.java
@@ -9,11 +9,11 @@ import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.common.util.DataXCaseEnvUtil;
import com.alibaba.datax.common.util.RetryUtil;
import com.alibaba.datax.plugin.writer.elasticsearchwriter.Key.ActionType;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.JSONArray;
-import com.alibaba.fastjson.JSONObject;
-import com.alibaba.fastjson.TypeReference;
-import com.alibaba.fastjson.serializer.SerializerFeature;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.JSONArray;
+import com.alibaba.fastjson2.JSONObject;
+import com.alibaba.fastjson2.TypeReference;
+import com.alibaba.fastjson2.JSONWriter;
import com.google.common.base.Joiner;
import io.searchbox.client.JestResult;
import io.searchbox.core.*;
@@ -927,9 +927,8 @@ public class ElasticSearchWriter extends Writer {
Index.Builder builder = null;
if (this.enableWriteNull) {
builder = new Index.Builder(
- JSONObject.toJSONString(data, SerializerFeature.WriteMapNullValue,
- SerializerFeature.QuoteFieldNames, SerializerFeature.SkipTransientField,
- SerializerFeature.WriteEnumUsingToString, SerializerFeature.SortField));
+ JSONObject.toJSONString(data, JSONWriter.Feature.WriteMapNullValue,
+ JSONWriter.Feature.WriteEnumUsingToString));
} else {
builder = new Index.Builder(JSONObject.toJSONString(data));
}
@@ -958,9 +957,8 @@ public class ElasticSearchWriter extends Writer {
if (this.enableWriteNull) {
// write: {a:"1",b:null}
update = new Update.Builder(
- JSONObject.toJSONString(updateDoc, SerializerFeature.WriteMapNullValue,
- SerializerFeature.QuoteFieldNames, SerializerFeature.SkipTransientField,
- SerializerFeature.WriteEnumUsingToString, SerializerFeature.SortField));
+ JSONObject.toJSONString(updateDoc, JSONWriter.Feature.WriteMapNullValue,
+ JSONWriter.Feature.WriteEnumUsingToString));
// 在DEFAULT_GENERATE_FEATURE基础上，只增加了SerializerFeature.WRITE_MAP_NULL_FEATURES
} else {
// write: {"a":"1"}
diff --git a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonPathUtil.java b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonPathUtil.java
index 49703435..e7619e7c 100644
--- a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonPathUtil.java
+++ b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonPathUtil.java
@@ -2,7 +2,7 @@ package com.alibaba.datax.plugin.writer.elasticsearchwriter;
import java.util.List;
-import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson2.JSONObject;
public class JsonPathUtil {
diff --git a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonUtil.java b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonUtil.java
index e73c87be..ad6c01be 100644
--- a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonUtil.java
+++ b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/JsonUtil.java
@@ -1,8 +1,8 @@
package com.alibaba.datax.plugin.writer.elasticsearchwriter;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.JSONException;
-import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.JSONException;
+import com.alibaba.fastjson2.JSONObject;
/**
* @author bozu
diff --git a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/Key.java b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/Key.java
index af197711..fcaac935 100644
--- a/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/Key.java
+++ b/elasticsearchwriter/src/main/java/com/alibaba/datax/plugin/writer/elasticsearchwriter/Key.java
@@ -1,8 +1,8 @@
package com.alibaba.datax.plugin.writer.elasticsearchwriter;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import org.apache.commons.lang3.StringUtils;
diff --git a/ftpreader/pom.xml b/ftpreader/pom.xml
index 7778d491..57bf889d 100755
--- a/ftpreader/pom.xml
+++ b/ftpreader/pom.xml
@@ -45,7 +45,7 @@
com.jcraft
jsch
- 0.1.51
+ 0.1.54
commons-net
@@ -89,4 +89,4 @@
-
+
diff --git a/ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/SftpHelper.java b/ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/SftpHelper.java
index d25b040c..6e42e10c 100644
--- a/ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/SftpHelper.java
+++ b/ftpreader/src/main/java/com/alibaba/datax/plugin/reader/ftpreader/SftpHelper.java
@@ -64,6 +64,8 @@ public class SftpHelper extends FtpHelper {
String message = String.format("请确认连接ftp服务器端口是否正确，错误的端口: [%s] ", port);
LOG.error(message);
throw DataXException.asDataXException(FtpReaderErrorCode.FAIL_LOGIN, message, e);
+ }else{
+ throw DataXException.asDataXException(FtpReaderErrorCode.COMMAND_FTP_IO_EXCEPTION, "", e);
}
}else {
if("Auth fail".equals(e.getMessage())){
diff --git a/ftpwriter/doc/ftpwriter.md b/ftpwriter/doc/ftpwriter.md
index 6b1b2687..a38a1052 100644
--- a/ftpwriter/doc/ftpwriter.md
+++ b/ftpwriter/doc/ftpwriter.md
@@ -24,7 +24,7 @@ FtpWriter实现了从DataX协议转为FTP文件功能，FTP文件本身是无结
我们不能做到：
-1. 单个文件不能支持并发写入。
+1. 单个文件并发写入。
## 3 功能说明
diff --git a/ftpwriter/pom.xml b/ftpwriter/pom.xml
index 69ec4a07..bf7ce83d 100644
--- a/ftpwriter/pom.xml
+++ b/ftpwriter/pom.xml
@@ -45,7 +45,7 @@
com.jcraft
jsch
- 0.1.51
+ 0.1.54
commons-net
diff --git a/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/SftpHelperImpl.java b/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/SftpHelperImpl.java
index e6d78629..e748f12c 100644
--- a/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/SftpHelperImpl.java
+++ b/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/SftpHelperImpl.java
@@ -14,8 +14,8 @@ import org.slf4j.LoggerFactory;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.plugin.writer.ftpwriter.FtpWriterErrorCode;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.serializer.SerializerFeature;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.JSONWriter;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
@@ -251,7 +251,7 @@ public class SftpHelperImpl implements IFtpHelper {
@SuppressWarnings("rawtypes")
Vector allFiles = this.channelSftp.ls(dir);
LOG.debug(String.format("ls: %s", JSON.toJSONString(allFiles,
- SerializerFeature.UseSingleQuotes)));
+ JSONWriter.Feature.UseSingleQuotes)));
for (int i = 0; i < allFiles.size(); i++) {
LsEntry le = (LsEntry) allFiles.get(i);
String strName = le.getFilename();
diff --git a/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/StandardFtpHelperImpl.java b/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/StandardFtpHelperImpl.java
index 8999b0a8..d5b9a746 100644
--- a/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/StandardFtpHelperImpl.java
+++ b/ftpwriter/src/main/java/com/alibaba/datax/plugin/writer/ftpwriter/util/StandardFtpHelperImpl.java
@@ -18,8 +18,8 @@ import org.slf4j.LoggerFactory;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.plugin.writer.ftpwriter.FtpWriterErrorCode;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.serializer.SerializerFeature;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.JSONWriter;
public class StandardFtpHelperImpl implements IFtpHelper {
private static final Logger LOG = LoggerFactory
@@ -244,7 +244,7 @@ public class StandardFtpHelperImpl implements IFtpHelper {
FTPFile[] fs = this.ftpClient.listFiles(dir);
// LOG.debug(JSON.toJSONString(this.ftpClient.listNames(dir)));
LOG.debug(String.format("ls: %s",
- JSON.toJSONString(fs, SerializerFeature.UseSingleQuotes)));
+ JSON.toJSONString(fs, JSONWriter.Feature.UseSingleQuotes)));
for (FTPFile ff : fs) {
String strName = ff.getName();
if (strName.startsWith(prefixFileName)) {
diff --git a/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/mapping/DefaultGdbMapper.java b/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/mapping/DefaultGdbMapper.java
index 73a94cf5..2c015879 100644
--- a/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/mapping/DefaultGdbMapper.java
+++ b/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/mapping/DefaultGdbMapper.java
@@ -19,8 +19,8 @@ import com.alibaba.datax.plugin.writer.gdbwriter.Key;
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbEdge;
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbElement;
import com.alibaba.datax.plugin.writer.gdbwriter.model.GdbVertex;
-import com.alibaba.fastjson.JSONArray;
-import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson2.JSONArray;
+import com.alibaba.fastjson2.JSONObject;
import lombok.extern.slf4j.Slf4j;
diff --git a/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/util/ConfigHelper.java b/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/util/ConfigHelper.java
index 178b5e7c..644f8898 100644
--- a/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/util/ConfigHelper.java
+++ b/gdbwriter/src/main/java/com/alibaba/datax/plugin/writer/gdbwriter/util/ConfigHelper.java
@@ -12,8 +12,8 @@ import org.apache.commons.lang3.StringUtils;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.plugin.writer.gdbwriter.GdbWriterErrorCode;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.JSONObject;
/**
* @author jerrywang
diff --git a/hbase094xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase094xreader/Hbase094xHelper.java b/hbase094xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase094xreader/Hbase094xHelper.java
index c3e2a212..b9f16b17 100644
--- a/hbase094xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase094xreader/Hbase094xHelper.java
+++ b/hbase094xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase094xreader/Hbase094xHelper.java
@@ -2,8 +2,8 @@ package com.alibaba.datax.plugin.reader.hbase094xreader;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.Validate;
import org.apache.hadoop.fs.Path;
diff --git a/hbase094xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase094xwriter/Hbase094xHelper.java b/hbase094xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase094xwriter/Hbase094xHelper.java
index f671d31d..00b128f3 100644
--- a/hbase094xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase094xwriter/Hbase094xHelper.java
+++ b/hbase094xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase094xwriter/Hbase094xHelper.java
@@ -2,8 +2,8 @@ package com.alibaba.datax.plugin.writer.hbase094xwriter;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.Validate;
import org.apache.hadoop.fs.Path;
diff --git a/hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xHelper.java b/hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xHelper.java
index 643072a9..82ad7122 100644
--- a/hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xHelper.java
+++ b/hbase11xreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xreader/Hbase11xHelper.java
@@ -2,8 +2,8 @@ package com.alibaba.datax.plugin.reader.hbase11xreader;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.Validate;
import org.apache.hadoop.hbase.HBaseConfiguration;
diff --git a/hbase11xsqlreader/doc/hbase11xsqlreader.md b/hbase11xsqlreader/doc/hbase11xsqlreader.md
index 03261a1f..9f70077f 100644
--- a/hbase11xsqlreader/doc/hbase11xsqlreader.md
+++ b/hbase11xsqlreader/doc/hbase11xsqlreader.md
@@ -60,12 +60,16 @@ hbase11xsqlreader插件实现了从Phoenix(HBase SQL)读取数据。在底层实
//填写连接Phoenix的hbase集群zk地址
"hbaseConfig": {
"hbase.zookeeper.quorum": "hb-proxy-xxx-002.hbase.rds.aliyuncs.com,hb-proxy-xxx-001.hbase.rds.aliyuncs.com,hb-proxy-xxx-003.hbase.rds.aliyuncs.com"
- },
+ },
+ //填写要读取的phoenix的命名空间
+ "schema": "TAG",
//填写要读取的phoenix的表名
"table": "US_POPULATION",
//填写要读取的列名，不填读取所有列
"column": [
- ]
+ ],
+ //查询条件
+ "where": "id="
}
},
"writer": {
@@ -92,11 +96,18 @@ hbase11xsqlreader插件实现了从Phoenix(HBase SQL)读取数据。在底层实
* 必选：是
+ * 默认值：无
+* **schema**
+
+ * 描述：编写Phoenix中的namespace，该值设置为''
+
+ * 必选：是
+
* 默认值：无
* **table**
- * 描述：编写Phoenix中的表名,如果有namespace，该值设置为'namespace.tablename'
+ * 描述：编写Phoenix中的表名，该值设置为'tablename'
* 必选：是
@@ -109,7 +120,13 @@ hbase11xsqlreader插件实现了从Phoenix(HBase SQL)读取数据。在底层实
* 必选：是
* 默认值：无
+* **where**
+
+ * 描述：填写需要从phoenix表中读取条件判断。
+ * 可选：是
+
+ * 默认值：无
### 3.3 类型转换
@@ -172,11 +189,14 @@ hbase11xsqlreader插件实现了从Phoenix(HBase SQL)读取数据。在底层实
"hbaseConfig": {
"hbase.zookeeper.quorum": "hb-proxy-xxx-002.hbase.rds.aliyuncs.com,hb-proxy-xxx-001.hbase.rds.aliyuncs.com,hb-proxy-xxx-003.hbase.rds.aliyuncs.com"
},
+ "schema": "TAG",
//填写要读取的phoenix的表名
"table": "US_POPULATION",
//填写要读取的列名，不填读取所有列
"column": [
- ]
+ ],
+ //查询条件
+ "where": "id="
}
},
"writer": {
@@ -204,7 +224,13 @@ hbase11xsqlreader插件实现了从Phoenix(HBase SQL)读取数据。在底层实
* 必选：是
* 默认值：无
-
+* **schema**
+
+ * 描述：编写Phoenix中的namespace，该值设置为''
+
+ * 必选：是
+
+ * 默认值：无
* **table**
* 描述：编写Phoenix中的表名,如果有namespace，该值设置为'namespace.tablename'
@@ -220,7 +246,13 @@ hbase11xsqlreader插件实现了从Phoenix(HBase SQL)读取数据。在底层实
* 必选：是
* 默认值：无
+ * **where**
+ * 描述：填写需要从phoenix表中读取条件判断。
+
+ * 可选：是
+
+ * 默认值：无
### 3.3 类型转换
diff --git a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLHelper.java b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLHelper.java
index 5309d1d9..cf4304ee 100644
--- a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLHelper.java
+++ b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLHelper.java
@@ -2,8 +2,8 @@ package com.alibaba.datax.plugin.reader.hbase11xsqlreader;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.mapreduce.InputSplit;
@@ -26,9 +26,7 @@ import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
public class HbaseSQLHelper {
@@ -50,11 +48,15 @@ public class HbaseSQLHelper {
String zkUrl = readerConfig.getZkUrl();
PhoenixConfigurationUtil.setInputClass(conf, PhoenixRecordWritable.class);
- PhoenixConfigurationUtil.setInputTableName(conf, table);
+
+ PhoenixConfigurationUtil.setInputTableName(conf, readerConfig.getSchema()+"."+table);
if (!columns.isEmpty()) {
PhoenixConfigurationUtil.setSelectColumnNames(conf, columns.toArray(new String[columns.size()]));
}
+ if(Objects.nonNull(readerConfig.getWhere())){
+ PhoenixConfigurationUtil.setInputTableConditions(conf,readerConfig.getWhere());
+ }
PhoenixEmbeddedDriver.ConnectionInfo info = null;
try {
info = PhoenixEmbeddedDriver.ConnectionInfo.create(zkUrl);
@@ -67,15 +69,19 @@ public class HbaseSQLHelper {
conf.setInt(HConstants.ZOOKEEPER_CLIENT_PORT, info.getPort());
if (info.getRootNode() != null)
conf.set(HConstants.ZOOKEEPER_ZNODE_PARENT, info.getRootNode());
+ conf.set(Key.NAME_SPACE_MAPPING_ENABLED,"true");
+ conf.set(Key.SYSTEM_TABLES_TO_NAMESPACE,"true");
return conf;
}
- public static List getPColumnNames(String connectionString, String tableName) throws SQLException {
- Connection con =
- DriverManager.getConnection(connectionString);
+ public static List getPColumnNames(String connectionString, String tableName,String schema) throws SQLException {
+ Properties pro = new Properties();
+ pro.put(Key.NAME_SPACE_MAPPING_ENABLED, true);
+ pro.put(Key.SYSTEM_TABLES_TO_NAMESPACE, true);
+ Connection con = DriverManager.getConnection(connectionString,pro);
PhoenixConnection phoenixConnection = con.unwrap(PhoenixConnection.class);
MetaDataClient metaDataClient = new MetaDataClient(phoenixConnection);
- PTable table = metaDataClient.updateCache("", tableName).getTable();
+ PTable table = metaDataClient.updateCache(schema, tableName).getTable();
List columnNames = new ArrayList();
for (PColumn pColumn : table.getColumns()) {
if (!pColumn.getName().getString().equals(SaltingUtil.SALTING_COLUMN_NAME))
diff --git a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderConfig.java b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderConfig.java
index ab06f6e1..37060986 100644
--- a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderConfig.java
+++ b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderConfig.java
@@ -9,6 +9,7 @@ import org.slf4j.LoggerFactory;
import java.sql.SQLException;
import java.util.List;
+import java.util.StringJoiner;
public class HbaseSQLReaderConfig {
private final static Logger LOG = LoggerFactory.getLogger(HbaseSQLReaderConfig.class);
@@ -27,6 +28,9 @@ public class HbaseSQLReaderConfig {
private String tableName;
private List columns; // 目的表的所有列的列名，包括主键和非主键，不包括时间列
+ private String where;//条件
+
+ private String schema;//
/**
* @return 获取原始的datax配置
*/
@@ -96,22 +100,27 @@ public class HbaseSQLReaderConfig {
}
String zkQuorum = zkCfg.getFirst();
String znode = zkCfg.getSecond();
+
if (zkQuorum == null || zkQuorum.isEmpty()) {
throw DataXException.asDataXException(
HbaseSQLReaderErrorCode.ILLEGAL_VALUE, "HBase的hbase.zookeeper.quorum配置不能为空" );
}
// 生成sql使用的连接字符串， 格式： jdbc:hbase:zk_quorum:2181:/znode_parent
- cfg.connectionString = "jdbc:phoenix:" + zkQuorum;
- cfg.zkUrl = zkQuorum + ":2181";
+ StringBuilder connectionString=new StringBuilder("jdbc:phoenix:");
+ connectionString.append(zkQuorum);
+ cfg.connectionString = connectionString.toString();
+ StringBuilder zkUrl =new StringBuilder(zkQuorum);
+ cfg.zkUrl = zkUrl.append(":2181").toString();
if (!znode.isEmpty()) {
- cfg.connectionString += cfg.connectionString + ":" + znode;
- cfg.zkUrl += cfg.zkUrl + ":" + znode;
+ cfg.connectionString = connectionString.append(":").append(znode).toString();
+ cfg.zkUrl=zkUrl.append(":").append(znode).toString();
}
}
private static void parseTableConfig(HbaseSQLReaderConfig cfg, Configuration dataxCfg) {
// 解析并检查表名
cfg.tableName = dataxCfg.getString(Key.TABLE);
+ cfg.schema = dataxCfg.getString(Key.SCHEMA);
if (cfg.tableName == null || cfg.tableName.isEmpty()) {
throw DataXException.asDataXException(
HbaseSQLReaderErrorCode.ILLEGAL_VALUE, "HBase的tableName配置不能为空,请检查并修改配置." );
@@ -124,13 +133,14 @@ public class HbaseSQLReaderConfig {
HbaseSQLReaderErrorCode.ILLEGAL_VALUE, "您配置的tableName含有非法字符{0}，请检查您的配置.");
} else if (cfg.columns.isEmpty()) {
try {
- cfg.columns = HbaseSQLHelper.getPColumnNames(cfg.connectionString, cfg.tableName);
+ cfg.columns = HbaseSQLHelper.getPColumnNames(cfg.connectionString, cfg.tableName,cfg.schema);
dataxCfg.set(Key.COLUMN, cfg.columns);
} catch (SQLException e) {
throw DataXException.asDataXException(
HbaseSQLReaderErrorCode.GET_PHOENIX_COLUMN_ERROR, "HBase的columns配置不能为空,请添加目标表的列名配置." + e.getMessage(), e);
}
}
+ cfg.where=dataxCfg.getString(Key.WHERE);
}
@Override
@@ -151,6 +161,8 @@ public class HbaseSQLReaderConfig {
ret.append(",");
}
ret.setLength(ret.length() - 1);
+ ret.append("[where=]").append(getWhere());
+ ret.append("[schema=]").append(getSchema());
ret.append("\n");
return ret.toString();
@@ -161,4 +173,20 @@ public class HbaseSQLReaderConfig {
*/
private HbaseSQLReaderConfig() {
}
+
+ public String getWhere() {
+ return where;
+ }
+
+ public void setWhere(String where) {
+ this.where = where;
+ }
+
+ public String getSchema() {
+ return schema;
+ }
+
+ public void setSchema(String schema) {
+ this.schema = schema;
+ }
}
diff --git a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderTask.java b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderTask.java
index 1ca22c6f..461649d1 100644
--- a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderTask.java
+++ b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/HbaseSQLReaderTask.java
@@ -19,10 +19,8 @@ import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.math.BigDecimal;
import java.sql.*;
-import java.util.HashMap;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
+import java.sql.Date;
+import java.util.*;
/**
* Created by admin on 1/3/18.
@@ -42,11 +40,14 @@ public class HbaseSQLReaderTask {
}
private void getPColumns() throws SQLException {
+ Properties pro = new Properties();
+ pro.put(Key.NAME_SPACE_MAPPING_ENABLED, true);
+ pro.put(Key.SYSTEM_TABLES_TO_NAMESPACE, true);
Connection con =
- DriverManager.getConnection(this.readerConfig.getConnectionString());
+ DriverManager.getConnection(this.readerConfig.getConnectionString(),pro);
PhoenixConnection phoenixConnection = con.unwrap(PhoenixConnection.class);
MetaDataClient metaDataClient = new MetaDataClient(phoenixConnection);
- PTable table = metaDataClient.updateCache("", this.readerConfig.getTableName()).getTable();
+ PTable table = metaDataClient.updateCache(this.readerConfig.getSchema(), this.readerConfig.getTableName()).getTable();
List columnNames = this.readerConfig.getColumns();
for (PColumn pColumn : table.getColumns()) {
if (columnNames.contains(pColumn.getName().getString())) {
diff --git a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/Key.java b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/Key.java
index 7987d6c8..f8453add 100644
--- a/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/Key.java
+++ b/hbase11xsqlreader/src/main/java/com/alibaba/datax/plugin/reader/hbase11xsqlreader/Key.java
@@ -24,5 +24,18 @@ public final class Key {
* 【必选】列配置
*/
public final static String COLUMN = "column";
+ /**
+ *
+ */
+ public static final String WHERE = "where";
+
+ /**
+ * 【可选】Phoenix表所属schema，默认为空
+ */
+ public static final String SCHEMA = "schema";
+
+ public static final String NAME_SPACE_MAPPING_ENABLED = "phoenix.schema.isNamespaceMappingEnabled";
+
+ public static final String SYSTEM_TABLES_TO_NAMESPACE = "phoenix.schema.mapSystemTablesToNamespace";
}
diff --git a/hbase11xsqlwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xsqlwriter/HbaseSQLHelper.java b/hbase11xsqlwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xsqlwriter/HbaseSQLHelper.java
index 41e57d4e..d1b23fdf 100644
--- a/hbase11xsqlwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xsqlwriter/HbaseSQLHelper.java
+++ b/hbase11xsqlwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xsqlwriter/HbaseSQLHelper.java
@@ -2,8 +2,8 @@ package com.alibaba.datax.plugin.writer.hbase11xsqlwriter;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.util.Pair;
diff --git a/hbase11xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xwriter/Hbase11xHelper.java b/hbase11xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xwriter/Hbase11xHelper.java
index 94b13b60..2889b647 100644
--- a/hbase11xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xwriter/Hbase11xHelper.java
+++ b/hbase11xwriter/src/main/java/com/alibaba/datax/plugin/writer/hbase11xwriter/Hbase11xHelper.java
@@ -2,8 +2,8 @@ package com.alibaba.datax.plugin.writer.hbase11xwriter;
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.TypeReference;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.TypeReference;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.Validate;
import org.apache.hadoop.hbase.HBaseConfiguration;
diff --git a/hdfsreader/doc/hdfsreader.md b/hdfsreader/doc/hdfsreader.md
index cd83c530..ca9a021f 100644
--- a/hdfsreader/doc/hdfsreader.md
+++ b/hdfsreader/doc/hdfsreader.md
@@ -166,20 +166,20 @@ HdfsReader实现了从Hadoop分布式文件系统Hdfs中读取文件数据并转
默认情况下，用户可以全部按照String类型读取数据，配置如下：
```json
- "column": ["*"]
+ "column": ["*"]
```
用户可以指定Column字段信息，配置如下：
```json
-{
- "type": "long",
- "index": 0 //从本地文件文本第一列获取int字段
-},
-{
- "type": "string",
- "value": "alibaba" //HdfsReader内部生成alibaba的字符串字段作为当前字段
-}
+ {
+ "type": "long",
+ "index": 0 //从本地文件文本第一列获取int字段
+ },
+ {
+ "type": "string",
+ "value": "alibaba" //HdfsReader内部生成alibaba的字符串字段作为当前字段
+ }
```
对于用户指定Column信息，type必须填写，index/value必须选择其一。
diff --git a/hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/DFSUtil.java b/hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/DFSUtil.java
index c39d3847..5ba572e1 100644
--- a/hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/DFSUtil.java
+++ b/hdfsreader/src/main/java/com/alibaba/datax/plugin/reader/hdfsreader/DFSUtil.java
@@ -8,8 +8,8 @@ import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.plugin.unstructuredstorage.reader.ColumnEntry;
import com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderErrorCode;
import com.alibaba.datax.plugin.unstructuredstorage.reader.UnstructuredStorageReaderUtil;
-import com.alibaba.fastjson.JSON;
-import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson2.JSON;
+import com.alibaba.fastjson2.JSONObject;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileStatus;
@@ -331,26 +331,30 @@ public class DFSUtil {
//If the network disconnected, will retry 45 times, each time the retry interval for 20 seconds
//Each file as a split
//TODO multy threads
- InputSplit[] splits = in.getSplits(conf, 1);
+ // OrcInputFormat getSplits params numSplits not used, splits size = block numbers
+ InputSplit[] splits = in.getSplits(conf, -1);
+ for (InputSplit split : splits) {
+ {
+ RecordReader reader = in.getRecordReader(split, conf, Reporter.NULL);
+ Object key = reader.createKey();
+ Object value = reader.createValue();
+ // 获取列信息
+ List fields = inspector.getAllStructFieldRefs();
- RecordReader reader = in.getRecordReader(splits[0], conf, Reporter.NULL);
- Object key = reader.createKey();
- Object value = reader.createValue();
- // 获取列信息
- List fields = inspector.getAllStructFieldRefs();
+ List
com.google.code.gson
gson
2.2.4
+
+ com.alibaba
+ fastjson
+ 1.2.83_noneautotype
+ compile
+
+
+
+ src/main/java
+
+ **/*.properties
+
+
+
@@ -98,10 +108,6 @@
maven-surefire-plugin
2.5
- all
- 10
- true
- -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=.
**/unittest/*.java
**/functiontest/*.java
@@ -111,4 +117,3 @@
-
diff --git a/otsreader/src/main/assembly/package.xml b/otsreader/src/main/assembly/package.xml
index 7ee305d1..cb90f3e8 100644
--- a/otsreader/src/main/assembly/package.xml
+++ b/otsreader/src/main/assembly/package.xml
@@ -12,8 +12,8 @@
src/main/resources
plugin.json
- plugin_job_template.json
-
+ plugin_job_template.json
+
plugin/reader/otsreader
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/IOtsReaderMasterProxy.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/IOtsReaderMasterProxy.java
new file mode 100644
index 00000000..ee622e16
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/IOtsReaderMasterProxy.java
@@ -0,0 +1,15 @@
+package com.alibaba.datax.plugin.reader.otsreader;
+
+import java.util.List;
+
+import com.alibaba.datax.common.util.Configuration;
+
+public interface IOtsReaderMasterProxy {
+
+ public void init(Configuration param) throws Exception;
+
+ public List split(int num) throws Exception;
+
+ public void close();
+
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/IOtsReaderSlaveProxy.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/IOtsReaderSlaveProxy.java
new file mode 100644
index 00000000..d1100a2a
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/IOtsReaderSlaveProxy.java
@@ -0,0 +1,26 @@
+package com.alibaba.datax.plugin.reader.otsreader;
+
+import com.alibaba.datax.common.plugin.RecordSender;
+import com.alibaba.datax.common.util.Configuration;
+
+/**
+ * OTS Reader工作进程接口
+ */
+public interface IOtsReaderSlaveProxy {
+ /**
+ * 初始化函数，解析配置、初始化相关资源
+ */
+ public void init(Configuration configuration);
+
+ /**
+ * 关闭函数，释放资源
+ */
+ public void close();
+
+ /**
+ * 数据导出函数
+ * @param recordSender
+ * @throws Exception
+ */
+ public void startRead(RecordSender recordSender) throws Exception;
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReader.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReader.java
index 8880c07e..c6bc44b8 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReader.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReader.java
@@ -1,45 +1,48 @@
package com.alibaba.datax.plugin.reader.otsreader;
-import java.util.List;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import com.alibaba.datax.common.exception.DataXException;
import com.alibaba.datax.common.plugin.RecordSender;
import com.alibaba.datax.common.spi.Reader;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.datax.plugin.reader.otsreader.utils.Common;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSMode;
+import com.alibaba.datax.plugin.reader.otsreader.utils.Constant;
+import com.alibaba.datax.plugin.reader.otsreader.utils.GsonParser;
+import com.alibaba.datax.plugin.reader.otsreader.utils.OtsReaderError;
+import com.alicloud.openservices.tablestore.TableStoreException;
import com.aliyun.openservices.ots.ClientException;
-import com.aliyun.openservices.ots.OTSException;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
public class OtsReader extends Reader {
public static class Job extends Reader.Job {
private static final Logger LOG = LoggerFactory.getLogger(Job.class);
- private OtsReaderMasterProxy proxy = new OtsReaderMasterProxy();
+ //private static final MessageSource MESSAGE_SOURCE = MessageSource.loadResourceBundle(OtsReader.class);
+ private IOtsReaderMasterProxy proxy = null;
+
@Override
- public void init() {
+ public void init() {
LOG.info("init() begin ...");
+
+ proxy = new OtsReaderMasterProxy();
try {
this.proxy.init(getPluginJobConf());
- } catch (OTSException e) {
- LOG.error("OTSException. ErrorCode:{}, ErrorMsg:{}, RequestId:{}",
- new Object[]{e.getErrorCode(), e.getMessage(), e.getRequestId()});
- LOG.error("Stack", e);
- throw DataXException.asDataXException(new OtsReaderError(e.getErrorCode(), "OTS端的错误"), Common.getDetailMessage(e), e);
+ } catch (TableStoreException e) {
+ LOG.error("OTSException: {}", e.toString(), e);
+ throw DataXException.asDataXException(new OtsReaderError(e.getErrorCode(), "OTS ERROR"), e.toString(), e);
} catch (ClientException e) {
- LOG.error("ClientException. ErrorCode:{}, ErrorMsg:{}",
- new Object[]{e.getErrorCode(), e.getMessage()});
- LOG.error("Stack", e);
- throw DataXException.asDataXException(new OtsReaderError(e.getErrorCode(), "OTS端的错误"), Common.getDetailMessage(e), e);
- } catch (IllegalArgumentException e) {
- LOG.error("IllegalArgumentException. ErrorMsg:{}", e.getMessage(), e);
- throw DataXException.asDataXException(OtsReaderError.INVALID_PARAM, Common.getDetailMessage(e), e);
+ LOG.error("ClientException: {}", e.toString(), e);
+ throw DataXException.asDataXException(OtsReaderError.ERROR, e.toString(), e);
} catch (Exception e) {
- LOG.error("Exception. ErrorMsg:{}", e.getMessage(), e);
- throw DataXException.asDataXException(OtsReaderError.ERROR, Common.getDetailMessage(e), e);
+ LOG.error("Exception. ErrorMsg:{}", e.toString(), e);
+ throw DataXException.asDataXException(OtsReaderError.ERROR, e.toString(), e);
}
+
LOG.info("init() end ...");
}
@@ -60,22 +63,9 @@ public class OtsReader extends Reader {
try {
confs = this.proxy.split(adviceNumber);
- } catch (OTSException e) {
- LOG.error("OTSException. ErrorCode:{}, ErrorMsg:{}, RequestId:{}",
- new Object[]{e.getErrorCode(), e.getMessage(), e.getRequestId()});
- LOG.error("Stack", e);
- throw DataXException.asDataXException(new OtsReaderError(e.getErrorCode(), "OTS端的错误"), Common.getDetailMessage(e), e);
- } catch (ClientException e) {
- LOG.error("ClientException. ErrorCode:{}, ErrorMsg:{}",
- new Object[]{e.getErrorCode(), e.getMessage()});
- LOG.error("Stack", e);
- throw DataXException.asDataXException(new OtsReaderError(e.getErrorCode(), "OTS端的错误"), Common.getDetailMessage(e), e);
- } catch (IllegalArgumentException e) {
- LOG.error("IllegalArgumentException. ErrorMsg:{}", e.getMessage(), e);
- throw DataXException.asDataXException(OtsReaderError.INVALID_PARAM, Common.getDetailMessage(e), e);
} catch (Exception e) {
LOG.error("Exception. ErrorMsg:{}", e.getMessage(), e);
- throw DataXException.asDataXException(OtsReaderError.ERROR, Common.getDetailMessage(e), e);
+ throw DataXException.asDataXException(OtsReaderError.ERROR, e.toString(), e);
}
LOG.info("split() end ...");
@@ -85,39 +75,60 @@ public class OtsReader extends Reader {
public static class Task extends Reader.Task {
private static final Logger LOG = LoggerFactory.getLogger(Task.class);
- private OtsReaderSlaveProxy proxy = new OtsReaderSlaveProxy();
+ //private static final MessageSource MESSAGE_SOURCE = MessageSource.loadResourceBundle(OtsReader.class);
+ private IOtsReaderSlaveProxy proxy = null;
@Override
public void init() {
+
+ OTSConf conf = GsonParser.jsonToConf((String) this.getPluginJobConf().get(Constant.ConfigKey.CONF));
+ // 是否使用新接口
+ if(conf.isNewVersion()) {
+ if (conf.getMode() == OTSMode.MULTI_VERSION) {
+ LOG.info("init OtsReaderSlaveProxyMultiVersion");
+ proxy = new OtsReaderSlaveMultiVersionProxy();
+ } else {
+ LOG.info("init OtsReaderSlaveProxyNormal");
+ proxy = new OtsReaderSlaveNormalProxy();
+ }
+
+ }
+ else{
+ String metaMode = conf.getMetaMode();
+ if (StringUtils.isNotBlank(metaMode) && !metaMode.equalsIgnoreCase("false")) {
+ LOG.info("init OtsMetaReaderSlaveProxy");
+ proxy = new OtsReaderSlaveMetaProxy();
+ } else {
+ LOG.info("init OtsReaderSlaveProxyOld");
+ proxy = new OtsReaderSlaveProxyOld();
+ }
+ }
+
+ proxy.init(this.getPluginJobConf());
}
@Override
public void destroy() {
+ try {
+ proxy.close();
+ } catch (Exception e) {
+ LOG.error("Exception. ErrorMsg:{}", e.toString(), e);
+ throw DataXException.asDataXException(OtsReaderError.ERROR, e.toString(), e);
+ }
}
@Override
public void startRead(RecordSender recordSender) {
- LOG.info("startRead() begin ...");
+
try {
- this.proxy.read(recordSender,getPluginJobConf());
- } catch (OTSException e) {
- LOG.error("OTSException. ErrorCode:{}, ErrorMsg:{}, RequestId:{}",
- new Object[]{e.getErrorCode(), e.getMessage(), e.getRequestId()});
- LOG.error("Stack", e);
- throw DataXException.asDataXException(new OtsReaderError(e.getErrorCode(), "OTS端的错误"), Common.getDetailMessage(e), e);
- } catch (ClientException e) {
- LOG.error("ClientException. ErrorCode:{}, ErrorMsg:{}",
- new Object[]{e.getErrorCode(), e.getMessage()});
- LOG.error("Stack", e);
- throw DataXException.asDataXException(new OtsReaderError(e.getErrorCode(), "OTS端的错误"), Common.getDetailMessage(e), e);
- } catch (IllegalArgumentException e) {
- LOG.error("IllegalArgumentException. ErrorMsg:{}", e.getMessage(), e);
- throw DataXException.asDataXException(OtsReaderError.INVALID_PARAM, Common.getDetailMessage(e), e);
+ proxy.startRead(recordSender);
} catch (Exception e) {
- LOG.error("Exception. ErrorMsg:{}", e.getMessage(), e);
- throw DataXException.asDataXException(OtsReaderError.ERROR, Common.getDetailMessage(e), e);
+ LOG.error("Exception. ErrorMsg:{}", e.toString(), e);
+ throw DataXException.asDataXException(OtsReaderError.ERROR, e.toString(), e);
}
- LOG.info("startRead() end ...");
+
+
+
}
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderMasterProxy.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderMasterProxy.java
index 2b758f06..4ecdd8c1 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderMasterProxy.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderMasterProxy.java
@@ -1,221 +1,243 @@
package com.alibaba.datax.plugin.reader.otsreader;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import com.alibaba.datax.common.util.Configuration;
import com.alibaba.datax.plugin.reader.otsreader.callable.GetFirstRowPrimaryKeyCallable;
-import com.alibaba.datax.plugin.reader.otsreader.callable.GetTableMetaCallable;
import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSConst;
import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
-import com.alibaba.datax.plugin.reader.otsreader.utils.ParamChecker;
-import com.alibaba.datax.plugin.reader.otsreader.utils.Common;
-import com.alibaba.datax.plugin.reader.otsreader.utils.GsonParser;
-import com.alibaba.datax.plugin.reader.otsreader.utils.ReaderModelParser;
-import com.alibaba.datax.plugin.reader.otsreader.utils.RangeSplit;
-import com.alibaba.datax.plugin.reader.otsreader.utils.RetryHelper;
-import com.aliyun.openservices.ots.OTSClient;
-import com.aliyun.openservices.ots.model.Direction;
-import com.aliyun.openservices.ots.model.PrimaryKeyValue;
-import com.aliyun.openservices.ots.model.RangeRowQueryCriteria;
-import com.aliyun.openservices.ots.model.RowPrimaryKey;
-import com.aliyun.openservices.ots.model.TableMeta;
+import com.alibaba.datax.plugin.reader.otsreader.utils.*;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.model.*;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataResponse;
+import com.alicloud.openservices.tablestore.model.timeseries.TimeseriesScanSplitInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
-public class OtsReaderMasterProxy {
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.List;
- private OTSConf conf = new OTSConf();
-
- private OTSRange range = null;
-
- private OTSClient ots = null;
-
- private TableMeta meta = null;
-
- private Direction direction = null;
+public class OtsReaderMasterProxy implements IOtsReaderMasterProxy {
private static final Logger LOG = LoggerFactory.getLogger(OtsReaderMasterProxy.class);
+ private OTSConf conf = null;
+ private TableMeta meta = null;
+ private SyncClientInterface ots = null;
+ private Direction direction = null;
- /**
- * 1.检查参数是否为
- * null，endpoint,accessid,accesskey,instance-name,table,column,range-begin,range-end,range-split
- * 2.检查参数是否为空字符串
- * endpoint,accessid,accesskey,instance-name,table
- * 3.检查是否为空数组
- * column
- * 4.检查Range的类型个个数是否和PrimaryKey匹配
- * column,range-begin,range-end
- * 5.检查Range Split 顺序和类型是否Range一致，类型是否于PartitionKey一致
- * column-split
- * @param param
- * @throws Exception
- */
- public void init(Configuration param) throws Exception {
- // 默认参数
- // 每次重试的时间都是上一次的一倍，当sleep时间大于30秒时，Sleep重试时间不在增长。18次能覆盖OTS的Failover时间5分钟
- conf.setRetry(param.getInt(OTSConst.RETRY, 18));
- conf.setSleepInMilliSecond(param.getInt(OTSConst.SLEEP_IN_MILLI_SECOND, 100));
-
- // 必选参数
- conf.setEndpoint(ParamChecker.checkStringAndGet(param, Key.OTS_ENDPOINT));
- conf.setAccessId(ParamChecker.checkStringAndGet(param, Key.OTS_ACCESSID));
- conf.setAccesskey(ParamChecker.checkStringAndGet(param, Key.OTS_ACCESSKEY));
- conf.setInstanceName(ParamChecker.checkStringAndGet(param, Key.OTS_INSTANCE_NAME));
- conf.setTableName(ParamChecker.checkStringAndGet(param, Key.TABLE_NAME));
-
- ots = new OTSClient(
- this.conf.getEndpoint(),
- this.conf.getAccessId(),
- this.conf.getAccesskey(),
- this.conf.getInstanceName());
-
- meta = getTableMeta(ots, conf.getTableName());
- LOG.info("Table Meta : {}", GsonParser.metaToJson(meta));
-
- conf.setColumns(ReaderModelParser.parseOTSColumnList(ParamChecker.checkListAndGet(param, Key.COLUMN, true)));
-
- Map rangeMap = ParamChecker.checkMapAndGet(param, Key.RANGE, true);
- conf.setRangeBegin(ReaderModelParser.parsePrimaryKey(ParamChecker.checkListAndGet(rangeMap, Key.RANGE_BEGIN, false)));
- conf.setRangeEnd(ReaderModelParser.parsePrimaryKey(ParamChecker.checkListAndGet(rangeMap, Key.RANGE_END, false)));
-
- range = ParamChecker.checkRangeAndGet(meta, this.conf.getRangeBegin(), this.conf.getRangeEnd());
-
- direction = ParamChecker.checkDirectionAndEnd(meta, range.getBegin(), range.getEnd());
- LOG.info("Direction : {}", direction);
-
- List points = ReaderModelParser.parsePrimaryKey(ParamChecker.checkListAndGet(rangeMap, Key.RANGE_SPLIT));
- ParamChecker.checkInputSplitPoints(meta, range, direction, points);
- conf.setRangeSplit(points);
- }
-
- public List split(int num) throws Exception {
- LOG.info("Expect split num : " + num);
-
- List configurations = new ArrayList();
-
- List ranges = null;
-
- if (this.conf.getRangeSplit() != null) { // 用户显示指定了拆分范围
- LOG.info("Begin userDefinedRangeSplit");
- ranges = userDefinedRangeSplit(meta, range, this.conf.getRangeSplit());
- LOG.info("End userDefinedRangeSplit");
- } else { // 采用默认的切分算法
- LOG.info("Begin defaultRangeSplit");
- ranges = defaultRangeSplit(ots, meta, range, num);
- LOG.info("End defaultRangeSplit");
- }
-
- // 解决大量的Split Point序列化消耗内存的问题
- // 因为slave中不会使用这个配置，所以置为空
- this.conf.setRangeSplit(null);
-
- for (OTSRange item : ranges) {
- Configuration configuration = Configuration.newDefault();
- configuration.set(OTSConst.OTS_CONF, GsonParser.confToJson(this.conf));
- configuration.set(OTSConst.OTS_RANGE, GsonParser.rangeToJson(item));
- configuration.set(OTSConst.OTS_DIRECTION, GsonParser.directionToJson(direction));
- configurations.add(configuration);
- }
-
- LOG.info("Configuration list count : " + configurations.size());
-
- return configurations;
- }
public OTSConf getConf() {
return conf;
}
+ public TableMeta getMeta() {
+ return meta;
+ }
+
+ public SyncClientInterface getOts() {
+ return ots;
+ }
+
+ public void setOts(SyncClientInterface ots) {
+ this.ots = ots;
+ }
+
+ /**
+ * 基于配置传入的配置文件，解析为对应的参数
+ *
+ * @param param
+ * @throws Exception
+ */
+ public void init(Configuration param) throws Exception {
+ // 基于预定义的Json格式,检查传入参数是否符合Conf定义规范
+ conf = OTSConf.load(param);
+
+ // Init ots
+ ots = OtsHelper.getOTSInstance(conf);
+
+ // 宽行表init
+ if (!conf.isTimeseriesTable()) {
+ // 获取TableMeta
+ meta = OtsHelper.getTableMeta(
+ ots,
+ conf.getTableName(),
+ conf.getRetry(),
+ conf.getRetryPauseInMillisecond());
+
+ // 基于Meta检查Conf是否正确
+ ParamChecker.checkAndSetOTSConf(conf, meta);
+ direction = ParamChecker.checkDirectionAndEnd(meta, conf.getRange().getBegin(), conf.getRange().getEnd());
+ }
+ // 时序表 检查tablestore SDK version
+ if (conf.isTimeseriesTable()){
+ Common.checkTableStoreSDKVersion();
+ }
+
+
+ }
+
+ public List split(int mandatoryNumber) throws Exception {
+ LOG.info("Expect split num : " + mandatoryNumber);
+
+ List configurations = new ArrayList();
+
+ if (conf.isTimeseriesTable()) {{ // 时序表全部采用默认切分策略
+ LOG.info("Begin timeseries table defaultRangeSplit");
+ configurations = getTimeseriesConfigurationBySplit(mandatoryNumber);
+ LOG.info("End timeseries table defaultRangeSplit");
+ }}
+ else if (this.conf.getRange().getSplit().size() != 0) { // 用户显示指定了拆分范围
+ LOG.info("Begin userDefinedRangeSplit");
+ configurations = getNormalConfigurationBySplit();
+ LOG.info("End userDefinedRangeSplit");
+ } else { // 采用默认的切分算法
+ LOG.info("Begin defaultRangeSplit");
+ configurations = getDefaultConfiguration(mandatoryNumber);
+ LOG.info("End defaultRangeSplit");
+ }
+
+ LOG.info("Expect split num: "+ mandatoryNumber +", and final configuration list count : " + configurations.size());
+ return configurations;
+ }
+
public void close() {
ots.shutdown();
}
- // private function
-
- private TableMeta getTableMeta(OTSClient ots, String tableName) throws Exception {
- return RetryHelper.executeWithRetry(
- new GetTableMetaCallable(ots, tableName),
+ /**
+ * timeseries split信息，根据切分数配置多个Task
+ */
+ private List getTimeseriesConfigurationBySplit(int mandatoryNumber) throws Exception {
+ List timeseriesScanSplitInfoList = OtsHelper.splitTimeseriesScan(
+ ots,
+ conf.getTableName(),
+ conf.getMeasurementName(),
+ mandatoryNumber,
conf.getRetry(),
- conf.getSleepInMilliSecond()
- );
+ conf.getRetryPauseInMillisecond());
+ List configurations = new ArrayList<>();
+
+ for (int i = 0; i < timeseriesScanSplitInfoList.size(); i++) {
+ Configuration configuration = Configuration.newDefault();
+ configuration.set(Constant.ConfigKey.CONF, GsonParser.confToJson(conf));
+ configuration.set(Constant.ConfigKey.SPLIT_INFO, GsonParser.timeseriesScanSplitInfoToString(timeseriesScanSplitInfoList.get(i)));
+ configurations.add(configuration);
+ }
+ return configurations;
}
- private RowPrimaryKey getPKOfFirstRow(
- OTSRange range , Direction direction) throws Exception {
+ /**
+ * 根据用户配置的split信息，将配置文件基于Range范围转换为多个Task的配置
+ */
+ private List getNormalConfigurationBySplit() {
+ List> primaryKeys = new ArrayList>();
+ primaryKeys.add(conf.getRange().getBegin());
+ for (PrimaryKeyColumn column : conf.getRange().getSplit()) {
+ List point = new ArrayList();
+ point.add(column);
+ ParamChecker.fillPrimaryKey(this.meta.getPrimaryKeyList(), point, PrimaryKeyValue.INF_MIN);
+ primaryKeys.add(point);
+ }
+ primaryKeys.add(conf.getRange().getEnd());
- RangeRowQueryCriteria cur = new RangeRowQueryCriteria(this.conf.getTableName());
- cur.setInclusiveStartPrimaryKey(range.getBegin());
- cur.setExclusiveEndPrimaryKey(range.getEnd());
- cur.setLimit(1);
- cur.setColumnsToGet(Common.getPrimaryKeyNameList(meta));
- cur.setDirection(direction);
+ List configurations = new ArrayList(primaryKeys.size() - 1);
- return RetryHelper.executeWithRetry(
- new GetFirstRowPrimaryKeyCallable(ots, meta, cur),
- conf.getRetry(),
- conf.getSleepInMilliSecond()
- );
+ for (int i = 0; i < primaryKeys.size() - 1; i++) {
+ OTSRange range = new OTSRange();
+ range.setBegin(primaryKeys.get(i));
+ range.setEnd(primaryKeys.get(i + 1));
+
+ Configuration configuration = Configuration.newDefault();
+ configuration.set(Constant.ConfigKey.CONF, GsonParser.confToJson(conf));
+ configuration.set(Constant.ConfigKey.RANGE, GsonParser.rangeToJson(range));
+ configuration.set(Constant.ConfigKey.META, GsonParser.metaToJson(meta));
+ configurations.add(configuration);
+ }
+ return configurations;
}
- private List defaultRangeSplit(OTSClient ots, TableMeta meta, OTSRange range, int num) throws Exception {
+ private List getDefaultConfiguration(int num) throws Exception {
if (num == 1) {
List ranges = new ArrayList();
+ OTSRange range = new OTSRange();
+ range.setBegin(conf.getRange().getBegin());
+ range.setEnd(conf.getRange().getEnd());
ranges.add(range);
- return ranges;
+
+ return getConfigurationsFromRanges(ranges);
}
-
+
OTSRange reverseRange = new OTSRange();
- reverseRange.setBegin(range.getEnd());
- reverseRange.setEnd(range.getBegin());
+ reverseRange.setBegin(conf.getRange().getEnd());
+ reverseRange.setEnd(conf.getRange().getBegin());
Direction reverseDirection = (direction == Direction.FORWARD ? Direction.BACKWARD : Direction.FORWARD);
- RowPrimaryKey realBegin = getPKOfFirstRow(range, direction);
- RowPrimaryKey realEnd = getPKOfFirstRow(reverseRange, reverseDirection);
-
+ List realBegin = getPKOfFirstRow(conf.getRange(), direction);
+ List realEnd = getPKOfFirstRow(reverseRange, reverseDirection);
+
// 因为如果其中一行为空，表示这个范围内至多有一行数据
// 所以不再细分，直接使用用户定义的范围
if (realBegin == null || realEnd == null) {
List ranges = new ArrayList();
- ranges.add(range);
- return ranges;
+ ranges.add(conf.getRange());
+ return getConfigurationsFromRanges(ranges);
}
-
+
// 如果出现realBegin，realEnd的方向和direction不一致的情况，直接返回range
int cmp = Common.compareRangeBeginAndEnd(meta, realBegin, realEnd);
Direction realDirection = cmp > 0 ? Direction.BACKWARD : Direction.FORWARD;
if (realDirection != direction) {
LOG.warn("Expect '" + direction + "', but direction of realBegin and readlEnd is '" + realDirection + "'");
List ranges = new ArrayList();
- ranges.add(range);
- return ranges;
+ ranges.add(conf.getRange());
+ return getConfigurationsFromRanges(ranges);
}
List ranges = RangeSplit.rangeSplitByCount(meta, realBegin, realEnd, num);
if (ranges.isEmpty()) { // 当PartitionKey相等时，工具内部不会切分Range
- ranges.add(range);
+ ranges.add(conf.getRange());
} else {
// replace first and last
OTSRange first = ranges.get(0);
OTSRange last = ranges.get(ranges.size() - 1);
- first.setBegin(range.getBegin());
- last.setEnd(range.getEnd());
+ first.setBegin(conf.getRange().getBegin());
+ last.setEnd(conf.getRange().getEnd());
}
-
- return ranges;
+
+ return getConfigurationsFromRanges(ranges);
}
- private List userDefinedRangeSplit(TableMeta meta, OTSRange range, List points) {
- List ranges = RangeSplit.rangeSplitByPoint(meta, range.getBegin(), range.getEnd(), points);
- if (ranges.isEmpty()) { // 当PartitionKey相等时，工具内部不会切分Range
- ranges.add(range);
+ private List getConfigurationsFromRanges(List ranges){
+ List configurationList = new ArrayList<>();
+ for (OTSRange range:ranges
+ ) {
+ Configuration configuration = Configuration.newDefault();
+ configuration.set(Constant.ConfigKey.CONF, GsonParser.confToJson(conf));
+ configuration.set(Constant.ConfigKey.RANGE, GsonParser.rangeToJson(range));
+ configuration.set(Constant.ConfigKey.META, GsonParser.metaToJson(meta));
+ configurationList.add(configuration);
}
- return ranges;
+ return configurationList;
}
+
+ private List getPKOfFirstRow(
+ OTSRange range , Direction direction) throws Exception {
+
+ RangeRowQueryCriteria cur = new RangeRowQueryCriteria(this.conf.getTableName());
+ cur.setInclusiveStartPrimaryKey(new PrimaryKey(range.getBegin()));
+ cur.setExclusiveEndPrimaryKey(new PrimaryKey(range.getEnd()));
+ cur.setLimit(1);
+ cur.addColumnsToGet(Common.getPrimaryKeyNameList(meta));
+ cur.setDirection(direction);
+ cur.setMaxVersions(1);
+
+ return RetryHelper.executeWithRetry(
+ new GetFirstRowPrimaryKeyCallable(ots, meta, cur),
+ conf.getRetry(),
+ conf.getRetryPauseInMillisecond()
+ );
+ }
+
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveMetaProxy.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveMetaProxy.java
new file mode 100644
index 00000000..f9860194
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveMetaProxy.java
@@ -0,0 +1,160 @@
+package com.alibaba.datax.plugin.reader.otsreader;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
+import com.alibaba.datax.plugin.reader.otsreader.utils.Constant;
+import com.alibaba.datax.plugin.reader.otsreader.utils.Key;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.alibaba.datax.common.element.Record;
+import com.alibaba.datax.common.element.StringColumn;
+import com.alibaba.datax.common.plugin.RecordSender;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.reader.otsreader.utils.ParamCheckerOld;
+import com.alibaba.datax.plugin.reader.otsreader.utils.ReaderModelParser;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
+import com.alibaba.datax.plugin.reader.otsreader.utils.DefaultNoRetry;
+import com.alibaba.datax.plugin.reader.otsreader.utils.GsonParser;
+import com.alibaba.fastjson.JSON;
+import com.aliyun.openservices.ots.OTSClient;
+import com.aliyun.openservices.ots.OTSServiceConfiguration;
+import com.aliyun.openservices.ots.model.DescribeTableRequest;
+import com.aliyun.openservices.ots.model.DescribeTableResult;
+import com.aliyun.openservices.ots.model.ListTableResult;
+import com.aliyun.openservices.ots.model.PrimaryKeyType;
+import com.aliyun.openservices.ots.model.ReservedThroughputDetails;
+import com.aliyun.openservices.ots.model.TableMeta;
+
+public class OtsReaderSlaveMetaProxy implements IOtsReaderSlaveProxy {
+
+ private OTSClient ots = null;
+ private OTSConf conf = null;
+ private OTSRange range = null;
+ private com.alicloud.openservices.tablestore.model.TableMeta meta = null;
+ private Configuration configuration = null;
+ private static final Logger LOG = LoggerFactory.getLogger(OtsReaderSlaveMetaProxy.class);
+
+
+ @Override
+ public void init(Configuration configuration) {
+ OTSServiceConfiguration configure = new OTSServiceConfiguration();
+ configure.setRetryStrategy(new DefaultNoRetry());
+
+ this.configuration = configuration;
+ conf = GsonParser.jsonToConf((String) configuration.get(Constant.ConfigKey.CONF));
+ range = GsonParser.jsonToRange((String) configuration.get(Constant.ConfigKey.RANGE));
+ meta = GsonParser.jsonToMeta((String) configuration.get(Constant.ConfigKey.META));
+
+ String endpoint = conf.getEndpoint();
+ String accessId = conf.getAccessId();
+ String accessKey = conf.getAccessKey();
+ String instanceName = conf.getInstanceName();
+
+ ots = new OTSClient(endpoint, accessId, accessKey, instanceName, null, configure, null);
+ }
+
+ @Override
+ public void close() {
+ ots.shutdown();
+ }
+
+ @Override
+ public void startRead(RecordSender recordSender) throws Exception {
+ List columns = ReaderModelParser
+ .parseOTSColumnList(ParamCheckerOld.checkListAndGet(configuration, Key.COLUMN, true));
+ String metaMode = conf.getMetaMode(); // column
+
+
+ ListTableResult listTableResult = null;
+ try {
+ listTableResult = ots.listTable();
+ LOG.info(String.format("ots listTable requestId:%s, traceId:%s", listTableResult.getRequestID(),
+ listTableResult.getTraceId()));
+ List allTables = listTableResult.getTableNames();
+ for (String eachTable : allTables) {
+ DescribeTableRequest describeTableRequest = new DescribeTableRequest();
+ describeTableRequest.setTableName(eachTable);
+ DescribeTableResult describeTableResult = ots.describeTable(describeTableRequest);
+ LOG.info(String.format("ots describeTable requestId:%s, traceId:%s", describeTableResult.getRequestID(),
+ describeTableResult.getTraceId()));
+
+ TableMeta tableMeta = describeTableResult.getTableMeta();
+ // table_name: first_table
+ // table primary key: type, data type: STRING
+ // table primary key: db_name, data type: STRING
+ // table primary key: table_name, data type: STRING
+ // Reserved throughput: read(0), write(0)
+ // last increase time: 1502881295
+ // last decrease time: None
+ // number of decreases today: 0
+
+ String tableName = tableMeta.getTableName();
+ Map primaryKey = tableMeta.getPrimaryKey();
+ ReservedThroughputDetails reservedThroughputDetails = describeTableResult
+ .getReservedThroughputDetails();
+ int reservedThroughputRead = reservedThroughputDetails.getCapacityUnit().getReadCapacityUnit();
+ int reservedThroughputWrite = reservedThroughputDetails.getCapacityUnit().getWriteCapacityUnit();
+ long lastIncreaseTime = reservedThroughputDetails.getLastIncreaseTime();
+ long lastDecreaseTime = reservedThroughputDetails.getLastDecreaseTime();
+ int numberOfDecreasesToday = reservedThroughputDetails.getNumberOfDecreasesToday();
+
+ Map allData = new HashMap();
+ allData.put("endpoint", conf.getEndpoint());
+ allData.put("instanceName", conf.getInstanceName());
+ allData.put("table", tableName);
+ // allData.put("primaryKey", JSON.toJSONString(primaryKey));
+ allData.put("reservedThroughputRead", reservedThroughputRead + "");
+ allData.put("reservedThroughputWrite", reservedThroughputWrite + "");
+ allData.put("lastIncreaseTime", lastIncreaseTime + "");
+ allData.put("lastDecreaseTime", lastDecreaseTime + "");
+ allData.put("numberOfDecreasesToday", numberOfDecreasesToday + "");
+
+ // 可扩展的可配置的形式
+ if ("column".equalsIgnoreCase(metaMode)) {
+ // 如果是列元数据模式并且column中配置的name是primaryKey，映射成多行DataX Record
+ List primaryKeyRecords = new ArrayList();
+ for (Entry eachPk : primaryKey.entrySet()) {
+ Record line = recordSender.createRecord();
+ for (OTSColumn col : columns) {
+ if (col.getColumnType() == OTSColumn.OTSColumnType.CONST) {
+ line.addColumn(col.getValue());
+ } else if ("primaryKey.name".equalsIgnoreCase(col.getName())) {
+ line.addColumn(new StringColumn(eachPk.getKey()));
+ } else if ("primaryKey.type".equalsIgnoreCase(col.getName())) {
+ line.addColumn(new StringColumn(eachPk.getValue().name()));
+ } else {
+ String v = allData.get(col.getName());
+ line.addColumn(new StringColumn(v));
+ }
+ }
+ LOG.debug("Reader send record : {}", line.toString());
+ recordSender.sendToWriter(line);
+ primaryKeyRecords.add(line);
+ }
+ } else {
+ Record line = recordSender.createRecord();
+ for (OTSColumn col : columns) {
+ if (col.getColumnType() == OTSColumn.OTSColumnType.CONST) {
+ line.addColumn(col.getValue());
+ } else {
+ String v = allData.get(col.getName());
+ line.addColumn(new StringColumn(v));
+ }
+ }
+ LOG.debug("Reader send record : {}", line.toString());
+ recordSender.sendToWriter(line);
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn(JSON.toJSONString(listTableResult), e);
+ }
+
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveMultiVersionProxy.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveMultiVersionProxy.java
new file mode 100644
index 00000000..818a507e
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveMultiVersionProxy.java
@@ -0,0 +1,102 @@
+package com.alibaba.datax.plugin.reader.otsreader;
+
+import com.alibaba.datax.common.element.LongColumn;
+import com.alibaba.datax.common.element.Record;
+import com.alibaba.datax.common.element.StringColumn;
+import com.alibaba.datax.common.plugin.RecordSender;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
+import com.alibaba.datax.plugin.reader.otsreader.utils.*;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.model.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class OtsReaderSlaveMultiVersionProxy implements IOtsReaderSlaveProxy {
+ private OTSConf conf = null;
+ private OTSRange range = null;
+ private TableMeta meta = null;
+ private SyncClientInterface ots = null;
+
+ private static final Logger LOG = LoggerFactory.getLogger(OtsReaderSlaveMultiVersionProxy.class);
+
+ @Override
+ public void init(Configuration configuration) {
+ conf = GsonParser.jsonToConf((String) configuration.get(Constant.ConfigKey.CONF));
+ range = GsonParser.jsonToRange((String) configuration.get(Constant.ConfigKey.RANGE));
+ meta = GsonParser.jsonToMeta((String) configuration.get(Constant.ConfigKey.META));
+
+ this.ots = OtsHelper.getOTSInstance(conf);
+ }
+
+ @Override
+ public void close() {
+ ots.shutdown();
+ }
+
+ private void sendToDatax(RecordSender recordSender, PrimaryKey pk, Column c) {
+ Record line = recordSender.createRecord();
+ //-------------------------
+ // 四元组 pk, column name, timestamp, value
+ //-------------------------
+
+ // pk
+ for( PrimaryKeyColumn pkc : pk.getPrimaryKeyColumns()) {
+ line.addColumn(TranformHelper.otsPrimaryKeyColumnToDataxColumn(pkc));
+ }
+ // column name
+ line.addColumn(new StringColumn(c.getName()));
+ // Timestamp
+ line.addColumn(new LongColumn(c.getTimestamp()));
+ // Value
+ line.addColumn(TranformHelper.otsColumnToDataxColumn(c));
+
+ recordSender.sendToWriter(line);
+ }
+
+ private void sendToDatax(RecordSender recordSender, Row row) {
+ PrimaryKey pk = row.getPrimaryKey();
+ for (Column c : row.getColumns()) {
+ sendToDatax(recordSender, pk, c);
+ }
+ }
+
+ /**
+ * 将获取到的数据采用4元组的方式传递给datax
+ * @param recordSender
+ * @param result
+ */
+ private void sendToDatax(RecordSender recordSender, GetRangeResponse result) {
+ LOG.debug("Per request get row count : " + result.getRows().size());
+ for (Row row : result.getRows()) {
+ sendToDatax(recordSender, row);
+ }
+ }
+
+ @Override
+ public void startRead(RecordSender recordSender) throws Exception {
+
+ PrimaryKey inclusiveStartPrimaryKey = new PrimaryKey(range.getBegin());
+ PrimaryKey exclusiveEndPrimaryKey = new PrimaryKey(range.getEnd());
+ PrimaryKey next = inclusiveStartPrimaryKey;
+
+ RangeRowQueryCriteria rangeRowQueryCriteria = new RangeRowQueryCriteria(conf.getTableName());
+ rangeRowQueryCriteria.setExclusiveEndPrimaryKey(exclusiveEndPrimaryKey);
+ rangeRowQueryCriteria.setDirection(Common.getDirection(range.getBegin(), range.getEnd()));
+ rangeRowQueryCriteria.setTimeRange(conf.getMulti().getTimeRange());
+ rangeRowQueryCriteria.setMaxVersions(conf.getMulti().getMaxVersion());
+ rangeRowQueryCriteria.addColumnsToGet(Common.toColumnToGet(conf.getColumn(), meta));
+
+ do{
+ rangeRowQueryCriteria.setInclusiveStartPrimaryKey(next);
+ GetRangeResponse result = OtsHelper.getRange(
+ ots,
+ rangeRowQueryCriteria,
+ conf.getRetry(),
+ conf.getRetryPauseInMillisecond());
+ sendToDatax(recordSender, result);
+ next = result.getNextStartPrimaryKey();
+ } while(next != null);
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveNormalProxy.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveNormalProxy.java
new file mode 100644
index 00000000..f7d89b15
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveNormalProxy.java
@@ -0,0 +1,256 @@
+package com.alibaba.datax.plugin.reader.otsreader;
+
+import com.alibaba.datax.common.element.LongColumn;
+import com.alibaba.datax.common.element.Record;
+import com.alibaba.datax.common.element.StringColumn;
+import com.alibaba.datax.common.exception.DataXException;
+import com.alibaba.datax.common.plugin.RecordSender;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSCriticalException;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
+import com.alibaba.datax.plugin.reader.otsreader.utils.*;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.core.utils.Pair;
+import com.alicloud.openservices.tablestore.model.*;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataRequest;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataResponse;
+import com.alicloud.openservices.tablestore.model.timeseries.TimeseriesRow;
+import com.alicloud.openservices.tablestore.model.timeseries.TimeseriesScanSplitInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+public class OtsReaderSlaveNormalProxy implements IOtsReaderSlaveProxy {
+ private static final Logger LOG = LoggerFactory.getLogger(OtsReaderSlaveNormalProxy.class);
+ private OTSConf conf = null;
+ private OTSRange range = null;
+ private TableMeta meta = null;
+ private SyncClientInterface ots = null;
+ private TimeseriesScanSplitInfo splitInfo = null;
+
+ @Override
+ public void init(Configuration configuration) {
+ conf = GsonParser.jsonToConf((String) configuration.get(Constant.ConfigKey.CONF));
+ if (!conf.isTimeseriesTable()) {
+ range = GsonParser.jsonToRange((String) configuration.get(Constant.ConfigKey.RANGE));
+ meta = GsonParser.jsonToMeta((String) configuration.get(Constant.ConfigKey.META));
+ } else {
+ splitInfo = GsonParser.stringToTimeseriesScanSplitInfo((String) configuration.get(Constant.ConfigKey.SPLIT_INFO));
+ // 时序表 检查tablestore SDK version
+ try{
+ Common.checkTableStoreSDKVersion();
+ }
+ catch (Exception e){
+ LOG.error("Exception. ErrorMsg:{}", e.getMessage(), e);
+ throw DataXException.asDataXException(OtsReaderError.ERROR, e.toString(), e);
+ }
+ }
+
+
+ this.ots = OtsHelper.getOTSInstance(conf);
+ }
+
+ @Override
+ public void close() {
+ ots.shutdown();
+ }
+
+ private void sendToDatax(RecordSender recordSender, Row row) {
+ Record line = recordSender.createRecord();
+
+ PrimaryKey pk = row.getPrimaryKey();
+ for (OTSColumn column : conf.getColumn()) {
+ if (column.getColumnType() == OTSColumn.OTSColumnType.NORMAL) {
+ // 获取指定的列
+ PrimaryKeyColumn value = pk.getPrimaryKeyColumn(column.getName());
+ if (value != null) {
+ line.addColumn(TranformHelper.otsPrimaryKeyColumnToDataxColumn(value));
+ } else {
+ Column c = row.getLatestColumn(column.getName());
+ if (c != null) {
+ line.addColumn(TranformHelper.otsColumnToDataxColumn(c));
+ } else {
+ // 这里使用StringColumn的无参构造函数构造对象，而不是用null，下
+ // 游（writer）应该通过获取Column，然后通过Column的数据接口的返回值
+ // 是否是null来判断改Column是否为null
+ // Datax其他插件的也是使用这种方式，约定俗成，并没有使用直接向record中注入null方式代表空
+ line.addColumn(new StringColumn());
+ }
+ }
+ } else {
+ line.addColumn(column.getValue());
+ }
+ }
+ recordSender.sendToWriter(line);
+ }
+
+ private void sendToDatax(RecordSender recordSender, TimeseriesRow row) {
+
+
+ Record line = recordSender.createRecord();
+ // 对于配置项中的每一列
+ for (int i = 0; i < conf.getColumn().size(); i++) {
+ OTSColumn column = conf.getColumn().get(i);
+ // 如果不是常数列
+ if (column.getColumnType() == OTSColumn.OTSColumnType.NORMAL) {
+ // 如果是tags内字段
+ if (conf.getColumn().get(i).getTimeseriesTag()) {
+ String s = row.getTimeseriesKey().getTags().get(column.getName());
+ line.addColumn(new StringColumn(s));
+ }
+ // 如果为measurement字段
+ else if (column.getName().equals(Constant.ConfigKey.TimeseriesPKColumn.MEASUREMENT_NAME)) {
+ String s = row.getTimeseriesKey().getMeasurementName();
+ line.addColumn(new StringColumn(s));
+ }
+ // 如果为dataSource字段
+ else if (column.getName().equals(Constant.ConfigKey.TimeseriesPKColumn.DATA_SOURCE)) {
+ String s = row.getTimeseriesKey().getDataSource();
+ line.addColumn(new StringColumn(s));
+ }
+ // 如果为tags字段
+ else if (column.getName().equals(Constant.ConfigKey.TimeseriesPKColumn.TAGS)) {
+ line.addColumn(new StringColumn(row.getTimeseriesKey().buildTagsString()));
+ }
+ else if (column.getName().equals(Constant.ConfigKey.TimeseriesPKColumn.TIME)) {
+ Long l = row.getTimeInUs();
+ line.addColumn(new LongColumn(l));
+ }
+ // 否则为field内字段
+ else {
+ ColumnValue c = row.getFields().get(column.getName());
+ if (c == null) {
+ LOG.warn("Get column {} : type {} failed, use empty string instead", column.getName(), conf.getColumn().get(i).getValueType());
+ line.addColumn(new StringColumn());
+ } else if (c.getType() != conf.getColumn().get(i).getValueType()) {
+ LOG.warn("Get column {} failed, expected type: {}, actual type: {}. Sending actual type to writer.", column.getName(), conf.getColumn().get(i).getValueType(), c.getType());
+ line.addColumn(TranformHelper.otsColumnToDataxColumn(c));
+ } else {
+ line.addColumn(TranformHelper.otsColumnToDataxColumn(c));
+ }
+ }
+ }
+ // 如果是常数列
+ else {
+ line.addColumn(column.getValue());
+ }
+ }
+ recordSender.sendToWriter(line);
+ }
+
+ /**
+ * 将获取到的数据根据用户配置Column的方式传递给datax
+ *
+ * @param recordSender
+ * @param result
+ */
+ private void sendToDatax(RecordSender recordSender, GetRangeResponse result) {
+ for (Row row : result.getRows()) {
+ sendToDatax(recordSender, row);
+ }
+ }
+
+ private void sendToDatax(RecordSender recordSender, ScanTimeseriesDataResponse result) {
+ for (TimeseriesRow row : result.getRows()) {
+ sendToDatax(recordSender, row);
+ }
+ }
+
+ @Override
+ public void startRead(RecordSender recordSender) throws Exception {
+ if (conf.isTimeseriesTable()) {
+ readTimeseriesTable(recordSender);
+ } else {
+ readNormalTable(recordSender);
+ }
+ }
+
+ public void readTimeseriesTable(RecordSender recordSender) throws Exception {
+
+ List timeseriesPkName = new ArrayList<>();
+ timeseriesPkName.add(Constant.ConfigKey.TimeseriesPKColumn.MEASUREMENT_NAME);
+ timeseriesPkName.add(Constant.ConfigKey.TimeseriesPKColumn.DATA_SOURCE);
+ timeseriesPkName.add(Constant.ConfigKey.TimeseriesPKColumn.TAGS);
+ timeseriesPkName.add(Constant.ConfigKey.TimeseriesPKColumn.TIME);
+
+ ScanTimeseriesDataRequest scanTimeseriesDataRequest = new ScanTimeseriesDataRequest(conf.getTableName());
+ List> fieldsToGet = new ArrayList<>();
+ for (int i = 0; i < conf.getColumn().size(); i++) {
+ /**
+ * 如果所配置列
+ * 1. 不是常量列（即列名不为null）
+ * 2. 列名不在["measurementName","dataSource","tags"]中
+ * 3. 不是tags内的字段
+ * 则为需要获取的field字段。
+ */
+ String fieldName = conf.getColumn().get(i).getName();
+ if (fieldName != null && !timeseriesPkName.contains(fieldName) && !conf.getColumn().get(i).getTimeseriesTag()) {
+ Pair pair = new Pair<>(fieldName, conf.getColumn().get(i).getValueType());
+ fieldsToGet.add(pair);
+ }
+ }
+ scanTimeseriesDataRequest.setFieldsToGet(fieldsToGet);
+ scanTimeseriesDataRequest.setSplitInfo(splitInfo);
+
+ while (true) {
+ ScanTimeseriesDataResponse response = OtsHelper.scanTimeseriesData(
+ ots,
+ scanTimeseriesDataRequest,
+ conf.getRetry(),
+ conf.getRetryPauseInMillisecond());
+ sendToDatax(recordSender, response);
+ if (response.getNextToken() == null) {
+ break;
+ }
+ scanTimeseriesDataRequest.setNextToken(response.getNextToken());
+ }
+ }
+
+ public void readNormalTable(RecordSender recordSender) throws Exception {
+ PrimaryKey inclusiveStartPrimaryKey = new PrimaryKey(range.getBegin());
+ PrimaryKey exclusiveEndPrimaryKey = new PrimaryKey(range.getEnd());
+ PrimaryKey next = inclusiveStartPrimaryKey;
+
+ RangeRowQueryCriteria rangeRowQueryCriteria = new RangeRowQueryCriteria(conf.getTableName());
+ rangeRowQueryCriteria.setExclusiveEndPrimaryKey(exclusiveEndPrimaryKey);
+ rangeRowQueryCriteria.setDirection(Common.getDirection(range.getBegin(), range.getEnd()));
+ rangeRowQueryCriteria.setMaxVersions(1);
+ rangeRowQueryCriteria.addColumnsToGet(Common.toColumnToGet(conf.getColumn(), meta));
+
+ do {
+ rangeRowQueryCriteria.setInclusiveStartPrimaryKey(next);
+ GetRangeResponse result = OtsHelper.getRange(
+ ots,
+ rangeRowQueryCriteria,
+ conf.getRetry(),
+ conf.getRetryPauseInMillisecond());
+ sendToDatax(recordSender, result);
+ next = result.getNextStartPrimaryKey();
+ } while (next != null);
+ }
+
+
+ public void setConf(OTSConf conf) {
+ this.conf = conf;
+ }
+
+
+ public void setRange(OTSRange range) {
+ this.range = range;
+ }
+
+
+ public void setMeta(TableMeta meta) {
+ this.meta = meta;
+ }
+
+
+ public void setOts(SyncClientInterface ots) {
+ this.ots = ots;
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveProxy.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveProxy.java
deleted file mode 100644
index e64b4e7e..00000000
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveProxy.java
+++ /dev/null
@@ -1,135 +0,0 @@
-package com.alibaba.datax.plugin.reader.otsreader;
-
-import java.util.List;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.alibaba.datax.common.element.Record;
-import com.alibaba.datax.common.plugin.RecordSender;
-import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.datax.plugin.reader.otsreader.callable.GetRangeCallable;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSConst;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
-import com.alibaba.datax.plugin.reader.otsreader.utils.Common;
-import com.alibaba.datax.plugin.reader.otsreader.utils.GsonParser;
-import com.alibaba.datax.plugin.reader.otsreader.utils.DefaultNoRetry;
-import com.alibaba.datax.plugin.reader.otsreader.utils.RetryHelper;
-import com.aliyun.openservices.ots.OTSClientAsync;
-import com.aliyun.openservices.ots.OTSServiceConfiguration;
-import com.aliyun.openservices.ots.model.Direction;
-import com.aliyun.openservices.ots.model.GetRangeRequest;
-import com.aliyun.openservices.ots.model.GetRangeResult;
-import com.aliyun.openservices.ots.model.OTSFuture;
-import com.aliyun.openservices.ots.model.RangeRowQueryCriteria;
-import com.aliyun.openservices.ots.model.Row;
-import com.aliyun.openservices.ots.model.RowPrimaryKey;
-
-public class OtsReaderSlaveProxy {
-
- class RequestItem {
- private RangeRowQueryCriteria criteria;
- private OTSFuture future;
-
- RequestItem(RangeRowQueryCriteria criteria, OTSFuture future) {
- this.criteria = criteria;
- this.future = future;
- }
-
- public RangeRowQueryCriteria getCriteria() {
- return criteria;
- }
-
- public OTSFuture getFuture() {
- return future;
- }
- }
-
- private static final Logger LOG = LoggerFactory.getLogger(OtsReaderSlaveProxy.class);
-
- private void rowsToSender(List rows, RecordSender sender, List columns) {
- for (Row row : rows) {
- Record line = sender.createRecord();
- line = Common.parseRowToLine(row, columns, line);
-
- LOG.debug("Reader send record : {}", line.toString());
-
- sender.sendToWriter(line);
- }
- }
-
- private RangeRowQueryCriteria generateRangeRowQueryCriteria(String tableName, RowPrimaryKey begin, RowPrimaryKey end, Direction direction, List columns) {
- RangeRowQueryCriteria criteria = new RangeRowQueryCriteria(tableName);
- criteria.setInclusiveStartPrimaryKey(begin);
- criteria.setDirection(direction);
- criteria.setColumnsToGet(columns);
- criteria.setLimit(-1);
- criteria.setExclusiveEndPrimaryKey(end);
- return criteria;
- }
-
- private RequestItem generateRequestItem(
- OTSClientAsync ots,
- OTSConf conf,
- RowPrimaryKey begin,
- RowPrimaryKey end,
- Direction direction,
- List columns) throws Exception {
- RangeRowQueryCriteria criteria = generateRangeRowQueryCriteria(conf.getTableName(), begin, end, direction, columns);
-
- GetRangeRequest request = new GetRangeRequest();
- request.setRangeRowQueryCriteria(criteria);
- OTSFuture future = ots.getRange(request);
-
- return new RequestItem(criteria, future);
- }
-
- public void read(RecordSender sender, Configuration configuration) throws Exception {
- LOG.info("read begin.");
-
- OTSConf conf = GsonParser.jsonToConf(configuration.getString(OTSConst.OTS_CONF));
- OTSRange range = GsonParser.jsonToRange(configuration.getString(OTSConst.OTS_RANGE));
- Direction direction = GsonParser.jsonToDirection(configuration.getString(OTSConst.OTS_DIRECTION));
-
- OTSServiceConfiguration configure = new OTSServiceConfiguration();
- configure.setRetryStrategy(new DefaultNoRetry());
-
- OTSClientAsync ots = new OTSClientAsync(
- conf.getEndpoint(),
- conf.getAccessId(),
- conf.getAccesskey(),
- conf.getInstanceName(),
- null,
- configure,
- null);
-
- RowPrimaryKey token = range.getBegin();
- List columns = Common.getNormalColumnNameList(conf.getColumns());
-
- RequestItem request = null;
-
- do {
- LOG.debug("Next token : {}", GsonParser.rowPrimaryKeyToJson(token));
- if (request == null) {
- request = generateRequestItem(ots, conf, token, range.getEnd(), direction, columns);
- } else {
- RequestItem req = request;
-
- GetRangeResult result = RetryHelper.executeWithRetry(
- new GetRangeCallable(ots, req.getCriteria(), req.getFuture()),
- conf.getRetry(),
- conf.getSleepInMilliSecond()
- );
- if ((token = result.getNextStartPrimaryKey()) != null) {
- request = generateRequestItem(ots, conf, token, range.getEnd(), direction, columns);
- }
-
- rowsToSender(result.getRows(), sender, conf.getColumns());
- }
- } while (token != null);
- ots.shutdown();
- LOG.info("read end.");
- }
-}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveProxyOld.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveProxyOld.java
new file mode 100644
index 00000000..72eb885e
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderSlaveProxyOld.java
@@ -0,0 +1,181 @@
+package com.alibaba.datax.plugin.reader.otsreader;
+
+import java.util.List;
+
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
+import com.alibaba.datax.plugin.reader.otsreader.utils.*;
+import com.alicloud.openservices.tablestore.model.PrimaryKeyColumn;
+import com.aliyun.openservices.ots.model.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.alibaba.datax.common.element.Record;
+import com.alibaba.datax.common.plugin.RecordSender;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.reader.otsreader.callable.GetRangeCallableOld;
+import com.aliyun.openservices.ots.OTSClientAsync;
+import com.aliyun.openservices.ots.OTSServiceConfiguration;
+
+public class OtsReaderSlaveProxyOld implements IOtsReaderSlaveProxy {
+
+
+ private OTSClientAsync ots = null;
+ private OTSConf conf = null;
+ private OTSRange range = null;
+
+ class RequestItem {
+ private RangeRowQueryCriteria criteria;
+ private OTSFuture future;
+
+ RequestItem(RangeRowQueryCriteria criteria, OTSFuture future) {
+ this.criteria = criteria;
+ this.future = future;
+ }
+
+ public RangeRowQueryCriteria getCriteria() {
+ return criteria;
+ }
+
+ public OTSFuture getFuture() {
+ return future;
+ }
+ }
+
+ private static final Logger LOG = LoggerFactory.getLogger(OtsReaderSlaveProxyOld.class);
+
+ private void rowsToSender(List rows, RecordSender sender, List columns) {
+ for (Row row : rows) {
+ Record line = sender.createRecord();
+ line = CommonOld.parseRowToLine(row, columns, line);
+
+ LOG.debug("Reader send record : {}", line.toString());
+
+ sender.sendToWriter(line);
+ }
+ }
+
+ private RangeRowQueryCriteria generateRangeRowQueryCriteria(String tableName, RowPrimaryKey begin, RowPrimaryKey end, Direction direction, List columns) {
+ RangeRowQueryCriteria criteria = new RangeRowQueryCriteria(tableName);
+ criteria.setInclusiveStartPrimaryKey(begin);
+ criteria.setDirection(direction);
+ criteria.setColumnsToGet(columns);
+ criteria.setLimit(-1);
+ criteria.setExclusiveEndPrimaryKey(end);
+ return criteria;
+ }
+
+ private RequestItem generateRequestItem(
+ OTSClientAsync ots,
+ OTSConf conf,
+ RowPrimaryKey begin,
+ RowPrimaryKey end,
+ Direction direction,
+ List columns) throws Exception {
+ RangeRowQueryCriteria criteria = generateRangeRowQueryCriteria(conf.getTableName(), begin, end, direction, columns);
+
+ GetRangeRequest request = new GetRangeRequest();
+ request.setRangeRowQueryCriteria(criteria);
+ OTSFuture future = ots.getRange(request);
+
+ return new RequestItem(criteria, future);
+ }
+
+ @Override
+ public void init(Configuration configuration) {
+ conf = GsonParser.jsonToConf(configuration.getString(Constant.ConfigKey.CONF));
+ range = GsonParser.jsonToRange(configuration.getString(Constant.ConfigKey.RANGE));
+
+ OTSServiceConfiguration configure = new OTSServiceConfiguration();
+ configure.setRetryStrategy(new DefaultNoRetry());
+
+ ots = new OTSClientAsync(
+ conf.getEndpoint(),
+ conf.getAccessId(),
+ conf.getAccessKey(),
+ conf.getInstanceName(),
+ null,
+ configure,
+ null);
+ }
+
+ @Override
+ public void close() {
+ ots.shutdown();
+ }
+
+ @Override
+ public void startRead(RecordSender recordSender) throws Exception {
+ RowPrimaryKey token = pKColumnList2RowPrimaryKey(range.getBegin());
+
+ List columns = CommonOld.getNormalColumnNameList(conf.getColumn());
+ Direction direction = null;
+ switch (Common.getDirection(range.getBegin(), range.getEnd())){
+ case FORWARD:
+ direction = Direction.FORWARD;
+ break;
+ case BACKWARD:
+ default:
+ direction = Direction.BACKWARD;
+ }
+ RequestItem request = null;
+
+ do {
+ LOG.debug("Next token : {}", GsonParser.rowPrimaryKeyToJson(token));
+ if (request == null) {
+ request = generateRequestItem(ots, conf, token, pKColumnList2RowPrimaryKey(range.getEnd()), direction, columns);
+ } else {
+ RequestItem req = request;
+
+ GetRangeResult result = RetryHelperOld.executeWithRetry(
+ new GetRangeCallableOld(ots, req.getCriteria(), req.getFuture()),
+ conf.getRetry(),
+ // TODO
+ 100
+ );
+ if ((token = result.getNextStartPrimaryKey()) != null) {
+ request = generateRequestItem(ots, conf, token, pKColumnList2RowPrimaryKey(range.getEnd()), direction, columns);
+ }
+
+ rowsToSender(result.getRows(), recordSender, conf.getColumn());
+ }
+ } while (token != null);
+ }
+
+ /**
+ * 将 {@link com.alicloud.openservices.tablestore.model.PrimaryKeyColumn}的列表转为{@link com.aliyun.openservices.ots.model.RowPrimaryKey}
+ * @param list
+ * @return
+ */
+ public RowPrimaryKey pKColumnList2RowPrimaryKey(List list){
+ RowPrimaryKey rowPrimaryKey = new RowPrimaryKey();
+ for(PrimaryKeyColumn pk : list){
+ PrimaryKeyValue v = null;
+ if(pk.getValue() == com.alicloud.openservices.tablestore.model.PrimaryKeyValue.INF_MAX){
+ v = PrimaryKeyValue.INF_MAX;
+ } else if (pk.getValue() == com.alicloud.openservices.tablestore.model.PrimaryKeyValue.INF_MIN) {
+ v = PrimaryKeyValue.INF_MIN;
+ }
+ // 非INF_MAX 或 INF_MIN
+ else{
+ switch (pk.getValue().getType()){
+ case STRING:
+ v = PrimaryKeyValue.fromString(pk.getValue().asString());
+ break;
+ case INTEGER:
+ v = PrimaryKeyValue.fromLong(pk.getValue().asLong());
+ break;
+ case BINARY:
+ v = PrimaryKeyValue.fromBinary(pk.getValue().asBinary());
+ break;
+ default:
+ throw new IllegalArgumentException("the pKColumnList to RowPrimaryKey conversion failed");
+ }
+ }
+
+ rowPrimaryKey.addPrimaryKeyColumn(pk.getName(),v);
+ }
+ return rowPrimaryKey;
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/ColumnAdaptor.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/ColumnAdaptor.java
new file mode 100644
index 00000000..b2e14b5c
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/ColumnAdaptor.java
@@ -0,0 +1,63 @@
+package com.alibaba.datax.plugin.reader.otsreader.adaptor;
+
+import com.alibaba.datax.common.element.*;
+import com.google.gson.*;
+import org.apache.commons.codec.binary.Base64;
+
+import java.lang.reflect.Type;
+
+public class ColumnAdaptor implements JsonDeserializer, JsonSerializer{
+ private final static String TYPE = "type";
+ private final static String RAW = "rawData";
+
+ @Override
+ public JsonElement serialize(Column obj, Type t,
+ JsonSerializationContext c) {
+ JsonObject json = new JsonObject();
+
+ String rawData = null;
+ switch (obj.getType()){
+ case BOOL:
+ rawData = String.valueOf(obj.getRawData()); break;
+ case BYTES:
+ rawData = Base64.encodeBase64String((byte[]) obj.getRawData()); break;
+ case DOUBLE:
+ rawData = String.valueOf(obj.getRawData());break;
+ case LONG:
+ rawData = String.valueOf(obj.getRawData());break;
+ case STRING:
+ rawData = String.valueOf(obj.getRawData());break;
+ default:
+ throw new IllegalArgumentException("Unsupport parse the column type:" + obj.getType().toString());
+
+ }
+ json.add(TYPE, new JsonPrimitive(obj.getType().toString()));
+ json.add(RAW, new JsonPrimitive(rawData));
+ return json;
+ }
+
+ @Override
+ public Column deserialize(JsonElement ele, Type t,
+ JsonDeserializationContext c) throws JsonParseException {
+ JsonObject obj = ele.getAsJsonObject();
+
+ String strType = obj.getAsJsonPrimitive(TYPE).getAsString();
+ String strRaw = obj.getAsJsonPrimitive(RAW).getAsString();
+ Column.Type type = Column.Type.valueOf(strType);
+ switch (type){
+ case BOOL:
+ return new BoolColumn(strRaw);
+ case BYTES:
+ return new BytesColumn(Base64.decodeBase64(strRaw));
+ case DOUBLE:
+ return new DoubleColumn(strRaw);
+ case LONG:
+ return new LongColumn(strRaw);
+ case STRING:
+ return new StringColumn(strRaw);
+ default:
+ throw new IllegalArgumentException("Unsupport parse the column type:" + type.toString());
+
+ }
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/OTSColumnAdaptor.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/OTSColumnAdaptor.java
deleted file mode 100644
index 25f9b682..00000000
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/OTSColumnAdaptor.java
+++ /dev/null
@@ -1,117 +0,0 @@
-package com.alibaba.datax.plugin.reader.otsreader.adaptor;
-
-import java.lang.reflect.Type;
-
-import org.apache.commons.codec.binary.Base64;
-
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
-import com.aliyun.openservices.ots.model.ColumnType;
-import com.google.gson.JsonDeserializationContext;
-import com.google.gson.JsonDeserializer;
-import com.google.gson.JsonElement;
-import com.google.gson.JsonObject;
-import com.google.gson.JsonParseException;
-import com.google.gson.JsonPrimitive;
-import com.google.gson.JsonSerializationContext;
-import com.google.gson.JsonSerializer;
-
-public class OTSColumnAdaptor implements JsonDeserializer, JsonSerializer{
- private final static String NAME = "name";
- private final static String COLUMN_TYPE = "column_type";
- private final static String VALUE_TYPE = "value_type";
- private final static String VALUE = "value";
-
- private void serializeConstColumn(JsonObject json, OTSColumn obj) {
- switch (obj.getValueType()) {
- case STRING :
- json.add(VALUE_TYPE, new JsonPrimitive(ColumnType.STRING.toString()));
- json.add(VALUE, new JsonPrimitive(obj.getValue().asString()));
- break;
- case INTEGER :
- json.add(VALUE_TYPE, new JsonPrimitive(ColumnType.INTEGER.toString()));
- json.add(VALUE, new JsonPrimitive(obj.getValue().asLong()));
- break;
- case DOUBLE :
- json.add(VALUE_TYPE, new JsonPrimitive(ColumnType.DOUBLE.toString()));
- json.add(VALUE, new JsonPrimitive(obj.getValue().asDouble()));
- break;
- case BOOLEAN :
- json.add(VALUE_TYPE, new JsonPrimitive(ColumnType.BOOLEAN.toString()));
- json.add(VALUE, new JsonPrimitive(obj.getValue().asBoolean()));
- break;
- case BINARY :
- json.add(VALUE_TYPE, new JsonPrimitive(ColumnType.BINARY.toString()));
- json.add(VALUE, new JsonPrimitive(Base64.encodeBase64String(obj.getValue().asBytes())));
- break;
- default:
- throw new IllegalArgumentException("Unsupport serialize the type : " + obj.getValueType() + "");
- }
- }
-
- private OTSColumn deserializeConstColumn(JsonObject obj) {
- String strType = obj.getAsJsonPrimitive(VALUE_TYPE).getAsString();
- ColumnType type = ColumnType.valueOf(strType);
-
- JsonPrimitive jsonValue = obj.getAsJsonPrimitive(VALUE);
-
- switch (type) {
- case STRING :
- return OTSColumn.fromConstStringColumn(jsonValue.getAsString());
- case INTEGER :
- return OTSColumn.fromConstIntegerColumn(jsonValue.getAsLong());
- case DOUBLE :
- return OTSColumn.fromConstDoubleColumn(jsonValue.getAsDouble());
- case BOOLEAN :
- return OTSColumn.fromConstBoolColumn(jsonValue.getAsBoolean());
- case BINARY :
- return OTSColumn.fromConstBytesColumn(Base64.decodeBase64(jsonValue.getAsString()));
- default:
- throw new IllegalArgumentException("Unsupport deserialize the type : " + type + "");
- }
- }
-
- private void serializeNormalColumn(JsonObject json, OTSColumn obj) {
- json.add(NAME, new JsonPrimitive(obj.getName()));
- }
-
- private OTSColumn deserializeNormarlColumn(JsonObject obj) {
- return OTSColumn.fromNormalColumn(obj.getAsJsonPrimitive(NAME).getAsString());
- }
-
- @Override
- public JsonElement serialize(OTSColumn obj, Type t,
- JsonSerializationContext c) {
- JsonObject json = new JsonObject();
-
- switch (obj.getColumnType()) {
- case CONST:
- json.add(COLUMN_TYPE, new JsonPrimitive(OTSColumn.OTSColumnType.CONST.toString()));
- serializeConstColumn(json, obj);
- break;
- case NORMAL:
- json.add(COLUMN_TYPE, new JsonPrimitive(OTSColumn.OTSColumnType.NORMAL.toString()));
- serializeNormalColumn(json, obj);
- break;
- default:
- throw new IllegalArgumentException("Unsupport serialize the type : " + obj.getColumnType() + "");
- }
- return json;
- }
-
- @Override
- public OTSColumn deserialize(JsonElement ele, Type t,
- JsonDeserializationContext c) throws JsonParseException {
- JsonObject obj = ele.getAsJsonObject();
- String strColumnType = obj.getAsJsonPrimitive(COLUMN_TYPE).getAsString();
- OTSColumn.OTSColumnType columnType = OTSColumn.OTSColumnType.valueOf(strColumnType);
-
- switch(columnType) {
- case CONST:
- return deserializeConstColumn(obj);
- case NORMAL:
- return deserializeNormarlColumn(obj);
- default:
- throw new IllegalArgumentException("Unsupport deserialize the type : " + columnType + "");
- }
- }
-}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/PrimaryKeyValueAdaptor.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/PrimaryKeyValueAdaptor.java
index 1a49ea47..240427ae 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/PrimaryKeyValueAdaptor.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/adaptor/PrimaryKeyValueAdaptor.java
@@ -1,18 +1,12 @@
package com.alibaba.datax.plugin.reader.otsreader.adaptor;
-import java.lang.reflect.Type;
+import com.alicloud.openservices.tablestore.model.ColumnType;
+import com.alicloud.openservices.tablestore.model.PrimaryKeyType;
+import com.alicloud.openservices.tablestore.model.PrimaryKeyValue;
+import com.google.gson.*;
+import org.apache.commons.codec.binary.Base64;
-import com.aliyun.openservices.ots.model.ColumnType;
-import com.aliyun.openservices.ots.model.PrimaryKeyType;
-import com.aliyun.openservices.ots.model.PrimaryKeyValue;
-import com.google.gson.JsonDeserializationContext;
-import com.google.gson.JsonDeserializer;
-import com.google.gson.JsonElement;
-import com.google.gson.JsonObject;
-import com.google.gson.JsonParseException;
-import com.google.gson.JsonPrimitive;
-import com.google.gson.JsonSerializationContext;
-import com.google.gson.JsonSerializer;
+import java.lang.reflect.Type;
/**
* {"type":"INF_MIN", "value":""}
@@ -31,27 +25,29 @@ public class PrimaryKeyValueAdaptor implements JsonDeserializer
JsonSerializationContext c) {
JsonObject json = new JsonObject();
- if (obj == PrimaryKeyValue.INF_MIN) {
+ if (obj.isInfMin()) {
json.add(TYPE, new JsonPrimitive(INF_MIN));
- json.add(VALUE, new JsonPrimitive(""));
return json;
}
- if (obj == PrimaryKeyValue.INF_MAX) {
+ if (obj.isInfMax()) {
json.add(TYPE, new JsonPrimitive(INF_MAX));
- json.add(VALUE, new JsonPrimitive(""));
return json;
}
switch (obj.getType()) {
case STRING :
- json.add(TYPE, new JsonPrimitive(ColumnType.STRING.toString()));
+ json.add(TYPE, new JsonPrimitive(ColumnType.STRING.toString()));
json.add(VALUE, new JsonPrimitive(obj.asString()));
break;
case INTEGER :
json.add(TYPE, new JsonPrimitive(ColumnType.INTEGER.toString()));
json.add(VALUE, new JsonPrimitive(obj.asLong()));
break;
+ case BINARY :
+ json.add(TYPE, new JsonPrimitive(ColumnType.BINARY.toString()));
+ json.add(VALUE, new JsonPrimitive(Base64.encodeBase64String(obj.asBinary())));
+ break;
default:
throw new IllegalArgumentException("Unsupport serialize the type : " + obj.getType() + "");
}
@@ -64,16 +60,17 @@ public class PrimaryKeyValueAdaptor implements JsonDeserializer
JsonObject obj = ele.getAsJsonObject();
String strType = obj.getAsJsonPrimitive(TYPE).getAsString();
- JsonPrimitive jsonValue = obj.getAsJsonPrimitive(VALUE);
- if (strType.equals(INF_MIN)) {
+ if (strType.equalsIgnoreCase(INF_MIN)) {
return PrimaryKeyValue.INF_MIN;
}
- if (strType.equals(INF_MAX)) {
+ if (strType.equalsIgnoreCase(INF_MAX)) {
return PrimaryKeyValue.INF_MAX;
}
+ JsonPrimitive jsonValue = obj.getAsJsonPrimitive(VALUE);
+
PrimaryKeyValue value = null;
PrimaryKeyType type = PrimaryKeyType.valueOf(strType);
switch(type) {
@@ -83,6 +80,9 @@ public class PrimaryKeyValueAdaptor implements JsonDeserializer
case INTEGER :
value = PrimaryKeyValue.fromLong(jsonValue.getAsLong());
break;
+ case BINARY :
+ value = PrimaryKeyValue.fromBinary(Base64.decodeBase64(jsonValue.getAsString()));
+ break;
default:
throw new IllegalArgumentException("Unsupport deserialize the type : " + type + "");
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetFirstRowPrimaryKeyCallable.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetFirstRowPrimaryKeyCallable.java
index f004c0ff..cdcae91a 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetFirstRowPrimaryKeyCallable.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetFirstRowPrimaryKeyCallable.java
@@ -1,53 +1,42 @@
package com.alibaba.datax.plugin.reader.otsreader.callable;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.model.*;
+
+import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
-import com.aliyun.openservices.ots.OTSClient;
-import com.aliyun.openservices.ots.model.ColumnType;
-import com.aliyun.openservices.ots.model.ColumnValue;
-import com.aliyun.openservices.ots.model.GetRangeRequest;
-import com.aliyun.openservices.ots.model.GetRangeResult;
-import com.aliyun.openservices.ots.model.PrimaryKeyType;
-import com.aliyun.openservices.ots.model.PrimaryKeyValue;
-import com.aliyun.openservices.ots.model.RangeRowQueryCriteria;
-import com.aliyun.openservices.ots.model.Row;
-import com.aliyun.openservices.ots.model.RowPrimaryKey;
-import com.aliyun.openservices.ots.model.TableMeta;
+public class GetFirstRowPrimaryKeyCallable implements Callable> {
-public class GetFirstRowPrimaryKeyCallable implements Callable{
-
- private OTSClient ots = null;
+ private SyncClientInterface ots = null;
private TableMeta meta = null;
private RangeRowQueryCriteria criteria = null;
-
- public GetFirstRowPrimaryKeyCallable(OTSClient ots, TableMeta meta, RangeRowQueryCriteria criteria) {
+
+ public GetFirstRowPrimaryKeyCallable(SyncClientInterface ots, TableMeta meta, RangeRowQueryCriteria criteria) {
this.ots = ots;
this.meta = meta;
this.criteria = criteria;
}
-
+
@Override
- public RowPrimaryKey call() throws Exception {
- RowPrimaryKey ret = new RowPrimaryKey();
+ public List call() throws Exception {
+ List ret = new ArrayList<>();
GetRangeRequest request = new GetRangeRequest();
request.setRangeRowQueryCriteria(criteria);
- GetRangeResult result = ots.getRange(request);
- List rows = result.getRows();
- if(rows.isEmpty()) {
+ GetRangeResponse response = ots.getRange(request);
+ List rows = response.getRows();
+ if (rows.isEmpty()) {
return null;// no data
- }
+ }
Row row = rows.get(0);
- Map pk = meta.getPrimaryKey();
- for (String key:pk.keySet()) {
- ColumnValue v = row.getColumns().get(key);
- if (v.getType() == ColumnType.INTEGER) {
- ret.addPrimaryKeyColumn(key, PrimaryKeyValue.fromLong(v.asLong()));
- } else {
- ret.addPrimaryKeyColumn(key, PrimaryKeyValue.fromString(v.asString()));
- }
+ Map pk = meta.getPrimaryKeyMap();
+
+ for (String key : pk.keySet()) {
+ PrimaryKeyColumn v = row.getPrimaryKey().getPrimaryKeyColumnsMap().get(key);
+ ret.add(v);
}
return ret;
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetRangeCallable.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetRangeCallable.java
index 2cd1398a..995d491c 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetRangeCallable.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetRangeCallable.java
@@ -1,35 +1,26 @@
package com.alibaba.datax.plugin.reader.otsreader.callable;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.model.GetRangeRequest;
+import com.alicloud.openservices.tablestore.model.GetRangeResponse;
+import com.alicloud.openservices.tablestore.model.RangeRowQueryCriteria;
+
import java.util.concurrent.Callable;
-import com.aliyun.openservices.ots.OTSClientAsync;
-import com.aliyun.openservices.ots.model.GetRangeRequest;
-import com.aliyun.openservices.ots.model.GetRangeResult;
-import com.aliyun.openservices.ots.model.OTSFuture;
-import com.aliyun.openservices.ots.model.RangeRowQueryCriteria;
-
-public class GetRangeCallable implements Callable {
+public class GetRangeCallable implements Callable {
- private OTSClientAsync ots;
+ private SyncClientInterface ots;
private RangeRowQueryCriteria criteria;
- private OTSFuture future;
- public GetRangeCallable(OTSClientAsync ots, RangeRowQueryCriteria criteria, OTSFuture future) {
+ public GetRangeCallable(SyncClientInterface ots, RangeRowQueryCriteria criteria) {
this.ots = ots;
this.criteria = criteria;
- this.future = future;
}
@Override
- public GetRangeResult call() throws Exception {
- try {
- return future.get();
- } catch (Exception e) {
- GetRangeRequest request = new GetRangeRequest();
- request.setRangeRowQueryCriteria(criteria);
- future = ots.getRange(request);
- throw e;
- }
+ public GetRangeResponse call() throws Exception {
+ GetRangeRequest request = new GetRangeRequest();
+ request.setRangeRowQueryCriteria(criteria);
+ return ots.getRange(request);
}
-
-}
+}
\ No newline at end of file
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetRangeCallableOld.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetRangeCallableOld.java
new file mode 100644
index 00000000..c0434126
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetRangeCallableOld.java
@@ -0,0 +1,35 @@
+package com.alibaba.datax.plugin.reader.otsreader.callable;
+
+import java.util.concurrent.Callable;
+
+import com.aliyun.openservices.ots.OTSClientAsync;
+import com.aliyun.openservices.ots.model.GetRangeRequest;
+import com.aliyun.openservices.ots.model.GetRangeResult;
+import com.aliyun.openservices.ots.model.OTSFuture;
+import com.aliyun.openservices.ots.model.RangeRowQueryCriteria;
+
+public class GetRangeCallableOld implements Callable {
+
+ private OTSClientAsync ots;
+ private RangeRowQueryCriteria criteria;
+ private OTSFuture future;
+
+ public GetRangeCallableOld(OTSClientAsync ots, RangeRowQueryCriteria criteria, OTSFuture future) {
+ this.ots = ots;
+ this.criteria = criteria;
+ this.future = future;
+ }
+
+ @Override
+ public GetRangeResult call() throws Exception {
+ try {
+ return future.get();
+ } catch (Exception e) {
+ GetRangeRequest request = new GetRangeRequest();
+ request.setRangeRowQueryCriteria(criteria);
+ future = ots.getRange(request);
+ throw e;
+ }
+ }
+
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetTableMetaCallable.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetTableMetaCallable.java
index 2884e12b..36a122c2 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetTableMetaCallable.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetTableMetaCallable.java
@@ -1,18 +1,19 @@
package com.alibaba.datax.plugin.reader.otsreader.callable;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.model.DescribeTableRequest;
+import com.alicloud.openservices.tablestore.model.DescribeTableResponse;
+import com.alicloud.openservices.tablestore.model.TableMeta;
+
import java.util.concurrent.Callable;
-import com.aliyun.openservices.ots.OTSClient;
-import com.aliyun.openservices.ots.model.DescribeTableRequest;
-import com.aliyun.openservices.ots.model.DescribeTableResult;
-import com.aliyun.openservices.ots.model.TableMeta;
public class GetTableMetaCallable implements Callable{
- private OTSClient ots = null;
+ private SyncClientInterface ots = null;
private String tableName = null;
- public GetTableMetaCallable(OTSClient ots, String tableName) {
+ public GetTableMetaCallable(SyncClientInterface ots, String tableName) {
this.ots = ots;
this.tableName = tableName;
}
@@ -21,9 +22,9 @@ public class GetTableMetaCallable implements Callable{
public TableMeta call() throws Exception {
DescribeTableRequest describeTableRequest = new DescribeTableRequest();
describeTableRequest.setTableName(tableName);
- DescribeTableResult result = ots.describeTable(describeTableRequest);
+ DescribeTableResponse result = ots.describeTable(describeTableRequest);
TableMeta tableMeta = result.getTableMeta();
return tableMeta;
}
-}
+}
\ No newline at end of file
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetTimeseriesSplitCallable.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetTimeseriesSplitCallable.java
new file mode 100644
index 00000000..96521c41
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/GetTimeseriesSplitCallable.java
@@ -0,0 +1,38 @@
+package com.alibaba.datax.plugin.reader.otsreader.callable;
+
+import com.alicloud.openservices.tablestore.SyncClient;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.TimeseriesClient;
+import com.alicloud.openservices.tablestore.model.timeseries.SplitTimeseriesScanTaskRequest;
+import com.alicloud.openservices.tablestore.model.timeseries.SplitTimeseriesScanTaskResponse;
+import com.alicloud.openservices.tablestore.model.timeseries.TimeseriesScanSplitInfo;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+
+public class GetTimeseriesSplitCallable implements Callable> {
+
+ private TimeseriesClient client = null;
+ private String timeseriesTableName = null;
+ private String measurementName = null;
+ private int splitCountHint = 1;
+
+
+ public GetTimeseriesSplitCallable(SyncClientInterface ots, String timeseriesTableName, String measurementName, int splitCountHint) {
+ this.client = ((SyncClient) ots).asTimeseriesClient();
+ this.timeseriesTableName = timeseriesTableName;
+ this.measurementName = measurementName;
+ this.splitCountHint = splitCountHint;
+ }
+
+ @Override
+ public List call() throws Exception {
+ SplitTimeseriesScanTaskRequest request = new SplitTimeseriesScanTaskRequest(timeseriesTableName, splitCountHint);
+ if (measurementName.length() != 0) {
+ request.setMeasurementName(measurementName);
+ }
+
+ SplitTimeseriesScanTaskResponse response = client.splitTimeseriesScanTask(request);
+ return response.getSplitInfos();
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/ScanTimeseriesDataCallable.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/ScanTimeseriesDataCallable.java
new file mode 100644
index 00000000..726d0e5d
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/callable/ScanTimeseriesDataCallable.java
@@ -0,0 +1,27 @@
+package com.alibaba.datax.plugin.reader.otsreader.callable;
+
+import com.alicloud.openservices.tablestore.SyncClient;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.TimeseriesClient;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataRequest;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataResponse;
+import com.alicloud.openservices.tablestore.model.timeseries.TimeseriesScanSplitInfo;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+
+public class ScanTimeseriesDataCallable implements Callable {
+
+ private TimeseriesClient client = null;
+ private ScanTimeseriesDataRequest request = null;
+
+ public ScanTimeseriesDataCallable(SyncClientInterface ots, ScanTimeseriesDataRequest scanTimeseriesDataRequest){
+ this.client = ((SyncClient) ots).asTimeseriesClient();
+ this.request = scanTimeseriesDataRequest;
+ }
+
+ @Override
+ public ScanTimeseriesDataResponse call() throws Exception {
+ return client.scanTimeseriesData(request);
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/DefaultNoRetry.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/DefaultNoRetry.java
new file mode 100644
index 00000000..b286472d
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/DefaultNoRetry.java
@@ -0,0 +1,32 @@
+package com.alibaba.datax.plugin.reader.otsreader.model;
+
+
+import com.alicloud.openservices.tablestore.model.DefaultRetryStrategy;
+import com.alicloud.openservices.tablestore.model.RetryStrategy;
+
+public class DefaultNoRetry extends DefaultRetryStrategy {
+
+ public DefaultNoRetry() {
+ super();
+ }
+
+ @Override
+ public RetryStrategy clone() {
+ return super.clone();
+ }
+
+ @Override
+ public int getRetries() {
+ return super.getRetries();
+ }
+
+ @Override
+ public boolean shouldRetry(String action, Exception ex) {
+ return false;
+ }
+
+ @Override
+ public long nextPause(String action, Exception ex) {
+ return super.nextPause(action, ex);
+ }
+}
\ No newline at end of file
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSColumn.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSColumn.java
index 129ccd2f..809f4c38 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSColumn.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSColumn.java
@@ -1,19 +1,18 @@
package com.alibaba.datax.plugin.reader.otsreader.model;
-import com.alibaba.datax.common.element.BoolColumn;
-import com.alibaba.datax.common.element.BytesColumn;
-import com.alibaba.datax.common.element.Column;
-import com.alibaba.datax.common.element.DoubleColumn;
-import com.alibaba.datax.common.element.LongColumn;
-import com.alibaba.datax.common.element.StringColumn;
-import com.aliyun.openservices.ots.model.ColumnType;
+import com.alibaba.datax.common.element.*;
+import com.alicloud.openservices.tablestore.model.ColumnType;
public class OTSColumn {
private String name;
private Column value;
+
private OTSColumnType columnType;
+
+ // 时序数据column配置
private ColumnType valueType;
-
+ private Boolean isTimeseriesTag;
+
public static enum OTSColumnType {
NORMAL, // 普通列
CONST // 常量列
@@ -24,10 +23,9 @@ public class OTSColumn {
this.columnType = OTSColumnType.NORMAL;
}
- private OTSColumn(Column value, ColumnType type) {
+ private OTSColumn(Column value) {
this.value = value;
this.columnType = OTSColumnType.CONST;
- this.valueType = type;
}
public static OTSColumn fromNormalColumn(String name) {
@@ -39,23 +37,23 @@ public class OTSColumn {
}
public static OTSColumn fromConstStringColumn(String value) {
- return new OTSColumn(new StringColumn(value), ColumnType.STRING);
+ return new OTSColumn(new StringColumn(value));
}
public static OTSColumn fromConstIntegerColumn(long value) {
- return new OTSColumn(new LongColumn(value), ColumnType.INTEGER);
+ return new OTSColumn(new LongColumn(value));
}
public static OTSColumn fromConstDoubleColumn(double value) {
- return new OTSColumn(new DoubleColumn(value), ColumnType.DOUBLE);
+ return new OTSColumn(new DoubleColumn(value));
}
public static OTSColumn fromConstBoolColumn(boolean value) {
- return new OTSColumn(new BoolColumn(value), ColumnType.BOOLEAN);
+ return new OTSColumn(new BoolColumn(value));
}
public static OTSColumn fromConstBytesColumn(byte[] value) {
- return new OTSColumn(new BytesColumn(value), ColumnType.BINARY);
+ return new OTSColumn(new BytesColumn(value));
}
public Column getValue() {
@@ -65,12 +63,25 @@ public class OTSColumn {
public OTSColumnType getColumnType() {
return columnType;
}
-
- public ColumnType getValueType() {
- return valueType;
- }
+
public String getName() {
return name;
}
-}
+
+ public ColumnType getValueType() {
+ return valueType;
+ }
+
+ public void setValueType(ColumnType valueType) {
+ this.valueType = valueType;
+ }
+
+ public Boolean getTimeseriesTag() {
+ return isTimeseriesTag;
+ }
+
+ public void setTimeseriesTag(Boolean timeseriesTag) {
+ isTimeseriesTag = timeseriesTag;
+ }
+}
\ No newline at end of file
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSConf.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSConf.java
index 8b109a39..cbfd8f6a 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSConf.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSConf.java
@@ -1,90 +1,245 @@
package com.alibaba.datax.plugin.reader.otsreader.model;
+import com.alibaba.datax.common.element.Column;
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.reader.otsreader.utils.Constant;
+import com.alibaba.datax.plugin.reader.otsreader.utils.Key;
+import com.alibaba.datax.plugin.reader.otsreader.utils.ParamChecker;
+import com.alicloud.openservices.tablestore.model.ColumnType;
+
import java.util.List;
-import com.aliyun.openservices.ots.model.PrimaryKeyValue;
-
public class OTSConf {
- private String endpoint= null;
+ private String endpoint = null;
private String accessId = null;
- private String accesskey = null;
+ private String accessKey = null;
private String instanceName = null;
private String tableName = null;
+ private OTSRange range = null;
+ private List column = null;
+ private OTSMode mode = null;
+
+ @Deprecated
+ private String metaMode = "";
+
+ private boolean newVersion = false;
+ /**
+ * 以下配置仅用于timeseries数据读取
+ */
+ private boolean isTimeseriesTable = false;
+ private String measurementName = null;
+ /**
+ * 以上配置仅用于timeseries数据读取
+ */
+ private OTSMultiVersionConf multi = null;
- private List rangeBegin = null;
- private List rangeEnd = null;
- private List rangeSplit = null;
-
- private List columns = null;
-
- private int retry;
- private int sleepInMilliSecond;
-
+ private int retry = Constant.ConfigDefaultValue.RETRY;
+ private int retryPauseInMillisecond = Constant.ConfigDefaultValue.RETRY_PAUSE_IN_MILLISECOND;
+ private int ioThreadCount = Constant.ConfigDefaultValue.IO_THREAD_COUNT;
+ private int maxConnectionCount = Constant.ConfigDefaultValue.MAX_CONNECTION_COUNT;
+ private int socketTimeoutInMillisecond = Constant.ConfigDefaultValue.SOCKET_TIMEOUT_IN_MILLISECOND;
+ private int connectTimeoutInMillisecond = Constant.ConfigDefaultValue.CONNECT_TIMEOUT_IN_MILLISECOND;
+
+ public int getIoThreadCount() {
+ return ioThreadCount;
+ }
+
+ public void setIoThreadCount(int ioThreadCount) {
+ this.ioThreadCount = ioThreadCount;
+ }
+
+ public int getMaxConnectCount() {
+ return maxConnectionCount;
+ }
+
+ public void setMaxConnectCount(int maxConnectCount) {
+ this.maxConnectionCount = maxConnectCount;
+ }
+
+ public int getSocketTimeoutInMillisecond() {
+ return socketTimeoutInMillisecond;
+ }
+
+ public void setSocketTimeoutInMillisecond(int socketTimeoutInMillisecond) {
+ this.socketTimeoutInMillisecond = socketTimeoutInMillisecond;
+ }
+
+ public int getConnectTimeoutInMillisecond() {
+ return connectTimeoutInMillisecond;
+ }
+
+ public void setConnectTimeoutInMillisecond(int connectTimeoutInMillisecond) {
+ this.connectTimeoutInMillisecond = connectTimeoutInMillisecond;
+ }
+
+ public int getRetry() {
+ return retry;
+ }
+
+ public void setRetry(int retry) {
+ this.retry = retry;
+ }
+
+ public int getRetryPauseInMillisecond() {
+ return retryPauseInMillisecond;
+ }
+
+ public void setRetryPauseInMillisecond(int sleepInMillisecond) {
+ this.retryPauseInMillisecond = sleepInMillisecond;
+ }
+
public String getEndpoint() {
return endpoint;
}
+
public void setEndpoint(String endpoint) {
this.endpoint = endpoint;
}
+
public String getAccessId() {
return accessId;
}
+
public void setAccessId(String accessId) {
this.accessId = accessId;
}
- public String getAccesskey() {
- return accesskey;
+
+ public String getAccessKey() {
+ return accessKey;
}
- public void setAccesskey(String accesskey) {
- this.accesskey = accesskey;
+
+ public void setAccessKey(String accessKey) {
+ this.accessKey = accessKey;
}
+
public String getInstanceName() {
return instanceName;
}
+
public void setInstanceName(String instanceName) {
this.instanceName = instanceName;
}
+
public String getTableName() {
return tableName;
}
+
public void setTableName(String tableName) {
this.tableName = tableName;
}
- public List getColumns() {
- return columns;
+ public OTSRange getRange() {
+ return range;
}
- public void setColumns(List columns) {
- this.columns = columns;
+
+ public void setRange(OTSRange range) {
+ this.range = range;
}
- public int getRetry() {
- return retry;
+
+ public OTSMode getMode() {
+ return mode;
}
- public void setRetry(int retry) {
- this.retry = retry;
+
+ public void setMode(OTSMode mode) {
+ this.mode = mode;
}
- public int getSleepInMilliSecond() {
- return sleepInMilliSecond;
+
+ public OTSMultiVersionConf getMulti() {
+ return multi;
}
- public void setSleepInMilliSecond(int sleepInMilliSecond) {
- this.sleepInMilliSecond = sleepInMilliSecond;
+
+ public void setMulti(OTSMultiVersionConf multi) {
+ this.multi = multi;
}
- public List getRangeBegin() {
- return rangeBegin;
+
+ public List getColumn() {
+ return column;
}
- public void setRangeBegin(List rangeBegin) {
- this.rangeBegin = rangeBegin;
+
+ public void setColumn(List column) {
+ this.column = column;
}
- public List getRangeEnd() {
- return rangeEnd;
+
+ public boolean isNewVersion() {
+ return newVersion;
}
- public void setRangeEnd(List rangeEnd) {
- this.rangeEnd = rangeEnd;
+
+ public void setNewVersion(boolean newVersion) {
+ this.newVersion = newVersion;
}
- public List getRangeSplit() {
- return rangeSplit;
+
+ @Deprecated
+ public String getMetaMode() {
+ return metaMode;
}
- public void setRangeSplit(List rangeSplit) {
- this.rangeSplit = rangeSplit;
+
+ @Deprecated
+ public void setMetaMode(String metaMode) {
+ this.metaMode = metaMode;
+ }
+
+ public boolean isTimeseriesTable() {
+ return isTimeseriesTable;
+ }
+
+ public void setTimeseriesTable(boolean timeseriesTable) {
+ isTimeseriesTable = timeseriesTable;
+ }
+
+ public String getMeasurementName() {
+ return measurementName;
+ }
+
+ public void setMeasurementName(String measurementName) {
+ this.measurementName = measurementName;
+ }
+
+ public static OTSConf load(Configuration param) throws OTSCriticalException {
+ OTSConf c = new OTSConf();
+
+ // account
+ c.setEndpoint(ParamChecker.checkStringAndGet(param, Key.OTS_ENDPOINT, true));
+ c.setAccessId(ParamChecker.checkStringAndGet(param, Key.OTS_ACCESSID, true));
+ c.setAccessKey(ParamChecker.checkStringAndGet(param, Key.OTS_ACCESSKEY, true));
+ c.setInstanceName(ParamChecker.checkStringAndGet(param, Key.OTS_INSTANCE_NAME, true));
+ c.setTableName(ParamChecker.checkStringAndGet(param, Key.TABLE_NAME, true));
+
+ c.setRetry(param.getInt(Constant.ConfigKey.RETRY, Constant.ConfigDefaultValue.RETRY));
+ c.setRetryPauseInMillisecond(param.getInt(Constant.ConfigKey.RETRY_PAUSE_IN_MILLISECOND, Constant.ConfigDefaultValue.RETRY_PAUSE_IN_MILLISECOND));
+ c.setIoThreadCount(param.getInt(Constant.ConfigKey.IO_THREAD_COUNT, Constant.ConfigDefaultValue.IO_THREAD_COUNT));
+ c.setMaxConnectCount(param.getInt(Constant.ConfigKey.MAX_CONNECTION_COUNT, Constant.ConfigDefaultValue.MAX_CONNECTION_COUNT));
+ c.setSocketTimeoutInMillisecond(param.getInt(Constant.ConfigKey.SOCKET_TIMEOUTIN_MILLISECOND, Constant.ConfigDefaultValue.SOCKET_TIMEOUT_IN_MILLISECOND));
+ c.setConnectTimeoutInMillisecond(param.getInt(Constant.ConfigKey.CONNECT_TIMEOUT_IN_MILLISECOND, Constant.ConfigDefaultValue.CONNECT_TIMEOUT_IN_MILLISECOND));
+
+ // range
+ c.setRange(ParamChecker.checkRangeAndGet(param));
+
+ // mode 可选参数
+ c.setMode(ParamChecker.checkModeAndGet(param));
+ //isNewVersion 可选参数
+ c.setNewVersion(param.getBool(Key.NEW_VERSION, false));
+ // metaMode 旧版本配置
+ c.setMetaMode(param.getString(Key.META_MODE, ""));
+
+
+
+ // 读时序表配置项
+ c.setTimeseriesTable(param.getBool(Key.IS_TIMESERIES_TABLE, false));
+ // column
+ if(!c.isTimeseriesTable()){
+ //非时序表
+ c.setColumn(ParamChecker.checkOTSColumnAndGet(param, c.getMode()));
+ }
+ else{
+ // 时序表
+ c.setMeasurementName(param.getString(Key.MEASUREMENT_NAME, ""));
+ c.setColumn(ParamChecker.checkTimeseriesColumnAndGet(param));
+ ParamChecker.checkTimeseriesMode(c.getMode(), c.isNewVersion());
+ }
+
+ if (c.getMode() == OTSMode.MULTI_VERSION) {
+ c.setMulti(OTSMultiVersionConf.load(param));
+ }
+ return c;
}
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSCriticalException.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSCriticalException.java
new file mode 100644
index 00000000..f02346bc
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSCriticalException.java
@@ -0,0 +1,24 @@
+package com.alibaba.datax.plugin.reader.otsreader.model;
+
+/**
+ * 插件错误异常，该异常主要用于描述插件的异常退出
+ * @author redchen
+ */
+public class OTSCriticalException extends Exception{
+
+ private static final long serialVersionUID = 5820460098894295722L;
+
+ public OTSCriticalException() {}
+
+ public OTSCriticalException(String message) {
+ super(message);
+ }
+
+ public OTSCriticalException(Throwable a) {
+ super(a);
+ }
+
+ public OTSCriticalException(String message, Throwable a) {
+ super(message, a);
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSErrorCode.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSErrorCode.java
new file mode 100644
index 00000000..0c537fce
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSErrorCode.java
@@ -0,0 +1,115 @@
+/**
+ * Copyright (C) Alibaba Cloud Computing
+ * All rights reserved.
+ *
+ * 版权所有 （C）阿里云计算有限公司
+ */
+
+package com.alibaba.datax.plugin.reader.otsreader.model;
+
+/**
+ * 表示来自开放结构化数据服务（Open Table Service，OTS）的错误代码。
+ *
+ */
+public class OTSErrorCode {
+ /**
+ * 用户身份验证失败。
+ */
+ public static final String AUTHORIZATION_FAILURE = "OTSAuthFailed";
+
+ /**
+ * 服务器内部错误。
+ */
+ public static final String INTERNAL_SERVER_ERROR = "OTSInternalServerError";
+
+ /**
+ * 参数错误。
+ */
+ public static final String INVALID_PARAMETER = "OTSParameterInvalid";
+
+ /**
+ * 整个请求过大。
+ */
+ public static final String REQUEST_TOO_LARGE = "OTSRequestBodyTooLarge";
+
+ /**
+ * 客户端请求超时。
+ */
+ public static final String REQUEST_TIMEOUT = "OTSRequestTimeout";
+
+ /**
+ * 用户的配额已经用满。
+ */
+ public static final String QUOTA_EXHAUSTED = "OTSQuotaExhausted";
+
+ /**
+ * 内部服务器发生failover，导致表的部分分区不可服务。
+ */
+ public static final String PARTITION_UNAVAILABLE = "OTSPartitionUnavailable";
+
+ /**
+ * 表刚被创建还无法立马提供服务。
+ */
+ public static final String TABLE_NOT_READY = "OTSTableNotReady";
+
+ /**
+ * 请求的表不存在。
+ */
+ public static final String OBJECT_NOT_EXIST = "OTSObjectNotExist";
+
+ /**
+ * 请求创建的表已经存在。
+ */
+ public static final String OBJECT_ALREADY_EXIST = "OTSObjectAlreadyExist";
+
+ /**
+ * 多个并发的请求写同一行数据，导致冲突。
+ */
+ public static final String ROW_OPEARTION_CONFLICT = "OTSRowOperationConflict";
+
+ /**
+ * 主键不匹配。
+ */
+ public static final String INVALID_PK = "OTSInvalidPK";
+
+ /**
+ * 读写能力调整过于频繁。
+ */
+ public static final String TOO_FREQUENT_RESERVED_THROUGHPUT_ADJUSTMENT = "OTSTooFrequentReservedThroughputAdjustment";
+
+ /**
+ * 该行总列数超出限制。
+ */
+ public static final String OUT_OF_COLUMN_COUNT_LIMIT = "OTSOutOfColumnCountLimit";
+
+ /**
+ * 该行所有列数据大小总和超出限制。
+ */
+ public static final String OUT_OF_ROW_SIZE_LIMIT = "OTSOutOfRowSizeLimit";
+
+ /**
+ * 剩余预留读写能力不足。
+ */
+ public static final String NOT_ENOUGH_CAPACITY_UNIT = "OTSNotEnoughCapacityUnit";
+
+ /**
+ * 预查条件检查失败。
+ */
+ public static final String CONDITION_CHECK_FAIL = "OTSConditionCheckFail";
+
+ /**
+ * 在OTS内部操作超时。
+ */
+ public static final String STORAGE_TIMEOUT = "OTSTimeout";
+
+ /**
+ * 在OTS内部有服务器不可访问。
+ */
+ public static final String SERVER_UNAVAILABLE = "OTSServerUnavailable";
+
+ /**
+ * OTS内部服务器繁忙。
+ */
+ public static final String SERVER_BUSY = "OTSServerBusy";
+
+}
\ No newline at end of file
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSMode.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSMode.java
new file mode 100644
index 00000000..88c6ee67
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSMode.java
@@ -0,0 +1,6 @@
+package com.alibaba.datax.plugin.reader.otsreader.model;
+
+public enum OTSMode {
+ NORMAL,
+ MULTI_VERSION
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSMultiVersionConf.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSMultiVersionConf.java
new file mode 100644
index 00000000..72a8e1b7
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSMultiVersionConf.java
@@ -0,0 +1,35 @@
+package com.alibaba.datax.plugin.reader.otsreader.model;
+
+import com.alibaba.datax.common.util.Configuration;
+import com.alibaba.datax.plugin.reader.otsreader.utils.Constant;
+import com.alibaba.datax.plugin.reader.otsreader.utils.ParamChecker;
+import com.alicloud.openservices.tablestore.model.TimeRange;
+
+public class OTSMultiVersionConf {
+
+ private TimeRange timeRange = null;
+ private int maxVersion = -1;
+
+ public TimeRange getTimeRange() {
+ return timeRange;
+ }
+
+ public void setTimeRange(TimeRange timeRange) {
+ this.timeRange = timeRange;
+ }
+
+ public int getMaxVersion() {
+ return maxVersion;
+ }
+
+ public void setMaxVersion(int maxVersion) {
+ this.maxVersion = maxVersion;
+ }
+
+ public static OTSMultiVersionConf load(Configuration param) throws OTSCriticalException {
+ OTSMultiVersionConf conf = new OTSMultiVersionConf();
+ conf.setTimeRange(ParamChecker.checkTimeRangeAndGet(param));
+ conf.setMaxVersion(param.getInt(Constant.ConfigKey.MAX_VERSION, Constant.ConfigDefaultValue.MAX_VERSION));
+ return conf;
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSPrimaryKeyColumn.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSPrimaryKeyColumn.java
index eaec50ce..44a37c0c 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSPrimaryKeyColumn.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSPrimaryKeyColumn.java
@@ -15,8 +15,41 @@ public class OTSPrimaryKeyColumn {
public PrimaryKeyType getType() {
return type;
}
+
+ public com.alicloud.openservices.tablestore.model.PrimaryKeyType getType(Boolean newVersion) {
+ com.alicloud.openservices.tablestore.model.PrimaryKeyType res = null;
+ switch (this.type){
+ case BINARY:
+ res = com.alicloud.openservices.tablestore.model.PrimaryKeyType.BINARY;
+ break;
+ case INTEGER:
+ res = com.alicloud.openservices.tablestore.model.PrimaryKeyType.INTEGER;
+ break;
+ case STRING:
+ default:
+ res = com.alicloud.openservices.tablestore.model.PrimaryKeyType.STRING;
+ break;
+ }
+ return res;
+ }
+
public void setType(PrimaryKeyType type) {
this.type = type;
}
+
+ public void setType(com.alicloud.openservices.tablestore.model.PrimaryKeyType type) {
+ switch (type){
+ case BINARY:
+ this.type = PrimaryKeyType.BINARY;
+ break;
+ case INTEGER:
+ this.type = PrimaryKeyType.INTEGER;
+ break;
+ case STRING:
+ default:
+ this.type = PrimaryKeyType.STRING;
+ break;
+ }
+ }
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSRange.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSRange.java
index 8ebfcf7e..eb3095e6 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSRange.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/model/OTSRange.java
@@ -1,29 +1,31 @@
package com.alibaba.datax.plugin.reader.otsreader.model;
-import com.aliyun.openservices.ots.model.RowPrimaryKey;
+import com.alicloud.openservices.tablestore.model.PrimaryKeyColumn;
+
+import java.util.List;
+
public class OTSRange {
+ private List begin = null;
+ private List end = null;
+ private List split = null;
- private RowPrimaryKey begin = null;
- private RowPrimaryKey end = null;
-
- public OTSRange() {}
-
- public OTSRange(RowPrimaryKey begin, RowPrimaryKey end) {
- this.begin = begin;
- this.end = end;
- }
-
- public RowPrimaryKey getBegin() {
+ public List getBegin() {
return begin;
}
- public void setBegin(RowPrimaryKey begin) {
+ public void setBegin(List begin) {
this.begin = begin;
}
- public RowPrimaryKey getEnd() {
+ public List getEnd() {
return end;
}
- public void setEnd(RowPrimaryKey end) {
+ public void setEnd(List end) {
this.end = end;
}
+ public List getSplit() {
+ return split;
+ }
+ public void setSplit(List split) {
+ this.split = split;
+ }
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Common.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Common.java
index fb8c7feb..90065d5d 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Common.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Common.java
@@ -1,26 +1,85 @@
package com.alibaba.datax.plugin.reader.otsreader.utils;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSCriticalException;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSPrimaryKeyColumn;
+import com.alicloud.openservices.tablestore.model.*;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataResponse;
+
+import java.lang.reflect.Field;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import com.alibaba.datax.common.element.BoolColumn;
-import com.alibaba.datax.common.element.BytesColumn;
-import com.alibaba.datax.common.element.DoubleColumn;
-import com.alibaba.datax.common.element.LongColumn;
-import com.alibaba.datax.common.element.Record;
-import com.alibaba.datax.common.element.StringColumn;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSPrimaryKeyColumn;
-import com.aliyun.openservices.ots.ClientException;
-import com.aliyun.openservices.ots.OTSException;
-import com.aliyun.openservices.ots.model.ColumnValue;
-import com.aliyun.openservices.ots.model.PrimaryKeyValue;
-import com.aliyun.openservices.ots.model.Row;
-import com.aliyun.openservices.ots.model.RowPrimaryKey;
-import com.aliyun.openservices.ots.model.TableMeta;
-
public class Common {
+ public static List toColumnToGet(List columns, TableMeta meta) {
+ Map pk = meta.getPrimaryKeyMap();
+ List names = new ArrayList();
+ for (OTSColumn c : columns) {
+ if (c.getColumnType() == OTSColumn.OTSColumnType.NORMAL && !pk.containsKey(c.getName())) {
+ names.add(c.getName());
+ }
+ }
+ return names;
+ }
+
+ public static List getPrimaryKeyNameList(TableMeta meta) {
+ List names = new ArrayList();
+ names.addAll(meta.getPrimaryKeyMap().keySet());
+ return names;
+ }
+
+ public static OTSPrimaryKeyColumn getPartitionKey(TableMeta meta) {
+ List keys = new ArrayList();
+ keys.addAll(meta.getPrimaryKeyMap().keySet());
+
+ String key = keys.get(0);
+
+ OTSPrimaryKeyColumn col = new OTSPrimaryKeyColumn();
+ col.setName(key);
+ col.setType(meta.getPrimaryKeyMap().get(key));
+ return col;
+ }
+
+ public static Direction getDirection(List begin, List end) throws OTSCriticalException {
+ int cmp = CompareHelper.comparePrimaryKeyColumnList(begin, end);
+ if (cmp < 0) {
+ return Direction.FORWARD;
+ } else if (cmp > 0) {
+ return Direction.BACKWARD;
+ } else {
+ throw new OTSCriticalException("Bug branch, the begin of range equals end of range.");
+ }
+ }
+
+ public static int compareRangeBeginAndEnd(TableMeta meta, List begin, List end) {
+ if (begin.size() != end.size()) {
+ throw new IllegalArgumentException("Input size of begin not equal size of end, begin size : " + begin.size() +
+ ", end size : " + end.size() + ".");
+ }
+
+ Map beginMap = new HashMap<>();
+ Map endMap = new HashMap<>();
+
+ for(PrimaryKeyColumn primaryKeyColumn : begin){
+ beginMap.put(primaryKeyColumn.getName(), primaryKeyColumn.getValue());
+ }
+ for(PrimaryKeyColumn primaryKeyColumn : end){
+ endMap.put(primaryKeyColumn.getName(), primaryKeyColumn.getValue());
+ }
+
+ for (String key : meta.getPrimaryKeyMap().keySet()) {
+ PrimaryKeyValue v1 = beginMap.get(key);
+ PrimaryKeyValue v2 = endMap.get(key);
+ int cmp = primaryKeyValueCmp(v1, v2);
+ if (cmp != 0) {
+ return cmp;
+ }
+ }
+ return 0;
+ }
+
public static int primaryKeyValueCmp(PrimaryKeyValue v1, PrimaryKeyValue v2) {
if (v1.getType() != null && v2.getType() != null) {
@@ -29,14 +88,14 @@ public class Common {
"Not same column type, column1:" + v1.getType() + ", column2:" + v2.getType());
}
switch (v1.getType()) {
- case INTEGER:
- Long l1 = Long.valueOf(v1.asLong());
- Long l2 = Long.valueOf(v2.asLong());
- return l1.compareTo(l2);
- case STRING:
- return v1.asString().compareTo(v2.asString());
- default:
- throw new IllegalArgumentException("Unsuporrt compare the type: " + v1.getType() + ".");
+ case INTEGER:
+ Long l1 = Long.valueOf(v1.asLong());
+ Long l2 = Long.valueOf(v2.asLong());
+ return l1.compareTo(l2);
+ case STRING:
+ return v1.asString().compareTo(v2.asString());
+ default:
+ throw new IllegalArgumentException("Unsuporrt compare the type: " + v1.getType() + ".");
}
} else {
if (v1 == v2) {
@@ -46,116 +105,31 @@ public class Common {
return -1;
} else if (v1 == PrimaryKeyValue.INF_MAX) {
return 1;
- }
+ }
if (v2 == PrimaryKeyValue.INF_MAX) {
return -1;
} else if (v2 == PrimaryKeyValue.INF_MIN) {
return 1;
- }
- }
- }
- return 0;
- }
-
- public static OTSPrimaryKeyColumn getPartitionKey(TableMeta meta) {
- List keys = new ArrayList();
- keys.addAll(meta.getPrimaryKey().keySet());
-
- String key = keys.get(0);
-
- OTSPrimaryKeyColumn col = new OTSPrimaryKeyColumn();
- col.setName(key);
- col.setType(meta.getPrimaryKey().get(key));
- return col;
- }
-
- public static List getPrimaryKeyNameList(TableMeta meta) {
- List names = new ArrayList();
- names.addAll(meta.getPrimaryKey().keySet());
- return names;
- }
-
- public static int compareRangeBeginAndEnd(TableMeta meta, RowPrimaryKey begin, RowPrimaryKey end) {
- if (begin.getPrimaryKey().size() != end.getPrimaryKey().size()) {
- throw new IllegalArgumentException("Input size of begin not equal size of end, begin size : " + begin.getPrimaryKey().size() +
- ", end size : " + end.getPrimaryKey().size() + ".");
- }
- for (String key : meta.getPrimaryKey().keySet()) {
- PrimaryKeyValue v1 = begin.getPrimaryKey().get(key);
- PrimaryKeyValue v2 = end.getPrimaryKey().get(key);
- int cmp = primaryKeyValueCmp(v1, v2);
- if (cmp != 0) {
- return cmp;
- }
- }
- return 0;
- }
-
- public static List getNormalColumnNameList(List columns) {
- List normalColumns = new ArrayList();
- for (OTSColumn col : columns) {
- if (col.getColumnType() == OTSColumn.OTSColumnType.NORMAL) {
- normalColumns.add(col.getName());
- }
- }
- return normalColumns;
- }
-
- public static Record parseRowToLine(Row row, List columns, Record line) {
- Map values = row.getColumns();
- for (OTSColumn col : columns) {
- if (col.getColumnType() == OTSColumn.OTSColumnType.CONST) {
- line.addColumn(col.getValue());
- } else {
- ColumnValue v = values.get(col.getName());
- if (v == null) {
- line.addColumn(new StringColumn(null));
- } else {
- switch(v.getType()) {
- case STRING: line.addColumn(new StringColumn(v.asString())); break;
- case INTEGER: line.addColumn(new LongColumn(v.asLong())); break;
- case DOUBLE: line.addColumn(new DoubleColumn(v.asDouble())); break;
- case BOOLEAN: line.addColumn(new BoolColumn(v.asBoolean())); break;
- case BINARY: line.addColumn(new BytesColumn(v.asBinary())); break;
- default:
- throw new IllegalArgumentException("Unsupported transform the type: " + col.getValue().getType() + ".");
- }
}
}
}
- return line;
+ return 0;
}
-
- public static String getDetailMessage(Exception exception) {
- if (exception instanceof OTSException) {
- OTSException e = (OTSException) exception;
- return "OTSException[ErrorCode:" + e.getErrorCode() + ", ErrorMessage:" + e.getMessage() + ", RequestId:" + e.getRequestId() + "]";
- } else if (exception instanceof ClientException) {
- ClientException e = (ClientException) exception;
- return "ClientException[ErrorCode:" + e.getErrorCode() + ", ErrorMessage:" + e.getMessage() + "]";
- } else if (exception instanceof IllegalArgumentException) {
- IllegalArgumentException e = (IllegalArgumentException) exception;
- return "IllegalArgumentException[ErrorMessage:" + e.getMessage() + "]";
- } else {
- return "Exception[ErrorMessage:" + exception.getMessage() + "]";
- }
- }
-
- public static long getDelaySendMillinSeconds(int hadRetryTimes, int initSleepInMilliSecond) {
- if (hadRetryTimes <= 0) {
- return 0;
- }
-
- int sleepTime = initSleepInMilliSecond;
- for (int i = 1; i < hadRetryTimes; i++) {
- sleepTime += sleepTime;
- if (sleepTime > 30000) {
- sleepTime = 30000;
+ public static void checkTableStoreSDKVersion() throws OTSCriticalException {
+ Field[] fields = ScanTimeseriesDataResponse.class.getFields();
+ String sdkVersion = null;
+ for (Field f : fields){
+ if (f.getName().equals("_VERSION_")){
+ sdkVersion = ScanTimeseriesDataResponse._VERSION_;
break;
- }
+ }
+ }
+ if (sdkVersion == null){
+ throw new OTSCriticalException("Check ots java SDK failed. Please check the version of tableStore maven dependency.");
+ }else if (Integer.parseInt(sdkVersion) < 20230111){
+ throw new OTSCriticalException("Check tableStore java SDK failed. The expected version number is greater than 20230111, actually version : " + sdkVersion + ".");
}
- return sleepTime;
}
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/CommonOld.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/CommonOld.java
new file mode 100644
index 00000000..d5c565f4
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/CommonOld.java
@@ -0,0 +1,112 @@
+package com.alibaba.datax.plugin.reader.otsreader.utils;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import com.alibaba.datax.common.element.BoolColumn;
+import com.alibaba.datax.common.element.BytesColumn;
+import com.alibaba.datax.common.element.DoubleColumn;
+import com.alibaba.datax.common.element.LongColumn;
+import com.alibaba.datax.common.element.Record;
+import com.alibaba.datax.common.element.StringColumn;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSPrimaryKeyColumn;
+import com.aliyun.openservices.ots.ClientException;
+import com.aliyun.openservices.ots.OTSException;
+import com.aliyun.openservices.ots.model.ColumnValue;
+import com.aliyun.openservices.ots.model.PrimaryKeyValue;
+import com.aliyun.openservices.ots.model.Row;
+import com.aliyun.openservices.ots.model.RowPrimaryKey;
+import com.aliyun.openservices.ots.model.TableMeta;
+
+public class CommonOld {
+ public static int primaryKeyValueCmp(PrimaryKeyValue v1, PrimaryKeyValue v2) {
+ if (v1.getType() != null && v2.getType() != null) {
+ if (v1.getType() != v2.getType()) {
+ throw new IllegalArgumentException(
+ "Not same column type, column1:" + v1.getType() + ", column2:" + v2.getType());
+ }
+ switch (v1.getType()) {
+ case INTEGER:
+ Long l1 = Long.valueOf(v1.asLong());
+ Long l2 = Long.valueOf(v2.asLong());
+ return l1.compareTo(l2);
+ case STRING:
+ return v1.asString().compareTo(v2.asString());
+ default:
+ throw new IllegalArgumentException("Unsuporrt compare the type: " + v1.getType() + ".");
+ }
+ } else {
+ if (v1 == v2) {
+ return 0;
+ } else {
+ if (v1 == PrimaryKeyValue.INF_MIN) {
+ return -1;
+ } else if (v1 == PrimaryKeyValue.INF_MAX) {
+ return 1;
+ }
+
+ if (v2 == PrimaryKeyValue.INF_MAX) {
+ return -1;
+ } else if (v2 == PrimaryKeyValue.INF_MIN) {
+ return 1;
+ }
+ }
+ }
+ return 0;
+ }
+
+
+ public static List getNormalColumnNameList(List columns) {
+ List normalColumns = new ArrayList();
+ for (OTSColumn col : columns) {
+ if (col.getColumnType() == OTSColumn.OTSColumnType.NORMAL) {
+ normalColumns.add(col.getName());
+ }
+ }
+ return normalColumns;
+ }
+
+ public static Record parseRowToLine(Row row, List columns, Record line) {
+ Map values = row.getColumns();
+ for (OTSColumn col : columns) {
+ if (col.getColumnType() == OTSColumn.OTSColumnType.CONST) {
+ line.addColumn(col.getValue());
+ } else {
+ ColumnValue v = values.get(col.getName());
+ if (v == null) {
+ line.addColumn(new StringColumn(null));
+ } else {
+ switch(v.getType()) {
+ case STRING: line.addColumn(new StringColumn(v.asString())); break;
+ case INTEGER: line.addColumn(new LongColumn(v.asLong())); break;
+ case DOUBLE: line.addColumn(new DoubleColumn(v.asDouble())); break;
+ case BOOLEAN: line.addColumn(new BoolColumn(v.asBoolean())); break;
+ case BINARY: line.addColumn(new BytesColumn(v.asBinary())); break;
+ default:
+ throw new IllegalArgumentException("Unsuporrt tranform the type: " + col.getValue().getType() + ".");
+ }
+ }
+ }
+ }
+ return line;
+ }
+
+ public static long getDelaySendMillinSeconds(int hadRetryTimes, int initSleepInMilliSecond) {
+
+ if (hadRetryTimes <= 0) {
+ return 0;
+ }
+
+ int sleepTime = initSleepInMilliSecond;
+ for (int i = 1; i < hadRetryTimes; i++) {
+ sleepTime += sleepTime;
+ if (sleepTime > 30000) {
+ sleepTime = 30000;
+ break;
+ }
+ }
+ return sleepTime;
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/CompareHelper.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/CompareHelper.java
new file mode 100644
index 00000000..19e06421
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/CompareHelper.java
@@ -0,0 +1,37 @@
+package com.alibaba.datax.plugin.reader.otsreader.utils;
+
+import com.alicloud.openservices.tablestore.model.PrimaryKeyColumn;
+
+import java.util.List;
+
+
+public class CompareHelper {
+ /**
+ * 比较PrimaryKeyColumn List的大小
+ * 返回
+ * -1 表示before小于after
+ * 0 表示before等于after
+ * 1 表示before大于after
+ *
+ * @param before
+ * @param after
+ * @return
+ */
+ public static int comparePrimaryKeyColumnList(List before, List after) {
+ int size = before.size() < after.size() ? before.size() : after.size();
+
+ for (int i = 0; i < size; i++) {
+ int cmp = before.get(i).compareTo(after.get(i));
+ if (cmp != 0) {
+ return cmp;
+ }
+ }
+
+ if (before.size() < after.size() ) {
+ return -1;
+ } else if (before.size() > after.size() ) {
+ return 1;
+ }
+ return 0;
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Constant.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Constant.java
new file mode 100644
index 00000000..90273bfb
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Constant.java
@@ -0,0 +1,92 @@
+package com.alibaba.datax.plugin.reader.otsreader.utils;
+
+public class Constant {
+ /**
+ * Json中的Key名字定义
+ */
+public class ConfigKey {
+ public static final String CONF = "conf";
+ public static final String RANGE = "range";
+ public static final String META = "meta";
+ public static final String SPLIT_INFO = "splitInfo";
+
+ public static final String TIME_RANGE = "timeRange";
+ public static final String MAX_VERSION = "maxVersion";
+
+ public static final String RETRY = "maxRetryTime";
+ public static final String RETRY_PAUSE_IN_MILLISECOND = "retryPauseInMillisecond";
+ public static final String IO_THREAD_COUNT = "ioThreadCount";
+ public static final String MAX_CONNECTION_COUNT = "maxConnectionCount";
+ public static final String SOCKET_TIMEOUTIN_MILLISECOND = "socketTimeoutInMillisecond";
+ public static final String CONNECT_TIMEOUT_IN_MILLISECOND = "connectTimeoutInMillisecond";
+
+ public class Range {
+ public static final String BEGIN = "begin";
+ public static final String END = "end";
+ public static final String SPLIT = "split";
+ };
+
+ public class PrimaryKeyColumn {
+ public static final String TYPE = "type";
+ public static final String VALUE = "value";
+ };
+
+ public class TimeseriesPKColumn {
+ public static final String MEASUREMENT_NAME = "_m_name";
+ public static final String DATA_SOURCE = "_data_source";
+ public static final String TAGS = "_tags";
+ public static final String TIME = "_time";
+ }
+
+ public class Column {
+ public static final String NAME = "name";
+ public static final String TYPE = "type";
+ public static final String VALUE = "value";
+ public static final String IS_TAG = "is_timeseries_tag";
+ };
+
+ public class TimeRange {
+ public static final String BEGIN = "begin";
+ public static final String END = "end";
+ }
+ };
+
+ /**
+ * 定义的配置文件中value type中可取的值
+ */
+ public class ValueType {
+ public static final String INF_MIN = "INF_MIN";
+ public static final String INF_MAX = "INF_MAX";
+ public static final String STRING = "string";
+ public static final String INTEGER = "int";
+ public static final String BINARY = "binary";
+ public static final String DOUBLE = "double";
+ public static final String BOOLEAN = "bool";
+ };
+
+ /**
+ * 全局默认常量定义
+ */
+ public class ConfigDefaultValue {
+ public static final int RETRY = 18;
+ public static final int RETRY_PAUSE_IN_MILLISECOND = 100;
+ public static final int IO_THREAD_COUNT = 1;
+ public static final int MAX_CONNECTION_COUNT = 1;
+ public static final int SOCKET_TIMEOUT_IN_MILLISECOND = 10000;
+ public static final int CONNECT_TIMEOUT_IN_MILLISECOND = 10000;
+
+ public static final int MAX_VERSION = Integer.MAX_VALUE;
+
+ public static final String DEFAULT_NAME = "DEFAULT_NAME";
+
+ public class Mode {
+ public static final String NORMAL = "normal";
+ public static final String MULTI_VERSION = "multiVersion";
+ }
+
+ public class TimeRange {
+ public static final long MIN = 0;
+ public static final long MAX = Long.MAX_VALUE;
+ }
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/GsonParser.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/GsonParser.java
index a82f3350..205f536d 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/GsonParser.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/GsonParser.java
@@ -1,23 +1,26 @@
package com.alibaba.datax.plugin.reader.otsreader.utils;
-import com.alibaba.datax.plugin.reader.otsreader.adaptor.OTSColumnAdaptor;
+import com.alibaba.datax.common.element.Column;
+import com.alibaba.datax.plugin.reader.otsreader.adaptor.ColumnAdaptor;
import com.alibaba.datax.plugin.reader.otsreader.adaptor.PrimaryKeyValueAdaptor;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSColumn;
import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
+import com.alicloud.openservices.tablestore.model.PrimaryKeyValue;
+import com.alicloud.openservices.tablestore.model.TableMeta;
+import com.alicloud.openservices.tablestore.model.timeseries.TimeseriesScanSplitInfo;
import com.aliyun.openservices.ots.model.Direction;
-import com.aliyun.openservices.ots.model.PrimaryKeyValue;
import com.aliyun.openservices.ots.model.RowPrimaryKey;
-import com.aliyun.openservices.ots.model.TableMeta;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
+import java.util.Map;
+
public class GsonParser {
private static Gson gsonBuilder() {
return new GsonBuilder()
- .registerTypeAdapter(OTSColumn.class, new OTSColumnAdaptor())
.registerTypeAdapter(PrimaryKeyValue.class, new PrimaryKeyValueAdaptor())
+ .registerTypeAdapter(Column.class, new ColumnAdaptor())
.create();
}
@@ -40,24 +43,39 @@ public class GsonParser {
Gson g = gsonBuilder();
return g.fromJson(jsonStr, OTSConf.class);
}
-
- public static String directionToJson (Direction direction) {
+
+ public static String metaToJson (TableMeta meta) {
Gson g = gsonBuilder();
- return g.toJson(direction);
+ return g.toJson(meta);
+ }
+
+ public static TableMeta jsonToMeta (String jsonStr) {
+ Gson g = gsonBuilder();
+ return g.fromJson(jsonStr, TableMeta.class);
+ }
+
+ public static String timeseriesScanSplitInfoToString(TimeseriesScanSplitInfo timeseriesScanSplitInfo){
+ Gson g = gsonBuilder();
+ return g.toJson(timeseriesScanSplitInfo);
+ }
+
+ public static TimeseriesScanSplitInfo stringToTimeseriesScanSplitInfo(String jsonStr){
+ Gson g = gsonBuilder();
+ return g.fromJson(jsonStr, TimeseriesScanSplitInfo.class);
}
public static Direction jsonToDirection (String jsonStr) {
Gson g = gsonBuilder();
return g.fromJson(jsonStr, Direction.class);
}
-
- public static String metaToJson (TableMeta meta) {
- Gson g = gsonBuilder();
- return g.toJson(meta);
- }
-
+
public static String rowPrimaryKeyToJson (RowPrimaryKey row) {
Gson g = gsonBuilder();
return g.toJson(row);
}
+
+ public static String mapToJson (Map map) {
+ Gson g = gsonBuilder();
+ return g.toJson(map);
+ }
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/Key.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Key.java
similarity index 81%
rename from otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/Key.java
rename to otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Key.java
index da6d4a5f..6628e4d3 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/Key.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/Key.java
@@ -14,7 +14,7 @@
* limitations under the License.
*/
-package com.alibaba.datax.plugin.reader.otsreader;
+package com.alibaba.datax.plugin.reader.otsreader.utils;
public final class Key {
/* ots account configuration */
@@ -46,5 +46,13 @@ public final class Key {
public final static String RANGE_END = "end";
public final static String RANGE_SPLIT = "split";
+
+ public final static String META_MODE = "metaMode";
+
+ public final static String MODE = "mode";
+ public final static String NEW_VERSION = "newVersion";
+
+ public final static String IS_TIMESERIES_TABLE = "isTimeseriesTable";
+ public final static String MEASUREMENT_NAME = "measurementName";
}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/OtsHelper.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/OtsHelper.java
new file mode 100644
index 00000000..060507b6
--- /dev/null
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/OtsHelper.java
@@ -0,0 +1,82 @@
+package com.alibaba.datax.plugin.reader.otsreader.utils;
+
+import com.alibaba.datax.plugin.reader.otsreader.callable.GetRangeCallable;
+import com.alibaba.datax.plugin.reader.otsreader.callable.GetTableMetaCallable;
+import com.alibaba.datax.plugin.reader.otsreader.callable.GetTimeseriesSplitCallable;
+import com.alibaba.datax.plugin.reader.otsreader.callable.ScanTimeseriesDataCallable;
+import com.alibaba.datax.plugin.reader.otsreader.model.DefaultNoRetry;
+import com.alibaba.datax.plugin.reader.otsreader.model.OTSConf;
+import com.alicloud.openservices.tablestore.ClientConfiguration;
+import com.alicloud.openservices.tablestore.SyncClient;
+import com.alicloud.openservices.tablestore.SyncClientInterface;
+import com.alicloud.openservices.tablestore.core.utils.Pair;
+import com.alicloud.openservices.tablestore.model.ColumnType;
+import com.alicloud.openservices.tablestore.model.GetRangeResponse;
+import com.alicloud.openservices.tablestore.model.RangeRowQueryCriteria;
+import com.alicloud.openservices.tablestore.model.TableMeta;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataRequest;
+import com.alicloud.openservices.tablestore.model.timeseries.ScanTimeseriesDataResponse;
+import com.alicloud.openservices.tablestore.model.timeseries.TimeseriesScanSplitInfo;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class OtsHelper {
+
+ public static SyncClientInterface getOTSInstance(OTSConf conf) {
+ ClientConfiguration clientConfigure = new ClientConfiguration();
+ clientConfigure.setIoThreadCount(conf.getIoThreadCount());
+ clientConfigure.setMaxConnections(conf.getMaxConnectCount());
+ clientConfigure.setSocketTimeoutInMillisecond(conf.getSocketTimeoutInMillisecond());
+ clientConfigure.setConnectionTimeoutInMillisecond(conf.getConnectTimeoutInMillisecond());
+ clientConfigure.setRetryStrategy(new DefaultNoRetry());
+
+ SyncClient ots = new SyncClient(
+ conf.getEndpoint(),
+ conf.getAccessId(),
+ conf.getAccessKey(),
+ conf.getInstanceName(),
+ clientConfigure);
+
+
+ Map extraHeaders = new HashMap();
+ extraHeaders.put("x-ots-sdk-type", "public");
+ extraHeaders.put("x-ots-request-source", "datax-otsreader");
+ ots.setExtraHeaders(extraHeaders);
+
+ return ots;
+ }
+
+ public static TableMeta getTableMeta(SyncClientInterface ots, String tableName, int retry, int sleepInMillisecond) throws Exception {
+ return RetryHelper.executeWithRetry(
+ new GetTableMetaCallable(ots, tableName),
+ retry,
+ sleepInMillisecond
+ );
+ }
+
+ public static GetRangeResponse getRange(SyncClientInterface ots, RangeRowQueryCriteria rangeRowQueryCriteria, int retry, int sleepInMillisecond) throws Exception {
+ return RetryHelper.executeWithRetry(
+ new GetRangeCallable(ots, rangeRowQueryCriteria),
+ retry,
+ sleepInMillisecond
+ );
+ }
+
+ public static List splitTimeseriesScan(SyncClientInterface ots, String tableName, String measurementName, int splitCountHint, int retry, int sleepInMillisecond) throws Exception {
+ return RetryHelper.executeWithRetry(
+ new GetTimeseriesSplitCallable(ots, tableName, measurementName, splitCountHint),
+ retry,
+ sleepInMillisecond
+ );
+ }
+
+ public static ScanTimeseriesDataResponse scanTimeseriesData(SyncClientInterface ots, ScanTimeseriesDataRequest scanTimeseriesDataRequest, int retry, int sleepInMillisecond) throws Exception {
+ return RetryHelper.executeWithRetry(
+ new ScanTimeseriesDataCallable(ots, scanTimeseriesDataRequest),
+ retry,
+ sleepInMillisecond
+ );
+ }
+}
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderError.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/OtsReaderError.java
similarity index 76%
rename from otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderError.java
rename to otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/OtsReaderError.java
index 05a13c1a..b578dcde 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/OtsReaderError.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/OtsReaderError.java
@@ -1,4 +1,4 @@
-package com.alibaba.datax.plugin.reader.otsreader;
+package com.alibaba.datax.plugin.reader.otsreader.utils;
import com.alibaba.datax.common.spi.ErrorCode;
@@ -14,10 +14,10 @@ public class OtsReaderError implements ErrorCode {
public final static OtsReaderError ERROR = new OtsReaderError(
"OtsReaderError",
- "该错误表示插件的内部错误，表示系统没有处理到的异常");
+ "This error represents an internal error of the otsreader plugin, which indicates that the system is not processed.");
public final static OtsReaderError INVALID_PARAM = new OtsReaderError(
"OtsReaderInvalidParameter",
- "该错误表示参数错误，表示用户输入了错误的参数格式等");
+ "This error represents a parameter error, indicating that the user entered the wrong parameter format.");
public OtsReaderError (String code) {
this.code = code;
diff --git a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/ParamChecker.java b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/ParamChecker.java
index fbcdc972..b2139fc1 100644
--- a/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/ParamChecker.java
+++ b/otsreader/src/main/java/com/alibaba/datax/plugin/reader/otsreader/utils/ParamChecker.java
@@ -1,162 +1,40 @@
package com.alibaba.datax.plugin.reader.otsreader.utils;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
+import com.alibaba.datax.common.element.Column;
import com.alibaba.datax.common.util.Configuration;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSPrimaryKeyColumn;
-import com.alibaba.datax.plugin.reader.otsreader.model.OTSRange;
-import com.aliyun.openservices.ots.model.Direction;
-import com.aliyun.openservices.ots.model.PrimaryKeyType;
-import com.aliyun.openservices.ots.model.PrimaryKeyValue;
-import com.aliyun.openservices.ots.model.RowPrimaryKey;
-import com.aliyun.openservices.ots.model.TableMeta;
+import com.alibaba.datax.plugin.reader.otsreader.model.*;
+import com.alicloud.openservices.tablestore.model.*;
+
+import java.util.*;
public class ParamChecker {
- private static void throwNotExistException(String key) {
- throw new IllegalArgumentException("The param '" + key + "' is not exist.");
+ private static void throwNotExistException() {
+ throw new IllegalArgumentException("missing the key.");
}
- private static void throwStringLengthZeroException(String key) {
- throw new IllegalArgumentException("The param length of '" + key + "' is zero.");
+ private static void throwStringLengthZeroException() {
+ throw new IllegalArgumentException("input the key is empty string.");
}
- private static void throwEmptyException(String key) {
- throw new IllegalArgumentException("The param '" + key + "' is empty.");
- }
-
- private static void throwNotListException(String key) {
- throw new IllegalArgumentException("The param '" + key + "' is not a json array.");
- }
-
- private static void throwNotMapException(String key) {
- throw new IllegalArgumentException("The param '" + key + "' is not a json map.");
- }
-
- public static String checkStringAndGet(Configuration param, String key) {
- String value = param.getString(key);
- if (null == value) {
- throwNotExistException(key);
- } else if (value.length() == 0) {
- throwStringLengthZeroException(key);
- }
- return value;
- }
-
- public static List