mirror of
https://github.com/alibaba/DataX.git
synced 2025-05-02 17:59:12 +08:00
Merge branch 'master' into starrocks_plugin
# Conflicts: # README.md # package.xml # pom.xml
This commit is contained in:
dingxiaobo
commit
10b7193f99
39
NOTICE
Normal file
39
NOTICE
Normal file
@ -0,0 +1,39 @@
|
|||||||
|
========================================================
|
||||||
|
DataX 是阿里云 DataWorks数据集成 的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
|
||||||
|
|
||||||
|
DataX is an open source offline data synchronization tool / platform widely used in Alibaba group and other companies. DataX implements efficient data synchronization between heterogeneous data sources including mysql, Oracle, oceanbase, sqlserver, postgre, HDFS, hive, ads, HBase, tablestore (OTS), maxcompute (ODPs), hologres, DRDS, etc.
|
||||||
|
|
||||||
|
Copyright 1999-2022 Alibaba Group Holding Ltd.
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
|
||||||
|
===================================================================
|
||||||
|
文级别引用，按许可证
|
||||||
|
This product contains various third-party components under other open source licenses.
|
||||||
|
This section summarizes those components and their licenses.
|
||||||
|
GNU Lesser General Public License
|
||||||
|
--------------------------------------
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/CliQuery.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/Connection4TSDB.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/DataPoint4TSDB.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/DumpSeries.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/OpenTSDBConnection.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/conn/OpenTSDBDump.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/Constant.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/Key.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/OpenTSDBReader.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/opentsdbreader/OpenTSDBReaderErrorCode.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/HttpUtils.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/TSDBUtils.java
|
||||||
|
opentsdbreader/src/main/java/com/alibaba/datax/plugin/reader/util/TimeUtils.java
|
||||||
|
===================================================================
|
||||||
157
README.md
157
README.md
@ -1,9 +1,154 @@
|
|||||||
# StarRocks writer for DataX
|
|
||||||
|
|
||||||
|
|
||||||
|
# DataX
|
||||||
|
|
||||||
|
DataX 是阿里云 [DataWorks数据集成](https://www.aliyun.com/product/bigdata/ide) 的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种异构数据源之间高效的数据同步功能。
|
||||||
|
|
||||||
|
# DataX 商业版本
|
||||||
|
阿里云DataWorks数据集成是DataX团队在阿里云上的商业化产品，致力于提供复杂网络环境下、丰富的异构数据源之间高速稳定的数据移动能力，以及繁杂业务背景下的数据同步解决方案。目前已经支持云上近3000家客户，单日同步数据超过3万亿条。DataWorks数据集成目前支持离线50+种数据源，可以进行整库迁移、批量上云、增量同步、分库分表等各类同步解决方案。2020年更新实时同步能力，2020年更新实时同步能力，支持10+种数据源的读写任意组合。提供MySQL，Oracle等多种数据源到阿里云MaxCompute，Hologres等大数据引擎的一键全增量同步解决方案。
|
||||||
|
|
||||||
|
商业版本参见： https://www.aliyun.com/product/bigdata/ide
|
||||||
|
|
||||||
|
|
||||||
|
# Features
|
||||||
|
|
||||||
|
DataX本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统, 每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# DataX详细介绍
|
||||||
|
|
||||||
|
##### 请参考：[DataX-Introduction](https://github.com/alibaba/DataX/blob/master/introduction.md)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Quick Start
|
||||||
|
|
||||||
|
##### Download [DataX下载地址](https://datax-opensource.oss-cn-hangzhou.aliyuncs.com/20220530/datax.tar.gz)
|
||||||
|
|
||||||
|
|
||||||
|
##### 请点击：[Quick Start](https://github.com/alibaba/DataX/blob/master/userGuid.md)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Support Data Channels
|
||||||
|
|
||||||
|
DataX目前已经有了比较全面的插件体系，主流的RDBMS数据库、NOSQL、大数据计算系统都已经接入，目前支持数据如下图，详情请点击：[DataX数据源参考指南](https://github.com/alibaba/DataX/wiki/DataX-all-data-channels)
|
||||||
|
|
||||||
|
| 类型 | 数据源 | Reader(读) | Writer(写) |文档|
|
||||||
|
| ------------ | ---------- | :-------: | :-------: |:-------: |
|
||||||
|
| RDBMS 关系型数据库 | MySQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md)|
|
||||||
|
| | Oracle | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/oraclereader/doc/oraclereader.md) 、[写](https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md)|
|
||||||
|
| | OceanBase | √ | √ |[读](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase) 、[写](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase)|
|
||||||
|
| | SQLServer | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/sqlserverreader/doc/sqlserverreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/sqlserverwriter/doc/sqlserverwriter.md)|
|
||||||
|
| | PostgreSQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/postgresqlwriter/doc/postgresqlwriter.md)|
|
||||||
|
| | DRDS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/drdsreader/doc/drdsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/drdswriter/doc/drdswriter.md)|
|
||||||
|
| | 通用RDBMS(支持所有关系型数据库) | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/rdbmsreader/doc/rdbmsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/rdbmswriter/doc/rdbmswriter.md)|
|
||||||
|
| 阿里云数仓数据存储 | ODPS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/odpsreader/doc/odpsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/odpswriter/doc/odpswriter.md)|
|
||||||
|
| | ADS | | √ |[写](https://github.com/alibaba/DataX/blob/master/adswriter/doc/adswriter.md)|
|
||||||
|
| | OSS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/ossreader/doc/ossreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/osswriter/doc/osswriter.md)|
|
||||||
|
| | OCS | | √ |[写](https://github.com/alibaba/DataX/blob/master/ocswriter/doc/ocswriter.md)|
|
||||||
|
| NoSQL数据存储 | OTS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/otsreader/doc/otsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/otswriter/doc/otswriter.md)|
|
||||||
|
| | Hbase0.94 | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase094xreader/doc/hbase094xreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase094xwriter/doc/hbase094xwriter.md)|
|
||||||
|
| | Hbase1.1 | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase11xreader/doc/hbase11xreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase11xwriter/doc/hbase11xwriter.md)|
|
||||||
|
| | Phoenix4.x | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase11xsqlreader/doc/hbase11xsqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase11xsqlwriter/doc/hbase11xsqlwriter.md)|
|
||||||
|
| | Phoenix5.x | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hbase20xsqlreader/doc/hbase20xsqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hbase20xsqlwriter/doc/hbase20xsqlwriter.md)|
|
||||||
|
| | MongoDB | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/mongodbreader/doc/mongodbreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mongodbwriter/doc/mongodbwriter.md)|
|
||||||
|
| | Hive | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hdfsreader/doc/hdfsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md)|
|
||||||
|
| | Cassandra | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/cassandrareader/doc/cassandrareader.md) 、[写](https://github.com/alibaba/DataX/blob/master/cassandrawriter/doc/cassandrawriter.md)|
|
||||||
|
| 无结构化数据存储 | TxtFile | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/txtfilereader/doc/txtfilereader.md) 、[写](https://github.com/alibaba/DataX/blob/master/txtfilewriter/doc/txtfilewriter.md)|
|
||||||
|
| | FTP | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/ftpreader/doc/ftpreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/ftpwriter/doc/ftpwriter.md)|
|
||||||
|
| | HDFS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/hdfsreader/doc/hdfsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md)|
|
||||||
|
| | Elasticsearch | | √ |[写](https://github.com/alibaba/DataX/blob/master/elasticsearchwriter/doc/elasticsearchwriter.md)|
|
||||||
|
| 时间序列数据库 | OpenTSDB | √ | |[读](https://github.com/alibaba/DataX/blob/master/opentsdbreader/doc/opentsdbreader.md)|
|
||||||
|
| | TSDB | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/tsdbreader/doc/tsdbreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/tsdbwriter/doc/tsdbhttpwriter.md)|
|
||||||
|
| | TDengine | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/tdenginereader/doc/tdenginereader-CN.md) 、[写](https://github.com/alibaba/DataX/blob/master/tdenginewriter/doc/tdenginewriter-CN.md)|
|
||||||
|
|
||||||
|
# 阿里云DataWorks数据集成
|
||||||
|
|
||||||
|
目前DataX的已有能力已经全部融和进阿里云的数据集成，并且比DataX更加高效、安全，同时数据集成具备DataX不具备的其它高级特性和功能。可以理解为数据集成是DataX的全面升级的商业化用版本，为企业可以提供稳定、可靠、安全的数据传输服务。与DataX相比，数据集成主要有以下几大突出特点：
|
||||||
|
|
||||||
|
支持实时同步：
|
||||||
|
|
||||||
|
- 功能简介：https://help.aliyun.com/document_detail/181912.html
|
||||||
|
- 支持的数据源：https://help.aliyun.com/document_detail/146778.html
|
||||||
|
- 支持数据处理：https://help.aliyun.com/document_detail/146777.html
|
||||||
|
|
||||||
|
离线同步数据源种类大幅度扩充：
|
||||||
|
|
||||||
|
- 新增比如：DB2、Kafka、Hologres、MetaQ、SAPHANA、达梦等等，持续扩充中
|
||||||
|
- 离线同步支持的数据源：https://help.aliyun.com/document_detail/137670.html
|
||||||
|
- 具备同步解决方案：
|
||||||
|
- 解决方案系统：https://help.aliyun.com/document_detail/171765.html
|
||||||
|
- 一键全增量：https://help.aliyun.com/document_detail/175676.html
|
||||||
|
- 整库迁移：https://help.aliyun.com/document_detail/137809.html
|
||||||
|
- 批量上云：https://help.aliyun.com/document_detail/146671.html
|
||||||
|
- 更新更多能力请访问：https://help.aliyun.com/document_detail/137663.html
|
||||||
|
|
||||||
|
|
||||||
|
# 我要开发新的插件
|
||||||
|
|
||||||
|
请点击：[DataX插件开发宝典](https://github.com/alibaba/DataX/blob/master/dataxPluginDev.md)
|
||||||
|
|
||||||
|
# 重要版本更新说明
|
||||||
|
|
||||||
|
DataX 后续计划月度迭代更新，也欢迎感兴趣的同学提交 Pull requests，月度更新内容会介绍介绍如下。
|
||||||
|
|
||||||
|
- [datax_v202205]（https://github.com/alibaba/DataX/releases/tag/datax_v202205)
|
||||||
|
- 涉及通道能力更新（MaxCompute、Hologres、OSS、Tdengine等）、安全漏洞更新、通用打包更新等
|
||||||
|
|
||||||
|
|
||||||
|
# 项目成员
|
||||||
|
|
||||||
|
核心Contributions: 言柏 、枕水、秋奇、青砾、一斅、云时
|
||||||
|
|
||||||
|
感谢天烬、光戈、祁然、巴真、静行对DataX做出的贡献。
|
||||||
|
|
||||||
|
# License
|
||||||
|
|
||||||
|
This software is free to use under the Apache License [Apache license](https://github.com/alibaba/DataX/blob/master/license.txt).
|
||||||
|
|
||||||
|
#
|
||||||
|
请及时提出issue给我们。请前往：[DataxIssue](https://github.com/alibaba/DataX/issues)
|
||||||
|
|
||||||
|
# 开源版DataX企业用户
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
```
|
||||||
|
长期招聘 联系邮箱：datax@alibabacloud.com
|
||||||
|
【JAVA开发职位】
|
||||||
|
职位名称：JAVA资深开发工程师/专家/高级专家
|
||||||
|
工作年限 : 2年以上
|
||||||
|
学历要求 : 本科（如果能力靠谱，这些都不是条件）
|
||||||
|
期望层级 : P6/P7/P8
|
||||||
|
|
||||||
|
岗位描述：
|
||||||
|
1. 负责阿里云大数据平台（数加）的开发设计。
|
||||||
|
2. 负责面向政企客户的大数据相关产品开发；
|
||||||
|
3. 利用大规模机器学习算法挖掘数据之间的联系，探索数据挖掘技术在实际场景中的产品应用 ；
|
||||||
|
4. 一站式大数据开发平台
|
||||||
|
5. 大数据任务调度引擎
|
||||||
|
6. 任务执行引擎
|
||||||
|
7. 任务监控告警
|
||||||
|
8. 海量异构数据同步
|
||||||
|
|
||||||
|
岗位要求：
|
||||||
|
1. 拥有3年以上JAVA Web开发经验；
|
||||||
|
2. 熟悉Java的基础技术体系。包括JVM、类装载、线程、并发、IO资源管理、网络；
|
||||||
|
3. 熟练使用常用Java技术框架、对新技术框架有敏锐感知能力；深刻理解面向对象、设计原则、封装抽象；
|
||||||
|
4. 熟悉HTML/HTML5和JavaScript；熟悉SQL语言；
|
||||||
|
5. 执行力强，具有优秀的团队合作精神、敬业精神；
|
||||||
|
6. 深刻理解设计模式及应用场景者加分；
|
||||||
|
7. 具有较强的问题分析和处理能力、比较强的动手能力，对技术有强烈追求者优先考虑；
|
||||||
|
8. 对高并发、高稳定可用性、高性能、大数据处理有过实际项目及产品经验者优先考虑；
|
||||||
|
9. 有大数据产品、云产品、中间件技术解决方案者优先考虑。
|
||||||
|
````
|
||||||
|
|
||||||
|
用户咨询支持：
|
||||||
|
|
||||||
|
钉钉群目前暂时受到了一些管控策略影响，建议大家有问题优先在这里提交问题 Issue，DataX研发和社区会定期回答Issue中的问题，知识库丰富后也能帮助到后来的使用者。
|
||||||
|
|
||||||
This is a repo forked from [DataX](https://github.com/alibaba/DataX), and maintained by [StarRocks](https://www.starrocks.com) (starrockswriter).
|
|
||||||
|
|
||||||
## How to use
|
|
||||||
|
|
||||||
1. Run `./build.sh` to gennerate the `starrockswriter.tar.gz`, then untar it into your own [DataX release](https://github.com/alibaba/DataX) directory(which will be `datax/plugin/writer/`).
|
|
||||||
2. Create a `job.json` to define the reader and writer. More details about the configurations, please refer to the [Documentations](https://docs.starrocks.com/en-us/main/loading/DataX-starrocks-writer).
|
|
||||||
3. Run `python datax/bin/datax.py --jvm="-Xms6G -Xmx6G" --loglevel=debug job.json` to start a job.
|
|
||||||
|
|||||||
@ -65,9 +65,9 @@ COPY命令将数据写入ADB PG数据库中。
|
|||||||
"writer": {
|
"writer": {
|
||||||
"name": "adbpgwriter",
|
"name": "adbpgwriter",
|
||||||
"parameter": {
|
"parameter": {
|
||||||
"username": "username",
|
"username": "",
|
||||||
"password": "password",
|
"password": "",
|
||||||
"host": "host",
|
"host": "127.0.0.1",
|
||||||
"port": "1234",
|
"port": "1234",
|
||||||
"database": "database",
|
"database": "database",
|
||||||
"schema": "schema",
|
"schema": "schema",
|
||||||
|
|||||||
@ -61,6 +61,14 @@
|
|||||||
</dependencies>
|
</dependencies>
|
||||||
|
|
||||||
<build>
|
<build>
|
||||||
|
<resources>
|
||||||
|
<resource>
|
||||||
|
<directory>src/main/java</directory>
|
||||||
|
<includes>
|
||||||
|
<include>**/*.properties</include>
|
||||||
|
</includes>
|
||||||
|
</resource>
|
||||||
|
</resources>
|
||||||
<plugins>
|
<plugins>
|
||||||
<plugin>
|
<plugin>
|
||||||
<artifactId>maven-compiler-plugin</artifactId>
|
<artifactId>maven-compiler-plugin</artifactId>
|
||||||
|
|||||||
@ -92,7 +92,13 @@ public class BoolColumn extends Column {
|
|||||||
throw DataXException.asDataXException(
|
throw DataXException.asDataXException(
|
||||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bool类型不能转为Date .");
|
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bool类型不能转为Date .");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Date asDate(String dateFormat) {
|
||||||
|
throw DataXException.asDataXException(
|
||||||
|
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bool类型不能转为Date .");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public byte[] asBytes() {
|
public byte[] asBytes() {
|
||||||
throw DataXException.asDataXException(
|
throw DataXException.asDataXException(
|
||||||
|
|||||||
@ -75,6 +75,12 @@ public class BytesColumn extends Column {
|
|||||||
throw DataXException.asDataXException(
|
throw DataXException.asDataXException(
|
||||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bytes类型不能转为Date .");
|
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bytes类型不能转为Date .");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Date asDate(String dateFormat) {
|
||||||
|
throw DataXException.asDataXException(
|
||||||
|
CommonErrorCode.CONVERT_NOT_SUPPORT, "Bytes类型不能转为Date .");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public Boolean asBoolean() {
|
public Boolean asBoolean() {
|
||||||
|
|||||||
@ -55,6 +55,8 @@ public abstract class Column {
|
|||||||
public abstract String asString();
|
public abstract String asString();
|
||||||
|
|
||||||
public abstract Date asDate();
|
public abstract Date asDate();
|
||||||
|
|
||||||
|
public abstract Date asDate(String dateFormat);
|
||||||
|
|
||||||
public abstract byte[] asBytes();
|
public abstract byte[] asBytes();
|
||||||
|
|
||||||
|
|||||||
@ -22,6 +22,11 @@ public final class ColumnCast {
|
|||||||
throws ParseException {
|
throws ParseException {
|
||||||
return StringCast.asDate(column);
|
return StringCast.asDate(column);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static Date string2Date(final StringColumn column, String dateFormat)
|
||||||
|
throws ParseException {
|
||||||
|
return StringCast.asDate(column, dateFormat);
|
||||||
|
}
|
||||||
|
|
||||||
public static byte[] string2Bytes(final StringColumn column)
|
public static byte[] string2Bytes(final StringColumn column)
|
||||||
throws UnsupportedEncodingException {
|
throws UnsupportedEncodingException {
|
||||||
@ -115,6 +120,16 @@ class StringCast {
|
|||||||
}
|
}
|
||||||
throw e;
|
throw e;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static Date asDate(final StringColumn column, String dateFormat) throws ParseException {
|
||||||
|
ParseException e;
|
||||||
|
try {
|
||||||
|
return FastDateFormat.getInstance(dateFormat, StringCast.timeZoner).parse(column.asString());
|
||||||
|
} catch (ParseException ignored) {
|
||||||
|
e = ignored;
|
||||||
|
}
|
||||||
|
throw e;
|
||||||
|
}
|
||||||
|
|
||||||
static byte[] asBytes(final StringColumn column)
|
static byte[] asBytes(final StringColumn column)
|
||||||
throws UnsupportedEncodingException {
|
throws UnsupportedEncodingException {
|
||||||
|
|||||||
@ -89,6 +89,11 @@ public class DateColumn extends Column {
|
|||||||
|
|
||||||
return new Date((Long)this.getRawData());
|
return new Date((Long)this.getRawData());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Date asDate(String dateFormat) {
|
||||||
|
return asDate();
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public byte[] asBytes() {
|
public byte[] asBytes() {
|
||||||
|
|||||||
@ -132,6 +132,12 @@ public class DoubleColumn extends Column {
|
|||||||
throw DataXException.asDataXException(
|
throw DataXException.asDataXException(
|
||||||
CommonErrorCode.CONVERT_NOT_SUPPORT, "Double类型无法转为Date类型 .");
|
CommonErrorCode.CONVERT_NOT_SUPPORT, "Double类型无法转为Date类型 .");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Date asDate(String dateFormat) {
|
||||||
|
throw DataXException.asDataXException(
|
||||||
|
CommonErrorCode.CONVERT_NOT_SUPPORT, "Double类型无法转为Date类型 .");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public byte[] asBytes() {
|
public byte[] asBytes() {
|
||||||
|
|||||||
@ -125,6 +125,11 @@ public class LongColumn extends Column {
|
|||||||
}
|
}
|
||||||
return new Date(this.asLong());
|
return new Date(this.asLong());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Date asDate(String dateFormat) {
|
||||||
|
return this.asDate();
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public byte[] asBytes() {
|
public byte[] asBytes() {
|
||||||
|
|||||||
@ -1,5 +1,7 @@
|
|||||||
package com.alibaba.datax.common.element;
|
package com.alibaba.datax.common.element;
|
||||||
|
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Created by jingxing on 14-8-24.
|
* Created by jingxing on 14-8-24.
|
||||||
*/
|
*/
|
||||||
@ -20,4 +22,8 @@ public interface Record {
|
|||||||
|
|
||||||
public int getMemorySize();
|
public int getMemorySize();
|
||||||
|
|
||||||
|
public void setMeta(Map<String, String> meta);
|
||||||
|
|
||||||
|
public Map<String, String> getMeta();
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -149,6 +149,16 @@ public class StringColumn extends Column {
|
|||||||
String.format("String[\"%s\"]不能转为Date .", this.asString()));
|
String.format("String[\"%s\"]不能转为Date .", this.asString()));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Date asDate(String dateFormat) {
|
||||||
|
try {
|
||||||
|
return ColumnCast.string2Date(this, dateFormat);
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw DataXException.asDataXException(CommonErrorCode.CONVERT_NOT_SUPPORT,
|
||||||
|
String.format("String[\"%s\"]不能转为Date .", this.asString()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public byte[] asBytes() {
|
public byte[] asBytes() {
|
||||||
|
|||||||
@ -16,6 +16,10 @@ public class DataXException extends RuntimeException {
|
|||||||
this.errorCode = errorCode;
|
this.errorCode = errorCode;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public DataXException(String errorMessage) {
|
||||||
|
super(errorMessage);
|
||||||
|
}
|
||||||
|
|
||||||
private DataXException(ErrorCode errorCode, String errorMessage, Throwable cause) {
|
private DataXException(ErrorCode errorCode, String errorMessage, Throwable cause) {
|
||||||
super(errorCode.toString() + " - " + getMessage(errorMessage) + " - " + getMessage(cause), cause);
|
super(errorCode.toString() + " - " + getMessage(errorMessage) + " - " + getMessage(cause), cause);
|
||||||
|
|
||||||
@ -26,6 +30,10 @@ public class DataXException extends RuntimeException {
|
|||||||
return new DataXException(errorCode, message);
|
return new DataXException(errorCode, message);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static DataXException asDataXException(String message) {
|
||||||
|
return new DataXException(message);
|
||||||
|
}
|
||||||
|
|
||||||
public static DataXException asDataXException(ErrorCode errorCode, String message, Throwable cause) {
|
public static DataXException asDataXException(ErrorCode errorCode, String message, Throwable cause) {
|
||||||
if (cause instanceof DataXException) {
|
if (cause instanceof DataXException) {
|
||||||
return (DataXException) cause;
|
return (DataXException) cause;
|
||||||
|
|||||||
@ -3,6 +3,8 @@ package com.alibaba.datax.common.plugin;
|
|||||||
import com.alibaba.datax.common.base.BaseObject;
|
import com.alibaba.datax.common.base.BaseObject;
|
||||||
import com.alibaba.datax.common.util.Configuration;
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
public abstract class AbstractPlugin extends BaseObject implements Pluginable {
|
public abstract class AbstractPlugin extends BaseObject implements Pluginable {
|
||||||
//作业的config
|
//作业的config
|
||||||
private Configuration pluginJobConf;
|
private Configuration pluginJobConf;
|
||||||
@ -15,6 +17,8 @@ public abstract class AbstractPlugin extends BaseObject implements Pluginable {
|
|||||||
|
|
||||||
private String peerPluginName;
|
private String peerPluginName;
|
||||||
|
|
||||||
|
private List<Configuration> readerPluginSplitConf;
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public String getPluginName() {
|
public String getPluginName() {
|
||||||
assert null != this.pluginConf;
|
assert null != this.pluginConf;
|
||||||
@ -84,4 +88,12 @@ public abstract class AbstractPlugin extends BaseObject implements Pluginable {
|
|||||||
public void postHandler(Configuration jobConfiguration){
|
public void postHandler(Configuration jobConfiguration){
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public List<Configuration> getReaderPluginSplitConf(){
|
||||||
|
return this.readerPluginSplitConf;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setReaderPluginSplitConf(List<Configuration> readerPluginSplitConf){
|
||||||
|
this.readerPluginSplitConf = readerPluginSplitConf;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -0,0 +1,37 @@
|
|||||||
|
package com.alibaba.datax.common.util;
|
||||||
|
|
||||||
|
import java.util.Arrays;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Set;
|
||||||
|
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
|
||||||
|
public class ConfigurationUtil {
|
||||||
|
private static final List<String> SENSITIVE_KEYS = Arrays.asList("password", "accessKey", "securityToken",
|
||||||
|
"AccessKeyId", "AccessKeySecert", "AccessKeySecret", "clientPassword");
|
||||||
|
|
||||||
|
public static Configuration filterSensitive(Configuration origin) {
|
||||||
|
// shell 任务configuration metric 可能为null。
|
||||||
|
if (origin == null) {
|
||||||
|
return origin;
|
||||||
|
}
|
||||||
|
// 确保不影响入参的对象
|
||||||
|
Configuration configuration = origin.clone();
|
||||||
|
Set<String> keys = configuration.getKeys();
|
||||||
|
for (final String key : keys) {
|
||||||
|
boolean isSensitive = false;
|
||||||
|
for (String sensitiveKey : SENSITIVE_KEYS) {
|
||||||
|
if (StringUtils.endsWithIgnoreCase(key, sensitiveKey)) {
|
||||||
|
isSensitive = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isSensitive && configuration.get(key) instanceof String) {
|
||||||
|
configuration.set(key, configuration.getString(key).replaceAll(".", "*"));
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
return configuration;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,5 +1,5 @@
|
|||||||
/**
|
/**
|
||||||
* (C) 2010-2014 Alibaba Group Holding Limited.
|
* (C) 2010-2022 Alibaba Group Holding Limited.
|
||||||
*
|
*
|
||||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
* you may not use this file except in compliance with the License.
|
* you may not use this file except in compliance with the License.
|
||||||
@ -14,342 +14,216 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package com.alibaba.datax.plugin.reader.odpsreader.util;
|
package com.alibaba.datax.common.util;
|
||||||
|
|
||||||
import javax.crypto.Cipher;
|
import javax.crypto.Cipher;
|
||||||
import javax.crypto.SecretKey;
|
import javax.crypto.SecretKey;
|
||||||
import javax.crypto.SecretKeyFactory;
|
import javax.crypto.SecretKeyFactory;
|
||||||
import javax.crypto.spec.DESKeySpec;
|
import javax.crypto.spec.DESKeySpec;
|
||||||
|
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
import java.security.SecureRandom;
|
import java.security.SecureRandom;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * DES加解密,支持与delphi交互(字符串编码需统一为UTF-8)
|
* DES加解密,支持与delphi交互(字符串编码需统一为UTF-8)
|
||||||
*
|
* 将这个工具类抽取到 common 中，方便后续代码复用
|
||||||
* *
|
|
||||||
*
|
|
||||||
* * @author wym
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public class DESCipher {
|
public class DESCipher {
|
||||||
|
private static Logger LOGGER = LoggerFactory.getLogger(DESCipher.class);
|
||||||
/**
|
/**
|
||||||
* * 密钥
|
* 密钥
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
public static final String KEY = "";
|
||||||
public static final String KEY = "u4Gqu4Z8";
|
|
||||||
|
|
||||||
private final static String DES = "DES";
|
private final static String DES = "DES";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 加密
|
* 加密
|
||||||
*
|
* @param src 明文(字节)
|
||||||
* *
|
* @param key 密钥，长度必须是8的倍数
|
||||||
*
|
* @return 密文(字节)
|
||||||
* * @param src
|
* @throws Exception
|
||||||
*
|
|
||||||
* * 明文(字节)
|
|
||||||
*
|
|
||||||
* * @param key
|
|
||||||
*
|
|
||||||
* * 密钥，长度必须是8的倍数
|
|
||||||
*
|
|
||||||
* * @return 密文(字节)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public static byte[] encrypt(byte[] src, byte[] key) throws Exception {
|
public static byte[] encrypt(byte[] src, byte[] key) throws Exception {
|
||||||
|
|
||||||
// DES算法要求有一个可信任的随机数源
|
// DES算法要求有一个可信任的随机数源
|
||||||
|
|
||||||
SecureRandom sr = new SecureRandom();
|
SecureRandom sr = new SecureRandom();
|
||||||
|
|
||||||
// 从原始密匙数据创建DESKeySpec对象
|
// 从原始密匙数据创建DESKeySpec对象
|
||||||
|
|
||||||
DESKeySpec dks = new DESKeySpec(key);
|
DESKeySpec dks = new DESKeySpec(key);
|
||||||
|
|
||||||
// 创建一个密匙工厂，然后用它把DESKeySpec转换成
|
// 创建一个密匙工厂，然后用它把DESKeySpec转换成
|
||||||
|
|
||||||
// 一个SecretKey对象
|
// 一个SecretKey对象
|
||||||
|
|
||||||
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(DES);
|
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(DES);
|
||||||
|
|
||||||
SecretKey securekey = keyFactory.generateSecret(dks);
|
SecretKey securekey = keyFactory.generateSecret(dks);
|
||||||
|
|
||||||
// Cipher对象实际完成加密操作
|
// Cipher对象实际完成加密操作
|
||||||
|
|
||||||
Cipher cipher = Cipher.getInstance(DES);
|
Cipher cipher = Cipher.getInstance(DES);
|
||||||
|
|
||||||
// 用密匙初始化Cipher对象
|
// 用密匙初始化Cipher对象
|
||||||
|
|
||||||
cipher.init(Cipher.ENCRYPT_MODE, securekey, sr);
|
cipher.init(Cipher.ENCRYPT_MODE, securekey, sr);
|
||||||
|
|
||||||
// 现在，获取数据并加密
|
// 现在，获取数据并加密
|
||||||
|
|
||||||
// 正式执行加密操作
|
// 正式执行加密操作
|
||||||
|
|
||||||
return cipher.doFinal(src);
|
return cipher.doFinal(src);
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 解密
|
* * 解密
|
||||||
*
|
* * @param src
|
||||||
* *
|
* * 密文(字节)
|
||||||
*
|
* * @param key
|
||||||
* * @param src
|
* * 密钥，长度必须是8的倍数
|
||||||
*
|
* * @return 明文(字节)
|
||||||
* * 密文(字节)
|
* * @throws Exception
|
||||||
*
|
|
||||||
* * @param key
|
|
||||||
*
|
|
||||||
* * 密钥，长度必须是8的倍数
|
|
||||||
*
|
|
||||||
* * @return 明文(字节)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public static byte[] decrypt(byte[] src, byte[] key) throws Exception {
|
public static byte[] decrypt(byte[] src, byte[] key) throws Exception {
|
||||||
|
|
||||||
// DES算法要求有一个可信任的随机数源
|
// DES算法要求有一个可信任的随机数源
|
||||||
|
|
||||||
SecureRandom sr = new SecureRandom();
|
SecureRandom sr = new SecureRandom();
|
||||||
|
|
||||||
// 从原始密匙数据创建一个DESKeySpec对象
|
// 从原始密匙数据创建一个DESKeySpec对象
|
||||||
|
|
||||||
DESKeySpec dks = new DESKeySpec(key);
|
DESKeySpec dks = new DESKeySpec(key);
|
||||||
|
|
||||||
// 创建一个密匙工厂，然后用它把DESKeySpec对象转换成
|
// 创建一个密匙工厂，然后用它把DESKeySpec对象转换成
|
||||||
|
|
||||||
// 一个SecretKey对象
|
// 一个SecretKey对象
|
||||||
|
|
||||||
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(DES);
|
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(DES);
|
||||||
|
|
||||||
SecretKey securekey = keyFactory.generateSecret(dks);
|
SecretKey securekey = keyFactory.generateSecret(dks);
|
||||||
|
|
||||||
// Cipher对象实际完成解密操作
|
// Cipher对象实际完成解密操作
|
||||||
|
|
||||||
Cipher cipher = Cipher.getInstance(DES);
|
Cipher cipher = Cipher.getInstance(DES);
|
||||||
|
|
||||||
// 用密匙初始化Cipher对象
|
// 用密匙初始化Cipher对象
|
||||||
|
|
||||||
cipher.init(Cipher.DECRYPT_MODE, securekey, sr);
|
cipher.init(Cipher.DECRYPT_MODE, securekey, sr);
|
||||||
|
|
||||||
// 现在，获取数据并解密
|
// 现在，获取数据并解密
|
||||||
|
|
||||||
// 正式执行解密操作
|
// 正式执行解密操作
|
||||||
|
|
||||||
return cipher.doFinal(src);
|
return cipher.doFinal(src);
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 加密
|
* 加密
|
||||||
*
|
* @param src * 明文(字节)
|
||||||
* *
|
* @return 密文(字节)
|
||||||
*
|
* @throws Exception
|
||||||
* * @param src
|
|
||||||
*
|
|
||||||
* * 明文(字节)
|
|
||||||
*
|
|
||||||
* * @return 密文(字节)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public static byte[] encrypt(byte[] src) throws Exception {
|
public static byte[] encrypt(byte[] src) throws Exception {
|
||||||
|
|
||||||
return encrypt(src, KEY.getBytes());
|
return encrypt(src, KEY.getBytes());
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 解密
|
* 解密
|
||||||
*
|
* @param src 密文(字节)
|
||||||
* *
|
* @return 明文(字节)
|
||||||
*
|
* @throws Exception
|
||||||
* * @param src
|
|
||||||
*
|
|
||||||
* * 密文(字节)
|
|
||||||
*
|
|
||||||
* * @return 明文(字节)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public static byte[] decrypt(byte[] src) throws Exception {
|
public static byte[] decrypt(byte[] src) throws Exception {
|
||||||
|
|
||||||
return decrypt(src, KEY.getBytes());
|
return decrypt(src, KEY.getBytes());
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 加密
|
* 加密
|
||||||
*
|
* @param src 明文(字符串)
|
||||||
* *
|
* @return 密文(16进制字符串)
|
||||||
*
|
* @throws Exception
|
||||||
* * @param src
|
|
||||||
*
|
|
||||||
* * 明文(字符串)
|
|
||||||
*
|
|
||||||
* * @return 密文(16进制字符串)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public final static String encrypt(String src) {
|
public final static String encrypt(String src) {
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
|
||||||
return byte2hex(encrypt(src.getBytes(), KEY.getBytes()));
|
return byte2hex(encrypt(src.getBytes(), KEY.getBytes()));
|
||||||
|
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
|
LOGGER.warn(e.getMessage(), e);
|
||||||
e.printStackTrace();
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 加密
|
||||||
|
* @param src 明文(字符串)
|
||||||
|
* @param encryptKey 加密用的秘钥
|
||||||
|
* @return 密文(16进制字符串)
|
||||||
|
* @throws Exception
|
||||||
|
*/
|
||||||
|
public final static String encrypt(String src, String encryptKey) {
|
||||||
|
try {
|
||||||
|
return byte2hex(encrypt(src.getBytes(), encryptKey.getBytes()));
|
||||||
|
} catch (Exception e) {
|
||||||
|
LOGGER.warn(e.getMessage(), e);
|
||||||
}
|
}
|
||||||
|
|
||||||
return null;
|
return null;
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 解密
|
* 解密
|
||||||
*
|
* @param src 密文(字符串)
|
||||||
* *
|
* @return 明文(字符串)
|
||||||
*
|
* @throws Exception
|
||||||
* * @param src
|
|
||||||
*
|
|
||||||
* * 密文(字符串)
|
|
||||||
*
|
|
||||||
* * @return 明文(字符串)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public final static String decrypt(String src) {
|
public final static String decrypt(String src) {
|
||||||
try {
|
try {
|
||||||
|
|
||||||
return new String(decrypt(hex2byte(src.getBytes()), KEY.getBytes()));
|
return new String(decrypt(hex2byte(src.getBytes()), KEY.getBytes()));
|
||||||
|
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
|
LOGGER.warn(e.getMessage(), e);
|
||||||
e.printStackTrace();
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 解密
|
||||||
|
* @param src 密文(字符串)
|
||||||
|
* @param decryptKey 解密用的秘钥
|
||||||
|
* @return 明文(字符串)
|
||||||
|
* @throws Exception
|
||||||
|
*/
|
||||||
|
public final static String decrypt(String src, String decryptKey) {
|
||||||
|
try {
|
||||||
|
return new String(decrypt(hex2byte(src.getBytes()), decryptKey.getBytes()));
|
||||||
|
} catch (Exception e) {
|
||||||
|
LOGGER.warn(e.getMessage(), e);
|
||||||
}
|
}
|
||||||
|
|
||||||
return null;
|
return null;
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 加密
|
* 加密
|
||||||
*
|
* @param src
|
||||||
* *
|
* 明文(字节)
|
||||||
*
|
* @return 密文(16进制字符串)
|
||||||
* * @param src
|
* @throws Exception
|
||||||
*
|
|
||||||
* * 明文(字节)
|
|
||||||
*
|
|
||||||
* * @return 密文(16进制字符串)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public static String encryptToString(byte[] src) throws Exception {
|
public static String encryptToString(byte[] src) throws Exception {
|
||||||
|
|
||||||
return encrypt(new String(src));
|
return encrypt(new String(src));
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* * 解密
|
* 解密
|
||||||
*
|
* @param src 密文(字节)
|
||||||
* *
|
* @return 明文(字符串)
|
||||||
*
|
* @throws Exception
|
||||||
* * @param src
|
|
||||||
*
|
|
||||||
* * 密文(字节)
|
|
||||||
*
|
|
||||||
* * @return 明文(字符串)
|
|
||||||
*
|
|
||||||
* * @throws Exception
|
|
||||||
*
|
|
||||||
*
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
public static String decryptToString(byte[] src) throws Exception {
|
public static String decryptToString(byte[] src) throws Exception {
|
||||||
|
|
||||||
return decrypt(new String(src));
|
return decrypt(new String(src));
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public static String byte2hex(byte[] b) {
|
public static String byte2hex(byte[] b) {
|
||||||
|
|
||||||
String hs = "";
|
String hs = "";
|
||||||
|
|
||||||
String stmp = "";
|
String stmp = "";
|
||||||
|
|
||||||
for (int n = 0; n < b.length; n++) {
|
for (int n = 0; n < b.length; n++) {
|
||||||
|
|
||||||
stmp = (Integer.toHexString(b[n] & 0XFF));
|
stmp = (Integer.toHexString(b[n] & 0XFF));
|
||||||
|
|
||||||
if (stmp.length() == 1)
|
if (stmp.length() == 1)
|
||||||
|
|
||||||
hs = hs + "0" + stmp;
|
hs = hs + "0" + stmp;
|
||||||
|
|
||||||
else
|
else
|
||||||
|
|
||||||
hs = hs + stmp;
|
hs = hs + stmp;
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return hs.toUpperCase();
|
return hs.toUpperCase();
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public static byte[] hex2byte(byte[] b) {
|
public static byte[] hex2byte(byte[] b) {
|
||||||
|
|
||||||
if ((b.length % 2) != 0)
|
if ((b.length % 2) != 0)
|
||||||
|
throw new IllegalArgumentException("The length is not an even number");
|
||||||
throw new IllegalArgumentException("长度不是偶数");
|
|
||||||
|
|
||||||
byte[] b2 = new byte[b.length / 2];
|
byte[] b2 = new byte[b.length / 2];
|
||||||
|
|
||||||
for (int n = 0; n < b.length; n += 2) {
|
for (int n = 0; n < b.length; n += 2) {
|
||||||
|
|
||||||
String item = new String(b, n, 2);
|
String item = new String(b, n, 2);
|
||||||
|
|
||||||
b2[n / 2] = (byte) Integer.parseInt(item, 16);
|
b2[n / 2] = (byte) Integer.parseInt(item, 16);
|
||||||
|
|
||||||
}
|
}
|
||||||
return b2;
|
return b2;
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/*
|
|
||||||
* public static void main(String[] args) { try { String src = "cheetah";
|
|
||||||
* String crypto = DESCipher.encrypt(src); System.out.println("密文[" + src +
|
|
||||||
* "]:" + crypto); System.out.println("解密后:" + DESCipher.decrypt(crypto)); }
|
|
||||||
* catch (Exception e) { e.printStackTrace(); } }
|
|
||||||
*/
|
|
||||||
}
|
}
|
||||||
@ -0,0 +1,33 @@
|
|||||||
|
package com.alibaba.datax.common.util;
|
||||||
|
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
|
||||||
|
public class DataXCaseEnvUtil {
|
||||||
|
|
||||||
|
private static final Logger LOGGER = LoggerFactory.getLogger(DataXCaseEnvUtil.class);
|
||||||
|
|
||||||
|
// datax回归测试效率提升
|
||||||
|
private static String DATAX_AUTOTEST_RETRY_TIME = System.getenv("DATAX_AUTOTEST_RETRY_TIME");
|
||||||
|
private static String DATAX_AUTOTEST_RETRY_INTERVAL = System.getenv("DATAX_AUTOTEST_RETRY_INTERVAL");
|
||||||
|
private static String DATAX_AUTOTEST_RETRY_EXPONENTIAL = System.getenv("DATAX_AUTOTEST_RETRY_EXPONENTIAL");
|
||||||
|
|
||||||
|
public static int getRetryTimes(int retryTimes) {
|
||||||
|
int actualRetryTimes = DATAX_AUTOTEST_RETRY_TIME != null ? Integer.valueOf(DATAX_AUTOTEST_RETRY_TIME) : retryTimes;
|
||||||
|
// LOGGER.info("The actualRetryTimes is {}", actualRetryTimes);
|
||||||
|
return actualRetryTimes;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static long getRetryInterval(long retryInterval) {
|
||||||
|
long actualRetryInterval = DATAX_AUTOTEST_RETRY_INTERVAL != null ? Long.valueOf(DATAX_AUTOTEST_RETRY_INTERVAL) : retryInterval;
|
||||||
|
// LOGGER.info("The actualRetryInterval is {}", actualRetryInterval);
|
||||||
|
return actualRetryInterval;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static boolean getRetryExponential(boolean retryExponential) {
|
||||||
|
boolean actualRetryExponential = DATAX_AUTOTEST_RETRY_EXPONENTIAL != null ? Boolean.valueOf(DATAX_AUTOTEST_RETRY_EXPONENTIAL) : retryExponential;
|
||||||
|
// LOGGER.info("The actualRetryExponential is {}", actualRetryExponential);
|
||||||
|
return actualRetryExponential;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,62 @@
|
|||||||
|
package com.alibaba.datax.common.util;
|
||||||
|
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.exception.DataXException;
|
||||||
|
|
||||||
|
public class IdAndKeyRollingUtil {
|
||||||
|
private static Logger LOGGER = LoggerFactory.getLogger(IdAndKeyRollingUtil.class);
|
||||||
|
public static final String SKYNET_ACCESSID = "SKYNET_ACCESSID";
|
||||||
|
public static final String SKYNET_ACCESSKEY = "SKYNET_ACCESSKEY";
|
||||||
|
|
||||||
|
public final static String ACCESS_ID = "accessId";
|
||||||
|
public final static String ACCESS_KEY = "accessKey";
|
||||||
|
|
||||||
|
public static String parseAkFromSkynetAccessKey() {
|
||||||
|
Map<String, String> envProp = System.getenv();
|
||||||
|
String skynetAccessID = envProp.get(IdAndKeyRollingUtil.SKYNET_ACCESSID);
|
||||||
|
String skynetAccessKey = envProp.get(IdAndKeyRollingUtil.SKYNET_ACCESSKEY);
|
||||||
|
String accessKey = null;
|
||||||
|
// follow 原有的判断条件
|
||||||
|
// 环境变量中，如果存在SKYNET_ACCESSID/SKYNET_ACCESSKEy（只要有其中一个变量，则认为一定是两个都存在的！
|
||||||
|
// if (StringUtils.isNotBlank(skynetAccessID) ||
|
||||||
|
// StringUtils.isNotBlank(skynetAccessKey)) {
|
||||||
|
// 检查严格，只有加密串不为空的时候才进去，不过 之前能跑的加密串都不应该为空
|
||||||
|
if (StringUtils.isNotBlank(skynetAccessKey)) {
|
||||||
|
LOGGER.info("Try to get accessId/accessKey from environment SKYNET_ACCESSKEY.");
|
||||||
|
accessKey = DESCipher.decrypt(skynetAccessKey);
|
||||||
|
if (StringUtils.isBlank(accessKey)) {
|
||||||
|
// 环境变量里面有，但是解析不到
|
||||||
|
throw DataXException.asDataXException(String.format(
|
||||||
|
"Failed to get the [accessId]/[accessKey] from the environment variable. The [accessId]=[%s]",
|
||||||
|
skynetAccessID));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (StringUtils.isNotBlank(accessKey)) {
|
||||||
|
LOGGER.info("Get accessId/accessKey from environment variables SKYNET_ACCESSKEY successfully.");
|
||||||
|
}
|
||||||
|
return accessKey;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static String getAccessIdAndKeyFromEnv(Configuration originalConfig) {
|
||||||
|
String accessId = null;
|
||||||
|
Map<String, String> envProp = System.getenv();
|
||||||
|
accessId = envProp.get(IdAndKeyRollingUtil.SKYNET_ACCESSID);
|
||||||
|
String accessKey = null;
|
||||||
|
if (StringUtils.isBlank(accessKey)) {
|
||||||
|
// 老的没有出异常，只是获取不到ak
|
||||||
|
accessKey = IdAndKeyRollingUtil.parseAkFromSkynetAccessKey();
|
||||||
|
}
|
||||||
|
|
||||||
|
if (StringUtils.isNotBlank(accessKey)) {
|
||||||
|
// 确认使用这个的都是 accessId、accessKey的命名习惯
|
||||||
|
originalConfig.set(IdAndKeyRollingUtil.ACCESS_ID, accessId);
|
||||||
|
originalConfig.set(IdAndKeyRollingUtil.ACCESS_KEY, accessKey);
|
||||||
|
}
|
||||||
|
return accessKey;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -6,6 +6,7 @@ import org.apache.commons.lang3.StringUtils;
|
|||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.Collections;
|
import java.util.Collections;
|
||||||
|
import java.util.HashSet;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@ -136,4 +137,25 @@ public final class ListUtil {
|
|||||||
|
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static Boolean checkIfHasSameValue(List<String> listA, List<String> listB) {
|
||||||
|
if (null == listA || listA.isEmpty() || null == listB || listB.isEmpty()) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (String oneValue : listA) {
|
||||||
|
if (listB.contains(oneValue)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static boolean checkIfAllSameValue(List<String> listA, List<String> listB) {
|
||||||
|
if (null == listA || listA.isEmpty() || null == listB || listB.isEmpty() || listA.size() != listB.size()) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return new HashSet<>(listA).containsAll(new HashSet<>(listB));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -0,0 +1,54 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||||
|
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||||
|
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||||
|
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||||
|
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||||
|
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||||
|
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||||
|
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||||
|
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||||
|
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||||
|
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||||
|
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||||
|
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||||
|
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||||
|
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1}, STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
@ -0,0 +1,53 @@
|
|||||||
|
very_like_yixiao=1{0}2{1}3
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=Configuration information error. The configuration file [{0}] you provided does not exist. Please check your configuration files.
|
||||||
|
configuration.2=Configuration information error. Failed to read the configuration file [{0}] you provided. Error reason: {1}. Please check the permission settings of your configuration files.
|
||||||
|
configuration.3=Please check your configuration files. Failed to read the configuration file you provided. Error reason: {0}. Please check the permission settings of your configuration files.
|
||||||
|
configuration.4=The configuration file you provided contains errors. [{0}] is a required parameter and cannot be empty or blank.
|
||||||
|
configuration.5=The configuration file you provided contains errors. [{0}] is a required parameter and cannot be empty or blank.
|
||||||
|
configuration.6=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the character type: {1}. Please check your configuration and make corrections.
|
||||||
|
configuration.7=The configuration information you provided contains errors. The value [{1}] obtained from [{0}] cannot be converted to the Bool type. Please check the source table configuration and make corrections.
|
||||||
|
configuration.8=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the integer type: {1}. Please check your configuration and make corrections.
|
||||||
|
configuration.9=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the integer type: {1}. Please check your configuration and make corrections.
|
||||||
|
configuration.10=Task reading configuration file error. Invalid configuration file path [{0}] value. The expected value should be of the floating-point type: {1}. Please check your configuration and make corrections.
|
||||||
|
configuration.11=The Key [{0}] for the configuration file does not exist. This is a code programming error. Please contact the DataX team.
|
||||||
|
configuration.12=The value [{0}] cannot adapt to the [{1}] you provided. This exception represents a system programming error. Please contact the DataX developer team.
|
||||||
|
configuration.13=The path cannot be null. This exception represents a system programming error. Please contact the DataX developer team.
|
||||||
|
configuration.14=The path [{0}] has an invalid value type [{1}]. This exception represents a system programming error. Please contact the DataX developer team.
|
||||||
|
configuration.15=This exception represents a system programming error. Please contact the DataX developer team.
|
||||||
|
configuration.16=The configuration file you provided contains errors. The path [{0}] requires you to configure a Map object in JSON format, but the actual type found on the node is [{1}]. Please check your configuration and make corrections.
|
||||||
|
configuration.17=The configuration file you provided contains errors. The value of the path [{0}] is null and DataX cannot recognize the configuration. Please check your configuration and make corrections.
|
||||||
|
configuration.18=The configuration file you provided contains errors. The path [{0}] requires you to configure a Map object in JSON format, but the actual type found on the node is [{1}]. Please check your configuration and make corrections.
|
||||||
|
configuration.19=System programming error. The list subscript must be of the numeric type, but the actual type found on this node is [{0}]. This exception represents a system programming error. Please contact the DataX developer team.
|
||||||
|
configuration.20=System programming error. This exception represents a system programming error. Please contact the DataX developer team.
|
||||||
|
configuration.21=System programming error. Invalid path [{0}]. No spaces are allowed between path layers.
|
||||||
|
configuration.22=Configuration information error. The configuration information you provided is not in a legal JSON format. JSON cannot be blank. Please provide the configuration information in the standard JSON format.
|
||||||
|
configuration.23=Configuration information error. The configuration information you provided is not in a valid JSON format: {0}. Please provide the configuration information in the standard JSON format.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=The job configuration you provided contains errors. The list cannot be empty.
|
||||||
|
listutil.2=The job configuration you provided contains errors. The list cannot be empty.
|
||||||
|
listutil.3=The job configuration information you provided contains errors. String: [{0}] is not allowed to be repeated in the list: [{1}].
|
||||||
|
listutil.4=The job configuration you provided contains errors. The list cannot be empty.
|
||||||
|
listutil.5=The job configuration you provided contains errors. The list cannot be empty.
|
||||||
|
listutil.6=The job configuration information you provided contains errors. String: [{0}] does not exist in the list: [{1}].
|
||||||
|
listutil.7=The job configuration you provided contains errors. The list cannot be empty.
|
||||||
|
listutil.8=The job configuration you provided contains errors. The list cannot be empty.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=The slice number cannot be less than 1. Here: [expectSliceNumber]=[{0}].
|
||||||
|
rangesplitutil.2=The left or right intervals of BigInteger character strings cannot be null when they are sliced. Here: [left]=[{0}], [right]=[{1}].
|
||||||
|
rangesplitutil.3=The [bigInteger] parameter cannot be null.
|
||||||
|
rangesplitutil.4=Only ASCII character strings are supported for character string slicing, but the [{0}] character string is not of the ASCII type.
|
||||||
|
rangesplitutil.5=The [bigInteger] parameter cannot be null.
|
||||||
|
rangesplitutil.6=Only ASCII character strings are supported for character string slicing, but the [{0}] character string is not of the ASCII type.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=System programming error. The “callable” input parameter cannot be null.
|
||||||
|
retryutil.2=System programming error. The “retrytime[%d]” input parameter cannot be less than 1.
|
||||||
|
retryutil.3=Exception when calling callable. Exception Msg: {0}
|
||||||
|
retryutil.4=Exception when calling callable. Retry Attempt: {0} will start soon. {1} attempts in total. This attempt planned to wait for [{2}]ms, and actually waited for [{3}]ms. Exception Msg: [{4}].
|
||||||
|
|
||||||
|
httpclientutil.1=Request address: {0}. Request method: {1}. STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=The remote interface returns -1. We will try again
|
||||||
@ -0,0 +1,53 @@
|
|||||||
|
very_like_yixiao=1{0}2{1}3
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||||
|
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||||
|
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||||
|
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||||
|
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||||
|
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||||
|
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||||
|
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||||
|
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||||
|
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||||
|
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||||
|
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||||
|
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||||
|
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||||
|
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
@ -0,0 +1,54 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||||
|
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||||
|
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||||
|
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||||
|
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||||
|
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||||
|
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||||
|
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||||
|
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||||
|
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||||
|
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||||
|
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||||
|
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||||
|
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||||
|
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
@ -0,0 +1,104 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||||
|
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||||
|
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||||
|
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||||
|
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||||
|
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||||
|
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||||
|
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||||
|
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||||
|
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||||
|
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||||
|
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||||
|
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||||
|
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||||
|
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||||
|
|
||||||
|
very_like_yixiao=一{0}二{1}三
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=配置資訊錯誤，您提供的配置檔案[{0}]不存在. 請檢查您的配置檔案.
|
||||||
|
configuration.2=配置資訊錯誤. 您提供配置檔案[{0}]讀取失敗，錯誤原因: {1}. 請檢查您的配置檔案的權限設定.
|
||||||
|
configuration.3=請檢查您的配置檔案. 您提供的配置檔案讀取失敗，錯誤原因: {0}. 請檢查您的配置檔案的權限設定.
|
||||||
|
configuration.4=您提供配置檔案有誤，[{0}]是必填參數，不允許為空或者留白 .
|
||||||
|
configuration.5=您提供配置檔案有誤，[{0}]是必填參數，不允許為空或者留白 .
|
||||||
|
configuration.6=任務讀取配置檔案出錯. 因為配置檔案路徑[{0}] 值不合法，期望是字符類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.7=您提供的配置資訊有誤，因為從[{0}]獲取的值[{1}]無法轉換為bool類型. 請檢查源表的配置並且做出相應的修改.
|
||||||
|
configuration.8=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.9=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.10=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是浮點類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.11=配置檔案對應Key[{0}]並不存在，該情況是代碼編程錯誤. 請聯絡DataX團隊的同學.
|
||||||
|
configuration.12=值[{0}]無法適配您提供[{1}]， 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!
|
||||||
|
configuration.13=Path不能為null，該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||||
|
configuration.14=路徑[{0}]出現不合法值類型[{1}]，該異常代表系統編程錯誤, 請聯絡DataX開發團隊! .
|
||||||
|
configuration.15=該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||||
|
configuration.16=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象，但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||||
|
configuration.17=您提供的配置檔案有誤. 路徑[{0}]值為null，datax無法識別該配置. 請檢查您的配置並作出修改.
|
||||||
|
configuration.18=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象，但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||||
|
configuration.19=系統編程錯誤，清單下標必須為數字類型，但該節點發現實際類型是[{0}] ，該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||||
|
configuration.20=系統編程錯誤, 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!.
|
||||||
|
configuration.21=系統編程錯誤, 路徑[{0}]不合法, 路徑層次之間不能出現空白字符 .
|
||||||
|
configuration.22=配置資訊錯誤. 因為您提供的配置資訊不是合法的JSON格式, JSON不能為空白. 請按照標準json格式提供配置資訊.
|
||||||
|
configuration.23=配置資訊錯誤. 您提供的配置資訊不是合法的JSON格式: {0}. 請按照標準json格式提供配置資訊.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=您提供的作業配置有誤，List不能為空.
|
||||||
|
listutil.2=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.3=您提供的作業配置資訊有誤, String:[{0}]不允許重複出現在清單中: [{1}].
|
||||||
|
listutil.4=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.5=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.6=您提供的作業配置資訊有誤, String:[{0}]不存在於清單中:[{1}].
|
||||||
|
listutil.7=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.8=您提供的作業配置有誤, List不能為空.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=切分份數不能小於1. 此處:expectSliceNumber=[{0}].
|
||||||
|
rangesplitutil.2=對 BigInteger 進行切分時，其左右區間不能為 null. 此處:left=[{0}],right=[{1}].
|
||||||
|
rangesplitutil.3=參數 bigInteger 不能為空.
|
||||||
|
rangesplitutil.4=根據字符串進行切分時僅支援 ASCII 字符串，而字符串:[{0}]非 ASCII 字符串.
|
||||||
|
rangesplitutil.5=參數 bigInteger 不能為空.
|
||||||
|
rangesplitutil.6=根據字符串進行切分時僅支援 ASCII 字符串，而字符串:[{0}]非 ASCII 字符串.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=系統編程錯誤, 入參callable不能為空 !
|
||||||
|
retryutil.2=系統編程錯誤, 入參retrytime[%d]不能小於1 !
|
||||||
|
retryutil.3=Exception when calling callable, 異常Msg:{0}
|
||||||
|
retryutil.4=Exception when calling callable, 即將嘗試執行第{0}次重試,共計重試{1}次.本次重試計劃等待[{2}]ms,實際等待[{3}]ms, 異常Msg:[{4}]
|
||||||
|
|
||||||
|
httpclientutil.1=\u8ACB\u6C42\u5730\u5740\uFF1A{0}, \u8ACB\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u9060\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C07\u91CD\u8A66
|
||||||
@ -0,0 +1,104 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef\uff0c\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6[{0}]\u4e0d\u5b58\u5728. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6.
|
||||||
|
configuration.2=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6[{0}]\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.3=\u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u8bfb\u53d6\u5931\u8d25\uff0c\u9519\u8bef\u539f\u56e0: {0}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u6587\u4ef6\u7684\u6743\u9650\u8bbe\u7f6e.
|
||||||
|
configuration.4=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.5=\u60a8\u63d0\u4f9b\u914d\u7f6e\u6587\u4ef6\u6709\u8bef\uff0c[{0}]\u662f\u5fc5\u586b\u53c2\u6570\uff0c\u4e0d\u5141\u8bb8\u4e3a\u7a7a\u6216\u8005\u7559\u767d .
|
||||||
|
configuration.6=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u56e0\u4e3a\u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5\uff0c\u671f\u671b\u662f\u5b57\u7b26\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.7=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u6709\u8bef\uff0c\u56e0\u4e3a\u4ece[{0}]\u83b7\u53d6\u7684\u503c[{1}]\u65e0\u6cd5\u8f6c\u6362\u4e3abool\u7c7b\u578b. \u8bf7\u68c0\u67e5\u6e90\u8868\u7684\u914d\u7f6e\u5e76\u4e14\u505a\u51fa\u76f8\u5e94\u7684\u4fee\u6539.
|
||||||
|
configuration.8=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.9=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6574\u6570\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.10=\u4efb\u52a1\u8bfb\u53d6\u914d\u7f6e\u6587\u4ef6\u51fa\u9519. \u914d\u7f6e\u6587\u4ef6\u8def\u5f84[{0}] \u503c\u975e\u6cd5, \u671f\u671b\u662f\u6d6e\u70b9\u7c7b\u578b: {1}. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.11=\u914d\u7f6e\u6587\u4ef6\u5bf9\u5e94Key[{0}]\u5e76\u4e0d\u5b58\u5728\uff0c\u8be5\u60c5\u51b5\u662f\u4ee3\u7801\u7f16\u7a0b\u9519\u8bef. \u8bf7\u8054\u7cfbDataX\u56e2\u961f\u7684\u540c\u5b66.
|
||||||
|
configuration.12=\u503c[{0}]\u65e0\u6cd5\u9002\u914d\u60a8\u63d0\u4f9b[{1}]\uff0c \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!
|
||||||
|
configuration.13=Path\u4e0d\u80fd\u4e3anull\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.14=\u8def\u5f84[{0}]\u51fa\u73b0\u975e\u6cd5\u503c\u7c7b\u578b[{1}]\uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f! .
|
||||||
|
configuration.15=\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.16=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.17=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u503c\u4e3anull\uff0cdatax\u65e0\u6cd5\u8bc6\u522b\u8be5\u914d\u7f6e. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.18=\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u6587\u4ef6\u6709\u8bef. \u8def\u5f84[{0}]\u9700\u8981\u914d\u7f6eJson\u683c\u5f0f\u7684Map\u5bf9\u8c61\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{1}]. \u8bf7\u68c0\u67e5\u60a8\u7684\u914d\u7f6e\u5e76\u4f5c\u51fa\u4fee\u6539.
|
||||||
|
configuration.19=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef\uff0c\u5217\u8868\u4e0b\u6807\u5fc5\u987b\u4e3a\u6570\u5b57\u7c7b\u578b\uff0c\u4f46\u8be5\u8282\u70b9\u53d1\u73b0\u5b9e\u9645\u7c7b\u578b\u662f[{0}] \uff0c\u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f !
|
||||||
|
configuration.20=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8be5\u5f02\u5e38\u4ee3\u8868\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8bf7\u8054\u7cfbDataX\u5f00\u53d1\u56e2\u961f!.
|
||||||
|
configuration.21=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u8def\u5f84[{0}]\u4e0d\u5408\u6cd5, \u8def\u5f84\u5c42\u6b21\u4e4b\u95f4\u4e0d\u80fd\u51fa\u73b0\u7a7a\u767d\u5b57\u7b26 .
|
||||||
|
configuration.22=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u56e0\u4e3a\u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f, JSON\u4e0d\u80fd\u4e3a\u7a7a\u767d. \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
configuration.23=\u914d\u7f6e\u4fe1\u606f\u9519\u8bef. \u60a8\u63d0\u4f9b\u7684\u914d\u7f6e\u4fe1\u606f\u4e0d\u662f\u5408\u6cd5\u7684JSON\u683c\u5f0f: {0} . \u8bf7\u6309\u7167\u6807\u51c6json\u683c\u5f0f\u63d0\u4f9b\u914d\u7f6e\u4fe1\u606f.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef\uff0cList\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.2=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.3=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5141\u8bb8\u91cd\u590d\u51fa\u73b0\u5728\u5217\u8868\u4e2d: [{1}].
|
||||||
|
listutil.4=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.5=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.6=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u4fe1\u606f\u6709\u8bef, String:[{0}] \u4e0d\u5b58\u5728\u4e8e\u5217\u8868\u4e2d:[{1}].
|
||||||
|
listutil.7=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
listutil.8=\u60a8\u63d0\u4f9b\u7684\u4f5c\u4e1a\u914d\u7f6e\u6709\u8bef, List\u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=\u5207\u5206\u4efd\u6570\u4e0d\u80fd\u5c0f\u4e8e1. \u6b64\u5904:expectSliceNumber=[{0}].
|
||||||
|
rangesplitutil.2=\u5bf9 BigInteger \u8fdb\u884c\u5207\u5206\u65f6\uff0c\u5176\u5de6\u53f3\u533a\u95f4\u4e0d\u80fd\u4e3a null. \u6b64\u5904:left=[{0}],right=[{1}].
|
||||||
|
rangesplitutil.3=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.4=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
rangesplitutil.5=\u53c2\u6570 bigInteger \u4e0d\u80fd\u4e3a\u7a7a.
|
||||||
|
rangesplitutil.6=\u6839\u636e\u5b57\u7b26\u4e32\u8fdb\u884c\u5207\u5206\u65f6\u4ec5\u652f\u6301 ASCII \u5b57\u7b26\u4e32\uff0c\u800c\u5b57\u7b26\u4e32:[{0}]\u975e ASCII \u5b57\u7b26\u4e32.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2callable\u4e0d\u80fd\u4e3a\u7a7a !
|
||||||
|
retryutil.2=\u7cfb\u7edf\u7f16\u7a0b\u9519\u8bef, \u5165\u53c2retrytime[%d]\u4e0d\u80fd\u5c0f\u4e8e1 !
|
||||||
|
retryutil.3=Exception when calling callable, \u5f02\u5e38Msg:{0}
|
||||||
|
retryutil.4=Exception when calling callable, \u5373\u5c06\u5c1d\u8bd5\u6267\u884c\u7b2c{0}\u6b21\u91cd\u8bd5,\u5171\u8ba1\u91cd\u8bd5{1}\u6b21.\u672c\u6b21\u91cd\u8bd5\u8ba1\u5212\u7b49\u5f85[{2}]ms,\u5b9e\u9645\u7b49\u5f85[{3}]ms, \u5f02\u5e38Msg:[{4}]
|
||||||
|
|
||||||
|
very_like_yixiao=一{0}二{1}三
|
||||||
|
|
||||||
|
|
||||||
|
configuration.1=配置資訊錯誤，您提供的配置檔案[{0}]不存在. 請檢查您的配置檔案.
|
||||||
|
configuration.2=配置資訊錯誤. 您提供配置檔案[{0}]讀取失敗，錯誤原因: {1}. 請檢查您的配置檔案的權限設定.
|
||||||
|
configuration.3=請檢查您的配置檔案. 您提供的配置檔案讀取失敗，錯誤原因: {0}. 請檢查您的配置檔案的權限設定.
|
||||||
|
configuration.4=您提供配置檔案有誤，[{0}]是必填參數，不允許為空或者留白 .
|
||||||
|
configuration.5=您提供配置檔案有誤，[{0}]是必填參數，不允許為空或者留白 .
|
||||||
|
configuration.6=任務讀取配置檔案出錯. 因為配置檔案路徑[{0}] 值不合法，期望是字符類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.7=您提供的配置資訊有誤，因為從[{0}]獲取的值[{1}]無法轉換為bool類型. 請檢查源表的配置並且做出相應的修改.
|
||||||
|
configuration.8=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.9=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是整數類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.10=任務讀取配置檔案出錯. 配置檔案路徑[{0}] 值不合法, 期望是浮點類型: {1}. 請檢查您的配置並作出修改.
|
||||||
|
configuration.11=配置檔案對應Key[{0}]並不存在，該情況是代碼編程錯誤. 請聯絡DataX團隊的同學.
|
||||||
|
configuration.12=值[{0}]無法適配您提供[{1}]， 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!
|
||||||
|
configuration.13=Path不能為null，該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||||
|
configuration.14=路徑[{0}]出現不合法值類型[{1}]，該異常代表系統編程錯誤, 請聯絡DataX開發團隊! .
|
||||||
|
configuration.15=該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||||
|
configuration.16=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象，但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||||
|
configuration.17=您提供的配置檔案有誤. 路徑[{0}]值為null，datax無法識別該配置. 請檢查您的配置並作出修改.
|
||||||
|
configuration.18=您提供的配置檔案有誤. 路徑[{0}]需要配置Json格式的Map對象，但該節點發現實際類型是[{1}]. 請檢查您的配置並作出修改.
|
||||||
|
configuration.19=系統編程錯誤，清單下標必須為數字類型，但該節點發現實際類型是[{0}] ，該異常代表系統編程錯誤, 請聯絡DataX開發團隊 !
|
||||||
|
configuration.20=系統編程錯誤, 該異常代表系統編程錯誤, 請聯絡DataX開發團隊!.
|
||||||
|
configuration.21=系統編程錯誤, 路徑[{0}]不合法, 路徑層次之間不能出現空白字符 .
|
||||||
|
configuration.22=配置資訊錯誤. 因為您提供的配置資訊不是合法的JSON格式, JSON不能為空白. 請按照標準json格式提供配置資訊.
|
||||||
|
configuration.23=配置資訊錯誤. 您提供的配置資訊不是合法的JSON格式: {0}. 請按照標準json格式提供配置資訊.
|
||||||
|
|
||||||
|
|
||||||
|
listutil.1=您提供的作業配置有誤，List不能為空.
|
||||||
|
listutil.2=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.3=您提供的作業配置資訊有誤, String:[{0}]不允許重複出現在清單中: [{1}].
|
||||||
|
listutil.4=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.5=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.6=您提供的作業配置資訊有誤, String:[{0}]不存在於清單中:[{1}].
|
||||||
|
listutil.7=您提供的作業配置有誤, List不能為空.
|
||||||
|
listutil.8=您提供的作業配置有誤, List不能為空.
|
||||||
|
|
||||||
|
|
||||||
|
rangesplitutil.1=切分份數不能小於1. 此處:expectSliceNumber=[{0}].
|
||||||
|
rangesplitutil.2=對 BigInteger 進行切分時，其左右區間不能為 null. 此處:left=[{0}],right=[{1}].
|
||||||
|
rangesplitutil.3=參數 bigInteger 不能為空.
|
||||||
|
rangesplitutil.4=根據字符串進行切分時僅支援 ASCII 字符串，而字符串:[{0}]非 ASCII 字符串.
|
||||||
|
rangesplitutil.5=參數 bigInteger 不能為空.
|
||||||
|
rangesplitutil.6=根據字符串進行切分時僅支援 ASCII 字符串，而字符串:[{0}]非 ASCII 字符串.
|
||||||
|
|
||||||
|
|
||||||
|
retryutil.1=系統編程錯誤, 入參callable不能為空 !
|
||||||
|
retryutil.2=系統編程錯誤, 入參retrytime[%d]不能小於1 !
|
||||||
|
retryutil.3=Exception when calling callable, 異常Msg:{0}
|
||||||
|
retryutil.4=Exception when calling callable, 即將嘗試執行第{0}次重試,共計重試{1}次.本次重試計劃等待[{2}]ms,實際等待[{3}]ms, 異常Msg:[{4}]
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
@ -0,0 +1,207 @@
|
|||||||
|
package com.alibaba.datax.common.util;
|
||||||
|
|
||||||
|
import java.text.MessageFormat;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.Locale;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.MissingResourceException;
|
||||||
|
import java.util.ResourceBundle;
|
||||||
|
import java.util.TimeZone;
|
||||||
|
|
||||||
|
import org.apache.commons.lang3.LocaleUtils;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
|
||||||
|
public class MessageSource {
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(MessageSource.class);
|
||||||
|
private static Map<String, ResourceBundle> resourceBundleCache = new HashMap<String, ResourceBundle>();
|
||||||
|
public static Locale locale = null;
|
||||||
|
public static TimeZone timeZone = null;
|
||||||
|
private ResourceBundle resourceBundle = null;
|
||||||
|
|
||||||
|
private MessageSource(ResourceBundle resourceBundle) {
|
||||||
|
this.resourceBundle = resourceBundle;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param baseName
|
||||||
|
* demo: javax.servlet.http.LocalStrings
|
||||||
|
*
|
||||||
|
* @throws MissingResourceException
|
||||||
|
* - if no resource bundle for the specified base name can be
|
||||||
|
* found
|
||||||
|
* */
|
||||||
|
public static MessageSource loadResourceBundle(String baseName) {
|
||||||
|
return loadResourceBundle(baseName, MessageSource.locale,
|
||||||
|
MessageSource.timeZone);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param clazz
|
||||||
|
* 根据其获取package name
|
||||||
|
* */
|
||||||
|
public static <T> MessageSource loadResourceBundle(Class<T> clazz) {
|
||||||
|
return loadResourceBundle(clazz.getPackage().getName());
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param clazz
|
||||||
|
* 根据其获取package name
|
||||||
|
* */
|
||||||
|
public static <T> MessageSource loadResourceBundle(Class<T> clazz,
|
||||||
|
Locale locale, TimeZone timeZone) {
|
||||||
|
return loadResourceBundle(clazz.getPackage().getName(), locale,
|
||||||
|
timeZone);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* warn:
|
||||||
|
* ok: ResourceBundle.getBundle("xxx.LocalStrings", Locale.getDefault(), LoadUtil.getJarLoader(PluginType.WRITER, "odpswriter"))
|
||||||
|
* error: ResourceBundle.getBundle("xxx.LocalStrings", Locale.getDefault(), LoadUtil.getJarLoader(PluginType.WRITER, "odpswriter"))
|
||||||
|
* @param baseName
|
||||||
|
* demo: javax.servlet.http.LocalStrings
|
||||||
|
*
|
||||||
|
* @throws MissingResourceException
|
||||||
|
* - if no resource bundle for the specified base name can be
|
||||||
|
* found
|
||||||
|
*
|
||||||
|
* */
|
||||||
|
public static MessageSource loadResourceBundle(String baseName,
|
||||||
|
Locale locale, TimeZone timeZone) {
|
||||||
|
ResourceBundle resourceBundle = null;
|
||||||
|
if (null == locale) {
|
||||||
|
locale = LocaleUtils.toLocale("en_US");
|
||||||
|
}
|
||||||
|
if (null == timeZone) {
|
||||||
|
timeZone = TimeZone.getDefault();
|
||||||
|
}
|
||||||
|
String resourceBaseName = String.format("%s.LocalStrings", baseName);
|
||||||
|
LOG.debug(
|
||||||
|
"initEnvironment MessageSource.locale[{}], MessageSource.timeZone[{}]",
|
||||||
|
MessageSource.locale, MessageSource.timeZone);
|
||||||
|
LOG.debug(
|
||||||
|
"loadResourceBundle with locale[{}], timeZone[{}], baseName[{}]",
|
||||||
|
locale, timeZone, resourceBaseName);
|
||||||
|
// warn: 这个map的维护需要考虑Local吗, no?
|
||||||
|
if (!MessageSource.resourceBundleCache.containsKey(resourceBaseName)) {
|
||||||
|
ClassLoader clazzLoader = Thread.currentThread()
|
||||||
|
.getContextClassLoader();
|
||||||
|
LOG.debug("loadResourceBundle classLoader:{}", clazzLoader);
|
||||||
|
resourceBundle = ResourceBundle.getBundle(resourceBaseName, locale,
|
||||||
|
clazzLoader);
|
||||||
|
MessageSource.resourceBundleCache.put(resourceBaseName,
|
||||||
|
resourceBundle);
|
||||||
|
} else {
|
||||||
|
resourceBundle = MessageSource.resourceBundleCache
|
||||||
|
.get(resourceBaseName);
|
||||||
|
}
|
||||||
|
|
||||||
|
return new MessageSource(resourceBundle);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static <T> boolean unloadResourceBundle(Class<T> clazz) {
|
||||||
|
String baseName = clazz.getPackage().getName();
|
||||||
|
String resourceBaseName = String.format("%s.LocalStrings", baseName);
|
||||||
|
if (!MessageSource.resourceBundleCache.containsKey(resourceBaseName)) {
|
||||||
|
return false;
|
||||||
|
} else {
|
||||||
|
MessageSource.resourceBundleCache.remove(resourceBaseName);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static <T> MessageSource reloadResourceBundle(Class<T> clazz) {
|
||||||
|
MessageSource.unloadResourceBundle(clazz);
|
||||||
|
return MessageSource.loadResourceBundle(clazz);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static void setEnvironment(Locale locale, TimeZone timeZone) {
|
||||||
|
// warn: 设置默认? @2018.03.21 将此处注释移除，否则在国际化多时区下会遇到问题
|
||||||
|
Locale.setDefault(locale);
|
||||||
|
TimeZone.setDefault(timeZone);
|
||||||
|
MessageSource.locale = locale;
|
||||||
|
MessageSource.timeZone = timeZone;
|
||||||
|
LOG.info("use Locale: {} timeZone: {}", locale, timeZone);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static void init(final Configuration configuration) {
|
||||||
|
Locale locale2Set = Locale.getDefault();
|
||||||
|
String localeStr = configuration.getString("common.column.locale", "zh_CN");// 默认操作系统的
|
||||||
|
if (StringUtils.isNotBlank(localeStr)) {
|
||||||
|
try {
|
||||||
|
locale2Set = LocaleUtils.toLocale(localeStr);
|
||||||
|
} catch (Exception e) {
|
||||||
|
LOG.warn("ignored locale parse exception: {}", e.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
TimeZone timeZone2Set = TimeZone.getDefault();
|
||||||
|
String timeZoneStr = configuration.getString("common.column.timeZone");// 默认操作系统的
|
||||||
|
if (StringUtils.isNotBlank(timeZoneStr)) {
|
||||||
|
try {
|
||||||
|
timeZone2Set = TimeZone.getTimeZone(timeZoneStr);
|
||||||
|
} catch (Exception e) {
|
||||||
|
LOG.warn("ignored timezone parse exception: {}", e.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
LOG.info("JVM TimeZone: {}, Locale: {}", timeZone2Set.getID(), locale2Set);
|
||||||
|
MessageSource.setEnvironment(locale2Set, timeZone2Set);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static void clearCache() {
|
||||||
|
MessageSource.resourceBundleCache.clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
public String message(String code) {
|
||||||
|
return this.messageWithDefaultMessage(code, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
public String message(String code, String args1) {
|
||||||
|
return this.messageWithDefaultMessage(code, null,
|
||||||
|
new Object[] { args1 });
|
||||||
|
}
|
||||||
|
|
||||||
|
public String message(String code, String args1, String args2) {
|
||||||
|
return this.messageWithDefaultMessage(code, null, new Object[] { args1,
|
||||||
|
args2 });
|
||||||
|
}
|
||||||
|
|
||||||
|
public String message(String code, String args1, String args2, String args3) {
|
||||||
|
return this.messageWithDefaultMessage(code, null, new Object[] { args1,
|
||||||
|
args2, args3 });
|
||||||
|
}
|
||||||
|
|
||||||
|
// 上面几个重载可以应对大多数情况, 避免使用这个可以提高性能的
|
||||||
|
public String message(String code, Object... args) {
|
||||||
|
return this.messageWithDefaultMessage(code, null, args);
|
||||||
|
}
|
||||||
|
|
||||||
|
public String messageWithDefaultMessage(String code, String defaultMessage) {
|
||||||
|
return this.messageWithDefaultMessage(code, defaultMessage,
|
||||||
|
new Object[] {});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param args
|
||||||
|
* MessageFormat会依次调用对应对象的toString方法
|
||||||
|
* */
|
||||||
|
public String messageWithDefaultMessage(String code, String defaultMessage,
|
||||||
|
Object... args) {
|
||||||
|
String messageStr = null;
|
||||||
|
try {
|
||||||
|
messageStr = this.resourceBundle.getString(code);
|
||||||
|
} catch (MissingResourceException e) {
|
||||||
|
messageStr = defaultMessage;
|
||||||
|
}
|
||||||
|
if (null != messageStr && null != args && args.length > 0) {
|
||||||
|
// warn: see loadResourceBundle set default locale
|
||||||
|
return MessageFormat.format(messageStr, args);
|
||||||
|
} else {
|
||||||
|
return messageStr;
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -206,4 +206,27 @@ public final class RangeSplitUtil {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* List拆分工具函数，主要用于reader插件的split拆分逻辑
|
||||||
|
* */
|
||||||
|
public static <T> List<List<T>> doListSplit(List<T> objects, int adviceNumber) {
|
||||||
|
List<List<T>> splitLists = new ArrayList<List<T>>();
|
||||||
|
if (null == objects) {
|
||||||
|
return splitLists;
|
||||||
|
}
|
||||||
|
long[] splitPoint = RangeSplitUtil.doLongSplit(0, objects.size(), adviceNumber);
|
||||||
|
for (int startIndex = 0; startIndex < splitPoint.length - 1; startIndex++) {
|
||||||
|
List<T> objectsForTask = new ArrayList<T>();
|
||||||
|
int endIndex = startIndex + 1;
|
||||||
|
for (long i = splitPoint[startIndex]; i < splitPoint[endIndex]; i++) {
|
||||||
|
objectsForTask.add(objects.get((int) i));
|
||||||
|
}
|
||||||
|
if (!objectsForTask.isEmpty()) {
|
||||||
|
splitLists.add(objectsForTask);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return splitLists;
|
||||||
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -100,6 +100,14 @@
|
|||||||
</dependencies>
|
</dependencies>
|
||||||
|
|
||||||
<build>
|
<build>
|
||||||
|
<resources>
|
||||||
|
<resource>
|
||||||
|
<directory>src/main/java</directory>
|
||||||
|
<includes>
|
||||||
|
<include>**/*.properties</include>
|
||||||
|
</includes>
|
||||||
|
</resource>
|
||||||
|
</resources>
|
||||||
<plugins>
|
<plugins>
|
||||||
<plugin>
|
<plugin>
|
||||||
<groupId>org.apache.maven.plugins</groupId>
|
<groupId>org.apache.maven.plugins</groupId>
|
||||||
|
|||||||
@ -1,23 +1,26 @@
|
|||||||
#!/usr/bin/env python
|
#!/usr/bin/env python
|
||||||
# -*- coding:utf-8 -*-
|
# -*- coding:utf-8 -*-
|
||||||
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
import signal
|
|
||||||
import subprocess
|
|
||||||
import time
|
|
||||||
import re
|
|
||||||
import socket
|
|
||||||
import json
|
|
||||||
from optparse import OptionParser
|
|
||||||
from optparse import OptionGroup
|
|
||||||
from string import Template
|
|
||||||
import codecs
|
import codecs
|
||||||
|
import json
|
||||||
|
import os
|
||||||
import platform
|
import platform
|
||||||
|
import re
|
||||||
|
import signal
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from optparse import OptionGroup
|
||||||
|
from optparse import OptionParser
|
||||||
|
from string import Template
|
||||||
|
|
||||||
|
ispy2 = sys.version_info.major == 2
|
||||||
|
|
||||||
def isWindows():
|
def isWindows():
|
||||||
return platform.system() == 'Windows'
|
return platform.system() == 'Windows'
|
||||||
|
|
||||||
|
|
||||||
DATAX_HOME = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
DATAX_HOME = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
DATAX_VERSION = 'DATAX-OPENSOURCE-3.0'
|
DATAX_VERSION = 'DATAX-OPENSOURCE-3.0'
|
||||||
@ -52,13 +55,19 @@ def getLocalIp():
|
|||||||
|
|
||||||
def suicide(signum, e):
|
def suicide(signum, e):
|
||||||
global child_process
|
global child_process
|
||||||
print >> sys.stderr, "[Error] DataX receive unexpected signal %d, starts to suicide." % (signum)
|
if ispy2:
|
||||||
|
print >> sys.stderr, "[Error] DataX receive unexpected signal %d, starts to suicide." % (signum)
|
||||||
|
else:
|
||||||
|
print("[Error] DataX receive unexpected signal %d, starts to suicide." % (signum), sys.stderr)
|
||||||
|
|
||||||
if child_process:
|
if child_process:
|
||||||
child_process.send_signal(signal.SIGQUIT)
|
child_process.send_signal(signal.SIGQUIT)
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
child_process.kill()
|
child_process.kill()
|
||||||
print >> sys.stderr, "DataX Process was killed ! you did ?"
|
if ispy2:
|
||||||
|
print >> sys.stderr, "DataX Process was killed ! you did ?"
|
||||||
|
else:
|
||||||
|
print("DataX Process was killed ! you did ?", sys.stderr)
|
||||||
sys.exit(RET_STATE["KILL"])
|
sys.exit(RET_STATE["KILL"])
|
||||||
|
|
||||||
|
|
||||||
@ -92,10 +101,10 @@ def getOptionParser():
|
|||||||
'if you have mutiple parameters: -p"-DtableName=your-table-name -DcolumnName=your-column-name".'
|
'if you have mutiple parameters: -p"-DtableName=your-table-name -DcolumnName=your-column-name".'
|
||||||
'Note: you should config in you job tableName with ${tableName}.')
|
'Note: you should config in you job tableName with ${tableName}.')
|
||||||
prodEnvOptionGroup.add_option("-r", "--reader", metavar="<parameter used in view job config[reader] template>",
|
prodEnvOptionGroup.add_option("-r", "--reader", metavar="<parameter used in view job config[reader] template>",
|
||||||
action="store", dest="reader",type="string",
|
action="store", dest="reader", type="string",
|
||||||
help='View job config[reader] template, eg: mysqlreader,streamreader')
|
help='View job config[reader] template, eg: mysqlreader,streamreader')
|
||||||
prodEnvOptionGroup.add_option("-w", "--writer", metavar="<parameter used in view job config[writer] template>",
|
prodEnvOptionGroup.add_option("-w", "--writer", metavar="<parameter used in view job config[writer] template>",
|
||||||
action="store", dest="writer",type="string",
|
action="store", dest="writer", type="string",
|
||||||
help='View job config[writer] template, eg: mysqlwriter,streamwriter')
|
help='View job config[writer] template, eg: mysqlwriter,streamwriter')
|
||||||
parser.add_option_group(prodEnvOptionGroup)
|
parser.add_option_group(prodEnvOptionGroup)
|
||||||
|
|
||||||
@ -108,45 +117,50 @@ def getOptionParser():
|
|||||||
parser.add_option_group(devEnvOptionGroup)
|
parser.add_option_group(devEnvOptionGroup)
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
|
|
||||||
def generateJobConfigTemplate(reader, writer):
|
def generateJobConfigTemplate(reader, writer):
|
||||||
readerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n" % (reader,reader,reader)
|
readerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n" % (
|
||||||
writerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n " % (writer,writer,writer)
|
reader, reader, reader)
|
||||||
print readerRef
|
writerRef = "Please refer to the %s document:\n https://github.com/alibaba/DataX/blob/master/%s/doc/%s.md \n " % (
|
||||||
print writerRef
|
writer, writer, writer)
|
||||||
|
print(readerRef)
|
||||||
|
print(writerRef)
|
||||||
jobGuid = 'Please save the following configuration as a json file and use\n python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json \nto run the job.\n'
|
jobGuid = 'Please save the following configuration as a json file and use\n python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json \nto run the job.\n'
|
||||||
print jobGuid
|
print(jobGuid)
|
||||||
jobTemplate={
|
jobTemplate = {
|
||||||
"job": {
|
"job": {
|
||||||
"setting": {
|
"setting": {
|
||||||
"speed": {
|
"speed": {
|
||||||
"channel": ""
|
"channel": ""
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"content": [
|
"content": [
|
||||||
{
|
{
|
||||||
"reader": {},
|
"reader": {},
|
||||||
"writer": {}
|
"writer": {}
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
readerTemplatePath = "%s/plugin/reader/%s/plugin_job_template.json" % (DATAX_HOME,reader)
|
readerTemplatePath = "%s/plugin/reader/%s/plugin_job_template.json" % (DATAX_HOME, reader)
|
||||||
writerTemplatePath = "%s/plugin/writer/%s/plugin_job_template.json" % (DATAX_HOME,writer)
|
writerTemplatePath = "%s/plugin/writer/%s/plugin_job_template.json" % (DATAX_HOME, writer)
|
||||||
try:
|
try:
|
||||||
readerPar = readPluginTemplate(readerTemplatePath);
|
readerPar = readPluginTemplate(readerTemplatePath)
|
||||||
except Exception, e:
|
except:
|
||||||
print "Read reader[%s] template error: can\'t find file %s" % (reader,readerTemplatePath)
|
print("Read reader[%s] template error: can\'t find file %s" % (reader, readerTemplatePath))
|
||||||
try:
|
try:
|
||||||
writerPar = readPluginTemplate(writerTemplatePath);
|
writerPar = readPluginTemplate(writerTemplatePath)
|
||||||
except Exception, e:
|
except:
|
||||||
print "Read writer[%s] template error: : can\'t find file %s" % (writer,writerTemplatePath)
|
print("Read writer[%s] template error: : can\'t find file %s" % (writer, writerTemplatePath))
|
||||||
jobTemplate['job']['content'][0]['reader'] = readerPar;
|
jobTemplate['job']['content'][0]['reader'] = readerPar
|
||||||
jobTemplate['job']['content'][0]['writer'] = writerPar;
|
jobTemplate['job']['content'][0]['writer'] = writerPar
|
||||||
print json.dumps(jobTemplate, indent=4, sort_keys=True)
|
print(json.dumps(jobTemplate, indent=4, sort_keys=True))
|
||||||
|
|
||||||
|
|
||||||
def readPluginTemplate(plugin):
|
def readPluginTemplate(plugin):
|
||||||
with open(plugin, 'r') as f:
|
with open(plugin, 'r') as f:
|
||||||
return json.load(f)
|
return json.load(f)
|
||||||
|
|
||||||
|
|
||||||
def isUrl(path):
|
def isUrl(path):
|
||||||
if not path:
|
if not path:
|
||||||
@ -168,7 +182,7 @@ def buildStartCommand(options, args):
|
|||||||
|
|
||||||
if options.remoteDebug:
|
if options.remoteDebug:
|
||||||
tempJVMCommand = tempJVMCommand + " " + REMOTE_DEBUG_CONFIG
|
tempJVMCommand = tempJVMCommand + " " + REMOTE_DEBUG_CONFIG
|
||||||
print 'local ip: ', getLocalIp()
|
print('local ip: ', getLocalIp())
|
||||||
|
|
||||||
if options.loglevel:
|
if options.loglevel:
|
||||||
tempJVMCommand = tempJVMCommand + " " + ("-Dloglevel=%s" % (options.loglevel))
|
tempJVMCommand = tempJVMCommand + " " + ("-Dloglevel=%s" % (options.loglevel))
|
||||||
@ -198,11 +212,11 @@ def buildStartCommand(options, args):
|
|||||||
|
|
||||||
|
|
||||||
def printCopyright():
|
def printCopyright():
|
||||||
print '''
|
print('''
|
||||||
DataX (%s), From Alibaba !
|
DataX (%s), From Alibaba !
|
||||||
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
|
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
|
||||||
|
|
||||||
''' % DATAX_VERSION
|
''' % DATAX_VERSION)
|
||||||
sys.stdout.flush()
|
sys.stdout.flush()
|
||||||
|
|
||||||
|
|
||||||
@ -211,7 +225,7 @@ if __name__ == "__main__":
|
|||||||
parser = getOptionParser()
|
parser = getOptionParser()
|
||||||
options, args = parser.parse_args(sys.argv[1:])
|
options, args = parser.parse_args(sys.argv[1:])
|
||||||
if options.reader is not None and options.writer is not None:
|
if options.reader is not None and options.writer is not None:
|
||||||
generateJobConfigTemplate(options.reader,options.writer)
|
generateJobConfigTemplate(options.reader, options.writer)
|
||||||
sys.exit(RET_STATE['OK'])
|
sys.exit(RET_STATE['OK'])
|
||||||
if len(args) != 1:
|
if len(args) != 1:
|
||||||
parser.print_help()
|
parser.print_help()
|
||||||
|
|||||||
@ -6,6 +6,7 @@ import com.alibaba.datax.common.spi.ErrorCode;
|
|||||||
import com.alibaba.datax.common.statistics.PerfTrace;
|
import com.alibaba.datax.common.statistics.PerfTrace;
|
||||||
import com.alibaba.datax.common.statistics.VMInfo;
|
import com.alibaba.datax.common.statistics.VMInfo;
|
||||||
import com.alibaba.datax.common.util.Configuration;
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.common.util.MessageSource;
|
||||||
import com.alibaba.datax.core.job.JobContainer;
|
import com.alibaba.datax.core.job.JobContainer;
|
||||||
import com.alibaba.datax.core.taskgroup.TaskGroupContainer;
|
import com.alibaba.datax.core.taskgroup.TaskGroupContainer;
|
||||||
import com.alibaba.datax.core.util.ConfigParser;
|
import com.alibaba.datax.core.util.ConfigParser;
|
||||||
@ -73,7 +74,7 @@ public class Engine {
|
|||||||
boolean traceEnable = allConf.getBool(CoreConstant.DATAX_CORE_CONTAINER_TRACE_ENABLE, true);
|
boolean traceEnable = allConf.getBool(CoreConstant.DATAX_CORE_CONTAINER_TRACE_ENABLE, true);
|
||||||
boolean perfReportEnable = allConf.getBool(CoreConstant.DATAX_CORE_REPORT_DATAX_PERFLOG, true);
|
boolean perfReportEnable = allConf.getBool(CoreConstant.DATAX_CORE_REPORT_DATAX_PERFLOG, true);
|
||||||
|
|
||||||
//standlone模式的datax shell任务不进行汇报
|
//standalone模式的 datax shell任务不进行汇报
|
||||||
if(instanceId == -1){
|
if(instanceId == -1){
|
||||||
perfReportEnable = false;
|
perfReportEnable = false;
|
||||||
}
|
}
|
||||||
@ -135,6 +136,9 @@ public class Engine {
|
|||||||
RUNTIME_MODE = cl.getOptionValue("mode");
|
RUNTIME_MODE = cl.getOptionValue("mode");
|
||||||
|
|
||||||
Configuration configuration = ConfigParser.parse(jobPath);
|
Configuration configuration = ConfigParser.parse(jobPath);
|
||||||
|
// 绑定i18n信息
|
||||||
|
MessageSource.init(configuration);
|
||||||
|
MessageSource.reloadResourceBundle(Configuration.class);
|
||||||
|
|
||||||
long jobId;
|
long jobId;
|
||||||
if (!"-1".equalsIgnoreCase(jobIdString)) {
|
if (!"-1".equalsIgnoreCase(jobIdString)) {
|
||||||
|
|||||||
@ -0,0 +1,5 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||||
|
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||||
|
|
||||||
@ -0,0 +1,5 @@
|
|||||||
|
very_like_yixiao=1{0}2{1}3
|
||||||
|
|
||||||
|
engine.1=A valid job ID must be provided in the URL for the non-standalone mode.
|
||||||
|
engine.2=\n\nThrough the intelligent analysis by DataX, the most likely error reason of this task is: \n{0}
|
||||||
|
|
||||||
@ -0,0 +1,5 @@
|
|||||||
|
very_like_yixiao=1{0}2{1}3
|
||||||
|
|
||||||
|
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||||
|
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||||
|
|
||||||
@ -0,0 +1,5 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||||
|
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||||
|
|
||||||
@ -0,0 +1,10 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||||
|
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||||
|
|
||||||
|
very_like_yixiao=一{0}二{1}三
|
||||||
|
|
||||||
|
engine.1=非 standalone 模式必須在 URL 中提供有效的 jobId.
|
||||||
|
engine.2=\n\n經DataX智能分析,該任務最可能的錯誤原因是:\n{0}
|
||||||
|
|
||||||
@ -0,0 +1,10 @@
|
|||||||
|
very_like_yixiao=\u4e00{0}\u4e8c{1}\u4e09
|
||||||
|
|
||||||
|
engine.1=\u975e standalone \u6a21\u5f0f\u5fc5\u987b\u5728 URL \u4e2d\u63d0\u4f9b\u6709\u6548\u7684 jobId.
|
||||||
|
engine.2=\n\n\u7ecfDataX\u667a\u80fd\u5206\u6790,\u8be5\u4efb\u52a1\u6700\u53ef\u80fd\u7684\u9519\u8bef\u539f\u56e0\u662f:\n{0}
|
||||||
|
|
||||||
|
very_like_yixiao=一{0}二{1}三
|
||||||
|
|
||||||
|
engine.1=非 standalone 模式必須在 URL 中提供有效的 jobId.
|
||||||
|
engine.2=\n\n經DataX智能分析,該任務最可能的錯誤原因是:\n{0}
|
||||||
|
|
||||||
@ -11,15 +11,18 @@ import java.math.BigInteger;
|
|||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.Date;
|
import java.util.Date;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
public class DirtyRecord implements Record {
|
public class DirtyRecord implements Record {
|
||||||
private List<Column> columns = new ArrayList<Column>();
|
private List<Column> columns = new ArrayList<Column>();
|
||||||
|
private Map<String, String> meta;
|
||||||
|
|
||||||
public static DirtyRecord asDirtyRecord(final Record record) {
|
public static DirtyRecord asDirtyRecord(final Record record) {
|
||||||
DirtyRecord result = new DirtyRecord();
|
DirtyRecord result = new DirtyRecord();
|
||||||
for (int i = 0; i < record.getColumnNumber(); i++) {
|
for (int i = 0; i < record.getColumnNumber(); i++) {
|
||||||
result.addColumn(record.getColumn(i));
|
result.addColumn(record.getColumn(i));
|
||||||
}
|
}
|
||||||
|
result.setMeta(record.getMeta());
|
||||||
|
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@ -65,6 +68,16 @@ public class DirtyRecord implements Record {
|
|||||||
"该方法不支持!");
|
"该方法不支持!");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void setMeta(Map<String, String> meta) {
|
||||||
|
this.meta = meta;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Map<String, String> getMeta() {
|
||||||
|
return this.meta;
|
||||||
|
}
|
||||||
|
|
||||||
public List<Column> getColumns() {
|
public List<Column> getColumns() {
|
||||||
return columns;
|
return columns;
|
||||||
}
|
}
|
||||||
@ -119,6 +132,12 @@ class DirtyColumn extends Column {
|
|||||||
throw DataXException.asDataXException(FrameworkErrorCode.RUNTIME_ERROR,
|
throw DataXException.asDataXException(FrameworkErrorCode.RUNTIME_ERROR,
|
||||||
"该方法不支持!");
|
"该方法不支持!");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Date asDate(String dateFormat) {
|
||||||
|
throw DataXException.asDataXException(FrameworkErrorCode.RUNTIME_ERROR,
|
||||||
|
"该方法不支持!");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public byte[] asBytes() {
|
public byte[] asBytes() {
|
||||||
|
|||||||
@ -27,6 +27,8 @@ public class DefaultRecord implements Record {
|
|||||||
// 首先是Record本身需要的内存
|
// 首先是Record本身需要的内存
|
||||||
private int memorySize = ClassSize.DefaultRecordHead;
|
private int memorySize = ClassSize.DefaultRecordHead;
|
||||||
|
|
||||||
|
private Map<String, String> meta;
|
||||||
|
|
||||||
public DefaultRecord() {
|
public DefaultRecord() {
|
||||||
this.columns = new ArrayList<Column>(RECORD_AVERGAE_COLUMN_NUMBER);
|
this.columns = new ArrayList<Column>(RECORD_AVERGAE_COLUMN_NUMBER);
|
||||||
}
|
}
|
||||||
@ -83,6 +85,16 @@ public class DefaultRecord implements Record {
|
|||||||
return memorySize;
|
return memorySize;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void setMeta(Map<String, String> meta) {
|
||||||
|
this.meta = meta;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Map<String, String> getMeta() {
|
||||||
|
return this.meta;
|
||||||
|
}
|
||||||
|
|
||||||
private void decrByteSize(final Column column) {
|
private void decrByteSize(final Column column) {
|
||||||
if (null == column) {
|
if (null == column) {
|
||||||
return;
|
return;
|
||||||
|
|||||||
@ -3,6 +3,8 @@ package com.alibaba.datax.core.transport.record;
|
|||||||
import com.alibaba.datax.common.element.Column;
|
import com.alibaba.datax.common.element.Column;
|
||||||
import com.alibaba.datax.common.element.Record;
|
import com.alibaba.datax.common.element.Record;
|
||||||
|
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 作为标示 生产者已经完成生产的标志
|
* 作为标示 生产者已经完成生产的标志
|
||||||
*
|
*
|
||||||
@ -41,6 +43,16 @@ public class TerminateRecord implements Record {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void setMeta(Map<String, String> meta) {
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Map<String, String> getMeta() {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void setColumn(int i, Column column) {
|
public void setColumn(int i, Column column) {
|
||||||
return;
|
return;
|
||||||
|
|||||||
@ -0,0 +1,58 @@
|
|||||||
|
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||||
|
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||||
|
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||||
|
|
||||||
|
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||||
|
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||||
|
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||||
|
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||||
|
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||||
|
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||||
|
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||||
|
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||||
|
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||||
|
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||||
|
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1}, STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||||
|
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||||
|
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||||
|
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||||
|
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||||
|
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||||
|
|
||||||
@ -0,0 +1,58 @@
|
|||||||
|
configparser.1=Failed to load the plug-in [{0},{1}]. We will retry in 1s... Exception: {2}
|
||||||
|
configparser.2=Failed to obtain the job configuration information: {0}
|
||||||
|
configparser.3=Failed to obtain the job configuration information: {0}
|
||||||
|
configparser.4=Failed to obtain the job configuration information: {0}
|
||||||
|
configparser.5=Failed to load the plug-in. Loading of the specific plug-in:{0} is not completed
|
||||||
|
configparser.6=Failed to load the plug-in. A duplicate plug-in: {0} exists
|
||||||
|
|
||||||
|
dataxserviceutil.1=Exception in creating signature. NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=Exception in creating signature. InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=Exception in creating signature. UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=The percentage of dirty data should be limited to within [0.0, 1.0]
|
||||||
|
errorrecordchecker.2=The number of dirty data entries should now be a nonnegative integer
|
||||||
|
errorrecordchecker.3=Check for the number of dirty data entries has not passed. The limit is [{0}] entries, but [{1}] entries have been captured.
|
||||||
|
errorrecordchecker.4=Check for the percentage of dirty data has not passed. The limit is [{0}], but [{1}] of dirty data has been captured.
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=Error in installing DataX engine. Please contact your O&M team to solve the problem.
|
||||||
|
errorcode.argument_error=Error in running DataX engine. This problem is generally caused by an internal programming error. Please contact the DataX developer team to solve the problem.
|
||||||
|
errorcode.runtime_error=The DataX engine encountered an error during running. For the specific cause, refer to the error diagnosis after DataX stops running.
|
||||||
|
errorcode.config_error=Error in DataX engine configuration. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||||
|
errorcode.secret_error=Error in DataX engine encryption or decryption. This problem is generally caused by a DataX key configuration error. Please contact your O&M team to solve the problem.
|
||||||
|
errorcode.hook_load_error=Error in loading the external hook. This problem is generally caused by the DataX installation.
|
||||||
|
errorcode.hook_fail_error=Error in executing the external hook
|
||||||
|
errorcode.plugin_install_error=Error in installing DataX plug-in. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||||
|
errorcode.plugin_not_found=Error in DataX plug-in configuration. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||||
|
errorcode.plugin_init_error=Error in DataX plug-in initialization. This problem is generally caused by a DataX installation error. Please contact your O&M team to solve the problem.
|
||||||
|
errorcode.plugin_runtime_error=The DataX plug-in encountered an error during running. For the specific cause, refer to the error diagnosis after DataX stops running.
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=The dirty data transmitted by DataX exceeds user expectations. This error often occurs when a lot dirty data exists in the source data. Please carefully check the dirty data log information reported by DataX, or you can tune up the dirty data threshold value.
|
||||||
|
errorcode.plugin_split_error=Error in DataX plug-in slicing. This problem is generally caused by a programming error in some DataX plug-in. Please contact the DataX developer team to solve the problem.
|
||||||
|
errorcode.kill_job_timeout_error=The kill task times out. Please contact the PE to solve the problem
|
||||||
|
errorcode.start_taskgroup_error=Failed to start the task group. Please contact the DataX developer team to solve the problem
|
||||||
|
errorcode.call_datax_service_failed=Error in requesting DataX Service.
|
||||||
|
errorcode.call_remote_failed=Remote call failure
|
||||||
|
errorcode.killed_exit_value=The job has received a Kill command.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=Request address: {0}. Request method: {1}. STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=The remote interface returns -1. We will try again
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=System programing error. Unsupported encryption type
|
||||||
|
secretutil.2=System programing error. Unsupported encryption type
|
||||||
|
secretutil.3=RSA encryption error
|
||||||
|
secretutil.4=RSA decryption error
|
||||||
|
secretutil.5=Triple DES encryption error
|
||||||
|
secretutil.6=RSA decryption error
|
||||||
|
secretutil.7=Error in building Triple DES key
|
||||||
|
secretutil.8=DataX configuration requires encryption and decryption, but unable to find the key configuration file
|
||||||
|
secretutil.9=Error in reading the encryption and decryption configuration file
|
||||||
|
secretutil.10=The version of the DataX-configured key is [{0}], but there is no configuration in the system. Error in task key configuration. The key version you configured does not exist
|
||||||
|
secretutil.11=The version of the DataX-configured key is [{0}], but there is no configuration in the system. There may be an error in task key configuration, or a problem in system maintenance
|
||||||
|
secretutil.12=The version of the DataX-configured key is [{0}], but there is no configuration in the system. Error in task key configuration. The key version you configured does not exist
|
||||||
|
secretutil.13=The version of the DataX-configured key is [{0}], but there is no configuration in the system. There may be an error in task key configuration, or a problem in system maintenance
|
||||||
|
secretutil.14=DataX configuration requires encryption and decryption, but some key in the configured key version [{0}] is empty
|
||||||
|
secretutil.15=DataX configuration requires encryption and decryption, but some configured public/private key pairs are empty and the version is [{0}]
|
||||||
|
secretutil.16=DataX configuration requires encryption and decryption, but the encryption and decryption configuration cannot be found
|
||||||
|
|
||||||
@ -0,0 +1,58 @@
|
|||||||
|
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||||
|
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||||
|
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||||
|
|
||||||
|
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||||
|
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||||
|
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||||
|
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||||
|
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||||
|
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||||
|
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||||
|
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||||
|
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||||
|
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||||
|
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||||
|
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||||
|
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||||
|
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||||
|
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||||
|
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||||
|
|
||||||
@ -0,0 +1,58 @@
|
|||||||
|
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||||
|
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||||
|
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||||
|
|
||||||
|
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||||
|
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||||
|
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||||
|
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||||
|
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||||
|
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||||
|
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||||
|
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||||
|
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||||
|
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||||
|
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||||
|
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||||
|
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||||
|
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||||
|
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||||
|
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||||
|
|
||||||
@ -0,0 +1,116 @@
|
|||||||
|
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||||
|
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||||
|
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||||
|
|
||||||
|
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||||
|
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||||
|
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||||
|
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||||
|
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||||
|
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||||
|
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||||
|
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||||
|
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||||
|
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||||
|
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||||
|
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||||
|
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||||
|
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||||
|
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||||
|
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||||
|
|
||||||
|
configparser.1=\u5916\u639B\u7A0B\u5F0F[{0},{1}]\u8F09\u5165\u5931\u6557\uFF0C1s\u5F8C\u91CD\u8A66... Exception:{2}
|
||||||
|
configparser.2=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||||
|
configparser.3=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||||
|
configparser.4=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||||
|
configparser.5=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u5916\u639B\u7A0B\u5F0F\u8F09\u5165:{0}
|
||||||
|
configparser.6=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557,\u5B58\u5728\u91CD\u8907\u5916\u639B\u7A0B\u5F0F:{0}
|
||||||
|
|
||||||
|
dataxserviceutil.1=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u9650\u5236\u61C9\u8A72\u5728[0.0, 1.0]\u4E4B\u9593
|
||||||
|
errorrecordchecker.2=\u9AD2\u6578\u64DA\u689D\u6578\u73FE\u5728\u61C9\u8A72\u70BA\u975E\u8CA0\u6574\u6578
|
||||||
|
errorrecordchecker.3=\u9AD2\u6578\u64DA\u689D\u6578\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\u689D\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u4E86[{1}]\u689D.
|
||||||
|
errorrecordchecker.4=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u5230[{1}].
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88DD\u932F\u8AA4, \u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.argument_error=DataX\u5F15\u64CE\u904B\u884C\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BC\u5167\u90E8\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A .
|
||||||
|
errorcode.runtime_error=DataX\u5F15\u64CE\u904B\u884C\u904E\u7A0B\u51FA\u932F\uFF0C\u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||||
|
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u932F\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.hook_load_error=\u8F09\u5165\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4\uFF0C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u5F15\u8D77\u7684
|
||||||
|
errorcode.hook_fail_error=\u57F7\u884C\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4
|
||||||
|
errorcode.plugin_install_error=DataX\u5916\u639B\u7A0B\u5F0F\u5B89\u88DD\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.plugin_not_found=DataX\u5916\u639B\u7A0B\u5F0F\u914D\u7F6E\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.plugin_init_error=DataX\u5916\u639B\u7A0B\u5F0F\u521D\u59CB\u5316\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.plugin_runtime_error=DataX\u5916\u639B\u7A0B\u5F0F\u904B\u884C\u6642\u51FA\u932F, \u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=DataX\u50B3\u8F38\u9AD2\u6578\u64DA\u8D85\u904E\u7528\u6236\u9810\u671F\uFF0C\u8A72\u932F\u8AA4\u901A\u5E38\u662F\u7531\u65BC\u6E90\u7AEF\u6578\u64DA\u5B58\u5728\u8F03\u591A\u696D\u52D9\u9AD2\u6578\u64DA\u5C0E\u81F4\uFF0C\u8ACB\u4ED4\u7D30\u6AA2\u67E5DataX\u5F59\u5831\u7684\u9AD2\u6578\u64DA\u65E5\u8A8C\u8CC7\u8A0A, \u6216\u8005\u60A8\u53EF\u4EE5\u9069\u7576\u8ABF\u5927\u9AD2\u6578\u64DA\u95BE\u503C .
|
||||||
|
errorcode.plugin_split_error=DataX\u5916\u639B\u7A0B\u5F0F\u5207\u5206\u51FA\u932F, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5404\u500B\u5916\u639B\u7A0B\u5F0F\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||||
|
errorcode.kill_job_timeout_error=kill \u4EFB\u52D9\u903E\u6642\uFF0C\u8ACB\u806F\u7D61PE\u89E3\u6C7A
|
||||||
|
errorcode.start_taskgroup_error=taskGroup\u555F\u52D5\u5931\u6557,\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||||
|
errorcode.call_datax_service_failed=\u8ACB\u6C42 DataX Service \u51FA\u932F.
|
||||||
|
errorcode.call_remote_failed=\u9060\u7A0B\u8ABF\u7528\u5931\u6557
|
||||||
|
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8ACB\u6C42\u5730\u5740\uFF1A{0}, \u8ACB\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u9060\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C07\u91CD\u8A66
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||||
|
secretutil.2=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||||
|
secretutil.3=rsa\u52A0\u5BC6\u51FA\u932F
|
||||||
|
secretutil.4=rsa\u89E3\u5BC6\u51FA\u932F
|
||||||
|
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u932F
|
||||||
|
secretutil.6=rsa\u89E3\u5BC6\u51FA\u932F
|
||||||
|
secretutil.7=\u69CB\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u932F
|
||||||
|
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u5BC6\u9470\u7684\u914D\u7F6E\u6A94\u6848
|
||||||
|
secretutil.9=\u8B80\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6A94\u6848\u51FA\u932F
|
||||||
|
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||||
|
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||||
|
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||||
|
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||||
|
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u9470\u70BA\u7A7A\u7684\u60C5\u6CC1
|
||||||
|
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u9470\u5C0D\u5B58\u5728\u70BA\u7A7A\u7684\u60C5\u6CC1\uFF0C\u7248\u672C[{0}]
|
||||||
|
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||||
|
|
||||||
@ -0,0 +1,116 @@
|
|||||||
|
configparser.1=\u63D2\u4EF6[{0},{1}]\u52A0\u8F7D\u5931\u8D25\uFF0C1s\u540E\u91CD\u8BD5... Exception:{2}
|
||||||
|
configparser.2=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.3=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.4=\u83B7\u53D6\u4F5C\u4E1A\u914D\u7F6E\u4FE1\u606F\u5931\u8D25:{0}
|
||||||
|
configparser.5=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u63D2\u4EF6\u52A0\u8F7D:{0}
|
||||||
|
configparser.6=\u63D2\u4EF6\u52A0\u8F7D\u5931\u8D25,\u5B58\u5728\u91CD\u590D\u63D2\u4EF6:{0}
|
||||||
|
|
||||||
|
dataxserviceutil.1=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=\u521B\u5EFA\u7B7E\u540D\u5F02\u5E38UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=\u810F\u6570\u636E\u767E\u5206\u6BD4\u9650\u5236\u5E94\u8BE5\u5728[0.0, 1.0]\u4E4B\u95F4
|
||||||
|
errorrecordchecker.2=\u810F\u6570\u636E\u6761\u6570\u73B0\u5728\u5E94\u8BE5\u4E3A\u975E\u8D1F\u6574\u6570
|
||||||
|
errorrecordchecker.3=\u810F\u6570\u636E\u6761\u6570\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\u6761\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u4E86[{1}]\u6761.
|
||||||
|
errorrecordchecker.4=\u810F\u6570\u636E\u767E\u5206\u6BD4\u68C0\u67E5\u4E0D\u901A\u8FC7\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5B9E\u9645\u4E0A\u6355\u83B7\u5230[{1}].
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88C5\u9519\u8BEF, \u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.argument_error=DataX\u5F15\u64CE\u8FD0\u884C\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8E\u5185\u90E8\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3 .
|
||||||
|
errorcode.runtime_error=DataX\u5F15\u64CE\u8FD0\u884C\u8FC7\u7A0B\u51FA\u9519\uFF0C\u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u9519\u8BEF\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u9519\uFF0C\u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.hook_load_error=\u52A0\u8F7D\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF\uFF0C\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u5F15\u8D77\u7684
|
||||||
|
errorcode.hook_fail_error=\u6267\u884C\u5916\u90E8Hook\u51FA\u73B0\u9519\u8BEF
|
||||||
|
errorcode.plugin_install_error=DataX\u63D2\u4EF6\u5B89\u88C5\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_not_found=DataX\u63D2\u4EF6\u914D\u7F6E\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_init_error=DataX\u63D2\u4EF6\u521D\u59CB\u5316\u9519\u8BEF, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5B89\u88C5\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFB\u60A8\u7684\u8FD0\u7EF4\u89E3\u51B3 .
|
||||||
|
errorcode.plugin_runtime_error=DataX\u63D2\u4EF6\u8FD0\u884C\u65F6\u51FA\u9519, \u5177\u4F53\u539F\u56E0\u8BF7\u53C2\u770BDataX\u8FD0\u884C\u7ED3\u675F\u65F6\u7684\u9519\u8BEF\u8BCA\u65AD\u4FE1\u606F .
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=DataX\u4F20\u8F93\u810F\u6570\u636E\u8D85\u8FC7\u7528\u6237\u9884\u671F\uFF0C\u8BE5\u9519\u8BEF\u901A\u5E38\u662F\u7531\u4E8E\u6E90\u7AEF\u6570\u636E\u5B58\u5728\u8F83\u591A\u4E1A\u52A1\u810F\u6570\u636E\u5BFC\u81F4\uFF0C\u8BF7\u4ED4\u7EC6\u68C0\u67E5DataX\u6C47\u62A5\u7684\u810F\u6570\u636E\u65E5\u5FD7\u4FE1\u606F, \u6216\u8005\u60A8\u53EF\u4EE5\u9002\u5F53\u8C03\u5927\u810F\u6570\u636E\u9608\u503C .
|
||||||
|
errorcode.plugin_split_error=DataX\u63D2\u4EF6\u5207\u5206\u51FA\u9519, \u8BE5\u95EE\u9898\u901A\u5E38\u662F\u7531\u4E8EDataX\u5404\u4E2A\u63D2\u4EF6\u7F16\u7A0B\u9519\u8BEF\u5F15\u8D77\uFF0C\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.kill_job_timeout_error=kill \u4EFB\u52A1\u8D85\u65F6\uFF0C\u8BF7\u8054\u7CFBPE\u89E3\u51B3
|
||||||
|
errorcode.start_taskgroup_error=taskGroup\u542F\u52A8\u5931\u8D25,\u8BF7\u8054\u7CFBDataX\u5F00\u53D1\u56E2\u961F\u89E3\u51B3
|
||||||
|
errorcode.call_datax_service_failed=\u8BF7\u6C42 DataX Service \u51FA\u9519.
|
||||||
|
errorcode.call_remote_failed=\u8FDC\u7A0B\u8C03\u7528\u5931\u8D25
|
||||||
|
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8BF7\u6C42\u5730\u5740\uFF1A{0}, \u8BF7\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u8FDC\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C06\u91CD\u8BD5
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.2=\u7CFB\u7EDF\u7F16\u7A0B\u9519\u8BEF,\u4E0D\u652F\u6301\u7684\u52A0\u5BC6\u7C7B\u578B
|
||||||
|
secretutil.3=rsa\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.4=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u9519
|
||||||
|
secretutil.6=rsa\u89E3\u5BC6\u51FA\u9519
|
||||||
|
secretutil.7=\u6784\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u9519
|
||||||
|
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u5BC6\u94A5\u7684\u914D\u7F6E\u6587\u4EF6
|
||||||
|
secretutil.9=\u8BFB\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6587\u4EF6\u51FA\u9519
|
||||||
|
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C
|
||||||
|
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C\u4E3A[{0}]\uFF0C\u4F46\u5728\u7CFB\u7EDF\u4E2D\u6CA1\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52A1\u5BC6\u94A5\u914D\u7F6E\u9519\u8BEF\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7EDF\u7EF4\u62A4\u95EE\u9898
|
||||||
|
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u94A5\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u94A5\u4E3A\u7A7A\u7684\u60C5\u51B5
|
||||||
|
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u94A5\u5BF9\u5B58\u5728\u4E3A\u7A7A\u7684\u60C5\u51B5\uFF0C\u7248\u672C[{0}]
|
||||||
|
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u65E0\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||||
|
|
||||||
|
configparser.1=\u5916\u639B\u7A0B\u5F0F[{0},{1}]\u8F09\u5165\u5931\u6557\uFF0C1s\u5F8C\u91CD\u8A66... Exception:{2}
|
||||||
|
configparser.2=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||||
|
configparser.3=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||||
|
configparser.4=\u7372\u53D6\u4F5C\u696D\u914D\u7F6E\u8CC7\u8A0A\u5931\u6557:{0}
|
||||||
|
configparser.5=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557\uFF0C\u672A\u5B8C\u6210\u6307\u5B9A\u5916\u639B\u7A0B\u5F0F\u8F09\u5165:{0}
|
||||||
|
configparser.6=\u5916\u639B\u7A0B\u5F0F\u8F09\u5165\u5931\u6557,\u5B58\u5728\u91CD\u8907\u5916\u639B\u7A0B\u5F0F:{0}
|
||||||
|
|
||||||
|
dataxserviceutil.1=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38NoSuchAlgorithmException, [{0}]
|
||||||
|
dataxserviceutil.2=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38InvalidKeyException, [{0}]
|
||||||
|
dataxserviceutil.3=\u5EFA\u7ACB\u7C3D\u540D\u7570\u5E38UnsupportedEncodingException, [{0}]
|
||||||
|
|
||||||
|
errorrecordchecker.1=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u9650\u5236\u61C9\u8A72\u5728[0.0, 1.0]\u4E4B\u9593
|
||||||
|
errorrecordchecker.2=\u9AD2\u6578\u64DA\u689D\u6578\u73FE\u5728\u61C9\u8A72\u70BA\u975E\u8CA0\u6574\u6578
|
||||||
|
errorrecordchecker.3=\u9AD2\u6578\u64DA\u689D\u6578\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\u689D\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u4E86[{1}]\u689D.
|
||||||
|
errorrecordchecker.4=\u9AD2\u6578\u64DA\u767E\u5206\u6BD4\u6AA2\u67E5\u4E0D\u901A\u904E\uFF0C\u9650\u5236\u662F[{0}]\uFF0C\u4F46\u5BE6\u969B\u4E0A\u6355\u7372\u5230[{1}].
|
||||||
|
|
||||||
|
|
||||||
|
errorcode.install_error=DataX\u5F15\u64CE\u5B89\u88DD\u932F\u8AA4, \u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.argument_error=DataX\u5F15\u64CE\u904B\u884C\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BC\u5167\u90E8\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A .
|
||||||
|
errorcode.runtime_error=DataX\u5F15\u64CE\u904B\u884C\u904E\u7A0B\u51FA\u932F\uFF0C\u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||||
|
errorcode.config_error=DataX\u5F15\u64CE\u914D\u7F6E\u932F\u8AA4\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.secret_error=DataX\u5F15\u64CE\u52A0\u89E3\u5BC6\u51FA\u932F\uFF0C\u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.hook_load_error=\u8F09\u5165\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4\uFF0C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u5F15\u8D77\u7684
|
||||||
|
errorcode.hook_fail_error=\u57F7\u884C\u5916\u90E8Hook\u51FA\u73FE\u932F\u8AA4
|
||||||
|
errorcode.plugin_install_error=DataX\u5916\u639B\u7A0B\u5F0F\u5B89\u88DD\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.plugin_not_found=DataX\u5916\u639B\u7A0B\u5F0F\u914D\u7F6E\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.plugin_init_error=DataX\u5916\u639B\u7A0B\u5F0F\u521D\u59CB\u5316\u932F\u8AA4, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5B89\u88DD\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61\u60A8\u7684\u904B\u7DAD\u89E3\u6C7A .
|
||||||
|
errorcode.plugin_runtime_error=DataX\u5916\u639B\u7A0B\u5F0F\u904B\u884C\u6642\u51FA\u932F, \u5177\u9AD4\u539F\u56E0\u8ACB\u53C3\u770BDataX\u904B\u884C\u7D50\u675F\u6642\u7684\u932F\u8AA4\u8A3A\u65B7\u8CC7\u8A0A .
|
||||||
|
errorcode.plugin_dirty_data_limit_exceed=DataX\u50B3\u8F38\u9AD2\u6578\u64DA\u8D85\u904E\u7528\u6236\u9810\u671F\uFF0C\u8A72\u932F\u8AA4\u901A\u5E38\u662F\u7531\u65BC\u6E90\u7AEF\u6578\u64DA\u5B58\u5728\u8F03\u591A\u696D\u52D9\u9AD2\u6578\u64DA\u5C0E\u81F4\uFF0C\u8ACB\u4ED4\u7D30\u6AA2\u67E5DataX\u5F59\u5831\u7684\u9AD2\u6578\u64DA\u65E5\u8A8C\u8CC7\u8A0A, \u6216\u8005\u60A8\u53EF\u4EE5\u9069\u7576\u8ABF\u5927\u9AD2\u6578\u64DA\u95BE\u503C .
|
||||||
|
errorcode.plugin_split_error=DataX\u5916\u639B\u7A0B\u5F0F\u5207\u5206\u51FA\u932F, \u8A72\u554F\u984C\u901A\u5E38\u662F\u7531\u65BCDataX\u5404\u500B\u5916\u639B\u7A0B\u5F0F\u7DE8\u7A0B\u932F\u8AA4\u5F15\u8D77\uFF0C\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||||
|
errorcode.kill_job_timeout_error=kill \u4EFB\u52D9\u903E\u6642\uFF0C\u8ACB\u806F\u7D61PE\u89E3\u6C7A
|
||||||
|
errorcode.start_taskgroup_error=taskGroup\u555F\u52D5\u5931\u6557,\u8ACB\u806F\u7D61DataX\u958B\u767C\u5718\u968A\u89E3\u6C7A
|
||||||
|
errorcode.call_datax_service_failed=\u8ACB\u6C42 DataX Service \u51FA\u932F.
|
||||||
|
errorcode.call_remote_failed=\u9060\u7A0B\u8ABF\u7528\u5931\u6557
|
||||||
|
errorcode.killed_exit_value=Job \u6536\u5230\u4E86 Kill \u547D\u4EE4.
|
||||||
|
|
||||||
|
|
||||||
|
httpclientutil.1=\u8ACB\u6C42\u5730\u5740\uFF1A{0}, \u8ACB\u6C42\u65B9\u6CD5\uFF1A{1},STATUS CODE = {2}, Response Entity: {3}
|
||||||
|
httpclientutil.2=\u9060\u7A0B\u63A5\u53E3\u8FD4\u56DE-1,\u5C07\u91CD\u8A66
|
||||||
|
|
||||||
|
|
||||||
|
secretutil.1=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||||
|
secretutil.2=\u7CFB\u7D71\u7DE8\u7A0B\u932F\u8AA4,\u4E0D\u652F\u63F4\u7684\u52A0\u5BC6\u985E\u578B
|
||||||
|
secretutil.3=rsa\u52A0\u5BC6\u51FA\u932F
|
||||||
|
secretutil.4=rsa\u89E3\u5BC6\u51FA\u932F
|
||||||
|
secretutil.5=3\u91CDDES\u52A0\u5BC6\u51FA\u932F
|
||||||
|
secretutil.6=rsa\u89E3\u5BC6\u51FA\u932F
|
||||||
|
secretutil.7=\u69CB\u5EFA\u4E09\u91CDDES\u5BC6\u5319\u51FA\u932F
|
||||||
|
secretutil.8=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u5BC6\u9470\u7684\u914D\u7F6E\u6A94\u6848
|
||||||
|
secretutil.9=\u8B80\u53D6\u52A0\u89E3\u5BC6\u914D\u7F6E\u6A94\u6848\u51FA\u932F
|
||||||
|
secretutil.10=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||||
|
secretutil.11=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||||
|
secretutil.12=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E0D\u5B58\u5728\u60A8\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C
|
||||||
|
secretutil.13=DataX\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C\u70BA[{0}]\uFF0C\u4F46\u5728\u7CFB\u7D71\u4E2D\u6C92\u6709\u914D\u7F6E\uFF0C\u53EF\u80FD\u662F\u4EFB\u52D9\u5BC6\u9470\u914D\u7F6E\u932F\u8AA4\uFF0C\u4E5F\u53EF\u80FD\u662F\u7CFB\u7D71\u7DAD\u8B77\u554F\u984C
|
||||||
|
secretutil.14=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u5BC6\u9470\u7248\u672C[{0}]\u5B58\u5728\u5BC6\u9470\u70BA\u7A7A\u7684\u60C5\u6CC1
|
||||||
|
secretutil.15=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u914D\u7F6E\u7684\u516C\u79C1\u9470\u5C0D\u5B58\u5728\u70BA\u7A7A\u7684\u60C5\u6CC1\uFF0C\u7248\u672C[{0}]
|
||||||
|
secretutil.16=DataX\u914D\u7F6E\u8981\u6C42\u52A0\u89E3\u5BC6\uFF0C\u4F46\u7121\u6CD5\u627E\u5230\u52A0\u89E3\u5BC6\u914D\u7F6E
|
||||||
|
|
||||||
@ -50,7 +50,7 @@ DRDS的插件目前DataX只适配了Mysql引擎的场景，DRDS对于DataX而言
|
|||||||
// 数据库连接密码
|
// 数据库连接密码
|
||||||
"password": "root",
|
"password": "root",
|
||||||
"column": [
|
"column": [
|
||||||
"id"，"name"
|
"id","name"
|
||||||
],
|
],
|
||||||
"connection": [
|
"connection": [
|
||||||
{
|
{
|
||||||
|
|||||||
@ -42,7 +42,7 @@
|
|||||||
<dependency>
|
<dependency>
|
||||||
<groupId>mysql</groupId>
|
<groupId>mysql</groupId>
|
||||||
<artifactId>mysql-connector-java</artifactId>
|
<artifactId>mysql-connector-java</artifactId>
|
||||||
<version>5.1.34</version>
|
<version>${mysql.driver.version}</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -44,7 +44,7 @@
|
|||||||
<dependency>
|
<dependency>
|
||||||
<groupId>mysql</groupId>
|
<groupId>mysql</groupId>
|
||||||
<artifactId>mysql-connector-java</artifactId>
|
<artifactId>mysql-connector-java</artifactId>
|
||||||
<version>5.1.34</version>
|
<version>${mysql.driver.version}</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
</dependencies>
|
</dependencies>
|
||||||
|
|
||||||
|
|||||||
@ -2,6 +2,6 @@
|
|||||||
"name": "hbase11xsqlreader",
|
"name": "hbase11xsqlreader",
|
||||||
"class": "com.alibaba.datax.plugin.reader.hbase11xsqlreader.HbaseSQLReader",
|
"class": "com.alibaba.datax.plugin.reader.hbase11xsqlreader.HbaseSQLReader",
|
||||||
"description": "useScene: prod. mechanism: Scan to read data.",
|
"description": "useScene: prod. mechanism: Scan to read data.",
|
||||||
"developer": "liwei.li, bug reported to : liwei.li@alibaba-inc.com"
|
"developer": "alibaba"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -203,19 +203,20 @@ HbaseWriter 插件实现了从向Hbase中写取数据。在底层实现上，Hba
|
|||||||
* 描述：要写入的hbase字段。index：指定该列对应reader端column的索引，从0开始；name：指定hbase表中的列，必须为 列族:列名 的格式；type：指定写入数据类型，用于转换HBase byte[]。配置格式如下：
|
* 描述：要写入的hbase字段。index：指定该列对应reader端column的索引，从0开始；name：指定hbase表中的列，必须为 列族:列名 的格式；type：指定写入数据类型，用于转换HBase byte[]。配置格式如下：
|
||||||
|
|
||||||
```
|
```
|
||||||
"column": [
|
|
||||||
{
|
"column": [
|
||||||
"index":1,
|
{
|
||||||
"name": "cf1:q1",
|
"index":1,
|
||||||
"type": "string"
|
"name": "cf1:q1",
|
||||||
},
|
"type": "string"
|
||||||
{
|
},
|
||||||
"index":2,
|
{
|
||||||
"name": "cf1:q2",
|
"index":2,
|
||||||
"type": "string"
|
"name": "cf1:q2",
|
||||||
}
|
"type": "string"
|
||||||
］
|
}
|
||||||
|
］
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
* 必选：是<br />
|
* 必选：是<br />
|
||||||
@ -227,17 +228,17 @@ HbaseWriter 插件实现了从向Hbase中写取数据。在底层实现上，Hba
|
|||||||
* 描述：要写入的hbase的rowkey列。index：指定该列对应reader端column的索引，从0开始，若为常量index为－1；type：指定写入数据类型，用于转换HBase byte[]；value：配置常量，常作为多个字段的拼接符。hbasewriter会将rowkeyColumn中所有列按照配置顺序进行拼接作为写入hbase的rowkey，不能全为常量。配置格式如下：
|
* 描述：要写入的hbase的rowkey列。index：指定该列对应reader端column的索引，从0开始，若为常量index为－1；type：指定写入数据类型，用于转换HBase byte[]；value：配置常量，常作为多个字段的拼接符。hbasewriter会将rowkeyColumn中所有列按照配置顺序进行拼接作为写入hbase的rowkey，不能全为常量。配置格式如下：
|
||||||
|
|
||||||
```
|
```
|
||||||
"rowkeyColumn": [
|
"rowkeyColumn": [
|
||||||
{
|
{
|
||||||
"index":0,
|
"index":0,
|
||||||
"type":"string"
|
"type":"string"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"index":-1,
|
"index":-1,
|
||||||
"type":"string",
|
"type":"string",
|
||||||
"value":"_"
|
"value":"_"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -250,19 +251,19 @@ HbaseWriter 插件实现了从向Hbase中写取数据。在底层实现上，Hba
|
|||||||
* 描述：指定写入hbase的时间戳。支持：当前时间、指定时间列，指定时间，三者选一。若不配置表示用当前时间。index：指定对应reader端column的索引，从0开始，需保证能转换为long,若是Date类型，会尝试用yyyy-MM-dd HH:mm:ss和yyyy-MM-dd HH:mm:ss SSS去解析；若为指定时间index为－1；value：指定时间的值,long值。配置格式如下：
|
* 描述：指定写入hbase的时间戳。支持：当前时间、指定时间列，指定时间，三者选一。若不配置表示用当前时间。index：指定对应reader端column的索引，从0开始，需保证能转换为long,若是Date类型，会尝试用yyyy-MM-dd HH:mm:ss和yyyy-MM-dd HH:mm:ss SSS去解析；若为指定时间index为－1；value：指定时间的值,long值。配置格式如下：
|
||||||
|
|
||||||
```
|
```
|
||||||
"versionColumn":{
|
"versionColumn":{
|
||||||
"index":1
|
"index":1
|
||||||
}
|
}
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
或者
|
或者
|
||||||
|
|
||||||
```
|
```
|
||||||
"versionColumn":{
|
"versionColumn":{
|
||||||
"index":－1,
|
"index":－1,
|
||||||
"value":123456789
|
"value":123456789
|
||||||
}
|
}
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@ -16,6 +16,17 @@
|
|||||||
<hadoop.version>2.7.1</hadoop.version>
|
<hadoop.version>2.7.1</hadoop.version>
|
||||||
</properties>
|
</properties>
|
||||||
<dependencies>
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.logging.log4j</groupId>
|
||||||
|
<artifactId>log4j-api</artifactId>
|
||||||
|
<version>2.17.1</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.logging.log4j</groupId>
|
||||||
|
<artifactId>log4j-core</artifactId>
|
||||||
|
<version>2.17.1</version>
|
||||||
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>com.alibaba.datax</groupId>
|
<groupId>com.alibaba.datax</groupId>
|
||||||
<artifactId>datax-common</artifactId>
|
<artifactId>datax-common</artifactId>
|
||||||
@ -51,6 +62,11 @@
|
|||||||
<artifactId>hadoop-yarn-common</artifactId>
|
<artifactId>hadoop-yarn-common</artifactId>
|
||||||
<version>${hadoop.version}</version>
|
<version>${hadoop.version}</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.aliyun.oss</groupId>
|
||||||
|
<artifactId>hadoop-aliyun</artifactId>
|
||||||
|
<version>2.7.2</version>
|
||||||
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.apache.hadoop</groupId>
|
<groupId>org.apache.hadoop</groupId>
|
||||||
<artifactId>hadoop-mapreduce-client-core</artifactId>
|
<artifactId>hadoop-mapreduce-client-core</artifactId>
|
||||||
|
|||||||
@ -19,6 +19,17 @@
|
|||||||
</properties>
|
</properties>
|
||||||
|
|
||||||
<dependencies>
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.logging.log4j</groupId>
|
||||||
|
<artifactId>log4j-api</artifactId>
|
||||||
|
<version>2.17.1</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.apache.logging.log4j</groupId>
|
||||||
|
<artifactId>log4j-core</artifactId>
|
||||||
|
<version>2.17.1</version>
|
||||||
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>com.alibaba.datax</groupId>
|
<groupId>com.alibaba.datax</groupId>
|
||||||
<artifactId>datax-common</artifactId>
|
<artifactId>datax-common</artifactId>
|
||||||
@ -30,6 +41,11 @@
|
|||||||
</exclusion>
|
</exclusion>
|
||||||
</exclusions>
|
</exclusions>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.aliyun.oss</groupId>
|
||||||
|
<artifactId>hadoop-aliyun</artifactId>
|
||||||
|
<version>2.7.2</version>
|
||||||
|
</dependency>
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.slf4j</groupId>
|
<groupId>org.slf4j</groupId>
|
||||||
<artifactId>slf4j-api</artifactId>
|
<artifactId>slf4j-api</artifactId>
|
||||||
@ -132,4 +148,4 @@
|
|||||||
</plugin>
|
</plugin>
|
||||||
</plugins>
|
</plugins>
|
||||||
</build>
|
</build>
|
||||||
</project>
|
</project>
|
||||||
|
|||||||
@ -6,10 +6,13 @@ import com.alibaba.datax.common.exception.DataXException;
|
|||||||
import com.alibaba.datax.common.plugin.RecordReceiver;
|
import com.alibaba.datax.common.plugin.RecordReceiver;
|
||||||
import com.alibaba.datax.common.plugin.TaskPluginCollector;
|
import com.alibaba.datax.common.plugin.TaskPluginCollector;
|
||||||
import com.alibaba.datax.common.util.Configuration;
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.unstructuredstorage.util.ColumnTypeUtil;
|
||||||
|
import com.alibaba.datax.plugin.unstructuredstorage.util.HdfsUtil;
|
||||||
import com.alibaba.fastjson.JSON;
|
import com.alibaba.fastjson.JSON;
|
||||||
import com.alibaba.fastjson.JSONObject;
|
import com.alibaba.fastjson.JSONObject;
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.apache.commons.lang3.Validate;
|
||||||
import org.apache.commons.lang3.tuple.MutablePair;
|
import org.apache.commons.lang3.tuple.MutablePair;
|
||||||
import org.apache.hadoop.fs.*;
|
import org.apache.hadoop.fs.*;
|
||||||
import org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat;
|
import org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat;
|
||||||
@ -24,6 +27,10 @@ import org.apache.hadoop.mapred.*;
|
|||||||
import org.apache.hadoop.security.UserGroupInformation;
|
import org.apache.hadoop.security.UserGroupInformation;
|
||||||
import org.slf4j.Logger;
|
import org.slf4j.Logger;
|
||||||
import org.slf4j.LoggerFactory;
|
import org.slf4j.LoggerFactory;
|
||||||
|
import parquet.schema.OriginalType;
|
||||||
|
import parquet.schema.PrimitiveType;
|
||||||
|
import parquet.schema.Types;
|
||||||
|
|
||||||
import java.io.IOException;
|
import java.io.IOException;
|
||||||
import java.text.SimpleDateFormat;
|
import java.text.SimpleDateFormat;
|
||||||
import java.util.*;
|
import java.util.*;
|
||||||
@ -556,4 +563,67 @@ public class HdfsHelper {
|
|||||||
transportResult.setLeft(recordList);
|
transportResult.setLeft(recordList);
|
||||||
return transportResult;
|
return transportResult;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
public static String generateParquetSchemaFromColumnAndType(List<Configuration> columns) {
|
||||||
|
Map<String, ColumnTypeUtil.DecimalInfo> decimalColInfo = new HashMap<>(16);
|
||||||
|
ColumnTypeUtil.DecimalInfo PARQUET_DEFAULT_DECIMAL_INFO = new ColumnTypeUtil.DecimalInfo(10, 2);
|
||||||
|
Types.MessageTypeBuilder typeBuilder = Types.buildMessage();
|
||||||
|
for (Configuration column : columns) {
|
||||||
|
String name = column.getString("name");
|
||||||
|
String colType = column.getString("type");
|
||||||
|
Validate.notNull(name, "column.name can't be null");
|
||||||
|
Validate.notNull(colType, "column.type can't be null");
|
||||||
|
switch (colType.toLowerCase()) {
|
||||||
|
case "tinyint":
|
||||||
|
case "smallint":
|
||||||
|
case "int":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.INT32).named(name);
|
||||||
|
break;
|
||||||
|
case "bigint":
|
||||||
|
case "long":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.INT64).named(name);
|
||||||
|
break;
|
||||||
|
case "float":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.FLOAT).named(name);
|
||||||
|
break;
|
||||||
|
case "double":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.DOUBLE).named(name);
|
||||||
|
break;
|
||||||
|
case "binary":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.BINARY).named(name);
|
||||||
|
break;
|
||||||
|
case "char":
|
||||||
|
case "varchar":
|
||||||
|
case "string":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.BINARY).as(OriginalType.UTF8).named(name);
|
||||||
|
break;
|
||||||
|
case "boolean":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.BOOLEAN).named(name);
|
||||||
|
break;
|
||||||
|
case "timestamp":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.INT96).named(name);
|
||||||
|
break;
|
||||||
|
case "date":
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.INT32).as(OriginalType.DATE).named(name);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
if (ColumnTypeUtil.isDecimalType(colType)) {
|
||||||
|
ColumnTypeUtil.DecimalInfo decimalInfo = ColumnTypeUtil.getDecimalInfo(colType, PARQUET_DEFAULT_DECIMAL_INFO);
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY)
|
||||||
|
.as(OriginalType.DECIMAL)
|
||||||
|
.precision(decimalInfo.getPrecision())
|
||||||
|
.scale(decimalInfo.getScale())
|
||||||
|
.length(HdfsUtil.computeMinBytesForPrecision(decimalInfo.getPrecision()))
|
||||||
|
.named(name);
|
||||||
|
|
||||||
|
decimalColInfo.put(name, decimalInfo);
|
||||||
|
} else {
|
||||||
|
typeBuilder.optional(PrimitiveType.PrimitiveTypeName.BINARY).named(name);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return typeBuilder.named("m").toString();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -9,9 +9,11 @@ import com.google.common.collect.Sets;
|
|||||||
import org.apache.commons.io.Charsets;
|
import org.apache.commons.io.Charsets;
|
||||||
import org.apache.commons.io.IOUtils;
|
import org.apache.commons.io.IOUtils;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.apache.commons.lang3.Validate;
|
||||||
import org.apache.hadoop.fs.Path;
|
import org.apache.hadoop.fs.Path;
|
||||||
import org.slf4j.Logger;
|
import org.slf4j.Logger;
|
||||||
import org.slf4j.LoggerFactory;
|
import org.slf4j.LoggerFactory;
|
||||||
|
import parquet.schema.MessageTypeParser;
|
||||||
|
|
||||||
import java.util.*;
|
import java.util.*;
|
||||||
|
|
||||||
@ -323,8 +325,55 @@ public class HdfsWriter extends Writer {
|
|||||||
}
|
}
|
||||||
return tmpFilePath;
|
return tmpFilePath;
|
||||||
}
|
}
|
||||||
|
public void unitizeParquetConfig(Configuration writerSliceConfig) {
|
||||||
|
String parquetSchema = writerSliceConfig.getString(Key.PARQUET_SCHEMA);
|
||||||
|
if (StringUtils.isNotBlank(parquetSchema)) {
|
||||||
|
LOG.info("parquetSchema has config. use parquetSchema:\n{}", parquetSchema);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
List<Configuration> columns = writerSliceConfig.getListConfiguration(Key.COLUMN);
|
||||||
|
if (columns == null || columns.isEmpty()) {
|
||||||
|
throw DataXException.asDataXException("parquetSchema or column can't be blank!");
|
||||||
|
}
|
||||||
|
|
||||||
|
parquetSchema = generateParquetSchemaFromColumn(columns);
|
||||||
|
// 为了兼容历史逻辑,对之前的逻辑做保留，但是如果配置的时候报错，则走新逻辑
|
||||||
|
try {
|
||||||
|
MessageTypeParser.parseMessageType(parquetSchema);
|
||||||
|
} catch (Throwable e) {
|
||||||
|
LOG.warn("The generated parquetSchema {} is illegal, try to generate parquetSchema in another way", parquetSchema);
|
||||||
|
parquetSchema = HdfsHelper.generateParquetSchemaFromColumnAndType(columns);
|
||||||
|
LOG.info("The last generated parquet schema is {}", parquetSchema);
|
||||||
|
}
|
||||||
|
writerSliceConfig.set(Key.PARQUET_SCHEMA, parquetSchema);
|
||||||
|
LOG.info("dataxParquetMode use default fields.");
|
||||||
|
writerSliceConfig.set(Key.DATAX_PARQUET_MODE, "fields");
|
||||||
|
}
|
||||||
|
|
||||||
|
private String generateParquetSchemaFromColumn(List<Configuration> columns) {
|
||||||
|
StringBuffer parquetSchemaStringBuffer = new StringBuffer();
|
||||||
|
parquetSchemaStringBuffer.append("message m {");
|
||||||
|
for (Configuration column: columns) {
|
||||||
|
String name = column.getString("name");
|
||||||
|
Validate.notNull(name, "column.name can't be null");
|
||||||
|
|
||||||
|
String type = column.getString("type");
|
||||||
|
Validate.notNull(type, "column.type can't be null");
|
||||||
|
|
||||||
|
String parquetColumn = String.format("optional %s %s;", type, name);
|
||||||
|
parquetSchemaStringBuffer.append(parquetColumn);
|
||||||
|
}
|
||||||
|
parquetSchemaStringBuffer.append("}");
|
||||||
|
String parquetSchema = parquetSchemaStringBuffer.toString();
|
||||||
|
LOG.info("generate parquetSchema:\n{}", parquetSchema);
|
||||||
|
return parquetSchema;
|
||||||
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
public static class Task extends Writer.Task {
|
public static class Task extends Writer.Task {
|
||||||
private static final Logger LOG = LoggerFactory.getLogger(Task.class);
|
private static final Logger LOG = LoggerFactory.getLogger(Task.class);
|
||||||
|
|
||||||
|
|||||||
@ -33,4 +33,17 @@ public class Key {
|
|||||||
public static final String KERBEROS_PRINCIPAL = "kerberosPrincipal";
|
public static final String KERBEROS_PRINCIPAL = "kerberosPrincipal";
|
||||||
// hadoop config
|
// hadoop config
|
||||||
public static final String HADOOP_CONFIG = "hadoopConfig";
|
public static final String HADOOP_CONFIG = "hadoopConfig";
|
||||||
|
|
||||||
|
// useOldRawDataTransf
|
||||||
|
public final static String PARQUET_FILE_USE_RAW_DATA_TRANSF = "useRawDataTransf";
|
||||||
|
|
||||||
|
public final static String DATAX_PARQUET_MODE = "dataxParquetMode";
|
||||||
|
|
||||||
|
// hdfs username 默认值 admin
|
||||||
|
public final static String HDFS_USERNAME = "hdfsUsername";
|
||||||
|
|
||||||
|
public static final String PROTECTION = "protection";
|
||||||
|
|
||||||
|
public static final String PARQUET_SCHEMA = "parquetSchema";
|
||||||
|
public static final String PARQUET_MERGE_RESULT = "parquetMergeResult";
|
||||||
}
|
}
|
||||||
|
|||||||
204
hologresjdbcwriter/doc/hologresjdbcwriter.md
Normal file
204
hologresjdbcwriter/doc/hologresjdbcwriter.md
Normal file
@ -0,0 +1,204 @@
|
|||||||
|
# DataX HologresJdbcWriter
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
## 1 快速介绍
|
||||||
|
|
||||||
|
HologresJdbcWriter 插件实现了写入数据到 Hologres目的表的功能。在底层实现上，HologresJdbcWriter通过JDBC连接远程 Hologres 数据库，并执行相应的 insert into ... on conflict sql 语句将数据写入 Hologres，内部会分批次提交入库。
|
||||||
|
|
||||||
|
<br />
|
||||||
|
|
||||||
|
* HologresJdbcWriter 只支持单表同步
|
||||||
|
|
||||||
|
## 2 实现原理
|
||||||
|
|
||||||
|
HologresJdbcWriter 通过 DataX 框架获取 Reader 生成的协议数据，根据你配置生成相应的SQL插入语句
|
||||||
|
|
||||||
|
* `insert into... on conflict `
|
||||||
|
|
||||||
|
|
||||||
|
## 3 功能说明
|
||||||
|
|
||||||
|
### 3.1 配置样例
|
||||||
|
|
||||||
|
* 这里使用一份从内存产生到 HologresJdbcWriter导入的数据。
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"job": {
|
||||||
|
"setting": {
|
||||||
|
"speed": {
|
||||||
|
"channel": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"content": [
|
||||||
|
{
|
||||||
|
"reader": {
|
||||||
|
"name": "streamreader",
|
||||||
|
"parameter": {
|
||||||
|
"column" : [
|
||||||
|
{
|
||||||
|
"value": "DataX",
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": 19880808,
|
||||||
|
"type": "long"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "1988-08-08 08:08:08",
|
||||||
|
"type": "date"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": true,
|
||||||
|
"type": "bool"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "test",
|
||||||
|
"type": "bytes"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sliceRecordCount": 1000
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"writer": {
|
||||||
|
"name": "hologresjdbcwriter",
|
||||||
|
"parameter": {
|
||||||
|
"username": "xx",
|
||||||
|
"password": "xx",
|
||||||
|
"column": [
|
||||||
|
"id",
|
||||||
|
"name"
|
||||||
|
],
|
||||||
|
"preSql": [
|
||||||
|
"delete from test"
|
||||||
|
],
|
||||||
|
"connection": [
|
||||||
|
{
|
||||||
|
"jdbcUrl": "jdbc:postgresql://127.0.0.1:3002/datax",
|
||||||
|
"table": [
|
||||||
|
"test"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"writeMode" : "REPLACE",
|
||||||
|
"client" : {
|
||||||
|
"writeThreadSize" : 3
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
### 3.2 参数说明
|
||||||
|
|
||||||
|
* **jdbcUrl**
|
||||||
|
|
||||||
|
* 描述：目的数据库的 JDBC 连接信息 ,jdbcUrl必须包含在connection配置单元中。
|
||||||
|
|
||||||
|
注意：1、在一个数据库上只能配置一个值。
|
||||||
|
2、jdbcUrl按照PostgreSQL官方规范，并可以填写连接附加参数信息。具体请参看PostgreSQL官方文档或者咨询对应 DBA。
|
||||||
|
|
||||||
|
|
||||||
|
* 必选：是 <br />
|
||||||
|
|
||||||
|
* 默认值：无 <br />
|
||||||
|
|
||||||
|
* **username**
|
||||||
|
|
||||||
|
* 描述：目的数据库的用户名 <br />
|
||||||
|
|
||||||
|
* 必选：是 <br />
|
||||||
|
|
||||||
|
* 默认值：无 <br />
|
||||||
|
|
||||||
|
* **password**
|
||||||
|
|
||||||
|
* 描述：目的数据库的密码 <br />
|
||||||
|
|
||||||
|
* 必选：是 <br />
|
||||||
|
|
||||||
|
* 默认值：无 <br />
|
||||||
|
|
||||||
|
* **table**
|
||||||
|
|
||||||
|
* 描述：目的表的表名称。只支持写入一个表。
|
||||||
|
|
||||||
|
注意：table 和 jdbcUrl 必须包含在 connection 配置单元中
|
||||||
|
|
||||||
|
* 必选：是 <br />
|
||||||
|
|
||||||
|
* 默认值：无 <br />
|
||||||
|
|
||||||
|
* **column**
|
||||||
|
|
||||||
|
* 描述：目的表需要写入数据的字段,字段之间用英文逗号分隔。例如: "column": ["id","name","age"]。如果要依次写入全部列，使用\*表示, 例如: "column": ["\*"]
|
||||||
|
|
||||||
|
注意：1、我们强烈不推荐你这样配置，因为当你目的表字段个数、类型等有改动时，你的任务可能运行不正确或者失败
|
||||||
|
2、此处 column 不能配置任何常量值
|
||||||
|
|
||||||
|
* 必选：是 <br />
|
||||||
|
|
||||||
|
* 默认值：否 <br />
|
||||||
|
|
||||||
|
* **preSql**
|
||||||
|
|
||||||
|
* 描述：写入数据到目的表前，会先执行这里的标准语句。如果 Sql 中有你需要操作到的表名称，请使用 `@table` 表示，这样在实际执行 Sql 语句时，会对变量按照实际表名称进行替换。 <br />
|
||||||
|
|
||||||
|
* 必选：否 <br />
|
||||||
|
|
||||||
|
* 默认值：无 <br />
|
||||||
|
|
||||||
|
* **postSql**
|
||||||
|
|
||||||
|
* 描述：写入数据到目的表后，会执行这里的标准语句。（原理同 preSql ） <br />
|
||||||
|
|
||||||
|
* 必选：否 <br />
|
||||||
|
|
||||||
|
* 默认值：无 <br />
|
||||||
|
|
||||||
|
* **batchSize**
|
||||||
|
|
||||||
|
* 描述：一次性批量提交的记录数大小，该值可以极大减少DataX与HologresJdbcWriter的网络交互次数，并提升整体吞吐量。但是该值设置过大可能会造成DataX运行进程OOM情况。<br />
|
||||||
|
|
||||||
|
* 必选：否 <br />
|
||||||
|
|
||||||
|
* 默认值：512 <br />
|
||||||
|
|
||||||
|
* **writeMode**
|
||||||
|
|
||||||
|
* 描述：当写入hologres有主键表时，控制主键冲突后的策略。REPLACE表示冲突后hologres表的所有字段都被覆盖（未在writer中配置的字段将填充null);UPDATE表示冲突后hologres表writer配置的字段将被覆盖；IGNORE表示冲突后丢弃新数据，不覆盖。 <br />
|
||||||
|
|
||||||
|
* 必选：否 <br />
|
||||||
|
|
||||||
|
* 默认值：REPLACE <br />
|
||||||
|
|
||||||
|
* **client.writeThreadSize**
|
||||||
|
|
||||||
|
* 描述：写入hologres的连接池大小，多个连接将并行写入数据。 <br />
|
||||||
|
|
||||||
|
* 必选：否 <br />
|
||||||
|
|
||||||
|
* 默认值：1 <br />
|
||||||
|
|
||||||
|
### 3.3 类型转换
|
||||||
|
|
||||||
|
目前 HologresJdbcWriter支持大部分 Hologres类型，但也存在部分没有支持的情况，请注意检查你的类型。
|
||||||
|
|
||||||
|
下面列出 HologresJdbcWriter针对 Hologres类型转换列表:
|
||||||
|
|
||||||
|
| DataX 内部类型| Hologres 数据类型 |
|
||||||
|
| -------- | ----- |
|
||||||
|
| Long |bigint, integer, smallint |
|
||||||
|
| Double |double precision, money, numeric, real |
|
||||||
|
| String |varchar, char, text, bit|
|
||||||
|
| Date |date, time, timestamp |
|
||||||
|
| Boolean |bool|
|
||||||
|
| Bytes |bytea|
|
||||||
90
hologresjdbcwriter/pom.xml
Normal file
90
hologresjdbcwriter/pom.xml
Normal file
@ -0,0 +1,90 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<parent>
|
||||||
|
<artifactId>datax-all</artifactId>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<version>0.0.1-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<artifactId>hologresjdbcwriter</artifactId>
|
||||||
|
<name>hologresjdbcwriter</name>
|
||||||
|
<packaging>jar</packaging>
|
||||||
|
<description>writer data into hologres using jdbc</description>
|
||||||
|
|
||||||
|
<properties>
|
||||||
|
<jdk-version>1.8
|
||||||
|
|
||||||
|
</jdk-version>
|
||||||
|
</properties>
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<artifactId>datax-common</artifactId>
|
||||||
|
<version>${datax-project-version}</version>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<artifactId>slf4j-log4j12</artifactId>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
<artifactId>slf4j-api</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>ch.qos.logback</groupId>
|
||||||
|
<artifactId>logback-classic</artifactId>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<artifactId>plugin-rdbms-util</artifactId>
|
||||||
|
<version>${datax-project-version}</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alibaba.hologres</groupId>
|
||||||
|
<artifactId>holo-client</artifactId>
|
||||||
|
<version>2.1.0</version>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
</dependencies>
|
||||||
|
<build>
|
||||||
|
<plugins>
|
||||||
|
<!-- compiler plugin -->
|
||||||
|
<plugin>
|
||||||
|
<artifactId>maven-compiler-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<source>${jdk-version}</source>
|
||||||
|
<target>${jdk-version}</target>
|
||||||
|
<encoding>${project-sourceEncoding}</encoding>
|
||||||
|
</configuration>
|
||||||
|
</plugin>
|
||||||
|
<!-- assembly plugin -->
|
||||||
|
<plugin>
|
||||||
|
<artifactId>maven-assembly-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<descriptors>
|
||||||
|
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||||
|
</descriptors>
|
||||||
|
<finalName>datax</finalName>
|
||||||
|
</configuration>
|
||||||
|
<executions>
|
||||||
|
<execution>
|
||||||
|
<id>dwzip</id>
|
||||||
|
<phase>package</phase>
|
||||||
|
<goals>
|
||||||
|
<goal>single</goal>
|
||||||
|
</goals>
|
||||||
|
</execution>
|
||||||
|
</executions>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</build>
|
||||||
|
</project>
|
||||||
35
hologresjdbcwriter/src/main/assembly/package.xml
Executable file
35
hologresjdbcwriter/src/main/assembly/package.xml
Executable file
@ -0,0 +1,35 @@
|
|||||||
|
<assembly
|
||||||
|
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||||
|
<id></id>
|
||||||
|
<formats>
|
||||||
|
<format>dir</format>
|
||||||
|
</formats>
|
||||||
|
<includeBaseDirectory>false</includeBaseDirectory>
|
||||||
|
<fileSets>
|
||||||
|
<fileSet>
|
||||||
|
<directory>src/main/resources</directory>
|
||||||
|
<includes>
|
||||||
|
<include>plugin.json</include>
|
||||||
|
<include>plugin_job_template.json</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/writer/hologresjdbcwriter</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>target/</directory>
|
||||||
|
<includes>
|
||||||
|
<include>hologresjdbcwriter-0.0.1-SNAPSHOT.jar</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/writer/hologresjdbcwriter</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
</fileSets>
|
||||||
|
|
||||||
|
<dependencySets>
|
||||||
|
<dependencySet>
|
||||||
|
<useProjectArtifact>false</useProjectArtifact>
|
||||||
|
<outputDirectory>plugin/writer/hologresjdbcwriter/libs</outputDirectory>
|
||||||
|
<scope>runtime</scope>
|
||||||
|
</dependencySet>
|
||||||
|
</dependencySets>
|
||||||
|
</assembly>
|
||||||
@ -0,0 +1,526 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.hologresjdbcwriter;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.element.Column;
|
||||||
|
import com.alibaba.datax.common.element.DateColumn;
|
||||||
|
import com.alibaba.datax.common.element.LongColumn;
|
||||||
|
import com.alibaba.datax.common.element.Record;
|
||||||
|
import com.alibaba.datax.common.exception.DataXException;
|
||||||
|
import com.alibaba.datax.common.plugin.RecordReceiver;
|
||||||
|
import com.alibaba.datax.common.plugin.TaskPluginCollector;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.common.util.RetryUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtilErrorCode;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
import com.alibaba.datax.plugin.writer.hologresjdbcwriter.util.ConfLoader;
|
||||||
|
import com.alibaba.datax.plugin.writer.hologresjdbcwriter.util.OriginalConfPretreatmentUtil;
|
||||||
|
import com.alibaba.datax.plugin.writer.hologresjdbcwriter.util.WriterUtil;
|
||||||
|
import com.alibaba.fastjson.JSONArray;
|
||||||
|
import com.alibaba.fastjson.JSONObject;
|
||||||
|
import com.alibaba.hologres.client.HoloClient;
|
||||||
|
import com.alibaba.hologres.client.HoloConfig;
|
||||||
|
import com.alibaba.hologres.client.Put;
|
||||||
|
import com.alibaba.hologres.client.exception.HoloClientWithDetailsException;
|
||||||
|
import com.alibaba.hologres.client.model.TableSchema;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.sql.DriverManager;
|
||||||
|
import java.sql.SQLException;
|
||||||
|
import java.sql.Time;
|
||||||
|
import java.sql.Timestamp;
|
||||||
|
import java.sql.Types;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Set;
|
||||||
|
|
||||||
|
public class BaseWriter {
|
||||||
|
|
||||||
|
protected static final Set<String> ignoreConfList;
|
||||||
|
|
||||||
|
static {
|
||||||
|
ignoreConfList = new HashSet<>();
|
||||||
|
ignoreConfList.add("jdbcUrl");
|
||||||
|
ignoreConfList.add("username");
|
||||||
|
ignoreConfList.add("password");
|
||||||
|
ignoreConfList.add("writeMode");
|
||||||
|
}
|
||||||
|
|
||||||
|
enum WriteMode {
|
||||||
|
IGNORE,
|
||||||
|
UPDATE,
|
||||||
|
REPLACE
|
||||||
|
}
|
||||||
|
|
||||||
|
private static WriteMode getWriteMode(String text) {
|
||||||
|
text = text.toUpperCase();
|
||||||
|
switch (text) {
|
||||||
|
case "IGNORE":
|
||||||
|
return WriteMode.IGNORE;
|
||||||
|
case "UPDATE":
|
||||||
|
return WriteMode.UPDATE;
|
||||||
|
case "REPLACE":
|
||||||
|
return WriteMode.REPLACE;
|
||||||
|
default:
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.ILLEGAL_VALUE, "writeMode只支持IGNORE,UPDATE,REPLACE,无法识别 " + text);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static class Job {
|
||||||
|
private DataBaseType dataBaseType;
|
||||||
|
|
||||||
|
private static final Logger LOG = LoggerFactory
|
||||||
|
.getLogger(BaseWriter.Job.class);
|
||||||
|
|
||||||
|
public Job(DataBaseType dataBaseType) {
|
||||||
|
this.dataBaseType = dataBaseType;
|
||||||
|
OriginalConfPretreatmentUtil.DATABASE_TYPE = this.dataBaseType;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void init(Configuration originalConfig) {
|
||||||
|
OriginalConfPretreatmentUtil.doPretreatment(originalConfig, this.dataBaseType);
|
||||||
|
checkConf(originalConfig);
|
||||||
|
LOG.debug("After job init(), originalConfig now is:[\n{}\n]",
|
||||||
|
originalConfig.toJSON());
|
||||||
|
}
|
||||||
|
|
||||||
|
private void checkConf(Configuration originalConfig) {
|
||||||
|
getWriteMode(originalConfig.getString(Key.WRITE_MODE, "REPLACE"));
|
||||||
|
List<String> userConfiguredColumns = originalConfig.getList(Key.COLUMN, String.class);
|
||||||
|
List<JSONObject> conns = originalConfig.getList(Constant.CONN_MARK,
|
||||||
|
JSONObject.class);
|
||||||
|
if (conns.size() > 1) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.ILLEGAL_VALUE, "只支持单表同步");
|
||||||
|
}
|
||||||
|
int tableNumber = originalConfig.getInt(Constant.TABLE_NUMBER_MARK);
|
||||||
|
if (tableNumber > 1) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.ILLEGAL_VALUE, "只支持单表同步");
|
||||||
|
}
|
||||||
|
JSONObject connConf = conns.get(0);
|
||||||
|
String jdbcUrl = connConf.getString(Key.JDBC_URL);
|
||||||
|
String username = originalConfig.getString(Key.USERNAME);
|
||||||
|
String password = originalConfig.getString(Key.PASSWORD);
|
||||||
|
|
||||||
|
String table = connConf.getJSONArray(Key.TABLE).getString(0);
|
||||||
|
|
||||||
|
Map<String, Object> clientConf = originalConfig.getMap("client");
|
||||||
|
|
||||||
|
HoloConfig config = new HoloConfig();
|
||||||
|
config.setJdbcUrl(jdbcUrl);
|
||||||
|
config.setUsername(username);
|
||||||
|
config.setPassword(password);
|
||||||
|
if (clientConf != null) {
|
||||||
|
try {
|
||||||
|
config = ConfLoader.load(clientConf, config, ignoreConfList);
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw DataXException
|
||||||
|
.asDataXException(
|
||||||
|
DBUtilErrorCode.CONF_ERROR,
|
||||||
|
"配置解析失败.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
try (HoloClient client = new HoloClient(config)) {
|
||||||
|
TableSchema schema = client.getTableSchema(table);
|
||||||
|
LOG.info("table {} column info:", schema.getTableNameObj().getFullName());
|
||||||
|
for (com.alibaba.hologres.client.model.Column column : schema.getColumnSchema()) {
|
||||||
|
LOG.info("name:{},type:{},typeName:{},nullable:{},defaultValue:{}", column.getName(), column.getType(), column.getTypeName(), column.getAllowNull(), column.getDefaultValue());
|
||||||
|
}
|
||||||
|
for (String userColumn : userConfiguredColumns) {
|
||||||
|
if (schema.getColumnIndex(userColumn) == null) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.CONF_ERROR, "配置的列 " + userColumn + " 不存在");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.CONN_DB_ERROR, "获取表schema失败", e);
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
// 一般来说，是需要推迟到 task 中进行pre 的执行（单表情况例外）
|
||||||
|
public void prepare(Configuration originalConfig) {
|
||||||
|
|
||||||
|
try {
|
||||||
|
String username = originalConfig.getString(Key.USERNAME);
|
||||||
|
String password = originalConfig.getString(Key.PASSWORD);
|
||||||
|
|
||||||
|
List<Object> conns = originalConfig.getList(Constant.CONN_MARK,
|
||||||
|
Object.class);
|
||||||
|
Configuration connConf = Configuration.from(conns.get(0)
|
||||||
|
.toString());
|
||||||
|
|
||||||
|
String jdbcUrl = connConf.getString(Key.JDBC_URL);
|
||||||
|
originalConfig.set(Key.JDBC_URL, jdbcUrl);
|
||||||
|
|
||||||
|
String table = connConf.getList(Key.TABLE, String.class).get(0);
|
||||||
|
originalConfig.set(Key.TABLE, table);
|
||||||
|
|
||||||
|
List<String> preSqls = originalConfig.getList(Key.PRE_SQL,
|
||||||
|
String.class);
|
||||||
|
List<String> renderedPreSqls = WriterUtil.renderPreOrPostSqls(
|
||||||
|
preSqls, table);
|
||||||
|
|
||||||
|
originalConfig.remove(Constant.CONN_MARK);
|
||||||
|
if (null != renderedPreSqls && !renderedPreSqls.isEmpty()) {
|
||||||
|
// 说明有 preSql 配置，则此处删除掉
|
||||||
|
originalConfig.remove(Key.PRE_SQL);
|
||||||
|
String tempJdbcUrl = jdbcUrl.replace("postgresql", "hologres");
|
||||||
|
try (Connection conn = DriverManager.getConnection(
|
||||||
|
tempJdbcUrl, username, password)) {
|
||||||
|
LOG.info("Begin to execute preSqls:[{}]. context info:{}.",
|
||||||
|
StringUtils.join(renderedPreSqls, ";"), tempJdbcUrl);
|
||||||
|
|
||||||
|
WriterUtil.executeSqls(conn, renderedPreSqls, tempJdbcUrl, dataBaseType);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
LOG.debug("After job prepare(), originalConfig now is:[\n{}\n]",
|
||||||
|
originalConfig.toJSON());
|
||||||
|
} catch (SQLException e) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.SQL_EXECUTE_FAIL, e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<Configuration> split(Configuration originalConfig,
|
||||||
|
int mandatoryNumber) {
|
||||||
|
return WriterUtil.doSplit(originalConfig, mandatoryNumber);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 一般来说，是需要推迟到 task 中进行post 的执行（单表情况例外）
|
||||||
|
public void post(Configuration originalConfig) {
|
||||||
|
|
||||||
|
String username = originalConfig.getString(Key.USERNAME);
|
||||||
|
String password = originalConfig.getString(Key.PASSWORD);
|
||||||
|
|
||||||
|
String jdbcUrl = originalConfig.getString(Key.JDBC_URL);
|
||||||
|
|
||||||
|
String table = originalConfig.getString(Key.TABLE);
|
||||||
|
|
||||||
|
List<String> postSqls = originalConfig.getList(Key.POST_SQL,
|
||||||
|
String.class);
|
||||||
|
List<String> renderedPostSqls = WriterUtil.renderPreOrPostSqls(
|
||||||
|
postSqls, table);
|
||||||
|
|
||||||
|
if (null != renderedPostSqls && !renderedPostSqls.isEmpty()) {
|
||||||
|
// 说明有 postSql 配置，则此处删除掉
|
||||||
|
originalConfig.remove(Key.POST_SQL);
|
||||||
|
String tempJdbcUrl = jdbcUrl.replace("postgresql", "hologres");
|
||||||
|
Connection conn = DBUtil.getConnection(this.dataBaseType,
|
||||||
|
tempJdbcUrl, username, password);
|
||||||
|
|
||||||
|
LOG.info(
|
||||||
|
"Begin to execute postSqls:[{}]. context info:{}.",
|
||||||
|
StringUtils.join(renderedPostSqls, ";"), tempJdbcUrl);
|
||||||
|
WriterUtil.executeSqls(conn, renderedPostSqls, tempJdbcUrl, dataBaseType);
|
||||||
|
DBUtil.closeDBResources(null, null, conn);
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
public void destroy(Configuration originalConfig) {
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
public static class Task {
|
||||||
|
protected static final Logger LOG = LoggerFactory
|
||||||
|
.getLogger(BaseWriter.Task.class);
|
||||||
|
|
||||||
|
protected DataBaseType dataBaseType;
|
||||||
|
|
||||||
|
protected String username;
|
||||||
|
protected String password;
|
||||||
|
protected String jdbcUrl;
|
||||||
|
protected String table;
|
||||||
|
protected List<String> columns;
|
||||||
|
protected int batchSize;
|
||||||
|
protected int batchByteSize;
|
||||||
|
protected int columnNumber = 0;
|
||||||
|
protected TaskPluginCollector taskPluginCollector;
|
||||||
|
|
||||||
|
// 作为日志显示信息时，需要附带的通用信息。比如信息所对应的数据库连接等信息，针对哪个表做的操作
|
||||||
|
protected static String BASIC_MESSAGE;
|
||||||
|
|
||||||
|
protected WriteMode writeMode;
|
||||||
|
protected String arrayDelimiter;
|
||||||
|
protected boolean emptyAsNull;
|
||||||
|
|
||||||
|
protected HoloConfig config;
|
||||||
|
|
||||||
|
public Task(DataBaseType dataBaseType) {
|
||||||
|
this.dataBaseType = dataBaseType;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void init(Configuration writerSliceConfig) {
|
||||||
|
this.username = writerSliceConfig.getString(Key.USERNAME);
|
||||||
|
this.password = writerSliceConfig.getString(Key.PASSWORD);
|
||||||
|
this.jdbcUrl = writerSliceConfig.getString(Key.JDBC_URL);
|
||||||
|
this.table = writerSliceConfig.getString(Key.TABLE);
|
||||||
|
|
||||||
|
this.columns = writerSliceConfig.getList(Key.COLUMN, String.class);
|
||||||
|
this.columnNumber = this.columns.size();
|
||||||
|
|
||||||
|
this.arrayDelimiter = writerSliceConfig.getString(Key.Array_Delimiter);
|
||||||
|
|
||||||
|
this.batchSize = writerSliceConfig.getInt(Key.BATCH_SIZE, Constant.DEFAULT_BATCH_SIZE);
|
||||||
|
this.batchByteSize = writerSliceConfig.getInt(Key.BATCH_BYTE_SIZE, Constant.DEFAULT_BATCH_BYTE_SIZE);
|
||||||
|
|
||||||
|
writeMode = getWriteMode(writerSliceConfig.getString(Key.WRITE_MODE, "REPLACE"));
|
||||||
|
emptyAsNull = writerSliceConfig.getBool(Key.EMPTY_AS_NULL, true);
|
||||||
|
|
||||||
|
Map<String, Object> clientConf = writerSliceConfig.getMap("client");
|
||||||
|
|
||||||
|
config = new HoloConfig();
|
||||||
|
config.setJdbcUrl(this.jdbcUrl);
|
||||||
|
config.setUsername(username);
|
||||||
|
config.setPassword(password);
|
||||||
|
config.setWriteMode(writeMode == WriteMode.IGNORE ? com.alibaba.hologres.client.model.WriteMode.INSERT_OR_IGNORE : (writeMode == WriteMode.UPDATE ? com.alibaba.hologres.client.model.WriteMode.INSERT_OR_UPDATE : com.alibaba.hologres.client.model.WriteMode.INSERT_OR_REPLACE));
|
||||||
|
config.setWriteBatchSize(this.batchSize);
|
||||||
|
config.setWriteBatchTotalByteSize(this.batchByteSize);
|
||||||
|
config.setMetaCacheTTL(3600000L);
|
||||||
|
config.setEnableDefaultForNotNullColumn(false);
|
||||||
|
config.setRetryCount(5);
|
||||||
|
config.setAppName("datax");
|
||||||
|
|
||||||
|
if (clientConf != null) {
|
||||||
|
try {
|
||||||
|
config = ConfLoader.load(clientConf, config, ignoreConfList);
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw DataXException
|
||||||
|
.asDataXException(
|
||||||
|
DBUtilErrorCode.CONF_ERROR,
|
||||||
|
"配置解析失败.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
BASIC_MESSAGE = String.format("jdbcUrl:[%s], table:[%s]",
|
||||||
|
this.jdbcUrl, this.table);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void prepare(Configuration writerSliceConfig) {
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
public void startWriteWithConnection(RecordReceiver recordReceiver, TaskPluginCollector taskPluginCollector) {
|
||||||
|
this.taskPluginCollector = taskPluginCollector;
|
||||||
|
|
||||||
|
try (HoloClient client = new HoloClient(config)) {
|
||||||
|
Record record;
|
||||||
|
TableSchema schema = RetryUtil.executeWithRetry(() -> client.getTableSchema(this.table), 3, 5000L, true);
|
||||||
|
while ((record = recordReceiver.getFromReader()) != null) {
|
||||||
|
if (record.getColumnNumber() != this.columnNumber) {
|
||||||
|
// 源头读取字段列数与目的表字段写入列数不相等，直接报错
|
||||||
|
throw DataXException
|
||||||
|
.asDataXException(
|
||||||
|
DBUtilErrorCode.CONF_ERROR,
|
||||||
|
String.format(
|
||||||
|
"列配置信息有错误. 因为您配置的任务中，源头读取字段数:%s 与 目的表要写入的字段数:%s 不相等. 请检查您的配置并作出修改.",
|
||||||
|
record.getColumnNumber(),
|
||||||
|
this.columnNumber));
|
||||||
|
}
|
||||||
|
Put put = convertToPut(record, schema);
|
||||||
|
if (null != put) {
|
||||||
|
try {
|
||||||
|
client.put(put);
|
||||||
|
} catch (HoloClientWithDetailsException detail) {
|
||||||
|
handleDirtyData(detail);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
client.flush();
|
||||||
|
} catch (HoloClientWithDetailsException detail) {
|
||||||
|
handleDirtyData(detail);
|
||||||
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw DataXException.asDataXException(
|
||||||
|
DBUtilErrorCode.WRITE_DATA_ERROR, e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void handleDirtyData(HoloClientWithDetailsException detail) {
|
||||||
|
for (int i = 0; i < detail.size(); ++i) {
|
||||||
|
com.alibaba.hologres.client.model.Record failRecord = detail.getFailRecord(i);
|
||||||
|
if (failRecord.getAttachmentList() != null) {
|
||||||
|
for (Object obj : failRecord.getAttachmentList()) {
|
||||||
|
taskPluginCollector.collectDirtyRecord((Record) obj, detail.getException(i));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public void startWrite(RecordReceiver recordReceiver,
|
||||||
|
TaskPluginCollector taskPluginCollector) {
|
||||||
|
startWriteWithConnection(recordReceiver, taskPluginCollector);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void post(Configuration writerSliceConfig) {
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
public void destroy(Configuration writerSliceConfig) {
|
||||||
|
}
|
||||||
|
|
||||||
|
// 直接使用了两个类变量：columnNumber,resultSetMetaData
|
||||||
|
protected Put convertToPut(Record record, TableSchema schema) {
|
||||||
|
try {
|
||||||
|
Put put = new Put(schema);
|
||||||
|
put.getRecord().addAttachment(record);
|
||||||
|
for (int i = 0; i < this.columnNumber; i++) {
|
||||||
|
fillColumn(put, schema, schema.getColumnIndex(this.columns.get(i)), record.getColumn(i));
|
||||||
|
}
|
||||||
|
return put;
|
||||||
|
} catch (Exception e) {
|
||||||
|
taskPluginCollector.collectDirtyRecord(record, e);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
protected void fillColumn(Put data, TableSchema schema, int index, Column column) throws SQLException {
|
||||||
|
com.alibaba.hologres.client.model.Column holoColumn = schema.getColumn(index);
|
||||||
|
switch (holoColumn.getType()) {
|
||||||
|
case Types.CHAR:
|
||||||
|
case Types.NCHAR:
|
||||||
|
case Types.CLOB:
|
||||||
|
case Types.NCLOB:
|
||||||
|
case Types.VARCHAR:
|
||||||
|
case Types.LONGVARCHAR:
|
||||||
|
case Types.NVARCHAR:
|
||||||
|
case Types.LONGNVARCHAR:
|
||||||
|
String value = column.asString();
|
||||||
|
if (emptyAsNull && value != null && value.length() == 0) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
} else {
|
||||||
|
data.setObject(index, value);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case Types.SMALLINT:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
data.setObject(index, column.asBigInteger().shortValue());
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.INTEGER:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
data.setObject(index, column.asBigInteger().intValue());
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.BIGINT:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
data.setObject(index, column.asBigInteger().longValue());
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.NUMERIC:
|
||||||
|
case Types.DECIMAL:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
data.setObject(index, column.asBigDecimal());
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.FLOAT:
|
||||||
|
case Types.REAL:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
data.setObject(index, column.asBigDecimal().floatValue());
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.DOUBLE:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
data.setObject(index, column.asDouble());
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.TIME:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
if (column instanceof LongColumn || column instanceof DateColumn) {
|
||||||
|
data.setObject(index, new Time(column.asLong()));
|
||||||
|
} else {
|
||||||
|
data.setObject(index, column.asString());
|
||||||
|
}
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.DATE:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
if (column instanceof LongColumn || column instanceof DateColumn) {
|
||||||
|
data.setObject(index, column.asLong());
|
||||||
|
} else {
|
||||||
|
data.setObject(index, column.asString());
|
||||||
|
}
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.TIMESTAMP:
|
||||||
|
if (column.getByteSize() > 0) {
|
||||||
|
if (column instanceof LongColumn || column instanceof DateColumn) {
|
||||||
|
data.setObject(index, new Timestamp(column.asLong()));
|
||||||
|
} else {
|
||||||
|
data.setObject(index, column.asString());
|
||||||
|
}
|
||||||
|
} else if (emptyAsNull) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case Types.BINARY:
|
||||||
|
case Types.VARBINARY:
|
||||||
|
case Types.BLOB:
|
||||||
|
case Types.LONGVARBINARY:
|
||||||
|
String byteValue = column.asString();
|
||||||
|
if (null != byteValue) {
|
||||||
|
data.setObject(index, column
|
||||||
|
.asBytes());
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.BOOLEAN:
|
||||||
|
case Types.BIT:
|
||||||
|
if (column.getByteSize() == 0) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
Boolean boolValue = column.asBoolean();
|
||||||
|
data.setObject(index, boolValue);
|
||||||
|
} catch (Exception e) {
|
||||||
|
data.setObject(index, !"0".equals(column.asString()));
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
case Types.ARRAY:
|
||||||
|
String arrayString = column.asString();
|
||||||
|
Object arrayObject = null;
|
||||||
|
if (null == arrayString || (emptyAsNull && "".equals(arrayString))) {
|
||||||
|
data.setObject(index, null);
|
||||||
|
break;
|
||||||
|
} else if (arrayDelimiter != null && arrayDelimiter.length() > 0) {
|
||||||
|
arrayObject = arrayString.split(this.arrayDelimiter);
|
||||||
|
} else {
|
||||||
|
arrayObject = JSONArray.parseArray(arrayString);
|
||||||
|
}
|
||||||
|
data.setObject(index, arrayObject);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
throw DataXException
|
||||||
|
.asDataXException(
|
||||||
|
DBUtilErrorCode.UNSUPPORTED_TYPE,
|
||||||
|
String.format(
|
||||||
|
"您的配置文件中的列配置信息有误. 因为DataX 不支持数据库写入这种字段类型. 字段名:[%s], 字段类型:[%d], 字段Java类型:[%s]. 请修改表中该字段的类型或者不同步该字段.",
|
||||||
|
holoColumn.getName(),
|
||||||
|
holoColumn.getType(),
|
||||||
|
holoColumn.getTypeName()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,15 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.hologresjdbcwriter;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 用于插件解析用户配置时，需要进行标识（MARK）的常量的声明.
|
||||||
|
*/
|
||||||
|
public final class Constant {
|
||||||
|
public static final int DEFAULT_BATCH_SIZE = 512;
|
||||||
|
|
||||||
|
public static final int DEFAULT_BATCH_BYTE_SIZE = 50 * 1024 * 1024;
|
||||||
|
|
||||||
|
public static String CONN_MARK = "connection";
|
||||||
|
|
||||||
|
public static String TABLE_NUMBER_MARK = "tableNumber";
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,78 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.hologresjdbcwriter;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.plugin.RecordReceiver;
|
||||||
|
import com.alibaba.datax.common.spi.Writer;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public class HologresJdbcWriter extends Writer {
|
||||||
|
private static final DataBaseType DATABASE_TYPE = DataBaseType.PostgreSQL;
|
||||||
|
|
||||||
|
public static class Job extends Writer.Job {
|
||||||
|
private Configuration originalConfig = null;
|
||||||
|
private BaseWriter.Job baseWriterMaster;
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void init() {
|
||||||
|
this.originalConfig = super.getPluginJobConf();
|
||||||
|
this.baseWriterMaster = new BaseWriter.Job(DATABASE_TYPE);
|
||||||
|
this.baseWriterMaster.init(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void prepare() {
|
||||||
|
this.baseWriterMaster.prepare(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public List<Configuration> split(int mandatoryNumber) {
|
||||||
|
return this.baseWriterMaster.split(this.originalConfig, mandatoryNumber);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void post() {
|
||||||
|
this.baseWriterMaster.post(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void destroy() {
|
||||||
|
this.baseWriterMaster.destroy(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
public static class Task extends Writer.Task {
|
||||||
|
private Configuration writerSliceConfig;
|
||||||
|
private BaseWriter.Task baseWriterSlave;
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void init() {
|
||||||
|
this.writerSliceConfig = super.getPluginJobConf();
|
||||||
|
this.baseWriterSlave = new BaseWriter.Task(DATABASE_TYPE);
|
||||||
|
this.baseWriterSlave.init(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void prepare() {
|
||||||
|
this.baseWriterSlave.prepare(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void startWrite(RecordReceiver recordReceiver) {
|
||||||
|
this.baseWriterSlave.startWrite(recordReceiver, super.getTaskPluginCollector());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void post() {
|
||||||
|
this.baseWriterSlave.post(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void destroy() {
|
||||||
|
this.baseWriterSlave.destroy(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,31 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.hologresjdbcwriter;
|
||||||
|
|
||||||
|
public final class Key {
|
||||||
|
public final static String JDBC_URL = "jdbcUrl";
|
||||||
|
|
||||||
|
public final static String USERNAME = "username";
|
||||||
|
|
||||||
|
public final static String PASSWORD = "password";
|
||||||
|
|
||||||
|
public final static String TABLE = "table";
|
||||||
|
|
||||||
|
public final static String COLUMN = "column";
|
||||||
|
|
||||||
|
public final static String Array_Delimiter = "arrayDelimiter";
|
||||||
|
|
||||||
|
public final static String WRITE_MODE = "writeMode";
|
||||||
|
|
||||||
|
public final static String PRE_SQL = "preSql";
|
||||||
|
|
||||||
|
public final static String POST_SQL = "postSql";
|
||||||
|
|
||||||
|
//默认值：256
|
||||||
|
public final static String BATCH_SIZE = "batchSize";
|
||||||
|
|
||||||
|
//默认值：50m
|
||||||
|
public final static String BATCH_BYTE_SIZE = "batchByteSize";
|
||||||
|
|
||||||
|
public final static String EMPTY_AS_NULL = "emptyAsNull";
|
||||||
|
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,59 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.hologresjdbcwriter.util;
|
||||||
|
|
||||||
|
import com.alibaba.hologres.client.model.WriteMode;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import java.lang.reflect.Field;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Set;
|
||||||
|
|
||||||
|
public class ConfLoader {
|
||||||
|
public static Logger LOG = LoggerFactory.getLogger(ConfLoader.class);
|
||||||
|
|
||||||
|
static public <T> T load(Map<String, Object> props, T config, Set<String> ignoreList) throws Exception {
|
||||||
|
Field[] fields = config.getClass().getDeclaredFields();
|
||||||
|
for (Map.Entry<String, Object> entry : props.entrySet()) {
|
||||||
|
String key = entry.getKey();
|
||||||
|
String value = entry.getValue().toString();
|
||||||
|
if (ignoreList.contains(key)) {
|
||||||
|
LOG.info("Config Skip {}", key);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
boolean match = false;
|
||||||
|
for (Field field : fields) {
|
||||||
|
if (field.getName().equals(key)) {
|
||||||
|
match = true;
|
||||||
|
field.setAccessible(true);
|
||||||
|
Class<?> type = field.getType();
|
||||||
|
if (type.equals(String.class)) {
|
||||||
|
field.set(config, value);
|
||||||
|
} else if (type.equals(int.class)) {
|
||||||
|
field.set(config, Integer.parseInt(value));
|
||||||
|
} else if (type.equals(long.class)) {
|
||||||
|
field.set(config, Long.parseLong(value));
|
||||||
|
} else if (type.equals(boolean.class)) {
|
||||||
|
field.set(config, Boolean.parseBoolean(value));
|
||||||
|
} else if (WriteMode.class.equals(type)) {
|
||||||
|
field.set(config, WriteMode.valueOf(value));
|
||||||
|
} else {
|
||||||
|
throw new Exception("invalid type " + type + " for param " + key);
|
||||||
|
}
|
||||||
|
if ("password".equals(key)) {
|
||||||
|
StringBuilder sb = new StringBuilder();
|
||||||
|
for (int i = 0; i < value.length(); ++i) {
|
||||||
|
sb.append("*");
|
||||||
|
}
|
||||||
|
LOG.info("Config {}={}", key, sb.toString());
|
||||||
|
} else {
|
||||||
|
LOG.info("Config {}={}", key, value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!match) {
|
||||||
|
throw new Exception("param " + key + " not found in HoloConfig");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return config;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,82 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.hologresjdbcwriter.util;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.exception.DataXException;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtilErrorCode;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.TableExpandUtil;
|
||||||
|
import com.alibaba.datax.plugin.writer.hologresjdbcwriter.Constant;
|
||||||
|
import com.alibaba.datax.plugin.writer.hologresjdbcwriter.Key;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public final class OriginalConfPretreatmentUtil {
|
||||||
|
private static final Logger LOG = LoggerFactory
|
||||||
|
.getLogger(OriginalConfPretreatmentUtil.class);
|
||||||
|
|
||||||
|
public static DataBaseType DATABASE_TYPE;
|
||||||
|
|
||||||
|
public static void doPretreatment(Configuration originalConfig, DataBaseType dataBaseType) {
|
||||||
|
// 检查 username/password 配置（必填）
|
||||||
|
originalConfig.getNecessaryValue(Key.USERNAME, DBUtilErrorCode.REQUIRED_VALUE);
|
||||||
|
originalConfig.getNecessaryValue(Key.PASSWORD, DBUtilErrorCode.REQUIRED_VALUE);
|
||||||
|
|
||||||
|
doCheckBatchSize(originalConfig);
|
||||||
|
simplifyConf(originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static void doCheckBatchSize(Configuration originalConfig) {
|
||||||
|
// 检查batchSize 配置（选填，如果未填写，则设置为默认值）
|
||||||
|
int batchSize = originalConfig.getInt(Key.BATCH_SIZE, Constant.DEFAULT_BATCH_SIZE);
|
||||||
|
if (batchSize < 1) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.ILLEGAL_VALUE, String.format(
|
||||||
|
"您的batchSize配置有误. 您所配置的写入数据库表的 batchSize:%s 不能小于1. 推荐配置范围为：[256-1024] (保持128的倍数), 该值越大, 内存溢出可能性越大. 请检查您的配置并作出修改.",
|
||||||
|
batchSize));
|
||||||
|
}
|
||||||
|
|
||||||
|
originalConfig.set(Key.BATCH_SIZE, batchSize);
|
||||||
|
}
|
||||||
|
|
||||||
|
public static void simplifyConf(Configuration originalConfig) {
|
||||||
|
List<Object> connections = originalConfig.getList(Constant.CONN_MARK,
|
||||||
|
Object.class);
|
||||||
|
|
||||||
|
int tableNum = 0;
|
||||||
|
|
||||||
|
for (int i = 0, len = connections.size(); i < len; i++) {
|
||||||
|
Configuration connConf = Configuration.from(connections.get(i).toString());
|
||||||
|
|
||||||
|
String jdbcUrl = connConf.getString(Key.JDBC_URL);
|
||||||
|
if (StringUtils.isBlank(jdbcUrl)) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.REQUIRED_VALUE, "您未配置的写入数据库表的 jdbcUrl.");
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> tables = connConf.getList(Key.TABLE, String.class);
|
||||||
|
|
||||||
|
if (null == tables || tables.isEmpty()) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.REQUIRED_VALUE,
|
||||||
|
"您未配置写入数据库表的表名称. 根据配置DataX找不到您配置的表. 请检查您的配置并作出修改.");
|
||||||
|
}
|
||||||
|
|
||||||
|
// 对每一个connection 上配置的table 项进行解析
|
||||||
|
List<String> expandedTables = TableExpandUtil
|
||||||
|
.expandTableConf(DATABASE_TYPE, tables);
|
||||||
|
|
||||||
|
if (null == expandedTables || expandedTables.isEmpty()) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.CONF_ERROR,
|
||||||
|
"您配置的写入数据库表名称错误. DataX找不到您配置的表，请检查您的配置并作出修改.");
|
||||||
|
}
|
||||||
|
|
||||||
|
tableNum += expandedTables.size();
|
||||||
|
|
||||||
|
originalConfig.set(String.format("%s[%d].%s", Constant.CONN_MARK,
|
||||||
|
i, Key.TABLE), expandedTables);
|
||||||
|
}
|
||||||
|
|
||||||
|
originalConfig.set(Constant.TABLE_NUMBER_MARK, tableNum);
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,111 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.hologresjdbcwriter.util;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.exception.DataXException;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtilErrorCode;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.RdbmsException;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.writer.Constant;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.writer.Key;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.sql.Statement;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Collections;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public final class WriterUtil {
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(WriterUtil.class);
|
||||||
|
|
||||||
|
//TODO 切分报错
|
||||||
|
public static List<Configuration> doSplit(Configuration simplifiedConf,
|
||||||
|
int adviceNumber) {
|
||||||
|
|
||||||
|
List<Configuration> splitResultConfigs = new ArrayList<Configuration>();
|
||||||
|
|
||||||
|
int tableNumber = simplifiedConf.getInt(Constant.TABLE_NUMBER_MARK);
|
||||||
|
|
||||||
|
//处理单表的情况
|
||||||
|
if (tableNumber == 1) {
|
||||||
|
//由于在之前的 master prepare 中已经把 table,jdbcUrl 提取出来，所以这里处理十分简单
|
||||||
|
for (int j = 0; j < adviceNumber; j++) {
|
||||||
|
splitResultConfigs.add(simplifiedConf.clone());
|
||||||
|
}
|
||||||
|
|
||||||
|
return splitResultConfigs;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (tableNumber != adviceNumber) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.CONF_ERROR,
|
||||||
|
String.format("您的配置文件中的列配置信息有误. 您要写入的目的端的表个数是:%s , 但是根据系统建议需要切分的份数是：%s. 请检查您的配置并作出修改.",
|
||||||
|
tableNumber, adviceNumber));
|
||||||
|
}
|
||||||
|
|
||||||
|
String jdbcUrl;
|
||||||
|
List<String> preSqls = simplifiedConf.getList(Key.PRE_SQL, String.class);
|
||||||
|
List<String> postSqls = simplifiedConf.getList(Key.POST_SQL, String.class);
|
||||||
|
|
||||||
|
List<Object> conns = simplifiedConf.getList(Constant.CONN_MARK,
|
||||||
|
Object.class);
|
||||||
|
|
||||||
|
for (Object conn : conns) {
|
||||||
|
Configuration sliceConfig = simplifiedConf.clone();
|
||||||
|
|
||||||
|
Configuration connConf = Configuration.from(conn.toString());
|
||||||
|
jdbcUrl = connConf.getString(Key.JDBC_URL);
|
||||||
|
sliceConfig.set(Key.JDBC_URL, jdbcUrl);
|
||||||
|
|
||||||
|
sliceConfig.remove(Constant.CONN_MARK);
|
||||||
|
|
||||||
|
List<String> tables = connConf.getList(Key.TABLE, String.class);
|
||||||
|
|
||||||
|
for (String table : tables) {
|
||||||
|
Configuration tempSlice = sliceConfig.clone();
|
||||||
|
tempSlice.set(Key.TABLE, table);
|
||||||
|
tempSlice.set(Key.PRE_SQL, renderPreOrPostSqls(preSqls, table));
|
||||||
|
tempSlice.set(Key.POST_SQL, renderPreOrPostSqls(postSqls, table));
|
||||||
|
|
||||||
|
splitResultConfigs.add(tempSlice);
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
return splitResultConfigs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static List<String> renderPreOrPostSqls(List<String> preOrPostSqls, String tableName) {
|
||||||
|
if (null == preOrPostSqls) {
|
||||||
|
return Collections.emptyList();
|
||||||
|
}
|
||||||
|
|
||||||
|
List<String> renderedSqls = new ArrayList<String>();
|
||||||
|
for (String sql : preOrPostSqls) {
|
||||||
|
//preSql为空时，不加入执行队列
|
||||||
|
if (StringUtils.isNotBlank(sql)) {
|
||||||
|
renderedSqls.add(sql.replace(Constant.TABLE_NAME_PLACEHOLDER, tableName));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return renderedSqls;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static void executeSqls(Connection conn, List<String> sqls, String basicMessage,DataBaseType dataBaseType) {
|
||||||
|
Statement stmt = null;
|
||||||
|
String currentSql = null;
|
||||||
|
try {
|
||||||
|
stmt = conn.createStatement();
|
||||||
|
for (String sql : sqls) {
|
||||||
|
currentSql = sql;
|
||||||
|
DBUtil.executeSqlWithoutResultSet(stmt, sql);
|
||||||
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
|
throw RdbmsException.asQueryException(dataBaseType,e,currentSql,null,null);
|
||||||
|
} finally {
|
||||||
|
DBUtil.closeDBResources(null, stmt, null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
6
hologresjdbcwriter/src/main/resources/plugin.json
Normal file
6
hologresjdbcwriter/src/main/resources/plugin.json
Normal file
@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"name": "hologreswriter",
|
||||||
|
"class": "com.alibaba.datax.plugin.writer.hologreswriter.HologresWriter",
|
||||||
|
"description": "",
|
||||||
|
"developer": "alibaba"
|
||||||
|
}
|
||||||
@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"name": "hologreswriter",
|
||||||
|
"parameter": {
|
||||||
|
"url": "",
|
||||||
|
"username": "",
|
||||||
|
"password": "",
|
||||||
|
"database": "",
|
||||||
|
"table": "",
|
||||||
|
"partition": ""
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
images/datax.logo.png
Normal file
BIN
images/datax.logo.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 102 KiB |
@ -36,6 +36,7 @@ DataX本身作为离线数据同步框架，采用Framework + plugin架构构建
|
|||||||
| ------------ | ---------- | :-------: | :-------: |:-------: |
|
| ------------ | ---------- | :-------: | :-------: |:-------: |
|
||||||
| RDBMS 关系型数据库 | MySQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md)|
|
| RDBMS 关系型数据库 | MySQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md)|
|
||||||
| | Oracle | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/oraclereader/doc/oraclereader.md) 、[写](https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md)|
|
| | Oracle | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/oraclereader/doc/oraclereader.md) 、[写](https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md)|
|
||||||
|
| | OceanBase | √ | √ |[读](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase) 、[写](https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/use-datax-to-full-migration-data-to-oceanbase)|
|
||||||
| | SQLServer | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/sqlserverreader/doc/sqlserverreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/sqlserverwriter/doc/sqlserverwriter.md)|
|
| | SQLServer | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/sqlserverreader/doc/sqlserverreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/sqlserverwriter/doc/sqlserverwriter.md)|
|
||||||
| | PostgreSQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/postgresqlwriter/doc/postgresqlwriter.md)|
|
| | PostgreSQL | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/postgresqlwriter/doc/postgresqlwriter.md)|
|
||||||
| | DRDS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/drdsreader/doc/drdsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/drdswriter/doc/drdswriter.md)|
|
| | DRDS | √ | √ |[读](https://github.com/alibaba/DataX/blob/master/drdsreader/doc/drdsreader.md) 、[写](https://github.com/alibaba/DataX/blob/master/drdswriter/doc/drdswriter.md)|
|
||||||
|
|||||||
@ -1,4 +1,4 @@
|
|||||||
Copyright 1999-2017 Alibaba Group Holding Ltd.
|
Copyright 1999-2022 Alibaba Group Holding Ltd.
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
you may not use this file except in compliance with the License.
|
you may not use this file except in compliance with the License.
|
||||||
|
|||||||
@ -8,7 +8,7 @@ MongoDBReader 插件利用 MongoDB 的java客户端MongoClient进行MongoDB的
|
|||||||
MongoDBReader通过Datax框架从MongoDB并行的读取数据，通过主控的JOB程序按照指定的规则对MongoDB中的数据进行分片，并行读取，然后将MongoDB支持的类型通过逐一判断转换成Datax支持的类型。
|
MongoDBReader通过Datax框架从MongoDB并行的读取数据，通过主控的JOB程序按照指定的规则对MongoDB中的数据进行分片，并行读取，然后将MongoDB支持的类型通过逐一判断转换成Datax支持的类型。
|
||||||
|
|
||||||
#### 3 功能说明
|
#### 3 功能说明
|
||||||
* 该示例从ODPS读一份数据到MongoDB。
|
* 该示例从MongoDB读一份数据到ODPS。
|
||||||
|
|
||||||
{
|
{
|
||||||
"job": {
|
"job": {
|
||||||
@ -127,6 +127,7 @@ MongoDBReader通过Datax框架从MongoDB并行的读取数据，通过主控的J
|
|||||||
* address： MongoDB的数据地址信息，因为MonogDB可能是个集群，则ip端口信息需要以Json数组的形式给出。【必填】
|
* address： MongoDB的数据地址信息，因为MonogDB可能是个集群，则ip端口信息需要以Json数组的形式给出。【必填】
|
||||||
* userName：MongoDB的用户名。【选填】
|
* userName：MongoDB的用户名。【选填】
|
||||||
* userPassword： MongoDB的密码。【选填】
|
* userPassword： MongoDB的密码。【选填】
|
||||||
|
* authDb: MongoDB认证数据库【选填】
|
||||||
* collectionName： MonogoDB的集合名。【必填】
|
* collectionName： MonogoDB的集合名。【必填】
|
||||||
* column：MongoDB的文档列名。【必填】
|
* column：MongoDB的文档列名。【必填】
|
||||||
* name：Column的名字。【必填】
|
* name：Column的名字。【必填】
|
||||||
@ -147,4 +148,4 @@ MongoDBReader通过Datax框架从MongoDB并行的读取数据，通过主控的J
|
|||||||
|
|
||||||
|
|
||||||
#### 6 性能报告
|
#### 6 性能报告
|
||||||
#### 7 测试报告
|
#### 7 测试报告
|
||||||
|
|||||||
@ -116,10 +116,10 @@ MongoDBWriter通过Datax框架获取Reader生成的数据，然后将Datax支持
|
|||||||
"type": "int"
|
"type": "int"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"upsertInfo": {
|
"writeMode": {
|
||||||
"isUpsert": "true",
|
"isReplace": "true",
|
||||||
"upsertKey": "unique_id"
|
"replaceKey": "unique_id"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -135,11 +135,11 @@ MongoDBWriter通过Datax框架获取Reader生成的数据，然后将Datax支持
|
|||||||
* collectionName： MonogoDB的集合名。【必填】
|
* collectionName： MonogoDB的集合名。【必填】
|
||||||
* column：MongoDB的文档列名。【必填】
|
* column：MongoDB的文档列名。【必填】
|
||||||
* name：Column的名字。【必填】
|
* name：Column的名字。【必填】
|
||||||
* type：Column的类型。【选填】
|
* type：Column的类型。【必填】
|
||||||
* splitter：特殊分隔符，当且仅当要处理的字符串要用分隔符分隔为字符数组时，才使用这个参数，通过这个参数指定的分隔符，将字符串分隔存储到MongoDB的数组中。【选填】
|
* splitter：特殊分隔符，当且仅当要处理的字符串要用分隔符分隔为字符数组时，才使用这个参数，通过这个参数指定的分隔符，将字符串分隔存储到MongoDB的数组中。【选填】
|
||||||
* upsertInfo：指定了传输数据时更新的信息。【选填】
|
* writeMode：指定了传输数据时更新的信息。【选填】
|
||||||
* isUpsert：当设置为true时，表示针对相同的upsertKey做更新操作。【选填】
|
* isReplace：当设置为true时，表示针对相同的replaceKey做更新操作。【选填】
|
||||||
* upsertKey：upsertKey指定了没行记录的业务主键。用来做更新时使用。【选填】
|
* replaceKey：replaceKey指定了每行记录的业务主键。用来做更新时使用。【选填】
|
||||||
|
|
||||||
#### 5 类型转换
|
#### 5 类型转换
|
||||||
|
|
||||||
@ -154,4 +154,4 @@ MongoDBWriter通过Datax框架获取Reader生成的数据，然后将Datax支持
|
|||||||
|
|
||||||
|
|
||||||
#### 6 性能报告
|
#### 6 性能报告
|
||||||
#### 7 测试报告
|
#### 7 测试报告
|
||||||
|
|||||||
@ -165,7 +165,7 @@ MysqlReader插件实现了从Mysql读取数据。在底层实现上，MysqlReade
|
|||||||
|
|
||||||
支持常量配置，用户需要按照Mysql SQL语法格式:
|
支持常量配置，用户需要按照Mysql SQL语法格式:
|
||||||
["id", "\`table\`", "1", "'bazhen.csy'", "null", "to_char(a + 1)", "2.3" , "true"]
|
["id", "\`table\`", "1", "'bazhen.csy'", "null", "to_char(a + 1)", "2.3" , "true"]
|
||||||
id为普通列名，\`table\`为包含保留在的列名，1为整形数字常量，'bazhen.csy'为字符串常量，null为空指针，to_char(a + 1)为表达式，2.3为浮点数，true为布尔值。
|
id为普通列名，\`table\`为包含保留字的列名，1为整形数字常量，'bazhen.csy'为字符串常量，null为空指针，to_char(a + 1)为表达式，2.3为浮点数，true为布尔值。
|
||||||
|
|
||||||
* 必选：是 <br />
|
* 必选：是 <br />
|
||||||
|
|
||||||
@ -197,9 +197,9 @@ MysqlReader插件实现了从Mysql读取数据。在底层实现上，MysqlReade
|
|||||||
|
|
||||||
* **querySql**
|
* **querySql**
|
||||||
|
|
||||||
* 描述：在有些业务场景下，where这一配置项不足以描述所筛选的条件，用户可以通过该配置型来自定义筛选SQL。当用户配置了这一项之后，DataX系统就会忽略table，column这些配置型，直接使用这个配置项的内容对数据进行筛选，例如需要进行多表join后同步数据，使用select a,b from table_a join table_b on table_a.id = table_b.id <br />
|
* 描述：在有些业务场景下，where这一配置项不足以描述所筛选的条件，用户可以通过该配置型来自定义筛选SQL。当用户配置了这一项之后，DataX系统就会忽略column这些配置型，直接使用这个配置项的内容对数据进行筛选，例如需要进行多表join后同步数据，使用select a,b from table_a join table_b on table_a.id = table_b.id <br />
|
||||||
|
|
||||||
`当用户配置querySql时，MysqlReader直接忽略table、column、where条件的配置`，querySql优先级大于table、column、where选项。
|
`当用户配置querySql时，MysqlReader直接忽略column、where条件的配置`，querySql优先级大于column、where选项。querySql和table不能同时存在
|
||||||
|
|
||||||
* 必选：否 <br />
|
* 必选：否 <br />
|
||||||
|
|
||||||
|
|||||||
@ -40,7 +40,7 @@
|
|||||||
<dependency>
|
<dependency>
|
||||||
<groupId>mysql</groupId>
|
<groupId>mysql</groupId>
|
||||||
<artifactId>mysql-connector-java</artifactId>
|
<artifactId>mysql-connector-java</artifactId>
|
||||||
<version>5.1.34</version>
|
<version>${mysql.driver.version}</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -40,7 +40,7 @@
|
|||||||
<dependency>
|
<dependency>
|
||||||
<groupId>mysql</groupId>
|
<groupId>mysql</groupId>
|
||||||
<artifactId>mysql-connector-java</artifactId>
|
<artifactId>mysql-connector-java</artifactId>
|
||||||
<version>5.1.34</version>
|
<version>${mysql.driver.version}</version>
|
||||||
</dependency>
|
</dependency>
|
||||||
</dependencies>
|
</dependencies>
|
||||||
|
|
||||||
|
|||||||
97
oceanbasev10reader/pom.xml
Normal file
97
oceanbasev10reader/pom.xml
Normal file
@ -0,0 +1,97 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<parent>
|
||||||
|
<artifactId>datax-all</artifactId>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<version>0.0.1-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
<artifactId>oceanbasev10reader</artifactId>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<version>0.0.1-SNAPSHOT</version>
|
||||||
|
<packaging>jar</packaging>
|
||||||
|
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<artifactId>datax-common</artifactId>
|
||||||
|
<version>${datax-project-version}</version>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<artifactId>slf4j-log4j12</artifactId>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
<artifactId>slf4j-api</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>ch.qos.logback</groupId>
|
||||||
|
<artifactId>logback-classic</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<artifactId>plugin-rdbms-util</artifactId>
|
||||||
|
<version>${datax-project-version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>mysql</groupId>
|
||||||
|
<artifactId>mysql-connector-java</artifactId>
|
||||||
|
<version>5.1.40</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>log4j</groupId>
|
||||||
|
<artifactId>log4j</artifactId>
|
||||||
|
<version>1.2.16</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>junit</groupId>
|
||||||
|
<artifactId>junit</artifactId>
|
||||||
|
<version>4.11</version>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
|
||||||
|
<build>
|
||||||
|
<resources>
|
||||||
|
<resource>
|
||||||
|
<directory>src/main/java</directory>
|
||||||
|
<includes>
|
||||||
|
<include>**/*.properties</include>
|
||||||
|
</includes>
|
||||||
|
</resource>
|
||||||
|
</resources>
|
||||||
|
<plugins>
|
||||||
|
<!-- compiler plugin -->
|
||||||
|
<plugin>
|
||||||
|
<artifactId>maven-compiler-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<source>${jdk-version}</source>
|
||||||
|
<target>${jdk-version}</target>
|
||||||
|
<encoding>${project-sourceEncoding}</encoding>
|
||||||
|
</configuration>
|
||||||
|
</plugin>
|
||||||
|
<!-- assembly plugin -->
|
||||||
|
<plugin>
|
||||||
|
<artifactId>maven-assembly-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<descriptors>
|
||||||
|
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||||
|
</descriptors>
|
||||||
|
<finalName>datax</finalName>
|
||||||
|
</configuration>
|
||||||
|
<executions>
|
||||||
|
<execution>
|
||||||
|
<id>dwzip</id>
|
||||||
|
<phase>package</phase>
|
||||||
|
<goals>
|
||||||
|
<goal>single</goal>
|
||||||
|
</goals>
|
||||||
|
</execution>
|
||||||
|
</executions>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</build>
|
||||||
|
</project>
|
||||||
42
oceanbasev10reader/src/main/assembly/package.xml
Normal file
42
oceanbasev10reader/src/main/assembly/package.xml
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
<assembly
|
||||||
|
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||||
|
<id></id>
|
||||||
|
<formats>
|
||||||
|
<format>dir</format>
|
||||||
|
</formats>
|
||||||
|
<includeBaseDirectory>false</includeBaseDirectory>
|
||||||
|
<fileSets>
|
||||||
|
<fileSet>
|
||||||
|
<directory>src/main/resources</directory>
|
||||||
|
<includes>
|
||||||
|
<include>plugin.json</include>
|
||||||
|
<include>plugin_job_template.json</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/reader/oceanbasev10reader</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>target/</directory>
|
||||||
|
<includes>
|
||||||
|
<include>oceanbasev10reader-0.0.1-SNAPSHOT.jar</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/reader/oceanbasev10reader</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>src/main/libs/</directory>
|
||||||
|
<includes>
|
||||||
|
<include>*.jar</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/reader/oceanbasev10reader/libs</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
</fileSets>
|
||||||
|
|
||||||
|
<dependencySets>
|
||||||
|
<dependencySet>
|
||||||
|
<useProjectArtifact>false</useProjectArtifact>
|
||||||
|
<outputDirectory>plugin/reader/oceanbasev10reader/libs</outputDirectory>
|
||||||
|
<scope>runtime</scope>
|
||||||
|
</dependencySet>
|
||||||
|
</dependencySets>
|
||||||
|
</assembly>
|
||||||
@ -0,0 +1,16 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader;
|
||||||
|
|
||||||
|
public interface Config {
|
||||||
|
// queryTimeoutSecond
|
||||||
|
String QUERY_TIMEOUT_SECOND = "memstoreCheckIntervalSecond";
|
||||||
|
|
||||||
|
int DEFAULT_QUERY_TIMEOUT_SECOND = 60 * 60 * 48;// 2天
|
||||||
|
|
||||||
|
// readBatchSize
|
||||||
|
String READ_BATCH_SIZE = "readBatchSize";
|
||||||
|
|
||||||
|
int DEFAULT_READ_BATCH_SIZE = 100000;// 10万
|
||||||
|
|
||||||
|
String RETRY_LIMIT = "retryLimit";
|
||||||
|
int DEFAULT_RETRY_LIMIT = 10;
|
||||||
|
}
|
||||||
@ -0,0 +1,147 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.ext.ObReaderKey;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.plugin.RecordSender;
|
||||||
|
import com.alibaba.datax.common.spi.Reader;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Constant;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Key;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.ext.ReaderJob;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.ext.ReaderTask;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.util.ObReaderUtils;
|
||||||
|
|
||||||
|
public class OceanBaseReader extends Reader {
|
||||||
|
|
||||||
|
public static class Job extends Reader.Job {
|
||||||
|
private Configuration originalConfig = null;
|
||||||
|
private ReaderJob readerJob;
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(Task.class);
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void init() {
|
||||||
|
this.originalConfig = super.getPluginJobConf();
|
||||||
|
|
||||||
|
Integer userConfigedFetchSize = this.originalConfig.getInt(Constant.FETCH_SIZE);
|
||||||
|
if (userConfigedFetchSize != null) {
|
||||||
|
LOG.warn("The [fetchSize] is not recognized, please use readBatchSize instead.");
|
||||||
|
}
|
||||||
|
this.originalConfig.set(Constant.FETCH_SIZE, Integer.MIN_VALUE);
|
||||||
|
setDatabaseType(originalConfig);
|
||||||
|
this.readerJob = new ReaderJob();
|
||||||
|
this.readerJob.init(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void prepare() {
|
||||||
|
//ObReaderUtils.DATABASE_TYPE获取当前数据库的语法模式
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void preCheck() {
|
||||||
|
init();
|
||||||
|
this.readerJob.preCheck(this.originalConfig, ObReaderUtils.databaseType);
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public List<Configuration> split(int adviceNumber) {
|
||||||
|
String splitPk = originalConfig.getString(Key.SPLIT_PK);
|
||||||
|
List<String> quotedColumns = originalConfig.getList(Key.COLUMN_LIST, String.class);
|
||||||
|
if (splitPk != null && splitPk.length() > 0 && quotedColumns != null) {
|
||||||
|
String escapeChar = ObReaderUtils.isOracleMode(originalConfig.getString(ObReaderKey.OB_COMPATIBILITY_MODE))
|
||||||
|
? "\"" : "`";
|
||||||
|
if (!splitPk.startsWith(escapeChar) && !splitPk.endsWith(escapeChar)) {
|
||||||
|
splitPk = escapeChar + splitPk + escapeChar;
|
||||||
|
}
|
||||||
|
for (String column : quotedColumns) {
|
||||||
|
if (column.equals(splitPk)) {
|
||||||
|
LOG.info("splitPk is an ob reserved keyword, set to {}", splitPk);
|
||||||
|
originalConfig.set(Key.SPLIT_PK, splitPk);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return this.readerJob.split(this.originalConfig, adviceNumber);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void post() {
|
||||||
|
this.readerJob.post(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void destroy() {
|
||||||
|
this.readerJob.destroy(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void setDatabaseType(Configuration config) {
|
||||||
|
String username = config.getString(Key.USERNAME);
|
||||||
|
String password = config.getString(Key.PASSWORD);
|
||||||
|
List<Object> conns = originalConfig.getList(Constant.CONN_MARK, Object.class);
|
||||||
|
Configuration connConf = Configuration.from(conns.get(0).toString());
|
||||||
|
List<String> jdbcUrls = connConf.getList(Key.JDBC_URL, String.class);
|
||||||
|
String jdbcUrl = jdbcUrls.get(0);
|
||||||
|
if (jdbcUrl.startsWith(com.alibaba.datax.plugin.rdbms.writer.Constant.OB10_SPLIT_STRING)) {
|
||||||
|
String[] ss = jdbcUrl.split(com.alibaba.datax.plugin.rdbms.writer.Constant.OB10_SPLIT_STRING_PATTERN);
|
||||||
|
if (ss.length != 3) {
|
||||||
|
LOG.warn("unrecognized jdbc url: " + jdbcUrl);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
username = ss[1].trim() + ":" + username;
|
||||||
|
jdbcUrl = ss[2];
|
||||||
|
}
|
||||||
|
// Use ob-client to get compatible mode.
|
||||||
|
try {
|
||||||
|
String obJdbcUrl = jdbcUrl.replace("jdbc:mysql:", "jdbc:oceanbase:");
|
||||||
|
Connection conn = DBUtil.getConnection(DataBaseType.OceanBase, obJdbcUrl, username, password);
|
||||||
|
String compatibleMode = ObReaderUtils.getCompatibleMode(conn);
|
||||||
|
config.set(ObReaderKey.OB_COMPATIBILITY_MODE, compatibleMode);
|
||||||
|
if (ObReaderUtils.isOracleMode(compatibleMode)) {
|
||||||
|
ObReaderUtils.compatibleMode = ObReaderUtils.OB_COMPATIBLE_MODE_ORACLE;
|
||||||
|
}
|
||||||
|
|
||||||
|
} catch (Exception e) {
|
||||||
|
LOG.warn("error in get compatible mode, using mysql as default: " + e.getMessage());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static class Task extends Reader.Task {
|
||||||
|
private Configuration readerSliceConfig;
|
||||||
|
private ReaderTask commonRdbmsReaderTask;
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(Task.class);
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void init() {
|
||||||
|
this.readerSliceConfig = super.getPluginJobConf();
|
||||||
|
this.commonRdbmsReaderTask = new ReaderTask(super.getTaskGroupId(), super.getTaskId());
|
||||||
|
this.commonRdbmsReaderTask.init(this.readerSliceConfig);
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void startRead(RecordSender recordSender) {
|
||||||
|
int fetchSize = this.readerSliceConfig.getInt(Constant.FETCH_SIZE);
|
||||||
|
this.commonRdbmsReaderTask.startRead(this.readerSliceConfig, recordSender, super.getTaskPluginCollector(),
|
||||||
|
fetchSize);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void post() {
|
||||||
|
this.commonRdbmsReaderTask.post(this.readerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void destroy() {
|
||||||
|
this.commonRdbmsReaderTask.destroy(this.readerSliceConfig);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,11 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.ext;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @author johnrobbet
|
||||||
|
*/
|
||||||
|
public class Constant {
|
||||||
|
|
||||||
|
public static String WEAK_READ_QUERY_SQL_TEMPLATE_WITHOUT_WHERE = "select /*+read_consistency(weak)*/ %s from %s ";
|
||||||
|
|
||||||
|
public static String WEAK_READ_QUERY_SQL_TEMPLATE = "select /*+read_consistency(weak)*/ %s from %s where (%s)";
|
||||||
|
}
|
||||||
@ -0,0 +1,16 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.ext;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @author johnrobbet
|
||||||
|
*/
|
||||||
|
public class ObReaderKey {
|
||||||
|
|
||||||
|
public final static String READ_BY_PARTITION = "readByPartition";
|
||||||
|
|
||||||
|
public final static String PARTITION_NAME = "partitionName";
|
||||||
|
|
||||||
|
public final static String PARTITION_TYPE = "partitionType";
|
||||||
|
|
||||||
|
public final static String OB_COMPATIBILITY_MODE = "obCompatibilityMode";
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,91 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.ext;
|
||||||
|
|
||||||
|
import java.util.Arrays;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.constant.CommonConstant;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Key;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Constant;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.OceanBaseReader;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.util.ObReaderUtils;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.util.PartitionSplitUtil;
|
||||||
|
import com.alibaba.fastjson.JSONObject;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
|
||||||
|
public class ReaderJob extends CommonRdbmsReader.Job {
|
||||||
|
private Logger LOG = LoggerFactory.getLogger(OceanBaseReader.Task.class);
|
||||||
|
|
||||||
|
public ReaderJob() {
|
||||||
|
super(ObReaderUtils.databaseType);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void init(Configuration originalConfig) {
|
||||||
|
//将config中的column和table中的关键字进行转义
|
||||||
|
List<String> columns = originalConfig.getList(Key.COLUMN, String.class);
|
||||||
|
ObReaderUtils.escapeDatabaseKeywords(columns);
|
||||||
|
originalConfig.set(Key.COLUMN, columns);
|
||||||
|
|
||||||
|
List<JSONObject> conns = originalConfig.getList(Constant.CONN_MARK, JSONObject.class);
|
||||||
|
for (int i = 0; i < conns.size(); i++) {
|
||||||
|
JSONObject conn = conns.get(i);
|
||||||
|
Configuration connConfig = Configuration.from(conn.toString());
|
||||||
|
List<String> tables = connConfig.getList(Key.TABLE, String.class);
|
||||||
|
|
||||||
|
// tables will be null when querySql is configured
|
||||||
|
if (tables != null) {
|
||||||
|
ObReaderUtils.escapeDatabaseKeywords(tables);
|
||||||
|
originalConfig.set(String.format("%s[%d].%s", Constant.CONN_MARK, i, Key.TABLE),
|
||||||
|
tables);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
super.init(originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public List<Configuration> split(Configuration originalConfig, int adviceNumber) {
|
||||||
|
List<Configuration> list;
|
||||||
|
// readByPartition is lower priority than splitPk.
|
||||||
|
// and readByPartition only works in table mode.
|
||||||
|
if (!isSplitPkValid(originalConfig) &&
|
||||||
|
originalConfig.getBool(Constant.IS_TABLE_MODE) &&
|
||||||
|
originalConfig.getBool(ObReaderKey.READ_BY_PARTITION, false)) {
|
||||||
|
LOG.info("try to split reader job by partition.");
|
||||||
|
list = PartitionSplitUtil.splitByPartition(originalConfig);
|
||||||
|
} else {
|
||||||
|
LOG.info("try to split reader job by splitPk.");
|
||||||
|
list = super.split(originalConfig, adviceNumber);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (Configuration config : list) {
|
||||||
|
String jdbcUrl = config.getString(Key.JDBC_URL);
|
||||||
|
String obRegionName = getObRegionName(jdbcUrl);
|
||||||
|
config.set(CommonConstant.LOAD_BALANCE_RESOURCE_MARK, obRegionName);
|
||||||
|
}
|
||||||
|
|
||||||
|
return list;
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean isSplitPkValid(Configuration originalConfig) {
|
||||||
|
String splitPk = originalConfig.getString(Key.SPLIT_PK);
|
||||||
|
return splitPk != null && splitPk.trim().length() > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
private String getObRegionName(String jdbcUrl) {
|
||||||
|
final String obJdbcDelimiter = com.alibaba.datax.plugin.rdbms.writer.Constant.OB10_SPLIT_STRING;
|
||||||
|
if (jdbcUrl.startsWith(obJdbcDelimiter)) {
|
||||||
|
String[] ss = jdbcUrl.split(obJdbcDelimiter);
|
||||||
|
if (ss.length >= 2) {
|
||||||
|
String tenant = ss[1].trim();
|
||||||
|
String[] sss = tenant.split(":");
|
||||||
|
return sss[0];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,297 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.ext;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.element.Column;
|
||||||
|
import com.alibaba.datax.common.element.Record;
|
||||||
|
import com.alibaba.datax.common.plugin.RecordSender;
|
||||||
|
import com.alibaba.datax.common.plugin.TaskPluginCollector;
|
||||||
|
import com.alibaba.datax.common.statistics.PerfRecord;
|
||||||
|
import com.alibaba.datax.common.statistics.PerfTrace;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Constant;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Key;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.RdbmsException;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.Config;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.util.ObReaderUtils;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.util.TaskContext;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import java.sql.*;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
public class ReaderTask extends CommonRdbmsReader.Task {
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(ReaderTask.class);
|
||||||
|
private int taskGroupId = -1;
|
||||||
|
private int taskId = -1;
|
||||||
|
|
||||||
|
private String username;
|
||||||
|
private String password;
|
||||||
|
private String jdbcUrl;
|
||||||
|
private String mandatoryEncoding;
|
||||||
|
private int queryTimeoutSeconds;// 查询超时 默认48小时
|
||||||
|
private int readBatchSize;
|
||||||
|
private int retryLimit = 0;
|
||||||
|
private String compatibleMode = ObReaderUtils.OB_COMPATIBLE_MODE_MYSQL;
|
||||||
|
private boolean reuseConn = false;
|
||||||
|
|
||||||
|
public ReaderTask(int taskGroupId, int taskId) {
|
||||||
|
super(ObReaderUtils.databaseType, taskGroupId, taskId);
|
||||||
|
this.taskGroupId = taskGroupId;
|
||||||
|
this.taskId = taskId;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void init(Configuration readerSliceConfig) {
|
||||||
|
/* for database connection */
|
||||||
|
username = readerSliceConfig.getString(Key.USERNAME);
|
||||||
|
password = readerSliceConfig.getString(Key.PASSWORD);
|
||||||
|
jdbcUrl = readerSliceConfig.getString(Key.JDBC_URL);
|
||||||
|
queryTimeoutSeconds = readerSliceConfig.getInt(Config.QUERY_TIMEOUT_SECOND,
|
||||||
|
Config.DEFAULT_QUERY_TIMEOUT_SECOND);
|
||||||
|
// ob10的处理
|
||||||
|
if (jdbcUrl.startsWith(com.alibaba.datax.plugin.rdbms.writer.Constant.OB10_SPLIT_STRING)) {
|
||||||
|
String[] ss = jdbcUrl.split(com.alibaba.datax.plugin.rdbms.writer.Constant.OB10_SPLIT_STRING_PATTERN);
|
||||||
|
if (ss.length == 3) {
|
||||||
|
LOG.info("this is ob1_0 jdbc url.");
|
||||||
|
username = ss[1].trim() + ":" + username;
|
||||||
|
jdbcUrl = ss[2];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
jdbcUrl = jdbcUrl.replace("jdbc:mysql:", "jdbc:oceanbase:") + "&socketTimeout=1800000&connectTimeout=60000"; //socketTimeout 半个小时
|
||||||
|
if (ObReaderUtils.compatibleMode.equals(ObReaderUtils.OB_COMPATIBLE_MODE_ORACLE)) {
|
||||||
|
compatibleMode = ObReaderUtils.OB_COMPATIBLE_MODE_ORACLE;
|
||||||
|
}
|
||||||
|
LOG.info("this is ob1_0 jdbc url. user=" + username + " :url=" + jdbcUrl);
|
||||||
|
mandatoryEncoding = readerSliceConfig.getString(Key.MANDATORY_ENCODING, "");
|
||||||
|
retryLimit = readerSliceConfig.getInt(Config.RETRY_LIMIT, Config.DEFAULT_RETRY_LIMIT);
|
||||||
|
LOG.info("retryLimit: " + retryLimit);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void buildSavePoint(TaskContext context) {
|
||||||
|
if (!ObReaderUtils.isUserSavePointValid(context)) {
|
||||||
|
LOG.info("user save point is not valid, set to null.");
|
||||||
|
context.setUserSavePoint(null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 如果isTableMode && table有PK
|
||||||
|
* <p>
|
||||||
|
* 则支持断点续读 (若pk不在原始的columns中,则追加到尾部,但不传给下游)
|
||||||
|
* <p>
|
||||||
|
* 否则,则使用旧模式
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void startRead(Configuration readerSliceConfig, RecordSender recordSender,
|
||||||
|
TaskPluginCollector taskPluginCollector, int fetchSize) {
|
||||||
|
String querySql = readerSliceConfig.getString(Key.QUERY_SQL);
|
||||||
|
String table = readerSliceConfig.getString(Key.TABLE);
|
||||||
|
PerfTrace.getInstance().addTaskDetails(taskId, table + "," + jdbcUrl);
|
||||||
|
List<String> columns = readerSliceConfig.getList(Key.COLUMN_LIST, String.class);
|
||||||
|
String where = readerSliceConfig.getString(Key.WHERE);
|
||||||
|
boolean weakRead = readerSliceConfig.getBool(Key.WEAK_READ, true); // default true, using weak read
|
||||||
|
String userSavePoint = readerSliceConfig.getString(Key.SAVE_POINT, null);
|
||||||
|
reuseConn = readerSliceConfig.getBool(Key.REUSE_CONN, false);
|
||||||
|
String partitionName = readerSliceConfig.getString(Key.PARTITION_NAME, null);
|
||||||
|
// 从配置文件中取readBatchSize,若无则用默认值
|
||||||
|
readBatchSize = readerSliceConfig.getInt(Config.READ_BATCH_SIZE, Config.DEFAULT_READ_BATCH_SIZE);
|
||||||
|
// 不能少于1万
|
||||||
|
if (readBatchSize < 10000) {
|
||||||
|
readBatchSize = 10000;
|
||||||
|
}
|
||||||
|
TaskContext context = new TaskContext(table, columns, where, fetchSize);
|
||||||
|
context.setQuerySql(querySql);
|
||||||
|
context.setWeakRead(weakRead);
|
||||||
|
context.setCompatibleMode(compatibleMode);
|
||||||
|
if (partitionName != null) {
|
||||||
|
context.setPartitionName(partitionName);
|
||||||
|
}
|
||||||
|
// Add the user save point into the context
|
||||||
|
context.setUserSavePoint(userSavePoint);
|
||||||
|
PerfRecord allPerf = new PerfRecord(taskGroupId, taskId, PerfRecord.PHASE.RESULT_NEXT_ALL);
|
||||||
|
allPerf.start();
|
||||||
|
boolean isTableMode = readerSliceConfig.getBool(Constant.IS_TABLE_MODE);
|
||||||
|
try {
|
||||||
|
startRead0(isTableMode, context, recordSender, taskPluginCollector);
|
||||||
|
} finally {
|
||||||
|
ObReaderUtils.close(null, null, context.getConn());
|
||||||
|
}
|
||||||
|
allPerf.end(context.getCost());
|
||||||
|
// 目前大盘是依赖这个打印，而之前这个Finish read record是包含了sql查询和result next的全部时间
|
||||||
|
LOG.info("finished read record by Sql: [{}\n] {}.", context.getQuerySql(), jdbcUrl);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void startRead0(boolean isTableMode, TaskContext context, RecordSender recordSender,
|
||||||
|
TaskPluginCollector taskPluginCollector) {
|
||||||
|
// 不是table模式 直接使用原来的做法
|
||||||
|
if (!isTableMode) {
|
||||||
|
doRead(recordSender, taskPluginCollector, context);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// check primary key index
|
||||||
|
Connection conn = DBUtil.getConnection(ObReaderUtils.databaseType, jdbcUrl, username, password);
|
||||||
|
ObReaderUtils.initConn4Reader(conn, queryTimeoutSeconds);
|
||||||
|
context.setConn(conn);
|
||||||
|
try {
|
||||||
|
ObReaderUtils.initIndex(conn, context);
|
||||||
|
ObReaderUtils.matchPkIndexs(conn, context);
|
||||||
|
} catch (Throwable e) {
|
||||||
|
LOG.warn("fetch PkIndexs fail,table=" + context.getTable(), e);
|
||||||
|
}
|
||||||
|
// 如果不是table 且 pk不存在 则仍然使用原来的做法
|
||||||
|
if (context.getPkIndexs() == null) {
|
||||||
|
doRead(recordSender, taskPluginCollector, context);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// setup the user defined save point
|
||||||
|
buildSavePoint(context);
|
||||||
|
|
||||||
|
// 从这里开始就是 断点续读功能
|
||||||
|
// while(true) {
|
||||||
|
// 正常读 (需 order by pk asc)
|
||||||
|
// 如果遇到失败,分两种情况:
|
||||||
|
// a)已读出记录,则开始走增量读逻辑
|
||||||
|
// b)未读出记录,则走正常读逻辑(仍然需要order by pk asc)
|
||||||
|
// 正常结束 则 break
|
||||||
|
// }
|
||||||
|
context.setReadBatchSize(readBatchSize);
|
||||||
|
String getFirstQuerySql = ObReaderUtils.buildFirstQuerySql(context);
|
||||||
|
String appendQuerySql = ObReaderUtils.buildAppendQuerySql(conn, context);
|
||||||
|
LOG.warn("start table scan key : {}", context.getIndexName() == null ? "primary" : context.getIndexName());
|
||||||
|
context.setQuerySql(getFirstQuerySql);
|
||||||
|
boolean firstQuery = true;
|
||||||
|
// 原来打算firstQuery时 limit 1 减少
|
||||||
|
// 后来经过对比发现其实是多余的,因为:
|
||||||
|
// 1.假如走gmt_modified辅助索引,则直接索引扫描 不需要topN的order by
|
||||||
|
// 2.假如不走辅助索引,而是pk table scan,则减少排序规模并没有好处,因为下一次仍然要排序
|
||||||
|
// 减少这个多余的优化tip 可以让代码更易读
|
||||||
|
int retryCount = 0;
|
||||||
|
while (true) {
|
||||||
|
try {
|
||||||
|
boolean finish = doRead(recordSender, taskPluginCollector, context);
|
||||||
|
if (finish) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
} catch (Throwable e) {
|
||||||
|
if (retryLimit == ++retryCount) {
|
||||||
|
throw RdbmsException.asQueryException(ObReaderUtils.databaseType, new Exception(e),
|
||||||
|
context.getQuerySql(), context.getTable(), username);
|
||||||
|
}
|
||||||
|
LOG.error("read fail, retry count " + retryCount + ", sleep 60 second, save point:" +
|
||||||
|
context.getSavePoint() + ", error: " + e.getMessage());
|
||||||
|
ObReaderUtils.sleep(60000); // sleep 10s
|
||||||
|
}
|
||||||
|
// 假如原来的查询有查出数据,则改成增量查询
|
||||||
|
if (firstQuery && context.getPkIndexs() != null && context.getSavePoint() != null) {
|
||||||
|
context.setQuerySql(appendQuerySql);
|
||||||
|
firstQuery = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
DBUtil.closeDBResources(null, context.getConn());
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean isConnectionAlive(Connection conn) {
|
||||||
|
if (conn == null) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
Statement stmt = null;
|
||||||
|
ResultSet rs = null;
|
||||||
|
String sql = "select 1" + (compatibleMode == ObReaderUtils.OB_COMPATIBLE_MODE_ORACLE ? " from dual" : "");
|
||||||
|
try {
|
||||||
|
stmt = conn.createStatement();
|
||||||
|
rs = stmt.executeQuery(sql);
|
||||||
|
rs.next();
|
||||||
|
} catch (Exception ex) {
|
||||||
|
LOG.info("connection is not alive: " + ex.getMessage());
|
||||||
|
return false;
|
||||||
|
} finally {
|
||||||
|
DBUtil.closeDBResources(rs, stmt, null);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean doRead(RecordSender recordSender, TaskPluginCollector taskPluginCollector, TaskContext context) {
|
||||||
|
LOG.info("exe sql: {}", context.getQuerySql());
|
||||||
|
Connection conn = context.getConn();
|
||||||
|
if (reuseConn && isConnectionAlive(conn)) {
|
||||||
|
LOG.info("connection is alive, will reuse this connection.");
|
||||||
|
} else {
|
||||||
|
LOG.info("Create new connection for reader.");
|
||||||
|
conn = DBUtil.getConnection(ObReaderUtils.databaseType, jdbcUrl, username, password);
|
||||||
|
ObReaderUtils.initConn4Reader(conn, queryTimeoutSeconds);
|
||||||
|
context.setConn(conn);
|
||||||
|
}
|
||||||
|
PreparedStatement ps = null;
|
||||||
|
ResultSet rs = null;
|
||||||
|
PerfRecord perfRecord = new PerfRecord(taskGroupId, taskId, PerfRecord.PHASE.SQL_QUERY);
|
||||||
|
perfRecord.start();
|
||||||
|
try {
|
||||||
|
ps = conn.prepareStatement(context.getQuerySql(),
|
||||||
|
ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
|
||||||
|
if (context.getPkIndexs() != null && context.getSavePoint() != null) {
|
||||||
|
Record savePoint = context.getSavePoint();
|
||||||
|
List<Column> point = ObReaderUtils.buildPoint(savePoint, context.getPkIndexs());
|
||||||
|
ObReaderUtils.binding(ps, point);
|
||||||
|
if (LOG.isWarnEnabled()) {
|
||||||
|
List<String> pointForLog = new ArrayList<String>();
|
||||||
|
for (Column c : point) {
|
||||||
|
pointForLog.add(c.asString());
|
||||||
|
}
|
||||||
|
LOG.warn("{} save point : {}", context.getTable(), StringUtils.join(pointForLog, ','));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 打开流式接口
|
||||||
|
ps.setFetchSize(context.getFetchSize());
|
||||||
|
rs = ps.executeQuery();
|
||||||
|
ResultSetMetaData metaData = rs.getMetaData();
|
||||||
|
int columnNumber = metaData.getColumnCount();
|
||||||
|
long lastTime = System.nanoTime();
|
||||||
|
int count = 0;
|
||||||
|
for (; rs.next(); count++) {
|
||||||
|
context.addCost(System.nanoTime() - lastTime);
|
||||||
|
Record row = buildRecord(recordSender, rs, metaData, columnNumber, mandatoryEncoding,
|
||||||
|
taskPluginCollector);
|
||||||
|
// // 如果第一个record重复了,则不需要发送
|
||||||
|
// if (count == 0 &&
|
||||||
|
// ObReaderUtils.isPkEquals(context.getSavePoint(), row,
|
||||||
|
// context.getPkIndexs())) {
|
||||||
|
// continue;
|
||||||
|
// }
|
||||||
|
// 如果是querySql
|
||||||
|
if (context.getTransferColumnNumber() == -1
|
||||||
|
|| row.getColumnNumber() == context.getTransferColumnNumber()) {
|
||||||
|
recordSender.sendToWriter(row);
|
||||||
|
} else {
|
||||||
|
Record newRow = recordSender.createRecord();
|
||||||
|
for (int i = 0; i < context.getTransferColumnNumber(); i++) {
|
||||||
|
newRow.addColumn(row.getColumn(i));
|
||||||
|
}
|
||||||
|
recordSender.sendToWriter(newRow);
|
||||||
|
}
|
||||||
|
context.setSavePoint(row);
|
||||||
|
lastTime = System.nanoTime();
|
||||||
|
}
|
||||||
|
LOG.info("end of sql: {}, " + count + "rows are read.", context.getQuerySql());
|
||||||
|
return context.getReadBatchSize() <= 0 || count < readBatchSize;
|
||||||
|
} catch (Exception e) {
|
||||||
|
ObReaderUtils.close(null, null, context.getConn());
|
||||||
|
context.setConn(null);
|
||||||
|
LOG.error("reader data fail", e);
|
||||||
|
throw RdbmsException.asQueryException(ObReaderUtils.databaseType, e, context.getQuerySql(),
|
||||||
|
context.getTable(), username);
|
||||||
|
} finally {
|
||||||
|
perfRecord.end();
|
||||||
|
if (reuseConn) {
|
||||||
|
ObReaderUtils.close(rs, ps, null);
|
||||||
|
} else {
|
||||||
|
ObReaderUtils.close(rs, ps, conn);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because one or more lines are too long
@ -0,0 +1,35 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.util;
|
||||||
|
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @author johnrobbet
|
||||||
|
*/
|
||||||
|
public class PartInfo {
|
||||||
|
|
||||||
|
private PartType partType;
|
||||||
|
|
||||||
|
List<String> partList;
|
||||||
|
|
||||||
|
public PartInfo(PartType partType) {
|
||||||
|
this.partType = partType;
|
||||||
|
this.partList = new ArrayList();
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getPartType () {
|
||||||
|
return partType.getTypeString();
|
||||||
|
}
|
||||||
|
|
||||||
|
public void addPart(List partList) {
|
||||||
|
this.partList.addAll(partList);
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<String> getPartList() {
|
||||||
|
return partList;
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean isPartitionTable() {
|
||||||
|
return partType != PartType.NONPARTITION && partList.size() > 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,23 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.util;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @author johnrobbet
|
||||||
|
*/
|
||||||
|
|
||||||
|
public enum PartType {
|
||||||
|
NONPARTITION("NONPARTITION"),
|
||||||
|
PARTITION("PARTITION"),
|
||||||
|
SUBPARTITION("SUBPARTITION");
|
||||||
|
|
||||||
|
private String typeString;
|
||||||
|
|
||||||
|
PartType (String typeString) {
|
||||||
|
this.typeString = typeString;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getTypeString() {
|
||||||
|
return typeString;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,165 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.util;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Constant;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Key;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.util.HintUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
import com.alibaba.datax.plugin.reader.oceanbasev10reader.ext.ObReaderKey;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.sql.ResultSet;
|
||||||
|
import java.sql.Statement;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @author johnrobbet
|
||||||
|
*/
|
||||||
|
public class PartitionSplitUtil {
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(PartitionSplitUtil.class);
|
||||||
|
|
||||||
|
public static List<Configuration> splitByPartition (Configuration configuration) {
|
||||||
|
List<Configuration> allSlices = new ArrayList<>();
|
||||||
|
List<Object> conns = configuration.getList(Constant.CONN_MARK, Object.class);
|
||||||
|
for (int i = 0, len = conns.size(); i < len; i++) {
|
||||||
|
Configuration sliceConfig = configuration.clone();
|
||||||
|
Configuration connConf = Configuration.from(conns.get(i).toString());
|
||||||
|
String jdbcUrl = connConf.getString(Key.JDBC_URL);
|
||||||
|
sliceConfig.set(Key.JDBC_URL, jdbcUrl);
|
||||||
|
sliceConfig.remove(Constant.CONN_MARK);
|
||||||
|
|
||||||
|
List<String> tables = connConf.getList(Key.TABLE, String.class);
|
||||||
|
for (String table : tables) {
|
||||||
|
Configuration tempSlice = sliceConfig.clone();
|
||||||
|
tempSlice.set(Key.TABLE, table);
|
||||||
|
allSlices.addAll(splitSinglePartitionTable(tempSlice));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return allSlices;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static List<Configuration> splitSinglePartitionTable(Configuration configuration) {
|
||||||
|
String table = configuration.getString(Key.TABLE);
|
||||||
|
String where = configuration.getString(Key.WHERE, null);
|
||||||
|
String column = configuration.getString(Key.COLUMN);
|
||||||
|
final boolean weakRead = configuration.getBool(Key.WEAK_READ, true);
|
||||||
|
|
||||||
|
List<Configuration> slices = new ArrayList();
|
||||||
|
PartInfo partInfo = getObPartInfoBySQL(configuration, table);
|
||||||
|
if (partInfo != null && partInfo.isPartitionTable()) {
|
||||||
|
String partitionType = partInfo.getPartType();
|
||||||
|
for (String partitionName : partInfo.getPartList()) {
|
||||||
|
LOG.info(String.format("add %s %s for table %s", partitionType, partitionName, table));
|
||||||
|
Configuration slice = configuration.clone();
|
||||||
|
slice.set(ObReaderKey.PARTITION_NAME, partitionName);
|
||||||
|
slice.set(ObReaderKey.PARTITION_TYPE, partitionType);
|
||||||
|
slice.set(Key.QUERY_SQL,
|
||||||
|
ObReaderUtils.buildQuerySql(weakRead, column,
|
||||||
|
String.format("%s partition(%s)", table, partitionName), where));
|
||||||
|
slices.add(slice);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
LOG.info("fail to get table part info or table is not partitioned, proceed as non-partitioned table.");
|
||||||
|
|
||||||
|
Configuration slice = configuration.clone();
|
||||||
|
slice.set(Key.QUERY_SQL, ObReaderUtils.buildQuerySql(weakRead, column, table, where));
|
||||||
|
slices.add(slice);
|
||||||
|
}
|
||||||
|
|
||||||
|
return slices;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static PartInfo getObPartInfoBySQL(Configuration config, String table) {
|
||||||
|
PartInfo partInfo = new PartInfo(PartType.NONPARTITION);
|
||||||
|
List<String> partList;
|
||||||
|
Connection conn = null;
|
||||||
|
try {
|
||||||
|
String jdbcUrl = config.getString(Key.JDBC_URL);
|
||||||
|
String username = config.getString(Key.USERNAME);
|
||||||
|
String password = config.getString(Key.PASSWORD);
|
||||||
|
String dbname = ObReaderUtils.getDbNameFromJdbcUrl(jdbcUrl);
|
||||||
|
String allTable = "__all_table";
|
||||||
|
|
||||||
|
conn = DBUtil.getConnection(DataBaseType.OceanBase, jdbcUrl, username, password);
|
||||||
|
String obVersion = getResultsFromSql(conn, "select version()").get(0);
|
||||||
|
|
||||||
|
LOG.info("obVersion: " + obVersion);
|
||||||
|
|
||||||
|
if (ObReaderUtils.compareObVersion("2.2.76", obVersion) < 0) {
|
||||||
|
allTable = "__all_table_v2";
|
||||||
|
}
|
||||||
|
|
||||||
|
String queryPart = String.format(
|
||||||
|
"select p.part_name " +
|
||||||
|
"from oceanbase.__all_part p, oceanbase.%s t, oceanbase.__all_database d " +
|
||||||
|
"where p.table_id = t.table_id " +
|
||||||
|
"and d.database_id = t.database_id " +
|
||||||
|
"and d.database_name = '%s' " +
|
||||||
|
"and t.table_name = '%s'", allTable, dbname, table);
|
||||||
|
String querySubPart = String.format(
|
||||||
|
"select p.sub_part_name " +
|
||||||
|
"from oceanbase.__all_sub_part p, oceanbase.%s t, oceanbase.__all_database d " +
|
||||||
|
"where p.table_id = t.table_id " +
|
||||||
|
"and d.database_id = t.database_id " +
|
||||||
|
"and d.database_name = '%s' " +
|
||||||
|
"and t.table_name = '%s'", allTable, dbname, table);
|
||||||
|
if (config.getString(ObReaderKey.OB_COMPATIBILITY_MODE).equals("ORACLE")) {
|
||||||
|
queryPart = String.format(
|
||||||
|
"select partition_name from all_tab_partitions where TABLE_OWNER = '%s' and table_name = '%s'",
|
||||||
|
dbname.toUpperCase(), table.toUpperCase());
|
||||||
|
querySubPart = String.format(
|
||||||
|
"select subpartition_name from all_tab_subpartitions where TABLE_OWNER = '%s' and table_name = '%s'",
|
||||||
|
dbname.toUpperCase(), table.toUpperCase());
|
||||||
|
}
|
||||||
|
|
||||||
|
PartType partType = PartType.SUBPARTITION;
|
||||||
|
|
||||||
|
// try subpartition first
|
||||||
|
partList = getResultsFromSql(conn, querySubPart);
|
||||||
|
|
||||||
|
// if table is not sub-partitioned, the try partition
|
||||||
|
if (partList.isEmpty()) {
|
||||||
|
partList = getResultsFromSql(conn, queryPart);
|
||||||
|
partType = PartType.PARTITION;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!partList.isEmpty()) {
|
||||||
|
partInfo = new PartInfo(partType);
|
||||||
|
partInfo.addPart(partList);
|
||||||
|
}
|
||||||
|
} catch (Exception ex) {
|
||||||
|
LOG.error("error when get partition list: " + ex.getMessage());
|
||||||
|
} finally {
|
||||||
|
DBUtil.closeDBResources(null, conn);
|
||||||
|
}
|
||||||
|
|
||||||
|
return partInfo;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static List<String> getResultsFromSql(Connection conn, String sql) {
|
||||||
|
List<String> list = new ArrayList();
|
||||||
|
Statement stmt = null;
|
||||||
|
ResultSet rs = null;
|
||||||
|
|
||||||
|
LOG.info("executing sql: " + sql);
|
||||||
|
|
||||||
|
try {
|
||||||
|
stmt = conn.createStatement();
|
||||||
|
rs = stmt.executeQuery(sql);
|
||||||
|
while (rs.next()) {
|
||||||
|
list.add(rs.getString(1));
|
||||||
|
}
|
||||||
|
} catch (Exception e) {
|
||||||
|
LOG.error("error when executing sql: " + e.getMessage());
|
||||||
|
} finally {
|
||||||
|
DBUtil.closeDBResources(rs, stmt, null);
|
||||||
|
}
|
||||||
|
|
||||||
|
return list;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,176 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.util;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.util.Collections;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.element.Record;
|
||||||
|
|
||||||
|
public class TaskContext {
|
||||||
|
private Connection conn;
|
||||||
|
private final String table;
|
||||||
|
private String indexName;
|
||||||
|
// 辅助索引的字段列表
|
||||||
|
private List<String> secondaryIndexColumns = Collections.emptyList();
|
||||||
|
private String querySql;
|
||||||
|
private final String where;
|
||||||
|
private final int fetchSize;
|
||||||
|
private long readBatchSize = -1;
|
||||||
|
private boolean weakRead = true;
|
||||||
|
private String userSavePoint;
|
||||||
|
private String compatibleMode = ObReaderUtils.OB_COMPATIBLE_MODE_MYSQL;
|
||||||
|
private String partitionName;
|
||||||
|
|
||||||
|
// 断点续读的保存点
|
||||||
|
private volatile Record savePoint;
|
||||||
|
|
||||||
|
// pk在column中的index,用于绑定变量时从savePoint中读取值
|
||||||
|
// 如果这个值为null,则表示 不是断点续读的场景
|
||||||
|
private int[] pkIndexs;
|
||||||
|
|
||||||
|
private final List<String> columns;
|
||||||
|
|
||||||
|
private String[] pkColumns;
|
||||||
|
|
||||||
|
private long cost;
|
||||||
|
|
||||||
|
private final int transferColumnNumber;
|
||||||
|
|
||||||
|
public TaskContext(String table, List<String> columns, String where, int fetchSize) {
|
||||||
|
super();
|
||||||
|
this.table = table;
|
||||||
|
this.columns = columns;
|
||||||
|
// 针对只有querySql的场景
|
||||||
|
this.transferColumnNumber = columns == null ? -1 : columns.size();
|
||||||
|
this.where = where;
|
||||||
|
this.fetchSize = fetchSize;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Connection getConn() {
|
||||||
|
return conn;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setConn(Connection conn) {
|
||||||
|
this.conn = conn;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getIndexName() {
|
||||||
|
return indexName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setIndexName(String indexName) {
|
||||||
|
this.indexName = indexName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<String> getSecondaryIndexColumns() {
|
||||||
|
return secondaryIndexColumns;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setSecondaryIndexColumns(List<String> secondaryIndexColumns) {
|
||||||
|
this.secondaryIndexColumns = secondaryIndexColumns;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getQuerySql() {
|
||||||
|
if (readBatchSize == -1 || ObReaderUtils.isOracleMode(compatibleMode)) {
|
||||||
|
return querySql;
|
||||||
|
} else {
|
||||||
|
return querySql + " limit " + readBatchSize;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setQuerySql(String querySql) {
|
||||||
|
this.querySql = querySql;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getWhere() {
|
||||||
|
return where;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Record getSavePoint() {
|
||||||
|
return savePoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setSavePoint(Record savePoint) {
|
||||||
|
this.savePoint = savePoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int[] getPkIndexs() {
|
||||||
|
return pkIndexs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setPkIndexs(int[] pkIndexs) {
|
||||||
|
this.pkIndexs = pkIndexs;
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<String> getColumns() {
|
||||||
|
return columns;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String[] getPkColumns() {
|
||||||
|
return pkColumns;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setPkColumns(String[] pkColumns) {
|
||||||
|
this.pkColumns = pkColumns;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getTable() {
|
||||||
|
return table;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int getFetchSize() {
|
||||||
|
return fetchSize;
|
||||||
|
}
|
||||||
|
|
||||||
|
public long getCost() {
|
||||||
|
return cost;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void addCost(long cost) {
|
||||||
|
this.cost += cost;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int getTransferColumnNumber() {
|
||||||
|
return transferColumnNumber;
|
||||||
|
}
|
||||||
|
|
||||||
|
public long getReadBatchSize() {
|
||||||
|
return readBatchSize;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setReadBatchSize(long readBatchSize) {
|
||||||
|
this.readBatchSize = readBatchSize;
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean getWeakRead() {
|
||||||
|
return weakRead;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setWeakRead(boolean weakRead) {
|
||||||
|
this.weakRead = weakRead;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getUserSavePoint() {
|
||||||
|
return userSavePoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setUserSavePoint(String userSavePoint) {
|
||||||
|
this.userSavePoint = userSavePoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getCompatibleMode() {
|
||||||
|
return compatibleMode;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setCompatibleMode(String compatibleMode) {
|
||||||
|
this.compatibleMode = compatibleMode;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getPartitionName() {
|
||||||
|
return partitionName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setPartitionName(String partitionName) {
|
||||||
|
this.partitionName = partitionName;
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
oceanbasev10reader/src/main/libs/oceanbase-client-1.1.10.jar
Normal file
BIN
oceanbasev10reader/src/main/libs/oceanbase-client-1.1.10.jar
Normal file
Binary file not shown.
6
oceanbasev10reader/src/main/resources/plugin.json
Normal file
6
oceanbasev10reader/src/main/resources/plugin.json
Normal file
@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"name": "oceanbasev10reader",
|
||||||
|
"class": "com.alibaba.datax.plugin.reader.oceanbasev10reader.OceanBaseReader",
|
||||||
|
"description": "read data from oceanbase with SQL interface",
|
||||||
|
"developer": "oceanbase"
|
||||||
|
}
|
||||||
@ -0,0 +1,22 @@
|
|||||||
|
package com.alibaba.datax.plugin.reader.oceanbasev10reader.util;
|
||||||
|
|
||||||
|
import org.junit.Test;
|
||||||
|
|
||||||
|
public class ObReaderUtilsTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
public void getDbTest() {
|
||||||
|
assert ObReaderUtils.getDbNameFromJdbcUrl("jdbc:mysql://127.0.0.1:3306/testdb").equalsIgnoreCase("testdb");
|
||||||
|
assert ObReaderUtils.getDbNameFromJdbcUrl("jdbc:oceanbase://127.0.0.1:2883/testdb").equalsIgnoreCase("testdb");
|
||||||
|
assert ObReaderUtils.getDbNameFromJdbcUrl("||_dsc_ob10_dsc_||obcluster:mysql||_dsc_ob10_dsc_||jdbc:mysql://127.0.0.1:3306/testdb").equalsIgnoreCase("testdb");
|
||||||
|
assert ObReaderUtils.getDbNameFromJdbcUrl("||_dsc_ob10_dsc_||obcluster:oracle||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:3306/testdb").equalsIgnoreCase("testdb");
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
public void compareObVersionTest() {
|
||||||
|
assert ObReaderUtils.compareObVersion("2.2.70", "3.2.2") == -1;
|
||||||
|
assert ObReaderUtils.compareObVersion("2.2.70", "2.2.50") == 1;
|
||||||
|
assert ObReaderUtils.compareObVersion("2.2.70", "3.1.2") == -1;
|
||||||
|
assert ObReaderUtils.compareObVersion("3.1.2", "3.1.2") == 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
126
oceanbasev10writer/pom.xml
Normal file
126
oceanbasev10writer/pom.xml
Normal file
@ -0,0 +1,126 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
|
<parent>
|
||||||
|
<artifactId>datax-all</artifactId>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<version>0.0.1-SNAPSHOT</version>
|
||||||
|
</parent>
|
||||||
|
<modelVersion>4.0.0</modelVersion>
|
||||||
|
|
||||||
|
<artifactId>oceanbasev10writer</artifactId>
|
||||||
|
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<version>0.0.1-SNAPSHOT</version>
|
||||||
|
|
||||||
|
<dependencies>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<artifactId>datax-common</artifactId>
|
||||||
|
<version>${datax-project-version}</version>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<artifactId>slf4j-log4j12</artifactId>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alibaba.datax</groupId>
|
||||||
|
<artifactId>plugin-rdbms-util</artifactId>
|
||||||
|
<version>${datax-project-version}</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.slf4j</groupId>
|
||||||
|
<artifactId>slf4j-api</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>ch.qos.logback</groupId>
|
||||||
|
<artifactId>logback-classic</artifactId>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.springframework</groupId>
|
||||||
|
<artifactId>spring-test</artifactId>
|
||||||
|
<version>4.0.4.RELEASE</version>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
<!--
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alipay.oceanbase</groupId>
|
||||||
|
<artifactId>oceanbase-partition</artifactId>
|
||||||
|
<version>0.0.5</version>
|
||||||
|
</dependency>
|
||||||
|
-->
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>com.alipay.oceanbase</groupId>
|
||||||
|
<artifactId>oceanbase-connector-java</artifactId>
|
||||||
|
<version>3.2.0</version>
|
||||||
|
<scope>system</scope>
|
||||||
|
<systemPath>${basedir}/src/main/libs/oceanbase-connector-java-3.2.0.jar</systemPath>
|
||||||
|
<exclusions>
|
||||||
|
<exclusion>
|
||||||
|
<groupId>com.alipay.oceanbase</groupId>
|
||||||
|
<artifactId>oceanbase-client</artifactId>
|
||||||
|
</exclusion>
|
||||||
|
</exclusions>
|
||||||
|
</dependency>
|
||||||
|
|
||||||
|
<dependency>
|
||||||
|
<groupId>log4j</groupId>
|
||||||
|
<artifactId>log4j</artifactId>
|
||||||
|
<version>1.2.16</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>org.json</groupId>
|
||||||
|
<artifactId>json</artifactId>
|
||||||
|
<version>20160810</version>
|
||||||
|
</dependency>
|
||||||
|
<dependency>
|
||||||
|
<groupId>junit</groupId>
|
||||||
|
<artifactId>junit</artifactId>
|
||||||
|
<version>4.11</version>
|
||||||
|
<scope>test</scope>
|
||||||
|
</dependency>
|
||||||
|
</dependencies>
|
||||||
|
|
||||||
|
<build>
|
||||||
|
<resources>
|
||||||
|
<resource>
|
||||||
|
<directory>src/main/java</directory>
|
||||||
|
<includes>
|
||||||
|
<include>**/*.properties</include>
|
||||||
|
</includes>
|
||||||
|
</resource>
|
||||||
|
</resources>
|
||||||
|
<plugins>
|
||||||
|
<!-- compiler plugin -->
|
||||||
|
<plugin>
|
||||||
|
<artifactId>maven-compiler-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<source>${jdk-version}</source>
|
||||||
|
<target>${jdk-version}</target>
|
||||||
|
<encoding>${project-sourceEncoding}</encoding>
|
||||||
|
</configuration>
|
||||||
|
</plugin>
|
||||||
|
<!-- assembly plugin -->
|
||||||
|
<plugin>
|
||||||
|
<artifactId>maven-assembly-plugin</artifactId>
|
||||||
|
<configuration>
|
||||||
|
<descriptors>
|
||||||
|
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||||
|
</descriptors>
|
||||||
|
<finalName>datax</finalName>
|
||||||
|
</configuration>
|
||||||
|
<executions>
|
||||||
|
<execution>
|
||||||
|
<id>dwzip</id>
|
||||||
|
<phase>package</phase>
|
||||||
|
<goals>
|
||||||
|
<goal>single</goal>
|
||||||
|
</goals>
|
||||||
|
</execution>
|
||||||
|
</executions>
|
||||||
|
</plugin>
|
||||||
|
</plugins>
|
||||||
|
</build>
|
||||||
|
</project>
|
||||||
42
oceanbasev10writer/src/main/assembly/package.xml
Normal file
42
oceanbasev10writer/src/main/assembly/package.xml
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
<assembly
|
||||||
|
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
|
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||||
|
<id></id>
|
||||||
|
<formats>
|
||||||
|
<format>dir</format>
|
||||||
|
</formats>
|
||||||
|
<includeBaseDirectory>false</includeBaseDirectory>
|
||||||
|
<fileSets>
|
||||||
|
<fileSet>
|
||||||
|
<directory>src/main/resources</directory>
|
||||||
|
<includes>
|
||||||
|
<include>plugin.json</include>
|
||||||
|
<include>plugin_job_template.json</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/writer/oceanbasev10writer</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>target/</directory>
|
||||||
|
<includes>
|
||||||
|
<include>oceanbasev10writer-0.0.1-SNAPSHOT.jar</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/writer/oceanbasev10writer</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
<fileSet>
|
||||||
|
<directory>src/main/libs</directory>
|
||||||
|
<includes>
|
||||||
|
<include>*.jar</include>
|
||||||
|
</includes>
|
||||||
|
<outputDirectory>plugin/writer/oceanbasev10writer/libs</outputDirectory>
|
||||||
|
</fileSet>
|
||||||
|
</fileSets>
|
||||||
|
|
||||||
|
<dependencySets>
|
||||||
|
<dependencySet>
|
||||||
|
<useProjectArtifact>false</useProjectArtifact>
|
||||||
|
<outputDirectory>plugin/writer/oceanbasev10writer/libs</outputDirectory>
|
||||||
|
<scope>runtime</scope>
|
||||||
|
</dependencySet>
|
||||||
|
</dependencySets>
|
||||||
|
</assembly>
|
||||||
@ -0,0 +1,62 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.oceanbasev10writer;
|
||||||
|
|
||||||
|
public interface Config {
|
||||||
|
|
||||||
|
String MEMSTORE_THRESHOLD = "memstoreThreshold";
|
||||||
|
|
||||||
|
double DEFAULT_MEMSTORE_THRESHOLD = 0.9d;
|
||||||
|
|
||||||
|
String MEMSTORE_CHECK_INTERVAL_SECOND = "memstoreCheckIntervalSecond";
|
||||||
|
|
||||||
|
long DEFAULT_MEMSTORE_CHECK_INTERVAL_SECOND = 30;
|
||||||
|
|
||||||
|
int DEFAULT_BATCH_SIZE = 100;
|
||||||
|
int MAX_BATCH_SIZE = 4096;
|
||||||
|
|
||||||
|
String FAIL_TRY_COUNT = "failTryCount";
|
||||||
|
|
||||||
|
int DEFAULT_FAIL_TRY_COUNT = 10000;
|
||||||
|
|
||||||
|
String WRITER_THREAD_COUNT = "writerThreadCount";
|
||||||
|
|
||||||
|
int DEFAULT_WRITER_THREAD_COUNT = 1;
|
||||||
|
|
||||||
|
String CONCURRENT_WRITE = "concurrentWrite";
|
||||||
|
|
||||||
|
boolean DEFAULT_CONCURRENT_WRITE = true;
|
||||||
|
|
||||||
|
String OB_VERSION = "obVersion";
|
||||||
|
String TIMEOUT = "timeout";
|
||||||
|
|
||||||
|
String PRINT_COST = "printCost";
|
||||||
|
boolean DEFAULT_PRINT_COST = false;
|
||||||
|
|
||||||
|
String COST_BOUND = "costBound";
|
||||||
|
long DEFAULT_COST_BOUND = 20;
|
||||||
|
|
||||||
|
String MAX_ACTIVE_CONNECTION = "maxActiveConnection";
|
||||||
|
int DEFAULT_MAX_ACTIVE_CONNECTION = 2000;
|
||||||
|
|
||||||
|
String WRITER_SUB_TASK_COUNT = "writerSubTaskCount";
|
||||||
|
int DEFAULT_WRITER_SUB_TASK_COUNT = 1;
|
||||||
|
int MAX_WRITER_SUB_TASK_COUNT = 4096;
|
||||||
|
|
||||||
|
String OB_WRITE_MODE = "obWriteMode";
|
||||||
|
String OB_COMPATIBLE_MODE = "obCompatibilityMode";
|
||||||
|
String OB_COMPATIBLE_MODE_ORACLE = "ORACLE";
|
||||||
|
String OB_COMPATIBLE_MODE_MYSQL = "MYSQL";
|
||||||
|
|
||||||
|
String OCJ_GET_CONNECT_TIMEOUT = "ocjGetConnectTimeout";
|
||||||
|
int DEFAULT_OCJ_GET_CONNECT_TIMEOUT = 5000; // 5s
|
||||||
|
|
||||||
|
String OCJ_PROXY_CONNECT_TIMEOUT = "ocjProxyConnectTimeout";
|
||||||
|
int DEFAULT_OCJ_PROXY_CONNECT_TIMEOUT = 5000; // 5s
|
||||||
|
|
||||||
|
String OCJ_CREATE_RESOURCE_TIMEOUT = "ocjCreateResourceTimeout";
|
||||||
|
int DEFAULT_OCJ_CREATE_RESOURCE_TIMEOUT = 60000; // 60s
|
||||||
|
|
||||||
|
String OB_UPDATE_COLUMNS = "obUpdateColumns";
|
||||||
|
|
||||||
|
String USE_PART_CALCULATOR = "usePartCalculator";
|
||||||
|
boolean DEFAULT_USE_PART_CALCULATOR = false;
|
||||||
|
}
|
||||||
@ -0,0 +1,258 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.oceanbasev10writer;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.plugin.RecordReceiver;
|
||||||
|
import com.alibaba.datax.common.spi.Writer;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.writer.Constant;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.writer.Key;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.writer.util.WriterUtil;
|
||||||
|
import com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask;
|
||||||
|
import com.alibaba.datax.plugin.writer.oceanbasev10writer.util.DbUtils;
|
||||||
|
import com.alibaba.datax.plugin.writer.oceanbasev10writer.util.ObWriterUtils;
|
||||||
|
import com.alibaba.fastjson.JSONObject;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 2016-04-07
|
||||||
|
* <p>
|
||||||
|
* 专门针对OceanBase1.0的Writer
|
||||||
|
*
|
||||||
|
* @author biliang.wbl
|
||||||
|
*
|
||||||
|
*/
|
||||||
|
public class OceanBaseV10Writer extends Writer {
|
||||||
|
private static DataBaseType DATABASE_TYPE = DataBaseType.OceanBase;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Job 中的方法仅执行一次，Task 中方法会由框架启动多个 Task 线程并行执行。
|
||||||
|
* <p/>
|
||||||
|
* 整个 Writer 执行流程是：
|
||||||
|
*
|
||||||
|
* <pre>
|
||||||
|
* Job类init-->prepare-->split
|
||||||
|
*
|
||||||
|
* Task类init-->prepare-->startWrite-->post-->destroy
|
||||||
|
* Task类init-->prepare-->startWrite-->post-->destroy
|
||||||
|
*
|
||||||
|
* Job类post-->destroy
|
||||||
|
* </pre>
|
||||||
|
*/
|
||||||
|
public static class Job extends Writer.Job {
|
||||||
|
private Configuration originalConfig = null;
|
||||||
|
private CommonRdbmsWriter.Job commonJob;
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(Job.class);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法仅执行一次。 最佳实践：通常在这里对用户的配置进行校验：是否缺失必填项？有无错误值？有没有无关配置项？...
|
||||||
|
* 并给出清晰的报错/警告提示。校验通常建议采用静态工具类进行，以保证本类结构清晰。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void init() {
|
||||||
|
this.originalConfig = super.getPluginJobConf();
|
||||||
|
checkCompatibleMode(originalConfig);
|
||||||
|
//将config中的column和table中的关键字进行转义
|
||||||
|
List<String> columns = originalConfig.getList(Key.COLUMN, String.class);
|
||||||
|
ObWriterUtils.escapeDatabaseKeywords(columns);
|
||||||
|
originalConfig.set(Key.COLUMN, columns);
|
||||||
|
|
||||||
|
List<JSONObject> conns = originalConfig.getList(Constant.CONN_MARK, JSONObject.class);
|
||||||
|
for (int i = 0; i < conns.size(); i++) {
|
||||||
|
JSONObject conn = conns.get(i);
|
||||||
|
Configuration connConfig = Configuration.from(conn.toString());
|
||||||
|
List<String> tables = connConfig.getList(Key.TABLE, String.class);
|
||||||
|
ObWriterUtils.escapeDatabaseKeywords(tables);
|
||||||
|
originalConfig.set(String.format("%s[%d].%s", Constant.CONN_MARK, i, Key.TABLE), tables);
|
||||||
|
}
|
||||||
|
this.commonJob = new CommonRdbmsWriter.Job(DATABASE_TYPE);
|
||||||
|
this.commonJob.init(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法仅执行一次。 最佳实践：如果 Job 中有需要进行数据同步之前的处理，可以在此处完成，如果没有必要则可以直接去掉。
|
||||||
|
*/
|
||||||
|
// 一般来说，是需要推迟到 task 中进行pre 的执行（单表情况例外）
|
||||||
|
@Override
|
||||||
|
public void prepare() {
|
||||||
|
int tableNumber = originalConfig.getInt(Constant.TABLE_NUMBER_MARK);
|
||||||
|
if (tableNumber == 1) {
|
||||||
|
this.commonJob.prepare(this.originalConfig);
|
||||||
|
final String version = fetchServerVersion(originalConfig);
|
||||||
|
originalConfig.set(Config.OB_VERSION, version);
|
||||||
|
}
|
||||||
|
|
||||||
|
String username = originalConfig.getString(Key.USERNAME);
|
||||||
|
String password = originalConfig.getString(Key.PASSWORD);
|
||||||
|
|
||||||
|
// 获取presql配置，并执行
|
||||||
|
List<String> preSqls = originalConfig.getList(Key.PRE_SQL, String.class);
|
||||||
|
if (preSqls == null || preSqls.size() == 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
List<Object> conns = originalConfig.getList(Constant.CONN_MARK, Object.class);
|
||||||
|
for (Object connConfObject : conns) {
|
||||||
|
Configuration connConf = Configuration.from(connConfObject.toString());
|
||||||
|
// 这里的 jdbcUrl 已经 append 了合适后缀参数
|
||||||
|
String jdbcUrl = connConf.getString(Key.JDBC_URL);
|
||||||
|
|
||||||
|
List<String> tableList = connConf.getList(Key.TABLE, String.class);
|
||||||
|
for (String table : tableList) {
|
||||||
|
List<String> renderedPreSqls = WriterUtil.renderPreOrPostSqls(preSqls, table);
|
||||||
|
if (null != renderedPreSqls && !renderedPreSqls.isEmpty()) {
|
||||||
|
Connection conn = DBUtil.getConnection(DATABASE_TYPE, jdbcUrl, username, password);
|
||||||
|
LOG.info("Begin to execute preSqls:[{}]. context info:{}.",
|
||||||
|
StringUtils.join(renderedPreSqls, ";"), jdbcUrl);
|
||||||
|
WriterUtil.executeSqls(conn, renderedPreSqls, jdbcUrl, DATABASE_TYPE);
|
||||||
|
ObWriterUtils.asyncClose(null, null, conn);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (LOG.isDebugEnabled()) {
|
||||||
|
LOG.debug("After job prepare(), originalConfig now is:[\n{}\n]", originalConfig.toJSON());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法仅执行一次。 最佳实践：通常采用工具静态类完成把 Job 配置切分成多个 Task 配置的工作。 这里的
|
||||||
|
* mandatoryNumber 是强制必须切分的份数。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public List<Configuration> split(int mandatoryNumber) {
|
||||||
|
int tableNumber = originalConfig.getInt(Constant.TABLE_NUMBER_MARK);
|
||||||
|
if (tableNumber == 1) {
|
||||||
|
return this.commonJob.split(this.originalConfig, mandatoryNumber);
|
||||||
|
}
|
||||||
|
Configuration simplifiedConf = this.originalConfig;
|
||||||
|
|
||||||
|
List<Configuration> splitResultConfigs = new ArrayList<Configuration>();
|
||||||
|
for (int j = 0; j < mandatoryNumber; j++) {
|
||||||
|
splitResultConfigs.add(simplifiedConf.clone());
|
||||||
|
}
|
||||||
|
return splitResultConfigs;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法仅执行一次。 最佳实践：如果 Job 中有需要进行数据同步之后的后续处理，可以在此处完成。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void post() {
|
||||||
|
int tableNumber = originalConfig.getInt(Constant.TABLE_NUMBER_MARK);
|
||||||
|
if (tableNumber == 1) {
|
||||||
|
commonJob.post(this.originalConfig);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
String username = originalConfig.getString(Key.USERNAME);
|
||||||
|
String password = originalConfig.getString(Key.PASSWORD);
|
||||||
|
List<Object> conns = originalConfig.getList(Constant.CONN_MARK, Object.class);
|
||||||
|
List<String> postSqls = originalConfig.getList(Key.POST_SQL, String.class);
|
||||||
|
if (postSqls == null || postSqls.size() == 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (Object connConfObject : conns) {
|
||||||
|
Configuration connConf = Configuration.from(connConfObject.toString());
|
||||||
|
String jdbcUrl = connConf.getString(Key.JDBC_URL);
|
||||||
|
List<String> tableList = connConf.getList(Key.TABLE, String.class);
|
||||||
|
|
||||||
|
for (String table : tableList) {
|
||||||
|
List<String> renderedPostSqls = WriterUtil.renderPreOrPostSqls(postSqls, table);
|
||||||
|
if (null != renderedPostSqls && !renderedPostSqls.isEmpty()) {
|
||||||
|
// 说明有 postSql 配置，则此处删除掉
|
||||||
|
Connection conn = DBUtil.getConnection(DATABASE_TYPE, jdbcUrl, username, password);
|
||||||
|
LOG.info("Begin to execute postSqls:[{}]. context info:{}.",
|
||||||
|
StringUtils.join(renderedPostSqls, ";"), jdbcUrl);
|
||||||
|
WriterUtil.executeSqls(conn, renderedPostSqls, jdbcUrl, DATABASE_TYPE);
|
||||||
|
ObWriterUtils.asyncClose(null, null, conn);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
originalConfig.remove(Key.POST_SQL);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法仅执行一次。 最佳实践：通常配合 Job 中的 post() 方法一起完成 Job 的资源释放。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void destroy() {
|
||||||
|
this.commonJob.destroy(this.originalConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
private String fetchServerVersion(Configuration config) {
|
||||||
|
final String fetchVersionSql = "show variables like 'version'";
|
||||||
|
return DbUtils.fetchSingleValueWithRetry(config, fetchVersionSql);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void checkCompatibleMode(Configuration configure) {
|
||||||
|
final String fetchCompatibleModeSql = "SHOW VARIABLES LIKE 'ob_compatibility_mode'";
|
||||||
|
String compatibleMode = DbUtils.fetchSingleValueWithRetry(configure, fetchCompatibleModeSql);
|
||||||
|
ObWriterUtils.setCompatibleMode(compatibleMode);
|
||||||
|
configure.set(Config.OB_COMPATIBLE_MODE, compatibleMode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static class Task extends Writer.Task {
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(Task.class);
|
||||||
|
private Configuration writerSliceConfig;
|
||||||
|
private CommonRdbmsWriter.Task writerTask;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法每个 Task 都会执行一次。 最佳实践：此处通过对 taskConfig 配置的读取，进而初始化一些资源为
|
||||||
|
* startWrite()做准备。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void init() {
|
||||||
|
this.writerSliceConfig = super.getPluginJobConf();
|
||||||
|
int tableNumber = writerSliceConfig.getInt(Constant.TABLE_NUMBER_MARK);
|
||||||
|
if (tableNumber == 1) {
|
||||||
|
// always use concurrentTableWriter
|
||||||
|
this.writerTask = new ConcurrentTableWriterTask(DATABASE_TYPE);
|
||||||
|
} else {
|
||||||
|
throw new RuntimeException("writing to multi-tables is not supported.");
|
||||||
|
}
|
||||||
|
LOG.info("tableNumber:" + tableNumber + ",writerTask Class:" + writerTask.getClass().getName());
|
||||||
|
this.writerTask.init(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法每个 Task 都会执行一次。 最佳实践：如果 Task
|
||||||
|
* 中有需要进行数据同步之前的处理，可以在此处完成，如果没有必要则可以直接去掉。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void prepare() {
|
||||||
|
this.writerTask.prepare(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法每个 Task 都会执行一次。 最佳实践：此处适当封装确保简洁清晰完成数据写入工作。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void startWrite(RecordReceiver recordReceiver) {
|
||||||
|
this.writerTask.startWrite(recordReceiver, this.writerSliceConfig, super.getTaskPluginCollector());
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法每个 Task 都会执行一次。 最佳实践：如果 Task 中有需要进行数据同步之后的后续处理，可以在此处完成。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void post() {
|
||||||
|
this.writerTask.post(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 注意：此方法每个 Task 都会执行一次。 最佳实践：通常配合Task 中的 post() 方法一起完成 Task 的资源释放。
|
||||||
|
*/
|
||||||
|
@Override
|
||||||
|
public void destroy() {
|
||||||
|
this.writerTask.destroy(this.writerSliceConfig);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,37 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.oceanbasev10writer.ext;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
|
||||||
|
public abstract class ConnHolder {
|
||||||
|
|
||||||
|
protected final Configuration config;
|
||||||
|
protected Connection conn;
|
||||||
|
|
||||||
|
public ConnHolder(Configuration config) {
|
||||||
|
this.config = config;
|
||||||
|
}
|
||||||
|
|
||||||
|
public abstract Connection initConnection();
|
||||||
|
|
||||||
|
public Configuration getConfig() {
|
||||||
|
return config;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Connection getConn() {
|
||||||
|
return conn;
|
||||||
|
}
|
||||||
|
|
||||||
|
public Connection reconnect() {
|
||||||
|
DBUtil.closeDBResources(null, conn);
|
||||||
|
return initConnection();
|
||||||
|
}
|
||||||
|
|
||||||
|
public abstract String getJdbcUrl();
|
||||||
|
|
||||||
|
public abstract String getUserName();
|
||||||
|
|
||||||
|
public abstract void destroy();
|
||||||
|
}
|
||||||
@ -0,0 +1,101 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.oceanbasev10writer.ext;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.LinkedList;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.element.Record;
|
||||||
|
import com.alibaba.datax.common.exception.DataXException;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtilErrorCode;
|
||||||
|
import com.alibaba.datax.plugin.writer.oceanbasev10writer.util.ObWriterUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
/**
|
||||||
|
*
|
||||||
|
* @author oceanbase
|
||||||
|
*
|
||||||
|
*/
|
||||||
|
public class DataBaseWriterBuffer {
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(DataBaseWriterBuffer.class);
|
||||||
|
|
||||||
|
private final ConnHolder connHolder;
|
||||||
|
private final String dbName;
|
||||||
|
private Map<String, LinkedList<Record>> tableBuffer = new HashMap<String, LinkedList<Record>>();
|
||||||
|
private long lastCheckMemstoreTime;
|
||||||
|
|
||||||
|
public DataBaseWriterBuffer(Configuration config,String jdbcUrl, String userName, String password,String dbName){
|
||||||
|
this.connHolder = new ObClientConnHolder(config, jdbcUrl, userName, password);
|
||||||
|
this.dbName=dbName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public ConnHolder getConnHolder(){
|
||||||
|
return connHolder;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void initTableBuffer(List<String> tableList) {
|
||||||
|
for (String table : tableList) {
|
||||||
|
tableBuffer.put(table, new LinkedList<Record>());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<String> getTableList(){
|
||||||
|
return new ArrayList<String>(tableBuffer.keySet());
|
||||||
|
}
|
||||||
|
|
||||||
|
public void addRecord(Record record, String tableName) {
|
||||||
|
LinkedList<Record> recordList = tableBuffer.get(tableName);
|
||||||
|
if (recordList == null) {
|
||||||
|
throw DataXException.asDataXException(DBUtilErrorCode.WRITE_DATA_ERROR,
|
||||||
|
String.format("The [table] calculated based on the rules does not exist. The calculated [tableName]=%s, [db]=%s. Please check the rules you configured.",
|
||||||
|
tableName, connHolder.getJdbcUrl()));
|
||||||
|
}
|
||||||
|
recordList.add(record);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Map<String, LinkedList<Record>> getTableBuffer() {
|
||||||
|
return tableBuffer;
|
||||||
|
}
|
||||||
|
|
||||||
|
public String getDbName() {
|
||||||
|
return dbName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public long getLastCheckMemstoreTime() {
|
||||||
|
return lastCheckMemstoreTime;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void setLastCheckMemstoreTime(long lastCheckMemstoreTime) {
|
||||||
|
this.lastCheckMemstoreTime = lastCheckMemstoreTime;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 检查当前DB的memstore使用状态
|
||||||
|
* <p>
|
||||||
|
* 若超过阈值,则休眠
|
||||||
|
*
|
||||||
|
* @param memstoreCheckIntervalSecond
|
||||||
|
* @param memstoreThreshold
|
||||||
|
*/
|
||||||
|
public synchronized void checkMemstore(long memstoreCheckIntervalSecond, double memstoreThreshold) {
|
||||||
|
long now = System.currentTimeMillis();
|
||||||
|
if (now - getLastCheckMemstoreTime() < 1000 * memstoreCheckIntervalSecond) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
LOG.debug(String.format("checking memstore usage: lastCheckTime=%d, now=%d, check interval=%d, threshold=%f",
|
||||||
|
getLastCheckMemstoreTime(), now, memstoreCheckIntervalSecond, memstoreThreshold));
|
||||||
|
|
||||||
|
Connection conn = getConnHolder().getConn();
|
||||||
|
while (ObWriterUtils.isMemstoreFull(conn, memstoreThreshold)) {
|
||||||
|
LOG.warn("OB memstore is full,sleep 60 seconds, jdbc=" + getConnHolder().getJdbcUrl()
|
||||||
|
+ ",threshold=" + memstoreThreshold);
|
||||||
|
ObWriterUtils.sleep(60000);
|
||||||
|
}
|
||||||
|
setLastCheckMemstoreTime(now);
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,190 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.oceanbasev10writer.ext;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
import java.sql.SQLException;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
|
import com.alibaba.datax.plugin.rdbms.reader.Key;
|
||||||
|
import com.alibaba.datax.plugin.writer.oceanbasev10writer.util.ObWriterUtils;
|
||||||
|
import org.slf4j.Logger;
|
||||||
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.exception.DataXException;
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.writer.oceanbasev10writer.Config;
|
||||||
|
import com.alipay.oceanbase.obproxy.datasource.ObGroupDataSource;
|
||||||
|
import com.alipay.oceanbase.obproxy.exception.ConnectionPropertiesNotSupportedException;
|
||||||
|
import com.alipay.oceanbase.obproxy.util.StringParser.IllegalFormatException;
|
||||||
|
import com.google.common.collect.Maps;
|
||||||
|
|
||||||
|
public class OBDataSourceV10 {
|
||||||
|
private static final Logger LOG = LoggerFactory.getLogger(OBDataSourceV10.class);
|
||||||
|
|
||||||
|
private static final Map<String, DataSourceHolder> dataSources = Maps.newHashMap();
|
||||||
|
|
||||||
|
private static int ocjGetConnectionTimeout = 0;
|
||||||
|
private static int ocjGlobalProxyroGetConnectionTimeout = 0;
|
||||||
|
private static int ocjMaxWaitOfCreateClusterResourceMs = 0;
|
||||||
|
|
||||||
|
private static Configuration taskConfig;
|
||||||
|
|
||||||
|
public static String genKey(String fullUserName, String dbName) {
|
||||||
|
//username@tenantName#clusterName/dbName
|
||||||
|
return fullUserName + "/" + dbName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static synchronized void init(Configuration configuration,
|
||||||
|
final String fullUsername,
|
||||||
|
final String password,
|
||||||
|
final String dbName) {
|
||||||
|
taskConfig = configuration;
|
||||||
|
final String rsUrl = "";
|
||||||
|
final String dataSourceKey = genKey(fullUsername, dbName);
|
||||||
|
final int maxActiveConnection = configuration.getInt(Config.MAX_ACTIVE_CONNECTION, Config.DEFAULT_MAX_ACTIVE_CONNECTION);
|
||||||
|
if (dataSources.containsKey(dataSourceKey)) {
|
||||||
|
dataSources.get(dataSourceKey).increseRefercnce();
|
||||||
|
} else {
|
||||||
|
long timeout = configuration.getInt(Config.TIMEOUT, 30);
|
||||||
|
if (timeout < 30) {
|
||||||
|
timeout = 30;
|
||||||
|
}
|
||||||
|
if (ocjGetConnectionTimeout == 0) {
|
||||||
|
ocjGetConnectionTimeout = configuration.getInt(Config.OCJ_GET_CONNECT_TIMEOUT,
|
||||||
|
Config.DEFAULT_OCJ_GET_CONNECT_TIMEOUT);
|
||||||
|
ocjGlobalProxyroGetConnectionTimeout = configuration.getInt(Config.OCJ_PROXY_CONNECT_TIMEOUT,
|
||||||
|
Config.DEFAULT_OCJ_PROXY_CONNECT_TIMEOUT);
|
||||||
|
ocjMaxWaitOfCreateClusterResourceMs = configuration.getInt(Config.OCJ_CREATE_RESOURCE_TIMEOUT,
|
||||||
|
Config.DEFAULT_OCJ_CREATE_RESOURCE_TIMEOUT);
|
||||||
|
|
||||||
|
LOG.info(String.format("initializing OCJ with ocjGetConnectionTimeout=%d, " +
|
||||||
|
"ocjGlobalProxyroGetConnectionTimeout=%d, ocjMaxWaitOfCreateClusterResourceMs=%d",
|
||||||
|
ocjGetConnectionTimeout, ocjGlobalProxyroGetConnectionTimeout, ocjMaxWaitOfCreateClusterResourceMs));
|
||||||
|
}
|
||||||
|
DataSourceHolder holder = null;
|
||||||
|
try {
|
||||||
|
holder = new DataSourceHolder(rsUrl, fullUsername, password, dbName, maxActiveConnection, timeout);
|
||||||
|
dataSources.put(dataSourceKey, holder);
|
||||||
|
} catch (ConnectionPropertiesNotSupportedException e) {
|
||||||
|
e.printStackTrace();
|
||||||
|
throw new DataXException(ObDataSourceErrorCode.DESC, "connect error");
|
||||||
|
} catch (IllegalArgumentException e) {
|
||||||
|
e.printStackTrace();
|
||||||
|
throw new DataXException(ObDataSourceErrorCode.DESC, "connect error");
|
||||||
|
} catch (IllegalFormatException e) {
|
||||||
|
e.printStackTrace();
|
||||||
|
throw new DataXException(ObDataSourceErrorCode.DESC, "connect error");
|
||||||
|
} catch (SQLException e) {
|
||||||
|
e.printStackTrace();
|
||||||
|
throw new DataXException(ObDataSourceErrorCode.DESC, "connect error");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static synchronized void destory(final String dataSourceKey){
|
||||||
|
DataSourceHolder holder = dataSources.get(dataSourceKey);
|
||||||
|
holder.decreaseReference();
|
||||||
|
if (holder.canClose()) {
|
||||||
|
dataSources.remove(dataSourceKey);
|
||||||
|
holder.close();
|
||||||
|
LOG.info(String.format("close datasource success [%s]", dataSourceKey));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static Connection getConnection(final String url) {
|
||||||
|
Connection conn = null;
|
||||||
|
try {
|
||||||
|
conn = dataSources.get(url).getconnection();
|
||||||
|
} catch (SQLException e) {
|
||||||
|
e.printStackTrace();
|
||||||
|
}
|
||||||
|
return conn;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static Map<String, String> buildJdbcProperty() {
|
||||||
|
Map<String, String> property = new HashMap<String, String>();
|
||||||
|
property.put("useServerPrepStmts", "false");
|
||||||
|
property.put("characterEncoding", "UTF-8");
|
||||||
|
property.put("useLocalSessionState", "false");
|
||||||
|
property.put("rewriteBatchedStatements", "true");
|
||||||
|
property.put("socketTimeout", "25000");
|
||||||
|
|
||||||
|
return property;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static class DataSourceHolder {
|
||||||
|
private volatile int reference;
|
||||||
|
private final ObGroupDataSource groupDataSource;
|
||||||
|
public static final Map<String, String> jdbcProperty = buildJdbcProperty();;
|
||||||
|
|
||||||
|
public DataSourceHolder(final String rsUrl,
|
||||||
|
final String fullUsername,
|
||||||
|
final String password,
|
||||||
|
final String dbName,
|
||||||
|
final int maxActive,
|
||||||
|
final long timeout) throws ConnectionPropertiesNotSupportedException, IllegalFormatException, IllegalArgumentException, SQLException {
|
||||||
|
this.reference = 1;
|
||||||
|
this.groupDataSource = new ObGroupDataSource();
|
||||||
|
this.groupDataSource.setUrl(rsUrl);
|
||||||
|
this.groupDataSource.setFullUsername(fullUsername);
|
||||||
|
this.groupDataSource.setPassword(password);
|
||||||
|
this.groupDataSource.setDatabase(dbName);
|
||||||
|
this.groupDataSource.setConnectionProperties(jdbcProperty);
|
||||||
|
this.groupDataSource.setGetConnectionTimeout(ocjGetConnectionTimeout);
|
||||||
|
this.groupDataSource.setGlobalProxyroGetConnectionTimeout(ocjGlobalProxyroGetConnectionTimeout);
|
||||||
|
this.groupDataSource.setMaxWaitOfCreateClusterResourceMs(ocjMaxWaitOfCreateClusterResourceMs);
|
||||||
|
this.groupDataSource.setMaxActive(maxActive);
|
||||||
|
this.groupDataSource.setGlobalSlowQueryThresholdUs(3000000); // 3s, sql with response time more than 3s will be logged
|
||||||
|
this.groupDataSource.setGlobalCleanLogFileEnabled(true); // enable log cleanup
|
||||||
|
this.groupDataSource.setGlobalLogFileSizeThreshold(17179869184L); // 16G, log file total size
|
||||||
|
this.groupDataSource.setGlobalCleanLogFileInterval(10000); // 10s, check interval
|
||||||
|
this.groupDataSource.setInitialSize(1);
|
||||||
|
|
||||||
|
List<String> initSqls = new ArrayList<String>();
|
||||||
|
if (taskConfig != null) {
|
||||||
|
List<String> sessionConfig = taskConfig.getList(Key.SESSION, new ArrayList(), String.class);
|
||||||
|
if (sessionConfig != null || sessionConfig.size() > 0) {
|
||||||
|
initSqls.addAll(sessionConfig);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// set up for writing timestamp columns
|
||||||
|
if (ObWriterUtils.isOracleMode()) {
|
||||||
|
initSqls.add("ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS';");
|
||||||
|
initSqls.add("ALTER SESSION SET NLS_TIMESTAMP_FORMAT='YYYY-MM-DD HH24:MI:SS.FF';");
|
||||||
|
initSqls.add("ALTER SESSION SET NLS_TIMESTAMP_TZ_FORMAT='YYYY-MM-DD HH24:MI:SS.FF TZR TZD';");
|
||||||
|
}
|
||||||
|
|
||||||
|
this.groupDataSource.setConnectionInitSqls(initSqls);
|
||||||
|
|
||||||
|
this.groupDataSource.init();
|
||||||
|
// this.groupDataSource;
|
||||||
|
LOG.info("Create GroupDataSource rsUrl=[{}], fullUserName=[{}], dbName=[{}], getConnectionTimeout= {}ms, maxActive={}",
|
||||||
|
rsUrl, fullUsername, dbName, 5000, maxActive);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Connection getconnection() throws SQLException {
|
||||||
|
return groupDataSource.getConnection();
|
||||||
|
}
|
||||||
|
|
||||||
|
public synchronized void increseRefercnce() {
|
||||||
|
this.reference++;
|
||||||
|
}
|
||||||
|
|
||||||
|
public synchronized void decreaseReference() {
|
||||||
|
this.reference--;
|
||||||
|
}
|
||||||
|
|
||||||
|
public synchronized boolean canClose() {
|
||||||
|
return reference == 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
public synchronized void close() {
|
||||||
|
if (this.canClose()) {
|
||||||
|
groupDataSource.destroy();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,55 @@
|
|||||||
|
package com.alibaba.datax.plugin.writer.oceanbasev10writer.ext;
|
||||||
|
|
||||||
|
import java.sql.Connection;
|
||||||
|
|
||||||
|
import com.alibaba.datax.common.util.Configuration;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DBUtil;
|
||||||
|
import com.alibaba.datax.plugin.rdbms.util.DataBaseType;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* wrap oceanbase java client
|
||||||
|
* @author oceanbase
|
||||||
|
*/
|
||||||
|
|
||||||
|
public class OCJConnHolder extends ConnHolder {
|
||||||
|
private ServerConnectInfo connectInfo;
|
||||||
|
private String dataSourceKey;
|
||||||
|
|
||||||
|
public OCJConnHolder (Configuration config, ServerConnectInfo connInfo) {
|
||||||
|
super(config);
|
||||||
|
this.connectInfo = connInfo;
|
||||||
|
this.dataSourceKey = OBDataSourceV10.genKey(connectInfo.getFullUserName(), connectInfo.databaseName);
|
||||||
|
OBDataSourceV10.init(config, connectInfo.getFullUserName(), connectInfo.password, connectInfo.databaseName);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Connection initConnection() {
|
||||||
|
conn = OBDataSourceV10.getConnection(dataSourceKey);
|
||||||
|
return conn;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Connection reconnect() {
|
||||||
|
DBUtil.closeDBResources(null, conn);
|
||||||
|
return initConnection();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Connection getConn() {
|
||||||
|
return conn;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public String getJdbcUrl() {
|
||||||
|
return connectInfo.jdbcUrl;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public String getUserName() {
|
||||||
|
return connectInfo.userName;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void destroy() {
|
||||||
|
OBDataSourceV10.destory(this.dataSourceKey);
|
||||||
|
}
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user