mirror of
https://github.com/alibaba/DataX.git
synced 2025-05-02 19:50:29 +08:00
Merge 65e1a8c11f into 0824b45c5e
This commit is contained in:
leehom
GitHub
commit
d2d53ff643
3
rdb2graph/README.md
Normal file
3
rdb2graph/README.md
Normal file
@ -0,0 +1,3 @@
|
||||
- 关系/图转换插件，该插件属于数据同构同步，模型一致，属性名称一致，优点是自动化高，不用多大干预，适合生成中间数据，
|
||||
特别是，关系数据库系统与图数据库系统不在同一网段，网络不稳定，同步尽量简单显得很重要
|
||||
- schema转换 4.3+
|
||||
368
rdb2graph/mysqlreader4graph/doc/mysqlreader.md
Normal file
368
rdb2graph/mysqlreader4graph/doc/mysqlreader.md
Normal file
@ -0,0 +1,368 @@
|
||||
|
||||
# MysqlReader 插件文档
|
||||
|
||||
|
||||
___
|
||||
|
||||
|
||||
|
||||
## 1 快速介绍
|
||||
|
||||
MysqlReader插件实现了从Mysql读取数据。在底层实现上，MysqlReader通过JDBC连接远程Mysql数据库，并执行相应的sql语句将数据从mysql库中SELECT出来。
|
||||
|
||||
**不同于其他关系型数据库，MysqlReader不支持FetchSize.**
|
||||
|
||||
## 2 实现原理
|
||||
|
||||
简而言之，MysqlReader通过JDBC连接器连接到远程的Mysql数据库，并根据用户配置的信息生成查询SELECT SQL语句，然后发送到远程Mysql数据库，并将该SQL执行返回结果使用DataX自定义的数据类型拼装为抽象的数据集，并传递给下游Writer处理。
|
||||
|
||||
对于用户配置Table、Column、Where的信息，MysqlReader将其拼接为SQL语句发送到Mysql数据库；对于用户配置querySql信息，MysqlReader直接将其发送到Mysql数据库。
|
||||
|
||||
|
||||
## 3 功能说明
|
||||
|
||||
### 3.1 配置样例
|
||||
|
||||
* 配置一个从Mysql数据库同步抽取数据到本地的作业:
|
||||
|
||||
```
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 3
|
||||
},
|
||||
"errorLimit": {
|
||||
"record": 0,
|
||||
"percentage": 0.02
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "mysqlreader",
|
||||
"parameter": {
|
||||
"username": "root",
|
||||
"password": "root",
|
||||
"column": [
|
||||
"id",
|
||||
"name"
|
||||
],
|
||||
"splitPk": "db_id",
|
||||
"connection": [
|
||||
{
|
||||
"table": [
|
||||
"table"
|
||||
],
|
||||
"jdbcUrl": [
|
||||
"jdbc:mysql://127.0.0.1:3306/database"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "streamwriter",
|
||||
"parameter": {
|
||||
"print":true
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
* 配置一个自定义SQL的数据库同步任务到本地内容的作业：
|
||||
|
||||
```
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel":1
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "mysqlreader",
|
||||
"parameter": {
|
||||
"username": "root",
|
||||
"password": "root",
|
||||
"connection": [
|
||||
{
|
||||
"querySql": [
|
||||
"select db_id,on_line_flag from db_info where db_id < 10;"
|
||||
],
|
||||
"jdbcUrl": [
|
||||
"jdbc:mysql://bad_ip:3306/database",
|
||||
"jdbc:mysql://127.0.0.1:bad_port/database",
|
||||
"jdbc:mysql://127.0.0.1:3306/database"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "streamwriter",
|
||||
"parameter": {
|
||||
"print": false,
|
||||
"encoding": "UTF-8"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### 3.2 参数说明
|
||||
|
||||
* **jdbcUrl**
|
||||
|
||||
* 描述：描述的是到对端数据库的JDBC连接信息，使用JSON的数组描述，并支持一个库填写多个连接地址。之所以使用JSON数组描述连接信息，是因为阿里集团内部支持多个IP探测，如果配置了多个，MysqlReader可以依次探测ip的可连接性，直到选择一个合法的IP。如果全部连接失败，MysqlReader报错。 注意，jdbcUrl必须包含在connection配置单元中。对于阿里集团外部使用情况，JSON数组填写一个JDBC连接即可。
|
||||
|
||||
jdbcUrl按照Mysql官方规范，并可以填写连接附件控制信息。具体请参看[Mysql官方文档](http://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html)。
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **username**
|
||||
|
||||
* 描述：数据源的用户名 <br />
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **password**
|
||||
|
||||
* 描述：数据源指定用户名的密码 <br />
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **table**
|
||||
|
||||
* 描述：所选取的需要同步的表。使用JSON的数组描述，因此支持多张表同时抽取。当配置为多张表时，用户自己需保证多张表是同一schema结构，MysqlReader不予检查表是否同一逻辑表。注意，table必须包含在connection配置单元中。<br />
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **column**
|
||||
|
||||
* 描述：所配置的表中需要同步的列名集合，使用JSON的数组描述字段信息。用户使用\*代表默认使用所有列配置，例如['\*']。
|
||||
|
||||
支持列裁剪，即列可以挑选部分列进行导出。
|
||||
|
||||
支持列换序，即列可以不按照表schema信息进行导出。
|
||||
|
||||
支持常量配置，用户需要按照Mysql SQL语法格式:
|
||||
["id", "\`table\`", "1", "'bazhen.csy'", "null", "to_char(a + 1)", "2.3" , "true"]
|
||||
id为普通列名，\`table\`为包含保留在的列名，1为整形数字常量，'bazhen.csy'为字符串常量，null为空指针，to_char(a + 1)为表达式，2.3为浮点数，true为布尔值。
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **splitPk**
|
||||
|
||||
* 描述：MysqlReader进行数据抽取时，如果指定splitPk，表示用户希望使用splitPk代表的字段进行数据分片，DataX因此会启动并发任务进行数据同步，这样可以大大提供数据同步的效能。
|
||||
|
||||
推荐splitPk用户使用表主键，因为表主键通常情况下比较均匀，因此切分出来的分片也不容易出现数据热点。
|
||||
|
||||
目前splitPk仅支持整形数据切分，`不支持浮点、字符串、日期等其他类型`。如果用户指定其他非支持类型，MysqlReader将报错！
|
||||
|
||||
如果splitPk不填写，包括不提供splitPk或者splitPk值为空，DataX视作使用单通道同步该表数据。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：空 <br />
|
||||
|
||||
* **where**
|
||||
|
||||
* 描述：筛选条件，MysqlReader根据指定的column、table、where条件拼接SQL，并根据这个SQL进行数据抽取。在实际业务场景中，往往会选择当天的数据进行同步，可以将where条件指定为gmt_create > $bizdate 。注意：不可以将where条件指定为limit 10，limit不是SQL的合法where子句。<br />
|
||||
|
||||
where条件可以有效地进行业务增量同步。如果不填写where语句，包括不提供where的key或者value，DataX均视作同步全量数据。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **querySql**
|
||||
|
||||
* 描述：在有些业务场景下，where这一配置项不足以描述所筛选的条件，用户可以通过该配置型来自定义筛选SQL。当用户配置了这一项之后，DataX系统就会忽略table，column这些配置型，直接使用这个配置项的内容对数据进行筛选，例如需要进行多表join后同步数据，使用select a,b from table_a join table_b on table_a.id = table_b.id <br />
|
||||
|
||||
`当用户配置querySql时，MysqlReader直接忽略table、column、where条件的配置`，querySql优先级大于table、column、where选项。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
|
||||
### 3.3 类型转换
|
||||
|
||||
目前MysqlReader支持大部分Mysql类型，但也存在部分个别类型没有支持的情况，请注意检查你的类型。
|
||||
|
||||
下面列出MysqlReader针对Mysql类型转换列表:
|
||||
|
||||
|
||||
| DataX 内部类型| Mysql 数据类型 |
|
||||
| -------- | ----- |
|
||||
| Long |int, tinyint, smallint, mediumint, int, bigint|
|
||||
| Double |float, double, decimal|
|
||||
| String |varchar, char, tinytext, text, mediumtext, longtext, year |
|
||||
| Date |date, datetime, timestamp, time |
|
||||
| Boolean |bit, bool |
|
||||
| Bytes |tinyblob, mediumblob, blob, longblob, varbinary |
|
||||
|
||||
|
||||
|
||||
请注意:
|
||||
|
||||
* `除上述罗列字段类型外，其他类型均不支持`。
|
||||
* `tinyint(1) DataX视作为整形`。
|
||||
* `year DataX视作为字符串类型`
|
||||
* `bit DataX属于未定义行为`。
|
||||
|
||||
## 4 性能报告
|
||||
|
||||
### 4.1 环境准备
|
||||
|
||||
#### 4.1.1 数据特征
|
||||
建表语句：
|
||||
|
||||
CREATE TABLE `tc_biz_vertical_test_0000` (
|
||||
`biz_order_id` bigint(20) NOT NULL COMMENT 'id',
|
||||
`key_value` varchar(4000) NOT NULL COMMENT 'Key-value的内容',
|
||||
`gmt_create` datetime NOT NULL COMMENT '创建时间',
|
||||
`gmt_modified` datetime NOT NULL COMMENT '修改时间',
|
||||
`attribute_cc` int(11) DEFAULT NULL COMMENT '防止并发修改的标志',
|
||||
`value_type` int(11) NOT NULL DEFAULT '0' COMMENT '类型',
|
||||
`buyer_id` bigint(20) DEFAULT NULL COMMENT 'buyerid',
|
||||
`seller_id` bigint(20) DEFAULT NULL COMMENT 'seller_id',
|
||||
PRIMARY KEY (`biz_order_id`,`value_type`),
|
||||
KEY `idx_biz_vertical_gmtmodified` (`gmt_modified`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=gbk COMMENT='tc_biz_vertical'
|
||||
|
||||
|
||||
单行记录类似于：
|
||||
|
||||
biz_order_id: 888888888
|
||||
key_value: ;orderIds:20148888888,2014888888813800;
|
||||
gmt_create: 2011-09-24 11:07:20
|
||||
gmt_modified: 2011-10-24 17:56:34
|
||||
attribute_cc: 1
|
||||
value_type: 3
|
||||
buyer_id: 8888888
|
||||
seller_id: 1
|
||||
|
||||
#### 4.1.2 机器参数
|
||||
|
||||
* 执行DataX的机器参数为:
|
||||
1. cpu: 24核 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
|
||||
2. mem: 48GB
|
||||
3. net: 千兆双网卡
|
||||
4. disc: DataX 数据不落磁盘，不统计此项
|
||||
|
||||
* Mysql数据库机器参数为:
|
||||
1. cpu: 32核 Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
|
||||
2. mem: 256GB
|
||||
3. net: 千兆双网卡
|
||||
4. disc: BTWL419303E2800RGN INTEL SSDSC2BB800G4 D2010370
|
||||
|
||||
#### 4.1.3 DataX jvm 参数
|
||||
|
||||
-Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError
|
||||
|
||||
|
||||
### 4.2 测试报告
|
||||
|
||||
#### 4.2.1 单表测试报告
|
||||
|
||||
|
||||
| 通道数| 是否按照主键切分| DataX速度(Rec/s)|DataX流量(MB/s)| DataX机器网卡进入流量(MB/s)|DataX机器运行负载|DB网卡流出流量(MB/s)|DB运行负载|
|
||||
|--------|--------| --------|--------|--------|--------|--------|--------|
|
||||
|1| 否 | 183185 | 18.11 | 29| 0.6 | 31| 0.6 |
|
||||
|1| 是 | 183185 | 18.11 | 29| 0.6 | 31| 0.6 |
|
||||
|4| 否 | 183185 | 18.11 | 29| 0.6 | 31| 0.6 |
|
||||
|4| 是 | 329733 | 32.60 | 58| 0.8 | 60| 0.76 |
|
||||
|8| 否 | 183185 | 18.11 | 29| 0.6 | 31| 0.6 |
|
||||
|8| 是 | 549556 | 54.33 | 115| 1.46 | 120| 0.78 |
|
||||
|
||||
说明：
|
||||
|
||||
1. 这里的单表，主键类型为 bigint(20),范围为：190247559466810-570722244711460，从主键范围划分看，数据分布均匀。
|
||||
2. 对单表如果没有安装主键切分，那么配置通道个数不会提升速度，效果与1个通道一样。
|
||||
|
||||
|
||||
#### 4.2.2 分表测试报告(2个分库，每个分库16张分表，共计32张分表)
|
||||
|
||||
|
||||
| 通道数| DataX速度(Rec/s)|DataX流量(MB/s)| DataX机器网卡进入流量(MB/s)|DataX机器运行负载|DB网卡流出流量(MB/s)|DB运行负载|
|
||||
|--------| --------|--------|--------|--------|--------|--------|
|
||||
|1| 202241 | 20.06 | 31.5| 1.0 | 32 | 1.1 |
|
||||
|4| 726358 | 72.04 | 123.9 | 3.1 | 132 | 3.6 |
|
||||
|8|1074405 | 106.56| 197 | 5.5 | 205| 5.1|
|
||||
|16| 1227892 | 121.79 | 229.2 | 8.1 | 233 | 7.3 |
|
||||
|
||||
## 5 约束限制
|
||||
|
||||
### 5.1 主备同步数据恢复问题
|
||||
|
||||
主备同步问题指Mysql使用主从灾备，备库从主库不间断通过binlog恢复数据。由于主备数据同步存在一定的时间差，特别在于某些特定情况，例如网络延迟等问题，导致备库同步恢复的数据与主库有较大差别，导致从备库同步的数据不是一份当前时间的完整镜像。
|
||||
|
||||
针对这个问题，我们提供了preSql功能，该功能待补充。
|
||||
|
||||
### 5.2 一致性约束
|
||||
|
||||
Mysql在数据存储划分中属于RDBMS系统，对外可以提供强一致性数据查询接口。例如当一次同步任务启动运行过程中，当该库存在其他数据写入方写入数据时，MysqlReader完全不会获取到写入更新数据，这是由于数据库本身的快照特性决定的。关于数据库快照特性，请参看[MVCC Wikipedia](https://en.wikipedia.org/wiki/Multiversion_concurrency_control)
|
||||
|
||||
上述是在MysqlReader单线程模型下数据同步一致性的特性，由于MysqlReader可以根据用户配置信息使用了并发数据抽取，因此不能严格保证数据一致性：当MysqlReader根据splitPk进行数据切分后，会先后启动多个并发任务完成数据同步。由于多个并发任务相互之间不属于同一个读事务，同时多个并发任务存在时间间隔。因此这份数据并不是`完整的`、`一致的`数据快照信息。
|
||||
|
||||
针对多线程的一致性快照需求，在技术上目前无法实现，只能从工程角度解决，工程化的方式存在取舍，我们提供几个解决思路给用户，用户可以自行选择：
|
||||
|
||||
1. 使用单线程同步，即不再进行数据切片。缺点是速度比较慢，但是能够很好保证一致性。
|
||||
|
||||
2. 关闭其他数据写入方，保证当前数据为静态数据，例如，锁表、关闭备库同步等等。缺点是可能影响在线业务。
|
||||
|
||||
### 5.3 数据库编码问题
|
||||
|
||||
Mysql本身的编码设置非常灵活，包括指定编码到库、表、字段级别，甚至可以均不同编码。优先级从高到低为字段、表、库、实例。我们不推荐数据库用户设置如此混乱的编码，最好在库级别就统一到UTF-8。
|
||||
|
||||
MysqlReader底层使用JDBC进行数据抽取，JDBC天然适配各类编码，并在底层进行了编码转换。因此MysqlReader不需用户指定编码，可以自动获取编码并转码。
|
||||
|
||||
对于Mysql底层写入编码和其设定的编码不一致的混乱情况，MysqlReader对此无法识别，对此也无法提供解决方案，对于这类情况，`导出有可能为乱码`。
|
||||
|
||||
### 5.4 增量数据同步
|
||||
|
||||
MysqlReader使用JDBC SELECT语句完成数据抽取工作，因此可以使用SELECT...WHERE...进行增量数据抽取，方式有多种：
|
||||
|
||||
* 数据库在线应用写入数据库时，填充modify字段为更改时间戳，包括新增、更新、删除(逻辑删)。对于这类应用，MysqlReader只需要WHERE条件跟上一同步阶段时间戳即可。
|
||||
* 对于新增流水型数据，MysqlReader可以WHERE条件后跟上一阶段最大自增ID即可。
|
||||
|
||||
对于业务上无字段区分新增、修改数据情况，MysqlReader也无法进行增量数据同步，只能同步全量数据。
|
||||
|
||||
### 5.5 Sql安全性
|
||||
|
||||
MysqlReader提供querySql语句交给用户自己实现SELECT抽取语句，MysqlReader本身对querySql不做任何安全性校验。这块交由DataX用户方自己保证。
|
||||
|
||||
## 6 FAQ
|
||||
|
||||
***
|
||||
|
||||
**Q: MysqlReader同步报错，报错信息为XXX**
|
||||
|
||||
A: 网络或者权限问题，请使用mysql命令行测试：
|
||||
|
||||
mysql -u<username> -p<password> -h<ip> -D<database> -e "select * from <表名>"
|
||||
|
||||
如果上述命令也报错，那可以证实是环境问题，请联系你的DBA。
|
||||
|
||||
|
||||
84
rdb2graph/mysqlreader4graph/pom.xml
Normal file
84
rdb2graph/mysqlreader4graph/pom.xml
Normal file
@ -0,0 +1,84 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-parent</artifactId>
|
||||
<version>${revision}</version>
|
||||
</parent>
|
||||
<artifactId>mysqlreader4graph</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-common</artifactId>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<artifactId>slf4j-log4j12</artifactId>
|
||||
<groupId>org.slf4j</groupId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-api</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>ch.qos.logback</groupId>
|
||||
<artifactId>logback-classic</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>mysql</groupId>
|
||||
<artifactId>mysql-connector-java</artifactId>
|
||||
<version>${mysql.driver.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-rdbms-util</artifactId>
|
||||
<version>${revision}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-common</artifactId>
|
||||
<version>${revision}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<!-- compiler plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<configuration>
|
||||
<source>${java.version}</source>
|
||||
<target>${java.version}</target>
|
||||
<encoding>${project-sourceEncoding}</encoding>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<!-- assembly plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-assembly-plugin</artifactId>
|
||||
<configuration>
|
||||
<descriptors>
|
||||
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||
</descriptors>
|
||||
<finalName>datax</finalName>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>dwzip</id>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>single</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
35
rdb2graph/mysqlreader4graph/src/main/assembly/package.xml
Normal file
35
rdb2graph/mysqlreader4graph/src/main/assembly/package.xml
Normal file
@ -0,0 +1,35 @@
|
||||
<assembly
|
||||
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||
<id></id>
|
||||
<formats>
|
||||
<format>dir</format>
|
||||
</formats>
|
||||
<includeBaseDirectory>false</includeBaseDirectory>
|
||||
<fileSets>
|
||||
<fileSet>
|
||||
<directory>src/main/resources</directory>
|
||||
<includes>
|
||||
<include>plugin.json</include>
|
||||
<include>plugin_job_template.json</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/reader/mysqlreader4graph</outputDirectory>
|
||||
</fileSet>
|
||||
<fileSet>
|
||||
<directory>target/</directory>
|
||||
<includes>
|
||||
<include>mysqlreader4graph-1.0.0-SNAPSHOT.jar</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/reader/mysqlreader4graph</outputDirectory>
|
||||
</fileSet>
|
||||
</fileSets>
|
||||
|
||||
<dependencySets>
|
||||
<dependencySet>
|
||||
<useProjectArtifact>false</useProjectArtifact>
|
||||
<outputDirectory>plugin/reader/mysqlreader4graph/libs</outputDirectory>
|
||||
<scope>runtime</scope>
|
||||
</dependencySet>
|
||||
</dependencySets>
|
||||
</assembly>
|
||||
@ -0,0 +1,110 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.reader.mysqlreader;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.alibaba.datax.common.plugin.RecordSender;
|
||||
import com.alibaba.datax.common.spi.Reader;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.CommonRdbms2GraphReader;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.Constant;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.util.DataBaseType;
|
||||
|
||||
/**
|
||||
* @类名: MysqlReader4Graph
|
||||
* @说明: mysql读入
|
||||
* 1.表数据同步，datax table模式，schema表写入datax的配置
|
||||
2.表关系同步，按表连接图分组，一对关系一组，表连接关系生成querySql，写入datax的querySql模式配置
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月30日 上午12:38:34
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class MysqlReader4Graph extends Reader {
|
||||
|
||||
private static final DataBaseType DATABASE_TYPE = DataBaseType.MySql;
|
||||
|
||||
public static class Job extends Reader.Job {
|
||||
private static final Logger LOG = LoggerFactory
|
||||
.getLogger(Job.class);
|
||||
|
||||
private Configuration originalConfig = null;
|
||||
private CommonRdbms2GraphReader.Job commonRdbmsReaderJob;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
this.originalConfig = super.getPluginJobConf();
|
||||
|
||||
Integer userConfigedFetchSize = this.originalConfig.getInt(Constant.FETCH_SIZE);
|
||||
if (userConfigedFetchSize != null) {
|
||||
LOG.warn("对 mysqlreader 不需要配置 fetchSize, mysqlreader 将会忽略这项配置. 如果您不想再看到此警告,请去除fetchSize 配置.");
|
||||
}
|
||||
|
||||
this.originalConfig.set(Constant.FETCH_SIZE, Integer.MIN_VALUE);
|
||||
|
||||
this.commonRdbmsReaderJob = new CommonRdbms2GraphReader.Job(DATABASE_TYPE);
|
||||
this.commonRdbmsReaderJob.init(this.originalConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void preCheck(){
|
||||
init();
|
||||
this.commonRdbmsReaderJob.preCheck(this.originalConfig,DATABASE_TYPE);
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Configuration> split(int adviceNumber) {
|
||||
return this.commonRdbmsReaderJob.split(this.originalConfig, adviceNumber);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
this.commonRdbmsReaderJob.post(this.originalConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
this.commonRdbmsReaderJob.destroy(this.originalConfig);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
public static class Task extends Reader.Task {
|
||||
|
||||
private Configuration readerSliceConfig;
|
||||
private CommonRdbms2GraphReader.Task commonRdbmsReaderTask;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
this.readerSliceConfig = super.getPluginJobConf();
|
||||
this.commonRdbmsReaderTask = new CommonRdbms2GraphReader.Task(DATABASE_TYPE,super.getTaskGroupId(), super.getTaskId());
|
||||
this.commonRdbmsReaderTask.init(this.readerSliceConfig);
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public void startRead(RecordSender recordSender) {
|
||||
int fetchSize = this.readerSliceConfig.getInt(Constant.FETCH_SIZE);
|
||||
|
||||
this.commonRdbmsReaderTask.startRead(this.readerSliceConfig, recordSender,
|
||||
super.getTaskPluginCollector(), fetchSize);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
this.commonRdbmsReaderTask.post(this.readerSliceConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
this.commonRdbmsReaderTask.destroy(this.readerSliceConfig);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.reader.mysqlreader;
|
||||
|
||||
import com.alibaba.datax.common.spi.ErrorCode;
|
||||
|
||||
public enum MysqlReaderErrorCode implements ErrorCode {
|
||||
;
|
||||
|
||||
private final String code;
|
||||
private final String description;
|
||||
|
||||
private MysqlReaderErrorCode(String code, String description) {
|
||||
this.code = code;
|
||||
this.description = description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getCode() {
|
||||
return this.code;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getDescription() {
|
||||
return this.description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return String.format("Code:[%s], Description:[%s]. ", this.code,
|
||||
this.description);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "mysqlreader4graph",
|
||||
"class": "com.leehom.arch.datax.plugin.rdb2graph.reader.mysqlreader.MysqlReader4Graph",
|
||||
"description": "",
|
||||
"developer": "leehom"
|
||||
}
|
||||
@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "mysqlreader4graph",
|
||||
"parameter": {
|
||||
"username": "",
|
||||
"password": "",
|
||||
"phase": "",
|
||||
"schemaUri": "",
|
||||
"column": [],
|
||||
"connection": [
|
||||
{
|
||||
"jdbcUrl": [],
|
||||
"table": []
|
||||
}
|
||||
],
|
||||
"where": ""
|
||||
}
|
||||
}
|
||||
4
rdb2graph/neo4jwriter/README.md
Normal file
4
rdb2graph/neo4jwriter/README.md
Normal file
@ -0,0 +1,4 @@
|
||||
|
||||
|
||||
|
||||
|
||||
84
rdb2graph/neo4jwriter/pom.xml
Normal file
84
rdb2graph/neo4jwriter/pom.xml
Normal file
@ -0,0 +1,84 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<parent>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-parent</artifactId>
|
||||
<version>${revision}</version>
|
||||
</parent>
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<artifactId>neo4jwriter</artifactId>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-common</artifactId>
|
||||
<version>${datax.version}</version>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<artifactId>slf4j-log4j12</artifactId>
|
||||
<groupId>org.slf4j</groupId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-api</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>ch.qos.logback</groupId>
|
||||
<artifactId>logback-classic</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.alibaba</groupId>
|
||||
<artifactId>fastjson</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-scanner</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-common</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
<build>
|
||||
<plugins>
|
||||
<!-- compiler plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<configuration>
|
||||
<source>${java.version}</source>
|
||||
<target>${java.version}</target>
|
||||
<encoding>${project-sourceEncoding}</encoding>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<!-- assembly plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-assembly-plugin</artifactId>
|
||||
<configuration>
|
||||
<descriptors>
|
||||
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||
</descriptors>
|
||||
<finalName>datax</finalName>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>dwzip</id>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>single</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
34
rdb2graph/neo4jwriter/src/main/assembly/package.xml
Normal file
34
rdb2graph/neo4jwriter/src/main/assembly/package.xml
Normal file
@ -0,0 +1,34 @@
|
||||
<assembly
|
||||
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||
<id></id>
|
||||
<formats>
|
||||
<format>dir</format>
|
||||
</formats>
|
||||
<includeBaseDirectory>false</includeBaseDirectory>
|
||||
<fileSets>
|
||||
<fileSet>
|
||||
<directory>src/main/resources</directory>
|
||||
<includes>
|
||||
<include>plugin.json</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/writer/neo4jwriter</outputDirectory>
|
||||
</fileSet>
|
||||
<fileSet>
|
||||
<directory>target/</directory>
|
||||
<includes>
|
||||
<include>neo4jwriter-1.0.0-SNAPSHOT.jar</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/writer/neo4jwriter</outputDirectory>
|
||||
</fileSet>
|
||||
</fileSets>
|
||||
|
||||
<dependencySets>
|
||||
<dependencySet>
|
||||
<useProjectArtifact>false</useProjectArtifact>
|
||||
<outputDirectory>plugin/writer/neo4jwriter/libs</outputDirectory>
|
||||
<scope>runtime</scope>
|
||||
</dependencySet>
|
||||
</dependencySets>
|
||||
</assembly>
|
||||
@ -0,0 +1,121 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.writer.neo4jwriter;
|
||||
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* @类名: Key
|
||||
* @说明: 配置项
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月27日 下午2:49:13
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public final class Key {
|
||||
|
||||
public static String database(Configuration conf) {
|
||||
return conf.getString("database", "");
|
||||
}
|
||||
|
||||
public static String schemaUri(Configuration conf) {
|
||||
return conf.getString("schemaUri", "");
|
||||
}
|
||||
|
||||
public static String uri(Configuration conf) {
|
||||
return conf.getString("uri", "");
|
||||
}
|
||||
|
||||
public static String userName(Configuration conf) {
|
||||
return conf.getString("username", "");
|
||||
}
|
||||
|
||||
public static String password(Configuration conf) {
|
||||
return conf.getString("password", "");
|
||||
}
|
||||
|
||||
public static int batchSize(Configuration conf) {
|
||||
return conf.getInt("batchSize", 1000);
|
||||
}
|
||||
|
||||
public static int getTrySize(Configuration conf) {
|
||||
return conf.getInt("trySize", 30);
|
||||
}
|
||||
|
||||
public static int getTimeout(Configuration conf) {
|
||||
return conf.getInt("timeout", 600000);
|
||||
}
|
||||
|
||||
public static boolean isCleanup(Configuration conf) {
|
||||
return conf.getBool("cleanup", false);
|
||||
}
|
||||
|
||||
public static boolean isDiscovery(Configuration conf) {
|
||||
return conf.getBool("discovery", false);
|
||||
}
|
||||
|
||||
public static boolean isCompression(Configuration conf) {
|
||||
return conf.getBool("compression", true);
|
||||
}
|
||||
|
||||
public static boolean isMultiThread(Configuration conf) {
|
||||
return conf.getBool("multiThread", true);
|
||||
}
|
||||
|
||||
public static String getIndexName(Configuration conf) {
|
||||
return conf.getNecessaryValue("index", Neo4jWriterErrorCode.BAD_CONFIG_VALUE);
|
||||
}
|
||||
|
||||
public static String getTypeName(Configuration conf) {
|
||||
String indexType = conf.getString("indexType");
|
||||
if(StringUtils.isBlank(indexType)){
|
||||
indexType = conf.getString("type", getIndexName(conf));
|
||||
}
|
||||
return indexType;
|
||||
}
|
||||
|
||||
|
||||
public static boolean isIgnoreWriteError(Configuration conf) {
|
||||
return conf.getBool("ignoreWriteError", false);
|
||||
}
|
||||
|
||||
public static boolean isIgnoreParseError(Configuration conf) {
|
||||
return conf.getBool("ignoreParseError", true);
|
||||
}
|
||||
|
||||
|
||||
public static boolean isHighSpeedMode(Configuration conf) {
|
||||
if ("highspeed".equals(conf.getString("mode", ""))) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
public static String getAlias(Configuration conf) {
|
||||
return conf.getString("alias", "");
|
||||
}
|
||||
|
||||
public static boolean isNeedCleanAlias(Configuration conf) {
|
||||
String mode = conf.getString("aliasMode", "append");
|
||||
if ("exclusive".equals(mode)) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
public static Map<String, Object> getSettings(Configuration conf) {
|
||||
return conf.getMap("settings", new HashMap<String, Object>());
|
||||
}
|
||||
|
||||
public static String getSplitter(Configuration conf) {
|
||||
return conf.getString("splitter", "-,-");
|
||||
}
|
||||
|
||||
public static boolean getDynamic(Configuration conf) {
|
||||
return conf.getBool("dynamic", false);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,424 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.writer.neo4jwriter;
|
||||
|
||||
import java.io.InputStream;
|
||||
import java.lang.reflect.InvocationTargetException;
|
||||
import java.sql.JDBCType;
|
||||
import java.text.MessageFormat;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Date;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.apache.commons.lang3.reflect.MethodUtils;
|
||||
import org.neo4j.driver.AuthTokens;
|
||||
import org.neo4j.driver.Driver;
|
||||
import org.neo4j.driver.GraphDatabase;
|
||||
import org.neo4j.driver.Query;
|
||||
import org.neo4j.driver.exceptions.Neo4jException;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.alibaba.datax.common.element.Column;
|
||||
import com.alibaba.datax.common.element.Column.Type;
|
||||
import com.alibaba.datax.common.element.Record;
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.plugin.RecordReceiver;
|
||||
import com.alibaba.datax.common.spi.Writer;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.ByteAndStreamUtils;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.ResourceLoaderUtil;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.serializer.Serializer;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.config.ScannerSerializerConfig;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds.Neo4jDao;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds.Neo4jQueryPattern;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds.ParamsUtils;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchema;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKField;
|
||||
|
||||
/**
|
||||
* @类名: Neo4jWriter
|
||||
* @说明: neo4j写入，支持两个阶段
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月21日 下午11:14:32
|
||||
* 修改记录：
|
||||
*
|
||||
* TODO
|
||||
* 1. 支持写入模式，@WriteMode replace/update
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class Neo4jWriter extends Writer {
|
||||
|
||||
public static class Job extends Writer.Job {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(Job.class);
|
||||
|
||||
private Configuration conf = null;
|
||||
|
||||
// 初始化环境和配置，为客户端准备环境
|
||||
@Override
|
||||
public void init() {
|
||||
this.conf = super.getPluginJobConf();
|
||||
}
|
||||
|
||||
// 准备
|
||||
// 这里可做schema转换
|
||||
@Override
|
||||
public void prepare() {
|
||||
//
|
||||
}
|
||||
|
||||
// 目标库只有一个，分片相当于并行写入，克隆即可
|
||||
@Override
|
||||
public List<Configuration> split(int mandatoryNumber) {
|
||||
List<Configuration> configurations = new ArrayList<Configuration>(mandatoryNumber);
|
||||
for (int i = 0; i < mandatoryNumber; i++) {
|
||||
configurations.add(conf);
|
||||
}
|
||||
return configurations;
|
||||
}
|
||||
|
||||
// 后处理
|
||||
@Override
|
||||
public void post() {
|
||||
// nothing
|
||||
}
|
||||
|
||||
// 释放资源
|
||||
@Override
|
||||
public void destroy() {
|
||||
// nothing
|
||||
}
|
||||
}
|
||||
|
||||
public static class Task extends Writer.Task {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(Job.class);
|
||||
|
||||
private Configuration conf;
|
||||
private Neo4jDao client = null;
|
||||
private int batchSize; // 支持批量
|
||||
private DbSchema rdbSchema; //
|
||||
private WriteMode writeMode; // 写入模式，
|
||||
private Serializer ser; //
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
//
|
||||
this.conf = super.getPluginJobConf();
|
||||
// 初始化 Neo4jDao
|
||||
String uri = Key.uri(conf);
|
||||
String un = Key.userName(conf);
|
||||
String pw = Key.password(conf);
|
||||
String db = Key.database(conf);
|
||||
Driver driver = GraphDatabase.driver(uri, AuthTokens.basic(un, pw));
|
||||
client = new Neo4jDao();
|
||||
client.setDriver(driver);
|
||||
client.setDatabase(db);
|
||||
// 初始化数据库schema序列化器
|
||||
ser = ScannerSerializerConfig.rdbSchemaXmlSerializer();
|
||||
// 写入批量
|
||||
batchSize = Key.batchSize(conf);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void prepare() {
|
||||
init(); // 作业容器并没有调用init方法
|
||||
// 载入关系数据库schema
|
||||
String schemaUri = Key.schemaUri(conf);
|
||||
InputStream is;
|
||||
try {
|
||||
is = ResourceLoaderUtil.getResourceStream(schemaUri);
|
||||
byte[] bytes = ByteAndStreamUtils.StreamToBytes(is);
|
||||
rdbSchema = (DbSchema)ser.Unmarshal(bytes);
|
||||
} catch (Exception e) {
|
||||
DataXException.asDataXException(Neo4jWriterErrorCode.ERROR_LOAD_RDBSCHEMA, e);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// 写入, 两个场景
|
||||
// 1. 表记录 TableRecord 2. 关系记录 RelRecord
|
||||
@Override
|
||||
public void startWrite(RecordReceiver recordReceiver) {
|
||||
List<Record> writerBuffer = new ArrayList<Record>(this.batchSize);
|
||||
Record record = null;
|
||||
// 交换器(Exchanger)接收到TerminateRecord返回null
|
||||
while ((record = recordReceiver.getFromReader()) != null) {
|
||||
writerBuffer.add(record);
|
||||
if (writerBuffer.size() >= this.batchSize) {
|
||||
this.doWrite(writerBuffer);
|
||||
writerBuffer.clear();
|
||||
}
|
||||
}
|
||||
if (!writerBuffer.isEmpty()) {
|
||||
this.doWrite(writerBuffer);
|
||||
writerBuffer.clear();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// 写入分流 节点 / 关系
|
||||
private void doWrite(List<Record> writerBuffer) {
|
||||
Record record = writerBuffer.get(0);
|
||||
try {
|
||||
if ("TableRecord".equals(record.getClass().getSimpleName())) {
|
||||
doBatchWriteNode(writerBuffer);
|
||||
return;
|
||||
}
|
||||
if ("RelRecord".equals(record.getClass().getSimpleName())) {
|
||||
doBatchWriteRel(writerBuffer);
|
||||
return;
|
||||
}
|
||||
// 系统sleep
|
||||
try {
|
||||
Thread.sleep(300);
|
||||
} catch (InterruptedException e) {
|
||||
|
||||
}
|
||||
} catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
|
||||
// 理论上不会发送，只记录
|
||||
log.error(e.getMessage());
|
||||
|
||||
} catch (DataXException e) {
|
||||
// 字段类型不合法，记录，其他抛出
|
||||
log.error(e.getMessage());
|
||||
|
||||
} catch (Neo4jException e) {
|
||||
// neo4j异常
|
||||
throw DataXException.asDataXException(Neo4jWriterErrorCode.WRONG_NEO4j_CLIENT,
|
||||
Neo4jWriterErrorCode.WRONG_NEO4j_CLIENT.getDescription(), e);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// 批量写入
|
||||
private void doBatchWriteNode(final List<Record> writerBuffer) throws NoSuchMethodException, IllegalAccessException, InvocationTargetException {
|
||||
List<Query> queries = new ArrayList<>();
|
||||
for (Record record : writerBuffer) {
|
||||
String tn = (String) MethodUtils.invokeMethod(record, "getTable");
|
||||
TableMetadata tbmd = rdbSchema.findTable(tn);
|
||||
if (tbmd.isLinkTable())
|
||||
continue;
|
||||
// 节点属性
|
||||
Map<String, Object> props = fillProperties(tbmd, record);
|
||||
Query q = calcNodeWriteQuery(tbmd, props);
|
||||
queries.add(q);
|
||||
}
|
||||
client.reTryRunInTransaction(queries, 5);
|
||||
|
||||
}
|
||||
|
||||
private void doBatchWriteRel(final List<Record> writerBuffer)
|
||||
throws NoSuchMethodException, IllegalAccessException, InvocationTargetException {
|
||||
List<Query> queries = new ArrayList<>();
|
||||
for (Record record : writerBuffer) {
|
||||
// 表名称
|
||||
String tn;
|
||||
String fk;
|
||||
|
||||
tn = (String) MethodUtils.invokeMethod(record, "getFromTable");
|
||||
fk = (String) MethodUtils.invokeMethod(record, "getFk");
|
||||
TableMetadata tbmd = rdbSchema.findTable(tn);
|
||||
// 关系起始表，外键
|
||||
TableMetadata from;
|
||||
TableMetadata to;
|
||||
String fkName;
|
||||
Map<String, Object> props;
|
||||
// 连接表
|
||||
if (tbmd.isLinkTable()) {
|
||||
// 作为起点的外键，连接表有且仅有2外键，其中一个作为起点，另一个为关系终点
|
||||
FKConstraintMetadata fromFkmd = tbmd.getLinkFrom();
|
||||
from = fromFkmd.getRefTable();
|
||||
//
|
||||
FKConstraintMetadata toFkmd = tbmd.getFks().get(0).getFkName().equals(fromFkmd.getFkName())
|
||||
? tbmd.getFks().get(1)
|
||||
: tbmd.getFks().get(0);
|
||||
to = toFkmd.getRefTable();
|
||||
fkName = tbmd.getName();
|
||||
props = fillProperties(tbmd, record);
|
||||
Query q = calcLinkRelWriteQuery(from, to, tbmd, fromFkmd, toFkmd, props);
|
||||
queries.add(q);
|
||||
} else {
|
||||
from = tbmd;
|
||||
FKConstraintMetadata fkmd = from.findFk(fk);
|
||||
to = fkmd.getRefTable();
|
||||
fkName = fkmd.getFkName();
|
||||
props = new HashMap<String, Object>();
|
||||
Query q = calcRelWriteQuery(from, to, record, fkName, props);
|
||||
queries.add(q);
|
||||
}
|
||||
// 构建查询
|
||||
|
||||
}
|
||||
client.reTryRunInTransaction(queries, 5);
|
||||
}
|
||||
|
||||
// 构建写入 query, 写入模式
|
||||
// @WriteMode
|
||||
private Query calcNodeWriteQuery(TableMetadata tbmd, Map<String, Object> props) {
|
||||
// insert
|
||||
String propsStr = ParamsUtils.params2String(props);
|
||||
String cql = MessageFormat.format(Neo4jQueryPattern.CREATE_NODE, tbmd.getName(), propsStr);
|
||||
return new Query(cql);
|
||||
}
|
||||
|
||||
private Query calcRelWriteQuery(TableMetadata from, TableMetadata to, Record record, String fkName, Map<String, Object> props) {
|
||||
// 连接属性
|
||||
String propsStr = ParamsUtils.params2String(props);
|
||||
// 节点过滤条件，使用主键
|
||||
String nodeWherePattern = "{0}.{1} = {2}";
|
||||
List<String> nodeWhereItems = new ArrayList<>();
|
||||
//
|
||||
List<FieldMetadata> pkfs = from.getPk().getFields();
|
||||
FKConstraintMetadata fk = from.findFk(fkName);
|
||||
//
|
||||
List<FKField> fkfs = fk.getFkFields();
|
||||
// from节点
|
||||
int i = 0;
|
||||
for(FieldMetadata f : pkfs) {
|
||||
Column col = record.getColumn(i);
|
||||
String item;
|
||||
if(col.getType()==Type.INT || col.getType()==Type.LONG || col.getType()==Type.DOUBLE) {
|
||||
item = MessageFormat.format(nodeWherePattern, "a", f.getName(), col.asLong().toString());
|
||||
} else { // 其他，字符增加引号
|
||||
item = MessageFormat.format(nodeWherePattern, "a", f.getName(), "'"+col.asString()+"'");
|
||||
}
|
||||
nodeWhereItems.add(item);
|
||||
i++;
|
||||
}
|
||||
// to节点
|
||||
for(FKField fkf : fkfs) {
|
||||
Column col = record.getColumn(i);
|
||||
String item;
|
||||
if(col.getType()==Type.INT || col.getType()==Type.LONG || col.getType()==Type.DOUBLE) {
|
||||
item = MessageFormat.format(nodeWherePattern, "b", fkf.getRefField().getName(), col.asLong().toString());
|
||||
} else { // 其他，字符增加引号
|
||||
item = MessageFormat.format(nodeWherePattern, "b", fkf.getRefField().getName(), "'"+col.asString()+"'");
|
||||
}
|
||||
nodeWhereItems.add(item);
|
||||
i++;
|
||||
|
||||
}
|
||||
String nodeWhere = Utils.relWhere("", nodeWhereItems, " and ");
|
||||
//
|
||||
String cql = MessageFormat.format(Neo4jQueryPattern.CREATE_REL,
|
||||
from.getName(), to.getName(), nodeWhere.toString(), fkName, propsStr);
|
||||
return new Query(cql);
|
||||
}
|
||||
|
||||
// 连接表关系
|
||||
private Query calcLinkRelWriteQuery(TableMetadata from, TableMetadata to, TableMetadata link,
|
||||
FKConstraintMetadata fromFkmd, FKConstraintMetadata toFkmd,
|
||||
Map<String, Object> props) {
|
||||
// 连接属性
|
||||
String propsStr = ParamsUtils.params2String(props);
|
||||
// 节点过滤条件，使用主键
|
||||
String nodeWherePattern = "{0}.{1} = {2}";
|
||||
List<String> nodeWhereItems = new ArrayList<>();
|
||||
//
|
||||
List<FKField> fromFkFs = fromFkmd.getFkFields();
|
||||
List<FKField> toFkFs = toFkmd.getFkFields();
|
||||
// from节点
|
||||
for(FKField fkf : fromFkFs) {
|
||||
String item;
|
||||
// from<-link
|
||||
// from表对应link外键名称
|
||||
String fn = fkf.getRefField().getName();
|
||||
String ln = fkf.getField().getName();
|
||||
Object v = props.get(ln);
|
||||
if(v instanceof Long || v instanceof Double) {
|
||||
item = MessageFormat.format(nodeWherePattern, "a", fn, v.toString());
|
||||
} else { // 其他，字符增加引号,
|
||||
item = MessageFormat.format(nodeWherePattern, "a", fn, "'"+v.toString()+"'");
|
||||
}
|
||||
nodeWhereItems.add(item);
|
||||
|
||||
}
|
||||
// to节点
|
||||
for(FKField fkf : toFkFs) {
|
||||
String item;
|
||||
// link->to
|
||||
// to表对应link外键名称
|
||||
String fn = fkf.getRefField().getName();
|
||||
String ln = fkf.getField().getName();
|
||||
Object v = props.get(ln);
|
||||
if(v instanceof Long || v instanceof Double) {
|
||||
item = MessageFormat.format(nodeWherePattern, "b", fn, v.toString());
|
||||
} else { // 其他，字符增加引号,
|
||||
item = MessageFormat.format(nodeWherePattern, "b", fn, "'"+v.toString()+"'");
|
||||
}
|
||||
nodeWhereItems.add(item);
|
||||
}
|
||||
String nodeWhere = Utils.relWhere("", nodeWhereItems, " and ");
|
||||
// from->to
|
||||
String cql = MessageFormat.format(Neo4jQueryPattern.CREATE_REL,
|
||||
from.getName(), to.getName(), nodeWhere.toString(), link.getName(), propsStr);
|
||||
return new Query(cql);
|
||||
}
|
||||
|
||||
// 节点或关系属性，类型转换
|
||||
// 本地缓存，查找
|
||||
protected Map<String, Object> fillProperties(TableMetadata tbmd, Record record) {
|
||||
Map<String, Object> props = new HashMap<>();
|
||||
int i = 0;
|
||||
for(FieldMetadata fmd : tbmd.getFields()) {
|
||||
String columnName = fmd.getName();
|
||||
JDBCType jdbcType = fmd.getType();
|
||||
Column column = record.getColumn(i);
|
||||
/* BAD, NULL, INT, LONG, DOUBLE, STRING, BOOL, DATE, BYTES */
|
||||
switch (column.getType()) {
|
||||
case INT:
|
||||
case LONG:
|
||||
props.put(columnName, column.asLong());
|
||||
break;
|
||||
case DOUBLE:
|
||||
props.put(columnName, column.asDouble());
|
||||
break;
|
||||
case STRING:
|
||||
// 转义
|
||||
String col = Utils.strFieldEscape(column.asString());
|
||||
props.put(columnName, col);
|
||||
break;
|
||||
case BOOL:
|
||||
props.put(fmd.getName(), column.asBoolean());
|
||||
break;
|
||||
case DATE:
|
||||
Date date = column.asDate();
|
||||
// LocalDateTime ldt = Utils.dateToLocalDateTime(date);
|
||||
props.put(fmd.getName(), date);
|
||||
break;
|
||||
case BYTES:
|
||||
log.warn(String.format("neo4j不支持二进制属性类型, 字段名:[%s], 字段类型:[%s]. ",
|
||||
fmd.getName(),
|
||||
jdbcType.getName()));
|
||||
break;
|
||||
default: // 其他，不支持类型
|
||||
throw DataXException.asDataXException(Neo4jWriterErrorCode.UNSUPPORTED_TYPE,
|
||||
String.format("neo4j不支持属性类型, 字段名:[%s], 字段类型:[%s]. ",
|
||||
fmd.getName(),
|
||||
jdbcType.getName()));
|
||||
} // end switch
|
||||
i++; // 下一个字段
|
||||
} // end for tbmd
|
||||
return props;
|
||||
}
|
||||
|
||||
// 后处理
|
||||
@Override
|
||||
public void post() {
|
||||
}
|
||||
|
||||
// 释放
|
||||
@Override
|
||||
public void destroy() {
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,36 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.writer.neo4jwriter;
|
||||
|
||||
import com.alibaba.datax.common.spi.ErrorCode;
|
||||
|
||||
public enum Neo4jWriterErrorCode implements ErrorCode {
|
||||
BAD_CONFIG_VALUE("Neo4jWriter-00", "配置的值不合法."),
|
||||
ERROR_LOAD_RDBSCHEMA("Neo4jWriter-01", "载入关系模式异常."),
|
||||
UNSUPPORTED_TYPE("Neo4jWriter-02", "不支持字段类型."),
|
||||
WRONG_RECORD_TYPE("Neo4jWriter-03, {}, {}", "错误记录类型."),
|
||||
WRONG_NEO4j_CLIENT("Neo4jWriter-04", "neo4j client访问异常."),
|
||||
;
|
||||
|
||||
private final String code;
|
||||
private final String description;
|
||||
|
||||
Neo4jWriterErrorCode(String code, String description) {
|
||||
this.code = code;
|
||||
this.description = description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getCode() {
|
||||
return this.code;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getDescription() {
|
||||
return this.description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return String.format("Code:[%s], Description:[%s]. ", this.code,
|
||||
this.description);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,61 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.writer.neo4jwriter;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.StringUtils;
|
||||
|
||||
public class Utils {
|
||||
|
||||
/** 字段分割*/
|
||||
public static final String FIELD_SEQ = " and ";
|
||||
|
||||
// 关系wehre
|
||||
public static String relWhere(List<String> fields) {
|
||||
return relWhere(null, fields);
|
||||
}
|
||||
|
||||
public static String relWhere(String prefix, List<String> fields) {
|
||||
return relWhere(prefix, fields, FIELD_SEQ);
|
||||
}
|
||||
|
||||
public static String relWhere(String prefix, List<String> fields, String seq) {
|
||||
if (fields == null || fields.size() == 0)
|
||||
return "";
|
||||
StringBuffer sb = new StringBuffer();
|
||||
for (int i = 0; i < fields.size(); i++) {
|
||||
String tmp = StringUtils.isNotEmpty(prefix) ? prefix+"."+fields.get(i) : fields.get(i) + seq;
|
||||
sb.append(tmp);
|
||||
}
|
||||
// 去掉最后seq
|
||||
sb.delete(sb.length()-seq.length(), sb.length());
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
private static Map<String, String> ESCAPES = new HashMap<String, String>();
|
||||
|
||||
static {
|
||||
// '\' -> '\\'
|
||||
// '"' -> '""'
|
||||
ESCAPES.put("\\", "\\\\\\\\");
|
||||
ESCAPES.put("\"", "\"\"");
|
||||
}
|
||||
|
||||
// 字符字段转义
|
||||
public static String strFieldEscape(String fieldString) {
|
||||
if (StringUtils.isEmpty(fieldString))
|
||||
return fieldString;
|
||||
String ed = fieldString;
|
||||
for (String key : ESCAPES.keySet()) {
|
||||
if(ed.contains(key)) {
|
||||
if("\\".equals(key))
|
||||
ed = ed.replaceAll("\\\\", ESCAPES.get(key));
|
||||
else
|
||||
ed = ed.replaceAll(key, ESCAPES.get(key));
|
||||
}
|
||||
}
|
||||
return ed;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,26 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.writer.neo4jwriter;
|
||||
|
||||
/**
|
||||
* @类名: WriteMode
|
||||
* @说明: 写入模式
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2020年9月3日 下午6:01:12s
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public enum WriteMode {
|
||||
|
||||
// CLEAR_BEFORE_INSERT, // 清除
|
||||
INSERT, // 直接写入
|
||||
INSERT_NOT_EXIST,
|
||||
REPALCE, // 每次删除后插入
|
||||
UPDATE // 更新模式, 不存在插入
|
||||
;
|
||||
|
||||
}
|
||||
6
rdb2graph/neo4jwriter/src/main/resources/plugin.json
Normal file
6
rdb2graph/neo4jwriter/src/main/resources/plugin.json
Normal file
@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "neo4jwriter",
|
||||
"class": "com.leehom.arch.datax.plugin.rdb2graph.writer.neo4jwriter.Neo4jWriter",
|
||||
"description": "",
|
||||
"developer": "leehom"
|
||||
}
|
||||
@ -0,0 +1,13 @@
|
||||
package com.leehom.arch.datax.rdb2graph.writer.neo4jwriter;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.writer.neo4jwriter.Utils;
|
||||
|
||||
public class EscapeTest {
|
||||
|
||||
public static void main(String[] args) {
|
||||
String escapeStr = "xx\\Gxx";
|
||||
String ed =Utils.strFieldEscape(escapeStr);
|
||||
System.out.print(ed);
|
||||
|
||||
}
|
||||
}
|
||||
350
rdb2graph/oraclereader4graph/oraclereader.md
Normal file
350
rdb2graph/oraclereader4graph/oraclereader.md
Normal file
@ -0,0 +1,350 @@
|
||||
|
||||
# OracleReader 插件文档
|
||||
|
||||
|
||||
___
|
||||
|
||||
|
||||
## 1 快速介绍
|
||||
|
||||
OracleReader插件实现了从Oracle读取数据。在底层实现上，OracleReader通过JDBC连接远程Oracle数据库，并执行相应的sql语句将数据从Oracle库中SELECT出来。
|
||||
|
||||
## 2 实现原理
|
||||
|
||||
简而言之，OracleReader通过JDBC连接器连接到远程的Oracle数据库，并根据用户配置的信息生成查询SELECT SQL语句并发送到远程Oracle数据库，并将该SQL执行返回结果使用DataX自定义的数据类型拼装为抽象的数据集，并传递给下游Writer处理。
|
||||
|
||||
对于用户配置Table、Column、Where的信息，OracleReader将其拼接为SQL语句发送到Oracle数据库；对于用户配置querySql信息，Oracle直接将其发送到Oracle数据库。
|
||||
|
||||
|
||||
## 3 功能说明
|
||||
|
||||
### 3.1 配置样例
|
||||
|
||||
* 配置一个从Oracle数据库同步抽取数据到本地的作业:
|
||||
|
||||
```
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
//设置传输速度 byte/s 尽量逼近这个速度但是不高于它.
|
||||
// channel 表示通道数量，byte表示通道速度，如果单通道速度1MB，配置byte为1048576表示一个channel
|
||||
"byte": 1048576
|
||||
},
|
||||
//出错限制
|
||||
"errorLimit": {
|
||||
//先选择record
|
||||
"record": 0,
|
||||
//百分比 1表示100%
|
||||
"percentage": 0.02
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "oraclereader",
|
||||
"parameter": {
|
||||
// 数据库连接用户名
|
||||
"username": "root",
|
||||
// 数据库连接密码
|
||||
"password": "root",
|
||||
"column": [
|
||||
"id","name"
|
||||
],
|
||||
//切分主键
|
||||
"splitPk": "db_id",
|
||||
"connection": [
|
||||
{
|
||||
"table": [
|
||||
"table"
|
||||
],
|
||||
"jdbcUrl": [
|
||||
"jdbc:oracle:thin:@[HOST_NAME]:PORT:[DATABASE_NAME]"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
//writer类型
|
||||
"name": "streamwriter",
|
||||
// 是否打印内容
|
||||
"parameter": {
|
||||
"print": true
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
* 配置一个自定义SQL的数据库同步任务到本地内容的作业：
|
||||
|
||||
```
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 5
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "oraclereader",
|
||||
"parameter": {
|
||||
"username": "root",
|
||||
"password": "root",
|
||||
"where": "",
|
||||
"connection": [
|
||||
{
|
||||
"querySql": [
|
||||
"select db_id,on_line_flag from db_info where db_id < 10"
|
||||
],
|
||||
"jdbcUrl": [
|
||||
"jdbc:oracle:thin:@[HOST_NAME]:PORT:[DATABASE_NAME]"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "streamwriter",
|
||||
"parameter": {
|
||||
"visible": false,
|
||||
"encoding": "UTF-8"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### 3.2 参数说明
|
||||
|
||||
* **jdbcUrl**
|
||||
|
||||
* 描述：描述的是到对端数据库的JDBC连接信息，使用JSON的数组描述，并支持一个库填写多个连接地址。之所以使用JSON数组描述连接信息，是因为阿里集团内部支持多个IP探测，如果配置了多个，OracleReader可以依次探测ip的可连接性，直到选择一个合法的IP。如果全部连接失败，OracleReader报错。 注意，jdbcUrl必须包含在connection配置单元中。对于阿里集团外部使用情况，JSON数组填写一个JDBC连接即可。
|
||||
|
||||
jdbcUrl按照Oracle官方规范，并可以填写连接附件控制信息。具体请参看[Oracle官方文档](http://www.oracle.com/technetwork/database/enterprise-edition/documentation/index.html)。
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **username**
|
||||
|
||||
* 描述：数据源的用户名 <br />
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **password**
|
||||
|
||||
* 描述：数据源指定用户名的密码 <br />
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **table**
|
||||
|
||||
* 描述：所选取的需要同步的表。使用JSON的数组描述，因此支持多张表同时抽取。当配置为多张表时，用户自己需保证多张表是同一schema结构，OracleReader不予检查表是否同一逻辑表。注意，table必须包含在connection配置单元中。<br />
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **column**
|
||||
|
||||
* 描述：所配置的表中需要同步的列名集合，使用JSON的数组描述字段信息。用户使用\*代表默认使用所有列配置，例如['\*']。
|
||||
|
||||
支持列裁剪，即列可以挑选部分列进行导出。
|
||||
|
||||
支持列换序，即列可以不按照表schema信息进行导出。
|
||||
|
||||
支持常量配置，用户需要按照JSON格式:
|
||||
["id", "`table`", "1", "'bazhen.csy'", "null", "to_char(a + 1)", "2.3" , "true"]
|
||||
id为普通列名，\`table\`为包含保留在的列名，1为整形数字常量，'bazhen.csy'为字符串常量，null为空指针，to_char(a + 1)为表达式，2.3为浮点数，true为布尔值。
|
||||
|
||||
Column必须显示填写，不允许为空！
|
||||
|
||||
* 必选：是 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **splitPk**
|
||||
|
||||
* 描述：OracleReader进行数据抽取时，如果指定splitPk，表示用户希望使用splitPk代表的字段进行数据分片，DataX因此会启动并发任务进行数据同步，这样可以大大提供数据同步的效能。
|
||||
|
||||
推荐splitPk用户使用表主键，因为表主键通常情况下比较均匀，因此切分出来的分片也不容易出现数据热点。
|
||||
|
||||
目前splitPk仅支持整形、字符串型数据切分，`不支持浮点、日期等其他类型`。如果用户指定其他非支持类型，OracleReader将报错！
|
||||
|
||||
splitPk如果不填写，将视作用户不对单表进行切分，OracleReader使用单通道同步全量数据。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **where**
|
||||
|
||||
* 描述：筛选条件，MysqlReader根据指定的column、table、where条件拼接SQL，并根据这个SQL进行数据抽取。在实际业务场景中，往往会选择当天的数据进行同步，可以将where条件指定为gmt_create > $bizdate 。注意：不可以将where条件指定为limit 10，limit不是SQL的合法where子句。<br />
|
||||
|
||||
where条件可以有效地进行业务增量同步。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **querySql**
|
||||
|
||||
* 描述：在有些业务场景下，where这一配置项不足以描述所筛选的条件，用户可以通过该配置型来自定义筛选SQL。当用户配置了这一项之后，DataX系统就会忽略table，column这些配置型，直接使用这个配置项的内容对数据进行筛选，例如需要进行多表join后同步数据，使用select a,b from table_a join table_b on table_a.id = table_b.id <br />
|
||||
|
||||
`当用户配置querySql时，OracleReader直接忽略table、column、where条件的配置`。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
* **fetchSize**
|
||||
|
||||
* 描述：该配置项定义了插件和数据库服务器端每次批量数据获取条数，该值决定了DataX和服务器端的网络交互次数，能够较大的提升数据抽取性能。<br />
|
||||
|
||||
`注意，该值过大(>2048)可能造成DataX进程OOM。`。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：1024 <br />
|
||||
|
||||
* **session**
|
||||
|
||||
* 描述：控制写入数据的时间格式，时区等的配置，如果表中有时间字段，配置该值以明确告知写入 oracle 的时间格式。通常配置的参数为：NLS_DATE_FORMAT,NLS_TIME_FORMAT。其配置的值为 json 格式，例如：
|
||||
```
|
||||
"session": [
|
||||
"alter session set NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'",
|
||||
"alter session set NLS_TIMESTAMP_FORMAT='yyyy-mm-dd hh24:mi:ss'",
|
||||
"alter session set NLS_TIMESTAMP_TZ_FORMAT='yyyy-mm-dd hh24:mi:ss'",
|
||||
"alter session set TIME_ZONE='US/Pacific'"
|
||||
]
|
||||
```
|
||||
`(注意"是 " 的转义字符串)`。
|
||||
|
||||
* 必选：否 <br />
|
||||
|
||||
* 默认值：无 <br />
|
||||
|
||||
|
||||
### 3.3 类型转换
|
||||
|
||||
目前OracleReader支持大部分Oracle类型，但也存在部分个别类型没有支持的情况，请注意检查你的类型。
|
||||
|
||||
下面列出OracleReader针对Oracle类型转换列表:
|
||||
|
||||
|
||||
| DataX 内部类型| Oracle 数据类型 |
|
||||
| -------- | ----- |
|
||||
| Long |NUMBER,INTEGER,INT,SMALLINT|
|
||||
| Double |NUMERIC,DECIMAL,FLOAT,DOUBLE PRECISION,REAL|
|
||||
| String |LONG,CHAR,NCHAR,VARCHAR,VARCHAR2,NVARCHAR2,CLOB,NCLOB,CHARACTER,CHARACTER VARYING,CHAR VARYING,NATIONAL CHARACTER,NATIONAL CHAR,NATIONAL CHARACTER VARYING,NATIONAL CHAR VARYING,NCHAR VARYING |
|
||||
| Date |TIMESTAMP,DATE |
|
||||
| Boolean |bit, bool |
|
||||
| Bytes |BLOB,BFILE,RAW,LONG RAW |
|
||||
|
||||
|
||||
|
||||
请注意:
|
||||
|
||||
* `除上述罗列字段类型外，其他类型均不支持`。
|
||||
|
||||
|
||||
## 4 性能报告
|
||||
|
||||
### 4.1 环境准备
|
||||
|
||||
#### 4.1.1 数据特征
|
||||
|
||||
为了模拟线上真实数据，我们设计两个Oracle数据表，分别为:
|
||||
|
||||
#### 4.1.2 机器参数
|
||||
|
||||
* 执行DataX的机器参数为:
|
||||
|
||||
* Oracle数据库机器参数为:
|
||||
|
||||
### 4.2 测试报告
|
||||
|
||||
#### 4.2.1 表1测试报告
|
||||
|
||||
|
||||
| 并发任务数| DataX速度(Rec/s)|DataX流量|网卡流量|DataX运行负载|DB运行负载|
|
||||
|--------| --------|--------|--------|--------|--------|
|
||||
|1| DataX 统计速度(Rec/s)|DataX统计流量|网卡流量|DataX运行负载|DB运行负载|
|
||||
|
||||
## 5 约束限制
|
||||
|
||||
### 5.1 主备同步数据恢复问题
|
||||
|
||||
主备同步问题指Oracle使用主从灾备，备库从主库不间断通过binlog恢复数据。由于主备数据同步存在一定的时间差，特别在于某些特定情况，例如网络延迟等问题，导致备库同步恢复的数据与主库有较大差别，导致从备库同步的数据不是一份当前时间的完整镜像。
|
||||
|
||||
针对这个问题，我们提供了preSql功能，该功能待补充。
|
||||
|
||||
### 5.2 一致性约束
|
||||
|
||||
Oracle在数据存储划分中属于RDBMS系统，对外可以提供强一致性数据查询接口。例如当一次同步任务启动运行过程中，当该库存在其他数据写入方写入数据时，OracleReader完全不会获取到写入更新数据，这是由于数据库本身的快照特性决定的。关于数据库快照特性，请参看[MVCC Wikipedia](https://en.wikipedia.org/wiki/Multiversion_concurrency_control)
|
||||
|
||||
上述是在OracleReader单线程模型下数据同步一致性的特性，由于OracleReader可以根据用户配置信息使用了并发数据抽取，因此不能严格保证数据一致性：当OracleReader根据splitPk进行数据切分后，会先后启动多个并发任务完成数据同步。由于多个并发任务相互之间不属于同一个读事务，同时多个并发任务存在时间间隔。因此这份数据并不是`完整的`、`一致的`数据快照信息。
|
||||
|
||||
针对多线程的一致性快照需求，在技术上目前无法实现，只能从工程角度解决，工程化的方式存在取舍，我们提供几个解决思路给用户，用户可以自行选择：
|
||||
|
||||
1. 使用单线程同步，即不再进行数据切片。缺点是速度比较慢，但是能够很好保证一致性。
|
||||
|
||||
2. 关闭其他数据写入方，保证当前数据为静态数据，例如，锁表、关闭备库同步等等。缺点是可能影响在线业务。
|
||||
|
||||
### 5.3 数据库编码问题
|
||||
|
||||
|
||||
OracleReader底层使用JDBC进行数据抽取，JDBC天然适配各类编码，并在底层进行了编码转换。因此OracleReader不需用户指定编码，可以自动获取编码并转码。
|
||||
|
||||
对于Oracle底层写入编码和其设定的编码不一致的混乱情况，OracleReader对此无法识别，对此也无法提供解决方案，对于这类情况，`导出有可能为乱码`。
|
||||
|
||||
### 5.4 增量数据同步
|
||||
|
||||
OracleReader使用JDBC SELECT语句完成数据抽取工作，因此可以使用SELECT...WHERE...进行增量数据抽取，方式有多种：
|
||||
|
||||
* 数据库在线应用写入数据库时，填充modify字段为更改时间戳，包括新增、更新、删除(逻辑删)。对于这类应用，OracleReader只需要WHERE条件跟上一同步阶段时间戳即可。
|
||||
* 对于新增流水型数据，OracleReader可以WHERE条件后跟上一阶段最大自增ID即可。
|
||||
|
||||
对于业务上无字段区分新增、修改数据情况，OracleReader也无法进行增量数据同步，只能同步全量数据。
|
||||
|
||||
### 5.5 Sql安全性
|
||||
|
||||
OracleReader提供querySql语句交给用户自己实现SELECT抽取语句，OracleReader本身对querySql不做任何安全性校验。这块交由DataX用户方自己保证。
|
||||
|
||||
## 6 FAQ
|
||||
|
||||
***
|
||||
|
||||
**Q: OracleReader同步报错，报错信息为XXX**
|
||||
|
||||
A: 网络或者权限问题，请使用Oracle命令行测试：
|
||||
sqlplus username/password@//host:port/sid
|
||||
|
||||
|
||||
如果上述命令也报错，那可以证实是环境问题，请联系你的DBA。
|
||||
|
||||
|
||||
**Q: OracleReader抽取速度很慢怎么办？**
|
||||
|
||||
A: 影响抽取时间的原因大概有如下几个：(来自专业 DBA 卫绾)
|
||||
1. 由于SQL的plan异常，导致的抽取时间长； 在抽取时，尽可能使用全表扫描代替索引扫描;
|
||||
2. 合理sql的并发度，减少抽取时间；根据表的大小，
|
||||
<50G可以不用并发，
|
||||
<100G添加如下hint: parallel(a,2）,
|
||||
>100G添加如下hint : parallel(a,4);
|
||||
3. 抽取sql要简单，尽量不用replace等函数，这个非常消耗cpu，会严重影响抽取速度;
|
||||
90
rdb2graph/oraclereader4graph/pom.xml
Normal file
90
rdb2graph/oraclereader4graph/pom.xml
Normal file
@ -0,0 +1,90 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<parent>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-parent</artifactId>
|
||||
<version>${revision}</version>
|
||||
</parent>
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<artifactId>oraclereader4graph</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-common</artifactId>
|
||||
<version>${datax.version}</version>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<artifactId>slf4j-log4j12</artifactId>
|
||||
<groupId>org.slf4j</groupId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.slf4j</groupId>
|
||||
<artifactId>slf4j-api</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>ch.qos.logback</groupId>
|
||||
<artifactId>logback-classic</artifactId>
|
||||
</dependency>
|
||||
|
||||
|
||||
<dependency>
|
||||
<groupId>com.oracle</groupId>
|
||||
<artifactId>ojdbc6</artifactId>
|
||||
<version>11.2.0.3</version>
|
||||
<scope>system</scope>
|
||||
<systemPath>${basedir}/src/main/lib/ojdbc6-11.2.0.3.jar</systemPath>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-scanner</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-rdbms-util</artifactId>
|
||||
<version>1.0.0-SNAPSHOT</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<!-- compiler plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<configuration>
|
||||
<source>${java.version}</source>
|
||||
<target>${java.version}</target>
|
||||
<encoding>${project-sourceEncoding}</encoding>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<!-- assembly plugin -->
|
||||
<plugin>
|
||||
<artifactId>maven-assembly-plugin</artifactId>
|
||||
<configuration>
|
||||
<descriptors>
|
||||
<descriptor>src/main/assembly/package.xml</descriptor>
|
||||
</descriptors>
|
||||
<finalName>datax</finalName>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>dwzip</id>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>single</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
42
rdb2graph/oraclereader4graph/src/main/assembly/package.xml
Normal file
42
rdb2graph/oraclereader4graph/src/main/assembly/package.xml
Normal file
@ -0,0 +1,42 @@
|
||||
<assembly
|
||||
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
|
||||
<id></id>
|
||||
<formats>
|
||||
<format>dir</format>
|
||||
</formats>
|
||||
<includeBaseDirectory>false</includeBaseDirectory>
|
||||
<fileSets>
|
||||
<fileSet>
|
||||
<directory>src/main/resources</directory>
|
||||
<includes>
|
||||
<include>plugin.json</include>
|
||||
<include>plugin_job_template.json</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/reader/oraclereader4graph</outputDirectory>
|
||||
</fileSet>
|
||||
<fileSet>
|
||||
<directory>src/main/lib</directory>
|
||||
<includes>
|
||||
<include>ojdbc6-11.2.0.3.jar</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/reader/oraclereader4graph/libs</outputDirectory>
|
||||
</fileSet>
|
||||
<fileSet>
|
||||
<directory>target/</directory>
|
||||
<includes>
|
||||
<include>oraclereader4graph-1.0.0-SNAPSHOT.jar</include>
|
||||
</includes>
|
||||
<outputDirectory>plugin/reader/oraclereader4graph</outputDirectory>
|
||||
</fileSet>
|
||||
</fileSets>
|
||||
|
||||
<dependencySets>
|
||||
<dependencySet>
|
||||
<useProjectArtifact>false</useProjectArtifact>
|
||||
<outputDirectory>plugin/reader/oraclereader4graph/libs</outputDirectory>
|
||||
<scope>runtime</scope>
|
||||
</dependencySet>
|
||||
</dependencySets>
|
||||
</assembly>
|
||||
@ -0,0 +1,7 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.reader.oraclereader;
|
||||
|
||||
public class Constant {
|
||||
|
||||
public static final int DEFAULT_FETCH_SIZE = 1024;
|
||||
|
||||
}
|
||||
@ -0,0 +1,143 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.reader.oraclereader;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.alibaba.datax.common.exception.DataXException;
|
||||
import com.alibaba.datax.common.plugin.RecordSender;
|
||||
import com.alibaba.datax.common.spi.Reader;
|
||||
import com.alibaba.datax.common.util.Configuration;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.CommonRdbms2GraphReader;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.Key;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.util.HintUtil;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.util.DBUtilErrorCode;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.rdbms.util.DataBaseType;
|
||||
|
||||
/**
|
||||
* @类名: OracleReader4Graph
|
||||
* @说明: oracle读入
|
||||
* 1.表数据同步，datax table模式，schema表写入datax的配置
|
||||
2.表关系同步，按表连接图分组，一对关系一组，表连接关系生成querySql，写入datax的querySql模式配置
|
||||
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月28日 下午6:30:15
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class OracleReader4Graph extends Reader {
|
||||
|
||||
private static final DataBaseType DATABASE_TYPE = DataBaseType.Oracle;
|
||||
|
||||
public static class Job extends Reader.Job {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(OracleReader4Graph.Job.class);
|
||||
|
||||
//
|
||||
private Configuration originalConfig = null;
|
||||
private CommonRdbms2GraphReader.Job commonRdbmsReaderJob;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
// 载入配置
|
||||
this.originalConfig = super.getPluginJobConf();
|
||||
//
|
||||
dealFetchSize(this.originalConfig);
|
||||
//
|
||||
this.commonRdbmsReaderJob = new CommonRdbms2GraphReader.Job(DATABASE_TYPE);
|
||||
//
|
||||
this.commonRdbmsReaderJob.init(this.originalConfig);
|
||||
|
||||
// 注意：要在 this.commonRdbmsReaderJob.init(this.originalConfig); 之后执行，这样可以直接快速判断是否是querySql 模式
|
||||
dealHint(this.originalConfig);
|
||||
}
|
||||
|
||||
// 检查连接/表是否可查询
|
||||
@Override
|
||||
public void preCheck(){
|
||||
init();
|
||||
// 检测
|
||||
this.commonRdbmsReaderJob.preCheck(this.originalConfig, DATABASE_TYPE);
|
||||
}
|
||||
|
||||
// 分片，包括多表分配，表分片
|
||||
@Override
|
||||
public List<Configuration> split(int adviceNumber) {
|
||||
return this.commonRdbmsReaderJob.split(this.originalConfig, adviceNumber);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
this.commonRdbmsReaderJob.post(this.originalConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
this.commonRdbmsReaderJob.destroy(this.originalConfig);
|
||||
}
|
||||
|
||||
// fetch size
|
||||
private void dealFetchSize(Configuration originalConfig) {
|
||||
int fetchSize = originalConfig.getInt(
|
||||
com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.Constant.FETCH_SIZE,
|
||||
Constant.DEFAULT_FETCH_SIZE);
|
||||
if (fetchSize < 1) {
|
||||
throw DataXException
|
||||
.asDataXException(DBUtilErrorCode.REQUIRED_VALUE,
|
||||
String.format("您配置的 fetchSize 有误，fetchSize:[%d] 值不能小于 1.",
|
||||
fetchSize));
|
||||
}
|
||||
originalConfig.set(
|
||||
com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.Constant.FETCH_SIZE, fetchSize);
|
||||
}
|
||||
|
||||
private void dealHint(Configuration originalConfig) {
|
||||
String hint = originalConfig.getString(Key.HINT);
|
||||
if (StringUtils.isNotBlank(hint)) {
|
||||
boolean isTableMode = originalConfig.getBool(com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.Constant.IS_TABLE_MODE).booleanValue();
|
||||
if(!isTableMode){
|
||||
throw DataXException.asDataXException(OracleReaderErrorCode.HINT_ERROR, "当且仅当非 querySql 模式读取 oracle 时才能配置 HINT.");
|
||||
}
|
||||
HintUtil.initHintConf(DATABASE_TYPE, originalConfig);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public static class Task extends Reader.Task {
|
||||
|
||||
private Configuration readerSliceConfig;
|
||||
private CommonRdbms2GraphReader.Task commonRdbmsReaderTask;
|
||||
|
||||
@Override
|
||||
public void init() {
|
||||
this.readerSliceConfig = super.getPluginJobConf();
|
||||
this.commonRdbmsReaderTask = new CommonRdbms2GraphReader.Task(
|
||||
DATABASE_TYPE ,super.getTaskGroupId(), super.getTaskId());
|
||||
this.commonRdbmsReaderTask.init(this.readerSliceConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void startRead(RecordSender recordSender) {
|
||||
int fetchSize = this.readerSliceConfig
|
||||
.getInt(com.leehom.arch.datax.plugin.rdb2graph.rdbms.reader.Constant.FETCH_SIZE);
|
||||
|
||||
this.commonRdbmsReaderTask.startRead(this.readerSliceConfig,
|
||||
recordSender, super.getTaskPluginCollector(), fetchSize);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void post() {
|
||||
this.commonRdbmsReaderTask.post(this.readerSliceConfig);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void destroy() {
|
||||
this.commonRdbmsReaderTask.destroy(this.readerSliceConfig);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,32 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.reader.oraclereader;
|
||||
|
||||
import com.alibaba.datax.common.spi.ErrorCode;
|
||||
|
||||
public enum OracleReaderErrorCode implements ErrorCode {
|
||||
HINT_ERROR("Oraclereader-00", "您的 Hint 配置出错."),
|
||||
;
|
||||
|
||||
private final String code;
|
||||
private final String description;
|
||||
|
||||
private OracleReaderErrorCode(String code, String description) {
|
||||
this.code = code;
|
||||
this.description = description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getCode() {
|
||||
return this.code;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getDescription() {
|
||||
return this.description;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return String.format("Code:[%s], Description:[%s]. ", this.code,
|
||||
this.description);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "oraclereader4graph",
|
||||
"class": "com.leehom.arch.datax.plugin.rdb2graph.reader.oraclereader.OracleReader4Graph",
|
||||
"description": "",
|
||||
"developer": "leehom"
|
||||
}
|
||||
@ -0,0 +1,16 @@
|
||||
{
|
||||
"name": "oraclereader4graph",
|
||||
"parameter": {
|
||||
"username": "",
|
||||
"password": "",
|
||||
"phase": "",
|
||||
"schemaUri": "",
|
||||
"column": [],
|
||||
"connection": [
|
||||
{
|
||||
"table": [],
|
||||
"jdbcUrl": []
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
146
rdb2graph/pom.xml
Normal file
146
rdb2graph/pom.xml
Normal file
@ -0,0 +1,146 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-all</artifactId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
</parent>
|
||||
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-parent</artifactId>
|
||||
<packaging>pom</packaging>
|
||||
<version>${revision}</version>
|
||||
|
||||
<properties>
|
||||
<revision>1.0.0-SNAPSHOT</revision>
|
||||
<java.version>1.8</java.version>
|
||||
<fastjson.version>1.2.36</fastjson.version>
|
||||
<xstream.version>1.4.10</xstream.version>
|
||||
<guava.version>27.0.1-jre</guava.version>
|
||||
<datax.version>0.0.1-SNAPSHOT</datax.version>
|
||||
<lombok.version>1.18.12</lombok.version>
|
||||
<druid.version>1.1.23</druid.version>
|
||||
<lombok.version>1.18.12</lombok.version>
|
||||
<base.version>1.0.1-SNAPSHOT</base.version>
|
||||
|
||||
</properties>
|
||||
|
||||
<modules>
|
||||
<module>rdb2graph-scanner</module>
|
||||
<module>rdb2graph-transformer</module>
|
||||
<module>neo4jwriter</module>
|
||||
<module>mysqlreader4graph</module>
|
||||
<module>oraclereader4graph</module>
|
||||
<module>rdb2grpah-rdbms-util</module>
|
||||
<module>rdb2graph-common</module>
|
||||
<module>rdb2graph-datax</module>
|
||||
</modules>
|
||||
|
||||
<distributionManagement>
|
||||
<repository>
|
||||
<id>rdc-releases</id>
|
||||
<url>https://repo.rdc.aliyun.com/repository/128947-release-JP7RVf/</url>
|
||||
</repository>
|
||||
<snapshotRepository>
|
||||
<id>rdc-snapshots</id>
|
||||
<url>https://repo.rdc.aliyun.com/repository/128947-snapshot-2gBBfM/</url>
|
||||
</snapshotRepository>
|
||||
</distributionManagement>
|
||||
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.alibaba</groupId>
|
||||
<artifactId>fastjson</artifactId>
|
||||
<version>${fastjson.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-core</artifactId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-common</artifactId>
|
||||
<version>0.0.1-SNAPSHOT</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<version>${lombok.version}</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.alibaba</groupId>
|
||||
<artifactId>druid</artifactId>
|
||||
<version>${druid.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.google.guava</groupId>
|
||||
<artifactId>guava</artifactId>
|
||||
<version>${guava.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.thoughtworks.xstream</groupId>
|
||||
<artifactId>xstream</artifactId>
|
||||
<version>${xstream.version}</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
<build>
|
||||
<pluginManagement>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-compiler-plugin</artifactId>
|
||||
<version>3.1</version>
|
||||
<configuration>
|
||||
<source>${java.version}</source>
|
||||
<target>${java.version}</target>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
<version>${spring-boot.version}</version>
|
||||
<configuration>
|
||||
<fork>true</fork>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</pluginManagement>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.codehaus.mojo</groupId>
|
||||
<artifactId>flatten-maven-plugin</artifactId>
|
||||
<version>1.1.0</version>
|
||||
<configuration>
|
||||
<!-- 是否更新pom文件，此处还有更高级的用法 -->
|
||||
<updatePomFile>true</updatePomFile>
|
||||
<flattenMode>oss</flattenMode>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>flatten</id>
|
||||
<phase>process-resources</phase>
|
||||
<goals>
|
||||
<goal>flatten</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
<execution>
|
||||
<id>flatten.clean</id>
|
||||
<phase>clean</phase>
|
||||
<goals>
|
||||
<goal>clean</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
54
rdb2graph/rdb2graph-common/pom.xml
Normal file
54
rdb2graph/rdb2graph-common/pom.xml
Normal file
@ -0,0 +1,54 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-parent</artifactId>
|
||||
<version>${revision}</version>
|
||||
</parent>
|
||||
<artifactId>rdb2graph-common</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<prometheus.client.version>0.9.0</prometheus.client.version>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<!-- -->
|
||||
<dependency>
|
||||
<groupId>com.google.guava</groupId>
|
||||
<artifactId>guava</artifactId>
|
||||
</dependency>
|
||||
<!-- tools -->
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
</dependency>
|
||||
<!-- -->
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-core</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.thoughtworks.xstream</groupId>
|
||||
<artifactId>xstream</artifactId>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-jar-plugin</artifactId>
|
||||
<version>3.0.2</version>
|
||||
<configuration>
|
||||
<excludes>
|
||||
<exclude>logback.xml</exclude>
|
||||
<exclude>application.yml</exclude>
|
||||
</excludes>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
@ -0,0 +1,158 @@
|
||||
/**
|
||||
*
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common;
|
||||
|
||||
import java.io.PrintStream;
|
||||
import java.lang.reflect.ParameterizedType;
|
||||
import java.lang.reflect.Type;
|
||||
import java.math.BigDecimal;
|
||||
import java.math.BigInteger;
|
||||
import java.util.Date;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* 类名: BeanUtils
|
||||
* 说明: Bean工具类
|
||||
*
|
||||
* @author leehom
|
||||
* @Date Nov 30, 2009 4:09:36 PM
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class BeanUtils {
|
||||
|
||||
/**
|
||||
* @说明：输出Bean的属性/深输出Bean的属性
|
||||
*
|
||||
* @param '输出流'
|
||||
* @author hjli
|
||||
*
|
||||
* @异常：
|
||||
*/
|
||||
public static void printBean(Object bean) {
|
||||
printBean(bean, System.out);
|
||||
}
|
||||
|
||||
public static void printBean(Object bean, PrintStream out) {
|
||||
out.println(bean.toString());
|
||||
}
|
||||
|
||||
public static void printBeanDeep(Object bean) {
|
||||
printBeanDeep(bean, System.out);
|
||||
}
|
||||
|
||||
public static void printBeanDeep(Object bean, PrintStream out) {
|
||||
try {
|
||||
Map m = org.apache.commons.beanutils.BeanUtils.describe(bean);
|
||||
for (Object o : m.keySet()) {
|
||||
if(o==null){
|
||||
out.println("Null value field");
|
||||
continue;
|
||||
}
|
||||
out.println(o.toString()+":"+m.get(o));
|
||||
}
|
||||
} catch(Exception ex) {
|
||||
throw new RuntimeException(ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
public static Long number2Long(Number num) {
|
||||
if(num!=null)
|
||||
return num.longValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Integer number2Integer(Number num) {
|
||||
if(num!=null)
|
||||
return num.intValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Double number2Double(Number num) {
|
||||
if(num!=null)
|
||||
return num.doubleValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Short number2Short(Number num) {
|
||||
if(num!=null)
|
||||
return num.shortValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Byte number2Byte(Number num) {
|
||||
if(num!=null)
|
||||
return num.byteValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Double bigDecimal2Double(BigDecimal num) {
|
||||
if(num!=null)
|
||||
return num.doubleValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Long bigDecimal2Long(BigDecimal num) {
|
||||
if(num!=null)
|
||||
return num.longValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Integer bigDecimal2Integer(BigDecimal num) {
|
||||
if(num!=null)
|
||||
return num.intValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Integer bigInteger2Integer(BigInteger num) {
|
||||
if(num!=null)
|
||||
return num.intValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Long bigInteger2Long(BigInteger num) {
|
||||
if(num!=null)
|
||||
return num.longValue();
|
||||
return null;
|
||||
|
||||
}
|
||||
|
||||
public static Date stringToDate(String dateStr, Class<?> clazz) {
|
||||
if(dateStr==null)
|
||||
return null;
|
||||
return DateTimeUtils.StringToDate(dateStr);
|
||||
|
||||
}
|
||||
|
||||
public static double doublePrecision(double d, short precision) {
|
||||
BigDecimal bg = new BigDecimal(d);
|
||||
double d2 = bg.setScale(precision, BigDecimal.ROUND_HALF_DOWN).doubleValue();
|
||||
return d2;
|
||||
}
|
||||
|
||||
public static Class getTemplateType(Object obj) {
|
||||
Class clazz = null;
|
||||
Class<?> c = obj.getClass();
|
||||
Type t = c.getGenericSuperclass();
|
||||
if (t instanceof ParameterizedType) {
|
||||
Type[] p = ((ParameterizedType) t).getActualTypeArguments();
|
||||
if(p[0] instanceof ParameterizedType )
|
||||
clazz = (Class) ((ParameterizedType) p[0]).getRawType();
|
||||
else
|
||||
clazz = (Class) p[0];
|
||||
}
|
||||
return clazz;
|
||||
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,125 @@
|
||||
/**
|
||||
* %流程框架%
|
||||
* %1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common;
|
||||
|
||||
import java.io.BufferedInputStream;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.ObjectInputStream;
|
||||
import java.io.ObjectOutputStream;
|
||||
import java.io.Serializable;
|
||||
|
||||
/**
|
||||
* @类名: ByteAndStreamUtils
|
||||
* @说明: 字节与流工具类
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2010-4-2 下午02:19:25
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class ByteAndStreamUtils {
|
||||
|
||||
/**
|
||||
* @说明：流转换成字节
|
||||
*
|
||||
* @author hjli
|
||||
* @param is
|
||||
* @return
|
||||
*
|
||||
* @异常：
|
||||
*/
|
||||
public static byte[] StreamToBytes(InputStream is) {
|
||||
BufferedInputStream bis = null;
|
||||
try {
|
||||
is = new BufferedInputStream(is);
|
||||
byte[] bytes = new byte[is.available()];
|
||||
int len = bytes.length;
|
||||
int offset = 0;
|
||||
int read = 0;
|
||||
while (offset < len
|
||||
&& (read = is.read(bytes, offset, len - offset)) > 0) {
|
||||
offset += read;
|
||||
}
|
||||
return bytes;
|
||||
} catch (Exception e) {
|
||||
return null;
|
||||
} finally {
|
||||
try {
|
||||
is.close();
|
||||
is = null;
|
||||
} catch (IOException e) {
|
||||
return null;
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
// A block of stream to bytes
|
||||
public static byte[] StreamBlockToBytes(InputStream is, long offset, int size) {
|
||||
BufferedInputStream bis = null;
|
||||
try {
|
||||
is = new BufferedInputStream(is);
|
||||
// Skip to the position where to start receiving bytes
|
||||
is.skip(offset);
|
||||
// Actual data size that would be get
|
||||
int datSize = is.available()< size ? is.available() : size;
|
||||
byte[] bytes = new byte[datSize];
|
||||
// Offset of data bytes which to start storing bytes
|
||||
int dataOffset = 0;
|
||||
int read = 0;
|
||||
while (dataOffset < size
|
||||
&& (read = is.read(bytes, dataOffset, datSize - dataOffset)) > 0) {
|
||||
dataOffset += read;
|
||||
}
|
||||
return bytes;
|
||||
} catch (Exception e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 从字节数组获取对象
|
||||
* @Author Sean.guo
|
||||
* @EditTime 2007-8-13 上午11:46:34
|
||||
*/
|
||||
public static Object objectFromBytes(byte[] objBytes) throws IOException, ClassNotFoundException {
|
||||
if (objBytes == null || objBytes.length == 0) {
|
||||
return null;
|
||||
}
|
||||
ByteArrayInputStream bi = new ByteArrayInputStream(objBytes);
|
||||
ObjectInputStream oi = new ObjectInputStream(bi);
|
||||
return oi.readObject();
|
||||
}
|
||||
|
||||
/**
|
||||
* 从对象获取一个字节数组
|
||||
* @Author Sean.guo
|
||||
* @EditTime 2007-8-13 上午11:46:56
|
||||
*/
|
||||
public static byte[] objectToBytes(Serializable obj) throws IOException {
|
||||
if (obj == null) {
|
||||
return null;
|
||||
}
|
||||
ByteArrayOutputStream bo = new ByteArrayOutputStream();
|
||||
ObjectOutputStream oo = new ObjectOutputStream(bo);
|
||||
oo.writeObject(obj);
|
||||
return bo.toByteArray();
|
||||
}
|
||||
|
||||
// 深克隆对象
|
||||
public static Object deepClone(Serializable obj) throws IOException, ClassNotFoundException {
|
||||
ByteArrayOutputStream bo = new ByteArrayOutputStream();
|
||||
ObjectOutputStream oo = new ObjectOutputStream(bo);
|
||||
oo.writeObject(obj);
|
||||
//
|
||||
ByteArrayInputStream bi = new ByteArrayInputStream(bo.toByteArray());
|
||||
ObjectInputStream oi = new ObjectInputStream(bi);
|
||||
return oi.readObject();
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,306 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common;
|
||||
|
||||
import java.text.SimpleDateFormat;
|
||||
import java.util.Calendar;
|
||||
import java.util.Date;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* 类名: DateTimeUtils.java
|
||||
* 说明: 日期时间工具类
|
||||
*
|
||||
* @author leehom
|
||||
* @Date Apr 23, 2009 5:36:41 AM
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class DateTimeUtils {
|
||||
|
||||
/**
|
||||
* 默认数据模式
|
||||
*/
|
||||
public static final String defaultPatten = "yyyy-MM-dd HH:mm:ss";
|
||||
public static final String defaultPattenMillis = "yyyy-MM-dd HH:mm:ss.SSS";
|
||||
public static final String defaultDatePatten = "yyyyMMdd";
|
||||
public static final String DatePatten2 = "yyyy-MM-dd";
|
||||
// ISO8601 Date Type Pattern
|
||||
public static final String ISO8601Patten = "yyyy-MM-dd'T'HH:mm:ssZZ";
|
||||
public static final String ISO8601PattenNoZone = "yyyy-MM-dd'T'HH:mm:ss";
|
||||
public static final String ISO8601PattenWithMillis = "yyyy-MM-dd'T'HH:mm:ss.SSSZZ";
|
||||
|
||||
|
||||
/**
|
||||
* 说明：Date to String
|
||||
*
|
||||
* @author hjli
|
||||
* @Param @param date
|
||||
* @Param @return
|
||||
* @Return String
|
||||
*
|
||||
*/
|
||||
public static String DateToString(Date date) {
|
||||
return DateToString(date, defaultPatten);
|
||||
}
|
||||
|
||||
public static String DateToString(Date date, String pattern) {
|
||||
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
|
||||
return sdf.format(date);
|
||||
}
|
||||
|
||||
/**
|
||||
* @说明：String to Date
|
||||
*
|
||||
* @author leehong
|
||||
* @param dateStr
|
||||
* @return Date
|
||||
*
|
||||
* @异常：
|
||||
*/
|
||||
public static Date StringToDate(String dateStr) {
|
||||
return StringToDate(dateStr, defaultPatten);
|
||||
}
|
||||
|
||||
public static Date StringToDate(String dateStr, String pattern) {
|
||||
if(dateStr==null)
|
||||
return null;
|
||||
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
|
||||
try {
|
||||
return sdf.parse(dateStr);
|
||||
} catch (Exception ex) {
|
||||
ex.printStackTrace();
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 获取当前日期
|
||||
*
|
||||
* @author hjli
|
||||
* @Param @return
|
||||
* @Return String
|
||||
*
|
||||
* TODO
|
||||
*
|
||||
*/
|
||||
public static String getCurrentDateTimeText(String pattern) {
|
||||
if ((pattern==null)||(pattern.equals(""))) {
|
||||
pattern = defaultPatten;
|
||||
}
|
||||
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
|
||||
return sdf.format(Calendar.getInstance().getTime());
|
||||
}
|
||||
|
||||
public static String getCurrentDateTimeText() {
|
||||
return getCurrentDateTimeText(defaultPatten);
|
||||
}
|
||||
|
||||
/**
|
||||
* 说明：获取前一小时时间
|
||||
*
|
||||
* @author hjli
|
||||
* @Param @param pattern
|
||||
* @Param @return
|
||||
* @Return String
|
||||
*
|
||||
* TODO
|
||||
*
|
||||
*/
|
||||
public static String getPreHourText(String pattern) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
return getPreHourText(c.getTime(), pattern);
|
||||
}
|
||||
|
||||
public static String getPreHourText(Date date, String pattern) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.HOUR_OF_DAY, -1);
|
||||
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
|
||||
return sdf.format(c.getTime());
|
||||
}
|
||||
|
||||
public static Date getPreHour(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.HOUR_OF_DAY, -1);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
/** 获取前n小时时间*/
|
||||
public static Date getPreHourN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.HOUR_OF_DAY, 0-n);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
public static Date getNextHour(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.HOUR_OF_DAY, 1);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
public static Date getNextHourN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.HOUR_OF_DAY, n);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* 说明：获取当前日前一日
|
||||
*
|
||||
* @author hjli
|
||||
* @Param @param pattern
|
||||
* @Param @return
|
||||
* @Return String
|
||||
*
|
||||
* TODO
|
||||
*
|
||||
*/
|
||||
public static String getPreDayText(String pattern) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.add(Calendar.DATE, -1);
|
||||
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
|
||||
return sdf.format(c.getTime());
|
||||
}
|
||||
|
||||
public static Date getPreDay(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.DATE, -1);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
/** 获取前n天时间*/
|
||||
public static Date getPreDayN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.DATE, 0-n);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
public static Date getNextDay(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.DATE, 1);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
public static Date getNextDayN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.DATE, n);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
public static Date getPreWeek(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.DATE, -7);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
/**
|
||||
* 说明：获取当前月前一月
|
||||
*
|
||||
* @author hjli
|
||||
* @Param @param pattern
|
||||
* @Param @return
|
||||
* @Return String
|
||||
*
|
||||
*/
|
||||
public static String getPreMonthText(String pattern) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.add(Calendar.MONTH, -1);
|
||||
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
|
||||
return sdf.format(c.getTime());
|
||||
}
|
||||
|
||||
public static Date getPreMonth(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.MONTH, -1);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
/** 获取前n月时间*/
|
||||
public static Date getPreMonthN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.MONTH, 0-n);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
public static Date getNextMonth(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.MONTH, 1);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
public static Date getNextMonthN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.MONTH, n);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
// 获取前一年
|
||||
public static String getPreYearText(String pattern) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.add(Calendar.YEAR, -1);
|
||||
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
|
||||
return sdf.format(c.getTime());
|
||||
}
|
||||
|
||||
public static Date getPreYear(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.YEAR, -1);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
/** 获取前n年*/
|
||||
public static Date getPreYearN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.YEAR, 0-n);
|
||||
return c.getTime();
|
||||
}
|
||||
|
||||
/** 获取下一年*/
|
||||
public static Date getNextYear(Date date) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.YEAR, 1);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
public static Date getNextYearN(Date date, int n) {
|
||||
Calendar c = Calendar.getInstance();
|
||||
c.setTime(date);
|
||||
c.add(Calendar.YEAR, n);
|
||||
return c.getTime();
|
||||
|
||||
}
|
||||
|
||||
// the right date type should be like "2012-12-15T00:00:00+08:00"
|
||||
public static void validateDateFormat(String dateString) throws Exception {
|
||||
Pattern pattern = Pattern.compile("\\d{4}\\-\\d{2}\\-\\d{2}T\\d{2}:\\d{2}:\\d{2}[\\+|\\-]\\d{2}:\\d{2}");
|
||||
Matcher matcher = pattern.matcher(dateString);
|
||||
boolean isValidated = matcher.matches();
|
||||
if (!isValidated) {
|
||||
throw new Exception("'" + dateString + "' is not a validate " +
|
||||
"date string. Please follow this sample:2012-12-15T00:00:00+08:00");
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,43 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common;
|
||||
|
||||
import com.alibaba.datax.core.transport.record.DefaultRecord;
|
||||
|
||||
/**
|
||||
* @类名: RelRecord
|
||||
* @说明: 关系record
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月26日 下午7:31:18
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class RelRecord extends DefaultRecord {
|
||||
|
||||
/** 关系起始表或者连接表*/
|
||||
private String fromTable;
|
||||
/**
|
||||
* 若连接表，fk则为连接的起点的关联表
|
||||
* 若为一般表，fk则为连接的外键，表可有多个外键
|
||||
*/
|
||||
private String fk;
|
||||
|
||||
public String getFk() {
|
||||
return fk;
|
||||
}
|
||||
public void setFk(String fk) {
|
||||
this.fk = fk;
|
||||
}
|
||||
public String getFromTable() {
|
||||
return fromTable;
|
||||
}
|
||||
public void setFromTable(String fromTable) {
|
||||
this.fromTable = fromTable;
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
@ -0,0 +1,251 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.MalformedURLException;
|
||||
import java.net.URISyntaxException;
|
||||
import java.net.URL;
|
||||
import java.util.Properties;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* @类名: ResourceLoaderUtil
|
||||
* @说明: 资源加载工具类
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2018-9-9 上午12:05:39
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class ResourceLoaderUtil {
|
||||
|
||||
private static Logger log = LoggerFactory.getLogger(ResourceLoaderUtil.class);
|
||||
|
||||
/**
|
||||
*
|
||||
* get a class with the name specified
|
||||
*
|
||||
* @paramclassName
|
||||
*
|
||||
* @return
|
||||
*
|
||||
*/
|
||||
public static Class loadClass(String className) {
|
||||
try {
|
||||
return getClassLoader().loadClass(className);
|
||||
}
|
||||
catch (ClassNotFoundException e) {
|
||||
throw new RuntimeException("class not found '" + className + "'", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
* get the class loader
|
||||
*
|
||||
* @return
|
||||
*
|
||||
*/
|
||||
public static ClassLoader getClassLoader() {
|
||||
return ResourceLoaderUtil.class.getClassLoader();
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
* get the file that from the path that related to current class path
|
||||
* example, there is the file root like this:
|
||||
* |--resource/xml/target.xml
|
||||
* |--classes
|
||||
* in order to get the xml file the api should called like this:
|
||||
* getStream("../resource/xml/target.xml");
|
||||
* @paramrelativePath
|
||||
*
|
||||
* @return
|
||||
*
|
||||
* @throwsIOException
|
||||
*
|
||||
* @throwsMalformedURLException
|
||||
*
|
||||
*/
|
||||
public static InputStream getResourceStream(String relativePath) throws MalformedURLException, IOException {
|
||||
if (!relativePath.contains("../")) {
|
||||
return getClassLoader().getResourceAsStream(relativePath);
|
||||
}
|
||||
else {
|
||||
return ResourceLoaderUtil.getStreamByExtendResource(relativePath);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
*
|
||||
* @paramurl
|
||||
*
|
||||
* @return
|
||||
*
|
||||
* @throwsIOException
|
||||
*
|
||||
*/
|
||||
public static InputStream getStream(URL url) throws IOException {
|
||||
if (url != null) {
|
||||
return url.openStream();
|
||||
}
|
||||
else {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
* get the file that from the path that related to current class path
|
||||
* example, there is the file root like this:
|
||||
* |--resource/xml/target.xml
|
||||
* |--classes
|
||||
* in order to get the xml file the api should called like this:
|
||||
* getStream("../resource/xml/target.xml");
|
||||
* @return
|
||||
*
|
||||
* @throwsMalformedURLException
|
||||
*
|
||||
* @throwsIOException
|
||||
*
|
||||
*/
|
||||
public static InputStream getStreamByExtendResource(String relativePath) throws MalformedURLException, IOException {
|
||||
return ResourceLoaderUtil.getStream(ResourceLoaderUtil.getExtendResource(relativePath));
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
* @paramresource
|
||||
*
|
||||
* @return
|
||||
*
|
||||
*/
|
||||
public static Properties getProperties(String resource) {
|
||||
Properties properties = new Properties();
|
||||
try {
|
||||
properties.load(getResourceStream(resource));
|
||||
}
|
||||
catch (IOException e) {
|
||||
throw new RuntimeException("couldn't load properties file '" + resource + "'", e);
|
||||
}
|
||||
return properties;
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
* get the absolute path of the class loader of this Class
|
||||
*
|
||||
*
|
||||
* @return
|
||||
*
|
||||
*/
|
||||
public static String getAbsolutePathOfClassLoaderClassPath() {
|
||||
//ClassLoaderUtil.log.info(ClassLoaderUtil.getClassLoader().getResource("").toString());
|
||||
return ResourceLoaderUtil.getClassLoader().getResource("").getPath();
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
* get the file that from the path that related to current class path
|
||||
* example, there is the file root like this:
|
||||
* |--resource/xml/target.xml
|
||||
* |--classes
|
||||
* in order to get the xml file the api should called like this:
|
||||
* getStream("../resource/xml/target.xml");
|
||||
*
|
||||
* @throwsMalformedURLException
|
||||
*
|
||||
*/
|
||||
public static URL getExtendResource(String relativePath) throws MalformedURLException {
|
||||
//ClassLoaderUtil.log.info("The income relative path:" + relativePath);
|
||||
// ClassLoaderUtil.log.info(Integer.valueOf(relativePath.indexOf("../"))) ;
|
||||
if (!relativePath.contains("../")) {
|
||||
return ResourceLoaderUtil.getResource(relativePath);
|
||||
}
|
||||
String classPathAbsolutePath = ResourceLoaderUtil.getAbsolutePathOfClassLoaderClassPath();
|
||||
if (relativePath.substring(0, 1).equals("/")) {
|
||||
relativePath = relativePath.substring(1);
|
||||
}
|
||||
//ClassLoaderUtil.log.info(Integer.valueOf(relativePath.lastIndexOf("../")));
|
||||
String wildcardString = relativePath.substring(0, relativePath.lastIndexOf("../") + 3);
|
||||
relativePath = relativePath.substring(relativePath.lastIndexOf("../") + 3);
|
||||
int containSum = ResourceLoaderUtil.containSum(wildcardString, "../");
|
||||
classPathAbsolutePath = ResourceLoaderUtil.cutLastString(classPathAbsolutePath, "/", containSum);
|
||||
String resourceAbsolutePath = classPathAbsolutePath + relativePath;
|
||||
//ClassLoaderUtil.log.info("The income absolute path:" + resourceAbsolutePath);
|
||||
URL resourceAbsoluteURL = new URL(resourceAbsolutePath);
|
||||
return resourceAbsoluteURL;
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
*
|
||||
* @paramsource
|
||||
*
|
||||
* @paramdest
|
||||
*
|
||||
* @return
|
||||
*
|
||||
*/
|
||||
private static int containSum(String source, String dest) {
|
||||
int containSum = 0;
|
||||
int destLength = dest.length();
|
||||
while (source.contains(dest)) {
|
||||
containSum = containSum + 1;
|
||||
source = source.substring(destLength);
|
||||
}
|
||||
return containSum;
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
*
|
||||
* @paramsource
|
||||
*
|
||||
* @paramdest
|
||||
*
|
||||
* @paramnum
|
||||
*
|
||||
* @return
|
||||
*
|
||||
*/
|
||||
private static String cutLastString(String source, String dest, int num) {
|
||||
// String cutSource=null;
|
||||
for (int i = 0; i < num; i++) {
|
||||
source = source.substring(0, source.lastIndexOf(dest, source.length() - 2) + 1);
|
||||
}
|
||||
return source;
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
*
|
||||
*
|
||||
* @paramresource
|
||||
*
|
||||
* @return
|
||||
*
|
||||
*/
|
||||
public static URL getResource(String resource) {
|
||||
//ClassLoaderUtil.log.info("The income classpath related path:" + resource);
|
||||
return ResourceLoaderUtil.getClassLoader().getResource(resource);
|
||||
}
|
||||
|
||||
public static File getFile(String resource) throws URISyntaxException {
|
||||
URL url = ResourceLoaderUtil.getClassLoader().getResource(resource);
|
||||
if(url==null)
|
||||
return null;
|
||||
File file=new File(url.getPath());
|
||||
return file;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,221 @@
|
||||
/**
|
||||
* %utils%
|
||||
* %1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common;
|
||||
|
||||
import java.io.UnsupportedEncodingException;
|
||||
import java.util.Arrays;
|
||||
import java.util.HashSet;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* @类名: StringUtils
|
||||
* @说明:
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2021年11月27日 下午4:47:54
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class StringUtils {
|
||||
|
||||
public static final String EMPTY = "";
|
||||
|
||||
/**
|
||||
* Pads a String <code>s</code> to take up <code>n</code> characters,
|
||||
* padding with char <code>c</code> on the left (<code>true</code>) or on
|
||||
* the right (<code>false</code>). Returns <code>null</code> if passed a
|
||||
* <code>null</code> String.
|
||||
**/
|
||||
public static String paddingString(String s, int n, char c, boolean paddingLeft) {
|
||||
if (s == null) {
|
||||
return s;
|
||||
}
|
||||
int add = n - s.length(); // may overflow int size... should not be a
|
||||
// problem in real life
|
||||
if (add <= 0) {
|
||||
return s;
|
||||
}
|
||||
StringBuffer str = new StringBuffer(s);
|
||||
char[] ch = new char[add];
|
||||
Arrays.fill(ch, c);
|
||||
if (paddingLeft) {
|
||||
str.insert(0, ch);
|
||||
} else {
|
||||
str.append(ch);
|
||||
}
|
||||
return str.toString();
|
||||
}
|
||||
|
||||
public static String padLeft(String s, int n, char c) {
|
||||
return paddingString(s, n, c, true);
|
||||
}
|
||||
|
||||
public static String padRight(String s, int n, char c) {
|
||||
return paddingString(s, n, c, false);
|
||||
}
|
||||
|
||||
public static boolean isInt(String s) {
|
||||
try {
|
||||
Integer.parseInt(s);
|
||||
return true;
|
||||
} catch (Exception ex) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @说明：字符串编码
|
||||
*
|
||||
* @author leehom
|
||||
* @param src
|
||||
* @return
|
||||
*
|
||||
* 异常：
|
||||
*/
|
||||
public static String strCoderGBk(String src) {
|
||||
try {
|
||||
if (src == null)
|
||||
return null;
|
||||
return new String(src.getBytes("ISO-8859-1"), "utf8");
|
||||
} catch (UnsupportedEncodingException e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public static String stringFromSet(Set<String> ss, String spliter) {
|
||||
if (ss == null || ss.size() == 0)
|
||||
return null;
|
||||
StringBuffer result = new StringBuffer();
|
||||
String[] a = ss.toArray(new String[0]);
|
||||
//
|
||||
result.append(a[0]);
|
||||
for (int i = 1; i < a.length; i++) {
|
||||
result.append(spliter);
|
||||
result.append(a[i]);
|
||||
}
|
||||
return result.toString();
|
||||
}
|
||||
|
||||
// 字符串->字符数组
|
||||
public static Set<String> str2Set(String s, String spliter) {
|
||||
if (s == null || s.equals(""))
|
||||
return null;
|
||||
String[] sa = s.split(spliter);
|
||||
List<String> sl = Arrays.asList(sa);
|
||||
Set<String> set = new HashSet<String>(sl);
|
||||
return set;
|
||||
}
|
||||
|
||||
// 字符串转换字符串数组
|
||||
public static String[] str2Strings(String s, String spliter) {
|
||||
if (s == null || s.equals(""))
|
||||
return new String[0];
|
||||
String[] sa = s.split(spliter);
|
||||
String[] ss = new String[sa.length];
|
||||
for(int i=0;i<sa.length;i++) {
|
||||
ss[i] = sa[i].trim();
|
||||
|
||||
}
|
||||
return ss;
|
||||
}
|
||||
|
||||
public static List<String> array2List(String[] strArry) {
|
||||
return Arrays.asList(strArry);
|
||||
}
|
||||
|
||||
public static List<String> str2List(String s, String spliter) {
|
||||
String[] arrayStr = str2Strings(s, spliter);
|
||||
return Arrays.asList(arrayStr);
|
||||
}
|
||||
|
||||
public static String array2Str(Object[] objs) {
|
||||
return array2Str(objs, "\"", ",");
|
||||
}
|
||||
|
||||
public static String list2Str(List<Object> objs, String quotor, String seq) {
|
||||
if (objs == null || objs.size() == 0)
|
||||
return "";
|
||||
String s = "";
|
||||
for (int i = 0; i < objs.size(); i++) {
|
||||
Object obj = objs.get(i);
|
||||
if (obj instanceof String)
|
||||
s = s + quotor + objs.get(i).toString() + quotor;
|
||||
else
|
||||
s = s + objs.get(i).toString();
|
||||
if (i != objs.size() - 1)
|
||||
s = s + seq;
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
public static String array2Str(Object[] objs, String quotor, String seq) {
|
||||
if (objs == null || objs.length == 0)
|
||||
return "";
|
||||
String s = "";
|
||||
for (int i = 0; i < objs.length; i++) {
|
||||
s = s + quotor + objs[i].toString() + quotor;
|
||||
if (i != objs.length - 1)
|
||||
s = s + seq;
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
// 去除文本中的的html标签
|
||||
public static String removeTagFromText(String content) {
|
||||
Pattern p = null;
|
||||
Matcher m = null;
|
||||
String value = null;
|
||||
// 去掉<>标签
|
||||
p = Pattern.compile("(<[^>]*>)");
|
||||
m = p.matcher(content);
|
||||
String temp = content;
|
||||
while (m.find()) {
|
||||
value = m.group(0);
|
||||
temp = temp.replace(value, "");
|
||||
}
|
||||
return temp;
|
||||
}
|
||||
|
||||
public static String escape(String target, Map<String, String> escapes) {
|
||||
Set<String> key = escapes.keySet();
|
||||
for (Iterator<String> it = key.iterator(); it.hasNext();) {
|
||||
String s = (String) it.next();
|
||||
target = target.replaceAll(s, escapes.get(s));
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
public static boolean isBlank(String str) {
|
||||
int strLen;
|
||||
if (str == null || (strLen = str.length()) == 0) {
|
||||
return true;
|
||||
}
|
||||
for (int i = 0; i < strLen; i++) {
|
||||
if ((!Character.isWhitespace(str.charAt(i)))) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
public static boolean isNotBlank(String str) {
|
||||
return !isBlank(str);
|
||||
}
|
||||
|
||||
public static boolean isNotEmpty(String str) {
|
||||
return str!=null && !"".equals(str);
|
||||
}
|
||||
|
||||
public static boolean isEmpty(String str) {
|
||||
return str==null || "".equals(str);
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,33 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common;
|
||||
|
||||
import com.alibaba.datax.core.transport.record.DefaultRecord;
|
||||
|
||||
/**
|
||||
* @类名: TableRecord
|
||||
* @说明: 表record
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月26日 下午7:31:18
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class TableRecord extends DefaultRecord {
|
||||
|
||||
/** 记录所属表*/
|
||||
private String table;
|
||||
|
||||
public String getTable() {
|
||||
return table;
|
||||
}
|
||||
|
||||
public void setTable(String table) {
|
||||
this.table = table;
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
@ -0,0 +1,48 @@
|
||||
/**
|
||||
* %基础类%
|
||||
* %1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common.serializer;
|
||||
|
||||
/**
|
||||
* @类名: Serializer
|
||||
* @说明: 序列化器接口
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2010-6-9 上午09:58:34
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
|
||||
public interface Serializer {
|
||||
|
||||
public static final String PREFIX_CDATA = "<![CDATA[";
|
||||
public static final String SUFFIX_CDATA = "]]>";
|
||||
|
||||
/**
|
||||
* @说明：序列化
|
||||
*
|
||||
* @author leehom
|
||||
* @param obj 目标对象
|
||||
* @throws SeriallizeException
|
||||
* @return void
|
||||
*
|
||||
* @异常：
|
||||
*/
|
||||
public byte[] Marshal(Object obj) throws SeriallizeException;
|
||||
|
||||
/**
|
||||
* @说明：反序列化
|
||||
*
|
||||
* @author leehong
|
||||
* @param xml XML字符串
|
||||
* typeAlias 类型关联，即XML与那个Class关联
|
||||
* @return
|
||||
* @throws SeriallizeException
|
||||
* @return Object
|
||||
*
|
||||
* @异常：
|
||||
*/
|
||||
public Object Unmarshal(byte[] xml) throws SeriallizeException;
|
||||
}
|
||||
@ -0,0 +1,27 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common.serializer;
|
||||
|
||||
|
||||
/**
|
||||
* @类名: SeriallizeException
|
||||
* @说明: 序列化基本异常
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2010-6-4 下午01:18:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class SeriallizeException extends Exception {
|
||||
|
||||
public SeriallizeException() {
|
||||
super();
|
||||
// TODO Auto-generated constructor stub
|
||||
}
|
||||
|
||||
public SeriallizeException(String message) {
|
||||
super(message);
|
||||
// TODO Auto-generated constructor stub
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
@ -0,0 +1,86 @@
|
||||
/**
|
||||
* %序列化器%
|
||||
* %1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common.serializer.xstream;
|
||||
|
||||
import java.sql.Timestamp;
|
||||
import java.util.Date;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.DateTimeUtils;
|
||||
import com.thoughtworks.xstream.converters.SingleValueConverter;
|
||||
|
||||
/**
|
||||
* @类名: DateConverter
|
||||
* @说明: 日期转换器
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2010-6-15 下午06:09:19
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class DateConverter implements SingleValueConverter {
|
||||
|
||||
/** 日期模式*/
|
||||
private String pattern;
|
||||
|
||||
/*
|
||||
* (non-Javadoc)
|
||||
*
|
||||
* @see
|
||||
* com.thoughtworks.xstream.converters.SingleValueConverter#fromString(java
|
||||
* .lang.String)
|
||||
*/
|
||||
|
||||
public Object fromString(String str) {
|
||||
return DateTimeUtils.StringToDate(str, pattern());
|
||||
}
|
||||
|
||||
/*
|
||||
* (non-Javadoc)
|
||||
*
|
||||
* @see
|
||||
* com.thoughtworks.xstream.converters.SingleValueConverter#toString(java
|
||||
* .lang.Object)
|
||||
*/
|
||||
|
||||
public String toString(Object obj) {
|
||||
return DateTimeUtils.DateToString((Date)obj, pattern());
|
||||
|
||||
}
|
||||
|
||||
private String pattern() {
|
||||
if(pattern==null)
|
||||
return DateTimeUtils.defaultPatten;
|
||||
else
|
||||
return pattern;
|
||||
}
|
||||
|
||||
/*
|
||||
* (non-Javadoc)
|
||||
*
|
||||
* @see
|
||||
* com.thoughtworks.xstream.converters.ConverterMatcher#canConvert(java.
|
||||
* lang.Class)
|
||||
*/
|
||||
|
||||
public boolean canConvert(Class type) {
|
||||
return type.equals(Date.class) || type.equals(Timestamp.class);
|
||||
}
|
||||
|
||||
/**
|
||||
* @return the pattern
|
||||
*/
|
||||
public String getPattern() {
|
||||
return pattern;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param pattern the pattern to set
|
||||
*/
|
||||
public void setPattern(String pattern) {
|
||||
this.pattern = pattern;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,172 @@
|
||||
/**
|
||||
* %serializer%
|
||||
* %1.2%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common.serializer.xstream;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.UnsupportedEncodingException;
|
||||
import java.text.MessageFormat;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.serializer.Serializer;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.serializer.SeriallizeException;
|
||||
import com.thoughtworks.xstream.XStream;
|
||||
import com.thoughtworks.xstream.converters.Converter;
|
||||
import com.thoughtworks.xstream.converters.SingleValueConverter;
|
||||
import com.thoughtworks.xstream.io.json.JettisonMappedXmlDriver;
|
||||
import com.thoughtworks.xstream.mapper.Mapper;
|
||||
|
||||
/**
|
||||
* @类名: JsonSerializerXStreamImpl
|
||||
* @说明: json序列化器XStream实现
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2010-6-9 上午10:58:40
|
||||
*
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class JsonSerializerXStreamImpl implements Serializer {
|
||||
|
||||
/** 序列化库*/
|
||||
protected XStream xs;
|
||||
/** 类型关联*/
|
||||
private Map<String, Class> typeAlias;
|
||||
/** 忽略字段*/
|
||||
private Map<String, Class> omitAlias;
|
||||
/**
|
||||
* 取消标记,
|
||||
* @see XStream.addImplicitCollection
|
||||
*/
|
||||
private Map<String, Class> imCols;
|
||||
/** 转换器*/
|
||||
private List<SingleValueConverter> converters;
|
||||
/** 转换器类型, 针对需要new的时候传入mapper参数的*/
|
||||
private List<Class<Converter>> converterClazz;
|
||||
/** 编码*/
|
||||
private String encoding = "utf-8";
|
||||
/** */
|
||||
private String header = "";
|
||||
|
||||
public JsonSerializerXStreamImpl() {
|
||||
|
||||
xs = new XStream(new JettisonMappedXmlDriver());
|
||||
xs.setMode(XStream.NO_REFERENCES);
|
||||
// 打开标注
|
||||
xs.autodetectAnnotations(true);
|
||||
|
||||
}
|
||||
|
||||
public List<SingleValueConverter> getConverters() {
|
||||
return converters;
|
||||
}
|
||||
|
||||
public void setConverters(List<SingleValueConverter> converters) {
|
||||
this.converters = converters;
|
||||
for(SingleValueConverter c : converters) {
|
||||
xs.registerConverter(c);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] Marshal(Object obj) throws SeriallizeException {
|
||||
String h = MessageFormat.format(header, new Object[]{encoding});
|
||||
try {
|
||||
byte[] objBs = xs.toXML(obj).getBytes(encoding);
|
||||
byte[] r = new byte[h.getBytes().length+objBs.length];
|
||||
System.arraycopy(h.getBytes(), 0, r, 0, h.getBytes().length);
|
||||
System.arraycopy(objBs, 0, r, h.getBytes().length, objBs.length);
|
||||
return r;
|
||||
} catch (UnsupportedEncodingException e) {
|
||||
throw new SeriallizeException("Serializer Marshal Exception: "+e.getMessage());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public Object Unmarshal(byte[] xml) throws SeriallizeException {
|
||||
return xs.fromXML(new ByteArrayInputStream(xml));
|
||||
|
||||
}
|
||||
|
||||
public Map<String, Class> getTypeAlias() {
|
||||
return typeAlias;
|
||||
}
|
||||
|
||||
public void setTypeAlias(Map<String, Class> typeAlias) {
|
||||
this.typeAlias = typeAlias;
|
||||
//
|
||||
for(String alias : typeAlias.keySet()) {
|
||||
xs.alias(alias, typeAlias.get(alias));
|
||||
}
|
||||
}
|
||||
|
||||
public String getHeader() {
|
||||
return header;
|
||||
}
|
||||
|
||||
public void setHeader(String header) {
|
||||
this.header = header;
|
||||
}
|
||||
|
||||
public Map<String, Class> getImCols() {
|
||||
return imCols;
|
||||
}
|
||||
|
||||
public void setImCols(Map<String, Class> imCols) {
|
||||
this.imCols = imCols;
|
||||
for(String imCol : imCols.keySet()) {
|
||||
xs.addImplicitCollection(imCols.get(imCol), imCol);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* @return the converterClazz
|
||||
*/
|
||||
public List<Class<Converter>> getConverterClazz() {
|
||||
return converterClazz;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param converterClazz the converterClazz to set
|
||||
*/
|
||||
public void setConverterClazz(List<Class<Converter>> converterClazzs) {
|
||||
for(Class<Converter> cc : converterClazzs) {
|
||||
Converter c;
|
||||
try {
|
||||
c = cc.getDeclaredConstructor(Mapper.class).newInstance(xs.getMapper());
|
||||
xs.registerConverter(c);
|
||||
} catch (Exception e) {
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* @return the omitAlias
|
||||
*/
|
||||
public Map<String, Class> getOmitAlias() {
|
||||
return omitAlias;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param omitAlias the omitAlias to set
|
||||
*/
|
||||
public void setOmitAlias(Map<String, Class> omitAlias) {
|
||||
this.omitAlias = omitAlias;
|
||||
}
|
||||
|
||||
public String getEncoding() {
|
||||
return encoding;
|
||||
}
|
||||
|
||||
public void setEncoding(String encoding) {
|
||||
this.encoding = encoding;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,196 @@
|
||||
/**
|
||||
* %commons%
|
||||
* %1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.common.serializer.xstream;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.UnsupportedEncodingException;
|
||||
import java.io.Writer;
|
||||
import java.text.MessageFormat;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.serializer.Serializer;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.serializer.SeriallizeException;
|
||||
import com.thoughtworks.xstream.XStream;
|
||||
import com.thoughtworks.xstream.converters.Converter;
|
||||
import com.thoughtworks.xstream.converters.SingleValueConverter;
|
||||
import com.thoughtworks.xstream.core.util.QuickWriter;
|
||||
import com.thoughtworks.xstream.io.HierarchicalStreamWriter;
|
||||
import com.thoughtworks.xstream.io.xml.PrettyPrintWriter;
|
||||
import com.thoughtworks.xstream.io.xml.StaxDriver;
|
||||
import com.thoughtworks.xstream.mapper.Mapper;
|
||||
|
||||
/**
|
||||
* @类名: SerializerXStreamImpl
|
||||
* @说明: 序列化器XStream实现
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2010-6-9 上午10:58:40
|
||||
* 修改记录：
|
||||
* *. 支持CData标签
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class XmlSerializerXStreamImpl implements Serializer {
|
||||
|
||||
/** 序列化库*/
|
||||
protected XStream xs;
|
||||
/** 类型关联*/
|
||||
private Map<String, Class> typeAlias;
|
||||
/** 忽略字段*/
|
||||
private Map<String, Class> omitAlias;
|
||||
/**
|
||||
* 取消标记,
|
||||
* @ XStream.addImplicitCollection
|
||||
*/
|
||||
private Map<String, Class> imCols;
|
||||
/** 转换器*/
|
||||
private List<SingleValueConverter> converters;
|
||||
/** 转换器类型, 针对需要new的时候传入mapper参数的*/
|
||||
private List<Class<Converter>> converterClazz;
|
||||
/** 编码*/
|
||||
private String encoding = "utf-8";
|
||||
/** xml文件头, {0}: endcoding*/
|
||||
private String header = "<?xml version=\"1.0\" encoding=\"{0}\"?>";
|
||||
|
||||
public XmlSerializerXStreamImpl() {
|
||||
|
||||
xs = new XStream(
|
||||
// StaxDriver支持namespace
|
||||
// 支持 CData
|
||||
new StaxDriver() {
|
||||
public HierarchicalStreamWriter createWriter(Writer out) {
|
||||
return new PrettyPrintWriter(out) {
|
||||
protected void writeText(QuickWriter writer, String text) {
|
||||
if(text.startsWith(PREFIX_CDATA)
|
||||
&& text.endsWith(SUFFIX_CDATA)) {
|
||||
writer.write(text);
|
||||
}else{
|
||||
super.writeText(writer, text);
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
// 打开标注
|
||||
//xs.autodetectAnnotations(true);
|
||||
|
||||
}
|
||||
|
||||
public List<SingleValueConverter> getConverters() {
|
||||
return converters;
|
||||
}
|
||||
|
||||
public void setConverters(List<SingleValueConverter> converters) {
|
||||
this.converters = converters;
|
||||
for(SingleValueConverter c : converters) {
|
||||
xs.registerConverter(c);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public byte[] Marshal(Object obj) throws SeriallizeException {
|
||||
xs.setClassLoader(obj.getClass().getClassLoader());
|
||||
String h = MessageFormat.format(header, new Object[]{encoding});
|
||||
try {
|
||||
byte[] objBs = xs.toXML(obj).getBytes(encoding);
|
||||
byte[] r = new byte[h.getBytes().length+objBs.length];
|
||||
System.arraycopy(h.getBytes(), 0, r, 0, h.getBytes().length);
|
||||
System.arraycopy(objBs, 0, r, h.getBytes().length, objBs.length);
|
||||
return r;
|
||||
} catch (UnsupportedEncodingException e) {
|
||||
throw new SeriallizeException("Serializer Marshal Exception: "+e.getMessage());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public Object Unmarshal(byte[] xml) throws SeriallizeException {
|
||||
xs.setClassLoader(this.getClass().getClassLoader());
|
||||
return xs.fromXML(new ByteArrayInputStream(xml));
|
||||
|
||||
}
|
||||
|
||||
public Map<String, Class> getTypeAlias() {
|
||||
return typeAlias;
|
||||
}
|
||||
|
||||
public void setTypeAlias(Map<String, Class> typeAlias) {
|
||||
this.typeAlias = typeAlias;
|
||||
//
|
||||
for(String alias : typeAlias.keySet()) {
|
||||
xs.alias(alias, typeAlias.get(alias));
|
||||
}
|
||||
}
|
||||
|
||||
public String getHeader() {
|
||||
return header;
|
||||
}
|
||||
|
||||
public void setHeader(String header) {
|
||||
this.header = header;
|
||||
}
|
||||
|
||||
public Map<String, Class> getImCols() {
|
||||
return imCols;
|
||||
}
|
||||
|
||||
public void setImCols(Map<String, Class> imCols) {
|
||||
this.imCols = imCols;
|
||||
for(String imCol : imCols.keySet()) {
|
||||
xs.addImplicitCollection(imCols.get(imCol), imCol);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* @return the converterClazz
|
||||
*/
|
||||
public List<Class<Converter>> getConverterClazz() {
|
||||
return converterClazz;
|
||||
}
|
||||
|
||||
/**
|
||||
* @ converterClazz the converterClazz to set
|
||||
*/
|
||||
public void setConverterClazz(List<Class<Converter>> converterClazzs) {
|
||||
for(Class<Converter> cc : converterClazzs) {
|
||||
Converter c;
|
||||
try {
|
||||
c = cc.getDeclaredConstructor(Mapper.class).newInstance(xs.getMapper());
|
||||
xs.registerConverter(c);
|
||||
} catch (Exception e) {
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* @return the omitAlias
|
||||
*/
|
||||
public Map<String, Class> getOmitAlias() {
|
||||
return omitAlias;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param omitAlias the omitAlias to set
|
||||
*/
|
||||
public void setOmitAlias(Map<String, Class> omitAlias) {
|
||||
this.omitAlias = omitAlias;
|
||||
}
|
||||
|
||||
public String getEncoding() {
|
||||
return encoding;
|
||||
}
|
||||
|
||||
public void setEncoding(String encoding) {
|
||||
this.encoding = encoding;
|
||||
}
|
||||
}
|
||||
0
rdb2graph/rdb2graph-datax/README.md
Normal file
0
rdb2graph/rdb2graph-datax/README.md
Normal file
49
rdb2graph/rdb2graph-datax/pom.xml
Normal file
49
rdb2graph/rdb2graph-datax/pom.xml
Normal file
@ -0,0 +1,49 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-parent</artifactId>
|
||||
<version>${revision}</version>
|
||||
</parent>
|
||||
<artifactId>rdb2graph-datax</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<prometheus.client.version>0.9.0</prometheus.client.version>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<!-- -->
|
||||
<dependency>
|
||||
<groupId>com.google.guava</groupId>
|
||||
<artifactId>guava</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
</dependency>
|
||||
<!-- -->
|
||||
<dependency>
|
||||
<groupId>com.alibaba.datax</groupId>
|
||||
<artifactId>datax-core</artifactId>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-jar-plugin</artifactId>
|
||||
<version>3.0.2</version>
|
||||
<configuration>
|
||||
<excludes>
|
||||
<exclude>logback.xml</exclude>
|
||||
<exclude>application.yml</exclude>
|
||||
</excludes>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
@ -0,0 +1,60 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.rdb2graph;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.alibaba.datax.core.Engine;
|
||||
|
||||
/**
|
||||
* @类名: EngineMain
|
||||
* @说明: datax engine主入口，执行两个engine
|
||||
* 1. 行数据同步
|
||||
* 2. 关系同步
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月26日 下午6:56:23
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class EngineMain {
|
||||
|
||||
public static final Logger log = LoggerFactory.getLogger(EngineMain.class);
|
||||
|
||||
public static void main(String[] args) {
|
||||
System.setProperty("datax.home", getCurrentClasspath());
|
||||
try {
|
||||
log.info("行数据同步开始>>>");
|
||||
String[] dataxArgs1 = {"-job", getCurrentClasspath() + "/job/mysql/row2node.json", "-mode", "standalone", "-jobid", "1001"};
|
||||
// Engine.entry(dataxArgs1);
|
||||
log.info("行数据同步完成.");
|
||||
} catch (Throwable e) {
|
||||
log.error(e.getMessage());
|
||||
}
|
||||
log.info("关系同步开始>>>");
|
||||
try {
|
||||
String[] dataxArgs2 = {"-job", getCurrentClasspath() + "/job/mysql/fk2rel.json", "-mode", "standalone", "-jobid", "1001"};
|
||||
Engine.entry(dataxArgs2);
|
||||
log.info("关系同步完成.");
|
||||
} catch (Throwable e) {
|
||||
log.error(e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
//
|
||||
public static String getCurrentClasspath() {
|
||||
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
|
||||
String currentClasspath = classLoader.getResource("").getPath();
|
||||
// 当前操作系统
|
||||
String osName = System.getProperty("os.name");
|
||||
if (osName.startsWith("Windows")) {
|
||||
// 删除path中最前面的/
|
||||
currentClasspath = currentClasspath.substring(1);
|
||||
}
|
||||
return currentClasspath;
|
||||
}
|
||||
}
|
||||
65
rdb2graph/rdb2graph-datax/src/main/resources/conf/core.json
Normal file
65
rdb2graph/rdb2graph-datax/src/main/resources/conf/core.json
Normal file
@ -0,0 +1,65 @@
|
||||
|
||||
{
|
||||
"entry": {
|
||||
"jvm": "-Xms4G -Xmx4G",
|
||||
"environment": {}
|
||||
},
|
||||
"common": {
|
||||
"column": {
|
||||
"datetimeFormat": "yyyy-MM-dd HH:mm:ss",
|
||||
"timeFormat": "HH:mm:ss",
|
||||
"dateFormat": "yyyy-MM-dd",
|
||||
"extraFormats":["yyyyMMdd"],
|
||||
"timeZone": "GMT+8",
|
||||
"encoding": "utf-8"
|
||||
}
|
||||
},
|
||||
"core": {
|
||||
"dataXServer": {
|
||||
"address": "http://localhost:7001/api",
|
||||
"timeout": 10000,
|
||||
"reportDataxLog": false,
|
||||
"reportPerfLog": false
|
||||
},
|
||||
"transport": {
|
||||
"channel": {
|
||||
"class": "com.alibaba.datax.core.transport.channel.memory.MemoryChannel",
|
||||
"speed": {
|
||||
"byte": -1,
|
||||
"record": -1
|
||||
},
|
||||
"flowControlInterval": 20,
|
||||
"capacity": 512,
|
||||
"byteCapacity": 67108864
|
||||
},
|
||||
"exchanger": {
|
||||
"class": "com.alibaba.datax.core.plugin.BufferedRecordExchanger",
|
||||
"bufferSize": 32
|
||||
}
|
||||
},
|
||||
"container": {
|
||||
"job": {
|
||||
"reportInterval": 10000
|
||||
},
|
||||
"taskGroup": {
|
||||
"channel": 5
|
||||
},
|
||||
"trace": {
|
||||
"enable": "false"
|
||||
}
|
||||
|
||||
},
|
||||
"statistics": {
|
||||
"collector": {
|
||||
"plugin": {
|
||||
"taskClass": "com.alibaba.datax.core.statistics.plugin.task.StdoutPluginCollector",
|
||||
"maxDirtyNumber": 10
|
||||
}
|
||||
}
|
||||
},
|
||||
"metrics": {
|
||||
"reporter": "nop",
|
||||
"url": "192.168.1.25:9091"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,46 @@
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 7
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "mysqlreader4graph",
|
||||
"parameter": {
|
||||
"username": "root",
|
||||
"password": "123456",
|
||||
"phase": "rel",
|
||||
"schemaUri": "sakila.xml",
|
||||
"column": ["*"],
|
||||
"connection": [
|
||||
{
|
||||
"querySql": [
|
||||
|
||||
],
|
||||
"jdbcUrl": [
|
||||
"jdbc:mysql://localhost:3306/sakila?remarks=true&useInformationSchema=false&serverTimezone=Asia/Shanghai"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "neo4jwriter",
|
||||
"parameter": {
|
||||
"username": "neo4j",
|
||||
"password": "neo4j1234",
|
||||
"phase": "rel",
|
||||
"database": "sakila",
|
||||
"batchSize": 100,
|
||||
"uri": "bolt://192.168.1.20:7687",
|
||||
"schemaUri": "sakila.xml",
|
||||
"writemode": "insert"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,46 @@
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 5
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "mysqlreader4graph",
|
||||
"parameter": {
|
||||
"username": "root",
|
||||
"password": "123456",
|
||||
"phase": "table",
|
||||
"schemaUri": "sakila.xml",
|
||||
"column": ["*"],
|
||||
"connection": [
|
||||
{
|
||||
"table": [
|
||||
|
||||
],
|
||||
"jdbcUrl": [
|
||||
"jdbc:mysql://localhost:3306/sakila?remarks=true&useInformationSchema=false&serverTimezone=Asia/Shanghai"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "neo4jwriter",
|
||||
"parameter": {
|
||||
"username": "neo4j",
|
||||
"password": "neo4j1234",
|
||||
"phase": "table",
|
||||
"database": "sakila",
|
||||
"batchSize": 100,
|
||||
"uri": "bolt://192.168.1.20:7687",
|
||||
"schemaUri": "sakila.xml",
|
||||
"writemode": "insert"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
76
rdb2graph/rdb2graph-datax/src/main/resources/logback.xml
Normal file
76
rdb2graph/rdb2graph-datax/src/main/resources/logback.xml
Normal file
@ -0,0 +1,76 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- scan 配置文件如果发生改变，将会被重新加载 scanPeriod 检测间隔时间-->
|
||||
<configuration scan="true" scanPeriod="60 seconds" debug="false">
|
||||
<contextName>spring-boot-log</contextName>
|
||||
<include resource="org/springframework/boot/logging/logback/base.xml"/>
|
||||
<!-- 控制台 -->
|
||||
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
|
||||
<!-- 日志格式 -->
|
||||
<encoder>
|
||||
<pattern>[%date] %clr([%level]) %clr([%logger]:%L) >>> %msg %n</pattern>
|
||||
<charset>utf-8</charset>
|
||||
</encoder>
|
||||
<!--此日志appender是为开发使用，只配置最底级别，控制台输出的日志级别是大于或等于此级别的日志信息-->
|
||||
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
|
||||
<level>INFO</level>
|
||||
</filter>
|
||||
</appender>
|
||||
<!-- 信息日志 -->
|
||||
<appender name="INFO_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
|
||||
<file>logs/info.log</file>
|
||||
<!-- 循环政策：基于时间创建日志文件 -->
|
||||
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
|
||||
<!-- 日志命名:单个文件大于128MB 按照时间+自增i 生成log文件 -->
|
||||
<fileNamePattern>logs/info-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
|
||||
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
|
||||
<maxFileSize>128MB</maxFileSize>
|
||||
</timeBasedFileNamingAndTriggeringPolicy>
|
||||
<!-- 最大保存时间：30天-->
|
||||
<maxHistory>30</maxHistory>
|
||||
</rollingPolicy>
|
||||
<append>true</append>
|
||||
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
|
||||
<pattern>[%date] %clr([%level]) %clr([%logger]:%L) >>> %msg %n</pattern>
|
||||
<charset>utf-8</charset>
|
||||
</encoder>
|
||||
<filter class="ch.qos.logback.classic.filter.LevelFilter">
|
||||
<level>info</level>
|
||||
<onMatch>ACCEPT</onMatch>
|
||||
<onMismatch>DENY</onMismatch>
|
||||
</filter>
|
||||
</appender>
|
||||
<!-- 错误日志 -->
|
||||
<appender name="ERROR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
|
||||
<file>logs/error.log</file>
|
||||
<!-- 循环政策：基于时间创建日志文件 -->
|
||||
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
|
||||
<!-- 日志命名:单个文件大于2MB 按照时间+自增i 生成log文件 -->
|
||||
<fileNamePattern>logs/error-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
|
||||
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
|
||||
<maxFileSize>2MB</maxFileSize>
|
||||
</timeBasedFileNamingAndTriggeringPolicy>
|
||||
<!-- 最大保存时间：180天-->
|
||||
<maxHistory>180</maxHistory>
|
||||
</rollingPolicy>
|
||||
<append>true</append>
|
||||
<!-- 日志格式 -->
|
||||
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
|
||||
<pattern>[%date] %clr([%level]) [%thread] %clr([%logger]:%L) >>> %msg %n</pattern>
|
||||
<charset>utf-8</charset>
|
||||
</encoder>
|
||||
<!-- 日志级别过滤器 -->
|
||||
<filter class="ch.qos.logback.classic.filter.LevelFilter">
|
||||
<!-- 过滤的级别 -->
|
||||
<level>ERROR</level>
|
||||
<!-- 匹配时的操作：接收（记录） -->
|
||||
<onMatch>ACCEPT</onMatch>
|
||||
<!-- 不匹配时的操作：拒绝（不记录） -->
|
||||
<onMismatch>DENY</onMismatch>
|
||||
</filter>
|
||||
</appender>
|
||||
<root level="INFO">
|
||||
<appender-ref ref="STDOUT" />
|
||||
<appender-ref ref="INFO_FILE" />
|
||||
<appender-ref ref="ERROR_FILE" />
|
||||
</root>
|
||||
</configuration>
|
||||
@ -0,0 +1,57 @@
|
||||
{
|
||||
"job": {
|
||||
"setting": {
|
||||
"speed": {
|
||||
"channel": 5
|
||||
}
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"reader": {
|
||||
"name": "oraclereader4graph",
|
||||
"phase": "table" // rel
|
||||
"parameter": {
|
||||
"username": "root",
|
||||
"password": "root",
|
||||
//*******多url支持分库分表******
|
||||
"connection": [
|
||||
{
|
||||
"table": [
|
||||
"table"
|
||||
],
|
||||
"jdbcUrl": [
|
||||
"jdbc:oracle:thin:@[HOST_NAME]:PORT:[DATABASE_NAME]"
|
||||
]
|
||||
}
|
||||
] 或
|
||||
"connection": [
|
||||
{
|
||||
"querySql": [
|
||||
|
||||
],
|
||||
"relFrom": [],
|
||||
"relFk": [],
|
||||
"jdbcUrl": [
|
||||
"jdbc:oracle:thin:@[HOST_NAME]:PORT:[DATABASE_NAME]"
|
||||
]
|
||||
}
|
||||
]
|
||||
//****************
|
||||
}
|
||||
},
|
||||
"writer": {
|
||||
"name": "neo4jwriter",
|
||||
"parameter": {
|
||||
"username": "root",
|
||||
"password": "root",
|
||||
"phase": # table/rel
|
||||
"database":
|
||||
"uri": "",
|
||||
"schemaUri": ""
|
||||
"writemode": "insert" // replace，update
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
1942
rdb2graph/rdb2graph-datax/src/main/resources/sakila.xml
Normal file
1942
rdb2graph/rdb2graph-datax/src/main/resources/sakila.xml
Normal file
File diff suppressed because it is too large
Load Diff
74
rdb2graph/rdb2graph-scanner/pom.xml
Normal file
74
rdb2graph/rdb2graph-scanner/pom.xml
Normal file
@ -0,0 +1,74 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-parent</artifactId>
|
||||
<version>${revision}</version>
|
||||
</parent>
|
||||
<artifactId>rdb2graph-scanner</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<!-- <neo4j-java-driver.version>4.3.6</neo4j-java-driver.version> -->
|
||||
<neo4j-java-driver.version>4.0.2</neo4j-java-driver.version>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.alibaba</groupId>
|
||||
<artifactId>druid</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>mysql</groupId>
|
||||
<artifactId>mysql-connector-java</artifactId>
|
||||
<version>${mysql.driver.version}</version>
|
||||
</dependency>
|
||||
<!-- neo4j -->
|
||||
<dependency>
|
||||
<groupId>org.neo4j.driver</groupId>
|
||||
<artifactId>neo4j-java-driver</artifactId>
|
||||
<version>${neo4j-java-driver.version}</version>
|
||||
</dependency>
|
||||
<!-- tools -->
|
||||
<dependency>
|
||||
<groupId>org.projectlombok</groupId>
|
||||
<artifactId>lombok</artifactId>
|
||||
<optional>true</optional>
|
||||
</dependency>
|
||||
<!-- -->
|
||||
<dependency>
|
||||
<groupId>com.google.guava</groupId>
|
||||
<artifactId>guava</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.oracle</groupId>
|
||||
<artifactId>ojdbc6</artifactId>
|
||||
<version>11.2.0.3</version>
|
||||
<scope>system</scope>
|
||||
<systemPath>${basedir}/src/main/lib/ojdbc6-11.2.0.3.jar</systemPath>
|
||||
</dependency>
|
||||
<!-- -->
|
||||
<dependency>
|
||||
<groupId>com.leehom.arch.datax.plugin</groupId>
|
||||
<artifactId>rdb2graph-common</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<!-- test -->
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<!-- for test -->
|
||||
<dependency>
|
||||
<groupId>com.alibaba</groupId>
|
||||
<artifactId>fastjson</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
|
||||
|
||||
</project>
|
||||
1849
rdb2graph/rdb2graph-scanner/sakila.xml
Normal file
1849
rdb2graph/rdb2graph-scanner/sakila.xml
Normal file
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,75 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner;
|
||||
|
||||
import java.sql.DatabaseMetaData;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.SQLException;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchema;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.NotNullConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.PKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.UniqueConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds.BaseDataSourceBean;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds.DruidConnectionProperties;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index.IndexMetadata;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: AbstractDbScanner
|
||||
* @说明: 抽象库扫描,
|
||||
* 获取schema，包括表，字段，索引，约束
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午2:21:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public abstract class AbstractDbScanner {
|
||||
|
||||
protected DruidConnectionProperties connProps;
|
||||
|
||||
// 扫描数据库模式
|
||||
public abstract DbSchema scan(BaseDataSourceBean dsBean) throws Exception;
|
||||
// 扫描数据库
|
||||
protected abstract DbSchema doScanDb(DatabaseMetaData dbmd, ResultSet dbrs) throws SQLException;
|
||||
// 扫描数据表
|
||||
protected abstract TableMetadata doScanTable(String dbName, DatabaseMetaData metadata, ResultSet tablers) throws SQLException;
|
||||
// 扫描表的索引和约束
|
||||
// 主键/外键/非空/唯一
|
||||
protected abstract Optional<PKConstraintMetadata> doScanPK(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException;
|
||||
// 扫描外键，在扫描表后
|
||||
protected abstract Map<String, List<FKConstraintMetadata>> doScanFK(String dbName, DatabaseMetaData dbmd, List<TableMetadata> tbmds) throws SQLException;
|
||||
protected abstract List<NotNullConstraintMetadata> doScanNN(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException;
|
||||
protected abstract List<UniqueConstraintMetadata> doScanUnique(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException;
|
||||
// 索引
|
||||
protected abstract List<IndexMetadata> doScanIndex(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException;
|
||||
|
||||
// 表扫描过滤
|
||||
protected boolean acceptTable(String tableName) {
|
||||
List<String> tbs = connProps.getTbs();
|
||||
if((tbs==null||tbs.size()==0)||tbs.contains(tableName))
|
||||
return true;
|
||||
return false;
|
||||
}
|
||||
// 查找外键引用表
|
||||
protected Optional<TableMetadata> findRefTable(String refName, List<TableMetadata> tbmds) {
|
||||
for(TableMetadata tbmd: tbmds) {
|
||||
if(refName.equals(tbmd.getName())) {
|
||||
return Optional.of(tbmd);
|
||||
}
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,322 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner;
|
||||
|
||||
import java.sql.Connection;
|
||||
import java.sql.DatabaseMetaData;
|
||||
import java.sql.JDBCType;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.SQLException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
|
||||
import javax.sql.DataSource;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchema;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchemaConsts;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.NotNullConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.PKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.UniqueConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKField;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds.BaseDataSourceBean;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index.IndexMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index.IndexType;
|
||||
|
||||
/**
|
||||
* @类名: MysqlScanner
|
||||
* @说明: 数据库扫描,
|
||||
* 表，字段，索引，约束
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午2:21:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class MysqlScanner extends AbstractDbScanner {
|
||||
|
||||
// 扫描数据库模式
|
||||
public DbSchema scan(BaseDataSourceBean dsBean) throws Exception {
|
||||
// 构建数据源，获取数据库连接
|
||||
DataSource ds = dsBean.toDataSource();
|
||||
Connection conn = ds.getConnection();
|
||||
//
|
||||
String db = connProps.getDb();
|
||||
// db元数据
|
||||
DatabaseMetaData dbmd = conn.getMetaData();
|
||||
// 模式结果集
|
||||
ResultSet schemars = dbmd.getCatalogs();
|
||||
//
|
||||
while (schemars.next()) {
|
||||
// 扫描数据库
|
||||
String dbName = schemars.getString(DbSchemaConsts.DB_NAME_INDEX);
|
||||
if(db.equals(dbName)) {
|
||||
DbSchema schema = this.doScanDb(dbmd, schemars);
|
||||
schemars.close();
|
||||
return schema;
|
||||
}
|
||||
}
|
||||
// TODO 使用统一异常设计
|
||||
throw new Exception("no db found!");
|
||||
|
||||
}
|
||||
|
||||
// 扫描数据库
|
||||
@Override
|
||||
public DbSchema doScanDb(DatabaseMetaData metadata, ResultSet dbrs) throws SQLException {
|
||||
//
|
||||
DbSchema dbmd = new DbSchema();
|
||||
//
|
||||
String dbName = dbrs.getString(DbSchemaConsts.DB_NAME_INDEX);
|
||||
dbmd.setName(dbName);
|
||||
// 表元数据, 只获取表
|
||||
ResultSet tablers = metadata.getTables(dbName, null, null, new String[] {DbSchemaConsts.TABLE_TYPE});
|
||||
List<TableMetadata> tbmds = new ArrayList<>();
|
||||
while (tablers.next()) {
|
||||
// 扫描表
|
||||
String tbName = tablers.getString(DbSchemaConsts.TABLE_NAME_INDEX);
|
||||
if(!this.acceptTable(tbName))
|
||||
continue;
|
||||
TableMetadata tbmd = this.doScanTable(dbName, metadata, tablers);
|
||||
tbmds.add(tbmd);
|
||||
|
||||
}
|
||||
dbmd.setTables(tbmds);
|
||||
tablers.close();
|
||||
// 表扫描完，扫描外键, 外键需引用其他表
|
||||
doScanFK(dbName, metadata, tbmds);
|
||||
//
|
||||
return dbmd;
|
||||
}
|
||||
|
||||
// 扫描数据表
|
||||
@Override
|
||||
public TableMetadata doScanTable(String dbName, DatabaseMetaData metadata, ResultSet tablers) throws SQLException {
|
||||
TableMetadata tbmd = new TableMetadata();
|
||||
String tbName = tablers.getString(DbSchemaConsts.TABLE_NAME_INDEX);
|
||||
tbmd.setName(tbName);
|
||||
// 扫描表字段
|
||||
List<FieldMetadata> fdmds = new ArrayList<>();
|
||||
ResultSet feildrs = metadata.getColumns(dbName, null, tbName, null);
|
||||
// 用于调试
|
||||
// ResultSetMetaData fmd = feildrs.getMetaData();
|
||||
while (feildrs.next()) {
|
||||
FieldMetadata fdmd = new FieldMetadata();
|
||||
// 字段名称/类型/注释
|
||||
String fieldName = feildrs.getString(DbSchemaConsts.FEILD_NAME_INDEX);
|
||||
fdmd.setName(fieldName);
|
||||
int type = feildrs.getInt(DbSchemaConsts.FEILD_DATA_TYPE_INDEX);
|
||||
fdmd.setType(JDBCType.valueOf(type));
|
||||
int length = feildrs.getInt(DbSchemaConsts.FEILD_LENGTH_INDEX);
|
||||
fdmd.setLength(length);
|
||||
String remark = feildrs.getString(DbSchemaConsts.FEILD_REMARK_INDEX);
|
||||
fdmd.setRemark(remark);
|
||||
fdmds.add(fdmd);
|
||||
}
|
||||
feildrs.close();
|
||||
tbmd.setFields(fdmds);
|
||||
// 扫描主键
|
||||
doScanPK(dbName, metadata, tbmd);
|
||||
// 非空约束
|
||||
doScanNN(dbName, metadata, tbmd);
|
||||
// 唯一约束
|
||||
doScanUnique(dbName, metadata, tbmd);
|
||||
// 索引
|
||||
doScanIndex(dbName, metadata, tbmd);
|
||||
//
|
||||
return tbmd;
|
||||
}
|
||||
|
||||
// 扫描约束，主键；外键；非空；唯一
|
||||
@Override
|
||||
protected Optional<PKConstraintMetadata> doScanPK(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
// 主键结果集
|
||||
ResultSet pkrs = dbmd.getPrimaryKeys(dbName, null, tbmd.getName());
|
||||
// 主键字段
|
||||
List<FieldMetadata> pkfs = new ArrayList<>();
|
||||
// 主键元数据
|
||||
// ResultSetMetaData pkmd = pkrs.getMetaData();
|
||||
while(pkrs.next()){
|
||||
String pkFn = pkrs.getString(DbSchemaConsts.PK_NAME_INDEX);
|
||||
Optional<FieldMetadata> op = tbmd.findField(pkFn);
|
||||
if(op.isPresent())
|
||||
pkfs.add(op.get());
|
||||
}
|
||||
// 无主键
|
||||
if(pkfs.size()==0) {
|
||||
return Optional.empty();
|
||||
}
|
||||
tbmd.setPk(new PKConstraintMetadata(tbmd, pkfs));
|
||||
return Optional.of(tbmd.getPk());
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Map<String, List<FKConstraintMetadata>> doScanFK(String dbName, DatabaseMetaData dbmd, List<TableMetadata> tbmds) throws SQLException {
|
||||
Map<String, List<FKConstraintMetadata>> r = new HashMap<>();
|
||||
// 循环扫描表
|
||||
for(TableMetadata tbmd: tbmds) {
|
||||
List<FKConstraintMetadata> fks = scanFKTable(dbName, dbmd, tbmd, tbmds);
|
||||
r.put(tbmd.getName(), fks);
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
/**
|
||||
* @说明：扫描表外键索引
|
||||
* 一个表可有多个外键，一个外键可有多个字段
|
||||
*
|
||||
* @author leehom
|
||||
* @param dbName
|
||||
* @param dbmd
|
||||
* @param tbmd 扫描目标表
|
||||
* @param tbmds 所有表
|
||||
* @return
|
||||
* @throws SQLException
|
||||
*
|
||||
*/
|
||||
private List<FKConstraintMetadata> scanFKTable(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd, List<TableMetadata> tbmds) throws SQLException {
|
||||
// 外键元数据结果集
|
||||
ResultSet fkrs = dbmd.getImportedKeys(dbName, dbName, tbmd.getName());
|
||||
// 主键元数据
|
||||
// ResultSetMetaData fkmd = fkrs.getMetaData();
|
||||
//
|
||||
List<FKConstraintMetadata> fkcmds = new ArrayList<>();
|
||||
// key: 外键名称
|
||||
Map<String, FKConstraintMetadata> fkcmdMap = new HashMap<>();
|
||||
while(fkrs.next()){
|
||||
// 外键名称
|
||||
String fkName = fkrs.getString(DbSchemaConsts.FK_NAME_INDEX);
|
||||
// 外键引用表
|
||||
String fkTableName = fkrs.getString(DbSchemaConsts.FK_REF_TABLE_INDEX);
|
||||
TableMetadata refTable = this.findRefTable(fkTableName, tbmds).get();
|
||||
// 主表外键字段
|
||||
String pfn = fkrs.getString(DbSchemaConsts.FK_PK_FIELD_INDEX);
|
||||
FieldMetadata pf = tbmd.findField(pfn).get();
|
||||
//
|
||||
String ffn = fkrs.getString(DbSchemaConsts.FK_REF_FIELD_INDEX);
|
||||
FieldMetadata ff = refTable.findField(ffn).get();
|
||||
//
|
||||
FKField fkf = new FKField(pf, ff);
|
||||
// 外键存在，即，多字段外键
|
||||
if(fkcmdMap.containsKey(fkName)) {
|
||||
FKConstraintMetadata fkcmd = fkcmdMap.get(fkName);
|
||||
fkcmd.getFields().add(pf);
|
||||
fkcmd.getFkFields().add(fkf);
|
||||
} else {
|
||||
List<FieldMetadata> fields = new ArrayList<>();
|
||||
fields.add(pf);
|
||||
List<FKField> fkFields = new ArrayList<>();
|
||||
fkFields.add(fkf);
|
||||
FKConstraintMetadata fkcmd = new FKConstraintMetadata(fkName, tbmd, refTable, fields, fkFields);
|
||||
fkcmds.add(fkcmd);
|
||||
fkcmdMap.put(fkName, fkcmd);
|
||||
}
|
||||
}
|
||||
// 外键放入表
|
||||
tbmd.setFks(fkcmds);
|
||||
return fkcmds;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected List<NotNullConstraintMetadata> doScanNN(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
//
|
||||
List<NotNullConstraintMetadata> nns = new ArrayList<>();
|
||||
// 字段元数据
|
||||
ResultSet feildrs = dbmd.getColumns(dbName, null, tbmd.getName(), null);
|
||||
while (feildrs.next()) {
|
||||
//
|
||||
String fieldName = feildrs.getString(DbSchemaConsts.FEILD_NAME_INDEX);
|
||||
int nullable = feildrs.getInt(DbSchemaConsts.FEILD_NULLABLE_INDEX);
|
||||
// nullable==0 0为法false，即不能为空
|
||||
if(nullable==0) {
|
||||
Optional<FieldMetadata> op = tbmd.findField(fieldName);
|
||||
if(op.isPresent()) {
|
||||
nns.add(new NotNullConstraintMetadata(tbmd, op.get()));
|
||||
}
|
||||
}
|
||||
}
|
||||
feildrs.close();
|
||||
tbmd.setNotNull(nns);
|
||||
return nns;
|
||||
}
|
||||
|
||||
// 唯一约束
|
||||
@Override
|
||||
protected List<UniqueConstraintMetadata> doScanUnique(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
List<UniqueConstraintMetadata> uqmds = new ArrayList<>();
|
||||
//
|
||||
ResultSet indexrs = dbmd.getIndexInfo(dbName, null, tbmd.getName(), false, false);
|
||||
// ResultSetMetaData indexmd = indexrs.getMetaData();
|
||||
//
|
||||
Map<String, UniqueConstraintMetadata> uqmdMap = new HashMap<>();
|
||||
while (indexrs.next()) {
|
||||
// 名称，与索引共用
|
||||
String unqName = indexrs.getString(DbSchemaConsts.INDEX_NAME_INDEX);
|
||||
// nonUnique=true, 即不唯一
|
||||
boolean nonUnique = indexrs.getBoolean(DbSchemaConsts.UNIQUE_INDEX);
|
||||
if (nonUnique)
|
||||
continue;
|
||||
String fn = indexrs.getString(DbSchemaConsts.UNIQUE_FIELD_INDEX);
|
||||
FieldMetadata field = tbmd.findField(fn).get();
|
||||
if(uqmdMap.containsKey(unqName)) {
|
||||
UniqueConstraintMetadata uqmd = uqmdMap.get(unqName);
|
||||
uqmd.getFields().add(field);
|
||||
} else {
|
||||
List<FieldMetadata> fields = new ArrayList<>();
|
||||
fields.add(field);
|
||||
UniqueConstraintMetadata uqmd = new UniqueConstraintMetadata(unqName, tbmd, fields);
|
||||
uqmds.add(uqmd);
|
||||
uqmdMap.put(unqName, uqmd);
|
||||
}
|
||||
}
|
||||
tbmd.setUnique(uqmds);
|
||||
return uqmds;
|
||||
}
|
||||
|
||||
// 索引
|
||||
@Override
|
||||
protected List<IndexMetadata> doScanIndex(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
List<IndexMetadata> indexmds = new ArrayList<>();
|
||||
//
|
||||
ResultSet indexrs = dbmd.getIndexInfo(dbName, null, tbmd.getName(), false, false);
|
||||
//
|
||||
// ResultSetMetaData indexrsmd = indexrs.getMetaData();
|
||||
// key: 索引名称
|
||||
Map<String, IndexMetadata> indexmdMap = new HashMap<>();
|
||||
while (indexrs.next()) {
|
||||
//
|
||||
String indexName = indexrs.getString(DbSchemaConsts.INDEX_NAME_INDEX);
|
||||
// 非空索引不处理，以非空约束处理
|
||||
boolean nonUnique = indexrs.getBoolean(DbSchemaConsts.UNIQUE_INDEX);
|
||||
if (!nonUnique)
|
||||
continue;
|
||||
// 索引字段
|
||||
String fn = indexrs.getString(DbSchemaConsts.INDEX_FIELD_INDEX);
|
||||
FieldMetadata field = tbmd.findField(fn).get();
|
||||
// 已有索引存储，多字段索引
|
||||
if(indexmdMap.containsKey(indexName)) {
|
||||
IndexMetadata indexmd = indexmdMap.get(indexName);
|
||||
indexmd.getFields().add(field);
|
||||
} else {
|
||||
List<FieldMetadata> fields = new ArrayList<>();
|
||||
fields.add(field);
|
||||
// 索引类型
|
||||
int indexType = indexrs.getInt(DbSchemaConsts.INDEX_TYPE_INDEX);
|
||||
IndexMetadata indexmd = new IndexMetadata(indexName, tbmd, fields, IndexType.values()[indexType]);
|
||||
indexmds.add(indexmd);
|
||||
indexmdMap.put(indexName, indexmd);
|
||||
}
|
||||
}
|
||||
tbmd.setIndexes(indexmds);
|
||||
return indexmds;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,308 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner;
|
||||
|
||||
import java.sql.Connection;
|
||||
import java.sql.DatabaseMetaData;
|
||||
import java.sql.JDBCType;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.ResultSetMetaData;
|
||||
import java.sql.SQLException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
|
||||
import javax.sql.DataSource;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchema;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchemaConsts;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.NotNullConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.PKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.UniqueConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKField;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds.BaseDataSourceBean;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index.IndexMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index.IndexType;
|
||||
|
||||
/**
|
||||
* @类名: OracleScanner
|
||||
* @说明: oracle数据库扫描,
|
||||
* 表，字段，索引，约束
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午2:21:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class OracleScanner extends AbstractDbScanner {
|
||||
|
||||
// 扫描数据库模式
|
||||
public DbSchema scan(BaseDataSourceBean dsBean) throws Exception {
|
||||
// 构建数据源，获取数据库连接
|
||||
DataSource ds = dsBean.toDataSource();
|
||||
Connection conn = ds.getConnection();
|
||||
String db = connProps.getDb();
|
||||
// db元数据
|
||||
DatabaseMetaData dbmd = conn.getMetaData();
|
||||
// 模式结果集
|
||||
ResultSet schemars = dbmd.getSchemas(); // dbmd.getCatalogs();
|
||||
//
|
||||
while (schemars.next()) {
|
||||
// 扫描数据库
|
||||
String dbName = schemars.getString(DbSchemaConsts.DB_NAME_INDEX);
|
||||
if(db.equals(dbName)) {
|
||||
DbSchema schema = this.doScanDb(dbmd, schemars);
|
||||
schemars.close();
|
||||
return schema;
|
||||
}
|
||||
}
|
||||
throw new Exception("no db found!");
|
||||
|
||||
}
|
||||
|
||||
// 扫描数据库
|
||||
@Override
|
||||
public DbSchema doScanDb(DatabaseMetaData metadata, ResultSet dbrs) throws SQLException {
|
||||
//
|
||||
DbSchema dbmd = new DbSchema();
|
||||
//
|
||||
String dbName = dbrs.getString(DbSchemaConsts.DB_NAME_INDEX);
|
||||
dbmd.setName(dbName);
|
||||
// 表元数据
|
||||
ResultSet tablers = metadata.getTables(null, dbName, null, null);
|
||||
List<TableMetadata> tbmds = new ArrayList<>();
|
||||
while (tablers.next()) {
|
||||
// 扫描表
|
||||
TableMetadata tbmd = this.doScanTable(dbName, metadata, tablers);
|
||||
tbmds.add(tbmd);
|
||||
}
|
||||
dbmd.setTables(tbmds);
|
||||
tablers.close();
|
||||
// 扫描外键
|
||||
doScanFK(dbName, metadata, tbmds);
|
||||
//
|
||||
return dbmd;
|
||||
}
|
||||
|
||||
// 扫描数据表
|
||||
@Override
|
||||
public TableMetadata doScanTable(String dbName, DatabaseMetaData metadata, ResultSet tablers) throws SQLException {
|
||||
TableMetadata tbmd = new TableMetadata();
|
||||
String tbName = tablers.getString(DbSchemaConsts.TABLE_NAME_INDEX);
|
||||
tbmd.setName(tbName);
|
||||
// 扫描表字段
|
||||
List<FieldMetadata> fdmds = new ArrayList<>();
|
||||
ResultSet feildrs = metadata.getColumns(null, dbName, tbName, null);
|
||||
//
|
||||
ResultSetMetaData fmd = feildrs.getMetaData();
|
||||
while (feildrs.next()) {
|
||||
FieldMetadata fdmd = new FieldMetadata();
|
||||
// 字段名称/类型/注释
|
||||
String fieldName = feildrs.getString(DbSchemaConsts.FEILD_NAME_INDEX);
|
||||
fdmd.setName(fieldName);
|
||||
int type = feildrs.getInt(DbSchemaConsts.FEILD_DATA_TYPE_INDEX);
|
||||
fdmd.setType(JDBCType.valueOf(type));
|
||||
int length = feildrs.getInt(DbSchemaConsts.FEILD_LENGTH_INDEX);
|
||||
fdmd.setLength(length);
|
||||
String remark = feildrs.getString(DbSchemaConsts.FEILD_REMARK_INDEX);
|
||||
fdmd.setRemark(remark);
|
||||
fdmds.add(fdmd);
|
||||
}
|
||||
feildrs.close();
|
||||
tbmd.setFields(fdmds);
|
||||
// 扫描主键
|
||||
doScanPK(dbName, metadata, tbmd);
|
||||
// 非空约束
|
||||
doScanNN(dbName, metadata, tbmd);
|
||||
//
|
||||
doScanUnique(dbName, metadata, tbmd);
|
||||
//
|
||||
doScanIndex(dbName, metadata, tbmd);
|
||||
//
|
||||
return tbmd;
|
||||
}
|
||||
|
||||
// 扫描约束，主键；外键；非空；唯一
|
||||
@Override
|
||||
protected Optional<PKConstraintMetadata> doScanPK(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
// 主键结果集
|
||||
ResultSet pkrs = dbmd.getPrimaryKeys(null, dbName, tbmd.getName());
|
||||
// 主键字段
|
||||
List<FieldMetadata> pkfs = new ArrayList<>();
|
||||
// 主键元数据
|
||||
// ResultSetMetaData pkmd = pkrs.getMetaData();
|
||||
while(pkrs.next()){
|
||||
String pkFn = pkrs.getString(DbSchemaConsts.PK_NAME_INDEX);
|
||||
Optional<FieldMetadata> op = tbmd.findField(pkFn);
|
||||
if(op.isPresent())
|
||||
pkfs.add(op.get());
|
||||
}
|
||||
// 无主键
|
||||
if(pkfs.size()==0) {
|
||||
return Optional.empty();
|
||||
}
|
||||
tbmd.setPk(new PKConstraintMetadata(tbmd, pkfs));
|
||||
return Optional.of(tbmd.getPk());
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Map<String, List<FKConstraintMetadata>> doScanFK(String dbName, DatabaseMetaData dbmd, List<TableMetadata> tbmds) throws SQLException {
|
||||
Map<String, List<FKConstraintMetadata>> r = new HashMap<>();
|
||||
// 循环扫描表
|
||||
for(TableMetadata tbmd: tbmds) {
|
||||
List<FKConstraintMetadata> fks = scanFKTable(dbName, dbmd, tbmd, tbmds);
|
||||
r.put(tbmd.getName(), fks);
|
||||
}
|
||||
return r;
|
||||
}
|
||||
|
||||
private List<FKConstraintMetadata> scanFKTable(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd, List<TableMetadata> tbmds) throws SQLException {
|
||||
// 外键元数据结果集
|
||||
ResultSet fkrs = dbmd.getImportedKeys(dbName, dbName, tbmd.getName());
|
||||
// 主键元数据
|
||||
// ResultSetMetaData fkmd = fkrs.getMetaData();
|
||||
//
|
||||
List<FKConstraintMetadata> fkcmds = new ArrayList<>();
|
||||
// key: 外键名称
|
||||
Map<String, FKConstraintMetadata> fkcmdMap = new HashMap<>();
|
||||
while(fkrs.next()){
|
||||
// 外键名称
|
||||
String fkName = fkrs.getString(DbSchemaConsts.FK_NAME_INDEX);
|
||||
// 外键引用表
|
||||
String fkTableName = fkrs.getString(DbSchemaConsts.FK_REF_TABLE_INDEX);
|
||||
TableMetadata refTable = this.findRefTable(fkTableName, tbmds).get();
|
||||
// 主表外键字段
|
||||
String pfn = fkrs.getString(DbSchemaConsts.FK_PK_FIELD_INDEX);
|
||||
FieldMetadata pf = tbmd.findField(pfn).get();
|
||||
//
|
||||
String ffn = fkrs.getString(DbSchemaConsts.FK_REF_FIELD_INDEX);
|
||||
FieldMetadata ff = refTable.findField(ffn).get();
|
||||
//
|
||||
FKField fkf = new FKField(pf, ff);
|
||||
// 外键存在，即，多字段外键
|
||||
if(fkcmdMap.containsKey(fkName)) {
|
||||
FKConstraintMetadata fkcmd = fkcmdMap.get(fkName);
|
||||
fkcmd.getFields().add(pf);
|
||||
fkcmd.getFkFields().add(fkf);
|
||||
} else {
|
||||
List<FieldMetadata> fields = new ArrayList<>();
|
||||
fields.add(pf);
|
||||
List<FKField> fkFields = new ArrayList<>();
|
||||
fkFields.add(fkf);
|
||||
FKConstraintMetadata fkcmd = new FKConstraintMetadata(fkName, tbmd, refTable, fields, fkFields);
|
||||
fkcmds.add(fkcmd);
|
||||
fkcmdMap.put(fkName, fkcmd);
|
||||
}
|
||||
}
|
||||
// 外键放入表
|
||||
tbmd.setFks(fkcmds);
|
||||
return fkcmds;
|
||||
}
|
||||
|
||||
@Override
|
||||
protected List<NotNullConstraintMetadata> doScanNN(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
//
|
||||
List<NotNullConstraintMetadata> nns = new ArrayList<>();
|
||||
// 字段元数据
|
||||
ResultSet feildrs = dbmd.getColumns(null, dbName, tbmd.getName(), null);
|
||||
while (feildrs.next()) {
|
||||
//
|
||||
String fieldName = feildrs.getString(DbSchemaConsts.FEILD_NAME_INDEX);
|
||||
int nullable = feildrs.getInt(DbSchemaConsts.FEILD_NULLABLE_INDEX);
|
||||
// nullable==0 不能为空
|
||||
if(nullable==0) {
|
||||
Optional<FieldMetadata> op = tbmd.findField(fieldName);
|
||||
if(op.isPresent()) {
|
||||
nns.add(new NotNullConstraintMetadata(tbmd, op.get()));
|
||||
}
|
||||
}
|
||||
}
|
||||
feildrs.close();
|
||||
tbmd.setNotNull(nns);
|
||||
return nns;
|
||||
}
|
||||
|
||||
// 唯一约束
|
||||
@Override
|
||||
protected List<UniqueConstraintMetadata> doScanUnique(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
List<UniqueConstraintMetadata> uqmds = new ArrayList<>();
|
||||
//
|
||||
ResultSet indexrs = dbmd.getIndexInfo(null, dbName, tbmd.getName(), false, false);
|
||||
// ResultSetMetaData indexmd = indexrs.getMetaData();
|
||||
//
|
||||
Map<String, UniqueConstraintMetadata> uqmdMap = new HashMap<>();
|
||||
while (indexrs.next()) {
|
||||
// 名称，与索引共用
|
||||
String unqName = indexrs.getString(DbSchemaConsts.INDEX_NAME_INDEX);
|
||||
if(unqName==null)
|
||||
continue;
|
||||
// nonUnique=true, 即不唯一
|
||||
boolean nonUnique = indexrs.getBoolean(DbSchemaConsts.UNIQUE_INDEX);
|
||||
if (nonUnique)
|
||||
continue;
|
||||
String fn = indexrs.getString(DbSchemaConsts.UNIQUE_FIELD_INDEX);
|
||||
FieldMetadata field = tbmd.findField(fn).get();
|
||||
if(uqmdMap.containsKey(unqName)) {
|
||||
UniqueConstraintMetadata uqmd = uqmdMap.get(unqName);
|
||||
uqmd.getFields().add(field);
|
||||
} else {
|
||||
List<FieldMetadata> fields = new ArrayList<>();
|
||||
fields.add(field);
|
||||
UniqueConstraintMetadata uqmd = new UniqueConstraintMetadata(unqName, tbmd, fields);
|
||||
uqmds.add(uqmd);
|
||||
uqmdMap.put(unqName, uqmd);
|
||||
}
|
||||
}
|
||||
tbmd.setUnique(uqmds);
|
||||
return uqmds;
|
||||
}
|
||||
|
||||
// 索引
|
||||
@Override
|
||||
protected List<IndexMetadata> doScanIndex(String dbName, DatabaseMetaData dbmd, TableMetadata tbmd) throws SQLException {
|
||||
List<IndexMetadata> indexmds = new ArrayList<>();
|
||||
//
|
||||
ResultSet indexrs = dbmd.getIndexInfo(null, dbName, tbmd.getName(), false, false);
|
||||
//
|
||||
// ResultSetMetaData indexrsmd = indexrs.getMetaData();
|
||||
// key: 索引名称
|
||||
Map<String, IndexMetadata> indexmdMap = new HashMap<>();
|
||||
while (indexrs.next()) {
|
||||
//
|
||||
String indexName = indexrs.getString(DbSchemaConsts.INDEX_NAME_INDEX);
|
||||
// 非空索引不处理，以非空约束处理
|
||||
boolean nonUnique = indexrs.getBoolean(DbSchemaConsts.UNIQUE_INDEX);
|
||||
if (!nonUnique)
|
||||
continue;
|
||||
// 索引字段
|
||||
String fn = indexrs.getString(DbSchemaConsts.INDEX_FIELD_INDEX);
|
||||
Optional<FieldMetadata> o = tbmd.findField(fn);
|
||||
if(!o.isPresent())
|
||||
continue;
|
||||
// 已有索引存储，多字段索引
|
||||
if(indexmdMap.containsKey(indexName)) {
|
||||
IndexMetadata indexmd = indexmdMap.get(indexName);
|
||||
indexmd.getFields().add(tbmd.findField(fn).get());
|
||||
} else {
|
||||
List<FieldMetadata> fields = new ArrayList<>();
|
||||
fields.add(tbmd.findField(fn).get());
|
||||
// 索引类型
|
||||
int indexType = indexrs.getInt(DbSchemaConsts.INDEX_TYPE_INDEX);
|
||||
IndexMetadata indexmd = new IndexMetadata(indexName, tbmd, fields, IndexType.values()[indexType]);
|
||||
indexmds.add(indexmd);
|
||||
indexmdMap.put(indexName, indexmd);
|
||||
}
|
||||
}
|
||||
tbmd.setIndexes(indexmds);
|
||||
return indexmds;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,23 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchema;
|
||||
|
||||
/**
|
||||
* @类名: SchemaRegistry
|
||||
* @说明: schema注册, 支持文件(序列化)，数据库，zk，nacos, redis等等
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月27日 下午3:57:27
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public interface SchemaRegistry {
|
||||
|
||||
public DbSchema registry(DbSchema schema);
|
||||
public DbSchema load(String uri);
|
||||
}
|
||||
@ -0,0 +1,66 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.StringUtils;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
|
||||
public class SchemaUtils {
|
||||
|
||||
/** 字段分割*/
|
||||
public static final String FIELD_SEQ = ",";
|
||||
|
||||
//
|
||||
public static String[] extractTableNames(String prefix, List<TableMetadata> tables) {
|
||||
if (tables == null || tables.size() == 0)
|
||||
return new String[0];
|
||||
String[] tns = new String[tables.size()];
|
||||
for (int i = 0; i < tables.size(); i++) {
|
||||
tns[i] = tables.get(i).getName();
|
||||
}
|
||||
return tns;
|
||||
}
|
||||
|
||||
// 字段名排列
|
||||
public static String extractFieldNames(List<FieldMetadata> fields) {
|
||||
return extractFieldNames(null, fields);
|
||||
}
|
||||
|
||||
public static String extractFieldNames(String prefix, List<FieldMetadata> fields) {
|
||||
return extractFieldNames(prefix, fields, FIELD_SEQ);
|
||||
}
|
||||
|
||||
public static String extractFieldNames(String prefix, List<FieldMetadata> fields, String seq) {
|
||||
if (fields == null || fields.size() == 0)
|
||||
return "";
|
||||
StringBuffer sb = new StringBuffer();
|
||||
for (int i = 0; i < fields.size(); i++) {
|
||||
String tmp = StringUtils.isNotEmpty(prefix) ? prefix+"."+fields.get(i).getName() : fields.get(i).getName() + seq;
|
||||
sb.append(tmp);
|
||||
}
|
||||
// 去掉最后seq
|
||||
sb.delete(sb.length()-seq.length(), sb.length());
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
//
|
||||
public static String extractFieldWhereNotNull(List<FieldMetadata> fields, String seq) {
|
||||
return extractFieldWhereNotNull("", fields, seq);
|
||||
|
||||
}
|
||||
public static String extractFieldWhereNotNull(String prefix, List<FieldMetadata> fields, String seq) {
|
||||
if (fields == null || fields.size() == 0)
|
||||
return "";
|
||||
StringBuffer sb = new StringBuffer();
|
||||
for (int i = 0; i < fields.size(); i++) {
|
||||
String fieldItem = fields.get(i).getName() + " is not null ";
|
||||
String tmp = StringUtils.isNotEmpty(prefix) ? prefix+"."+fieldItem : fieldItem + seq;
|
||||
sb.append(tmp);
|
||||
}
|
||||
// 去掉最后seq
|
||||
sb.delete(sb.length()-seq.length(), sb.length());
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,24 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.config;
|
||||
|
||||
import org.neo4j.driver.AuthTokens;
|
||||
import org.neo4j.driver.Driver;
|
||||
import org.neo4j.driver.GraphDatabase;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds.Neo4jDao;
|
||||
|
||||
public class Neo4jDaoConfiguration {
|
||||
|
||||
public static Driver neo4jDriver(final Neo4jDriverProperties driverProperties) {
|
||||
return GraphDatabase.driver(driverProperties.getUri(),
|
||||
AuthTokens.basic(driverProperties.getUsername(),driverProperties.getPassword()));
|
||||
}
|
||||
|
||||
public static Neo4jDao neo4jDao(final Neo4jDriverProperties driverProperties) {
|
||||
Driver driver = Neo4jDaoConfiguration.neo4jDriver(driverProperties);
|
||||
Neo4jDao neo4jDao = new Neo4jDao();
|
||||
neo4jDao.setDatabase(driverProperties.getDatabase());
|
||||
neo4jDao.setDriver(driver);
|
||||
return neo4jDao;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,24 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.config;
|
||||
|
||||
import java.net.URI;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
@Data
|
||||
public class Neo4jDriverProperties {
|
||||
|
||||
/**
|
||||
* The uri this driver should connect to. The driver supports bolt or neo4j as schemes.
|
||||
*/
|
||||
private URI uri;
|
||||
private String database;
|
||||
/**
|
||||
* The login of the user connecting to the database.
|
||||
*/
|
||||
private String username;
|
||||
|
||||
/**
|
||||
* The password of the user connecting to the database.
|
||||
*/
|
||||
private String password;
|
||||
}
|
||||
@ -0,0 +1,17 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.config;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.AbstractDbScanner;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.MysqlScanner;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.OracleScanner;
|
||||
|
||||
class ScannerConfiguration {
|
||||
|
||||
AbstractDbScanner oracleScanner() {
|
||||
return new OracleScanner();
|
||||
}
|
||||
|
||||
AbstractDbScanner mysqlScanner() {
|
||||
return new MysqlScanner();
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,53 @@
|
||||
/**
|
||||
*
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.config;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.serializer.Serializer;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.serializer.xstream.XmlSerializerXStreamImpl;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchema;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.NotNullConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.UniqueConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index.IndexMetadata;
|
||||
|
||||
/**
|
||||
* @类名: ScannerSerializerConfig
|
||||
* @说明: 序列化器配置
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2019年10月8日 下午4:21:34
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class ScannerSerializerConfig {
|
||||
|
||||
public static Serializer rdbSchemaXmlSerializer() {
|
||||
XmlSerializerXStreamImpl ser = new XmlSerializerXStreamImpl();
|
||||
// typeAlias
|
||||
Map<String, Class> typeAlias = new HashMap<String, Class>();
|
||||
// schema
|
||||
typeAlias.put("schema", DbSchema.class);
|
||||
// TableMetadata
|
||||
typeAlias.put("table", TableMetadata.class);
|
||||
// FieldMetadata
|
||||
typeAlias.put("field", FieldMetadata.class);
|
||||
// IndexMetadata
|
||||
typeAlias.put("index", IndexMetadata.class);
|
||||
// UniqueConstraintMetadata
|
||||
typeAlias.put("uq", UniqueConstraintMetadata.class);
|
||||
// FKConstraintMetadata
|
||||
typeAlias.put("fk", FKConstraintMetadata.class);
|
||||
// NotNullConstraintMetadata
|
||||
typeAlias.put("nn", NotNullConstraintMetadata.class);
|
||||
ser.setTypeAlias(typeAlias);
|
||||
return ser;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,23 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.graph;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.Graph.Edge;
|
||||
|
||||
/**
|
||||
* @类名: Callback
|
||||
* @说明: 遍历回调
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月29日 下午5:28:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public interface Callback<T, R> {
|
||||
|
||||
public R execute(Edge<T> e);
|
||||
|
||||
}
|
||||
@ -0,0 +1,111 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.graph;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: Graph
|
||||
* @说明: 图，邻接表
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月18日 上午10:32:17
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class Graph {
|
||||
|
||||
@SuppressWarnings("rawtypes")
|
||||
Vertex[] vexs; // 顶点数组
|
||||
|
||||
/* 顶点*/
|
||||
@Data
|
||||
public static class Vertex<T> {
|
||||
|
||||
public Vertex() {
|
||||
|
||||
}
|
||||
public Vertex(T data) {
|
||||
this.data = data;
|
||||
}
|
||||
@SuppressWarnings("rawtypes")
|
||||
Edge first;// 该点指向的第一条边
|
||||
boolean selfRef; // 自关联
|
||||
T data; // 节点信息
|
||||
}
|
||||
|
||||
/* 边*/
|
||||
@Data
|
||||
public static class Edge<V> {
|
||||
|
||||
public Edge(int adjVex) {
|
||||
this.adjVex = adjVex;
|
||||
}
|
||||
public Edge(int adjVex, V data) {
|
||||
this.adjVex = adjVex;
|
||||
this.data = data;
|
||||
}
|
||||
|
||||
int adjVex; // 指向的顶点
|
||||
Edge<V> next; // 指向下一个边节点
|
||||
V data;
|
||||
}
|
||||
|
||||
// 增加顶点连接
|
||||
public void link(Vertex v, Edge e) {
|
||||
if(v.first==null) {
|
||||
v.first = e;
|
||||
return;
|
||||
}
|
||||
Edge l = v.first;
|
||||
while (l.next != null) {
|
||||
l = l.next;
|
||||
}
|
||||
l.next = e;
|
||||
}
|
||||
|
||||
/*
|
||||
* 创建图 vexs 顶点；vrs 边
|
||||
*/
|
||||
public static <T, V> Graph buildGraph(Vertex<T>[] vexs, VertexRelationship<V>[] vrs) {
|
||||
Graph g = new Graph();
|
||||
g.vexs = vexs;
|
||||
for(VertexRelationship<V> vr : vrs) {
|
||||
Edge<V> e = new Edge<V>(vr.getTo(), vr.getData());
|
||||
g.link(vexs[vr.getFrom()], e);
|
||||
}
|
||||
return g;
|
||||
}
|
||||
|
||||
// 遍历，集成guava event
|
||||
public <X, R> void traversal(Callback<X, R> callback) {
|
||||
for (int i = 0; i < vexs.length; i++) {
|
||||
// 顶点
|
||||
Edge e = vexs[i].first;
|
||||
while (e != null) {
|
||||
//
|
||||
callback.execute(e);
|
||||
e = e.next;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 打印
|
||||
public void print() {
|
||||
System.out.printf("Graph:\n");
|
||||
for (int i = 0; i < vexs.length; i++) {
|
||||
// 顶点
|
||||
System.out.printf("%d(%s) ", i, vexs[i].data);
|
||||
Edge e = vexs[i].first;
|
||||
while (e != null) {
|
||||
System.out.printf("->%d(%s)", e.adjVex, vexs[e.adjVex].data);
|
||||
e = e.next;
|
||||
}
|
||||
System.out.printf("\n");
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,29 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.graph;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: VertexRelationship
|
||||
* @说明: 顶点关系
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@AllArgsConstructor
|
||||
public class VertexRelationship<V> {
|
||||
|
||||
public VertexRelationship(int from, int to) {
|
||||
this.from = from;
|
||||
this.to = to;
|
||||
}
|
||||
private int from;
|
||||
private int to;
|
||||
private V data;
|
||||
|
||||
}
|
||||
@ -0,0 +1,28 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: Neo4jSchema
|
||||
* @说明: neo4j图库模式
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public class Neo4jSchema {
|
||||
|
||||
/** 图库名称*/
|
||||
private String name;
|
||||
/** 图库描述*/
|
||||
private String remark;
|
||||
/** 图结构，邻接表*/
|
||||
private List<NodeMetadata> nodes;
|
||||
|
||||
}
|
||||
@ -0,0 +1,25 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: NodeMetadata
|
||||
* @说明: 节点元数据
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public class NodeMetadata {
|
||||
|
||||
private String label;
|
||||
private String remark;
|
||||
private List<RelationshipMetadata> relationships;
|
||||
|
||||
}
|
||||
@ -0,0 +1,25 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.type.Neo4jType;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: PropertyMetadata
|
||||
* @说明: 属性元数据
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public class PropertyMetadata {
|
||||
|
||||
private String name;
|
||||
private Neo4jType type;
|
||||
private String remark;
|
||||
|
||||
}
|
||||
@ -0,0 +1,20 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: RelationshipMetadata
|
||||
* @说明: 关系元数据
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public class RelationshipMetadata {
|
||||
|
||||
|
||||
}
|
||||
@ -0,0 +1,137 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import org.neo4j.driver.AccessMode;
|
||||
import org.neo4j.driver.Driver;
|
||||
import org.neo4j.driver.Query;
|
||||
import org.neo4j.driver.Record;
|
||||
import org.neo4j.driver.Session;
|
||||
import org.neo4j.driver.SessionConfig;
|
||||
import org.neo4j.driver.SessionConfig.Builder;
|
||||
import org.neo4j.driver.Transaction;
|
||||
import org.neo4j.driver.exceptions.Neo4jException;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.StringUtils;
|
||||
|
||||
import lombok.Getter;
|
||||
import lombok.Setter;
|
||||
|
||||
/**
|
||||
* @类名: Neo4jDao
|
||||
* @说明: neo4j dao
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月22日 上午11:13:36
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*
|
||||
* TODO
|
||||
* 1. 返回支持反射转换为bean
|
||||
*/
|
||||
public class Neo4jDao {
|
||||
|
||||
private final static Logger log = LoggerFactory.getLogger(Neo4jDao.class);
|
||||
|
||||
@Setter
|
||||
@Getter
|
||||
private String database;
|
||||
|
||||
@Setter
|
||||
private Driver driver;
|
||||
|
||||
// 查询，返回结果集
|
||||
public List<Record> runQuery(Query query) {
|
||||
|
||||
Builder builder = SessionConfig.builder().withDefaultAccessMode(AccessMode.READ);
|
||||
SessionConfig sessionConfig;
|
||||
if(StringUtils.isEmpty(database))
|
||||
sessionConfig = builder.build();
|
||||
else
|
||||
sessionConfig = builder.withDatabase(database).build();
|
||||
try (Session session = driver.session(sessionConfig)) {
|
||||
return session.run(query).list();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// 查询，返回一个结果
|
||||
public Record runQuerySingle(Query query) {
|
||||
|
||||
SessionConfig sessionConfig = SessionConfig.builder().withDefaultAccessMode(AccessMode.READ).withDatabase(database).build();
|
||||
try (Session session = driver.session(sessionConfig)) {
|
||||
return session.run(query).single();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// 执行
|
||||
public void executeQuery(Query query) {
|
||||
|
||||
SessionConfig sessionConfig = sessionConfig(AccessMode.WRITE);
|
||||
try (Session session = driver.session(sessionConfig)) {
|
||||
session.run(query);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// 事务
|
||||
public void runInTransaction(List<Query> queries) {
|
||||
|
||||
SessionConfig sessionConfig = SessionConfig.builder().withDefaultAccessMode(AccessMode.WRITE).withDatabase(database).build();
|
||||
try (Session session = driver.session(sessionConfig)) {
|
||||
try(Transaction t = session.beginTransaction()) {
|
||||
for(Query q : queries) {
|
||||
t.run(q);
|
||||
}
|
||||
t.commit();
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
public void reTryRunInTransaction(List<Query> queries, int retries) {
|
||||
int r = 0;
|
||||
while(true) {
|
||||
try {
|
||||
SessionConfig sessionConfig = SessionConfig.builder().withDefaultAccessMode(AccessMode.WRITE).withDatabase(database).build();
|
||||
try (Session session = driver.session(sessionConfig)) {
|
||||
try(Transaction t = session.beginTransaction()) {
|
||||
for(Query q : queries) {
|
||||
t.run(q);
|
||||
}
|
||||
t.commit();
|
||||
}
|
||||
}
|
||||
break;
|
||||
} catch (Neo4jException e) {
|
||||
r++;
|
||||
if(r>retries) {
|
||||
log.error("neo4j批量query异常，重试次数超限[{}]，{}", r, e.getMessage());
|
||||
break;
|
||||
}
|
||||
log.error("neo4j批量query异常[{}]，{}", r, e.getMessage());
|
||||
try {
|
||||
Thread.sleep(300);
|
||||
} catch (InterruptedException e1) {
|
||||
|
||||
}
|
||||
}
|
||||
} // end while
|
||||
|
||||
}
|
||||
|
||||
private SessionConfig sessionConfig(AccessMode mode) {
|
||||
Builder builder = SessionConfig.builder().withDefaultAccessMode(mode);
|
||||
SessionConfig sessionConfig;
|
||||
if(StringUtils.isEmpty(database))
|
||||
sessionConfig = builder.build();
|
||||
else
|
||||
sessionConfig = builder.withDatabase(database).build();
|
||||
return sessionConfig;
|
||||
}
|
||||
}
|
||||
|
||||
@ -0,0 +1,58 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds;
|
||||
|
||||
|
||||
/**
|
||||
* @类名: Neo4jQueryPattern
|
||||
* @说明: neo4j查询
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2013-4-10 下午2:31:41
|
||||
* @修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public interface Neo4jQueryPattern {
|
||||
|
||||
/** 数据库*/
|
||||
public static final String USE_DB = "USE $dbname";
|
||||
public static final String CREATE_DB = "CREATE DATABASE $dbname IF NOT EXISTS";
|
||||
|
||||
|
||||
/** node属性别名*/
|
||||
public static final String PROPERTTY_ALIAS = "n.";
|
||||
/** 关系属性别名*/
|
||||
public static final String REL_PROPERTTY_ALIAS = "r.";
|
||||
|
||||
/**
|
||||
* 创建唯一约束
|
||||
* constraintname;labelname;properties
|
||||
*/
|
||||
public static final String CREATE_UNQ_CONSTRAINT = "CREATE CONSTRAINT {0} IF NOT EXISTS "
|
||||
+ "FOR (n:{1}) "
|
||||
+ "REQUIRE ({2}) IS UNIQUE";
|
||||
|
||||
/** 创建非空约束，constraintname;labelname;properties*/
|
||||
public static final String CREATE_NN_CONSTRAINT = "CREATE CONSTRAINT {0} IF NOT EXISTS "
|
||||
+ "FOR (n:{1}) "
|
||||
+ "REQUIRE ({2}) IS NOT NULL";
|
||||
|
||||
// node 索引，0, 索引名称；1, n:Lable; 2, n.x, 节点属性
|
||||
public static final String CREATE_NODE_INDEX = "CREATE INDEX {0} IF NOT EXISTS FOR (n:{1}) ON ({2})";
|
||||
// 关系 索引， 0, 索引名称；1, n:Lable; 2, n.x, 关系属性
|
||||
public static final String CREATE_REL_INDEX = "CREATE INDEX {0} IF NOT EXISTS FOR ()-[n:{1}]-() ON ({2})";
|
||||
|
||||
// node,
|
||||
// 0, label; 1, 属性, {"key", "value"}
|
||||
public static final String CREATE_NODE = "CREATE (n:{0} {1})";
|
||||
public static final String UPDATE_NODE = "CREATE (n:{0} {1})";
|
||||
public static final String DELETE_NODE = "MATCH (n:{0} {1}) DELETE n";
|
||||
|
||||
// rel，0，关系from node label；1，关系to node label；2， where， from/to；3, 关系type；4， 关系属性
|
||||
public static final String CREATE_REL = "MATCH (a:{0}), (b:{1}) WHERE {2} "
|
||||
+ "CREATE (a)-[r:{3} {4}]->(b) ";
|
||||
|
||||
// query all
|
||||
public static final String MATCH_ALL = "MATCH (n) RETURN n";
|
||||
|
||||
|
||||
}
|
||||
@ -0,0 +1,63 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.time.LocalDateTime;
|
||||
import java.time.ZoneId;
|
||||
import java.util.Date;
|
||||
import java.util.Map;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.DateTimeUtils;
|
||||
|
||||
/**
|
||||
* @类名: ParamsUtils
|
||||
* @说明: 参数工具类
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年5月5日 下午3:48:14
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class ParamsUtils {
|
||||
|
||||
// 日期类型转换
|
||||
public static LocalDateTime dateToLocalDateTime(Date date) {
|
||||
if(date==null)
|
||||
return null;
|
||||
Instant instant = date.toInstant();
|
||||
ZoneId zoneId = ZoneId.systemDefault();
|
||||
return instant.atZone(zoneId).toLocalDateTime();
|
||||
}
|
||||
|
||||
/** */
|
||||
public static final String FIELD_SEQ = ",";
|
||||
/** */
|
||||
public static final String FIELD_QUOTA = "'";
|
||||
|
||||
// 参数字符串
|
||||
public static String params2String(Map<String, Object> params) {
|
||||
if (params == null || params.size() == 0)
|
||||
return "";
|
||||
StringBuffer sb = new StringBuffer("{");
|
||||
for(Map.Entry<String, Object> entry : params.entrySet()){
|
||||
String name = entry.getKey();
|
||||
Object v = entry.getValue();
|
||||
if(v==null)
|
||||
continue;
|
||||
// 数字，int；long；double，不需要引号
|
||||
if(Number.class.isAssignableFrom(v.getClass())) {
|
||||
sb.append(name).append(":").append(v.toString());
|
||||
} else if(v instanceof Date) {
|
||||
String dataStr = DateTimeUtils.DateToString((Date)v, DateTimeUtils.ISO8601PattenNoZone);
|
||||
sb.append(name).append(":").append("'").append(dataStr).append("'");
|
||||
} else { //
|
||||
sb.append(name).append(":").append("'").append(v.toString()).append("'");
|
||||
}
|
||||
sb.append(FIELD_SEQ);
|
||||
}
|
||||
sb.delete(sb.length()-1, sb.length());
|
||||
sb.append("}");
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,21 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds;
|
||||
|
||||
/**
|
||||
* @类名: TypeEnum
|
||||
* @说明: 参数类型枚举
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2020年9月3日 下午6:01:12s
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public enum QueryType {
|
||||
|
||||
QUERY, SINGLE_QUERY, INSERT, UPDATE, DELETE;
|
||||
|
||||
}
|
||||
@ -0,0 +1,32 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.ds;
|
||||
|
||||
import org.neo4j.driver.Query;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: QueryWrapper
|
||||
* @说明: neo4j query包装
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月22日 下午6:07:23
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@AllArgsConstructor
|
||||
public class QueryWrapper {
|
||||
|
||||
private Query query;
|
||||
private QueryType queryType;
|
||||
|
||||
public static QueryWrapper wrap(Query query, QueryType queryType) {
|
||||
return new QueryWrapper(query, queryType);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,24 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.type;
|
||||
|
||||
import java.sql.JDBCType;
|
||||
|
||||
/**
|
||||
* @类名: JdbcTypeCast
|
||||
* @说明: jdbc类型转换为neo4j类型
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2020年9月3日 下午3:42:26
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@FunctionalInterface
|
||||
public interface JdbcTypeCast {
|
||||
|
||||
public Neo4jType cast(JDBCType jdbcType);
|
||||
|
||||
}
|
||||
@ -0,0 +1,19 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.type;
|
||||
|
||||
/**
|
||||
* @类名: JdbcTypeCastRegistry
|
||||
* @说明: jdbc类型转换注册器
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2020年9月3日 下午3:42:26
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class JdbcTypeCastRegistry {
|
||||
|
||||
}
|
||||
@ -0,0 +1,78 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.neo4j.type;
|
||||
|
||||
import java.util.Date;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* @类名: TypeEnum
|
||||
* @说明: 参数类型枚举
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2020年9月3日 下午6:01:12s
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
/**
|
||||
* neo4j类型 v4.0
|
||||
* https://neo4j.com/docs/cypher-manual/4.0/syntax/values/
|
||||
*/
|
||||
public enum Neo4jType {
|
||||
|
||||
BOOLEAN("boolean", "布尔型", Boolean.class),
|
||||
SHORT("short", "短整型", Short.class),
|
||||
INT("int", "整型", Integer.class),
|
||||
LONG("long", "长整型", Long.class),
|
||||
FLOAT("float", "浮点型", Float.class),
|
||||
DOUBLE("double", "双精度浮点型", Double.class),
|
||||
STRING("string", "字符串型", String.class),
|
||||
DATE("date", "日期型", Date.class),
|
||||
ENUMCONST("enumconst", "枚举项", Enum.class),
|
||||
ENUM("enum", "枚举", Enum.class),
|
||||
// composite type
|
||||
STRINGS("strings", "字符串型数组", String[].class);
|
||||
|
||||
private static Map<String, Class<?>> nameClazzMap;
|
||||
private static Map<String, Neo4jType> typeMap;
|
||||
|
||||
static {
|
||||
nameClazzMap = new HashMap<>();
|
||||
nameClazzMap.put("BOOLEAN", Boolean.class);
|
||||
nameClazzMap.put("SHORT", Short.class);
|
||||
nameClazzMap.put("INT", Integer.class);
|
||||
nameClazzMap.put("LONG", Long.class);
|
||||
nameClazzMap.put("FLOAT", Float.class);
|
||||
nameClazzMap.put("DOUBLE", Double.class);
|
||||
nameClazzMap.put("STRING", String.class);
|
||||
nameClazzMap.put("DATE", Date.class);
|
||||
//
|
||||
typeMap = new HashMap<>();
|
||||
typeMap.put("BOOLEAN", BOOLEAN);
|
||||
typeMap.put("SHORT", SHORT);
|
||||
typeMap.put("INT", INT);
|
||||
typeMap.put("LONG", LONG);
|
||||
typeMap.put("FLOAT", FLOAT);
|
||||
typeMap.put("DOUBLE", DOUBLE);
|
||||
|
||||
}
|
||||
|
||||
// 获取类型
|
||||
public static Class<?> nameToClazz(String name) {
|
||||
return nameClazzMap.get(name);
|
||||
}
|
||||
|
||||
//
|
||||
public static Neo4jType nameToType(String name) {
|
||||
return typeMap.get(name);
|
||||
}
|
||||
|
||||
Neo4jType(String name, String alias, Class<?> clazz) {
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,106 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.Graph;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.VertexRelationship;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.Graph.Vertex;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKField;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: DbSchema
|
||||
* @说明: 数据库模式
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public class DbSchema {
|
||||
|
||||
/** */
|
||||
private String name;
|
||||
/** 注释*/
|
||||
private String remark;
|
||||
/** 数据库表*/
|
||||
private List<TableMetadata> tables;
|
||||
|
||||
/**
|
||||
* @说明：设置/去除表为连接表
|
||||
*
|
||||
* @author leehom
|
||||
* @param table
|
||||
* @param linkFromFK
|
||||
*
|
||||
*/
|
||||
public void setLinkTable(TableMetadata table, FKConstraintMetadata linkFromFK) {
|
||||
// 验证是否符合连接表， node->edge->edge
|
||||
// TODO 统一异常规范
|
||||
if(table.getFks()==null||!(table.getFks().size()==2)) {
|
||||
throw new RuntimeException("不符合连接表形态，node->edge->edge");
|
||||
}
|
||||
table.setLinkTable(true);
|
||||
table.setLinkFrom(linkFromFK);
|
||||
}
|
||||
|
||||
public void unSetLinkTable(TableMetadata table) {
|
||||
table.setLinkTable(false);
|
||||
table.setLinkFrom(null);
|
||||
}
|
||||
|
||||
// 计算表连接图
|
||||
@SuppressWarnings("unchecked")
|
||||
public Graph buildTableGraph() {
|
||||
Vertex<String>[] vxs = new Vertex[tables.size()];
|
||||
// 构建顶点
|
||||
for(int i=0;i<vxs.length;i++) {
|
||||
vxs[i] = new Vertex<>(tables.get(i).getName());
|
||||
}
|
||||
// 构建关系
|
||||
List<VertexRelationship<List<FKField>>> vrs = new ArrayList<>();
|
||||
for(int i=0;i<vxs.length;i++) {
|
||||
TableMetadata table= tables.get(i);
|
||||
List<FKConstraintMetadata> fks = table.getFks();
|
||||
int from = i;
|
||||
for(FKConstraintMetadata fk : fks) {
|
||||
//
|
||||
int to = locate(fk.getRefTable().getName(), vxs);
|
||||
// 构建关系
|
||||
VertexRelationship<List<FKField>> vr = new VertexRelationship<List<FKField>>(from, to, fk.getFkFields());
|
||||
vrs.add(vr);
|
||||
}
|
||||
|
||||
}
|
||||
// 构建表连接图
|
||||
return Graph.buildGraph(vxs, vrs.toArray(new VertexRelationship[0]));
|
||||
}
|
||||
|
||||
// 获取表邻接数组位置
|
||||
private int locate(String tableName, Vertex<String>[] vxs) {
|
||||
int l = 0;
|
||||
for(Vertex<String> vx : vxs) {
|
||||
if(tableName.equals(vx.getData()))
|
||||
break;
|
||||
l++;
|
||||
}
|
||||
return l;
|
||||
|
||||
}
|
||||
|
||||
public TableMetadata findTable(String name) {
|
||||
for(TableMetadata t : tables) {
|
||||
if(t.getName().equals(name)) {
|
||||
return t;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,60 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb;
|
||||
|
||||
|
||||
/**
|
||||
* @类名: DbSchemaConsts
|
||||
* @说明: 数据库元数据常量
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2013-4-10 下午2:31:41
|
||||
* @修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public interface DbSchemaConsts {
|
||||
|
||||
/** */
|
||||
public static final int DB_NAME_INDEX = 1;
|
||||
public static final int DB_COMMENTS_INDEX = 9;
|
||||
|
||||
public static final int TABLE_NAME_INDEX = 3;
|
||||
public static final int TABLE_COMMENTS_INDEX = 5;
|
||||
// 字段
|
||||
public static final int FEILD_NAME_INDEX = 4;
|
||||
public static final int FEILD_DATA_TYPE_INDEX = 5;
|
||||
public static final int FEILD_LENGTH_INDEX = 7;
|
||||
public static final int FEILD_NULLABLE_INDEX = 11;
|
||||
public static final int FEILD_REMARK_INDEX = 12;
|
||||
|
||||
/** 主键约束*/
|
||||
public static final int PK_NAME_INDEX = 4;
|
||||
|
||||
/** 外键约束*/
|
||||
/** */
|
||||
public static final int FK_FIELD_COUNT = 14;
|
||||
/** 主表键名称*/
|
||||
public static final int FK_PK_FIELD_INDEX = 8;
|
||||
/** 外键引用表名称*/
|
||||
public static final int FK_REF_TABLE_INDEX = 3;
|
||||
/** 外键引用字段名称*/
|
||||
public static final int FK_REF_FIELD_INDEX = 4;
|
||||
/** 外键名称*/
|
||||
public static final int FK_NAME_INDEX = 12;
|
||||
|
||||
// 唯一约束
|
||||
/** 元数据结果集字段数*/
|
||||
public static final int UNIQUE_FIELD_COUNT = 14;
|
||||
public static final int UNIQUE_INDEX = 4;
|
||||
public static final int UNIQUE_FIELD_INDEX = 9;
|
||||
|
||||
/** 索引*/
|
||||
public static final int INDEX_FIELD_COUNT = 14;
|
||||
public static final int INDEX_NAME_INDEX = 6;
|
||||
public static final int INDEX_FIELD_INDEX = 9;
|
||||
public static final int INDEX_TYPE_INDEX = 7;
|
||||
|
||||
// 表类型，
|
||||
public static final String TABLE_TYPE = "TABLE";
|
||||
|
||||
|
||||
}
|
||||
@ -0,0 +1,29 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb;
|
||||
|
||||
import java.sql.JDBCType;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: FieldMetadata
|
||||
* @说明: 字段元数据
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public class FieldMetadata {
|
||||
|
||||
private String name;
|
||||
/** 类型*/
|
||||
private JDBCType type;
|
||||
/** 长度*/
|
||||
private Integer length;
|
||||
/** 注释*/
|
||||
private String remark;
|
||||
|
||||
}
|
||||
@ -0,0 +1,35 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: MultiFieldsDbSchemaComponent
|
||||
* @说明: 多字段数据库构件
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class MultiFieldsTableComponent extends TableComponent {
|
||||
|
||||
public MultiFieldsTableComponent(TableMetadata tbmd, List<FieldMetadata> fields) {
|
||||
super(tbmd);
|
||||
this.fields = fields;
|
||||
}
|
||||
|
||||
/** 约束作用字段*/
|
||||
private List<FieldMetadata> fields;
|
||||
|
||||
}
|
||||
@ -0,0 +1,33 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: SingleFieldDbSchemaComponent
|
||||
* @说明: 单字段数据库构件
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class SingleFieldTableComponent extends TableComponent {
|
||||
|
||||
public SingleFieldTableComponent(TableMetadata tbmd, FieldMetadata field) {
|
||||
super(tbmd);
|
||||
this.field = field;
|
||||
}
|
||||
|
||||
/** 约束作用字段*/
|
||||
private FieldMetadata field;
|
||||
|
||||
}
|
||||
@ -0,0 +1,29 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: TableComponent
|
||||
* @说明: 表功能构件
|
||||
* 约束，保证数据完整性
|
||||
* 索引，加快数据搜索
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@AllArgsConstructor
|
||||
public abstract class TableComponent {
|
||||
|
||||
/** 所属表*/
|
||||
private TableMetadata tbmd;
|
||||
}
|
||||
@ -0,0 +1,70 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.NotNullConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.PKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.UniqueConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk.FKConstraintMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index.IndexMetadata;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.Getter;
|
||||
|
||||
/**
|
||||
* @类名: TableMetadata
|
||||
* @说明: 数据库表元数据
|
||||
*
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年1月7日 下午1:35:00
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
public class TableMetadata {
|
||||
|
||||
/** 表名称*/
|
||||
private String name;
|
||||
/** 注释*/
|
||||
private String remark;
|
||||
/** 字段元数据*/
|
||||
private List<FieldMetadata> fields;
|
||||
// 约束
|
||||
/** 主键*/
|
||||
private PKConstraintMetadata pk;
|
||||
/** 外键*/
|
||||
private List<FKConstraintMetadata> fks;
|
||||
/** 非空*/
|
||||
private List<NotNullConstraintMetadata> notNull;
|
||||
/** 唯一*/
|
||||
private List<UniqueConstraintMetadata> unique;
|
||||
// 索引
|
||||
private List<IndexMetadata> indexes;
|
||||
/**
|
||||
* 是否连接表
|
||||
* link->t1->t2
|
||||
*/
|
||||
private boolean isLinkTable;
|
||||
/** 连接表有效，连接关系起点外键*/
|
||||
private FKConstraintMetadata linkFrom;
|
||||
|
||||
public Optional<FieldMetadata> findField(String name) {
|
||||
for(FieldMetadata fmd : fields) {
|
||||
if(fmd.getName().equals(name))
|
||||
return Optional.of(fmd);
|
||||
}
|
||||
return Optional.empty();
|
||||
}
|
||||
|
||||
public FKConstraintMetadata findFk(String name) {
|
||||
for(FKConstraintMetadata fkmd : fks) {
|
||||
if(fkmd.getFkName().equals(name))
|
||||
return fkmd;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,37 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.SingleFieldTableComponent;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: NotNullConstraintMetadata
|
||||
* @说明: 非空约束元数据
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class NotNullConstraintMetadata extends SingleFieldTableComponent {
|
||||
|
||||
public static final String NN_NAME = "_NN";
|
||||
|
||||
public NotNullConstraintMetadata(TableMetadata tbmd, FieldMetadata field) {
|
||||
super(tbmd, field);
|
||||
this.nnName = field.getName().toUpperCase()+NN_NAME;
|
||||
}
|
||||
|
||||
private String nnName;
|
||||
|
||||
}
|
||||
@ -0,0 +1,33 @@
|
||||
/**
|
||||
*
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.MultiFieldsTableComponent;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: PKConstraintMetadata
|
||||
* @说明: 主键约束元数据
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class PKConstraintMetadata extends MultiFieldsTableComponent {
|
||||
|
||||
public PKConstraintMetadata(TableMetadata table, List<FieldMetadata> fields) {
|
||||
super(table, fields);
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,37 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.MultiFieldsTableComponent;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: UniqueConstraintMetadata
|
||||
* @说明: 唯一约束元数据, 多字段
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class UniqueConstraintMetadata extends MultiFieldsTableComponent {
|
||||
|
||||
public UniqueConstraintMetadata(String unqName, TableMetadata table, List<FieldMetadata> fields) {
|
||||
super(table, fields);
|
||||
this.unqName = unqName;
|
||||
}
|
||||
|
||||
private String unqName;
|
||||
|
||||
}
|
||||
@ -0,0 +1,43 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.MultiFieldsTableComponent;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: FKConstraintMetadata
|
||||
* @说明: 外键约束元数据
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class FKConstraintMetadata extends MultiFieldsTableComponent {
|
||||
|
||||
public FKConstraintMetadata(String name, TableMetadata table, TableMetadata refTable, List<FieldMetadata> fields, List<FKField> fkFields) {
|
||||
super(table, fields);
|
||||
this.fkName = name;
|
||||
this.refTable = refTable;
|
||||
this.fkFields = fkFields;
|
||||
}
|
||||
/** 外键名称*/
|
||||
private String fkName;
|
||||
/** */
|
||||
private TableMetadata refTable;
|
||||
/** 字段元数据*/
|
||||
private List<FKField> fkFields;
|
||||
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.constraint.fk;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: FKField
|
||||
* @说明: 外键字段
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@AllArgsConstructor
|
||||
public class FKField {
|
||||
|
||||
/** */
|
||||
private FieldMetadata field;
|
||||
/** 参考字段*/
|
||||
private FieldMetadata refField;
|
||||
|
||||
}
|
||||
@ -0,0 +1,36 @@
|
||||
/**
|
||||
* %项目描述%
|
||||
* %ver%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: DruidConnectionProperties
|
||||
* @说明: druid连接池属性
|
||||
*
|
||||
*
|
||||
* @author: leehom
|
||||
* @Date 2018年10月22日 下午4:39:19
|
||||
* @修改记录：
|
||||
*
|
||||
* @see
|
||||
*
|
||||
*
|
||||
*/
|
||||
@Data
|
||||
public class BaseConnectionProperties {
|
||||
|
||||
private Long id;
|
||||
private String name;
|
||||
/** jdbc连接url*/
|
||||
private String url;
|
||||
/** 驱动类类型*/
|
||||
private String driverClass;
|
||||
/** 账号&密码*/
|
||||
private String userName;
|
||||
private String password;
|
||||
private String remark;
|
||||
|
||||
}
|
||||
@ -0,0 +1,29 @@
|
||||
/**
|
||||
* %项目描述%
|
||||
* %ver%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds;
|
||||
|
||||
import javax.sql.DataSource;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: BasicDataSource
|
||||
* @说明: 数据源
|
||||
*
|
||||
*
|
||||
* @author: leehom
|
||||
* @Date 2018年10月22日 下午4:35:30
|
||||
* @修改记录：
|
||||
*
|
||||
* @see
|
||||
*
|
||||
*
|
||||
*/
|
||||
@Data
|
||||
public abstract class BaseDataSourceBean {
|
||||
|
||||
public abstract DataSource toDataSource() throws Exception;
|
||||
|
||||
}
|
||||
@ -0,0 +1,65 @@
|
||||
/**
|
||||
* %项目描述%
|
||||
* %ver%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: DruidConnectionProperties
|
||||
* @说明: druid连接池属性
|
||||
*
|
||||
*
|
||||
* @author: leehom
|
||||
* @Date 2018年10月22日 下午4:39:19
|
||||
* @修改记录：
|
||||
*
|
||||
* @see
|
||||
*
|
||||
*
|
||||
*/
|
||||
@Data
|
||||
public class DruidConnectionProperties {
|
||||
private Long id;
|
||||
private String name;
|
||||
/** 目标数据库*/
|
||||
private String db;
|
||||
/** 表，白名单*/
|
||||
private List<String> tbs;
|
||||
/** jdbc连接url*/
|
||||
private String url;
|
||||
/** 驱动类类型*/
|
||||
private String driverClass;
|
||||
/** 账号&密码*/
|
||||
private String userName;
|
||||
private String password;
|
||||
private String remark;
|
||||
/** 初始连接数*/
|
||||
private int initialSize = 1;
|
||||
/** 最大请求等待时间*/
|
||||
private int maxWait = 60000;
|
||||
/** 最少空闲连接数*/
|
||||
private int minIdle = 1;
|
||||
/** 检查空闲连接频率*/
|
||||
private int timeBetweenEvictionRunsMillis = 60000;
|
||||
/** 最小空闲时间, 10分钟*/
|
||||
private int minEvictableIdleTimeMillis = 600000;
|
||||
/** 最大空闲时间, 60分钟*/
|
||||
private int maxEvictableIdleTimeMillis = 3600000;
|
||||
/** */
|
||||
private String validationQuery = "select 'x'";
|
||||
/** 有效性测试*/
|
||||
private boolean testWhileIdle = true;
|
||||
/** 有效性测试*/
|
||||
private boolean testOnBorrow = false;
|
||||
/** 有效性测试*/
|
||||
private boolean testOnReturn = false;
|
||||
/** 缓存sql*/
|
||||
private boolean poolPreparedStatements = true;
|
||||
/** 最多缓存SQL数*/
|
||||
private int maxOpenPreparedStatements = 20;
|
||||
|
||||
}
|
||||
@ -0,0 +1,59 @@
|
||||
/**
|
||||
* %项目描述%
|
||||
* %ver%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds;
|
||||
|
||||
import javax.sql.DataSource;
|
||||
|
||||
import com.alibaba.druid.pool.DruidDataSource;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: DataSource
|
||||
* @说明: 数据源
|
||||
*
|
||||
*
|
||||
* @author: leehom
|
||||
* @Date 2018年10月22日 下午4:35:30
|
||||
* @修改记录：
|
||||
*
|
||||
* @see
|
||||
*
|
||||
*
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class DruidDataSourceBean extends BaseDataSourceBean {
|
||||
|
||||
/** 连接属性*/
|
||||
private DruidConnectionProperties connProps;
|
||||
|
||||
@Override
|
||||
public DataSource toDataSource() throws Exception {
|
||||
DruidDataSource druidDataSource = new DruidDataSource();
|
||||
druidDataSource.setDriverClassName(connProps.getDriverClass());
|
||||
druidDataSource.setUrl(connProps.getUrl());
|
||||
druidDataSource.setUsername(connProps.getUserName());
|
||||
druidDataSource.setPassword(connProps.getPassword());
|
||||
druidDataSource.setInitialSize(connProps.getInitialSize());
|
||||
druidDataSource.setMinIdle(connProps.getMinIdle());
|
||||
// druidDataSource.setMaxActive(Integer.parseInt(propertyResolver.getProperty("maxActive")));
|
||||
druidDataSource.setMaxWait(connProps.getMaxWait());
|
||||
druidDataSource.setTimeBetweenEvictionRunsMillis(connProps.getTimeBetweenEvictionRunsMillis());
|
||||
druidDataSource.setMinEvictableIdleTimeMillis(connProps.getMinEvictableIdleTimeMillis());
|
||||
druidDataSource.setValidationQuery(connProps.getValidationQuery());
|
||||
druidDataSource.setTestWhileIdle(connProps.isTestWhileIdle());
|
||||
druidDataSource.setTestOnBorrow(connProps.isTestOnBorrow());
|
||||
druidDataSource.setTestOnReturn(connProps.isTestOnReturn());
|
||||
druidDataSource.setPoolPreparedStatements(connProps.isPoolPreparedStatements());
|
||||
druidDataSource.setMaxPoolPreparedStatementPerConnectionSize(connProps.getMaxOpenPreparedStatements());
|
||||
druidDataSource.setKeepAlive(true);
|
||||
druidDataSource.init();
|
||||
return druidDataSource;
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,39 @@
|
||||
/**
|
||||
* %%
|
||||
* %%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.FieldMetadata;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.MultiFieldsTableComponent;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.TableMetadata;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.EqualsAndHashCode;
|
||||
|
||||
/**
|
||||
* @类名: IndexMetadata
|
||||
* @说明: 索引元数据，多字段
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月16日 下午6:03:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@EqualsAndHashCode(callSuper=false)
|
||||
public class IndexMetadata extends MultiFieldsTableComponent {
|
||||
|
||||
public IndexMetadata(String name, TableMetadata table, List<FieldMetadata> fields, IndexType type) {
|
||||
super(table, fields);
|
||||
this.name = name;
|
||||
this.type = type;
|
||||
}
|
||||
|
||||
private String name;
|
||||
private IndexType type;
|
||||
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index;
|
||||
|
||||
|
||||
/**
|
||||
* @类名: IndexOrder
|
||||
* @说明:
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月19日 下午12:06:13
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public enum IndexOrder {
|
||||
|
||||
ASC("ASC"), DESC("DESC");
|
||||
|
||||
String value;
|
||||
|
||||
private IndexOrder(String value) {
|
||||
this.value = value;
|
||||
}
|
||||
|
||||
public String value() {
|
||||
return this.value;
|
||||
}
|
||||
|
||||
public String getValue() {
|
||||
return this.value;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,40 @@
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index;
|
||||
|
||||
|
||||
/**
|
||||
* @类名: IndexMode
|
||||
* @说明:
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月19日 下午12:06:13
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
|
||||
//public static final short tableIndexStatistic = 0;
|
||||
//
|
||||
//public static final short tableIndexClustered = 1;
|
||||
//
|
||||
//public static final short tableIndexHashed = 2;
|
||||
//
|
||||
//public static final short tableIndexOther = 3;
|
||||
|
||||
public enum IndexType {
|
||||
|
||||
STATISTIC("statistic"), CLUSTERED("clustered"), HASHED("hashed"), OTHER("other");
|
||||
|
||||
private String value;
|
||||
|
||||
private IndexType(String value) {
|
||||
this.value = value;
|
||||
}
|
||||
|
||||
public String value() {
|
||||
return this.value;
|
||||
}
|
||||
|
||||
public String getValue() {
|
||||
return this.value;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,25 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: NodeIndexMetadata
|
||||
* @说明:
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月25日 上午9:57:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@AllArgsConstructor
|
||||
public class NodeIndexMetadata {
|
||||
|
||||
private IndexMetadata indexmd;
|
||||
}
|
||||
@ -0,0 +1,25 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.index;
|
||||
|
||||
import lombok.AllArgsConstructor;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* @类名: RelIndexMetadata
|
||||
* @说明:
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月25日 上午9:57:20
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
@Data
|
||||
@AllArgsConstructor
|
||||
public class RelIndexMetadata {
|
||||
|
||||
private IndexMetadata indexmd;
|
||||
}
|
||||
76
rdb2graph/rdb2graph-scanner/src/main/resources/logback.xml
Normal file
76
rdb2graph/rdb2graph-scanner/src/main/resources/logback.xml
Normal file
@ -0,0 +1,76 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- scan 配置文件如果发生改变，将会被重新加载 scanPeriod 检测间隔时间-->
|
||||
<configuration scan="true" scanPeriod="60 seconds" debug="false">
|
||||
<contextName>spring-boot-log</contextName>
|
||||
<include resource="org/springframework/boot/logging/logback/base.xml"/>
|
||||
<!-- 控制台 -->
|
||||
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
|
||||
<!-- 日志格式 -->
|
||||
<encoder>
|
||||
<pattern>[%date] %clr([%level]) %clr([%logger]:%L) >>> %msg %n</pattern>
|
||||
<charset>utf-8</charset>
|
||||
</encoder>
|
||||
<!--此日志appender是为开发使用，只配置最底级别，控制台输出的日志级别是大于或等于此级别的日志信息-->
|
||||
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
|
||||
<level>INFO</level>
|
||||
</filter>
|
||||
</appender>
|
||||
<!-- 信息日志 -->
|
||||
<appender name="INFO_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
|
||||
<file>logs/info.log</file>
|
||||
<!-- 循环政策：基于时间创建日志文件 -->
|
||||
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
|
||||
<!-- 日志命名:单个文件大于128MB 按照时间+自增i 生成log文件 -->
|
||||
<fileNamePattern>logs/info-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
|
||||
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
|
||||
<maxFileSize>128MB</maxFileSize>
|
||||
</timeBasedFileNamingAndTriggeringPolicy>
|
||||
<!-- 最大保存时间：30天-->
|
||||
<maxHistory>30</maxHistory>
|
||||
</rollingPolicy>
|
||||
<append>true</append>
|
||||
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
|
||||
<pattern>[%date] %clr([%level]) %clr([%logger]:%L) >>> %msg %n</pattern>
|
||||
<charset>utf-8</charset>
|
||||
</encoder>
|
||||
<filter class="ch.qos.logback.classic.filter.LevelFilter">
|
||||
<level>info</level>
|
||||
<onMatch>ACCEPT</onMatch>
|
||||
<onMismatch>DENY</onMismatch>
|
||||
</filter>
|
||||
</appender>
|
||||
<!-- 错误日志 -->
|
||||
<appender name="ERROR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
|
||||
<file>logs/error.log</file>
|
||||
<!-- 循环政策：基于时间创建日志文件 -->
|
||||
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
|
||||
<!-- 日志命名:单个文件大于2MB 按照时间+自增i 生成log文件 -->
|
||||
<fileNamePattern>logs/error-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
|
||||
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
|
||||
<maxFileSize>2MB</maxFileSize>
|
||||
</timeBasedFileNamingAndTriggeringPolicy>
|
||||
<!-- 最大保存时间：180天-->
|
||||
<maxHistory>180</maxHistory>
|
||||
</rollingPolicy>
|
||||
<append>true</append>
|
||||
<!-- 日志格式 -->
|
||||
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
|
||||
<pattern>[%date] %clr([%level]) [%thread] %clr([%logger]:%L) >>> %msg %n</pattern>
|
||||
<charset>utf-8</charset>
|
||||
</encoder>
|
||||
<!-- 日志级别过滤器 -->
|
||||
<filter class="ch.qos.logback.classic.filter.LevelFilter">
|
||||
<!-- 过滤的级别 -->
|
||||
<level>ERROR</level>
|
||||
<!-- 匹配时的操作：接收（记录） -->
|
||||
<onMatch>ACCEPT</onMatch>
|
||||
<!-- 不匹配时的操作：拒绝（不记录） -->
|
||||
<onMismatch>DENY</onMismatch>
|
||||
</filter>
|
||||
</appender>
|
||||
<root level="INFO">
|
||||
<appender-ref ref="STDOUT" />
|
||||
<appender-ref ref="INFO_FILE" />
|
||||
<appender-ref ref="ERROR_FILE" />
|
||||
</root>
|
||||
</configuration>
|
||||
1849
rdb2graph/rdb2graph-scanner/src/main/resources/sakila.xml
Normal file
1849
rdb2graph/rdb2graph-scanner/src/main/resources/sakila.xml
Normal file
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,48 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.Graph;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.VertexRelationship;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.Graph.Vertex;
|
||||
|
||||
/**
|
||||
* @类名: GraphTest
|
||||
* @说明: 图测试
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2022年4月18日 上午10:50:36
|
||||
* 修改记录：
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class GraphTest {
|
||||
/**
|
||||
测试图：
|
||||
a->b
|
||||
->c
|
||||
d->e
|
||||
f
|
||||
*/
|
||||
public static void main(String[] args) {
|
||||
// 构建顶点
|
||||
Vertex<String> va = new Vertex<>("a"); // 0
|
||||
Vertex<String> vb = new Vertex<>("b"); // 1
|
||||
Vertex<String> vc = new Vertex<>("c"); // 2
|
||||
Vertex<String> vd = new Vertex<>("d"); // 3
|
||||
Vertex<String> ve = new Vertex<>("e"); // 4
|
||||
Vertex<String> vf = new Vertex<>("f"); // 5
|
||||
// 构建关系
|
||||
VertexRelationship vr1 = new VertexRelationship(0, 1);
|
||||
VertexRelationship vr2 = new VertexRelationship(0, 2);
|
||||
VertexRelationship vr3 = new VertexRelationship(3, 4);
|
||||
//
|
||||
Graph g = Graph.buildGraph(new Vertex[] {va,vb,vc,vd,ve,vf}, new VertexRelationship[] {vr1,vr2,vr3});
|
||||
// 打印
|
||||
g.print();
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,89 @@
|
||||
/**
|
||||
* %datax-graph%
|
||||
* %v1.0%
|
||||
*/
|
||||
package com.leehom.arch.datax.plugin.rdb2graph.scanner;
|
||||
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.common.BeanUtils;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.graph.Graph;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.DbSchema;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds.BaseDataSourceBean;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds.DruidConnectionProperties;
|
||||
import com.leehom.arch.datax.plugin.rdb2graph.scanner.rdb.ds.DruidDataSourceBean;
|
||||
|
||||
|
||||
/**
|
||||
* @类名: MysqlScannerTest
|
||||
* @说明: 数据库扫描引测试
|
||||
*
|
||||
* @author leehom
|
||||
* @Date 2012-8-29 下午5:55:46
|
||||
*
|
||||
*
|
||||
* @see
|
||||
*/
|
||||
public class MysqlScannerTest {
|
||||
|
||||
private AbstractDbScanner mysqlScanner;
|
||||
private DruidDataSourceBean dsBean;
|
||||
|
||||
// db: sakila
|
||||
// url: jdbc:mysql://localhost:3306/?remarks=true&useInformationSchema=false&serverTimezone=Asia/Shanghai
|
||||
// tbs:
|
||||
// username: root
|
||||
// password: root
|
||||
// driverClass: com.mysql.cj.jdbc.Driver
|
||||
// filters: stat
|
||||
// maxActive: 20
|
||||
// initialSize: 1
|
||||
// maxWait: 60000
|
||||
// minIdle: 1
|
||||
// timeBetweenEvictionRunsMillis: 60000
|
||||
// minEvictableIdleTimeMillis: 300000
|
||||
// validationQuery: select 'x'
|
||||
// testWhileIdle: true
|
||||
// testOnBorrow: false
|
||||
// testOnReturn: false
|
||||
// poolPreparedStatements: true
|
||||
// maxOpenPreparedStatements: 20
|
||||
|
||||
@Before
|
||||
public void init() {
|
||||
dsBean = new DruidDataSourceBean();
|
||||
DruidConnectionProperties connProps = new DruidConnectionProperties();
|
||||
connProps.setDb("sakila");
|
||||
connProps.setUrl("jdbc:mysql://localhost:3306/?remarks=true&useInformationSchema=false&serverTimezone=Asia/Shanghai");
|
||||
connProps.setUserName("root");
|
||||
connProps.setPassword("123456");
|
||||
connProps.setDriverClass("com.mysql.jdbc.Driver");
|
||||
//
|
||||
dsBean.setConnProps(connProps);
|
||||
//
|
||||
mysqlScanner = new MysqlScanner();
|
||||
mysqlScanner.setConnProps(connProps);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testDbScan() throws Exception {
|
||||
DbSchema dbmds = mysqlScanner.scan(dsBean);
|
||||
BeanUtils.printBean(dbmds);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetLinkTable() throws Exception {
|
||||
DbSchema dbmds = mysqlScanner.scan(dsBean);
|
||||
BeanUtils.printBean(dbmds);
|
||||
}
|
||||
|
||||
// 构建表连接图测试
|
||||
@Test
|
||||
public void testBuildGraph() throws Exception {
|
||||
DbSchema dbSch = mysqlScanner.scan(dsBean);
|
||||
Graph g = dbSch.buildTableGraph();
|
||||
g.print();
|
||||
}
|
||||
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user