github/DataX
5
0
Fork 0
mirror of https://github.com/alibaba/DataX.git synced 2025-05-03 06:21:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

support for "splitFactor" parameterization

Browse Source
This commit is contained in:
daizihao 2020-11-17 19:12:24 +08:00
parent dabb9a2fc3
commit 3fb7fe7451
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms/reader/util/ReaderSplitUtil.java
View File

@ -70,7 +70,7 @@ public final class ReaderSplitUtil {
//考虑其他比率数字?(splitPk is null, 忽略此长尾) //考虑其他比率数字?(splitPk is null, 忽略此长尾)
//eachTableShouldSplittedNumber = eachTableShouldSplittedNumber * 5; //eachTableShouldSplittedNumber = eachTableShouldSplittedNumber * 5;
//为避免导入hive小文件 默认基数为5，也就是channel配置几个就是几个task,可以通过 splitFactor 配置基数 //为避免导入hive小文件 默认基数为5，可以通过 splitFactor 配置基数
// 最终task数为(channel/tableNum)向上取整*splitFactor // 最终task数为(channel/tableNum)向上取整*splitFactor
Integer splitFactor = originalSliceConfig.getInt(Key.SPLIT_FACTOR, Constant.SPLIT_FACTOR); Integer splitFactor = originalSliceConfig.getInt(Key.SPLIT_FACTOR, Constant.SPLIT_FACTOR);
eachTableShouldSplittedNumber = eachTableShouldSplittedNumber * splitFactor; eachTableShouldSplittedNumber = eachTableShouldSplittedNumber * splitFactor;