github/sqoop
5
0
Fork 0
mirror of https://github.com/apache/sqoop.git synced 2025-05-03 22:02:31 +08:00
Code Issues Projects Releases Wiki Activity
76 Commits 21 Branches 30 Tags 24 MiB
0fde7dff8d
Commit Graph

76 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Bayer
0fde7dff8d Hive DDL generator uses INTEGER when it means INT. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149878 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:40 +00:00
Andrew Bayer
9374ed8398 Add support for mysqlimport-based export jobs. 
Using --direct in conjunction with --export-dir on a MySQL database will use
mysqlimport to emit the data to the database.
DirectMySQLManager now creates instances of MySQLExportJob.
src/test/.../MySQLUtils is renamed to MySQLTestUtils to avoid conflict with
src/java/.../MySQLUtils added by this patch.
MySQLUtils contains methods factored out of import-specific code for sharing
with exports.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149877 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:40 +00:00
Andrew Bayer
f5fc5a5f7e Enable cobertura coverage reporting. 
To use this, run:
ant cobertura -Dcobertura.home=/path/to/cobertura

That will run the standard smoke tests. To get a full report, you'll
need to re-run it on the thirdparty tests:

ant cobertura -Dcobertura.home=/path/to/cobertura -Dthirdparty=true \
    -Dsqoop.thirdparty.lib.dir=/path/to/jdbc/drivers

The results of both runs will be combined.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149876 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:40 +00:00
Andrew Bayer
abbfab1941 Refactor ExportJob to facilitate multiple kinds of export jobs. 
Add 'JobBase' which attempts to unify ExportJob with ImportJobBase.
ImportJobBase and ExportJobBase override job-type specific behavior.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149875 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:40 +00:00
Andrew Bayer
8147d262d8 Enable findbugs on build and fix all warnings. 
Some spurious warnings (and inconsequential warnings in test code)
have been disabled by src/test/findbugsExcludeFile.xml.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149874 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:39 +00:00
Andrew Bayer
7214230695 Support BINARY, VARBINARY, and RAW (Oracle) types 
Added support for importing byte array columns as BytesWritable.
Tested with MySQL, Oracle, HSQLDB.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149873 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:39 +00:00
Andrew Bayer
2240be8807 Cache connections to Oracle across ConnManagers. 
OracleManager now caches Connection instances for subsequent OracleManager
instances.
Refactored uses of ConnManager to call close() before discarding them.
This allows the Oracle JUnit tests to sleep less frequently to wait for Oracle
to reap closed server-side connection resources, improving Oracle test speed
by 50%.

Sleeping cannot be fully eliminated because MapReduce-side Connections are not
governed by this caching mechanism.

Also added some debugging advice re. this topic to OracleManagerTest's comment.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149872 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:39 +00:00
Andrew Bayer
43f9e2f2b0 Added unit test to check network setup needed for postgres tests. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149871 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:38 +00:00
Andrew Bayer
6cbe7572e9 If --hive-import and --generate-only are specified, create a ddl script file. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149870 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:38 +00:00
Andrew Bayer
df76e995e8 Users can precisely control export parallelism. 
Uses CombineFileInputFormat to run exports over a target number
of mappers independent of the number of input files.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149869 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:38 +00:00
Andrew Bayer
a0dd7e7490 Changed license headers to reference Cloudera instead of the ASF. 
Adds NOTICE.txt file

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149868 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:36 +00:00
Andrew Bayer
b72a134b52 Show imported row count after job completion. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149867 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:36 +00:00
Andrew Bayer
bb29ce9492 Support for CLOB/BLOB data in external files. 
CLOB/BLOB data may now be stored in additional files in HDFS which are
accessible through streams if the data cannot be fully materialized in RAM.
Adds tests for external large objects.
Refactored large object loading into the map() method from readFields().

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149866 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:35 +00:00
Andrew Bayer
32a67749b1 Run mysqldump in map task instead of on the client. 
Major refactoring of DataDrivenImportJob to support mysqldump in mappers.
ImportJobBase added below DataDrivenImportJob.
MySQLDumpImportJob added on top of ImportJobBase.
LocalMySQLManager -> renamed to -> DirectMySQLManager now just runs MysqldumpIJ.
MySQLDumpImportJob configures MySQLDumpMapper to run mysqldump instances on
multiple nodes and is split-aware (via MySQLDumpInputFormat).
TestImportJob works with new ImportJobBase framework.
Added test that imports a subset of columns in mysql imports.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149865 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:35 +00:00
Andrew Bayer
bdd405f756 Improve batch testrunner support for third-party tests. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149864 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:35 +00:00
Andrew Bayer
14c9e0bf88 Use DataDrivenDBInputFormat with Oracle. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149863 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:34 +00:00
Andrew Bayer
7482c71cf9 Initial support for CLOB/BLOB types 
Tests pass in Oracle and MySQL compatibility suites

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149862 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:34 +00:00
Andrew Bayer
71b01cdb7f Compilation, dependency resolution, and tests pass. 
Modified build.xml to run without Hadoop's build-contrib wrapper.
Added MRUnit jar from Hadoop MapReduce (not exposed via mvn).
Added 'package' and 'tar' targets for redistribution.
Added ivy settings files for direct dependencies.
Added gitignores where appropriate.
Move documentation from /doc to /src/docs.
Add LICENSE.txt.
Move readme.txt to README.txt.
Provide more fine-grained control of third-party redistributables
via 'redist' ivy configuration.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149861 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:34 +00:00
Andrew Bayer
af2ec3a03f MAPREDUCE-1445. Refactor Sqoop tests to support better ConnManager testing. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149860 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
c5c613cb92 MAPREDUCE-1444. Sqoop ConnManager instances can leak Statement objects. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149859 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
61d5da2500 MAPREDUCE-1467. Add a --verbose flag to Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149858 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
6a215d0fbc MAPREDUCE-1469. Sqoop should disable speculative execution in export. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149857 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
a625fd478c MAPREDUCE-1341. Sqoop should have an option to create hive tables and skip the table import step. Contributed by Leonid Furman. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149856 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:32 +00:00
Andrew Bayer
de836d714a MAPREDUCE-1356. Allow user-specified hive table name in sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149855 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:32 +00:00
Andrew Bayer
69f04fff8b MAPREDUCE-1395. Sqoop does not check return value of Job.waitForCompletion(). Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149854 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
40da832916 MAPREDUCE-1394. Sqoop generates incorrect URIs in paths sent to Hive. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149853 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
e7a8e519f3 MAPREDUCE-1327. Fix Sqoop handling of Oracle timezone with timestamp data 
types in import. Contributed by Leonid Furman

From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149852 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
6174268d28 MAPREDUCE-1313. Fix NPE in Sqoop when table with null fields uses escape 
during import. Contributed by Aaron Kimball

From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149851 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
c7f64e4f8c MAPREDUCE-1212. Mapreduce contrib project ivy dependencies are not included in binary target. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149850 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
05929a73e5 MAPREDUCE-1310. CREATE TABLE statements for Hive do not correctly specify delimiters. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149849 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:30 +00:00
Andrew Bayer
b74084196f MAPREDUCE-1235. Fix a MySQL timestamp incompatibility in Sqoop. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149848 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:30 +00:00
Andrew Bayer
681461461a MAPREDUCE-1174. Sqoop improperly handles table/column names which are reserved sql words. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149847 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:30 +00:00
Andrew Bayer
8b483c6ded MAPREDUCE-1146. Sqoop dependencies break Eclipse build on Linux. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149846 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:29 +00:00
Andrew Bayer
4686e0fee7 MAPREDUCE-1148. SQL identifiers are a superset of Java identifiers. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149845 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:29 +00:00
Andrew Bayer
12827a1765 MAPREDUCE-1224. Calling "SELECT t.* from <table> AS t" to get meta information is too expensive for big tables. Contributed by Spencer Ho. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149844 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:29 +00:00
Andrew Bayer
8e813b95a4 MAPREDUCE-1168. Export data to databases via Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149843 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
2312eeff5a MAPREDUCE-1239. Fix contrib components build dependencies 
From: Giridharan Kesavan <gkesavan@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149842 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
b451865d53 HADOOP-5107. Use Maven ant tasks to publish artifacts. (Giridharan Kesavan 
via omalley)

From: Owen O'Malley <omalley@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149841 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
ec8f687d97 MAPREDUCE-1169. Improvements to mysqldump use in Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149840 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
84adbeea26 MAPREDUCE-1037. Continue running contrib tests if Sqoop tests fail. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149839 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:27 +00:00
Andrew Bayer
a0229d9738 MAPREDUCE-1036. Document Sqoop API. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149838 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:27 +00:00
Andrew Bayer
9afc7a8aee MAPREDUCE-1069. Implement Sqoop API refactoring. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149837 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:27 +00:00
Andrew Bayer
93dab4998e MAPREDUCE-1017. Compression and output splitting for Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149836 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
8dcc748853 MAPREDUCE-947. Added commitJob and abortJob apis to OutputCommitter. Enhanced FileOutputCommitter to create a _SUCCESS file for successful jobs. Contributed by Amar Kamat & Jothi Padmanabhan. 
From: Arun Murthy <acmurthy@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149835 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
4b193ec488 MAPREDUCE-906. Update Sqoop documentation. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149834 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
4a689cf491 MAPREDUCE-999. Improve Sqoop test speed and refactor tests. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149833 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
940db9f6ce MAPREDUCE-980. Modify JobHistory to use Avro for serialization. 
From: Douglass Cutting <cutting@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149832 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00
Andrew Bayer
fd4f25baa4 MAPREDUCE-923. Sqoop classpath breaks for jar files with a plus sign in their names. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149831 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00
Andrew Bayer
7e3284691d MAPREDUCE-849. Rename configuration properties. Contributed by Amareshwari Sriramadasu. 
From: Sharad Agarwal <sharad@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149830 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00
Andrew Bayer
8071445237 MAPREDUCE-777. Brand new apis to track and query jobs as a replacement for JobClient. Contributed by Amareshwari Sriramadasu. 
From: Arun Murthy <acmurthy@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149829 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00