github/sqoop
5
0
Fork 0
mirror of https://github.com/apache/sqoop.git synced 2025-05-04 04:49:43 +08:00
Code Issues Projects Releases Wiki Activity
67 Commits 21 Branches 30 Tags 24 MiB
df76e995e8
Commit Graph

67 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Bayer
df76e995e8 Users can precisely control export parallelism. 
Uses CombineFileInputFormat to run exports over a target number
of mappers independent of the number of input files.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149869 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:38 +00:00
Andrew Bayer
a0dd7e7490 Changed license headers to reference Cloudera instead of the ASF. 
Adds NOTICE.txt file

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149868 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:36 +00:00
Andrew Bayer
b72a134b52 Show imported row count after job completion. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149867 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:36 +00:00
Andrew Bayer
bb29ce9492 Support for CLOB/BLOB data in external files. 
CLOB/BLOB data may now be stored in additional files in HDFS which are
accessible through streams if the data cannot be fully materialized in RAM.
Adds tests for external large objects.
Refactored large object loading into the map() method from readFields().

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149866 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:35 +00:00
Andrew Bayer
32a67749b1 Run mysqldump in map task instead of on the client. 
Major refactoring of DataDrivenImportJob to support mysqldump in mappers.
ImportJobBase added below DataDrivenImportJob.
MySQLDumpImportJob added on top of ImportJobBase.
LocalMySQLManager -> renamed to -> DirectMySQLManager now just runs MysqldumpIJ.
MySQLDumpImportJob configures MySQLDumpMapper to run mysqldump instances on
multiple nodes and is split-aware (via MySQLDumpInputFormat).
TestImportJob works with new ImportJobBase framework.
Added test that imports a subset of columns in mysql imports.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149865 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:35 +00:00
Andrew Bayer
bdd405f756 Improve batch testrunner support for third-party tests. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149864 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:35 +00:00
Andrew Bayer
14c9e0bf88 Use DataDrivenDBInputFormat with Oracle. 
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149863 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:34 +00:00
Andrew Bayer
7482c71cf9 Initial support for CLOB/BLOB types 
Tests pass in Oracle and MySQL compatibility suites

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149862 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:34 +00:00
Andrew Bayer
71b01cdb7f Compilation, dependency resolution, and tests pass. 
Modified build.xml to run without Hadoop's build-contrib wrapper.
Added MRUnit jar from Hadoop MapReduce (not exposed via mvn).
Added 'package' and 'tar' targets for redistribution.
Added ivy settings files for direct dependencies.
Added gitignores where appropriate.
Move documentation from /doc to /src/docs.
Add LICENSE.txt.
Move readme.txt to README.txt.
Provide more fine-grained control of third-party redistributables
via 'redist' ivy configuration.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149861 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:34 +00:00
Andrew Bayer
af2ec3a03f MAPREDUCE-1445. Refactor Sqoop tests to support better ConnManager testing. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149860 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
c5c613cb92 MAPREDUCE-1444. Sqoop ConnManager instances can leak Statement objects. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149859 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
61d5da2500 MAPREDUCE-1467. Add a --verbose flag to Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149858 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
6a215d0fbc MAPREDUCE-1469. Sqoop should disable speculative execution in export. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149857 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:33 +00:00
Andrew Bayer
a625fd478c MAPREDUCE-1341. Sqoop should have an option to create hive tables and skip the table import step. Contributed by Leonid Furman. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149856 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:32 +00:00
Andrew Bayer
de836d714a MAPREDUCE-1356. Allow user-specified hive table name in sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149855 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:32 +00:00
Andrew Bayer
69f04fff8b MAPREDUCE-1395. Sqoop does not check return value of Job.waitForCompletion(). Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149854 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
40da832916 MAPREDUCE-1394. Sqoop generates incorrect URIs in paths sent to Hive. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149853 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
e7a8e519f3 MAPREDUCE-1327. Fix Sqoop handling of Oracle timezone with timestamp data 
types in import. Contributed by Leonid Furman

From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149852 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
6174268d28 MAPREDUCE-1313. Fix NPE in Sqoop when table with null fields uses escape 
during import. Contributed by Aaron Kimball

From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149851 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
c7f64e4f8c MAPREDUCE-1212. Mapreduce contrib project ivy dependencies are not included in binary target. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149850 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:31 +00:00
Andrew Bayer
05929a73e5 MAPREDUCE-1310. CREATE TABLE statements for Hive do not correctly specify delimiters. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149849 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:30 +00:00
Andrew Bayer
b74084196f MAPREDUCE-1235. Fix a MySQL timestamp incompatibility in Sqoop. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149848 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:30 +00:00
Andrew Bayer
681461461a MAPREDUCE-1174. Sqoop improperly handles table/column names which are reserved sql words. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149847 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:30 +00:00
Andrew Bayer
8b483c6ded MAPREDUCE-1146. Sqoop dependencies break Eclipse build on Linux. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149846 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:29 +00:00
Andrew Bayer
4686e0fee7 MAPREDUCE-1148. SQL identifiers are a superset of Java identifiers. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149845 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:29 +00:00
Andrew Bayer
12827a1765 MAPREDUCE-1224. Calling "SELECT t.* from <table> AS t" to get meta information is too expensive for big tables. Contributed by Spencer Ho. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149844 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:29 +00:00
Andrew Bayer
8e813b95a4 MAPREDUCE-1168. Export data to databases via Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149843 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
2312eeff5a MAPREDUCE-1239. Fix contrib components build dependencies 
From: Giridharan Kesavan <gkesavan@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149842 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
b451865d53 HADOOP-5107. Use Maven ant tasks to publish artifacts. (Giridharan Kesavan 
via omalley)

From: Owen O'Malley <omalley@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149841 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
ec8f687d97 MAPREDUCE-1169. Improvements to mysqldump use in Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149840 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:28 +00:00
Andrew Bayer
84adbeea26 MAPREDUCE-1037. Continue running contrib tests if Sqoop tests fail. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149839 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:27 +00:00
Andrew Bayer
a0229d9738 MAPREDUCE-1036. Document Sqoop API. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149838 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:27 +00:00
Andrew Bayer
9afc7a8aee MAPREDUCE-1069. Implement Sqoop API refactoring. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149837 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:27 +00:00
Andrew Bayer
93dab4998e MAPREDUCE-1017. Compression and output splitting for Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149836 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
8dcc748853 MAPREDUCE-947. Added commitJob and abortJob apis to OutputCommitter. Enhanced FileOutputCommitter to create a _SUCCESS file for successful jobs. Contributed by Amar Kamat & Jothi Padmanabhan. 
From: Arun Murthy <acmurthy@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149835 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
4b193ec488 MAPREDUCE-906. Update Sqoop documentation. Contributed by Aaron Kimball 
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149834 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
4a689cf491 MAPREDUCE-999. Improve Sqoop test speed and refactor tests. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149833 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:26 +00:00
Andrew Bayer
940db9f6ce MAPREDUCE-980. Modify JobHistory to use Avro for serialization. 
From: Douglass Cutting <cutting@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149832 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00
Andrew Bayer
fd4f25baa4 MAPREDUCE-923. Sqoop classpath breaks for jar files with a plus sign in their names. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149831 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00
Andrew Bayer
7e3284691d MAPREDUCE-849. Rename configuration properties. Contributed by Amareshwari Sriramadasu. 
From: Sharad Agarwal <sharad@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149830 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00
Andrew Bayer
8071445237 MAPREDUCE-777. Brand new apis to track and query jobs as a replacement for JobClient. Contributed by Amareshwari Sriramadasu. 
From: Arun Murthy <acmurthy@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149829 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:25 +00:00
Andrew Bayer
37cadedc8f MAPREDUCE-907. Sqoop should use more intelligent splits. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149828 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:24 +00:00
Andrew Bayer
629559c3fb MAPREDUCE-157. Refactor job history APIs and change the history format to JSON. Contributed by Jothi Padmanabhan. 
From: Sharad Agarwal <sharad@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149827 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:24 +00:00
Andrew Bayer
b34b2acc10 MAPREDUCE-938. Postgresql support for Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149826 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:24 +00:00
Andrew Bayer
2bbf349f4c MAPREDUCE-918. Test hsqldb server should be memory-only. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149825 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:24 +00:00
Andrew Bayer
9e8955ede1 MAPREDUCE-876. Sqoop import of large tables can time out. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149824 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:23 +00:00
Andrew Bayer
89241b591a MAPREDUCE-750. Extensible ConnManager factory API. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149823 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:23 +00:00
Andrew Bayer
e7cf0ed69c MAPREDUCE-749. Make Sqoop unit tests more Hudson-friendly. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149822 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:23 +00:00
Andrew Bayer
a87e225891 MAPREDUCE-877. Added avro as a dependency to contrib ivy settings. Contributed by Tsz Wo (Nicholas) Sze. 
From: Hemanth Yamijala <yhemanth@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149821 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:23 +00:00
Andrew Bayer
6cf87c5147 MAPREDUCE-789. Oracle support for Sqoop. Contributed by Aaron Kimball. 
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149820 13f79535-47bb-0310-9956-ffa450edef68
 2011-07-22 20:03:23 +00:00