5
0
mirror of https://github.com/apache/sqoop.git synced 2025-05-18 01:40:43 +08:00
Commit Graph

68 Commits

Author SHA1 Message Date
Andrew Bayer
f4eba3dd8c SQOOP-32. sqoop-export documentation in SqoopUserGuide refers to 'import-args'.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149936 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:04 +00:00
Andrew Bayer
7e0ccb4008 SQOOP-14. Allow Sqoop to import data into HBase.
Added FieldMappable and FieldMapProcessor interfaces.
Added ProcessingException class.
Added NullOutputCommitter class.
SqoopRecord now has delegate() method which calls a FieldMapProcessor.
ClassWriter now generates getFieldMap() method for SqoopRecords.
Added HBasePutProcessor to transform SqoopRecords into Put commands,
implementing FieldMapProcessor.
Added PutTransformer interface class and ToStringPutTransformer implementation.
Added DelegatingOutputFormat that uses a FieldMapProcessor.
Added HBase deps to build.xml via hbase.home property.
Added HBase, ZooKeeper to the dependency net added by configure-sqoop.
Added HBaseImportJob, HBaseImportMapper.
ImportJobBase now has jobSetup() step executed just before job submission.
ImportJobContext now holds a reference to the ConnManager.
DataDrivenImportJob retrieves ConnManager from ImportJobContext, it no longer
creates a new one.
Added HBase table import configuration parameters to SqoopOptions, ImportTool.
SqlManager.importQuery() needs to set ConnManager in ImportJobContext.
Added HBase import user documentation.
Described PutTransformer API in developer docs.
Added HBase unit tests.
Added ANT_ARGUMENTS env variable to Hudson test scripts to allow freeform parameters.
Added HBASE_HOME and ZOOKEEPER_HOME variables to hudson scripts.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149935 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:03 +00:00
Andrew Bayer
6ea1665716 SQOOP-4. Add update-mode export process.
Add --update-key argument to sqoop-export tool.
Refactor ExportOutputFormat into AsyncSqlOutputFormat.
Added UpdateOutputFormat.
ClassWriter now allows alternate serialization order in database write() method.
SqoopOptions holds column list for alternate db serialization order.
Added TestExportUpdate unit test battery.
AsyncSqlRecordWriter now allows "batch" execution mode.
Updated documentation for export updates.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149933 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:02 +00:00
Andrew Bayer
b0cd632b58 SQOOP-3. Add ability to import arbitrary queries.
Add ConnManager.importQuery() API.
Change BaseSqoopTool.DEBUG_SQL_CMD_ARG to SQL_QUERY_ARG to reflect
the broader applicability of the argument.
Change 'debugSqlCmd' member of SqoopOptions to 'sqlQuery'.
CompilationManager now sets jar name based on specified class name.
Add tests for query support.
Add documentation for query-based import.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149932 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:02 +00:00
Andrew Bayer
e75b1fe94f Add release audit capability.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149923 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:59 +00:00
Andrew Bayer
3edcf20d33 Add webdocs target to build release docs website.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149922 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:59 +00:00
Andrew Bayer
1d80c4ded7 Minor manpage fixes.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149915 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:57 +00:00
Andrew Bayer
1f9ca86a2f Add documentation for --append and --target-dir.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149914 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:57 +00:00
Andrew Bayer
74d034cf36 Fix documentation format for use with asciidoc 8.5.
Comment blocks (for the license header) must appear at the end
of a man page asciidoc file; they cannot appear near the top.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149910 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:55 +00:00
Andrew Bayer
42875119dd Move org.apache.hadoop.sqoop to com.cloudera.sqoop
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149906 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:52 +00:00
Andrew Bayer
0abcdb0af9 Add new manual pages for all Sqoop tools.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149903 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:50 +00:00
Andrew Bayer
faabc51a90 Rewrite documentation for updated Sqoop API
Add documentation for all SqoopTool implementations.
Add database compatibility notes.
Separate user guide from the develpment guide.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149902 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:50 +00:00
Andrew Bayer
9374ed8398 Add support for mysqlimport-based export jobs.
Using --direct in conjunction with --export-dir on a MySQL database will use
mysqlimport to emit the data to the database.
DirectMySQLManager now creates instances of MySQLExportJob.
src/test/.../MySQLUtils is renamed to MySQLTestUtils to avoid conflict with
src/java/.../MySQLUtils added by this patch.
MySQLUtils contains methods factored out of import-specific code for sharing
with exports.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149877 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:40 +00:00
Andrew Bayer
6cbe7572e9 If --hive-import and --generate-only are specified, create a ddl script file.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149870 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:38 +00:00
Andrew Bayer
a0dd7e7490 Changed license headers to reference Cloudera instead of the ASF.
Adds NOTICE.txt file

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149868 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:36 +00:00
Andrew Bayer
bb29ce9492 Support for CLOB/BLOB data in external files.
CLOB/BLOB data may now be stored in additional files in HDFS which are
accessible through streams if the data cannot be fully materialized in RAM.
Adds tests for external large objects.
Refactored large object loading into the map() method from readFields().

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149866 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:35 +00:00
Andrew Bayer
14c9e0bf88 Use DataDrivenDBInputFormat with Oracle.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149863 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:34 +00:00
Andrew Bayer
71b01cdb7f Compilation, dependency resolution, and tests pass.
Modified build.xml to run without Hadoop's build-contrib wrapper.
Added MRUnit jar from Hadoop MapReduce (not exposed via mvn).
Added 'package' and 'tar' targets for redistribution.
Added ivy settings files for direct dependencies.
Added gitignores where appropriate.
Move documentation from /doc to /src/docs.
Add LICENSE.txt.
Move readme.txt to README.txt.
Provide more fine-grained control of third-party redistributables
via 'redist' ivy configuration.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149861 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:34 +00:00