This change updates the OracleManager to use catalog views for resolving
the necessary metadata.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1150006 13f79535-47bb-0310-9956-ffa450edef68
Sqoop user guide inaccurately claims that Hive does not support escaping
of characters. This change updates the user guide to fix this and make the
claim based on the current capabilities of Hive.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1150004 13f79535-47bb-0310-9956-ffa450edef68
This change introduces the ability to use a staging table for intermediate
storage during execution for regular export jobs in insert mode. This allows
all of exported data to first be populated in the staging table and then
inserted into the destination table in a single transaction. Thus if a failure
were to occur during export, it is less likely to corrupt the destination
table data. Moreover, the staging table is emptied before the export
job starts populating it, which ensures that re-running the job does not
require any special clean up.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1150002 13f79535-47bb-0310-9956-ffa450edef68
This change allows Sqoop to load options from an options file. An
options file is specified using --options-file. All options that
are otherwise specified on the command line should be specified
in this file in the order they would otherwise appear on the command
line. Options files can contain empty lines and comments for
readability. More than one options file may be used for a single
tool invocation if so preferred. Leading and trailing spaces are
ignored unless they appear within single or double quotes. Quoted
options extending into multiple lines are not supported.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149999 13f79535-47bb-0310-9956-ffa450edef68
This fix allows the user to optionally specify different null
representations. It addresses both the import and export use
cases, in addition to both string and non-string column types.
From: Ahmed Radwan <ahmed@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149996 13f79535-47bb-0310-9956-ffa450edef68
Documentation updated to explicitly state the limitations of the
free-form query based import facility. Also, fixed a documentation
example that was missing the 'WHERE $CONDITIONS' clause.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149995 13f79535-47bb-0310-9956-ffa450edef68
This fix allows the user to optionally specify the connection
manager class to be used, instead of inferring it from the
connection string.
From: Ahmed Radwan <ahmed@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149993 13f79535-47bb-0310-9956-ffa450edef68
This change removes the MySQL JDBC driver distribution that was
bundled with Sqoop previously. This is done to make sure that the
Sqoop distribution is completely Apache 2.0 compliant.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149987 13f79535-47bb-0310-9956-ffa450edef68
Adds 'merge' tool.
Adds MergeJob, Merge*Mapper, MergeReducer.
Merge-specific arguments added to SqoopOptions, BaseSqoopTool.
Add TestMerge to test that this tool functions as expected.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149980 13f79535-47bb-0310-9956-ffa450edef68
Eliminates "webdocs" target from build, merged with "docs" target.
Web-based format for document delivery is used for local documentation as well.
Links to release notes, API documentation added to documentation homepage.
Release notes are generated for non-SNAPSHOT builds as part of "package" step.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149971 13f79535-47bb-0310-9956-ffa450edef68
Updated the "supported databases" section of the user guide to reflect the
current driver installation process.
Updated "support" section of the user guide to mention current issue tracker.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149969 13f79535-47bb-0310-9956-ffa450edef68
Added manual pages and user guide sections for sqoop-job and sqoop-metastore.
Updated sqoop-import documentation to describe incremental imports.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149968 13f79535-47bb-0310-9956-ffa450edef68
Modify sqoop-eval to use generic execute() instead of
executeQuery()/executeUpdate().
The --update argument to sqoop-eval is unnecessary and has been removed.
Modify ResultSetPrinter to properly display left border on output.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149963 13f79535-47bb-0310-9956-ffa450edef68
Added --update argument to sqoop-eval tool, which runs
the user's statement in update mode rather than select mode.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149952 13f79535-47bb-0310-9956-ffa450edef68
Added FieldMappable and FieldMapProcessor interfaces.
Added ProcessingException class.
Added NullOutputCommitter class.
SqoopRecord now has delegate() method which calls a FieldMapProcessor.
ClassWriter now generates getFieldMap() method for SqoopRecords.
Added HBasePutProcessor to transform SqoopRecords into Put commands,
implementing FieldMapProcessor.
Added PutTransformer interface class and ToStringPutTransformer implementation.
Added DelegatingOutputFormat that uses a FieldMapProcessor.
Added HBase deps to build.xml via hbase.home property.
Added HBase, ZooKeeper to the dependency net added by configure-sqoop.
Added HBaseImportJob, HBaseImportMapper.
ImportJobBase now has jobSetup() step executed just before job submission.
ImportJobContext now holds a reference to the ConnManager.
DataDrivenImportJob retrieves ConnManager from ImportJobContext, it no longer
creates a new one.
Added HBase table import configuration parameters to SqoopOptions, ImportTool.
SqlManager.importQuery() needs to set ConnManager in ImportJobContext.
Added HBase import user documentation.
Described PutTransformer API in developer docs.
Added HBase unit tests.
Added ANT_ARGUMENTS env variable to Hudson test scripts to allow freeform parameters.
Added HBASE_HOME and ZOOKEEPER_HOME variables to hudson scripts.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149935 13f79535-47bb-0310-9956-ffa450edef68
Add --update-key argument to sqoop-export tool.
Refactor ExportOutputFormat into AsyncSqlOutputFormat.
Added UpdateOutputFormat.
ClassWriter now allows alternate serialization order in database write() method.
SqoopOptions holds column list for alternate db serialization order.
Added TestExportUpdate unit test battery.
AsyncSqlRecordWriter now allows "batch" execution mode.
Updated documentation for export updates.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149933 13f79535-47bb-0310-9956-ffa450edef68
Add ConnManager.importQuery() API.
Change BaseSqoopTool.DEBUG_SQL_CMD_ARG to SQL_QUERY_ARG to reflect
the broader applicability of the argument.
Change 'debugSqlCmd' member of SqoopOptions to 'sqlQuery'.
CompilationManager now sets jar name based on specified class name.
Add tests for query support.
Add documentation for query-based import.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149932 13f79535-47bb-0310-9956-ffa450edef68
Comment blocks (for the license header) must appear at the end
of a man page asciidoc file; they cannot appear near the top.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149910 13f79535-47bb-0310-9956-ffa450edef68
Add documentation for all SqoopTool implementations.
Add database compatibility notes.
Separate user guide from the develpment guide.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149902 13f79535-47bb-0310-9956-ffa450edef68
Using --direct in conjunction with --export-dir on a MySQL database will use
mysqlimport to emit the data to the database.
DirectMySQLManager now creates instances of MySQLExportJob.
src/test/.../MySQLUtils is renamed to MySQLTestUtils to avoid conflict with
src/java/.../MySQLUtils added by this patch.
MySQLUtils contains methods factored out of import-specific code for sharing
with exports.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149877 13f79535-47bb-0310-9956-ffa450edef68
CLOB/BLOB data may now be stored in additional files in HDFS which are
accessible through streams if the data cannot be fully materialized in RAM.
Adds tests for external large objects.
Refactored large object loading into the map() method from readFields().
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149866 13f79535-47bb-0310-9956-ffa450edef68
Modified build.xml to run without Hadoop's build-contrib wrapper.
Added MRUnit jar from Hadoop MapReduce (not exposed via mvn).
Added 'package' and 'tar' targets for redistribution.
Added ivy settings files for direct dependencies.
Added gitignores where appropriate.
Move documentation from /doc to /src/docs.
Add LICENSE.txt.
Move readme.txt to README.txt.
Provide more fine-grained control of third-party redistributables
via 'redist' ivy configuration.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149861 13f79535-47bb-0310-9956-ffa450edef68
