This change allows Sqoop to load options from an options file. An
options file is specified using --options-file. All options that
are otherwise specified on the command line should be specified
in this file in the order they would otherwise appear on the command
line. Options files can contain empty lines and comments for
readability. More than one options file may be used for a single
tool invocation if so preferred. Leading and trailing spaces are
ignored unless they appear within single or double quotes. Quoted
options extending into multiple lines are not supported.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149999 13f79535-47bb-0310-9956-ffa450edef68
This fix allows the user to optionally specify different null
representations. It addresses both the import and export use
cases, in addition to both string and non-string column types.
From: Ahmed Radwan <ahmed@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149996 13f79535-47bb-0310-9956-ffa450edef68
This change removes the ShimLoader and various Shim classes such as CDH3Shim
etc. It introduces a couple of new classes - ConfigurationConstants and
ConfigurationHelper - that provide a unique place for articulating interface
related details such as configuration keys that can likely change from version
to version of Hadoop.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149994 13f79535-47bb-0310-9956-ffa450edef68
This fix allows the user to optionally specify the connection
manager class to be used, instead of inferring it from the
connection string.
From: Ahmed Radwan <ahmed@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149993 13f79535-47bb-0310-9956-ffa450edef68
This change fixes a problem that prevented users from importing into Hive
data extracted using a free form query.
From: Ahmed Radwan <ahmed@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149992 13f79535-47bb-0310-9956-ffa450edef68
This is a fix for a case of malformed command line arguments,
where the tool name is correct, the option is also correct,
but the value of the option is missing.
From: Ahmed Radwan <ahmed@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149990 13f79535-47bb-0310-9956-ffa450edef68
Hive allows the use of keywords as column and table names as long as they are
escaped using back-ticks. This change makes Sqoop always escape table and
column names using back-ticks thereby allowing Sqoop to work with Hive tables
that use keywords for either the table name or column names.
(Lars Francke via arvind)
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149989 13f79535-47bb-0310-9956-ffa450edef68
The TextSplitter implementation used when creating splits on top
of String based columns, has a bug in its logic which causes
the bounds for splits to be created incorrectly. This results
in the import of duplicate data. This change fixes the TextSplitter
in order to ensure that the bound checks are created correctly.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149988 13f79535-47bb-0310-9956-ffa450edef68
This change modifies the ClassWriter implementation to provide
support for interpeting empty strings as NULL values for datatypes
other than String. For String datatype, an explicit string 'null'
is interpreted as NULL value and empty string is not. This is
because certain databases distinguish between NULL and empty
strings.
The clone implementation generated by ClassWriter has also been
modified to make it more defensive against the presence of NULL
values.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149985 13f79535-47bb-0310-9956-ffa450edef68
Current CDH3 build includes version 0.20.3 of Hadoop which is now
mapped to CDH3Shim loader. Apart from that, this change includes
a change in build.xml and OracleUtils test class that allows the
ability to override connect string for Oracle tests.
From: Arvind Prabhakar <arvind@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149984 13f79535-47bb-0310-9956-ffa450edef68
Adds 'merge' tool.
Adds MergeJob, Merge*Mapper, MergeReducer.
Merge-specific arguments added to SqoopOptions, BaseSqoopTool.
Add TestMerge to test that this tool functions as expected.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149980 13f79535-47bb-0310-9956-ffa450edef68
Create nonce dirs as /tmp/sqoop-${user.name}/compile.
Remove unused compilation directories at end of execution.
Do not memoize nonce compilation directory names in metastore.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149975 13f79535-47bb-0310-9956-ffa450edef68
Files in conf/managers.d/ are treated as configuration files that specify
classes for sqoop.connection.factories. If this property is unset, these
files are processed in order.
ClassLoaderStack no longer attempts to load a jar if the test class is
already available with the current set of ClassLoaders.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149966 13f79535-47bb-0310-9956-ffa450edef68
Modify sqoop-eval to use generic execute() instead of
executeQuery()/executeUpdate().
The --update argument to sqoop-eval is unnecessary and has been removed.
Modify ResultSetPrinter to properly display left border on output.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149963 13f79535-47bb-0310-9956-ffa450edef68
Added ManagerFactory.accept(SessionData) API to allow ManagerFactory
to inspect the chosen SqoopTool.
Deprecated ManagerFactory.accept(SqoopOptions).
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149958 13f79535-47bb-0310-9956-ffa450edef68
Add com.cloudera.sqoop.io.NamedFifo class to represent named FIFO objects.
Added TestNamedFifo as unit test.
MySQLExportMapper now uses this utility.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149954 13f79535-47bb-0310-9956-ffa450edef68
ImportJobContext and ExportJobContext now contain references to the
already-instantiated ConnManager. JobBase subclasses should no
longer use a ConnFactory, but should use the existing ConnManager.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149953 13f79535-47bb-0310-9956-ffa450edef68
Added --update argument to sqoop-eval tool, which runs
the user's statement in update mode rather than select mode.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149952 13f79535-47bb-0310-9956-ffa450edef68
.class and .jar generation occurs in a per-process anonymous
directory. Generation of .java files now occurs in the bin directory;
.java files are moved to the src directory after compilation.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149947 13f79535-47bb-0310-9956-ffa450edef68
Modify ImportTool, SessionTool, to support incremental imports.
Add TestIncrementalImport to unit test incremental imports.
SqoopOptions now implements Cloneable.
SQOOP-44. Bugfix in ClassWriter: fix NPE if the case of column names specified
with --columns do not match the case reported by the database.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149944 13f79535-47bb-0310-9956-ffa450edef68
Sqoop, shim, JDBC driver, and $SQOOP_HOME/lib/*.jar are now all
placed in the DistributedCache for each job.
Removes generation of the 'lib/' subdir in generated jar files.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149943 13f79535-47bb-0310-9956-ffa450edef68
Copy DataDrivenDBInputFormat, its dependencies, and tests into Sqoop
in the com.cloudera.sqoop.mapreduce.db package.
Reformatted code to match Sqoop style guide and eliminate findbugs warnings.
Changed existing Sqoop code to use this implementation rather than Hadoop's.
Modified TestDataDrivenDBInputFormat to use mem-only hsqldb to prevent
Hudson race condition.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149941 13f79535-47bb-0310-9956-ffa450edef68
Added SessionStorage API.
Added SessonData API.
Added SessionStorageFactory.
Added SessionTool for create/delete/execute/show/list operations on sessions.
SqoopOptions can read and write all "sticky" state to a Properties instance.
Added HsqldbSessionStorage to implement SessionStorage API.
Added AutoHsqldbStorage to auto-instantiate a local metastore for the user.
Added client metastore connection parameters to sqoop-site.xml.
Added metastore tool (MetastoreTool).
Added HsqldbMetaStore for standalone metastore instance.
Added metastore properties to sqoop-default.xml.
Added TestSessions unit tests of session API.
Renamed conf/sqoop-default.xml to conf/sqoop-site-template.xml.
Added conf/.gitignore for sqoop-site.xml.
Tests run:
Tested all metastore operations on an import session.
Tested that ~/.sqoop/-based storage will be auto-created by the metastore.
Tested that 'sqoop metastore'-based metastores can be connected to
by external clients.
Tested that 'sqoop metastore --shutdown' will gracefully shut down a running
metastore instance.
Tested that passwords are not stored in the metastore by default, and the
user is prompted for the password when executing that saved session.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149940 13f79535-47bb-0310-9956-ffa450edef68
If jobs are already retired, display a useful message on how to
enable the completed job store and suppress performance counter
usage, rather than crashing with a NullPointerException.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149939 13f79535-47bb-0310-9956-ffa450edef68
Added FieldMappable and FieldMapProcessor interfaces.
Added ProcessingException class.
Added NullOutputCommitter class.
SqoopRecord now has delegate() method which calls a FieldMapProcessor.
ClassWriter now generates getFieldMap() method for SqoopRecords.
Added HBasePutProcessor to transform SqoopRecords into Put commands,
implementing FieldMapProcessor.
Added PutTransformer interface class and ToStringPutTransformer implementation.
Added DelegatingOutputFormat that uses a FieldMapProcessor.
Added HBase deps to build.xml via hbase.home property.
Added HBase, ZooKeeper to the dependency net added by configure-sqoop.
Added HBaseImportJob, HBaseImportMapper.
ImportJobBase now has jobSetup() step executed just before job submission.
ImportJobContext now holds a reference to the ConnManager.
DataDrivenImportJob retrieves ConnManager from ImportJobContext, it no longer
creates a new one.
Added HBase table import configuration parameters to SqoopOptions, ImportTool.
SqlManager.importQuery() needs to set ConnManager in ImportJobContext.
Added HBase import user documentation.
Described PutTransformer API in developer docs.
Added HBase unit tests.
Added ANT_ARGUMENTS env variable to Hudson test scripts to allow freeform parameters.
Added HBASE_HOME and ZOOKEEPER_HOME variables to hudson scripts.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149935 13f79535-47bb-0310-9956-ffa450edef68
Add --update-key argument to sqoop-export tool.
Refactor ExportOutputFormat into AsyncSqlOutputFormat.
Added UpdateOutputFormat.
ClassWriter now allows alternate serialization order in database write() method.
SqoopOptions holds column list for alternate db serialization order.
Added TestExportUpdate unit test battery.
AsyncSqlRecordWriter now allows "batch" execution mode.
Updated documentation for export updates.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149933 13f79535-47bb-0310-9956-ffa450edef68
Add ConnManager.importQuery() API.
Change BaseSqoopTool.DEBUG_SQL_CMD_ARG to SQL_QUERY_ARG to reflect
the broader applicability of the argument.
Change 'debugSqlCmd' member of SqoopOptions to 'sqlQuery'.
CompilationManager now sets jar name based on specified class name.
Add tests for query support.
Add documentation for query-based import.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149932 13f79535-47bb-0310-9956-ffa450edef68
Maven pushes sqoop jar and shim jars.
Renamed shim jar files to conform to maven artifact-version naming.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149925 13f79535-47bb-0310-9956-ffa450edef68
If the Hive CliDriver is present on the classpath, Sqoop will
call it directly in the same JVM, rather than invoke a separate
JVM instance for Hive.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149916 13f79535-47bb-0310-9956-ffa450edef68
Moves TaskId to com.cloudera.sqoop.util.
Add com.cloudera.sqoop.lib.DelimiterSet.
Rewrite FieldFormatter, RecordParser, to use DelimiterSet.
Add generated class version id to SqoopRecord.
From: Aaron Kimball <aaron@cloudera.com>
git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149907 13f79535-47bb-0310-9956-ffa450edef68
