5
0
mirror of https://github.com/apache/sqoop.git synced 2025-05-20 19:00:48 +08:00
Commit Graph

226 Commits

Author SHA1 Message Date
Andrew Bayer
5c088f32ec SQOOP-73. Generated records should handle more boolean value formats.
Add BooleanParser to parse more strings than Boolean.valueOf() does.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149960 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:11 +00:00
Andrew Bayer
7f53d52f4f SQOOP-69. Improve data accessible to ManagerFactory.accept().
Added ManagerFactory.accept(SessionData) API to allow ManagerFactory
to inspect the chosen SqoopTool.
Deprecated ManagerFactory.accept(SqoopOptions).

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149958 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:11 +00:00
Andrew Bayer
402a2cd61a SQOOP-68. Delimiters in strings are not escaped properly.
Use escape character to protect delimiters if enclosing character is
unavailable.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149957 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:10 +00:00
Andrew Bayer
1749a84f68 SQOOP-64. Refactor named FIFO creation into standalone utility.
Add com.cloudera.sqoop.io.NamedFifo class to represent named FIFO objects.
Added TestNamedFifo as unit test.
MySQLExportMapper now uses this utility.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149954 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:10 +00:00
Andrew Bayer
e3fb49775d SQOOP-56. Improve parameterization of ManagerCompatTestCase.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149951 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:09 +00:00
Andrew Bayer
87d0dab003 SQOOP-52. Stop TestHBaseImport from timing out.
Restart MiniHBaseCluster between each test to prevent triggering
livelock at end of test battery.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149950 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:08 +00:00
Andrew Bayer
bf92ba41dd SQOOP-55. BaseSqoopTestCase.createTableWithColTypes() calls close() twice.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149949 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:08 +00:00
Andrew Bayer
6efcec0da2 SQOOP-41. Add support for incremental imports.
Modify ImportTool, SessionTool, to support incremental imports.
Add TestIncrementalImport to unit test incremental imports.
SqoopOptions now implements Cloneable.
SQOOP-44. Bugfix in ClassWriter: fix NPE if the case of column names specified
with --columns do not match the case reported by the database.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149944 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:07 +00:00
Andrew Bayer
1a5f52f063 SQOOP-24. Copy o.a.h.mapreduce.lib.db classes into Sqoop.
Copy DataDrivenDBInputFormat, its dependencies, and tests into Sqoop
in the com.cloudera.sqoop.mapreduce.db package.
Reformatted code to match Sqoop style guide and eliminate findbugs warnings.
Changed existing Sqoop code to use this implementation rather than Hadoop's.
Modified TestDataDrivenDBInputFormat to use mem-only hsqldb to prevent
Hudson race condition.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149941 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:05 +00:00
Andrew Bayer
06b94587af SQOOP-39. (SIP-6) Session Management API.
Added SessionStorage API.
Added SessonData API.
Added SessionStorageFactory.
Added SessionTool for create/delete/execute/show/list operations on sessions.
SqoopOptions can read and write all "sticky" state to a Properties instance.
Added HsqldbSessionStorage to implement SessionStorage API.
Added AutoHsqldbStorage to auto-instantiate a local metastore for the user.
Added client metastore connection parameters to sqoop-site.xml.
Added metastore tool (MetastoreTool).
Added HsqldbMetaStore for standalone metastore instance.
Added metastore properties to sqoop-default.xml.
Added TestSessions unit tests of session API.
Renamed conf/sqoop-default.xml to conf/sqoop-site-template.xml.
Added conf/.gitignore for sqoop-site.xml.

Tests run:

Tested all metastore operations on an import session.
Tested that ~/.sqoop/-based storage will be auto-created by the metastore.
Tested that 'sqoop metastore'-based metastores can be connected to
by external clients.
Tested that 'sqoop metastore --shutdown' will gracefully shut down a running
metastore instance.
Tested that passwords are not stored in the metastore by default, and the
user is prompted for the password when executing that saved session.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149940 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:05 +00:00
Andrew Bayer
7e0ccb4008 SQOOP-14. Allow Sqoop to import data into HBase.
Added FieldMappable and FieldMapProcessor interfaces.
Added ProcessingException class.
Added NullOutputCommitter class.
SqoopRecord now has delegate() method which calls a FieldMapProcessor.
ClassWriter now generates getFieldMap() method for SqoopRecords.
Added HBasePutProcessor to transform SqoopRecords into Put commands,
implementing FieldMapProcessor.
Added PutTransformer interface class and ToStringPutTransformer implementation.
Added DelegatingOutputFormat that uses a FieldMapProcessor.
Added HBase deps to build.xml via hbase.home property.
Added HBase, ZooKeeper to the dependency net added by configure-sqoop.
Added HBaseImportJob, HBaseImportMapper.
ImportJobBase now has jobSetup() step executed just before job submission.
ImportJobContext now holds a reference to the ConnManager.
DataDrivenImportJob retrieves ConnManager from ImportJobContext, it no longer
creates a new one.
Added HBase table import configuration parameters to SqoopOptions, ImportTool.
SqlManager.importQuery() needs to set ConnManager in ImportJobContext.
Added HBase import user documentation.
Described PutTransformer API in developer docs.
Added HBase unit tests.
Added ANT_ARGUMENTS env variable to Hudson test scripts to allow freeform parameters.
Added HBASE_HOME and ZOOKEEPER_HOME variables to hudson scripts.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149935 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:03 +00:00
Andrew Bayer
7e473cafa0 SQOOP-22. Sqoop date tests are too permissive.
Ensure that tests involving dates/times use proper ANSI SQL Date/Time escape
formatting (yyyy-mm-dd or hh:mm:ss). After Java 1.6u17, dates of the form
yyyy-m-dd or hhⓂ️ss are not parsed by java.sql.Date/Time and throw
IllegalArgumentException.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149934 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:03 +00:00
Andrew Bayer
6ea1665716 SQOOP-4. Add update-mode export process.
Add --update-key argument to sqoop-export tool.
Refactor ExportOutputFormat into AsyncSqlOutputFormat.
Added UpdateOutputFormat.
ClassWriter now allows alternate serialization order in database write() method.
SqoopOptions holds column list for alternate db serialization order.
Added TestExportUpdate unit test battery.
AsyncSqlRecordWriter now allows "batch" execution mode.
Updated documentation for export updates.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149933 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:02 +00:00
Andrew Bayer
b0cd632b58 SQOOP-3. Add ability to import arbitrary queries.
Add ConnManager.importQuery() API.
Change BaseSqoopTool.DEBUG_SQL_CMD_ARG to SQL_QUERY_ARG to reflect
the broader applicability of the argument.
Change 'debugSqlCmd' member of SqoopOptions to 'sqlQuery'.
CompilationManager now sets jar name based on specified class name.
Add tests for query support.
Add documentation for query-based import.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149932 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:02 +00:00
Andrew Bayer
d215cbac6e SQOOP-2. MySQLExportMapper does not send username.
If specified, pass --user argument to mysqlimport in MySQLExportMapper.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149930 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:04:01 +00:00
Andrew Bayer
568a827a1c Append mode import and target-dir output
Signed-off-by: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149913 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:56 +00:00
Andrew Bayer
383e46201c Eliminate checkstyle warnings.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149909 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:55 +00:00
Andrew Bayer
3ab9ebd354 SIP-4. Refactor classes for public API 1.0.0.
Moves TaskId to com.cloudera.sqoop.util.
Add com.cloudera.sqoop.lib.DelimiterSet.
Rewrite FieldFormatter, RecordParser, to use DelimiterSet.
Add generated class version id to SqoopRecord.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149907 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:54 +00:00
Andrew Bayer
42875119dd Move org.apache.hadoop.sqoop to com.cloudera.sqoop
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149906 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:52 +00:00
Andrew Bayer
0b96b5f1c3 Add checkstyle to build process.
Fix the overwhelming majority of existing violations.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149899 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:47 +00:00
Andrew Bayer
e926bf1fe0 New ExportOutputFormat with improved JDBC performance.
Added ExportOutputFormat.
JdbcExportJob uses ExportOutputFormat instead of DBOutputFormat.
SqoopRecord now supports write() to multi-row INSERT statement, clone().
ClassWriter generates methods to fulfill expanded SqoopRecord contract.
Added Jdbc MySQL tests to test multi-row statements.
BlobRef, ClobRef classes now support Cloneable interface.
Added ExportStressTest.
Added Oracle-specific ExportOutputFormat subclass.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149898 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:46 +00:00
Andrew Bayer
1eb4226230 SIP-3. File format for large object (LOB) storage.
Introduce LobFile format for storing large objects.
Implemented LobFile.Reader, LobFile.Writer classes.
Added a performance test of LobFile reading/writing speed.
Build system: fix cobertura build deps.
Remove unused utility classes from o.a.h.s.io.
Use LobFile for external storage in {B,C}lobRef.
Added LobReaderCache.
Converted BlobRef to read from LobFiles (through LobReaderCache).
LargeObjectLoader writes to LobFiles.
Common code from BlobRef and ClobRef factored out into LobRef abstract
base class.
Updated Test{B,C}lobRef and TestLargeObjectLoader for new external LOB storage.
Updated *ImportMappers to close LargeObjectLoaders when they're done.
Added performance tests to build.
Added script to run perf tests; factored out common logic into config script.
Fixed ivy dependency resolution to use multiple configuration inheritance.
Added LobFileStressTest.
Added readme with instructions to src/perftest directory.
Added CodecMap that abstracts compression codec classes to names.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149897 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:46 +00:00
Andrew Bayer
b4d59da671 Fix dead store in SqoopOptions.setTmpDir().
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149895 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:46 +00:00
Andrew Bayer
b7a3f47222 SIP-1. Refactor operations into distinct tools.
Introduced SqoopTool interface.
Added cli package for option parsing:
includes RelatedOptions, ToolOptions, SqoopParser.
'Sqoop' is now a wrapper that invokes a SqoopTool.
Added setter methods for all fields of SqoopOptions.
Added commons-cli 1.2 build dependency.
Argument parsing is removed from SqoopOptions and pushed into individual tools.
Added HelpTool to display basic usage information for Sqoop and
usage for subcommands.
Added ImportTool to perform imports.
Added EvalSqlTool.
Added ExportTool.
Added ImportAllTablesTool.
Added ListDatabasesTool, ListTablesTool.
Added CodeGenTool.
Added CreateHiveTableTool.
Small changes to orm.ClassWriter.
Auto-generate bin scripts for all tools; include in release package.
Allow user to provide build properties in a file.
Shim use of GenericOptionsParser to allow cross-compilation.
Fix Hive testcases to pass under CDH.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149894 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:45 +00:00
Andrew Bayer
c37e2d0568 Set Oracle session timezone based on property.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149893 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:45 +00:00
Andrew Bayer
7be68f8418 Oracle connection pool caches closed connections
OracleConnManager now nullifies references to connection instances
after returning them to the connection cache or closing them.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149891 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:44 +00:00
Andrew Bayer
13e32e71cf Add better exception logging to BaseSqoopTestCase
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149889 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:44 +00:00
Andrew Bayer
9a7dcdeafb MapReduce-API specific classes moved to shim jars.
All classes which depend on MapReduce APIs which change from
interfaces to classes between 0.20 and 0.22 are moved to distribution-
specific shim jars.
"Common" shim classes are now compiled multiple times against different
Hadoop distributions.
Shim classes are broken out into separate jars; ShimLoader now picks
the appropriate jar to load at runtime.
Configuration constants moved into HadoopShim.
BlobRef/ClobRef methods changed to use Mapper.Context for binary compatibility.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149884 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:42 +00:00
Andrew Bayer
b55cb598da Add shim classes to allow compilation against different Hadoop distributions
Version-incompatible code now moved to HadoopShim subclasses.
HadoopShim singleton instance dynamically loaded based on VersionInfo.
Separate MRUnit builds from Apache and CDH placed in /lib subdirs.
Modified 'ant package' target to properly include all shims.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149880 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:41 +00:00
Andrew Bayer
9374ed8398 Add support for mysqlimport-based export jobs.
Using --direct in conjunction with --export-dir on a MySQL database will use
mysqlimport to emit the data to the database.
DirectMySQLManager now creates instances of MySQLExportJob.
src/test/.../MySQLUtils is renamed to MySQLTestUtils to avoid conflict with
src/java/.../MySQLUtils added by this patch.
MySQLUtils contains methods factored out of import-specific code for sharing
with exports.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149877 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:40 +00:00
Andrew Bayer
8147d262d8 Enable findbugs on build and fix all warnings.
Some spurious warnings (and inconsequential warnings in test code)
have been disabled by src/test/findbugsExcludeFile.xml.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149874 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:39 +00:00
Andrew Bayer
7214230695 Support BINARY, VARBINARY, and RAW (Oracle) types
Added support for importing byte array columns as BytesWritable.
Tested with MySQL, Oracle, HSQLDB.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149873 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:39 +00:00
Andrew Bayer
2240be8807 Cache connections to Oracle across ConnManagers.
OracleManager now caches Connection instances for subsequent OracleManager
instances.
Refactored uses of ConnManager to call close() before discarding them.
This allows the Oracle JUnit tests to sleep less frequently to wait for Oracle
to reap closed server-side connection resources, improving Oracle test speed
by 50%.

Sleeping cannot be fully eliminated because MapReduce-side Connections are not
governed by this caching mechanism.

Also added some debugging advice re. this topic to OracleManagerTest's comment.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149872 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:39 +00:00
Andrew Bayer
43f9e2f2b0 Added unit test to check network setup needed for postgres tests.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149871 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:38 +00:00
Andrew Bayer
6cbe7572e9 If --hive-import and --generate-only are specified, create a ddl script file.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149870 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:38 +00:00
Andrew Bayer
df76e995e8 Users can precisely control export parallelism.
Uses CombineFileInputFormat to run exports over a target number
of mappers independent of the number of input files.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149869 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:38 +00:00
Andrew Bayer
a0dd7e7490 Changed license headers to reference Cloudera instead of the ASF.
Adds NOTICE.txt file

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149868 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:36 +00:00
Andrew Bayer
bb29ce9492 Support for CLOB/BLOB data in external files.
CLOB/BLOB data may now be stored in additional files in HDFS which are
accessible through streams if the data cannot be fully materialized in RAM.
Adds tests for external large objects.
Refactored large object loading into the map() method from readFields().

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149866 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:35 +00:00
Andrew Bayer
32a67749b1 Run mysqldump in map task instead of on the client.
Major refactoring of DataDrivenImportJob to support mysqldump in mappers.
ImportJobBase added below DataDrivenImportJob.
MySQLDumpImportJob added on top of ImportJobBase.
LocalMySQLManager -> renamed to -> DirectMySQLManager now just runs MysqldumpIJ.
MySQLDumpImportJob configures MySQLDumpMapper to run mysqldump instances on
multiple nodes and is split-aware (via MySQLDumpInputFormat).
TestImportJob works with new ImportJobBase framework.
Added test that imports a subset of columns in mysql imports.

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149865 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:35 +00:00
Andrew Bayer
bdd405f756 Improve batch testrunner support for third-party tests.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149864 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:35 +00:00
Andrew Bayer
14c9e0bf88 Use DataDrivenDBInputFormat with Oracle.
From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149863 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:34 +00:00
Andrew Bayer
7482c71cf9 Initial support for CLOB/BLOB types
Tests pass in Oracle and MySQL compatibility suites

From: Aaron Kimball <aaron@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149862 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:34 +00:00
Andrew Bayer
af2ec3a03f MAPREDUCE-1445. Refactor Sqoop tests to support better ConnManager testing. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149860 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:33 +00:00
Andrew Bayer
a625fd478c MAPREDUCE-1341. Sqoop should have an option to create hive tables and skip the table import step. Contributed by Leonid Furman.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149856 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:32 +00:00
Andrew Bayer
de836d714a MAPREDUCE-1356. Allow user-specified hive table name in sqoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149855 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:32 +00:00
Andrew Bayer
69f04fff8b MAPREDUCE-1395. Sqoop does not check return value of Job.waitForCompletion(). Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149854 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:31 +00:00
Andrew Bayer
e7a8e519f3 MAPREDUCE-1327. Fix Sqoop handling of Oracle timezone with timestamp data
types in import. Contributed by Leonid Furman

From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149852 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:31 +00:00
Andrew Bayer
6174268d28 MAPREDUCE-1313. Fix NPE in Sqoop when table with null fields uses escape
during import. Contributed by Aaron Kimball

From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149851 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:31 +00:00
Andrew Bayer
05929a73e5 MAPREDUCE-1310. CREATE TABLE statements for Hive do not correctly specify delimiters. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149849 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:30 +00:00
Andrew Bayer
b74084196f MAPREDUCE-1235. Fix a MySQL timestamp incompatibility in Sqoop. Contributed by Aaron Kimball
From: Christopher Douglas <cdouglas@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149848 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:30 +00:00
Andrew Bayer
681461461a MAPREDUCE-1174. Sqoop improperly handles table/column names which are reserved sql words. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149847 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:30 +00:00
Andrew Bayer
4686e0fee7 MAPREDUCE-1148. SQL identifiers are a superset of Java identifiers. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149845 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:29 +00:00
Andrew Bayer
8e813b95a4 MAPREDUCE-1168. Export data to databases via Sqoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149843 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:28 +00:00
Andrew Bayer
ec8f687d97 MAPREDUCE-1169. Improvements to mysqldump use in Sqoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149840 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:28 +00:00
Andrew Bayer
9afc7a8aee MAPREDUCE-1069. Implement Sqoop API refactoring. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149837 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:27 +00:00
Andrew Bayer
93dab4998e MAPREDUCE-1017. Compression and output splitting for Sqoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149836 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:26 +00:00
Andrew Bayer
8dcc748853 MAPREDUCE-947. Added commitJob and abortJob apis to OutputCommitter. Enhanced FileOutputCommitter to create a _SUCCESS file for successful jobs. Contributed by Amar Kamat & Jothi Padmanabhan.
From: Arun Murthy <acmurthy@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149835 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:26 +00:00
Andrew Bayer
4a689cf491 MAPREDUCE-999. Improve Sqoop test speed and refactor tests. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149833 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:26 +00:00
Andrew Bayer
7e3284691d MAPREDUCE-849. Rename configuration properties. Contributed by Amareshwari Sriramadasu.
From: Sharad Agarwal <sharad@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149830 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:25 +00:00
Andrew Bayer
37cadedc8f MAPREDUCE-907. Sqoop should use more intelligent splits. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149828 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:24 +00:00
Andrew Bayer
b34b2acc10 MAPREDUCE-938. Postgresql support for Sqoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149826 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:24 +00:00
Andrew Bayer
2bbf349f4c MAPREDUCE-918. Test hsqldb server should be memory-only. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149825 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:24 +00:00
Andrew Bayer
9e8955ede1 MAPREDUCE-876. Sqoop import of large tables can time out. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149824 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:23 +00:00
Andrew Bayer
89241b591a MAPREDUCE-750. Extensible ConnManager factory API. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149823 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:23 +00:00
Andrew Bayer
e7cf0ed69c MAPREDUCE-749. Make Sqoop unit tests more Hudson-friendly. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149822 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:23 +00:00
Andrew Bayer
6cf87c5147 MAPREDUCE-789. Oracle support for Sqoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149820 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:23 +00:00
Andrew Bayer
ecbb765019 MAPREDUCE-816. Rename "local" mysql import to "direct" in Sqoop. (Aaron Kimball via matei).
From: Matei Alexandru Zaharia <matei@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149819 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:22 +00:00
Andrew Bayer
3d39962dfa MAPREDUCE-705. User-configurable quote and delimiter characters for Sqoop records and record reparsing. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149818 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:22 +00:00
Andrew Bayer
3c322c9969 MAPREDUCE-710. Sqoop should read and transmit passwords in a more secure manner. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149817 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:22 +00:00
Andrew Bayer
120e06f0a0 MAPREDUCE-713. Sqoop has some superfluous imports. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149816 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:21 +00:00
Andrew Bayer
595bf13670 MAPREDUCE-685. Sqoop will fail with OutOfMemory on large tables using mysql. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149815 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:21 +00:00
Andrew Bayer
d427bca2de MAPREDUCE-675. Sqoop should allow user-defined class and package names. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149814 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:21 +00:00
Andrew Bayer
9dfc7ba898 MAPREDUCE-674. Sqoop should allow a "where" clause to avoid having to export entire tables. Contributed by Kevin Weil.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149813 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:21 +00:00
Andrew Bayer
dc191132a5 HADOOP-5887. Sqoop should create tables in Hive metastore after importing to HDFS. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149807 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:20 +00:00
Andrew Bayer
bf65299ba1 HADOOP-5844. Use mysqldump when connecting to local mysql instance in Sqoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149806 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:19 +00:00
Andrew Bayer
c58670cbdb HADOOP-5815. Sqoop: A database import tool for Hadoop. Contributed by Aaron Kimball.
From: Thomas White <tomwhite@apache.org>

git-svn-id: https://svn.apache.org/repos/asf/incubator/sqoop/trunk@1149805 13f79535-47bb-0310-9956-ffa450edef68
2011-07-22 20:03:19 +00:00