github/sqoop
5
0
Fork 0
mirror of https://github.com/apache/sqoop.git synced 2025-05-17 01:11:07 +08:00
Code Issues Projects Releases Wiki Activity

SQOOP-1717: Sqoop2: Remove Data class from docs

Browse Source 
(Veena Basavaraj via Abraham Elmahrek)
This commit is contained in:
Abraham Elmahrek 2014-11-18 15:48:23 -08:00
parent 77eb52e240
commit 6bd8fe3e93
1 changed files with 33 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
docs/src/site/sphinx/ConnectorDevelopment.rst
View File

@ -171,7 +171,8 @@ Extractor (E for ETL) extracts data from a given data source
JobConfiguration jobConfiguration, JobConfiguration jobConfiguration,
SqoopPartition partition); SqoopPartition partition);
The ``extract`` method extracts data from the data source using the link and job configuration properties and writes it to the ``DataWriter`` (provided by the extractor context) as the default `Intermediate representation`_ . The ``extract`` method extracts data from the data source using the link and job configuration properties and writes it to the ``SqoopMapDataWriter`` (provided in the extractor context given to the extract method).
The ``SqoopMapDataWriter`` has the ``SqoopWritable`` thats holds the data read from the data source in the `Intermediate Data Format representation`_
Extractors use Writer's provided by the ExtractorContext to send a record through the sqoop system. Extractors use Writer's provided by the ExtractorContext to send a record through the sqoop system.
:: ::
@ -225,7 +226,7 @@ A loader (L for ETL) receives data from the ``From`` instance of the sqoop conne
ConnectionConfiguration connectionConfiguration, ConnectionConfiguration connectionConfiguration,
JobConfiguration jobConfiguration) throws Exception; JobConfiguration jobConfiguration) throws Exception;
The ``load`` method reads data from ``DataReader`` (provided by context) in the default `Intermediate representation`_ and loads it to data source. The ``load`` method reads data from ``SqoopOutputFormatDataReader`` (provided in the loader context of the load methods). It reads the data in the `Intermediate Data Format representation`_ and loads it to the data source.
Loader must iterate in the ``load`` method until the data from ``DataReader`` is exhausted. Loader must iterate in the ``load`` method until the data from ``DataReader`` is exhausted.
:: ::
@ -414,15 +415,15 @@ The diagram below describes the map phase of a job.
| extract | | | extract | |
|-------------------->| | |-------------------->| |
| | | | | |
read from DB | | read from Data Source | |
<-------------------------------| write* | <-------------------------------| write* |
| |------------------->| | |------------------->|
| | | ,----. | | | ,-------------.
| | |---------->|Data| | | |---------->|SqoopWritable|
| | | `-+--' | | | `----+--------'
| | | | | | |
| | | context.write | | | | context.write(writable, ..)
| | |--------------------------> | | | |---------------------------->
The diagram below decribes the reduce phase of a job. The diagram below decribes the reduce phase of a job.
``OutputFormat`` invokes ``To`` connector's loader's ``load`` method (via ``SqoopOutputFormatLoadExecutor`` ). ``OutputFormat`` invokes ``To`` connector's loader's ``load`` method (via ``SqoopOutputFormatLoadExecutor`` ).
@ -433,15 +434,14 @@ The diagram below decribes the reduce phase of a job.
`---+--------' `----------+----------' `---+--------' `----------+----------'
| | ,-----------------------------. | | ,-----------------------------.
| |-> |SqoopOutputFormatLoadExecutor| | |-> |SqoopOutputFormatLoadExecutor|
| | `--------------+--------------' ,----. | | `--------------+--------------' |
| | |---------------------> |Data| | | | |
| | | `-+--' | | | ,-----------------. ,-------------.
| | | ,-----------------. | | | |-> |SqoopRecordWriter|-->|SqoopWritable|
| | |-> |SqoopRecordWriter| | getRecordWriter | | `--------+--------' `---+---------'
getRecordWriter | | `--------+--------' |
----------------------->| getRecordWriter | | | ----------------------->| getRecordWriter | | |
| |----------------->| | | ,--------------. | |----------------->| | | ,--------------.
| | |-----------------------------> |ConsumerThread| | | |---------------------------------->|ConsumerThread|
| | | | | `------+-------' | | | | | `------+-------'
| |<- - - - - - - - -| | | | ,------. | |<- - - - - - - - -| | | | ,------.
<- - - - - - - - - - - -| | | | |--->|Loader| <- - - - - - - - - - - -| | | | |--->|Loader|
@ -451,12 +451,12 @@ The diagram below decribes the reduce phase of a job.
run | | | | | |------>| run | | | | | |------>|
----->| | write | | | | | ----->| | write | | | | |
|------------------------------------------------>| setContent | | read* | |------------------------------------------------>| setContent | | read* |
| | | |----------->| getContent |<------| | | | |--------------->| getContent |<------|
| | | | |<-----------| | | | | | |<-----------| |
| | | | | | - - ->| | | | | | | - - ->|
| | | | | | | write into DB | | | | | | | write into Data Source
| | | | | | |--------------> | | | | | | |----------------------->
.. _`Intermediate representation`: https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Intermediate+representation .. _`Intermediate Data Format representation`: https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Intermediate+representation