github/sqoop
5
0
Fork 0
mirror of https://github.com/apache/sqoop.git synced 2025-05-04 23:12:38 +08:00
Code Issues Projects Releases Wiki Activity

SQOOP-2736: Sqoop2: Document change command line shell interface

Browse Source 
(Colin Ma via Jarek Jarcec Cecho)
This commit is contained in:
Jarek Jarcec Cecho 2015-12-15 14:32:42 +01:00
parent d35efd3c3a
commit d89f64adc8
4 changed files with 194 additions and 188 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

50
docs/src/site/sphinx/dev/ClientAPI.rst
View File

@ -34,9 +34,9 @@ Workflow
Given workflow has to be followed for executing a sqoop job in Sqoop server. Given workflow has to be followed for executing a sqoop job in Sqoop server.
1. Create LINK object for a given connectorId - Creates Link object and returns linkId (lid) 1. Create LINK object for a given connector name - Creates Link object and returns it
2. Create a JOB for a given "from" and "to" linkId - Create Job object and returns jobId (jid) 2. Create a JOB for a given "from" and "to" link name - Create Job object and returns it
3. Start the JOB for a given jobId - Start Job on the server and creates a submission record 3. Start the JOB for a given job name - Start Job on the server and creates a submission record
Project Dependencies Project Dependencies
==================== ====================
@ -76,13 +76,12 @@ you can get the list of all the config/inputs using `Display Config and Input Na
Save Link Save Link
--------- ---------
First create a new link by invoking ``createLink(cid)`` method with connector Id and it returns a MLink object with dummy id and the unfilled link config inputs for that connector. Then fill the config inputs with relevant values. Invoke ``saveLink`` passing it the filled MLink object. First create a new link by invoking ``createLink(connectorName)`` method with connector name and it returns a MLink object with dummy id and the unfilled link config inputs for that connector. Then fill the config inputs with relevant values. Invoke ``saveLink`` passing it the filled MLink object.
:: ::
// create a placeholder for link // create a placeholder for link
long connectorId = 1; MLink link = client.createLink("connectorName");
MLink link = client.createLink(connectorId);
link.setName("Vampire"); link.setName("Vampire");
link.setCreationUser("Buffy"); link.setCreationUser("Buffy");
MLinkConfig linkConfig = link.getConnectorLinkConfig(); MLinkConfig linkConfig = link.getConnectorLinkConfig();
@ -94,21 +93,21 @@ First create a new link by invoking ``createLink(cid)`` method with connector Id
// save the link object that was filled // save the link object that was filled
Status status = client.saveLink(link); Status status = client.saveLink(link);
if(status.canProceed()) { if(status.canProceed()) {
System.out.println("Created Link with Link Id : " + link.getPersistenceId()); System.out.println("Created Link with Link Name : " + link.getName());
} else { } else {
System.out.println("Something went wrong creating the link"); System.out.println("Something went wrong creating the link");
} }
``status.canProceed()`` returns true if status is OK or a WARNING. Before sending the status, the link config values are validated using the corresponding validator associated with th link config inputs. ``status.canProceed()`` returns true if status is OK or a WARNING. Before sending the status, the link config values are validated using the corresponding validator associated with th link config inputs.
On successful execution of the saveLink method, new link Id is assigned to the link object else an exception is thrown. ``link.getPersistenceId()`` method returns the unique Id for this object persisted in the sqoop repository. On successful execution of the saveLink method, new link name is assigned to the link object else an exception is thrown. ``link.getName()`` method returns the unique name for this object persisted in the sqoop repository.
User can retrieve a link using the following methods User can retrieve a link using the following methods
+----------------------------+--------------------------------------+ +----------------------------+--------------------------------------+
| Method | Description | | Method | Description |
+============================+======================================+ +============================+======================================+
| ``getLink(lid)`` | Returns a link by id | | ``getLink(linkName)`` | Returns a link by name |
+----------------------------+--------------------------------------+ +----------------------------+--------------------------------------+
| ``getLinks()`` | Returns list of links in the sqoop | | ``getLinks()`` | Returns list of links in the sqoop |
+----------------------------+--------------------------------------+ +----------------------------+--------------------------------------+
@ -118,7 +117,7 @@ Job
A sqoop job holds the ``From`` and ``To`` parts for transferring data from the ``From`` data source to the ``To`` data source. Both the ``From`` and the ``To`` are uniquely identified by their corresponding connector Link Ids. i.e when creating a job we have to specifiy the ``FromLinkId`` and the ``ToLinkId``. Thus the pre-requisite for creating a job is to first create the links as described above. A sqoop job holds the ``From`` and ``To`` parts for transferring data from the ``From`` data source to the ``To`` data source. Both the ``From`` and the ``To`` are uniquely identified by their corresponding connector Link Ids. i.e when creating a job we have to specifiy the ``FromLinkId`` and the ``ToLinkId``. Thus the pre-requisite for creating a job is to first create the links as described above.
Once the linkIds for the ``From`` and ``To`` are given, then the job configs for the associated connector for the link object have to be filled. You can get the list of all the from and to job config/inputs using `Display Config and Input Names For Connector`_ for that connector. A connector can have one or more links. We then use the links in the ``From`` and ``To`` direction to populate the corresponding ``MFromConfig`` and ``MToConfig`` respectively. Once the link names for the ``From`` and ``To`` are given, then the job configs for the associated connector for the link object have to be filled. You can get the list of all the from and to job config/inputs using `Display Config and Input Names For Connector`_ for that connector. A connector can have one or more links. We then use the links in the ``From`` and ``To`` direction to populate the corresponding ``MFromConfig`` and ``MToConfig`` respectively.
In addition to filling the job configs for the ``From`` and the ``To`` representing the link, we also need to fill the driver configs that control the job execution engine environment. For example, if the job execution engine happens to be the MapReduce we will specifiy the number of mappers to be used in reading data from the ``From`` data source. In addition to filling the job configs for the ``From`` and the ``To`` representing the link, we also need to fill the driver configs that control the job execution engine environment. For example, if the job execution engine happens to be the MapReduce we will specifiy the number of mappers to be used in reading data from the ``From`` data source.
@ -130,9 +129,7 @@ Here is the code to create and then save a job
String url = "http://localhost:12000/sqoop/"; String url = "http://localhost:12000/sqoop/";
SqoopClient client = new SqoopClient(url); SqoopClient client = new SqoopClient(url);
//Creating dummy job object //Creating dummy job object
long fromLinkId = 1;// for jdbc connector MJob job = client.createJob("fromLinkName", "toLinkName");
long toLinkId = 2; // for HDFS connector
MJob job = client.createJob(fromLinkId, toLinkId);
job.setName("Vampire"); job.setName("Vampire");
job.setCreationUser("Buffy"); job.setCreationUser("Buffy");
// set the "FROM" link job config values // set the "FROM" link job config values
@ -149,7 +146,7 @@ Here is the code to create and then save a job
Status status = client.saveJob(job); Status status = client.saveJob(job);
if(status.canProceed()) { if(status.canProceed()) {
System.out.println("Created Job with Job Id: "+ job.getPersistenceId()); System.out.println("Created Job with Job Name: "+ job.getName());
} else { } else {
System.out.println("Something went wrong creating the job"); System.out.println("Something went wrong creating the job");
} }
@ -159,7 +156,7 @@ User can retrieve a job using the following methods
+----------------------------+--------------------------------------+ +----------------------------+--------------------------------------+
| Method | Description | | Method | Description |
+============================+======================================+ +============================+======================================+
| ``getJob(jid)`` | Returns a job by id | | ``getJob(jobName)`` | Returns a job by name |
+----------------------------+--------------------------------------+ +----------------------------+--------------------------------------+
| ``getJobs()`` | Returns list of jobs in the sqoop | | ``getJobs()`` | Returns list of jobs in the sqoop |
+----------------------------+--------------------------------------+ +----------------------------+--------------------------------------+
@ -219,23 +216,22 @@ After creating link or job in the repository, you can update or delete a link or
+==================================+====================================================================================+ +==================================+====================================================================================+
| ``updateLink(link)`` | Invoke update with link and check status for any errors or warnings | | ``updateLink(link)`` | Invoke update with link and check status for any errors or warnings |
+----------------------------------+------------------------------------------------------------------------------------+ +----------------------------------+------------------------------------------------------------------------------------+
| ``deleteLink(lid)`` | Delete link. Deletes only if specified link is not used by any job | | ``deleteLink(linkName)`` | Delete link. Deletes only if specified link is not used by any job |
+----------------------------------+------------------------------------------------------------------------------------+ +----------------------------------+------------------------------------------------------------------------------------+
| ``updateJob(job)`` | Invoke update with job and check status for any errors or warnings | | ``updateJob(job)`` | Invoke update with job and check status for any errors or warnings |
+----------------------------------+------------------------------------------------------------------------------------+ +----------------------------------+------------------------------------------------------------------------------------+
| ``deleteJob(jid)`` | Delete job | | ``deleteJob(jobName)`` | Delete job |
+----------------------------------+------------------------------------------------------------------------------------+ +----------------------------------+------------------------------------------------------------------------------------+
Job Start Job Start
============== ==============
Starting a job requires a job id. On successful start, getStatus() method returns "BOOTING" or "RUNNING". Starting a job requires a job name. On successful start, getStatus() method returns "BOOTING" or "RUNNING".
:: ::
//Job start //Job start
long jobId = 1; MSubmission submission = client.startJob("jobName");
MSubmission submission = client.startJob(jobId);
System.out.println("Job Submission Status : " + submission.getStatus()); System.out.println("Job Submission Status : " + submission.getStatus());
if(submission.getStatus().isRunning() && submission.getProgress() != -1) { if(submission.getStatus().isRunning() && submission.getProgress() != -1) {
System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100)); System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100));
@ -262,15 +258,15 @@ Starting a job requires a job id. On successful start, getStatus() method return
//Check job status for a running job //Check job status for a running job
MSubmission submission = client.getJobStatus(jobId); MSubmission submission = client.getJobStatus("jobName");
if(submission.getStatus().isRunning() && submission.getProgress() != -1) { if(submission.getStatus().isRunning() && submission.getProgress() != -1) {
System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100)); System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100));
} }
//Stop a running job //Stop a running job
submission.stopJob(jobId); submission.stopJob("jobName");
Above code block, job start is asynchronous. For synchronous job start, use ``startJob(jid, callback, pollTime)`` method. If you are not interested in getting the job status, then invoke the same method with "null" as the value for the callback parameter and this returns the final job status. ``pollTime`` is the request interval for getting the job status from sqoop server and the value should be greater than zero. We will frequently hit the sqoop server if a low value is given for the ``pollTime``. When a synchronous job is started with a non null callback, it first invokes the callback's ``submitted(MSubmission)`` method on successful start, after every poll time interval, it then invokes the ``updated(MSubmission)`` method on the callback API and finally on finishing the job executuon it invokes the ``finished(MSubmission)`` method on the callback API. Above code block, job start is asynchronous. For synchronous job start, use ``startJob(jobName, callback, pollTime)`` method. If you are not interested in getting the job status, then invoke the same method with "null" as the value for the callback parameter and this returns the final job status. ``pollTime`` is the request interval for getting the job status from sqoop server and the value should be greater than zero. We will frequently hit the sqoop server if a low value is given for the ``pollTime``. When a synchronous job is started with a non null callback, it first invokes the callback's ``submitted(MSubmission)`` method on successful start, after every poll time interval, it then invokes the ``updated(MSubmission)`` method on the callback API and finally on finishing the job executuon it invokes the ``finished(MSubmission)`` method on the callback API.
Display Config and Input Names For Connector Display Config and Input Names For Connector
============================================ ============================================
@ -281,13 +277,13 @@ You can view the config/input names for the link and job config types per connec
String url = "http://localhost:12000/sqoop/"; String url = "http://localhost:12000/sqoop/";
SqoopClient client = new SqoopClient(url); SqoopClient client = new SqoopClient(url);
long connectorId = 1; String connectorName = "connectorName";
// link config for connector // link config for connector
describe(client.getConnector(connectorId).getLinkConfig().getConfigs(), client.getConnectorConfigBundle(connectorId)); describe(client.getConnector(connectorName).getLinkConfig().getConfigs(), client.getConnectorConfigBundle(connectorName));
// from job config for connector // from job config for connector
describe(client.getConnector(connectorId).getFromConfig().getConfigs(), client.getConnectorConfigBundle(connectorId)); describe(client.getConnector(connectorName).getFromConfig().getConfigs(), client.getConnectorConfigBundle(connectorName));
// to job config for the connector // to job config for the connector
describe(client.getConnector(connectorId).getToConfig().getConfigs(), client.getConnectorConfigBundle(connectorId)); describe(client.getConnector(connectorName).getToConfig().getConfigs(), client.getConnectorConfigBundle(connectorName));
void describe(List<MConfig> configs, ResourceBundle resource) { void describe(List<MConfig> configs, ResourceBundle resource) {
for (MConfig config : configs) { for (MConfig config : configs) {

128
docs/src/site/sphinx/dev/RESTAPI.rst
View File

@ -110,12 +110,13 @@ Each input object in a config is structured below:
To send a filled config in the request, you should always use config id and input id to map the values to their correspondig names. To send a filled config in the request, you should always use config id and input id to map the values to their correspondig names.
For example, the following request contains an input value ``com.mysql.jdbc.Driver`` with input id ``7`` inside a config with id ``4`` that belongs to a link with id ``3`` For example, the following request contains an input value ``com.mysql.jdbc.Driver`` with input id ``7`` inside a config with id ``4`` that belongs to a link with name ``linkName``
:: ::
link: { link: {
id: 3, id : 3,
name: "linkName",
enabled: true, enabled: true,
link-config-values: [{ link-config-values: [{
id: 4, id: 4,
@ -353,10 +354,10 @@ Get all the connectors registered in Sqoop
}] }]
} }
/v1/connector/[cname] or /v1/connector/[cid] - [GET] - Get Connector /v1/connector/[cname] - [GET] - Get Connector
--------------------------------------------------------------------- ---------------------------------------------------------------------
Provide the id or unique name of the connector in the url ``[cid]`` or ``[cname]`` part. Provide the unique name of the connector in the url ``[cname]`` part.
* Method: ``GET`` * Method: ``GET``
* Format: ``JSON`` * Format: ``JSON``
@ -367,7 +368,7 @@ Provide the id or unique name of the connector in the url ``[cid]`` or ``[cname]
+--------------------------+----------------------------------------------------------------------------------------+ +--------------------------+----------------------------------------------------------------------------------------+
| Field | Description | | Field | Description |
+==========================+========================================================================================+ +==========================+========================================================================================+
| ``id`` | The id for the connector ( registered as a configurable ) | | ``name`` | The name for the connector ( registered as a configurable ) |
+--------------------------+----------------------------------------------------------------------------------------+ +--------------------------+----------------------------------------------------------------------------------------+
| ``job-config`` | Connector job config and inputs for both FROM and TO | | ``job-config`` | Connector job config and inputs for both FROM and TO |
+--------------------------+----------------------------------------------------------------------------------------+ +--------------------------+----------------------------------------------------------------------------------------+
@ -385,6 +386,7 @@ Provide the id or unique name of the connector in the url ``[cid]`` or ``[cname]
{ {
connector: { connector: {
id: 1, id: 1,
name: "connectorName",
job-config: { job-config: {
TO: [{ TO: [{
id: 3, id: 3,
@ -550,7 +552,7 @@ Get all the links created in Sqoop
link-config-values: [], link-config-values: [],
name: "First Link", name: "First Link",
creation-date: 1415309361756, creation-date: 1415309361756,
connector-id: 1, connector-name: "connectorName1",
update-date: 1415309361756, update-date: 1415309361756,
creation-user: "root" creation-user: "root"
}, },
@ -561,7 +563,7 @@ Get all the links created in Sqoop
link-config-values: [], link-config-values: [],
name: "Second Link", name: "Second Link",
creation-date: 1415309390807, creation-date: 1415309390807,
connector-id: 2, connector-name: "connectorName2",
update-date: 1415309390807, update-date: 1415309390807,
creation-user: "root" creation-user: "root"
} }
@ -574,12 +576,12 @@ Get all the links created in Sqoop
Get all the links for a given connector identified by ``[cname]`` part. Get all the links for a given connector identified by ``[cname]`` part.
/v1/link/[lname] or /v1/link/[lid] - [GET] - Get Link /v1/link/[lname] - [GET] - Get Link
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Provide the id or unique name of the link in the url ``[lid]`` or ``[lname]`` part. Provide the unique name of the link in the url ``[lname]`` part.
Get all the details of the link including the id, name, type and the corresponding config input values for the link Get all the details of the link including the name, type and the corresponding config input values for the link
* Method: ``GET`` * Method: ``GET``
@ -610,7 +612,7 @@ Get all the details of the link including the id, name, type and the correspondi
update-user: "root", update-user: "root",
name: "First Link", name: "First Link",
creation-date: 1415287846371, creation-date: 1415287846371,
connector-id: 1, connector-name: "connectorName",
update-date: 1415287846371, update-date: 1415287846371,
creation-user: "root" creation-user: "root"
} }
@ -673,7 +675,7 @@ Create a new link object. Provide values to the link config inputs for the ones
update-user: "root", update-user: "root",
name: "testLink", name: "testLink",
creation-date: 1415202223048, creation-date: 1415202223048,
connector-id: 1, connector-name: "connectorName",
update-date: 1415202223048, update-date: 1415202223048,
creation-user: "root" creation-user: "root"
} }
@ -684,7 +686,7 @@ Create a new link object. Provide values to the link config inputs for the ones
+---------------------------+--------------------------------------------------------------------------------------+ +---------------------------+--------------------------------------------------------------------------------------+
| Field | Description | | Field | Description |
+===========================+======================================================================================+ +===========================+======================================================================================+
| ``id`` | The id assigned for this new created link | | ``name`` | The name assigned for this new created link |
+---------------------------+--------------------------------------------------------------------------------------+ +---------------------------+--------------------------------------------------------------------------------------+
| ``validation-result`` | The validation status for the link config inputs given in the post data | | ``validation-result`` | The validation status for the link config inputs given in the post data |
+---------------------------+--------------------------------------------------------------------------------------+ +---------------------------+--------------------------------------------------------------------------------------+
@ -707,10 +709,10 @@ Create a new link object. Provide values to the link config inputs for the ones
} }
/v1/link/[lname] or /v1/link/[lid] - [PUT] - Update Link /v1/link/[lname] - [PUT] - Update Link
--------------------------------------------------------- ---------------------------------------------------------
Update an existing link object with name [lname] or id [lid]. To make the procedure of filling inputs easier, the general practice Update an existing link object with name [lname]. To make the procedure of filling inputs easier, the general practice
is get the link first and then change some of the values for the inputs. is get the link first and then change some of the values for the inputs.
* Method: ``PUT`` * Method: ``PUT``
@ -726,30 +728,30 @@ is get the link first and then change some of the values for the inputs.
] ]
} }
/v1/link/[lname] or /v1/link/[lid] - [DELETE] - Delete Link /v1/link/[lname] - [DELETE] - Delete Link
----------------------------------------------------------------- -----------------------------------------------------------------
Delete a link with name [lname] or id [lid] Delete a link with name [lname]
* Method: ``DELETE`` * Method: ``DELETE``
* Format: ``JSON`` * Format: ``JSON``
* Request Content: ``None`` * Request Content: ``None``
* Response Content: ``None`` * Response Content: ``None``
/v1/link/[lid]/enable or /v1/link/[lname]/enable - [PUT] - Enable Link /v1/link/[lname]/enable - [PUT] - Enable Link
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Enable a link with id ``lid`` or name ``lname`` Enable a link with name ``lname``
* Method: ``PUT`` * Method: ``PUT``
* Format: ``JSON`` * Format: ``JSON``
* Request Content: ``None`` * Request Content: ``None``
* Response Content: ``None`` * Response Content: ``None``
/v1/link/[lid]/disable - [PUT] - Disable Link /v1/link/[lname]/disable - [PUT] - Disable Link
--------------------------------------------------------- ---------------------------------------------------------
Disable a link with id ``lid`` or name ``lname`` Disable a link with name ``lname``
* Method: ``PUT`` * Method: ``PUT``
* Format: ``JSON`` * Format: ``JSON``
@ -773,33 +775,33 @@ Get all the jobs created in Sqoop
jobs: [{ jobs: [{
driver-config-values: [], driver-config-values: [],
enabled: true, enabled: true,
from-connector-id: 1, from-connector-name: "fromConnectorName",
update-user: "root", update-user: "root",
to-config-values: [], to-config-values: [],
to-connector-id: 2, to-connector-name: "toConnectorName",
creation-date: 1415310157618, creation-date: 1415310157618,
update-date: 1415310157618, update-date: 1415310157618,
creation-user: "root", creation-user: "root",
id: 1, id: 1,
to-link-id: 2, to-link-name: "toLinkName",
from-config-values: [], from-config-values: [],
name: "First Job", name: "First Job",
from-link-id: 1 from-link-name: "fromLinkName"
},{ },{
driver-config-values: [], driver-config-values: [],
enabled: true, enabled: true,
from-connector-id: 2, from-connector-name: "fromConnectorName",
update-user: "root", update-user: "root",
to-config-values: [], to-config-values: [],
to-connector-id: 1, to-connector-name: "toConnectorName",
creation-date: 1415310650600, creation-date: 1415310650600,
update-date: 1415310650600, update-date: 1415310650600,
creation-user: "root", creation-user: "root",
id: 2, id: 2,
to-link-id: 1, to-link-name: "toLinkName",
from-config-values: [], from-config-values: [],
name: "Second Job", name: "Second Job",
from-link-id: 2 from-link-name: "fromLinkName"
}] }]
} }
@ -808,11 +810,10 @@ Get all the jobs created in Sqoop
Get all the jobs for a given connector identified by ``[cname]`` part. Get all the jobs for a given connector identified by ``[cname]`` part.
/v1/job/[jname] or /v1/job/[jid] - [GET] - Get Job /v1/job/[jname] - [GET] - Get Job
----------------------------------------------------- -----------------------------------------------------
Provide the name or the id of the job in the url [jname] Provide the name of the job in the url [jname] part.
part or [jid] part.
* Method: ``GET`` * Method: ``GET``
* Format: ``JSON`` * Format: ``JSON``
@ -843,7 +844,7 @@ part or [jid] part.
type: "JOB" type: "JOB"
}], }],
enabled: true, enabled: true,
from-connector-id: 1, from-connector-name: "fromConnectorName",
update-user: "root", update-user: "root",
to-config-values: [{ to-config-values: [{
id: 6, id: 6,
@ -887,12 +888,12 @@ part or [jid] part.
name: "toJobConfig", name: "toJobConfig",
type: "JOB" type: "JOB"
}], }],
to-connector-id: 2, to-connector-name: "toConnectorName",
creation-date: 1415310157618, creation-date: 1415310157618,
update-date: 1415310157618, update-date: 1415310157618,
creation-user: "root", creation-user: "root",
id: 1, id: 1,
to-link-id: 2, to-link-name: "toLinkName",
from-config-values: [{ from-config-values: [{
id: 2, id: 2,
inputs: [{ inputs: [{
@ -907,7 +908,7 @@ part or [jid] part.
type: "JOB" type: "JOB"
}], }],
name: "First Job", name: "First Job",
from-link- id: 1 from-link-name: "fromLinkName"
} }
} }
@ -928,9 +929,9 @@ Create a new job object with the corresponding config values.
+==========================+======================================================================================+ +==========================+======================================================================================+
| ``job`` | The root of the post data in JSON | | ``job`` | The root of the post data in JSON |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``from-link-id`` | The id of the from link for the job | | ``from-link-name`` | The name of the from link for the job |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``to-link-id`` | The id of the to link for the job | | ``to-link-name`` | The name of the to link for the job |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``id`` | The id of the link can be left blank in the post data | | ``id`` | The id of the link can be left blank in the post data |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
@ -952,9 +953,10 @@ Create a new job object with the corresponding config values.
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``driver-config-values`` | Config input values for driver | | ``driver-config-values`` | Config input values for driver |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``connector-id`` | The id of the connector used for this link | | ``from-connector-name`` | The name of the from connector for the job |
+--------------------------+--------------------------------------------------------------------------------------+
| ``to-connector-name`` | The name of the to connector for the job |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
* Request Example: * Request Example:
@ -986,7 +988,7 @@ Create a new job object with the corresponding config values.
} }
], ],
enabled: true, enabled: true,
from-connector-id: 1, from-connector-name: "fromConnectorName",
update-user: "root", update-user: "root",
to-config-values: [ to-config-values: [
{ {
@ -1039,12 +1041,12 @@ Create a new job object with the corresponding config values.
type: "JOB" type: "JOB"
} }
], ],
to-connector-id: 2, to-connector-name: "toConnectorName",
creation-date: 1415310157618, creation-date: 1415310157618,
update-date: 1415310157618, update-date: 1415310157618,
creation-user: "root", creation-user: "root",
id: -1, id: -1,
to-link-id: 2, to-link-name: "toLinkName",
from-config-values: [ from-config-values: [
{ {
id: 2, id: 2,
@ -1063,7 +1065,7 @@ Create a new job object with the corresponding config values.
} }
], ],
name: "Test Job", name: "Test Job",
from-link-id: 1 from-link-name: "fromLinkName"
} }
} }
@ -1072,7 +1074,7 @@ Create a new job object with the corresponding config values.
+---------------------------+--------------------------------------------------------------------------------------+ +---------------------------+--------------------------------------------------------------------------------------+
| Field | Description | | Field | Description |
+===========================+======================================================================================+ +===========================+======================================================================================+
| ``id`` | The id assigned for this new created job | | ``name`` | The name assigned for this new created job |
+--------------------------+---------------------------------------------------------------------------------------+ +--------------------------+---------------------------------------------------------------------------------------+
| ``validation-result`` | The validation status for the job config and driver config inputs in the post data | | ``validation-result`` | The validation status for the job config and driver config inputs in the post data |
+---------------------------+--------------------------------------------------------------------------------------+ +---------------------------+--------------------------------------------------------------------------------------+
@ -1096,10 +1098,10 @@ Create a new job object with the corresponding config values.
} }
/v1/job/[jid] - [PUT] - Update Job /v1/job/[jname] - [PUT] - Update Job
--------------------------------------------------------- ---------------------------------------------------------
Update an existing job object with id [jid]. To make the procedure of filling inputs easier, the general practice Update an existing job object with name [jname]. To make the procedure of filling inputs easier, the general practice
is get the existing job object first and then change some of the inputs. is get the existing job object first and then change some of the inputs.
* Method: ``PUT`` * Method: ``PUT``
@ -1118,30 +1120,30 @@ The same as Create Job.
} }
/v1/job/[jid] - [DELETE] - Delete Job /v1/job/[jname] - [DELETE] - Delete Job
--------------------------------------------------------- ---------------------------------------------------------
Delete a job with id ``jid``. Delete a job with name ``jname``.
* Method: ``DELETE`` * Method: ``DELETE``
* Format: ``JSON`` * Format: ``JSON``
* Request Content: ``None`` * Request Content: ``None``
* Response Content: ``None`` * Response Content: ``None``
/v1/job/[jid]/enable - [PUT] - Enable Job /v1/job/[jname]/enable - [PUT] - Enable Job
--------------------------------------------------------- ---------------------------------------------------------
Enable a job with id ``jid``. Enable a job with name ``jname``.
* Method: ``PUT`` * Method: ``PUT``
* Format: ``JSON`` * Format: ``JSON``
* Request Content: ``None`` * Request Content: ``None``
* Response Content: ``None`` * Response Content: ``None``
/v1/job/[jid]/disable - [PUT] - Disable Job /v1/job/[jname]/disable - [PUT] - Disable Job
--------------------------------------------------------- ---------------------------------------------------------
Disable a job with id ``jid``. Disable a job with name ``jname``.
* Method: ``PUT`` * Method: ``PUT``
* Format: ``JSON`` * Format: ``JSON``
@ -1149,10 +1151,10 @@ Disable a job with id ``jid``.
* Response Content: ``None`` * Response Content: ``None``
/v1/job/[jid]/start or /v1/job/[jname]/start - [PUT]- Start Job /v1/job/[jname]/start - [PUT]- Start Job
--------------------------------------------------------------------------------- ---------------------------------------------------------------------------------
Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execution Start a job with name ``[jname]`` to trigger the job execution
* Method: ``POST`` * Method: ``POST``
* Format: ``JSON`` * Format: ``JSON``
@ -1170,6 +1172,7 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
"external-id": "job_1412137947693_0004", "external-id": "job_1412137947693_0004",
"status": "BOOTING", "status": "BOOTING",
"job": 2, "job": 2,
"job-name": "jobName",
"creation-date": 1415312531188, "creation-date": 1415312531188,
"to-schema": { "to-schema": {
"created": 1415312531426, "created": 1415312531426,
@ -1210,6 +1213,7 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
external-id: "job_1412137947693_0004", external-id: "job_1412137947693_0004",
status: "SUCCEEDED", status: "SUCCEEDED",
job: 2, job: 2,
job-name: "jobName",
creation-date: 1415312531188, creation-date: 1415312531188,
external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/", external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/",
counters: { counters: {
@ -1271,6 +1275,7 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
"status": "FAILURE_ON_SUBMIT", "status": "FAILURE_ON_SUBMIT",
"error-summary": "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run", "error-summary": "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run",
"job": 1, "job": 1,
"job-name": "jobName",
"creation-date": 1415312390570, "creation-date": 1415312390570,
"to-schema": { "to-schema": {
"created": 1415312390797, "created": 1415312390797,
@ -1301,20 +1306,20 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
} }
} }
/v1/job/[jid]/stop or /v1/job/[jname]/stop - [PUT]- Stop Job /v1/job/[jname]/stop - [PUT]- Stop Job
--------------------------------------------------------------------------------- ---------------------------------------------------------------------------------
Stop a job with name ``[janme]`` or with id ``[jid]`` to abort the running job. Stop a job with name ``[jname]`` to abort the running job.
* Method: ``PUT`` * Method: ``PUT``
* Format: ``JSON`` * Format: ``JSON``
* Request Content: ``None`` * Request Content: ``None``
* Response Content: ``Submission Record`` * Response Content: ``Submission Record``
/v1/job/[jid]/status or /v1/job/[jname]/status - [GET]- Get Job Status /v1/job/[jname]/status - [GET]- Get Job Status
--------------------------------------------------------------------------------- ---------------------------------------------------------------------------------
Get status of the running job with name ``[janme]`` or with id ``[jid]`` Get status of the running job with name ``[jname]``
* Method: ``GET`` * Method: ``GET``
* Format: ``JSON`` * Format: ``JSON``
@ -1330,6 +1335,7 @@ Get status of the running job with name ``[janme]`` or with id ``[jid]``
"external-id": "job_1412137947693_0004", "external-id": "job_1412137947693_0004",
"status": "RUNNING", "status": "RUNNING",
"job": 2, "job": 2,
"job-name": "jobName",
"creation-date": 1415312531188, "creation-date": 1415312531188,
"external-link": "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/" "external-link": "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/"
} }
@ -1359,6 +1365,8 @@ Provide the name of the job in the url [jname] part.
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``job`` | The id of the Sqoop job | | ``job`` | The id of the Sqoop job |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``job-name`` | The name of the Sqoop job |
+--------------------------+--------------------------------------------------------------------------------------+
| ``creation-date`` | The submission timestamp | | ``creation-date`` | The submission timestamp |
+--------------------------+--------------------------------------------------------------------------------------+ +--------------------------+--------------------------------------------------------------------------------------+
| ``last-update-date`` | The timestamp of the last status update | | ``last-update-date`` | The timestamp of the last status update |
@ -1382,6 +1390,7 @@ Provide the name of the job in the url [jname] part.
external-id: "job_1412137947693_0004", external-id: "job_1412137947693_0004",
status: "SUCCEEDED", status: "SUCCEEDED",
job: 2, job: 2,
job-name: "jobName",
creation-date: 1415312531188, creation-date: 1415312531188,
external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/", external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/",
counters: { counters: {
@ -1435,6 +1444,7 @@ Provide the name of the job in the url [jname] part.
status: "FAILURE_ON_SUBMIT", status: "FAILURE_ON_SUBMIT",
error-summary: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run", error-summary: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run",
job: 1, job: 1,
job-name: "jobName",
creation-date: 1415312390570, creation-date: 1415312390570,
error-details: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner...." error-details: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner...."
} }

140
docs/src/site/sphinx/user/CommandLineClient.rst
View File

@ -34,7 +34,7 @@ Sqoop client script is expected to contain valid Sqoop client commands, empty li
set server --host sqoop2.company.net set server --host sqoop2.company.net
# Executing given job # Executing given job
start job --jid 1 start job --name 1
.. contents:: Table of Contents .. contents:: Table of Contents
@ -269,12 +269,12 @@ Show persisted link objects.
+=======================+======================================================+ +=======================+======================================================+
| ``-a``, ``--all`` | Show all available links | | ``-a``, ``--all`` | Show all available links |
+-----------------------+------------------------------------------------------+ +-----------------------+------------------------------------------------------+
| ``-x``, ``--lid <x>`` | Show link with id ``<x>`` | | ``-n``, ``--name <x>``| Show link with name ``<x>`` |
+-----------------------+------------------------------------------------------+ +-----------------------+------------------------------------------------------+
Example: :: Example: ::
show link --all or show link show link --all or show link --name linkName
Show Job Function Show Job Function
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
@ -286,31 +286,31 @@ Show persisted job objects.
+=======================+==============================================+ +=======================+==============================================+
| ``-a``, ``--all`` | Show all available jobs | | ``-a``, ``--all`` | Show all available jobs |
+-----------------------+----------------------------------------------+ +-----------------------+----------------------------------------------+
| ``-j``, ``--jid <x>`` | Show job with id ``<x>`` | | ``-n``, ``--name <x>``| Show job with name ``<x>`` |
+-----------------------+----------------------------------------------+ +-----------------------+----------------------------------------------+
Example: :: Example: ::
show job --all or show job show job --all or show job --name jobName
Show Submission Function Show Submission Function
~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~
Show persisted job submission objects. Show persisted job submission objects.
+-----------------------+---------------------------------------------+ +-----------------------+-----------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+=============================================+ +=======================+===============================================+
| ``-j``, ``--jid <x>`` | Show available submissions for given job | | ``-j``, ``--job <x>`` | Show available submissions for given job name |
+-----------------------+---------------------------------------------+ +-----------------------+-----------------------------------------------+
| ``-d``, ``--detail`` | Show job submissions in full details | | ``-d``, ``--detail`` | Show job submissions in full details |
+-----------------------+---------------------------------------------+ +-----------------------+-----------------------------------------------+
Example: :: Example: ::
show submission show submission
show submission --jid 1 show submission --j jobName
show submission --jid 1 --detail show submission --job jobName --detail
Create Command Create Command
-------------- --------------
@ -332,16 +332,16 @@ Create Link Function
Create new link object. Create new link object.
+------------------------+-------------------------------------------------------------+ +------------------------------+-------------------------------------------------------------+
| Argument | Description | | Argument | Description |
+========================+=============================================================+ +==============================+=============================================================+
| ``-c``, ``--cid <x>`` | Create new link object for connector with id ``<x>`` | | ``-c``, ``--connector <x>`` | Create new link object for connector with name ``<x>`` |
+------------------------+-------------------------------------------------------------+ +------------------------------+-------------------------------------------------------------+
Example: :: Example: ::
create link --cid 1 or create link -c 1 create link --connector connectorName or create link -c connectorName
Create Job Function Create Job Function
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
@ -351,14 +351,14 @@ Create new job object.
+------------------------+------------------------------------------------------------------+ +------------------------+------------------------------------------------------------------+
| Argument | Description | | Argument | Description |
+========================+==================================================================+ +========================+==================================================================+
| ``-f``, ``--from <x>`` | Create new job object with a FROM link with id ``<x>`` | | ``-f``, ``--from <x>`` | Create new job object with a FROM link with name ``<x>`` |
+------------------------+------------------------------------------------------------------+ +------------------------+------------------------------------------------------------------+
| ``-t``, ``--to <t>`` | Create new job object with a TO link with id ``<x>`` | | ``-t``, ``--to <t>`` | Create new job object with a TO link with name ``<x>`` |
+------------------------+------------------------------------------------------------------+ +------------------------+------------------------------------------------------------------+
Example: :: Example: ::
create job --from 1 --to 2 or create job --f 1 --t 2 create job --from fromLinkName --to toLinkName or create job --f fromLinkName --t toLinkName
Update Command Update Command
-------------- --------------
@ -373,27 +373,27 @@ Update existing link object.
+-----------------------+---------------------------------------------+ +-----------------------+---------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+=============================================+ +=======================+=============================================+
| ``-x``, ``--lid <x>`` | Update existing link with id ``<x>`` | | ``-n``, ``--name <x>``| Update existing link with name ``<x>`` |
+-----------------------+---------------------------------------------+ +-----------------------+---------------------------------------------+
Example: :: Example: ::
update link --lid 1 update link --name linkName
Update Job Function Update Job Function
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
Update existing job object. Update existing job object.
+-----------------------+--------------------------------------------+ +-------------------------+----------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+============================================+ +=========================+==============================================+
| ``-j``, ``--jid <x>`` | Update existing job object with id ``<x>`` | | ``-n``, ``--name <x>`` | Update existing job object with name ``<x>`` |
+-----------------------+--------------------------------------------+ +-------------------------+----------------------------------------------+
Example: :: Example: ::
update job --jid 1 update job --name jobName
Delete Command Delete Command
@ -406,15 +406,15 @@ Delete Link Function
Delete existing link object. Delete existing link object.
+-----------------------+-------------------------------------------+ +-------------------------+-------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+===========================================+ +=========================+===========================================+
| ``-x``, ``--lid <x>`` | Delete link object with id ``<x>`` | | ``-n``, ``--name <x>`` | Delete link object with name ``<x>`` |
+-----------------------+-------------------------------------------+ +-------------------------+-------------------------------------------+
Example: :: Example: ::
delete link --lid 1 delete link --name linkName
Delete Job Function Delete Job Function
@ -422,15 +422,15 @@ Delete Job Function
Delete existing job object. Delete existing job object.
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+==========================================+ +=========================+==========================================+
| ``-j``, ``--jid <x>`` | Delete job object with id ``<x>`` | | ``-n``, ``--name <x>`` | Delete job object with name ``<x>`` |
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
Example: :: Example: ::
delete job --jid 1 delete job --name jobName
Clone Command Clone Command
@ -443,15 +443,15 @@ Clone Link Function
Clone existing link object. Clone existing link object.
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+==========================================+ +=========================+==========================================+
| ``-x``, ``--lid <x>`` | Clone link object with id ``<x>`` | | ``-n``, ``--name <x>`` | Clone link object with name ``<x>`` |
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
Example: :: Example: ::
clone link --lid 1 clone link --name linkName
Clone Job Function Clone Job Function
@ -459,15 +459,15 @@ Clone Job Function
Clone existing job object. Clone existing job object.
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+==========================================+ +=========================+==========================================+
| ``-j``, ``--jid <x>`` | Clone job object with id ``<x>`` | | ``-n``, ``--name <x>`` | Clone job object with name ``<x>`` |
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
Example: :: Example: ::
clone job --jid 1 clone job --name jobName
Start Command Start Command
------------- -------------
@ -482,15 +482,15 @@ Start job (submit new submission). Starting already running job is considered as
+----------------------------+----------------------------+ +----------------------------+----------------------------+
| Argument | Description | | Argument | Description |
+============================+============================+ +============================+============================+
| ``-j``, ``--jid <x>`` | Start job with id ``<x>`` | | ``-n``, ``--name <x>`` | Start job with name ``<x>``|
+----------------------------+----------------------------+ +----------------------------+----------------------------+
| ``-s``, ``--synchronous`` | Synchoronous job execution | | ``-s``, ``--synchronous`` | Synchoronous job execution |
+----------------------------+----------------------------+ +----------------------------+----------------------------+
Example: :: Example: ::
start job --jid 1 start job --name jobName
start job --jid 1 --synchronous start job --name jobName --synchronous
Stop Command Stop Command
------------ ------------
@ -502,15 +502,15 @@ Stop Job Function
Interrupt running job. Interrupt running job.
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+==========================================+ +=========================+==========================================+
| ``-j``, ``--jid <x>`` | Interrupt running job with id ``<x>`` | | ``-n``, ``--name <x>`` | Interrupt running job with name ``<x>`` |
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
Example: :: Example: ::
stop job --jid 1 stop job --name jobName
Status Command Status Command
-------------- --------------
@ -522,12 +522,12 @@ Status Job Function
Retrieve last status for given job. Retrieve last status for given job.
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
| Argument | Description | | Argument | Description |
+=======================+==========================================+ +=========================+==========================================+
| ``-j``, ``--jid <x>`` | Retrieve status for job with id ``<x>`` | | ``-n``, ``--name <x>`` | Retrieve status for job with name ``<x>``|
+-----------------------+------------------------------------------+ +-------------------------+------------------------------------------+
Example: :: Example: ::
status job --jid 1 status job --name jobName

64
docs/src/site/sphinx/user/Sqoop5MinutesDemo.rst
View File

@ -76,20 +76,20 @@ Creating Link Object
Check for the registered connectors on your Sqoop server: :: Check for the registered connectors on your Sqoop server: ::
sqoop:000> show connector sqoop:000> show connector
+----+------------------------+----------------+------------------------------------------------------+----------------------+ +------------------------+----------------+------------------------------------------------------+----------------------+
| Id | Name | Version | Class | Supported Directions | | Name | Version | Class | Supported Directions |
+----+------------------------+----------------+------------------------------------------------------+----------------------+ +------------------------+----------------+------------------------------------------------------+----------------------+
| 1 | hdfs-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO | | hdfs-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO |
| 2 | generic-jdbc-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO | | generic-jdbc-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO |
+----+------------------------+----------------+------------------------------------------------------+----------------------+ +------------------------+----------------+------------------------------------------------------+----------------------+
Our example contains two connectors. The one with connector Id 2 is called the ``generic-jdbc-connector``. This is a basic connector relying on the Java JDBC interface for communicating with data sources. It should work with the most common databases that are providing JDBC drivers. Please note that you must install JDBC drivers separately. They are not bundled in Sqoop due to incompatible licenses. Our example contains two connectors. The ``generic-jdbc-connector`` is a basic connector relying on the Java JDBC interface for communicating with data sources. It should work with the most common databases that are providing JDBC drivers. Please note that you must install JDBC drivers separately. They are not bundled in Sqoop due to incompatible licenses.
Generic JDBC Connector in our example has a persistence Id 2 and we will use this value to create new link object for this connector. Note that the link name should be unique. Generic JDBC Connector in our example has a name ``generic-jdbc-connector`` and we will use this value to create new link object for this connector. Note that the link name should be unique.
:: ::
sqoop:000> create link -c 2 sqoop:000> create link -connector generic-jdbc-connector
Creating link for connector with id 2 Creating link for connector with name generic-jdbc-connector
Please fill following values to create new link object Please fill following values to create new link object
Name: First Link Name: First Link
@ -101,22 +101,22 @@ Generic JDBC Connector in our example has a persistence Id 2 and we will use thi
JDBC Connection Properties: JDBC Connection Properties:
There are currently 0 values in the map: There are currently 0 values in the map:
entry#protocol=tcp entry#protocol=tcp
New link was successfully created with validation status OK and persistent id 1 New link was successfully created with validation status OK name First Link
Our new link object was created with assigned id 1. Our new link object was created with assigned name First Link.
In the ``show connector -all`` we see that there is a hdfs-connector registered in sqoop with the persistent id 1. Let us create another link object but this time for the hdfs-connector instead. In the ``show connector -all`` we see that there is a hdfs-connector registered. Let us create another link object but this time for the hdfs-connector instead.
:: ::
sqoop:000> create link -c 1 sqoop:000> create link -connector hdfs-connector
Creating link for connector with id 1 Creating link for connector with name hdfs-connector
Please fill following values to create new link object Please fill following values to create new link object
Name: Second Link Name: Second Link
Link configuration Link configuration
HDFS URI: hdfs://nameservice1:8020/ HDFS URI: hdfs://nameservice1:8020/
New link was successfully created with validation status OK and persistent id 2 New link was successfully created with validation status OK and name Second Link
Creating Job Object Creating Job Object
=================== ===================
@ -127,8 +127,8 @@ Connectors implement the ``From`` for reading data from and/or ``To`` for writin
sqoop:000> show link --all sqoop:000> show link --all
2 link(s) to show: 2 link(s) to show:
link with id 1 and name First Link (Enabled: true, Created by root at 11/4/14 4:27 PM, Updated by root at 11/4/14 4:27 PM) link with name First Link (Enabled: true, Created by root at 11/4/14 4:27 PM, Updated by root at 11/4/14 4:27 PM)
Using Connector id 2 Using Connector with name generic-jdbc-connector
Link configuration Link configuration
JDBC Driver Class: com.mysql.jdbc.Driver JDBC Driver Class: com.mysql.jdbc.Driver
JDBC Connection String: jdbc:mysql://mysql.ent.cloudera.com/sqoop JDBC Connection String: jdbc:mysql://mysql.ent.cloudera.com/sqoop
@ -136,16 +136,16 @@ Connectors implement the ``From`` for reading data from and/or ``To`` for writin
Password: Password:
JDBC Connection Properties: JDBC Connection Properties:
protocol = tcp protocol = tcp
link with id 2 and name Second Link (Enabled: true, Created by root at 11/4/14 4:38 PM, Updated by root at 11/4/14 4:38 PM) link with name Second Link (Enabled: true, Created by root at 11/4/14 4:38 PM, Updated by root at 11/4/14 4:38 PM)
Using Connector id 1 Using Connector with name hdfs-connector
Link configuration Link configuration
HDFS URI: hdfs://nameservice1:8020/ HDFS URI: hdfs://nameservice1:8020/
Next, we can use the two link Ids to associate the ``From`` and ``To`` for the job. Next, we can use the two link names to associate the ``From`` and ``To`` for the job.
:: ::
sqoop:000> create job -f 1 -t 2 sqoop:000> create job -f "First Link" -t "Second Link"
Creating job for links with from id 1 and to id 2 Creating job for links with from name First Link and to name Second Link
Please fill following values to create new job object Please fill following values to create new job object
Name: Sqoopy Name: Sqoopy
@ -182,9 +182,9 @@ Next, we can use the two link Ids to associate the ``From`` and ``To`` for the j
Driver Config Driver Config
Extractors:(Optional) 2 Extractors:(Optional) 2
Loaders:(Optional) 2 Loaders:(Optional) 2
New job was successfully created with validation status OK and persistent id 1 New job was successfully created with validation status OK and name jobName
Our new job object was created with assigned id 1. Note that if null value is allowed for the partition column, Our new job object was created with assigned name Sqoopy. Note that if null value is allowed for the partition column,
at least 2 extractors are needed for Sqoop to carry out the data transfer. On specifying 1 extractor in this at least 2 extractors are needed for Sqoop to carry out the data transfer. On specifying 1 extractor in this
scenario, Sqoop shall ignore this setting and continue with 2 extractors. scenario, Sqoop shall ignore this setting and continue with 2 extractors.
@ -194,9 +194,9 @@ Start Job ( a.k.a Data transfer )
You can start a sqoop job with the following command: You can start a sqoop job with the following command:
:: ::
sqoop:000> start job -j 1 sqoop:000> start job -name Sqoopy
Submission details Submission details
Job ID: 1 Job Name: Sqoopy
Server URL: http://localhost:12000/sqoop/ Server URL: http://localhost:12000/sqoop/
Created by: root Created by: root
Creation date: 2014-11-04 19:43:29 PST Creation date: 2014-11-04 19:43:29 PST
@ -209,9 +209,9 @@ You can iteratively check your running job status with ``status job`` command:
:: ::
sqoop:000> status job -j 1 sqoop:000> status job -n Sqoopy
Submission details Submission details
Job ID: 1 Job Name: Sqoopy
Server URL: http://localhost:12000/sqoop/ Server URL: http://localhost:12000/sqoop/
Created by: root Created by: root
Creation date: 2014-11-04 19:43:29 PST Creation date: 2014-11-04 19:43:29 PST
@ -224,9 +224,9 @@ Alternatively you can start a sqoop job and observe job running status with the
:: ::
sqoop:000> start job -j 1 -s sqoop:000> start job -n Sqoopy -s
Submission details Submission details
Job ID: 1 Job Name: Sqoopy
Server URL: http://localhost:12000/sqoop/ Server URL: http://localhost:12000/sqoop/
Created by: root Created by: root
Creation date: 2014-11-04 19:43:29 PST Creation date: 2014-11-04 19:43:29 PST
@ -239,4 +239,4 @@ Alternatively you can start a sqoop job and observe job running status with the
And finally you can stop running the job at any time using ``stop job`` command: :: And finally you can stop running the job at any time using ``stop job`` command: ::
sqoop:000> stop job -j 1 sqoop:000> stop job -n Sqoopy