mirror of
https://github.com/apache/sqoop.git
synced 2025-05-04 12:40:34 +08:00
SQOOP-2736: Sqoop2: Document change command line shell interface
(Colin Ma via Jarek Jarcec Cecho)
This commit is contained in:
Jarek Jarcec Cecho
parent
d35efd3c3a
commit
d89f64adc8
@ -34,9 +34,9 @@ Workflow
|
||||
|
||||
Given workflow has to be followed for executing a sqoop job in Sqoop server.
|
||||
|
||||
1. Create LINK object for a given connectorId - Creates Link object and returns linkId (lid)
|
||||
2. Create a JOB for a given "from" and "to" linkId - Create Job object and returns jobId (jid)
|
||||
3. Start the JOB for a given jobId - Start Job on the server and creates a submission record
|
||||
1. Create LINK object for a given connector name - Creates Link object and returns it
|
||||
2. Create a JOB for a given "from" and "to" link name - Create Job object and returns it
|
||||
3. Start the JOB for a given job name - Start Job on the server and creates a submission record
|
||||
|
||||
Project Dependencies
|
||||
====================
|
||||
@ -76,13 +76,12 @@ you can get the list of all the config/inputs using `Display Config and Input Na
|
||||
Save Link
|
||||
---------
|
||||
|
||||
First create a new link by invoking ``createLink(cid)`` method with connector Id and it returns a MLink object with dummy id and the unfilled link config inputs for that connector. Then fill the config inputs with relevant values. Invoke ``saveLink`` passing it the filled MLink object.
|
||||
First create a new link by invoking ``createLink(connectorName)`` method with connector name and it returns a MLink object with dummy id and the unfilled link config inputs for that connector. Then fill the config inputs with relevant values. Invoke ``saveLink`` passing it the filled MLink object.
|
||||
|
||||
::
|
||||
|
||||
// create a placeholder for link
|
||||
long connectorId = 1;
|
||||
MLink link = client.createLink(connectorId);
|
||||
MLink link = client.createLink("connectorName");
|
||||
link.setName("Vampire");
|
||||
link.setCreationUser("Buffy");
|
||||
MLinkConfig linkConfig = link.getConnectorLinkConfig();
|
||||
@ -94,21 +93,21 @@ First create a new link by invoking ``createLink(cid)`` method with connector Id
|
||||
// save the link object that was filled
|
||||
Status status = client.saveLink(link);
|
||||
if(status.canProceed()) {
|
||||
System.out.println("Created Link with Link Id : " + link.getPersistenceId());
|
||||
System.out.println("Created Link with Link Name : " + link.getName());
|
||||
} else {
|
||||
System.out.println("Something went wrong creating the link");
|
||||
}
|
||||
|
||||
``status.canProceed()`` returns true if status is OK or a WARNING. Before sending the status, the link config values are validated using the corresponding validator associated with th link config inputs.
|
||||
|
||||
On successful execution of the saveLink method, new link Id is assigned to the link object else an exception is thrown. ``link.getPersistenceId()`` method returns the unique Id for this object persisted in the sqoop repository.
|
||||
On successful execution of the saveLink method, new link name is assigned to the link object else an exception is thrown. ``link.getName()`` method returns the unique name for this object persisted in the sqoop repository.
|
||||
|
||||
User can retrieve a link using the following methods
|
||||
|
||||
+----------------------------+--------------------------------------+
|
||||
| Method | Description |
|
||||
+============================+======================================+
|
||||
| ``getLink(lid)`` | Returns a link by id |
|
||||
| ``getLink(linkName)`` | Returns a link by name |
|
||||
+----------------------------+--------------------------------------+
|
||||
| ``getLinks()`` | Returns list of links in the sqoop |
|
||||
+----------------------------+--------------------------------------+
|
||||
@ -118,7 +117,7 @@ Job
|
||||
|
||||
A sqoop job holds the ``From`` and ``To`` parts for transferring data from the ``From`` data source to the ``To`` data source. Both the ``From`` and the ``To`` are uniquely identified by their corresponding connector Link Ids. i.e when creating a job we have to specifiy the ``FromLinkId`` and the ``ToLinkId``. Thus the pre-requisite for creating a job is to first create the links as described above.
|
||||
|
||||
Once the linkIds for the ``From`` and ``To`` are given, then the job configs for the associated connector for the link object have to be filled. You can get the list of all the from and to job config/inputs using `Display Config and Input Names For Connector`_ for that connector. A connector can have one or more links. We then use the links in the ``From`` and ``To`` direction to populate the corresponding ``MFromConfig`` and ``MToConfig`` respectively.
|
||||
Once the link names for the ``From`` and ``To`` are given, then the job configs for the associated connector for the link object have to be filled. You can get the list of all the from and to job config/inputs using `Display Config and Input Names For Connector`_ for that connector. A connector can have one or more links. We then use the links in the ``From`` and ``To`` direction to populate the corresponding ``MFromConfig`` and ``MToConfig`` respectively.
|
||||
|
||||
In addition to filling the job configs for the ``From`` and the ``To`` representing the link, we also need to fill the driver configs that control the job execution engine environment. For example, if the job execution engine happens to be the MapReduce we will specifiy the number of mappers to be used in reading data from the ``From`` data source.
|
||||
|
||||
@ -130,9 +129,7 @@ Here is the code to create and then save a job
|
||||
String url = "http://localhost:12000/sqoop/";
|
||||
SqoopClient client = new SqoopClient(url);
|
||||
//Creating dummy job object
|
||||
long fromLinkId = 1;// for jdbc connector
|
||||
long toLinkId = 2; // for HDFS connector
|
||||
MJob job = client.createJob(fromLinkId, toLinkId);
|
||||
MJob job = client.createJob("fromLinkName", "toLinkName");
|
||||
job.setName("Vampire");
|
||||
job.setCreationUser("Buffy");
|
||||
// set the "FROM" link job config values
|
||||
@ -149,7 +146,7 @@ Here is the code to create and then save a job
|
||||
|
||||
Status status = client.saveJob(job);
|
||||
if(status.canProceed()) {
|
||||
System.out.println("Created Job with Job Id: "+ job.getPersistenceId());
|
||||
System.out.println("Created Job with Job Name: "+ job.getName());
|
||||
} else {
|
||||
System.out.println("Something went wrong creating the job");
|
||||
}
|
||||
@ -159,7 +156,7 @@ User can retrieve a job using the following methods
|
||||
+----------------------------+--------------------------------------+
|
||||
| Method | Description |
|
||||
+============================+======================================+
|
||||
| ``getJob(jid)`` | Returns a job by id |
|
||||
| ``getJob(jobName)`` | Returns a job by name |
|
||||
+----------------------------+--------------------------------------+
|
||||
| ``getJobs()`` | Returns list of jobs in the sqoop |
|
||||
+----------------------------+--------------------------------------+
|
||||
@ -219,23 +216,22 @@ After creating link or job in the repository, you can update or delete a link or
|
||||
+==================================+====================================================================================+
|
||||
| ``updateLink(link)`` | Invoke update with link and check status for any errors or warnings |
|
||||
+----------------------------------+------------------------------------------------------------------------------------+
|
||||
| ``deleteLink(lid)`` | Delete link. Deletes only if specified link is not used by any job |
|
||||
| ``deleteLink(linkName)`` | Delete link. Deletes only if specified link is not used by any job |
|
||||
+----------------------------------+------------------------------------------------------------------------------------+
|
||||
| ``updateJob(job)`` | Invoke update with job and check status for any errors or warnings |
|
||||
+----------------------------------+------------------------------------------------------------------------------------+
|
||||
| ``deleteJob(jid)`` | Delete job |
|
||||
| ``deleteJob(jobName)`` | Delete job |
|
||||
+----------------------------------+------------------------------------------------------------------------------------+
|
||||
|
||||
Job Start
|
||||
==============
|
||||
|
||||
Starting a job requires a job id. On successful start, getStatus() method returns "BOOTING" or "RUNNING".
|
||||
Starting a job requires a job name. On successful start, getStatus() method returns "BOOTING" or "RUNNING".
|
||||
|
||||
::
|
||||
|
||||
//Job start
|
||||
long jobId = 1;
|
||||
MSubmission submission = client.startJob(jobId);
|
||||
MSubmission submission = client.startJob("jobName");
|
||||
System.out.println("Job Submission Status : " + submission.getStatus());
|
||||
if(submission.getStatus().isRunning() && submission.getProgress() != -1) {
|
||||
System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100));
|
||||
@ -262,15 +258,15 @@ Starting a job requires a job id. On successful start, getStatus() method return
|
||||
|
||||
|
||||
//Check job status for a running job
|
||||
MSubmission submission = client.getJobStatus(jobId);
|
||||
MSubmission submission = client.getJobStatus("jobName");
|
||||
if(submission.getStatus().isRunning() && submission.getProgress() != -1) {
|
||||
System.out.println("Progress : " + String.format("%.2f %%", submission.getProgress() * 100));
|
||||
}
|
||||
|
||||
//Stop a running job
|
||||
submission.stopJob(jobId);
|
||||
submission.stopJob("jobName");
|
||||
|
||||
Above code block, job start is asynchronous. For synchronous job start, use ``startJob(jid, callback, pollTime)`` method. If you are not interested in getting the job status, then invoke the same method with "null" as the value for the callback parameter and this returns the final job status. ``pollTime`` is the request interval for getting the job status from sqoop server and the value should be greater than zero. We will frequently hit the sqoop server if a low value is given for the ``pollTime``. When a synchronous job is started with a non null callback, it first invokes the callback's ``submitted(MSubmission)`` method on successful start, after every poll time interval, it then invokes the ``updated(MSubmission)`` method on the callback API and finally on finishing the job executuon it invokes the ``finished(MSubmission)`` method on the callback API.
|
||||
Above code block, job start is asynchronous. For synchronous job start, use ``startJob(jobName, callback, pollTime)`` method. If you are not interested in getting the job status, then invoke the same method with "null" as the value for the callback parameter and this returns the final job status. ``pollTime`` is the request interval for getting the job status from sqoop server and the value should be greater than zero. We will frequently hit the sqoop server if a low value is given for the ``pollTime``. When a synchronous job is started with a non null callback, it first invokes the callback's ``submitted(MSubmission)`` method on successful start, after every poll time interval, it then invokes the ``updated(MSubmission)`` method on the callback API and finally on finishing the job executuon it invokes the ``finished(MSubmission)`` method on the callback API.
|
||||
|
||||
Display Config and Input Names For Connector
|
||||
============================================
|
||||
@ -281,13 +277,13 @@ You can view the config/input names for the link and job config types per connec
|
||||
|
||||
String url = "http://localhost:12000/sqoop/";
|
||||
SqoopClient client = new SqoopClient(url);
|
||||
long connectorId = 1;
|
||||
String connectorName = "connectorName";
|
||||
// link config for connector
|
||||
describe(client.getConnector(connectorId).getLinkConfig().getConfigs(), client.getConnectorConfigBundle(connectorId));
|
||||
describe(client.getConnector(connectorName).getLinkConfig().getConfigs(), client.getConnectorConfigBundle(connectorName));
|
||||
// from job config for connector
|
||||
describe(client.getConnector(connectorId).getFromConfig().getConfigs(), client.getConnectorConfigBundle(connectorId));
|
||||
describe(client.getConnector(connectorName).getFromConfig().getConfigs(), client.getConnectorConfigBundle(connectorName));
|
||||
// to job config for the connector
|
||||
describe(client.getConnector(connectorId).getToConfig().getConfigs(), client.getConnectorConfigBundle(connectorId));
|
||||
describe(client.getConnector(connectorName).getToConfig().getConfigs(), client.getConnectorConfigBundle(connectorName));
|
||||
|
||||
void describe(List<MConfig> configs, ResourceBundle resource) {
|
||||
for (MConfig config : configs) {
|
||||
|
||||
@ -110,12 +110,13 @@ Each input object in a config is structured below:
|
||||
|
||||
|
||||
To send a filled config in the request, you should always use config id and input id to map the values to their correspondig names.
|
||||
For example, the following request contains an input value ``com.mysql.jdbc.Driver`` with input id ``7`` inside a config with id ``4`` that belongs to a link with id ``3``
|
||||
For example, the following request contains an input value ``com.mysql.jdbc.Driver`` with input id ``7`` inside a config with id ``4`` that belongs to a link with name ``linkName``
|
||||
|
||||
::
|
||||
|
||||
link: {
|
||||
id: 3,
|
||||
id : 3,
|
||||
name: "linkName",
|
||||
enabled: true,
|
||||
link-config-values: [{
|
||||
id: 4,
|
||||
@ -353,10 +354,10 @@ Get all the connectors registered in Sqoop
|
||||
}]
|
||||
}
|
||||
|
||||
/v1/connector/[cname] or /v1/connector/[cid] - [GET] - Get Connector
|
||||
/v1/connector/[cname] - [GET] - Get Connector
|
||||
---------------------------------------------------------------------
|
||||
|
||||
Provide the id or unique name of the connector in the url ``[cid]`` or ``[cname]`` part.
|
||||
Provide the unique name of the connector in the url ``[cname]`` part.
|
||||
|
||||
* Method: ``GET``
|
||||
* Format: ``JSON``
|
||||
@ -367,7 +368,7 @@ Provide the id or unique name of the connector in the url ``[cid]`` or ``[cname]
|
||||
+--------------------------+----------------------------------------------------------------------------------------+
|
||||
| Field | Description |
|
||||
+==========================+========================================================================================+
|
||||
| ``id`` | The id for the connector ( registered as a configurable ) |
|
||||
| ``name`` | The name for the connector ( registered as a configurable ) |
|
||||
+--------------------------+----------------------------------------------------------------------------------------+
|
||||
| ``job-config`` | Connector job config and inputs for both FROM and TO |
|
||||
+--------------------------+----------------------------------------------------------------------------------------+
|
||||
@ -385,6 +386,7 @@ Provide the id or unique name of the connector in the url ``[cid]`` or ``[cname]
|
||||
{
|
||||
connector: {
|
||||
id: 1,
|
||||
name: "connectorName",
|
||||
job-config: {
|
||||
TO: [{
|
||||
id: 3,
|
||||
@ -550,7 +552,7 @@ Get all the links created in Sqoop
|
||||
link-config-values: [],
|
||||
name: "First Link",
|
||||
creation-date: 1415309361756,
|
||||
connector-id: 1,
|
||||
connector-name: "connectorName1",
|
||||
update-date: 1415309361756,
|
||||
creation-user: "root"
|
||||
},
|
||||
@ -561,7 +563,7 @@ Get all the links created in Sqoop
|
||||
link-config-values: [],
|
||||
name: "Second Link",
|
||||
creation-date: 1415309390807,
|
||||
connector-id: 2,
|
||||
connector-name: "connectorName2",
|
||||
update-date: 1415309390807,
|
||||
creation-user: "root"
|
||||
}
|
||||
@ -574,12 +576,12 @@ Get all the links created in Sqoop
|
||||
Get all the links for a given connector identified by ``[cname]`` part.
|
||||
|
||||
|
||||
/v1/link/[lname] or /v1/link/[lid] - [GET] - Get Link
|
||||
/v1/link/[lname] - [GET] - Get Link
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
Provide the id or unique name of the link in the url ``[lid]`` or ``[lname]`` part.
|
||||
Provide the unique name of the link in the url ``[lname]`` part.
|
||||
|
||||
Get all the details of the link including the id, name, type and the corresponding config input values for the link
|
||||
Get all the details of the link including the name, type and the corresponding config input values for the link
|
||||
|
||||
|
||||
* Method: ``GET``
|
||||
@ -610,7 +612,7 @@ Get all the details of the link including the id, name, type and the correspondi
|
||||
update-user: "root",
|
||||
name: "First Link",
|
||||
creation-date: 1415287846371,
|
||||
connector-id: 1,
|
||||
connector-name: "connectorName",
|
||||
update-date: 1415287846371,
|
||||
creation-user: "root"
|
||||
}
|
||||
@ -673,7 +675,7 @@ Create a new link object. Provide values to the link config inputs for the ones
|
||||
update-user: "root",
|
||||
name: "testLink",
|
||||
creation-date: 1415202223048,
|
||||
connector-id: 1,
|
||||
connector-name: "connectorName",
|
||||
update-date: 1415202223048,
|
||||
creation-user: "root"
|
||||
}
|
||||
@ -684,7 +686,7 @@ Create a new link object. Provide values to the link config inputs for the ones
|
||||
+---------------------------+--------------------------------------------------------------------------------------+
|
||||
| Field | Description |
|
||||
+===========================+======================================================================================+
|
||||
| ``id`` | The id assigned for this new created link |
|
||||
| ``name`` | The name assigned for this new created link |
|
||||
+---------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``validation-result`` | The validation status for the link config inputs given in the post data |
|
||||
+---------------------------+--------------------------------------------------------------------------------------+
|
||||
@ -707,10 +709,10 @@ Create a new link object. Provide values to the link config inputs for the ones
|
||||
}
|
||||
|
||||
|
||||
/v1/link/[lname] or /v1/link/[lid] - [PUT] - Update Link
|
||||
/v1/link/[lname] - [PUT] - Update Link
|
||||
---------------------------------------------------------
|
||||
|
||||
Update an existing link object with name [lname] or id [lid]. To make the procedure of filling inputs easier, the general practice
|
||||
Update an existing link object with name [lname]. To make the procedure of filling inputs easier, the general practice
|
||||
is get the link first and then change some of the values for the inputs.
|
||||
|
||||
* Method: ``PUT``
|
||||
@ -726,30 +728,30 @@ is get the link first and then change some of the values for the inputs.
|
||||
]
|
||||
}
|
||||
|
||||
/v1/link/[lname] or /v1/link/[lid] - [DELETE] - Delete Link
|
||||
/v1/link/[lname] - [DELETE] - Delete Link
|
||||
-----------------------------------------------------------------
|
||||
|
||||
Delete a link with name [lname] or id [lid]
|
||||
Delete a link with name [lname]
|
||||
|
||||
* Method: ``DELETE``
|
||||
* Format: ``JSON``
|
||||
* Request Content: ``None``
|
||||
* Response Content: ``None``
|
||||
|
||||
/v1/link/[lid]/enable or /v1/link/[lname]/enable - [PUT] - Enable Link
|
||||
/v1/link/[lname]/enable - [PUT] - Enable Link
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
Enable a link with id ``lid`` or name ``lname``
|
||||
Enable a link with name ``lname``
|
||||
|
||||
* Method: ``PUT``
|
||||
* Format: ``JSON``
|
||||
* Request Content: ``None``
|
||||
* Response Content: ``None``
|
||||
|
||||
/v1/link/[lid]/disable - [PUT] - Disable Link
|
||||
/v1/link/[lname]/disable - [PUT] - Disable Link
|
||||
---------------------------------------------------------
|
||||
|
||||
Disable a link with id ``lid`` or name ``lname``
|
||||
Disable a link with name ``lname``
|
||||
|
||||
* Method: ``PUT``
|
||||
* Format: ``JSON``
|
||||
@ -773,33 +775,33 @@ Get all the jobs created in Sqoop
|
||||
jobs: [{
|
||||
driver-config-values: [],
|
||||
enabled: true,
|
||||
from-connector-id: 1,
|
||||
from-connector-name: "fromConnectorName",
|
||||
update-user: "root",
|
||||
to-config-values: [],
|
||||
to-connector-id: 2,
|
||||
to-connector-name: "toConnectorName",
|
||||
creation-date: 1415310157618,
|
||||
update-date: 1415310157618,
|
||||
creation-user: "root",
|
||||
id: 1,
|
||||
to-link-id: 2,
|
||||
to-link-name: "toLinkName",
|
||||
from-config-values: [],
|
||||
name: "First Job",
|
||||
from-link-id: 1
|
||||
from-link-name: "fromLinkName"
|
||||
},{
|
||||
driver-config-values: [],
|
||||
enabled: true,
|
||||
from-connector-id: 2,
|
||||
from-connector-name: "fromConnectorName",
|
||||
update-user: "root",
|
||||
to-config-values: [],
|
||||
to-connector-id: 1,
|
||||
to-connector-name: "toConnectorName",
|
||||
creation-date: 1415310650600,
|
||||
update-date: 1415310650600,
|
||||
creation-user: "root",
|
||||
id: 2,
|
||||
to-link-id: 1,
|
||||
to-link-name: "toLinkName",
|
||||
from-config-values: [],
|
||||
name: "Second Job",
|
||||
from-link-id: 2
|
||||
from-link-name: "fromLinkName"
|
||||
}]
|
||||
}
|
||||
|
||||
@ -808,11 +810,10 @@ Get all the jobs created in Sqoop
|
||||
Get all the jobs for a given connector identified by ``[cname]`` part.
|
||||
|
||||
|
||||
/v1/job/[jname] or /v1/job/[jid] - [GET] - Get Job
|
||||
/v1/job/[jname] - [GET] - Get Job
|
||||
-----------------------------------------------------
|
||||
|
||||
Provide the name or the id of the job in the url [jname]
|
||||
part or [jid] part.
|
||||
Provide the name of the job in the url [jname] part.
|
||||
|
||||
* Method: ``GET``
|
||||
* Format: ``JSON``
|
||||
@ -843,7 +844,7 @@ part or [jid] part.
|
||||
type: "JOB"
|
||||
}],
|
||||
enabled: true,
|
||||
from-connector-id: 1,
|
||||
from-connector-name: "fromConnectorName",
|
||||
update-user: "root",
|
||||
to-config-values: [{
|
||||
id: 6,
|
||||
@ -887,12 +888,12 @@ part or [jid] part.
|
||||
name: "toJobConfig",
|
||||
type: "JOB"
|
||||
}],
|
||||
to-connector-id: 2,
|
||||
to-connector-name: "toConnectorName",
|
||||
creation-date: 1415310157618,
|
||||
update-date: 1415310157618,
|
||||
creation-user: "root",
|
||||
id: 1,
|
||||
to-link-id: 2,
|
||||
to-link-name: "toLinkName",
|
||||
from-config-values: [{
|
||||
id: 2,
|
||||
inputs: [{
|
||||
@ -907,7 +908,7 @@ part or [jid] part.
|
||||
type: "JOB"
|
||||
}],
|
||||
name: "First Job",
|
||||
from-link- id: 1
|
||||
from-link-name: "fromLinkName"
|
||||
}
|
||||
}
|
||||
|
||||
@ -928,9 +929,9 @@ Create a new job object with the corresponding config values.
|
||||
+==========================+======================================================================================+
|
||||
| ``job`` | The root of the post data in JSON |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``from-link-id`` | The id of the from link for the job |
|
||||
| ``from-link-name`` | The name of the from link for the job |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``to-link-id`` | The id of the to link for the job |
|
||||
| ``to-link-name`` | The name of the to link for the job |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``id`` | The id of the link can be left blank in the post data |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
@ -952,9 +953,10 @@ Create a new job object with the corresponding config values.
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``driver-config-values`` | Config input values for driver |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``connector-id`` | The id of the connector used for this link |
|
||||
| ``from-connector-name`` | The name of the from connector for the job |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``to-connector-name`` | The name of the to connector for the job |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
|
||||
|
||||
* Request Example:
|
||||
|
||||
@ -986,7 +988,7 @@ Create a new job object with the corresponding config values.
|
||||
}
|
||||
],
|
||||
enabled: true,
|
||||
from-connector-id: 1,
|
||||
from-connector-name: "fromConnectorName",
|
||||
update-user: "root",
|
||||
to-config-values: [
|
||||
{
|
||||
@ -1039,12 +1041,12 @@ Create a new job object with the corresponding config values.
|
||||
type: "JOB"
|
||||
}
|
||||
],
|
||||
to-connector-id: 2,
|
||||
to-connector-name: "toConnectorName",
|
||||
creation-date: 1415310157618,
|
||||
update-date: 1415310157618,
|
||||
creation-user: "root",
|
||||
id: -1,
|
||||
to-link-id: 2,
|
||||
to-link-name: "toLinkName",
|
||||
from-config-values: [
|
||||
{
|
||||
id: 2,
|
||||
@ -1063,7 +1065,7 @@ Create a new job object with the corresponding config values.
|
||||
}
|
||||
],
|
||||
name: "Test Job",
|
||||
from-link-id: 1
|
||||
from-link-name: "fromLinkName"
|
||||
}
|
||||
}
|
||||
|
||||
@ -1072,7 +1074,7 @@ Create a new job object with the corresponding config values.
|
||||
+---------------------------+--------------------------------------------------------------------------------------+
|
||||
| Field | Description |
|
||||
+===========================+======================================================================================+
|
||||
| ``id`` | The id assigned for this new created job |
|
||||
| ``name`` | The name assigned for this new created job |
|
||||
+--------------------------+---------------------------------------------------------------------------------------+
|
||||
| ``validation-result`` | The validation status for the job config and driver config inputs in the post data |
|
||||
+---------------------------+--------------------------------------------------------------------------------------+
|
||||
@ -1096,10 +1098,10 @@ Create a new job object with the corresponding config values.
|
||||
}
|
||||
|
||||
|
||||
/v1/job/[jid] - [PUT] - Update Job
|
||||
/v1/job/[jname] - [PUT] - Update Job
|
||||
---------------------------------------------------------
|
||||
|
||||
Update an existing job object with id [jid]. To make the procedure of filling inputs easier, the general practice
|
||||
Update an existing job object with name [jname]. To make the procedure of filling inputs easier, the general practice
|
||||
is get the existing job object first and then change some of the inputs.
|
||||
|
||||
* Method: ``PUT``
|
||||
@ -1118,30 +1120,30 @@ The same as Create Job.
|
||||
}
|
||||
|
||||
|
||||
/v1/job/[jid] - [DELETE] - Delete Job
|
||||
/v1/job/[jname] - [DELETE] - Delete Job
|
||||
---------------------------------------------------------
|
||||
|
||||
Delete a job with id ``jid``.
|
||||
Delete a job with name ``jname``.
|
||||
|
||||
* Method: ``DELETE``
|
||||
* Format: ``JSON``
|
||||
* Request Content: ``None``
|
||||
* Response Content: ``None``
|
||||
|
||||
/v1/job/[jid]/enable - [PUT] - Enable Job
|
||||
/v1/job/[jname]/enable - [PUT] - Enable Job
|
||||
---------------------------------------------------------
|
||||
|
||||
Enable a job with id ``jid``.
|
||||
Enable a job with name ``jname``.
|
||||
|
||||
* Method: ``PUT``
|
||||
* Format: ``JSON``
|
||||
* Request Content: ``None``
|
||||
* Response Content: ``None``
|
||||
|
||||
/v1/job/[jid]/disable - [PUT] - Disable Job
|
||||
/v1/job/[jname]/disable - [PUT] - Disable Job
|
||||
---------------------------------------------------------
|
||||
|
||||
Disable a job with id ``jid``.
|
||||
Disable a job with name ``jname``.
|
||||
|
||||
* Method: ``PUT``
|
||||
* Format: ``JSON``
|
||||
@ -1149,10 +1151,10 @@ Disable a job with id ``jid``.
|
||||
* Response Content: ``None``
|
||||
|
||||
|
||||
/v1/job/[jid]/start or /v1/job/[jname]/start - [PUT]- Start Job
|
||||
/v1/job/[jname]/start - [PUT]- Start Job
|
||||
---------------------------------------------------------------------------------
|
||||
|
||||
Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execution
|
||||
Start a job with name ``[jname]`` to trigger the job execution
|
||||
|
||||
* Method: ``POST``
|
||||
* Format: ``JSON``
|
||||
@ -1170,6 +1172,7 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
|
||||
"external-id": "job_1412137947693_0004",
|
||||
"status": "BOOTING",
|
||||
"job": 2,
|
||||
"job-name": "jobName",
|
||||
"creation-date": 1415312531188,
|
||||
"to-schema": {
|
||||
"created": 1415312531426,
|
||||
@ -1210,6 +1213,7 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
|
||||
external-id: "job_1412137947693_0004",
|
||||
status: "SUCCEEDED",
|
||||
job: 2,
|
||||
job-name: "jobName",
|
||||
creation-date: 1415312531188,
|
||||
external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/",
|
||||
counters: {
|
||||
@ -1271,6 +1275,7 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
|
||||
"status": "FAILURE_ON_SUBMIT",
|
||||
"error-summary": "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run",
|
||||
"job": 1,
|
||||
"job-name": "jobName",
|
||||
"creation-date": 1415312390570,
|
||||
"to-schema": {
|
||||
"created": 1415312390797,
|
||||
@ -1301,20 +1306,20 @@ Start a job with name ``[jname]`` or with id ``[jid]`` to trigger the job execut
|
||||
}
|
||||
}
|
||||
|
||||
/v1/job/[jid]/stop or /v1/job/[jname]/stop - [PUT]- Stop Job
|
||||
/v1/job/[jname]/stop - [PUT]- Stop Job
|
||||
---------------------------------------------------------------------------------
|
||||
|
||||
Stop a job with name ``[janme]`` or with id ``[jid]`` to abort the running job.
|
||||
Stop a job with name ``[jname]`` to abort the running job.
|
||||
|
||||
* Method: ``PUT``
|
||||
* Format: ``JSON``
|
||||
* Request Content: ``None``
|
||||
* Response Content: ``Submission Record``
|
||||
|
||||
/v1/job/[jid]/status or /v1/job/[jname]/status - [GET]- Get Job Status
|
||||
/v1/job/[jname]/status - [GET]- Get Job Status
|
||||
---------------------------------------------------------------------------------
|
||||
|
||||
Get status of the running job with name ``[janme]`` or with id ``[jid]``
|
||||
Get status of the running job with name ``[jname]``
|
||||
|
||||
* Method: ``GET``
|
||||
* Format: ``JSON``
|
||||
@ -1330,6 +1335,7 @@ Get status of the running job with name ``[janme]`` or with id ``[jid]``
|
||||
"external-id": "job_1412137947693_0004",
|
||||
"status": "RUNNING",
|
||||
"job": 2,
|
||||
"job-name": "jobName",
|
||||
"creation-date": 1415312531188,
|
||||
"external-link": "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/"
|
||||
}
|
||||
@ -1359,6 +1365,8 @@ Provide the name of the job in the url [jname] part.
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``job`` | The id of the Sqoop job |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``job-name`` | The name of the Sqoop job |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``creation-date`` | The submission timestamp |
|
||||
+--------------------------+--------------------------------------------------------------------------------------+
|
||||
| ``last-update-date`` | The timestamp of the last status update |
|
||||
@ -1382,6 +1390,7 @@ Provide the name of the job in the url [jname] part.
|
||||
external-id: "job_1412137947693_0004",
|
||||
status: "SUCCEEDED",
|
||||
job: 2,
|
||||
job-name: "jobName",
|
||||
creation-date: 1415312531188,
|
||||
external-link: "http://vbsqoop-1.ent.cloudera.com:8088/proxy/application_1412137947693_0004/",
|
||||
counters: {
|
||||
@ -1435,6 +1444,7 @@ Provide the name of the job in the url [jname] part.
|
||||
status: "FAILURE_ON_SUBMIT",
|
||||
error-summary: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner run",
|
||||
job: 1,
|
||||
job-name: "jobName",
|
||||
creation-date: 1415312390570,
|
||||
error-details: "org.apache.sqoop.common.SqoopException: GENERIC_HDFS_CONNECTOR_0000:Error occurs during partitioner...."
|
||||
}
|
||||
|
||||
@ -34,7 +34,7 @@ Sqoop client script is expected to contain valid Sqoop client commands, empty li
|
||||
set server --host sqoop2.company.net
|
||||
|
||||
# Executing given job
|
||||
start job --jid 1
|
||||
start job --name 1
|
||||
|
||||
|
||||
.. contents:: Table of Contents
|
||||
@ -269,12 +269,12 @@ Show persisted link objects.
|
||||
+=======================+======================================================+
|
||||
| ``-a``, ``--all`` | Show all available links |
|
||||
+-----------------------+------------------------------------------------------+
|
||||
| ``-x``, ``--lid <x>`` | Show link with id ``<x>`` |
|
||||
| ``-n``, ``--name <x>``| Show link with name ``<x>`` |
|
||||
+-----------------------+------------------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
show link --all or show link
|
||||
show link --all or show link --name linkName
|
||||
|
||||
Show Job Function
|
||||
~~~~~~~~~~~~~~~~~
|
||||
@ -286,31 +286,31 @@ Show persisted job objects.
|
||||
+=======================+==============================================+
|
||||
| ``-a``, ``--all`` | Show all available jobs |
|
||||
+-----------------------+----------------------------------------------+
|
||||
| ``-j``, ``--jid <x>`` | Show job with id ``<x>`` |
|
||||
| ``-n``, ``--name <x>``| Show job with name ``<x>`` |
|
||||
+-----------------------+----------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
show job --all or show job
|
||||
show job --all or show job --name jobName
|
||||
|
||||
Show Submission Function
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Show persisted job submission objects.
|
||||
|
||||
+-----------------------+---------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+=============================================+
|
||||
| ``-j``, ``--jid <x>`` | Show available submissions for given job |
|
||||
+-----------------------+---------------------------------------------+
|
||||
| ``-d``, ``--detail`` | Show job submissions in full details |
|
||||
+-----------------------+---------------------------------------------+
|
||||
+-----------------------+-----------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+===============================================+
|
||||
| ``-j``, ``--job <x>`` | Show available submissions for given job name |
|
||||
+-----------------------+-----------------------------------------------+
|
||||
| ``-d``, ``--detail`` | Show job submissions in full details |
|
||||
+-----------------------+-----------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
show submission
|
||||
show submission --jid 1
|
||||
show submission --jid 1 --detail
|
||||
show submission --j jobName
|
||||
show submission --job jobName --detail
|
||||
|
||||
Create Command
|
||||
--------------
|
||||
@ -332,16 +332,16 @@ Create Link Function
|
||||
|
||||
Create new link object.
|
||||
|
||||
+------------------------+-------------------------------------------------------------+
|
||||
| Argument | Description |
|
||||
+========================+=============================================================+
|
||||
| ``-c``, ``--cid <x>`` | Create new link object for connector with id ``<x>`` |
|
||||
+------------------------+-------------------------------------------------------------+
|
||||
+------------------------------+-------------------------------------------------------------+
|
||||
| Argument | Description |
|
||||
+==============================+=============================================================+
|
||||
| ``-c``, ``--connector <x>`` | Create new link object for connector with name ``<x>`` |
|
||||
+------------------------------+-------------------------------------------------------------+
|
||||
|
||||
|
||||
Example: ::
|
||||
|
||||
create link --cid 1 or create link -c 1
|
||||
create link --connector connectorName or create link -c connectorName
|
||||
|
||||
Create Job Function
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
@ -351,14 +351,14 @@ Create new job object.
|
||||
+------------------------+------------------------------------------------------------------+
|
||||
| Argument | Description |
|
||||
+========================+==================================================================+
|
||||
| ``-f``, ``--from <x>`` | Create new job object with a FROM link with id ``<x>`` |
|
||||
| ``-f``, ``--from <x>`` | Create new job object with a FROM link with name ``<x>`` |
|
||||
+------------------------+------------------------------------------------------------------+
|
||||
| ``-t``, ``--to <t>`` | Create new job object with a TO link with id ``<x>`` |
|
||||
| ``-t``, ``--to <t>`` | Create new job object with a TO link with name ``<x>`` |
|
||||
+------------------------+------------------------------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
create job --from 1 --to 2 or create job --f 1 --t 2
|
||||
create job --from fromLinkName --to toLinkName or create job --f fromLinkName --t toLinkName
|
||||
|
||||
Update Command
|
||||
--------------
|
||||
@ -373,27 +373,27 @@ Update existing link object.
|
||||
+-----------------------+---------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+=============================================+
|
||||
| ``-x``, ``--lid <x>`` | Update existing link with id ``<x>`` |
|
||||
| ``-n``, ``--name <x>``| Update existing link with name ``<x>`` |
|
||||
+-----------------------+---------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
update link --lid 1
|
||||
update link --name linkName
|
||||
|
||||
Update Job Function
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Update existing job object.
|
||||
|
||||
+-----------------------+--------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+============================================+
|
||||
| ``-j``, ``--jid <x>`` | Update existing job object with id ``<x>`` |
|
||||
+-----------------------+--------------------------------------------+
|
||||
+-------------------------+----------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=========================+==============================================+
|
||||
| ``-n``, ``--name <x>`` | Update existing job object with name ``<x>`` |
|
||||
+-------------------------+----------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
update job --jid 1
|
||||
update job --name jobName
|
||||
|
||||
|
||||
Delete Command
|
||||
@ -406,15 +406,15 @@ Delete Link Function
|
||||
|
||||
Delete existing link object.
|
||||
|
||||
+-----------------------+-------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+===========================================+
|
||||
| ``-x``, ``--lid <x>`` | Delete link object with id ``<x>`` |
|
||||
+-----------------------+-------------------------------------------+
|
||||
+-------------------------+-------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=========================+===========================================+
|
||||
| ``-n``, ``--name <x>`` | Delete link object with name ``<x>`` |
|
||||
+-------------------------+-------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
delete link --lid 1
|
||||
delete link --name linkName
|
||||
|
||||
|
||||
Delete Job Function
|
||||
@ -422,15 +422,15 @@ Delete Job Function
|
||||
|
||||
Delete existing job object.
|
||||
|
||||
+-----------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+==========================================+
|
||||
| ``-j``, ``--jid <x>`` | Delete job object with id ``<x>`` |
|
||||
+-----------------------+------------------------------------------+
|
||||
+-------------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=========================+==========================================+
|
||||
| ``-n``, ``--name <x>`` | Delete job object with name ``<x>`` |
|
||||
+-------------------------+------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
delete job --jid 1
|
||||
delete job --name jobName
|
||||
|
||||
|
||||
Clone Command
|
||||
@ -443,15 +443,15 @@ Clone Link Function
|
||||
|
||||
Clone existing link object.
|
||||
|
||||
+-----------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+==========================================+
|
||||
| ``-x``, ``--lid <x>`` | Clone link object with id ``<x>`` |
|
||||
+-----------------------+------------------------------------------+
|
||||
+-------------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=========================+==========================================+
|
||||
| ``-n``, ``--name <x>`` | Clone link object with name ``<x>`` |
|
||||
+-------------------------+------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
clone link --lid 1
|
||||
clone link --name linkName
|
||||
|
||||
|
||||
Clone Job Function
|
||||
@ -459,15 +459,15 @@ Clone Job Function
|
||||
|
||||
Clone existing job object.
|
||||
|
||||
+-----------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+==========================================+
|
||||
| ``-j``, ``--jid <x>`` | Clone job object with id ``<x>`` |
|
||||
+-----------------------+------------------------------------------+
|
||||
+-------------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=========================+==========================================+
|
||||
| ``-n``, ``--name <x>`` | Clone job object with name ``<x>`` |
|
||||
+-------------------------+------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
clone job --jid 1
|
||||
clone job --name jobName
|
||||
|
||||
Start Command
|
||||
-------------
|
||||
@ -482,15 +482,15 @@ Start job (submit new submission). Starting already running job is considered as
|
||||
+----------------------------+----------------------------+
|
||||
| Argument | Description |
|
||||
+============================+============================+
|
||||
| ``-j``, ``--jid <x>`` | Start job with id ``<x>`` |
|
||||
| ``-n``, ``--name <x>`` | Start job with name ``<x>``|
|
||||
+----------------------------+----------------------------+
|
||||
| ``-s``, ``--synchronous`` | Synchoronous job execution |
|
||||
+----------------------------+----------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
start job --jid 1
|
||||
start job --jid 1 --synchronous
|
||||
start job --name jobName
|
||||
start job --name jobName --synchronous
|
||||
|
||||
Stop Command
|
||||
------------
|
||||
@ -502,15 +502,15 @@ Stop Job Function
|
||||
|
||||
Interrupt running job.
|
||||
|
||||
+-----------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+==========================================+
|
||||
| ``-j``, ``--jid <x>`` | Interrupt running job with id ``<x>`` |
|
||||
+-----------------------+------------------------------------------+
|
||||
+-------------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=========================+==========================================+
|
||||
| ``-n``, ``--name <x>`` | Interrupt running job with name ``<x>`` |
|
||||
+-------------------------+------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
stop job --jid 1
|
||||
stop job --name jobName
|
||||
|
||||
Status Command
|
||||
--------------
|
||||
@ -522,12 +522,12 @@ Status Job Function
|
||||
|
||||
Retrieve last status for given job.
|
||||
|
||||
+-----------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=======================+==========================================+
|
||||
| ``-j``, ``--jid <x>`` | Retrieve status for job with id ``<x>`` |
|
||||
+-----------------------+------------------------------------------+
|
||||
+-------------------------+------------------------------------------+
|
||||
| Argument | Description |
|
||||
+=========================+==========================================+
|
||||
| ``-n``, ``--name <x>`` | Retrieve status for job with name ``<x>``|
|
||||
+-------------------------+------------------------------------------+
|
||||
|
||||
Example: ::
|
||||
|
||||
status job --jid 1
|
||||
status job --name jobName
|
||||
|
||||
@ -76,20 +76,20 @@ Creating Link Object
|
||||
Check for the registered connectors on your Sqoop server: ::
|
||||
|
||||
sqoop:000> show connector
|
||||
+----+------------------------+----------------+------------------------------------------------------+----------------------+
|
||||
| Id | Name | Version | Class | Supported Directions |
|
||||
+----+------------------------+----------------+------------------------------------------------------+----------------------+
|
||||
| 1 | hdfs-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO |
|
||||
| 2 | generic-jdbc-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO |
|
||||
+----+------------------------+----------------+------------------------------------------------------+----------------------+
|
||||
+------------------------+----------------+------------------------------------------------------+----------------------+
|
||||
| Name | Version | Class | Supported Directions |
|
||||
+------------------------+----------------+------------------------------------------------------+----------------------+
|
||||
| hdfs-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO |
|
||||
| generic-jdbc-connector | 2.0.0-SNAPSHOT | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO |
|
||||
+------------------------+----------------+------------------------------------------------------+----------------------+
|
||||
|
||||
Our example contains two connectors. The one with connector Id 2 is called the ``generic-jdbc-connector``. This is a basic connector relying on the Java JDBC interface for communicating with data sources. It should work with the most common databases that are providing JDBC drivers. Please note that you must install JDBC drivers separately. They are not bundled in Sqoop due to incompatible licenses.
|
||||
Our example contains two connectors. The ``generic-jdbc-connector`` is a basic connector relying on the Java JDBC interface for communicating with data sources. It should work with the most common databases that are providing JDBC drivers. Please note that you must install JDBC drivers separately. They are not bundled in Sqoop due to incompatible licenses.
|
||||
|
||||
Generic JDBC Connector in our example has a persistence Id 2 and we will use this value to create new link object for this connector. Note that the link name should be unique.
|
||||
Generic JDBC Connector in our example has a name ``generic-jdbc-connector`` and we will use this value to create new link object for this connector. Note that the link name should be unique.
|
||||
::
|
||||
|
||||
sqoop:000> create link -c 2
|
||||
Creating link for connector with id 2
|
||||
sqoop:000> create link -connector generic-jdbc-connector
|
||||
Creating link for connector with name generic-jdbc-connector
|
||||
Please fill following values to create new link object
|
||||
Name: First Link
|
||||
|
||||
@ -101,22 +101,22 @@ Generic JDBC Connector in our example has a persistence Id 2 and we will use thi
|
||||
JDBC Connection Properties:
|
||||
There are currently 0 values in the map:
|
||||
entry#protocol=tcp
|
||||
New link was successfully created with validation status OK and persistent id 1
|
||||
New link was successfully created with validation status OK name First Link
|
||||
|
||||
Our new link object was created with assigned id 1.
|
||||
Our new link object was created with assigned name First Link.
|
||||
|
||||
In the ``show connector -all`` we see that there is a hdfs-connector registered in sqoop with the persistent id 1. Let us create another link object but this time for the hdfs-connector instead.
|
||||
In the ``show connector -all`` we see that there is a hdfs-connector registered. Let us create another link object but this time for the hdfs-connector instead.
|
||||
|
||||
::
|
||||
|
||||
sqoop:000> create link -c 1
|
||||
Creating link for connector with id 1
|
||||
sqoop:000> create link -connector hdfs-connector
|
||||
Creating link for connector with name hdfs-connector
|
||||
Please fill following values to create new link object
|
||||
Name: Second Link
|
||||
|
||||
Link configuration
|
||||
HDFS URI: hdfs://nameservice1:8020/
|
||||
New link was successfully created with validation status OK and persistent id 2
|
||||
New link was successfully created with validation status OK and name Second Link
|
||||
|
||||
Creating Job Object
|
||||
===================
|
||||
@ -127,8 +127,8 @@ Connectors implement the ``From`` for reading data from and/or ``To`` for writin
|
||||
|
||||
sqoop:000> show link --all
|
||||
2 link(s) to show:
|
||||
link with id 1 and name First Link (Enabled: true, Created by root at 11/4/14 4:27 PM, Updated by root at 11/4/14 4:27 PM)
|
||||
Using Connector id 2
|
||||
link with name First Link (Enabled: true, Created by root at 11/4/14 4:27 PM, Updated by root at 11/4/14 4:27 PM)
|
||||
Using Connector with name generic-jdbc-connector
|
||||
Link configuration
|
||||
JDBC Driver Class: com.mysql.jdbc.Driver
|
||||
JDBC Connection String: jdbc:mysql://mysql.ent.cloudera.com/sqoop
|
||||
@ -136,16 +136,16 @@ Connectors implement the ``From`` for reading data from and/or ``To`` for writin
|
||||
Password:
|
||||
JDBC Connection Properties:
|
||||
protocol = tcp
|
||||
link with id 2 and name Second Link (Enabled: true, Created by root at 11/4/14 4:38 PM, Updated by root at 11/4/14 4:38 PM)
|
||||
Using Connector id 1
|
||||
link with name Second Link (Enabled: true, Created by root at 11/4/14 4:38 PM, Updated by root at 11/4/14 4:38 PM)
|
||||
Using Connector with name hdfs-connector
|
||||
Link configuration
|
||||
HDFS URI: hdfs://nameservice1:8020/
|
||||
|
||||
Next, we can use the two link Ids to associate the ``From`` and ``To`` for the job.
|
||||
Next, we can use the two link names to associate the ``From`` and ``To`` for the job.
|
||||
::
|
||||
|
||||
sqoop:000> create job -f 1 -t 2
|
||||
Creating job for links with from id 1 and to id 2
|
||||
sqoop:000> create job -f "First Link" -t "Second Link"
|
||||
Creating job for links with from name First Link and to name Second Link
|
||||
Please fill following values to create new job object
|
||||
Name: Sqoopy
|
||||
|
||||
@ -182,9 +182,9 @@ Next, we can use the two link Ids to associate the ``From`` and ``To`` for the j
|
||||
Driver Config
|
||||
Extractors:(Optional) 2
|
||||
Loaders:(Optional) 2
|
||||
New job was successfully created with validation status OK and persistent id 1
|
||||
New job was successfully created with validation status OK and name jobName
|
||||
|
||||
Our new job object was created with assigned id 1. Note that if null value is allowed for the partition column,
|
||||
Our new job object was created with assigned name Sqoopy. Note that if null value is allowed for the partition column,
|
||||
at least 2 extractors are needed for Sqoop to carry out the data transfer. On specifying 1 extractor in this
|
||||
scenario, Sqoop shall ignore this setting and continue with 2 extractors.
|
||||
|
||||
@ -194,9 +194,9 @@ Start Job ( a.k.a Data transfer )
|
||||
You can start a sqoop job with the following command:
|
||||
::
|
||||
|
||||
sqoop:000> start job -j 1
|
||||
sqoop:000> start job -name Sqoopy
|
||||
Submission details
|
||||
Job ID: 1
|
||||
Job Name: Sqoopy
|
||||
Server URL: http://localhost:12000/sqoop/
|
||||
Created by: root
|
||||
Creation date: 2014-11-04 19:43:29 PST
|
||||
@ -209,9 +209,9 @@ You can iteratively check your running job status with ``status job`` command:
|
||||
|
||||
::
|
||||
|
||||
sqoop:000> status job -j 1
|
||||
sqoop:000> status job -n Sqoopy
|
||||
Submission details
|
||||
Job ID: 1
|
||||
Job Name: Sqoopy
|
||||
Server URL: http://localhost:12000/sqoop/
|
||||
Created by: root
|
||||
Creation date: 2014-11-04 19:43:29 PST
|
||||
@ -224,9 +224,9 @@ Alternatively you can start a sqoop job and observe job running status with the
|
||||
|
||||
::
|
||||
|
||||
sqoop:000> start job -j 1 -s
|
||||
sqoop:000> start job -n Sqoopy -s
|
||||
Submission details
|
||||
Job ID: 1
|
||||
Job Name: Sqoopy
|
||||
Server URL: http://localhost:12000/sqoop/
|
||||
Created by: root
|
||||
Creation date: 2014-11-04 19:43:29 PST
|
||||
@ -239,4 +239,4 @@ Alternatively you can start a sqoop job and observe job running status with the
|
||||
|
||||
And finally you can stop running the job at any time using ``stop job`` command: ::
|
||||
|
||||
sqoop:000> stop job -j 1
|
||||
sqoop:000> stop job -n Sqoopy
|
||||
|
||||
Loading…
Reference in New Issue
Block a user