Rundeck Manual Overview

Rundeck Manual
Schlumberger-Private
Author(s): Alexander Griffith (Slides # 17 – 27)[ Source: Rundeck_MDMv1.4.pptx]
Praveen Kotla (Slides # 1-16) / Modifications to Slides# 17-27 as per SAPITT Data validation project
Agenda
 Rundeck Overview
 Create New job
– Setup Commands / Script file execution
– Job Scheduling
– Job Notification
 Execute a job
 Monitor a job
 Additional features
Overview - Rundeck
 An open source software that is developed on Github. Its prime use is to perform automation of routine operation procedures in
data center or cloud environments.
 Allows you to run tasks on any number of nodes from a web-based or command-line interface. Rundeck also includes other
features that make it easy to scale up your automation efforts including: access control, workflow building, scheduling, logging,
and integration with external sources for node and option data.
 Features:
• Web API
• distributed command execution
• pluggable execution system (SSH by default)
• multi-step workflows
• job execution with on demand or scheduled runs
• graphical web console for command and job execution
• role-based access control policy with support for LDAP/Active Directory
• history and auditing logs
• open integration to external host inventory tools
• command line interface tools
Login to Rundeck
Rundeck URL: http://rundeck-dev.mdm.slb.com:4440
Useful Info:
Every user in Rundeck is made a part of a group through which access policy on the Rundeck Application and the Projects is controlled.
For MDM group we have 2 Groups
mdm_user – Read access to all MDM IDQ projects. Can execute and monitor jobs.
mdm_admin – Read & Write access to all MDM & IDQ Projects.
In addition we have a super user group named “admin”. This group has full control over rundeck application and all projects.
Create New Job
New Job - Definition
Job Name: Provide a meaningful name. In the current example we are setting up a ODQ domain & repository backup job in production.
Description: Define the frequency, the specific days it run followed by the high level activities performed in this job.
Also mention if this job has any dependency upstream or downstream.
Options: Options are basically values that can be used as arguments to the commands or scripts to be executed within the job definition.
Workflow: This is the main body of the job definition wherein the commands or scripts to be executed are specified.
Specify the action to be taken if a step fails.
New Job - Definition
Add a Step: A Step under the workflow could be A Command, A Script, Job Reference (Call to another Job within the same Rundeck Project),
Copy File, Local command etc.
Script (Inline Script): Enter the linux script code here.
Arguments: Options are basically values that can be used as arguments to the commands or scripts to be executed within the job definition.
Workflow: This is the main body of the job definition wherein the commands or scripts to be executed are specified.
Specify the action to be taken if a step fails.
New Job –> Workflow –> Step -> Inline Script
Script:
#!/bin/bash
source ~/.bash_profile
echo "Backing up IDQ Domain on" @node.name@
echo "$INFA_BATCH_HOME/scripts/exec_domrep_bkp.sh"
sh $INFA_BATCH_HOME/scripts/exec_domrep_bkp.sh
RETURN_CODE=$?
if [ $RETURN_CODE -ne 0 ]
then
echo "Domain backup failed“
exit $RETURN_CODE
else
echo "Domain backup Successful“
fi
Script: In the above example we are performing Backup of the domain on IDQ Server. We are executing the script
$INFA_BATCH_HOME/scripts/exec_domrep_bkp.sh. Appropriate success or failure messages are displayed based on the script execution.
New Job –> Workflow –> Step -> Inline Script
Arguments: Specify any arguments that needs to be passed in to the script or command
Step Label: This is the label defining the step
Save: On completion of the workflow definition, save the workflow section.
Nodes: Post the Workflow save, specify the execution methodology whether it would be on the local Rundeck node or any other nodes defined and configured on
Rundeck server.
New Job –> Notification
Send Notification: Change it to “Yes” if you want email notifications.
Notifications can be sent on following events - Success, Failure or on Start.
On Success/ On Failure/ On Start: Send Email
Set this option as required for your process.
To: Specify the email IDs separate by comma with a subject
Subject: Mention a short note on the job & the success or failure message.
Attach Output Log: Output of script that is executed in the job can be attached with the email if this option is checked.
New Job –> Scheduling
Send to run repeatedly: No – for Ad-hoc jobs Yes – Means these jobs will be run on a specific schedule
Scheduling can be done in 2 ways one is the traditional Crontab method and the other is a simple user friendly way to define a schedule.
Specify the Time of the day on which an execution for this job needs to be planned.
Specify the Months for execution, default is “Every month”.
Enable Scheduling: Set to Yes
Enable Execution: Set to Yes
Log Level: Set to Normal unless logs from rundeck in Debug mode are required.
New Job –> Scheduling
Multiple Executions: No – Set to No if simultaneous executions are not required.
Timeout: This is only required if you want to specify a time limit for an execution of this job.
Retry: Only required to be specified if you want a execution to be retried in case of job failure/ timeout.
Log Output Limit: Required to limit the logs output in terms of lines or size.
When all of the above are specified then click on “Create” button. To Create the job.
Job –> Execution & Monitoring
Run Job Now: To begin immediate execution of a Job.
Recent: To view all recent executions of this job.
Running: To view currently running execution of the job.
failed: To view all failed executions of this job
By You: To view all executions kick started by your user id.
Job –> Execution & Monitoring
The above view is of a currently executing job “IDQ_Domain_Backup_D”.
Summary: Shows the job summary

Monitor: Shows the step that is completed, and step getting executed.
Log Output: Displays the output from the linux prompt.
Job –> Sample Notification email on job completion
The above is an email from Rundeck after Domain & repository backup if performed at 1:30 PM CET every day from Monday – Friday.
The email comes with useful information like the execution time, if there are multiple steps then step level information and average execution duration based on all past
executions.
Job –> Additional Features
Duplicate this Job: This option allows to duplicate the job definition in the same project. To be used when you are trying to create a job with exactly very similar
definition with minor changes that can be done after copying the definition from an existing job
Disable Execution: This option stops a job from being scheduled or ad-hoc runs
Download Definition: This options enables export of job definition as XML/YAML. A useful option for migration of jobs from lower to higher environment.
Rundeck Project Structure
Project folder structure on Rundeck Server:
$RDECK_BASE/projects/SAP_Data_Validation
|---acls
|---etc
| |---json
| | |--*.json
| |--project.properties
| |--resources.xml
Rundeck Project Structure - Nodes
Rundeck Nodes are network hosts on which the jobs will execute. For MDM, these
are IDQ and MDM servers.
The system is “agentless”, meaning SSH keys are used to connect to the servers and
run commands directly as sv-infa_[dev/prod]. No additional software is required on
any other machine.
Rundeck Project Structure - Nodes
Nodes: Node configuration (resources.xml):

• Rundeck
<node name="IDQ_DEV"
• IDQ_DEV
description="Informatica Data Quality Development Server"
• IDQ_QA tags="IDQ,SRC_TO_CN,CN_TO_CL,CL_TO_LDG,LDG_TO_STG"
• IDQ_PROD hostname="idq-dev.mdm.slb.com"
osFamily="unix" osName="Linux"
• MDM Dev
username="sv-infa_dev"
• MDM QA environment="development">
• MDM PROD </node>
Rundeck Jobs – SAP_Data_Validation
Job Group Structure
|---IDQ
| |--Run IDQ Mapping
| |--Run IDQ Profile
| |--Run IDQ Scorecard
|
Rundeck Jobs – SAP_Data_Validation
JSON files are used to drive workflows and

select options for ad-hoc job runs.
Options can be dependent on one another – i.e.

select an IDQ Application, and the available
list of mappings changes.
All jobs have the same set of “environment”

options: development, QA, and production.
Rundeck Methodology – SAP_Data_Validation
“Base” jobs for: JSON files for dynamic options:
• IDQ Mapping • IDQ Application/Mapping Name

• Deploy IDQ Application • IDQ Applications
• IDQ Profiles
• IDQ Scorecards
Base jobs contain scripts

used to execute Informatica
IDQ or MDM commands.
Workflow jobs contain

references to the base jobs,
with specific options pre-
selected.
In workflows, job options can be set at the workflow level, or when calling an
individual job
Jobs can be run in any environment,

using the “environment” option.
Verify this for every job, every time.
Notifications
Notifications sent to: mdm-nightly@slb.com Start Notification: Nightly Process

Success/Failure Notification:
<Need specific email id from SAPITT_DV
- Mapping Execution
Team>
- Profile Execution
- Scorecard Execution
Subject will be: SAPITT DV - <JOBNAME>
<Started/Completed/Failed>
Failed jobs will have the Rundeck output log

attached.
Future Enhancements
• Custom email templates for start, success, • Improved error handling (retries)
and failed jobs
• Switch to GNU Parallel instead of xargs for
• Additional IDQ or MDM log/DB data for bash scripts (better control over execution
failed jobs
output)
• Process data by subject area (Asset, Party,

etc) • Better workflow orchestration:
• Run Job A
• Start Job B, wait for touch file from Job A
• Add support for parameter files to Run IDQ
Mapping. • Continue running Job B

Rundeck Manual Overview

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Rundeck Manual Overview

Diunggah oleh

Hak Cipta:

Format Tersedia

Rundeck Manual

Summary: Shows the job summary

Project folder structure on Rundeck Server:

Nodes: Node configuration (resources.xml):

• MDM PROD </node>

JSON files are used to drive workflows and

Options can be dependent on one another – i.e.

All jobs have the same set of “environment”

“Base” jobs for: JSON files for dynamic options:

• IDQ Mapping • IDQ Application/Mapping Name

Base jobs contain scripts

Workflow jobs contain

Jobs can be run in any environment,

Verify this for every job, every time.

Notifications sent to: mdm-nightly@slb.com Start Notification: Nightly Process

Failed jobs will have the Rundeck output log

• Process data by subject area (Asset, Party,

Anda mungkin juga menyukai