Anda di halaman 1dari 10

A Iramework Ior Iault tolerant job scheduling in distributed

systems
Sandip Kumar Goyal
1,*
and Manpreet Singh
2
1
Research Scholar, Department of Computer Science & Engineering, M. M. Engineering CollegeM. M Universitv, Ambala, Harvana, India
2
Professor, Department of Computer Science & Engineering, M. M. University, Sadopur, Ambala,Harvana,India
Abstract
Distributed systems are more prone to Iailures than traditional computing systems due to its dynamicity and
high heterogeneity. The resource Iailures aIIect the job execution drastically; resulting in Iault tolerance as an
essential service in such systems. In this paper, a Iramework is proposed that integrates Iault tolerance and job
scheduling in distributed system. The proposed Iault tolerant Iramework makes the use oI checkpointing
approach Ior Iailure recovery.
2013 Elsevier Science. All rights reserved.
Keywords: Distributed system, Fault tolerance, Check pointing, Job scheduling.
1. Introduction
Distributed systems usually contain number oI machines which are geographically apart and are susceptible to
process Iailure, machine crashes and network Iailures. This may lead to job Iailures, violating timing deadlines
and degraded user expected quality oI service. The Iailure rate grows higher when the scale oI the
distributed system becomes larger |1||3|. To achieve the promising potentials oI distributed computing, the Iault
tolerance is Iundamentally important. Replication and check pointing are the major techniques used in any Iault-
tolerant distributed system |2||4|. In job replication, the same job is replicated to be executed on multiple
undependable resources to guard the job against a single point oI Iailure. In check pointing, the state oI a running
job is saved to a stable storage. This state can be used later in case oI any Iault to resume execution oI the job
instead oI restarting it. In this paper, a Iramework Ior scheduling oI jobs on heterogeneous resources oI a
distributed system in a Iault tolerant manner is presented. The scheduling policy considers the Iault history oI
resources during the allocation oI a resource to next incoming job. Moreover, the check pointing is used as a
recovery mechanism in case oI Iault occurrence at resources.
2. Related Work
A hybrid Fault Tolerance Techniques (FTTs) that are called alternate task with checkpoint and alternate
task with retry is proposed |5|. These hybrid FTTs inherit the good Ieatures and overcome the limitations oI
workIlow level FTT and task level FTT. In |6|, authors presented a model Ior job scheduling and Iault tolerance
service in the grid to improve dependability with respect to economic eIIiciency. A three layered detection
service is proposed to improve Iailure coverage and reduce the probability oI Iaults.
In |7|, authors highlight the importance oI genetic algorithm to design schedulers that minimizes the waiting
time and maximizes the resource utilization and provides Iairness in the grid environment. The idea oI Iault
tolerance in the crash Iault environment is implemented based on precautionary method and real time restoration.
Proceedings oI International ConIerence on Computing Sciences
WILKES100 ICCS 2013
ISBN: 978-93-5107-172-3
559 Elsevier Publications, 2013
*
Corresponding author. Sandip Kumar Goyal
Sandip Kumar Goyal and Manpreet Singh

A dynamic scheduling algorithm that switches between periodic checkpointing and replication to exploit the
advantages oI both techniques and to reduce the overhead is proposed |8|. Furthermore, several novel heuristics
are discussed that perIorm on-line adaptive tuning oI the checkpointing period based on historical inIormation on
resource behavior. The problem oI Iault tolerance in grid computing is addressed and a novel adaptive task
replication based Iault tolerant job scheduling strategy Ior economy driven grid is proposed |9|. The
proposed strategy maintains Iault history oI the resources termed as resource Iault index.
3. System Model
In proposed Iault tolerant job scheduling Iramework, resources are clustered into domains around a set
oI meta-schedulers, inIormation servers, Iault handlers and system monitors in a Iully decentralized Iashion
as shown in Figure 1. Each resources provider Iurnishes the inIormation oI resources it wishes to provide and
register it with InIormation Server. The user will submit his jobs to the meta-scheduler which combine diIIerent
jobs in the Iorm oI batch and then select Ieasible resources Irom its domain Ior these batches and Iinally generate
job-to-resource mapping using scheduling policy. The dispatcher component allocate the jobs oI given batch
waiting in the waiting queue to corresponding resources on Iirst come Iirst basis. Fault handler unit comprising
oI Iault manager and replica manager is the heart oI the Iramework that provides Iault tolerance behavior. Fault
manager maintain a table storing the Iault index value oI all resources available in that domain. The Iault index
oI a resource is computed by dividing the number oI jobs executed successIully with the total number oI jobs
submitted to it. System monitor maintains the load status oI all registered resources in terms oI job queue
length and propagate this inIormation to Iault handler.
Meta Scheduler
Schedule
Manager
Scheduling Policy
Dispatcher
Execution
Queue
Information Server
Fault Manager
Replica
Manager
System
Monitor
R0
R1
Resource
Pool
User 1
User n
User 3
User 2
Fault Handler
Rn
Fig. 1. Structure oI i
th
domain
Checkpoint manager keeps a record oI next ranked resource to which a job, not successIully
executed on allocated resource, can be assigned to resume its execution without any Iurther delay. The
selection oI next ranked resource Ior every job is computed by considering the Iault index, load and
relative location oI resources Irom the site where replica manger module is running. Checkpoint manager also
maintains the status oI partially executed jobs aIter certain interval oI time in order to ensure the completion oI a
job, even iI, the Iailure oI currently allocated resource takes place.
560 Elsevier Publications, 2013
A Framework for Fault Tolerant Job Scheduling in Distributed Systems

4. Proposed Algorithm
The main objective oI proposed algorithm is to achieve Iault tolerance with well- balanced load among
all resources. The idea is to reduce the vulnerability oI underlying system against Iaults through proactive
scheduling. The notations used in t h e description oI algorithm are illustrated in Table 1.
Table 1. Notations used
For any domain i (where i can assume value 1 to m) various phases according to Iunction perIormed by
components oI domain i are given below.
561 Elsevier Publications, 2013
Sandip Kumar Goyal and Manpreet Singh

Proposed Algorithm: The Iollowing steps are perIormed by algorithm in any domain i where i 1 to m.
Step 1: Initialize the value oI n
i
, b
i
, N
i,k
, ,
i
j
,
i
j
, L
i
j
,
i
j
.
Step 2: All the users submit their tasks to
i
S
m with the help oI web portal and
i
S
m constitute
i
b batches such
that Size ( j )
j i
N
,
where j (1, 2...
i
b ) .
Step 3: All
i
j
R register their various characteristics to
i
IS and whenever status oI any
i
j
R gets changed then it
give inIormation to
i
IS by using
s
d
nf where s
i
j
R and d
i
IS .
Step 4: SM m
i
S
. use the prescribed scheduling policy on k
th
batch oI jobs in order to obtain
i
k
Om
, 1
and
i
k
Om
, 2
such that
i
k
Om
, 1
= (<
i
k j
T
,
,
i
l
R , etc (
i
k j
T
,
,
i
l
R )>) where
i
k j
T
,
(1, 2,.,
k i
N
,
) and
i
k
Om
, 2
= (<
i
k j
T
,
,
i
o
R , etc (
i
k j
T
,
,
i
o
R )>) where
i
k j
T
,
(1, 2,.,
k i
N
,
) .
Step 5: All oI
i
b wait Ior their turn in EQ m
i
S
. and Di m
i
S
. dispatches tasks to corresponding resource
according to supplied condition whether priority execution or priority deadline.
Step 6: Check the priority choice described by user.
Step 7: II (choice is not priority deadline) then perIorm Iollowing steps until EQ m
i
S
. become empty:
i) Di m
i
S
. obtain the next waiting task
i
k j
T
,
Irom EQ m
i
S
. i.e.
i
k j
T
,
next ( EQ m
i
S
. ).
ii) Di m
i
S
. dispatch this task to two resources (
i
l
R ,
i
o
R ) called primary resource & alternate resource
respectively such that primary resources belong to
i
k
Om
, 1
and alternate resource belongs to
i
k
Om
, 2
.
iii) Primary resource (
i
l
R ) start execution oI
i
k j
T
,
.
iv) During execution II (I (
i
l
R ) = False) then perIorm Iollowing steps:
a)
i
l
R notiIy its status to
i
IS using
s
d
nf where s =
i
l
R & d
i
IS which in turn give this inIormation to
i
fm using
s
d
nf where s
i
IS & d
i
fm and it will update its
i
l
value.
b)
i
fm give this inIormation to SM m
i
S
. using
s
d
nf where s
i
fm & d SM m
i
S
. .
c) SM m
i
S
. initiates the execution oI
i
k j
T
,
on alternate resource (
i
o
R ) Irom scratch.
v) AIter successIul completion oI
i
k j
T
,
, the corresponding resource
i
o
R notiIy its status to
i
IS using
s
d
nf where
s =
i
o
R & d
i
IS which in turn give this inIormation to
i
fm using
s
d
nf where s
i
IS & d
i
fm and
i
fm
will update its
i
l
value.
vi)
i
o
R will send back the results oI
i
k j
T
,
to
i
S
m which in turn gives the results to corresponding user.
Step 8: II (choice is priority deadline) then perIorm Iollowing steps until EQ m
i
S
. become empty:
562 Elsevier Publications, 2013
A Framework for Fault Tolerant Job Scheduling in Distributed Systems

On the Meta scheduler side:
i) Di m
i
S
. obtain the next waiting task
i
k j
T
,
Irom EQ m
i
S
. i.e.
i
k j
T
,
next ( EQ m
i
S
. ).
ii) Di m
i
S
. dispatch this task to resources
i
l
R called primary resource such that primary resource belongs to
i
k
Om
, 1
.
iii) Primary resource (
i
l
R ) start execution oI
i
k j
T
,
.
On System monitor side:
i)
i
SM
maintain a table which store value oI
i
j
L against
i
j
R as its entries.
ii) It will supply
i
j
L oI each
i
j
R to
i
m
r which is used in making decision about alternate resource Ior replica
location.
On the Fault handler side:
i)
i
fm maintain a table which store value oI
i
j
,
i
j
and
i
j
as its entries against each
i
j
R .
ii) For each
i
j
R , the entries oI
i
fm can be obtained using Iollowing procedure:
a) Whenever I (
i
j
R ) returns true value then
i
j
gets incremented by 1 i.e.
i
j

i
j
1.
b) Whenever I (
i
j
R ) returns Ialse value then
i
j
gets incremented by 1 i.e.
i
j

i
j
1.
c) It calculates Iault index value
i
j
=
i
j
/ (
i
j
+
i
j
) where 0
i
j
1.
iii)
i
m
r maintain three tables
1
.t r
i
m
,
2
.t r
i
m
and
3
.t r
i
m
where
1
.t r
i
m
store the record oI
i
l k j , ,
against
i
l
R and
2
.t r
i
m
store the value oI hop count oI all
i
l
R oI resource pool Irom
i
m
r and
3
.t r
i
m
store the record oI
alternate resource
i
o
R corresponding to
i
k j
T
,
executing on
i
l
R and.
iv) For each
i
l
R , the entries oI
1
.t r
i
m
can be obtained using Iollowing procedure:
a)
i
m
r obtain value
i
l
Irom
i
fm .
b) Calculate
i
l k j , ,
/
i
l
.
v) TransIer the value oI
i
l k j , ,
to
i
l
R
vi)
i
l
R will send the status oI conIiguration Iile to
i
m
r aIter
i
l k j , ,
time units.
vii) For each
i
j
R , the entries oI
2
.t r
i
m
having Iorm (
i
j
R ,
i
j
hc ) can be obtained using Iollowing Iormula:
i
j
hc = No. oI sites that occur in the shortest path Irom
i
m
r to
i
j
R .
563 Elsevier Publications, 2013
Sandip Kumar Goyal and Manpreet Singh

viii) For each
i
j
R , the entries oI
3
.t r
i
m
having Iorm (
i
j
R ,
i
k o
T
,
,
i
l
R ) can be obtained using Iollowing procedure:
a)
i
m
r obtain
i
j
L value Irom
i
SM
and
i
j
hc value Irom
2
.t r
i
m
and
i
j
value Irom
i
fm .
b) Set
i
k o
T
,

i
k l
T
,
(<
i
k l
T
,
,
i
j
R , etc (
i
k l
T
,
,
i
j
R )>)
i
k
Om
, 1
.
c) Compute
i
j
av = (
i
j
L
i
j
hc
i
j
) / 3.
d) Let RL = {
i
o
R } where o c (1, 2,.,
i
n ) and RL1 RL -
i
j
R .
e) Compute RL2 ) 1 (RL
av

i
R
1
,
i
R
2
,..
i
n
R
1
}.
I) Set
3 , , 3
.
j
i
m
t r
i
l
R =
i
R
1
and each updated copy oI conIiguration Iile oI
i
j
R will be sent to
i
l
R .
During exection II (I (
i
l
R ) = False) then perIorm Iollowing steps:
a)
i
l
R notiIy its status to
i
IS using
s
d
nf where s=
i
l
R & d
i
IS which in turn give this inIormation to
i
fm using
s
d
nf where s
i
IS & d
i
fm and it will update its
i
l
value.
b)
i
fm give this inIormation to
i
m
r .
c)
i
m
r will instruct the alternate resource
3 , , 3
.
j
i
m
t r having the updated conIiguration inIormation oI
i
k j
T
,
to start the execution Irom the saved state.
d)
i
m
r notiIy the new mapping to SM m
i
S
. using
s
d
nf where s
i
m
r & d SM m
i
S
. .
e) AIter successIul completion oI
i
k j
T
,
, the corresponding resource
i
l
R notiIy its status to
i
IS using
s
d
nf
where s=
i
l
R & d
i
IS which in turn give this inIormation to
i
fm using
s
d
nf where s
i
IS & d
i
fm
and
i
fm will update its
i
l
value.
I)
i
l
R will send back the results oI
i
k j
T
,
to
i
S
m which in turn gives the results to corresponding user.
Step 9: Stop.
5. Performance Analysis
The proposed Iault tolerant Iramework is simulated in Java and its perIormance is measured with varying
load and scalability. Table 2 speciIies the simulation environment. The results shown in Fig. 2
conIirmed that the proposed Iramework improves the number oI successIul job completion. This is due
to the Iact that the proposed Iramework restores the job execution quickly thereby enabling more number oI
jobs to be completed successIully as compared to the system Without Fault Tolerance (WFT).
564 Elsevier Publications, 2013
A Framework for Fault Tolerant Job Scheduling in Distributed Systems

Table 2. Simulation Parameters
Simulation Runs 50
Number oI Resources 10 40
Number oI Jobs 50 200
Fault Index oI Resource 0 1
Deadline Duration (hrs.) 1 25
Job Size (MI) 100000 - 1000000
Resource Processing Power (MIPS) 100 - 500
Fig. 2. PerIormance comparisons in terms oI successIul job completion
565 Elsevier Publications, 2013
Sandip Kumar Goyal and Manpreet Singh

Fig. 3. EIIect oI load variation on Iailure probability percentage
Fig. 4. EIIect oI scalability on Iailure probability percentage
Fig. 3 and Fig. 4 show less Iailure probability percentage under proposed approach in terms oI variation
in jobs and resources respectively. The rationale behind this is reliable resource selection as the resources
with large Iault index are rarely selected Ior job execution.
566 Elsevier Publications, 2013
A Framework for Fault Tolerant Job Scheduling in Distributed Systems

6. Conclusion
The ability to tolerate Iailures while eIIectively exploiting the distributed computing resources in a scalable
manner must be an integral part oI distributed systems. To this end, a novel Iault tolerant job scheduling
Iramework is proposed. The proposed Iramework is simulated and results show that the proposed
approach provides a superior perIormance in terms oI number oI successIul job completion and reduced
Iailure probability to the one without considering Iault tolerance.
References
|1| Abawajy J. Fault-Tolerant Dynamic Job Scheduling Policy, Di s t r i but e d a nd Pa r a l l e l Comput i ng, Lecture
Notes in Computer Science, vol. 3719, pp. 165-173, 2005.
|2| Guerraoui R and Schiper A. Fault-tolerance by replication in distributed systems, P r o c e e d i n g o I
I n t e r n a t i o n a l C o n I e r e n c e o n Reliable SoItware Technologies, Lecture Notes in Computer Science, vol. 1088,
pp. 38-57, 1996.
|3| Storm C. Fault T o l e r a n c e i n D i s t r i b u t e d Co m p u t i n g . SpeciIication a n d A n a l y t i c a l Evaluation oI
Heterogeneous Dynamic Quorum-Based Data Replication Schemes, Springer, pp. 13-79, 2012.
|4| Litke A, Tserpes K, Dolkas K and Varvarigou T. A Task Replication and Fair Resource Management Scheme Ior
Fault Tolerant Grids, Advances in Grid Computing - EGC 2005, Lecture Notes in Computer Science, vol. 3470, pp. 1022-
1031, 2005.
|5| Qureshi K , Khan F , Manuel P and Nazir B. A hybrid Iault tolerance technique in grid computing system, The
Journal oI Supercomputing, vol. 56, pp. 106-128, 2011.
|6| Arasteh B and Hosseini M. A Dependable and EIIicient Scheduling Model and Fault Tolerance Service Ior Critical
Applications on Grid Systems, Future InIormation Technology Communications in Computer and InIormation Science,
vol. 184, pp. 110-122, 2011.
|7| Jacob M, Lakshmi S and Masilamani R. Improving the Grid Scheduling PerIormance with Fault Tolerance Using
Genetic Algorithm, Advances in Intelligent and SoIt Computing, vol. 132, pp. 11-20, 2012.
|8| Chtepen M, Claeys F, Dhoedt B, Turck F, Vanrolleghem P and Demeester P. Providing Fault-Tolerance in Unreliable
Grid Systems Through Adaptive Checkpointing and Replication, Computational Science ICCS 2007, Lecture Notes in
Computer Science, vol. 4487, pp. 454-461, 2007.
|9| Nazir B, Qureshi K and Manuel P. Replication based Iault tolerant job scheduling strategy Ior economy driven grid, The
Journal oI Supercomputing, vol. 62, pp. 855-873, 2012.
567 Elsevier Publications, 2013
Index

F
Fault tolerance techniques (FTTs), 559
algorithm, 561564
performance analysis, 564566
system model, 560