Anda di halaman 1dari 5

A Study on Risk Management for Software

Engineering
Luanna Lopes Lobato1,2, Paulo Anselmo da Mota Silveira Neto1, Ivan do Carmo Machado3
1
Informatics Center. Federal University of Pernambuco. Recife PE. Brazil
2
Computer Science Department. Federal University of Gois. Catalo GO. Brazil
3
Computer Science Department. Federal University of Bahia. Salvador BA. Brazil
{lll,pamsn}@cin.ufpe.br, {ivanmachado}@dcc.ufba.br
Abstract Explicit Risk Management (RM) in Software Product
Lines Engineering (SPL) is considered an open question, as posed
in literature, and confirmed by industrial practices, unlike Single
System Development (SSD), which contains a large set of
evidence. The goal of this research is to synthesize the available
evidence gathered in previous research, in a form of two scoping
studies, which considered RM in SPL and SSD, using the
narrative synthesis method. Through the synthesis we could
identify common risks to both development paradigm, as well as
RM activities and practices most commonly used to apply RM in
projects. In addition, was observed that RM to SPL is still an
open question if compared to SSD.
Software Product Lines; Single System Development; Risk
Management; Narrative Synthesis.

I.

INTRODUCTION

Risk Management (RM) has been used during decades as a


way to reduce the problems faced during to Single System
Development (SSD) to Software Engineering (SE) and others
fields. However, in the context of Software Product Lines
(SPL) engineering the use of RM is not explored in the same
proportions, lacking of studies that address issues related to
empirical evaluations on RM in SPL [1].
In order to understand the state-of-the-art of RM, this paper
presents the analyze and synthesize based on the results from
two scoping studies (also called mapping studies) on RM in
software development: firstly, addressing SPL [SSD]; and next,
SPL [5]. The data synthesis from these studies is focused on
identifying the most common practices applied to RM, and to
identify which risks are likely to occur during a software
development project [5].
The scoping studies were performed based on Arksey and
OMalley [6] and Kitchenham and Charters [11]. The narrative
synthesis (NS) method was applied, since this method is
efficiently used to compare systematic literature reviews [7].
The synthesis was performed based on the guidelines provided
by Rodgers et al. [8] and some experiences described by
Cruzes and Dyb [7].
The remainder of this paper is structured as follows.
Section II describes the related work. Section III presents the
theoretical background for the study and the evidence synthesis
method applied in this research. In Section IV are described the
mains findings. A discussion is presented in Section V, and the
limitations of this research in Section VI. Finally, the last
section concludes the paper with final remarks and direction for
future work.

Proceedings of the EASE 2012 - Published by the IET


ISBN 978-1-84919-541-6

II.

RELATED WORK

Cruzes and Dyb [7] present a tertiary study where were


analyzed literature reviews in the context of SE. The focus is to
understand which are the challenges in synthesizing SE
research and what are the implications for the progress of
research and practice. They concluded that little researches has
been conducted on study synthesis in SE and highlighted the
importance in comparing and contrasting evidence from more
than one study during the synthesis.
Rodgers et al. [8] conducted research based on blinded
comparison of NS against a meta-analysis of the same study
data. The authors proposed a framework as a guidance to
perform NS to synthesize evidence from primary studies
analyzed through literature reviews. They observed that when
rigorously performed, narrative synthesis could add meaningful
and valuable data to a study.
The researches, previously presented, are different from our
study, since in our case a synthesis was performed on results
identified from scoping studies in software development.
III.

BACKGROUND

This section provides the underlying concepts of RM for


SSD [4] and SPL [5], and presents an overview on scoping
study and how it is analyzed using evidence-based SE, through
the NS.
A. RM in SSD and SPL
The SPL development is more complex than SSD since it
demands the management of different products, domains, and
so more initial effort is spent [2]. Thus, the main consequence
is that SPL require much more than new technical practices. As
highlighted by Linden et al. [9], the initial investment in SPL is
higher than in SSD, however, after around three products
produced, on the basis of the same SPL, this paradigm becomes
more rewarding. The same holds true when it comes to RM
practices, due to the increased complexity of a range of
products, which composes the product line [5].
In terms of RM, similar characteristics fit well in both SPL
and SSD. However, a number of issues should be considered
when working with SPL projects, which are following
described: Assets reuse among different products; Variability
specification and management; Traceability on the assets;
Scope and Requirements definition for the whole domain;
Product development; and Architecture development, etc.

47

B. Scoping Studies to RM in SSD and SPL


Scoping studies are valuable to provide a narrative or
descriptive account of available research [11]. However, there
is limitation with the use of scoping studies. It does not
appraise the quality of evidence in the studies reports in any
formal sense and does not address the issue of synthesis [6]
[11]. The lack of research synthesis and quality appraisal is
what distinguishes these studies from systematic reviews [7].
Thus, the previous studies on RM in SSD [4] and SPL [5]
were performed using scoping studies practices combined with
systematic review practices, in order to assess the primary
studies quality [5].
C. Narrative Synthesis Guidance
According to Rodgers et al. [8] NS is applied to reviews of
qualitative and/or quantitative studies, providing a narrative
summary about the findings from the studies. As presented by
Cruzes and Dyb [7] the drawbacks of this method is that are
described several variants and lack procedures/standards to
perform it, which may be reliant on bias of the reviewer.

TABLE I.

IV.

FINDINGS

In this section, we present the results identified through the


evidence analysis from two scoping studies to RM in the SPL
[1] and SSD [4].
A. Synthesizing the results
In order to achieve the goals of this research, we analyzed
the extracted data in both qualitative and quantitative ways
from 30 primary studies in the first scoping study (SS-SPL)
and 74 in the second one (SS-SSD). The results were tabulated
and narrative descriptions were provided in order to present an
overview of the findings.
In this section, we describe the risks identified during the
scoping studies and synthesize these achievements in order to
provide a risk list to SPL. In addition, we present information
about the reviews in order to sketch the state-of-the-art of RM
in software development.
1) Publication Source.
In this sub-section, are presented the publication channels
used to collect the primary studies in both scoping studies. This
comparison is justified since we would like to know the
sources in which the risks were found. Table I presents the
sources of the primary studies.

NUMBER OF STUDIES BY PUBLICATION CHANNEL OF SS-SPL AND SS-SSD

Publication channel of the SS-SPL


Conferences
Num
Journals
SPLC
4
IEEE Software
Euromicro
2
IST
ESEC/FSE
2
JSS
ICSE
2
Journal Computer
REFSQ
1
CACM
Computer Standards &
IWPSE
1
Interfaces
Science of Computer
HICSS
1
Programming
QoSA
1
Total:
APSEC
1
PFE
1
QUTE1
SWAP

Num
4
3
1
1
1
1

Publication channel of the SS-SSD


Conferences Num
Journals
HICSS
4
JSS
ESEM
3
IEEE Software
METRICS
2
CACM
ICCSIT
2
ACM SIGSOFT
ICSE
2
IST
ISESE

RCIS

12

ICCIT
SACLA
ICGSE

1
1
1

FITME

SCM

CSNT

Total:

18

SERP
ITNG
ICMIT
SEDM
IEMC
ICIME
ICCRD
ASE
ICCASM
ICCET
ICIMT
Total

1
1
1
1
1
1
1
1
1
1
1
31

The scoping studies shared a number of publication


channels as sources to select the primary studies, since both
were performed in the SE area. However, despite the
commonality in the area, there are some publication channels

Proceedings of the EASE 2012 - Published by the IET


ISBN 978-1-84919-541-6

TSE
Information and
Management
Technovation
JSW

Num
10
8
4
3
3
2
2
1
1

Software Process
Improvement
IEEE Security and
Privacy
JATIT
IEEE Computer
AJBAS
EDPACS
Total

1
1
1
1
40

Reference Lists
*Technical Reports
Total

Num
3
3

1
1

specific each context, for example, SPLC (Software Product


Line Conference) was not addressed in SS-SSD.
Despite the commonality in the sources, few primary
studies were found in the same venue. It can be justified by the

48

fact that RM is an activity that is not as explored as other


software development activities. Thus, this issue turns out to be
very specific
2) Publication Source.
We looked to for the publication channels used to collect
the primary studies in both scoping studies. This comparison is
justified since we would like to know the sources in which the
risks were found [1].
There are some publication channels specific to each
context, for example, SPLC (Software Product Line
Conference) was not addressed in SS-SSD. Despite the
commonality in the sources, few primary studies were found in
the same venue. It can be justified by the fact that RM is an
activity that is not as explored to SPL as other development
activities. Thus, this issue turns out to be very specific [1].
B. Identified Risks
As a way to understand what has been studied in RM
through the available literature about software development
projects, we defined a research question that guided the data
collection. These question were based on that ones presents in
our scoping studies [1] [4].

TABLE II.

Management risk

Implementation risk

Categor
ies
Cost

ID

Risks Name

R1.1
R2.1
R2.2
R2.3
R2.4
R2.5
R2.6
R2.7
R2.8
R2.9
R2.10
R2.11
R2.12
R2.13
R2.14
R2.15

Inaccurate cost estimation


Core assets instability
Deliver fewer functions than promised
Immature architecture
Immature scoping
Immature process
Immature requirements
Implementation errors
Inadequate features definition
Inadequate quality of the artifacts
Inappropriate reuse
Inadequate core assets traceability
Lack of architecture documentation
Platform not Mutable
Pollution of the platform (gold plating)
Unnecessary Variability

R3.1
R3.2
R3.3
R3.4
R3.5
R3.6
R3.7
R3.8
R3.9
R3.10
R3.11
R3.12
R3.13
R3.14
R3.15

Bad practices in management


Failure to include new tasks
Failure to prioritize artifacts
Inadequate configuration management
Inadequate planning
Inadequate resource allocation
Inadequate risk management
Inadequate team size
Lack of SPL background
Lack of planning
Lack of project understanding
Lack of risk management
Lack of training
Inadequate training
No products focus

Proceedings of the EASE 2012 - Published by the IET


ISBN 978-1-84919-541-6

RQ1. Which risk were identified and reported? It aims to


identify the risks addressed by the studies. The risks are
classified into categories for better visualization of the current
RM scenario to software development.
In our previous study, we presented a definition of a risk
category [4]. Thus, the risks were classified in risks categories
[5] to facilitate the management, since is know the source of
the risks and the stakeholder involved can be identified. The
categories can also classified as target areas for applying risk
control activities. Table II presents the total amount of primary
studies that addressed the risks in the scoping studies. A total
of 32 different risks have been identified in the SS-SPL, while
the SS-SSD pointed out 56 risks. Some of the risks identified in
SPL, also appeared in SSD. These risks were also classified in
SPL activities, Core Assets Development (CAD), Product
Development (PD) and Management (M) [10], based on our
expertise in an industrial case study [3]. All disagreements
were solved among the authors of this paper. Since the risk
occurrence can vary according the project and the manager, the
risk occurrence in a specific activity is not absolutely true to all
SPL projects.

IDENTIFIED RISKS
SSSPL

SSSPL
0
2
0
3
4
1
0
0
2
1
4
0
0
1
2
2
5
0
0
2
0
0
0
0
1
0
0
0
0
1
3

SSSSD

SSSSD
16
0
3
0
6
0
22
2
0
0
0
3
1
0
4
0
21
1
1
1
1
4
2
1
0
7
28
6
8
0
0

CAD

SPL Activities
PD
M

49

Operatio
nal

Organiz
ational

User risks

Technical risk

Schedul
e risk

R3.16
R3.17
R3.18
R4.1
R4.2
R4.3
R4.4
R5.1
R5.2
R5.3
R5.4
R5.5
R6.1
R6.2
R6.3
R6.4
R7.1
R7.2
R7.3
R7.4
R7.5
R7.6
R7.7
R7.8
R7.9
R7.10
R7.11
R7.12
R7.13
R7.14
R7.15
R7.16

Rework
Unexpected project scope expansions
Workload on staff
Inadequate communication
Inadequate system integration
Legacy system integration issues
Security and privacy issues
Bureaucracy issues
Difficulties in introducing SPL
Immature organization
Infrastructure unavailability
Limited development resources
Delay in time-to-market
Missed schedule
Slow process of change
Tight schedule for the project
Absence of test
Failure in requirements identification
Immature domain
Inaccurate of changes estimation
Inadequate continuous process improvement
Inadequate system performance
Inadequate project monitoring
Immature process
Inadequate technology, methods and process
Inadequate technical documentation
Lack of documentation standards
Lack of support tools for RM
Project complexity
SPL complexity
Test complexity
Third-party components not certified

R8.1
R8.2
R8.3
R8.4
R8.5
R8.6
R8.7
R8.8
R8.9
R8.10
R8.11

Absence of domain experts


Absence of scope experts
Customer dissatisfaction
Difficulties in acquiring knowledge
Instability of staff productivity
Lack of customer involvement
Lack of expertise in management
Lack of team commitment
Not qualified staff
Staff turnover
Team expertise diversity issues

V.

DISCUSSION

This work contributes to a better understanding about the


challenges in apply RM to SPL projects. The main findings
reported are related to both the risks identified and the SPL
activities used to manage these risks. It was observed a number
of obstacles faced to introduce SPL, from cultural
organizational barriers to problems with the team.
Only one risk was classified into the Cost risk category,
which can occur during the three SPL activities. Regarding to
Implementation risk, we can observe that all risks were
classified as being a risk that happens during CAD. This
happens due to the fact that the CAD is the activity in which
the assets are developed, and put together to compose the

Proceedings of the EASE 2012 - Published by the IET


ISBN 978-1-84919-541-6

0
0
1

2
5
1

2
0
0
0

8
11
1
1

0
2
0
0
2
2
5
1
0
0
4
3
0
0
0
0
1
0
4
0
2
0
5
3
1
1
0
0
0
0
0
0
5
1
0
0

1
0
11
1
6
1
4
0
15
2
18
0
38
5
6
10
0
20
6
4
2
15
0
0
0
6
12
2
1
2
11
2
18
20
7
4

products. The risks of this category were also encompassed in


PD, and, finally, a small number was classified in the M.
Risks related to Management risk are those that occur due
to failures during project management, and some of them were
classified into CAD and PD. Operational and Organization
risks are resulting mainly from bad management. Schedule risk
are those for which the time is an impact factor to project
performance. Technical risks occurred during CAD, PD and M.
Finally, Users risks are those that occur by interference of the
users, and can be classified into all three SPL activities.
Thus, during the SPL project development, those three
activities must be constantly monitored in order to capture the
possibilities of risks occurrence. If the risks became real in the
project, they need to be quickly solved. In order to define
which risk must be managed first, they can be assessed in order

50

VI.

LIMITATIONS AND THREATS

The following limitations may be considered: Several


primary studies lack sufficient information regarding RM.
Thus, during the data collection, we had to infer on the
available findings; The same research team was responsible to
both studies (RM-SPL and RM-SSD) as well as the analysis of
evidence, so the results can potentially have been biased; As
the NS is a subjective method, different viewpoints could be
considered if other researchers had performed the analysis, and
so develop different syntheses. In order to reduce the threats,
three researchers had been actively involved in this work.
VII.

CONCLUSIONS

The scenario about the lack of RM during development


projects is more real in companies that develop software based
on SPL paradigm [3]. It was verified through the execution of
two scoping studies in SPL and SSD to RM. It was observed
the few studies reporting experiences in applying RM for SPL
projects compared with SSD [5]. In addition, the quality of the
studies selected to SSD is higher than in SPL, since in SPL few
researches have been directly linked with results from RM.
In order to define the categories that best set the SPL
aspects, we proposed a risk categorization [5]. The risks were
grouped in categories to make easier manage them, once that
risk list can become extensive. In addition, the risks were
classified in SPL activities. This was an attempt to formalize
the development stages in that specific risks should be most
verified. The decision about the most appropriated SPL activity
was taken in concordance of the authors of this paper.
Thus, it was observed that RM is a practice that deserves
greater attention during SPL development. As future work, we
intend to define a approach for RM in SPL. This approach will
present the activities, practices and steps that must be follow to
perform RM, as well as the possible risks that can be found
during the scoping and requirements disciplines in SPL.

Proceedings of the EASE 2012 - Published by the IET


ISBN 978-1-84919-541-6

REFERENCES
[1]

Lobato, L. L., O'Leary, P., Almeida, E. S., Meira, S. R. de L. (2010).


The Importance of Documentation, Design and Reuse in Risk
Management for SPL. In: 28th ACM International Conference on
Design of Communication (SIGDOC), 2010, So Carlos.
[2] Clements, P. and Northrop, L. M. (2001). Software Product Lines:
Practices and Patterns. Boston MA U.S.A.: Addison-Wesley, Aug 2001
[3] Lobato, L. L., Silveira Neto, P. A. M., Machado, I. C., Almeida, E. S.
and Meira, S. R. L. (2012a). Risk Management in Software Product
Lines: An Industrial Case Study. In: International Conference on
Software and Systems Process (ICSSP), 2012, Zurich. Washington DC:
IEEE Computer Society, 2012.
[4] Lobato, L. L., Almeida, E. S. and Meira, S. R. L. (2012c). Risk
Management in Software Engineering: A Scoping Study. In: 16th
International Conference on Evaluation & Assessment in Software
Engineering (EASE), 2012, Ciudad Real, Espanha.
[5] Lobato, L. L. (2012). An approach for Risk Management in Software
Product Lines. Ph. D. thesis proposal. Federal University of
Pernambuco, Brazil, 2012, pp 214p.
[6] Arksey, H. and OMalley, L. (2005). Scoping studies: towards a
methodological framework, International Journal of Social Research
Methodology 8 (1) (2005) 1932.
[7] Cruzes, D. S. and Dyb, T. (2011). Research Synthesis in Software
Engineering: A Tertiary Study, Information and Software Technology,
In Press, Accepted Manuscript, Available online 18 January 2011, ISSN
0950-5849.
[8] Rodgers, M., Sowden, A., Petticrew, M., Arai, L., Roberts, H., Britten,
N., and Popay, J. (2009). Testing Methodological Guidance on the
Conduct of Narrative Synthesis in Systematic Reviews, Evaluation,
15(1): 4974.
[9] Linden, F. J. v. d., Schmid, K. and Rommes, E. (2007). Software
Product Lines in Action: The Best Industrial Praczice in Product Line
Engineering. Secaucus, NJ, USA: Springer-Verlag New York, Inc..
[10] Northrop, L. M. (2002). SEI's Software Product Line Tenets. IEEE
Softw. 19, 4 (July 2002), 32-40.
[11] Cruzes, D. S. and Dyb, T. (2010). Synthesizing evidence in software
engineering research, in: Proceedings of the 2010 ACM-IEEE
International Symposium on Empirical Software Engineering and
Measurement, ESEM 10, ACM, New York, NY, USA, 2010.

51