energies

Article
Augmented Reality Applications for Substation
Management by Utilizing Standards-Compliant
SCADA Communication
Miro Antonijević *, Stjepan Sučić and Hrvoje Keserica
Končar-Power Plant and Electric Traction Engineering Inc., Fallerovo 22, 10000 Zagreb, Croatia;
stjepan.sucic@koncar-ket.hr (S.S.); hrvoje.keserica@koncar-ket.hr (H.K.)
* Correspondence: miro.antonijevic@koncar-ket.hr; Tel.: +385-99-364-8470

Received: 17 February 2018; Accepted: 6 March 2018; Published: 8 March 2018

Abstract: Most electrical substations are remotely monitored and controlled by using Supervisory
Control and Data Acquisition (SCADA) applications. Current SCADA systems have been significantly
enhanced by utilizing standardized communication protocols and the most prominent is the IEC 61850
international standard. These enhancements enable improvements in different domains of SCADA
systems such as communication engineering, data management and visualization of automation
process data in SCADA applications. Process data visualization is usually achieved through Human
Machine Interface (HMI) screens in substation control centres. However, this visualization method
sometimes makes supervision, control and maintenance procedures executed by engineers slow and
error-prone because it separates equipment from its automation data. Augmented reality (AR) and
mixed reality (MR) visualization techniques have matured enough to provide new possibilities of
displaying relevant data wherever needed. This paper presents a novel methodology for visualizing
process related SCADA data to enhance and facilitate human-centric activities in substations such as
regular equipment maintenance. The proposed solution utilizes AR visualization techniques together
with standards-based communication protocols used in substations. The developed proof-of-concept
AR application that enables displaying SCADA data on the corresponding substation equipment with
the help of AR markers demonstrates originality and benefits of the proposed visualization method.
Additionally, the application enables displaying widgets and 3D models of substation equipment to
make the visualization more user-friendly and intuitive. The visualized SCADA data needs to be
refreshed considering soft real-time data delivery restrictions. Therefore, the proposed solution is
thoroughly tested to demonstrate the applicability of proposed methodology in real substations.

Keywords: SCADA; augmented reality; IEC 61850; substation management

1. Introduction
In recent years, augmented reality (AR) and mixed reality (MR) technologies and devices (e.g.,
Google Glass [1], Microsoft Hololens [2]) have introduced a significant amount of technological
hype by offering the ability to augment a user’s real-world vision field with contextual information.
These technologies, apart from some technical limitations, have the potential to be applicable
in many different areas since they can move domain-specific visualizations from the computer
screen to the real world. This paper presents a novel methodology for visualizing process related
SCADA data to enhance and facilitate human-centric activities in substations such as regular
equipment maintenance. The proposed solution utilizes AR visualization techniques together with
standards-based communication protocols used in substations.
Research of human-centric industrial automation AR applications has been addressed by several
research projects and activities. Tamaazousti et al. propose a solution to real-time augmentation of a 3D

Energies 2018, 11, 599; doi:10.3390/en11030599 www.mdpi.com/journal/energies

Energies 2018, 11, 599 2 of 17

object by using a framework called Constrained SLAM (Simultaneous Localization And Mapping) [3].
They also created an intractable prototype application for training and education of the industrial
automation field workers. S. Kim et al. tried to enhance sensory driven actions like driving a car with
AR and Haptic technologies [4]. They concluded that technology-controlled sensory augmentation
could impact human perception and improve the execution of those actions but depending on the
targeted population and the amount of augmentation, can have negative effects like overloading the
user with too much information. Sebillo et al. have used AR technologies to improve training of
workers in the emergency domains [5]. Emergency domains have certainly benefited from the usage
of information technologies regarding improving the emergency control processes by providing tools
which support decision making in critical situations. The problem which arises is the fact that these
critical situations cause stress and lower the ability to use these crisis systems especially if the workers
are not familiar with them. The authors have proposed a solution to this problem by providing an
AR-based training system which uses mobile technology to virtually offer two visualization techniques
which train the users with visual metaphors.
Majority of AR/MR applications are not focused on a specific industry, therefore, exploring
generic solutions in data visualization. However, each application domain introduces a new set of
requirements with specific prerequisites. Considering the application of AR in the industry, Constanza
et al. have researched its ways to use AR in the manufacturing and remote collaboration [6]. Tang et al.
found that AR aided assembly tasks were improved by a significant amount when contrasting them
with regular tasks processes being executed alongside manuals [7]. Ferrarini and Dede explain that
virtual 3D visualization is a good way to model manufacturing systems and since these systems are
usually used for supervision, control and maintenance they fall in the category of SCADA systems [8].
Soete et al. have created a functional AR monitoring application which uses a static camera, AR
markers, OpenCV (Open Source Computer Vision Library, an open source computer vision and
machine learning library) technology and computer vision techniques to provide enhancement of
traditional SCADA functionalities [9]. They proved that AR and MR could, in fact, be used very
efficiently in the monitoring domain of SCADA systems. Their concerns were the impracticality of
hardware and software equipment they used stating that exporting similar functionalities to smaller
devices like smartphones and tablets or even head worn equipment would significantly improve
results. Also, improving the software by making it easier and more optimized would also offer
enhancements to their concepts.
Applying state-of-the-art technologies in traditionally conservative power system automation
domain is one of the Smart Grid pillars as already shown for the next-generation SCADA systems [10]
and equipment monitoring [11,12]. This paper presents a novel methodology for visualizing process
related SCADA data to enhance and facilitate human-centric activities in substations such as regular
equipment maintenance. The proposed solution utilizes AR visualization techniques together with
standards-based communication protocols used in substations and other power systems automation
domains [13]. The developed proof-of-concept AR application enables displaying SCADA data on
the corresponding substation equipment with the help of AR markers demonstrates originality and
benefits of the proposed visualization method. The visualized SCADA data needs to be refreshed
considering soft real-time data delivery restrictions. Therefore, the proposed solution is thoroughly
tested to demonstrate the applicability of proposed methodology in real substations. The results
have shown that the developed applications can be used to apply AR almost anywhere in a SCADA
monitored system, removing the need for static hardware like a camera and that the applications can
not only enhance monitoring but control and maintenance procedures. Faster and more user-friendly
tools for development of these applications can be used nowadays, thus making the creation and
maintenance process of the applications quicker and less error-prone.
Energies 2018, 11, 599 3 of 17

2. Augmented Reality Applied in Industrial Systems

By using dedicated monitoring and control computers that allow full or partial automation of
complex operations and can aid in making manual control more safe and accurate, automation and
continuous feedback are made possible and industrial machines have become advanced tools [14].
With the readily available real-time information and the ability to control many parameters through
computer interfaces, automation enables an operator to monitor multiple machines at the same time,
reducing the number of staff for a machine pool. Nevertheless, many vital operations cannot be fully
automated and operators may need to simultaneously monitor numerous quickly changing parameters
while visually following and interactively controlling components of the ongoing procedure.
AR technologies have matured enough to be used in distinct industrial automation environments
and successfully applied in manufacturing [14], smart building management [15], automotive
and aerospace industries [16]. AR technologies are used in in substation environments in [17,18].
Nevertheless, AR is used for simulation and training purposes in both articles and none of the already
mentioned articles deals with process related SCADA information used for substation maintenance
purposes as proposed in this paper.

3. Augmented Reality and the IEC 61850 Standard

Frequently considered as just another remote control protocol for electric utilities, IEC 61850 [19]
is more than a set of rules and encodings for information retrieval from field instruments as it
determines automation architecture conditions for utility subsystems to implement communication and
semantic interoperability among multi-vendor equipment. Even though it was primarily developed
for substations [9,20], IEC 61850 has now been broadened for the wind power plant domain [21],
Distributed Energy Resources (DERs) [22] and hydroelectric power plants [23]. This paper is based on
analyzing the implemented IEC 61850 stack developed by Končar-Power Plant and Electric Traction
Engineering Inc., Fallerovo 22, 10000 Zagreb, Croatia for its SCADA system PROZA NET v3.4.

3.1. IEC 61850 Standard

3.1.1. The Data Model

IEC 61850 offers data semantics which is closely linked to device functionalities in utility
subsystems such as DERs [22]. Data models [24] defined by IEC 61850 use object-oriented modelling
of relevant process automation data. Figure 1 shows connections between IEC 61850 data model
classes. A physical device, i.e., a device controller, is represented by the highest parent class, the Server.
The top parent class, the Server, is comprised of one or more Logical Devices (LD), which are virtual
representations of devices used for control, protection and supervision of automated systems. Piecing
together several Logical Nodes (LN) representing many different device functionalities creates an LD.
LNs represent a vital component of IEC 61850 data semantics. More about the IEC 61850 data model
can be read in

3.1.2. Exchange of Data

IEC 61850 introduced a new paradigm to describe data exchange procedures in subsystems of
the utility industry (substations) called Abstract Communication Service Interface (ACSI) [24]. ACSI
model classes define horizontal and vertical communication abstract information services used by
IEC 61850 devices. The ACSI paradigm should not be considered a protocol but a method to link
application layer protocols like MMS (Manufacturing Message Specification, an international ISO
9506 standard for transferring real time process data and supervisory control information between
networked devices or computer applications) to IEC 61850 generic services [25].
Any IEC 61850 client software can fully exploit their remote control by utilizing ACSI model
classes as standardized interfaces to IEC 61860 server enabled devices. Several authors have analysed
Energies 2018, 11, 599 4 of 17
Energies 2018, 11, x FOR PEER REVIEW 4 of 17

IEC 61850IEC
analysed data exchange
61850 with the help
data exchange with of
thenetwork simulation
help of network environment
simulation such as OPNET
environment [26–28].
such as OPNET
However, this paper is based on testing in laboratory setup with real-world equipment. [24].
[26–28]. However, this paper is based on testing in laboratory setup with real-world equipment.

Information modeling scope

Server
1…n
Logical Device (LD)
1…n
Logical Node (LN)
Application scope 1…n 1…n
0…n 1…n
Data Set (DS) Data Object (DO)
1…n 1…n
1 1…n
Functional constraint (FC) Data Attribute (DA)

Figure 1. IEC 61850 data class model.

Figure 1. IEC 61850 data class model.

3.1.3. Management of IEC 61850 Systems

3.1.3. Management of IEC 61850 Systems
The exchange of XML documents formatted according to the System Configuration description
The exchange of XML documents formatted according to the System Configuration description
Language (SCL) [29] is the basis of the engineering process for IEC 61850 systems. Depending on if
Language (SCL) [29] is the basis of the engineering process for IEC 61850 systems. Depending on
they describe equipment or the integrated system itself, there can be distinguished several SCL
if they describe equipment or the integrated system itself, there can be distinguished several SCL
document types. Communication system and electric network topology are predefined while the
document types. Communication system and electric network topology are predefined while the
engineering process based on SCL document exchange is somewhat static and most commonly
engineering process based on SCL document exchange is somewhat static and most commonly applied
applied in substation automation systems.
in substation automation systems.
3.2. Augmented Reality
3.2. Augmented Reality
AR is a computer science technology that visually alters real-world environments with virtual
AR is a computer science technology that visually alters real-world environments with virtual
elements [16]. Finding best ways to apply AR in the energy industry will only be possible after a
elements [16]. Finding best ways to apply AR in the energy industry will only be possible after a
thorough analysis of AR and its capabilities. Main features of AR are as follows:
thorough analysis of AR and its capabilities. Main features of AR are as follows:
• By using a tracking system, it is possible to track the user’s view and location.
• By using a tracking system, it is possible to track the user’s view and location.
• Virtual objects are placed into the view of the real world.
• Virtual objects are placed into the view of the real world.
• Virtual objects placed into the real world are of correct position, orientation and scale each frame.
•• Virtual objectstoplaced
It is possible renderinto
boththe real world
virtual objects areand
of correct position,view
the real-world orientation and scale each frame.
in real time.
• It is possible to render both virtual objects and the real-world view in real time.
The need to use advanced visualization techniques in substation automation environments is
The needobserving
visible while to use advanced
computers visualization
whose primarytechniquestask inis substation
to control automation
and monitorenvironments
processes. The is
visible
amountwhile observing computers
of information whose primary
shown is considerable andtask is tobe
it can control
safelyand monitor
assumed thatprocesses. The amount
in time this
of information
will showndue
keep increasing is considerable
to the growth and
of itthe
can be safely
systems’ assumed
size that in time
and complexity. It this
is ofamount will keep
vital importance
increasing due to the growth of the systems’ size and complexity. It
that only relevant information is shown so that the operators are not unnecessarily burdened is of vital importance that with
only
relevant
extra data.information is shown so that the operators are not unnecessarily burdened with extra data.
Control
Control computers
computersdisplay
displayprocess
processdata
data onon a computer
a computer screen and and
screen also use
alsothe useconventional
the conventionalinput
mechanisms like a keyboard or touch screens. Typical procedures often
input mechanisms like a keyboard or touch screens. Typical procedures often demand the operator demand the operator to watch
the process
to watch thethrough
processathrough
machine’s safety glass
a machine’s and at
safety the and
glass sameattime
the operate
same time a computer
operate atocomputer
control the to
process and to receive automation feedback. This type of workflow creates
control the process and to receive automation feedback. This type of workflow creates the problem the problem of splitting
the attentionthe
of splitting of the operator
attention of when the needwhen
the operator arisestheto simultaneously
need arises to observe the computer
simultaneously observe screen
the
for keeping
computer track for
screen of automation
keeping track values and to visually
of automation valueswatchand tothe process.
visually Placing
watch the displays next to
process. Placing
the machine
displays nextcanto partly addresscan
the machine thispartly
issue but it inherently
address this issue separates the processseparates
but it inherently from its automation
the process
data
from[14].
its automation data [14].
This paper
paper research
researchobjective
objectiveisistotooffer
offer a solution
a solution of seamlessly
of seamlesslyintegrating
integrating automation
automation data data
into
the workspace beyond the safety glass using features of AR technology.
into the workspace beyond the safety glass using features of AR technology. AR technology enablesAR technology enables the use
of
thecomputer-generated
use of computer-generated interactive virtual objects
interactive virtualwith real with
objects physical
realenvironments
physical environmentsincluding including
industrial
industrial environments. The technology is very applicable in immediate attention demanding
Energies 2018, 11, 599 5 of 17

environments. The technology is very applicable in immediate attention demanding industrial machines
because it allows easy to follow representation of relevant process information in real time at the spot
where it would be needed the most.
Some examples where the option of annotating real-world procedures with real-time process
automation data would be useful are as follows. Emphasizing locations of an industrial environment
to a student or a co-worker. It would also be beneficial to annotate some aspects of the workplace
with virtual information such as showing virtual labels with the type of machine, its current state
of operation and any relevant information about its current job. Virtual visual warnings can show
whenever any dangers with the ongoing activity are noticed. Virtual simulations of machine processes
could also increase safety through learning essential methods before actually attempting them in
real environments.
To get the best possible results from combining computer graphics with real environments,
the execution of using AR needs to be seamless and without encumbering tools. Looking at the
history of Virtual Reality (VR) and AR systems head-mounted displays (HMD) [14] which require
complex tracking and large tools to be worn [30]. One such modern example is the Oculus Rift [31].
Video see-through systems have gone up in popularity recently because software libraries and products
such as ARToolkit [32] (a cross-platform SDK for developing augmented reality applications) and
the more modern Vuforia [33] have enabled rapid and easy prototype implementation. Spatial AR
systems have been in research for some time now, for example, the ASTOR system [34] (a spatial
augmented reality system developed by Alex Olwal et al.). Major AR systems today have developed
into marker-based applications, meaning that they use AR markers to show virtual information and
track real-world points of interest. This paper explores a fusion of markers, specifically Quick Response
(QR) codes [35] and Vuforia frame markers [33] to create a new type of AR marker used in the first
prototype application built as part of this research. Later versions of the application use Vumarks [36],
a different type of Vuforia markers to achieve similar results.

3.2.1. AR Markers
Although there are many types of AR markers suitable for certain situations, the QR code, a 2D
matrix barcode developed by Denso Wave Corporation in 1994 [35,37], name abbreviated from Quick
Response (QR) Code, is used a more frequently than its adversaries.

Fusion Markers
Despite being able to encode up to 7089 numeric and 4296 alphanumeric characters, a QR code can
be read by a simple smartphone or PC application with camera access that is a significant advantage
over other 2D barcodes that require special scanners for decoding. It can also be generated by using
one of the many free online tools or the Python qrcode library [38].
Another, much more sophisticated, option for markers are markers available in the library called
Vuforia [33]. There are a couple of different marker types in Vuforia, mainly frame markers, image
targets and Vumarks [36]. When the first version of the AR application prototype was in development,
Vumarks were not available, so a combination of Vuforia frame markers and QR code markers was
used. The goal was to use Vuforia frame markers for fast detection and QR code markers for encoding
information inside the marker.
The first AR application prototype used QR codes inside Vuforia frame markers. Figure 2 shows a
Vuforia frame marker, while Figure 3 shows a QR code marker. Since almost all AR markers including
Vuforia frame markers do not encode data, they need to be preregistered. That means that each
marker’s function needs to be a priori decided. Encoding data inside a marker eliminates this step.
That is why using QR codes is more optimal than other markers. A significant problem with QR codes
is that QR code detection is slow. That is why we put them inside fast detectable Vuforia frame markers
like in Figure 4.
Energies 2018, 11, 599 6 of 17
Energies 2018, 11, x FOR PEER REVIEW 6 of 17
Energies 2018, 11, x FOR PEER REVIEW 6 of 17

Energies 2018, 11, x FOR PEER REVIEW 6 of 17

Figure 2. Vuforia frame marker.

Figure2.2.Vuforia
Figure Vuforiaframe
framemarker.
marker.

Alignement
Figure 2. Vuforia frame marker.
Alignement
pattern
pattern
Alignement
Position Area
Position
pattern Area
Position Area
Data Area
Data Area
Data Area

Figure 3. Quick Response (QR) marker with its decoding features highlighted.
Figure3.3.Quick
Figure QuickResponse
Response(QR)
(QR)marker
markerwith
withits
itsdecoding
decodingfeatures
featureshighlighted.
highlighted.
Figure 3. Quick Response (QR) marker with its decoding features highlighted.

Figure 4. Vuforia frame and QR marker fusion.

Figure 4. Vuforia frame and QR marker fusion.
Figure 4.
Figure 3 shows an example ofVuforia frame
a QR code and
with its QR marker
parts fusion. The most important parts of
highlighted.
Figure 4. Vuforia frame and QR marker fusion.
the QR code are three large squares, containing smaller black squares with a white border, located in
Figure 3 shows an example of a QR code with its parts highlighted. The most important parts of
Figure 3 shows an example of a QR code with its parts highlighted. The most important parts of
the QR code are three large squares, containing smaller black squares with a white border, located in
the QR code are three large squares, containing smaller black squares with a white border, located in
Energies 2018, 11, 599 7 of 17

Figure 3 shows an example of a QR code with its parts highlighted. The most important parts of
the QR code are three large squares, containing smaller black squares with a white border, located in
three corners of the code that are used to determine the position of the code. Version 2 (and above)
added a square used to align the code after detection and the rest of the code is used to write a
considerable number of small squares that encrypt the information in the marker.
Another asset of this form is the ability to reconstruct the code despite it being damaged or
missing which is crucial when scanning with a camera as the lens will never be positioned perfectly
thus omitting parts of the code. As seen in Table 1, higher correction levels result in allowing excluding
more of the code in exchange for increased code size [27]. Table 2 [27] presents dominance of the QR
code over other fiducial markers. Everything stated above advocates using QR codes in the substation
automation AR application prototype.

Table 1. QR marker error correction levels [27].

Level Percentage of Codewords That Can be Restored (Approximation)

L 7%
M 15%
H 25%
Q 30%

Table 2. Feature comparison of QR markers with other fiducial markers [27].

Feature QR Code Fiducial Marker

Need to pre-register No Yes
Model storing Internet Local
Limited number of markers Larger Smaller
Universality Universal barcode Standalone

Vumarks
In August 2016, Vuforia released a new type of marker called a VuMark [36]. This marker retained
the fast detection that other Vuforia markers had and it enabled data to be encoded inside the marker.
That is precisely why fusion of QR and frame markers were used in the first prototype, so a transition to
this new type of marker was apparent. Not only does it encode text data like a QR code but a Vumark
can also encode unique identifiers, strings and bytes making it more flexible than a QR code marker.
It is constructed in vector graphics designer tools (specifically Adobe Illustrator [39]) and can be made
visually very appealing as seen in Figure 5. The only downside to a Vumark is that the number of
characters it encodes when you choose to encode a string is maximum 100 that is much less than a QR
code. However, for proposed AR application needs, 100 characters are sufficient. A Vumark’s most
essential components are shown in Figure 6.
The tests conducted in this research showed excellent detection and instantaneous decoding
capabilities of the Vuforia library when using VuMarks. As soon as the Vuforia library detects the
marker, it also manages to decode the ID inside the marker processing the same frame in which the
marker was detected. High-quality cameras enable detection under very unfavourable conditions, for
example, a distance of a couple of meters from the marker or a high angle in regard to the marker—30
and even fewer degrees). That means that decoding is instantaneous and this fact sets Vumarks as
the most superior AR markers as long as the code is of smaller length (ASCII 100 or fewer characters,
1000 numbers or 100 bytes).
Energies 2018, 11, 599 8 of 17
Energies
Energies2018,
2018,11,
11,xxFOR
FORPEER
PEERREVIEW
REVIEW 88ofof17
17

Figure
Figure5.5.
Figure 5.Examples
Examplesof
Examples ofpreviously
of previouslycreated
previously createdVumarks.
created Vumarks.
Vumarks.

Figure 6.AA
Figure6.6.
Figure Vumark
AVumark with
Vumarkwith its
withits decoding
itsdecoding features
decodingfeatures highlighted.
featureshighlighted.
highlighted.

4.
4. Augmented
4.Augmented Reality
AugmentedReality IEC
RealityIEC 61850
IEC61850 Application
61850Application
Application
Development
Development
Developmentof of the
ofthe IEC
theIEC 61850
IEC61850
61850AR AR application
ARapplication
applicationuses uses vertical
usesvertical
verticalIEC IEC 61850
IEC61850 communication
61850communication
communicationwhile while
while
also
also utilizing
alsoutilizing
utilizingSCLSCL data
SCLdata engineering
dataengineering
engineeringand and
andAR AR marker
ARmarker encodings
markerencodings
encodingsto to identify
toidentify components
identifycomponents
componentsof of substation
ofsubstation
substation
automation
automation equipment
automationequipment
equipmentabidingabiding
abidingthe the rules
therules
rulesofof IEC
ofIEC 61850
IEC61850 data
61850data semantics.
datasemantics.
semantics.
Information
Information about semantically
Information aboutsemantically annotated
semanticallyannotated process
annotatedprocess data
processdata is
dataisprovided
isprovided
provided by by
the
bythecontents
thecontents of the
contents ofofSCL
the files.
theSCL
SCL
The
files.information
files. The
Theinformation is thenisis
information correlated
then with decoded
thencorrelated
correlated with data found
withdecoded
decoded data
datainfound
AR markers
found ininARARthat werethat
markers
markers put
thaton
were
weresubstation
put
puton on
equipment
substation (circuit
equipment breakers,
(circuit feeders,
breakers, transformers,
feeders, switches,
transformers, etc.). Using
switches,
substation equipment (circuit breakers, feeders, transformers, switches, etc.). Using the software that the
etc.). software
Using thethat recognizes
software that
AR markersAR
recognizes
recognizes in markers
AR an imagein
markers and
inanandecodes
image
imageand thedecodes
and contents
decodesthe written in thewritten
thecontents
contents markers
written inas
in theAR
the projector
markers
markers as draws
asAR virtual
ARprojector
projector
information
draws
draws virtual
virtual oninformation
the smartphone
information onon thescreen
the of a maintenance
smartphone
smartphone screen
screen of engineer.
of aa maintenance By using
maintenance this workflow,
engineer.
engineer. By
By using process
using this
this
information
workflow,
workflow, process is placed overlaying
process information substation
information isis placed equipment
placed overlaying it relates
overlaying substation to,
substation equipment directly connecting
equipment itit relates components
relates to,to, directly
directly
with their corresponding
connecting
connecting components
components with process
with their data
their in real time fetched
corresponding
corresponding process
processthrough
data
data in a
inSCADA
real
real time
timesystem.
fetched
fetchedIt isthrough
clear that
through aa
by
SCADA integrating
system. AR Itinto
is substation
clear that byenvironment,
integrating its
AR maintenance
into substation activities
SCADA system. It is clear that by integrating AR into substation environment, its maintenance will
environment, be simplified
its and
maintenance will
enable
activities unambiguous
activities will
will be identification
be simplified
simplified and of
and components
will
will enable while presenting
enable unambiguous
unambiguous real-time measurements
identification
identification of
of components
components data, thus
while
while
easing
presenting substation
presenting real-time
real-timemonitoring.
measurements
measurementsdata, data,thus
thuseasing
easingsubstation
substationmonitoring.
monitoring.

4.1.
4.1. Architecture
4.1.Architecture
Architecture
There
There were
wereaaafew
Therewere fewiterations
few iterationsofofof
iterations the
theAR
theAR
AR application
applicationprototype.
application prototype.
prototype. The most
The
The important
most
most twotwo
important
important are the
two arefirst
are the
the
and
first second
first and major
and second versions
second major that
major versions were
versions that developed
that were for the
were developed Android
developed for for the platform
the Android using the
Android platform Java programming
platform using
using the
the Java
Java
language.
programming
programming Thelanguage.
first iteration
language. Theuses
The firstVuforia
first iteration frame
iteration usesmarker
uses Vuforia
Vuforia and QR marker
frame
frame code fusion
marker andand
and QRQRit shows
code
code promising
fusion
fusion and
anditit
results
shows for marker-based
promising results AR
for process data
marker-based visualization.
AR process The
data second major
visualization. iteration
The was
second
shows promising results for marker-based AR process data visualization. The second major iteration made
major with the
iteration
Unity
was
wasmade game
made engine
with
with theframework
the Unity
Unitygame
game[40] and it
engine
engine is also utilizing
framework
framework [40] Vuforia
[40]and
and libraries
ititisisalso but with
alsoutilizing
utilizing different
Vuforia
Vuforia markers
libraries
librariesbut
but
with
with different
different markers
markers called
calledVuMarks.
VuMarks. Since
Since itit was
was developed
developedwith
with Unity,
Unity,itit can
can be
be built
built for
for aa
Energies 2018, 11, 599 9 of 17

Energies 2018, 11, x FOR PEER REVIEW 9 of 17

called VuMarks. Since it was developed with Unity, it can be built for a variety of platforms including
variety
PC, Mac, ofLinux,
platforms including
Android, PC,aMac,
iOS and Linux,and
few others Android, iOS and aisfew
the application others as
functional and theas
long application is
the platform
functional as long as
has access to a camera. the platform has access to a camera.
The
The architecture
architecture of of the
the application
application isis comprised
comprised of of several parts. A
several parts. A device’s
device’s camera
camera scans
scans aa
marker
marker and and decodes
decodes itit into
into aa process
process data
data identifier
identifier based
based on IEC 61850.
on IEC 61850. The
The identifier
identifier uniquely
uniquely
represents the path to the semantically annotated process data compliant to the IEC
represents the path to the semantically annotated process data compliant to the IEC 61850 standard. 61850 standard.
Then,
Then, the
the acquired
acquired data
data is
is correlated
correlated with
with process
process data
data from
from the
the SCADA
SCADA system
system and
and the
the SCADA
SCADA
itself
itself is responsible for fetching its data from its internal database. Intelligent Electronic Devices(IED)
is responsible for fetching its data from its internal database. Intelligent Electronic Devices (IED)
are
are then
then contacted
contacted byby the
the SCADA
SCADA system
systemto togather
gatherthis
thisdata
datavia
via vertical
verticalIEC
IEC61850
61850communication.
communication.
Figure
Figure 77 shows
shows the
the application
application architecture.
architecture.

Substation SCADA system

AR application screen

PhV.phsA.
cVal.mag =
5kA

Smart phone
Transformer

Figure
Figure 7.
7. AR
ARIEC
IEC61850
61850application
application architecture.
architecture.

4.2.
4.2. Implementation
Implementation
Application
Application implementation
implementation includes
includes several
several steps.
steps. The
Theprerequisite
prerequisiteisis to to have
have aa substation
substation
automation
automation environment with a variety of equipment and some IEDs connected to aa central
environment with a variety of equipment and some IEDs connected to central SCADA
SCADA
system. The SCADA system retrieves data in real time from IEDs via
system. The SCADA system retrieves data in real time from IEDs via vertical IEC 61850 communicationvertical IEC 61850
communication which is the most critical prerequisite for AR
which is the most critical prerequisite for AR application implementation. application implementation.
Markers
Markers need
need to to be
be printed
printed (Figure
(Figure 88 shows
shows one
one example
exampleof ofaa Vumark
Vumark withwith IEC
IEC 61850
61850 encoded
encoded
data)
data) and positioned on substation equipment to enrich it with AR visualization of process data
and positioned on substation equipment to enrich it with AR visualization of process data
related
related to
to that
that piece
piece ofof equipment.
equipment. Markers
Markers have
have their
their corresponding
corresponding IEC IEC 61850
61850 identifiers
identifiers encoded
encoded
which
which describe
describethe thepath
pathtoto uniquely
uniquely identified process
identified data
process according
data accordingto the
to IEC
the 61850 standard.
IEC 61850 The
standard.
identifiers correspond to process data of substation equipment and they are
The identifiers correspond to process data of substation equipment and they are positioned on.positioned on. Vumarks
that are created
Vumarks that areusing Adobe
created usingIllustrator and the Vuforia
Adobe Illustrator and the official
Vuforiawebsite [33,36,39].
official website A portable
[33,36,39]. device
A portable
with a camera
device is needed
with a camera (in this
is needed (inpaper an Android
this paper an Androiddevice
deviceSamsung
Samsung Galaxy
Galaxy Tab AA2016
Tab 2016was
wasused)
used)
where
where the AR application was previously installed. Third, the substation environment needs to
the AR application was previously installed. Third, the substation environment needs to have
have
Wi-Fi
Wi-Fi access
access toto its
its internal
internal network
network where
where aa commercial
commercial SCADASCADA system
systemisisconnected
connectedand andavailable.
available.
Integration
Integration with
with SCADA
SCADA application
application isis done
done via
via SCADA
SCADA manufacturer
manufacturer provided
provided API interface [30].
API interface [30].
Energies 2018, 11, 599 10 of 17
Energies 2018, 11, x FOR PEER REVIEW 10 of 17

Figure 8.
Figure A Vumark
8. A Vumark encoding
encoding an IEC 61850 identifier.

The
The substation
substation environment
environment can can be be scanned
scanned by by using
using aa device
device withwith aa camera.
camera. The application
The application
uses
uses an instance of the Vuforia library to process each frame image from the camera. Once aa marker
an instance of the Vuforia library to process each frame image from the camera. Once marker
is
is detected
detected in in aa frame,
frame, the
the Vuforia
Vuforia library
library alsoalso starts
starts processing
processing the the VuMark
VuMark and and decodes
decodes the the string
string
data encoded in the VuMark. This data is an IEC 61850 process data
data encoded in the VuMark. This data is an IEC 61850 process data path that describes the process path that describes the process
data
data path
path that
that is is related
related toto the
the corresponding
corresponding piece piece ofof equipment.
equipment. Now Now the the application
application knowsknows which
which
information it needs to overlay over the view of the marker but it
information it needs to overlay over the view of the marker but it stills needs to fetch this SCADA stills needs to fetch this SCADA
information.
information. ItIt then then connects
connects via via Wi-Fi
Wi-Fi to to the
the SCADA
SCADA system system and and onceonce aa successful
successful connection
connection is is
made,
made, the application asks the SCADA system to fetch all information with the process data
the application asks the SCADA system to fetch all information with the process data path.
path.
The SCADAsystem
The SCADA systemthen then communicates
communicates with with
IEDsIEDs and requests
and requests the appropriate
the appropriate device fordevice for the
the required
required information. Once the IED sends the data, the SCADA
information. Once the IED sends the data, the SCADA system forwards the data over Wi-Fi to the system forwards the data over Wi-Fi
AR
to the AR application. The application maintains its connection to
application. The application maintains its connection to the SCADA system and tasks it with sending the SCADA system and tasks it
with sending real-time updates of changes made to the required data.
real-time updates of changes made to the required data. Anytime information related to the process Anytime information related
to
datathe process
path changes, data thepath
SCADA changes,
system thewillSCADA
notify thesystem will notifyand
AR application thethe
AR application
application willand the
update
application
the change on willitsupdate
display.the Thischange
way aon its display.
real-time display Thisof way a real-time
substation IEC 61850display
dataofissubstation
achieved. IEC
61850Nowdatathatis achieved.
the application has the data required, it needs to display that data over the view of the
Now
marker. that the
Earlier application
application has the
versions useddata required,
OpenGL ES2it[31]
needs to display
technology (a that data
subset of over
OpenGL the view of the
technology,
marker. Earlier application
a cross platform application versions used OpenGL
programming ES2 [31]
interface for technology
rendering 2D (a subset
and 3D of OpenGL technology,
vector graphics) for
adisplaying
cross platform application programming interface for rendering
complex models. The current version uses game engine framework Unity [40]. It is 2D and 3D vector graphics) fora
displaying
flexible system complex models.all
that enables The current
kinds version models
of graphical uses game to beengine
shownframework
and animated. UnityThe [40]. It is a
intuitive
flexible system thatdata
way of displaying enables
is toall kinds aofrectangle
overlay graphicalover models to be shown
the marker with the anddata
animated.
writtenThe on intuitive
it like in
way of displaying data is to overlay a rectangle over the marker
Figure 9. Moreover, a custom texture is created with the information written on it and applied with the data written on it like in
to the
Figure
rectangle9. Moreover,
using OpenGL a custom
ES2.texture
In Unity is engine,
created a with the information
default text Object iswritten createdonandit and applied
overlaid overto the
the
rectangle using OpenGL ES2. In Unity engine, a default text Object
rectangle with the information as its text. These techniques enable immediate update of the required is created and overlaid over the
rectangle
data and showwith the datainformation as its This
to the operator. text. method
These techniques
was used for enable
the immediate update of the required
testing application.
data and showhelper
Different data tovisual
the operator.
effects canThissignificantly
method wasimprove used for thethe testing
time needed application.
to observe essential
Different helper visual effects can significantly improve
data and its changes. Figure 10 shows a pie chart next to the data. This visualization the time needed to observe
method essential
is used
data and its changes. Figure 10 shows a pie chart next to the data.
whenever the data, according to the semantics of the IEC 61850 standard, represents a percentage This visualization method is used
whenever thethe
value. It fills data,
chartaccording
with theto the semanticspercentage
corresponding of the IECand 61850 standard,
keeps represents
it updated a percentage
along with the data
value. It fills the chart with the corresponding percentage and
overlaid over the marker. This approach achieves additional visual feedback, shortening the keeps it updated along with the time
data
overlaid
needed toover observethe marker. This approach
the information. Full video achieves
of demo additional
applicationvisualcanfeedback,
be seen inshortening
[32]. the time
needed to observe the information. Full video of demo application can be seen in [32].
Energies 2018, 11, 599 11 of 17
Energies 2018, 11, x FOR PEER REVIEW 11 of 17
Energies 2018, 11, x FOR PEER REVIEW 11 of 17

Figure 9. A Vumark marker (left) and a screenshot of the camera view (right) while using the AR
Figure 9. A Vumark marker (left) and a screenshot of the camera view (right) while using the AR
application
Figure showing
9. A showing IEC
Vumark IEC 61850
marker information.
(left) and a screenshot of the camera view (right) while using the AR
application 61850 information.
application showing IEC 61850 information.

Figure 10. Screenshot of the AR application in use (the camera view) virtually showing real-time IEC
61850 percentage
Figure dataof
10. Screenshot (value and
the AR a pie chart).
application in use (the camera view) virtually showing real-time IEC
Figure 10. Screenshot of the AR application in use (the camera view) virtually showing real-time IEC
61850 percentage data (value and a pie chart).
61850 percentage data (value and a pie chart).
Another example of a sophisticated visualization method is shown in Figure 11. The figure
shows a modelexample
Another of a diesel-electric generatorvisualization
of a sophisticated next to a marker. Anyispiece
method shown of equipment in a The
in Figure 11. substation
figure
shows Another
environment example
a modelcan
of abe of a sophisticated
modelled this
diesel-electric visualization
way and
generator nextit to method
is apossible is
marker. to
Anyshown in
highlight Figure 11. The
important parts
piece of equipment figure
andshows
add
in a substation
ainformation
model of avisually.
environment diesel-electric
can be modelled generator
Highlighting waynext
this essential
andtoelements
ita is
marker. andAny
possible to piece
adding of equipment
IEC
highlight 61850 in
real-time
important a substation
parts data
and on adda
environment
information can becan
piece of equipment
visually. modelled
be usedthis
Highlighting way andelements
for essential
maintenance it ispurposes.
possible to highlight
An
and addingoperator important
can
IEC 61850 quickly parts and
viewdata
real-time the onadd
dataa
information
needed
piece visually.
of when goingcan
equipment Highlighting
through essential
thefor
be used process elements
of checking
maintenance and adding
equipment.
purposes. IEC 61850
Animations
An operator real-time data
can alsoview
can quickly on a piece
be added
the dataof
to
equipment
showcase
needed can
when be used
specific
going for maintenance
maintenance
through the processpurposes.
procedures An operator
and
of checking for can quickly
learning
equipment. view the
purposes.
Animations candata
For needed
example,
also be addedwhen
every
to
going through
inexperienced
showcase the process
operator
specific of checking equipment.
needs to procedures
maintenance go through and Animations
a learning can also be
process purposes.
for learning added
to be ableFor to showcase
to handle
example, specific
different
every
maintenance
equipment inprocedures
inexperienced aoperator
substationand
needsforto
learning
environment. purposes.
go through For example,
These aanimated
learning models
processevery
to inexperienced
can then
be ablebe used operator
as
to handle a quickneeds
and
different
to go
visualthrough
way of a learning
enhancing process
the to be
education able to handle
process. different equipment in
equipment in a substation environment. These animated models can then be used as a quick anda substation environment.
These
visual animated models can
way of enhancing thethen be usedprocess.
education as a quick and visual way of enhancing the education process.
Energies 2018, 11, 599 12 of 17
Energies 2018,
Energies 2018, 11,
11, xx FOR
FOR PEER
PEER REVIEW
REVIEW 12 of
12 of 17
17

Figure 11.
Figure 11. Screenshot
Screenshot of
of the
the AR
AR application
application in
in use
use (the
(the camera
camera view)
view) virtually
virtually showing
showing real-time
real-time IEC
IEC
Figure
61850 11. Screenshot
temperature dataofwith
the the
ARmodel
application
of its in use (the camera
corresponding view)
object—a virtually showing
diesel-electric real-time
generator.
61850 temperature data with the model of its corresponding object—a diesel-electric generator.
IEC 61850 temperature data with the model of its corresponding object—a diesel-electric generator.
4.3. Prototype
4.3. Prototype Evaluation
Evaluation
4.3. Prototype Evaluation
AA detailed
detailed model
model of of aa substation
substation was
was also
also developed
developed whichwhich is is displayed
displayed whenever
whenever the the camera
camera
A detailed
sees aa marker
marker with model of a
with aa specific substation
specific ID was
ID encoded. also
encoded. This developed
This model
model can which
can be
be seenis
seen in displayed
in Figure whenever
Figure 12.
12. After the
After the camera
the model
model is is
sees
sees
shown, a marker
the with
user can azoom
specific
in ID encoded.
and out and This model
rotate it in anycan be seenthey
direction in Figure
prefer12.
to After
get a the model
better view is
of
shown, the user can zoom in and out and rotate it in any direction they prefer to get a better view of
shown, the userIEC
the substation.
substation. can61850
zoom real-time
in and outinformation
and rotate itwasin any
alsodirection
added totothey prefer tohighlighting
the model,
model, get a better parts
view of
of
the IEC 61850 real-time information was also added the highlighting parts of
the substation.
the equipment
equipment thatIEC 61850
that the real-time
the chosen
chosen data information
data semantics was
semantics relate also
relate to. added
to. This to the
This feature model,
feature can highlighting
can be
be useful
useful when parts
when the of the
the user
user
the
equipment
needs to that
get a the chosen
general data semantics
overview of the relatesubstation
entire to. This feature
with can possibility
the be useful when
of the user needs
monitoring to
certain
needs to get a general overview of the entire substation with the possibility of monitoring certain
get a general
parts ofof the overview
the substation
substation andof the entire substation
and micromanaging
micromanaging them. with
them. It the
It can possibility
can also
also be
be used of
used to monitoring
to control
control thecertain
the whole parts of
whole substationthe
substation
parts
substation
remotely. and micromanaging them. It can also be used to control the whole substation remotely.
remotely.

Figure 12.
Figure 12. Model
Model of aa substation
substation containing
containing four
four transformers
transformers with
with their IEC
IEC 61850
61850 data
data annotated
annotated
Figure 12. Model of
of a substation containing four transformers with their
their IEC 61850 data annotated
over the transformers.
over transformers.
over the
the transformers.

Employing the
Employing the developed
developed application
application prototype
prototype in
in aa real-world
real-world substation
substation is
is the
the footing
footing of
of the
the
Employing
assessment. The the developed
first results applicationthat
demonstrate prototype in a with
VuMarks, real-world substation
immediate is theresults
decoding footing of QR
and the
assessment. The first results demonstrate that VuMarks, with immediate decoding results and QR
assessment. Theslower
codes, although
although first results demonstrate
than VuMarks, that VuMarks,
VuMarks, combined
combined with immediate
with Vuforia
Vuforia decoding
frame markers
markers results
display and QR
an excellent
excellent
codes, slower than with frame display an
choice as
choice as markers
markers for
for IEC
IEC 61850
61850 data
data semantics.
semantics.
Energies 2018, 11, 599 13 of 17

Energies 2018,
codes, 11, x FOR
although PEER than
slower REVIEWVuMarks, combined with Vuforia frame markers display an excellent 13 of 17

choice as markers for IEC 61850 data semantics.

The application
The applicationhas hasevolved
evolved since
since the the
first first prototype
prototype versionversion
and itsand itsofusage
usage of IEC
IEC 61850 61850
real-time
real-time data has also changed from just showing data as text to using graphical
data has also changed from just showing data as text to using graphical models with annotated data. models with
annotated data. Using the semantics of IEC 61850 data, it is possible to get any
Using the semantics of IEC 61850 data, it is possible to get any information from the SCADA systemsinformation from the
SCADA
inside systems inside
substations. substations.
The visual aspect The
of the visual aspect of
application hasthe application
also has also
been changed. been
From changed.
ARToolkit’s
From ARToolkit’s standard markers to Vuforia frame markers combined
standard markers to Vuforia frame markers combined with QR codes, to VuMarks as the fastest with QR codes, to
VuMarks as the fastest detectable markers with the ability to encode different types
detectable markers with the ability to encode different types of data. The graphics were done using the of data. The
graphics ES2
OpenGL weretechnology
done using the OpenGL
explicitly ES2 the
targeting technology
Androidexplicitly
platform targeting the Android
[33]. The current versionplatform
draws
[33]. The current version draws graphics using the game engine framework
graphics using the game engine framework Unity which is then also used to port the application Unity which is then also
to
used to port the
different platforms. application to different platforms.

5. Augmented
5. Augmented Reality
Reality and
and Real-Time
Real-Time Communication
Communication
By definition,
By definition,real-time
real-time computing
computing (RTC),
(RTC), or reactive
or reactive computing
computing describesdescribes
hardware hardware and
and software
software systems subject to a “real-time constraint”, for example from event
systems subject to a “real-time constraint”, for example from event to system response. Real-time to system response.
Real-time must
programs programs must guarantee
guarantee a responseawithin
response withintime
specified specified time constraints
constraints [30]. The system
[30]. The real-time real-timeis
system is one which sends updates to clients as soon as they happen and can be
one which sends updates to clients as soon as they happen and can be said to be real-time constrained said to be real-time
constrained
by by a certain
a certain (usually (usually
relatively short)relatively short)the
period called period called the
“deadline”. That“deadline”.
means that That
as soonmeans that as
as a change
soon as a the
happens, change happens,
client the client
is guaranteed is guaranteed
to receive to receive
the update the update
notification at thenotification at the endperiod
end of the deadline of the
deadline period that starts when
that starts when the change occurred. the change occurred.
In presented
In presentedresearch,
research,thethe tests
tests measured
measured the time
the time neededneeded
for AR forapplication
AR application prototypes
prototypes to
to receive
receive
an updatean ofupdate
a dataofchange
a data in
change in the SCADA
the SCADA system. Twosystem. Two experiments
experiments were done;were
thedone; the
first is first is
SCADA
SCADA response time when transferring data over a 100 MBit local Virtual
response time when transferring data over a 100 MBit local Virtual Private Network (VPN) and the Private Network (VPN)
and the issecond
second is the
the same same response
response time using timethe
using4G the 4G mobile
mobile Internet Internet as a network
as a network mediator.
mediator. Both
Both tests
tests measured
measured SCADA SCADA
changechange
responseresponse
times by times by changing
changing a data instance
a data instance 1000
1000 times. Thetimes.
graphThe graph
shown in
shown in Figure 13 shows results from
Figure 13 shows results from both of these tests. both of these tests.

Figure 13.
Figure 13. SCADA
SCADA response time test
response time test results
results using
using local
local VPN
VPN and
and the
the 4G
4G mobile
mobile Internet.
Internet.

The vertical axis of the graph represents the time needed to receive a change that happened in
The vertical axis of the graph represents the time needed to receive a change that happened in the
the SCADA system. As can be observed in Figure 13, this period spans from 0 ms to 500 ms. The
SCADA system. As can be observed in Figure 13, this period spans from 0 ms to 500 ms. The horizontal
horizontal axis represents ordinal numbers of data changes in the current test. There are several
axis represents ordinal numbers of data changes in the current test. There are several graphs plotted on
graphs plotted on Figure 13; the blue graph represents the test over the local network and the orange
Figure 13; the blue graph represents the test over the local network and the orange over the 4G mobile
over the 4G mobile Internet. Median graphs represent linear and quadratic approximations of the
Internet. Median graphs represent linear and quadratic approximations of the two main ones and serve
two main ones and serve as a means to simplify and eliminate peaks which happen in the orange
as a means to simplify and eliminate peaks which happen in the orange and blue graphs.
and blue graphs.
As expected, using a local VPN is much faster than using the 4G mobile Internet and peaks
happen far less frequently than over the Internet. The worst response time using a local VPN was
169 ms. The best response time over a local VPN was approximately 34 ms and means that changes
Energies 2018, 11, 599 14 of 17

As expected, using a local VPN is much faster than using the 4G mobile Internet and peaks happen
Energies
far less2018, 11, x FORthan
frequently PEER over
REVIEW the Internet. The worst response time using a local VPN was 169 14 ofms.
17
The best response time over a local VPN was approximately 34 ms and means that changes will always
willreceived
be always be received
in the period in between
the period 34between
ms and 169 34 msmsand 169
after themschange
after the change physically
physically occurs.
occurs. Analysis
Analysis
of of theresponse
the Internet Internet response
times shows times shows
that that the
the worst worst response
response time 478 time
ms and 478themsbest
and one
the best
is 69 one
ms.
is 69 ms. Comparison of the two best-worst case scenario periods shows that
Comparison of the two best-worst case scenario periods shows that the 4G mobile Internet response the 4G mobile Internet
response
times can times
be more canthan
be more
twice than
as longtwice as long
as local as local
network timesnetwork
what istimes whatofisuncontrollable
the result the result of
uncontrollable
4G mobile Internet 4G mobile
network Internet network
availability. availability. Approximation
Approximation graphs comparison graphs comparison
shows that localshows
VPN
that local VPN response times are usually around 92 ms on average,
response times are usually around 92 ms on average, while 4G mobile Internet response times while 4G mobile Internet
are
response times 136
approximately are ms.
approximately 136 ms.
Figure 14
Figure 14 shows
shows thethe tests
tests histograms.
histograms. The The histogram shows test
histogram shows test times
times inin ms
ms onon its
its x-axis
x-axis and
and the
the
number of test queries on the y-axis. It is readily observable that using a local
number of test queries on the y-axis. It is readily observable that using a local VPN 652 out of 1000 VPN 652 out of 1000
test
test queries
queries werewere achieved
achieved in 50 inms50toms
100toms,100287
ms, 287 queries
queries took 100tookms100 ms ms
to 150 to 150
andms and
only 10only
took10 150took
ms
150 ms to 200 ms. On the 4G Internet network, just 52 queries took 50 ms to 100
to 200 ms. On the 4G Internet network, just 52 queries took 50 ms to 100 ms, 663 queries took 100 ms ms, 663 queries took
100150
to msms,
to 150
252ms,took252 took
160 ms160 ms ms,
to 200 to 20027 ms,
took27200
took
ms200 ms to
to 250 ms,250 ms, 1250
1 took tookms 250toms
300toms,
3001ms,
also1took
also
tookms
300 300toms350toms350andmsthe
andworst
the worst
3 took3 450
tookto450
500toms.
500 ms.

SCADA response time histogram test results using local VPN and the 4G mobile Internet.
Figure 14. SCADA

The last
The last graph
graph on on Figure
Figure 1515 shows
shows aa boxplot
boxplot ofof the
the same
same results.
results. On the x-axis,
On the x-axis, the
the time
time inin ms
ms is
is
plotted but
plotted but the
the vertical
vertical axis
axis does
does not
not serve
serve the
the same
same function
function as as on
on the
the other
other graphs.
graphs. The upper box
The upper box
shows quartiles of Internet response times and the lower one shows the same
shows quartiles of Internet response times and the lower one shows the same for a local VPN network. for a local VPN
network.
The medianTheismedian
shown is shown
with with avertical
a yellow yellowline.
vertical line.
It can nowIt can now be observed
be observed that the that the median
median of
of private
private network response time is 92 ms and the median of 4G mobile Internet
network response time is 92 ms and the median of 4G mobile Internet response time is 136 ms. Outlier response time is
136 ms. Outlier values are also clearly visible for both tests with maximum values
values are also clearly visible for both tests with maximum values corresponding to maximum values corresponding to
maximum
in Figure 14.values in Figure
The lower 14.ofThe
quartile lower
local VPNquartile
responseoftimelocalis VPN
70 ms,response
while thetime
upper is quartile
70 ms, while
is 101 the
ms.
upper quartile is 101 ms. The lower quartile 4G mobile Internet response time
The lower quartile 4G mobile Internet response time is 112 ms, while the upper quartile is 154 ms. is 112 ms, while the
upper
This quartile is means
information 154 ms. This
that 50%information
of responsemeans that a50%
times with localofVPN
response
network timesarewith a local70
in between VPN
ms
network are in between 70 ms and 101 ms and that 50% of response times with
and 101 ms and that 50% of response times with 4G mobile Internet is in between 112 ms and 154 ms. 4G mobile Internet is
in between 112 ms and 154 ms.
Energies 2018, 11, 599 15 of 17
Energies 2018, 11, x FOR PEER REVIEW 15 of 17

Figure
Figure 15.
15. SCADA
SCADAresponse
response time
time boxplot
boxplot test
test results
results using
using local
local VPN
VPN and
and the
the 4G
4G mobile
mobile Internet.
Internet.

6.
6. Conclusions
Conclusions
This
This paper
paperpresents
presentsa anovel
novelapproach
approach in in
visualizing
visualizingprocess related
process SCADA
related SCADA datadata
to enhance and
to enhance
facilitate supervision,
and facilitate control control
supervision, and human-centric maintenance
and human-centric activities inactivities
maintenance power system
in powersubstations.
system
The proposed solution utilizes AR visualization techniques together with standards-based
substations. The proposed solution utilizes AR visualization techniques together with standards-based
communication
communication defined
defined by
by IEC
IEC 61850.
61850. The
The developed
developed proof-of-concept
proof-of-concept AR AR prototype
prototype application
application
was
was tested
tested in
in KONČAR’s
KONČAR’s (Končar-Power
(Končar-Power Plant
Plant and
and Electric
ElectricTraction
Traction Engineering
EngineeringInc.,Inc., Fallerovo
Fallerovo22,
22,
10000
10000 Zagreb,
Zagreb, Croatia)
Croatia) in-house
in-house research
research laboratories
laboratories and
and aa real-world
real-world substation
substation environment.
environment. TheThe
application
application enables
enables visualization
visualization ofof automation
automation datadata anywhere
anywhere where
where itit is
is relevant
relevant by
by combining
combining
standards-based communication to fetch real-time SCADA information
standards-based communication to fetch real-time SCADA information and show it near its
and show it near its
corresponding equipment. It also enables using full 3D equipment models along with helpful
corresponding equipment. It also enables using full 3D equipment models along with helpful animated
animated widgets to quicken the interaction with the user. Maintenance engineers that participated
widgets to quicken the interaction with the user. Maintenance engineers that participated in application
in application testing were able to execute their common procedures significantly faster and
testing were able to execute their common procedures significantly faster and requiring less effort
requiring less effort confirming the usefulness of the applications. The solution is also tested
confirming the usefulness of the applications. The solution is also tested regarding the real-time SCADA
regarding the real-time SCADA response requirements. The mean response times for fetching data
response requirements. The mean response times for fetching data are 92 ms over a local Virtual Private
are 92 ms over a local Virtual Private Network (VPN) and 136 ms over the 4G mobile Internet.
Network (VPN) and 136 ms over the 4G mobile Internet. Future work will be based on developing a
Future work will be based on developing a software platform capable of making maintenance
software platform capable of making maintenance activities in substation environments more accessible
activities in substation environments more accessible by providing an adaptable workflow
by providing an adaptable workflow procedure to increase user satisfaction. Since the proposed
procedure to increase user satisfaction. Since the proposed solution is based on IEC 61850 standard,
solution is based on IEC 61850 standard, any domain that uses such communication, including
any domain that uses such communication, including centralized generation, wind power plants,
centralized generation, wind power plants, DERs or battery systems could be easily integrated into the
DERs or battery systems could be easily integrated into the future versions of the application.
future versions of the application.
Author
Author Contributions:
Contributions: The
The corresponding
corresponding author,
author, Miro
Miro Antonijević,
Antonijević, contributed
contributedto to all
all aspects
aspectsof
ofthe
theresearch,
research,
from literature overview,
from literature overview,totodomain
domain specific
specific research,
research, to software
to software application
application development
development and testing
and testing to
to article
creation.
article The author
creation. The Stjepan
author Sučić
Stjepancontributed to domaintospecific
Sučić contributed domainresearch
specificsuch as the such
research IEC 61850 standard
as the and
IEC 61850
article creation.
standard The author
and article Hrvoje
creation. TheKeserica
author contributed to the contributed
Hrvoje Keserica developmenttoand thetesting of software
development andapplications.
testing of
software
Conflictsapplications.
of Interest: The authors declare no conflict of interest.
Conflicts of Interest: The authors declare no conflict of interest.
References
References
1. Glass. Available online: https://x.company/glass/ (accessed on 17 February 2018).
2.
1. Microsoft
Glass. HoloLens.
Available online:Available online: https://www.microsoft.com/en-us/hololens
https://x.company/glass/ (accessed on 17 February 2018). (accessed on 20
2. November 2017).
Microsoft HoloLens. Available online: https://www.microsoft.com/en-us/hololens (accessed on 20
November 2017).
3. Tamaazousti, M.; Naudet-Collette, S.; Gay-Bellile, V.; Bourgeois, S.; Besbes, B.; Dhome, M. The constrained
SLAM framework for non-instrumented augmented reality. Multimed. Tools Appl. 2016, 75, 9511–9547.
Energies 2018, 11, 599 16 of 17

3. Tamaazousti, M.; Naudet-Collette, S.; Gay-Bellile, V.; Bourgeois, S.; Besbes, B.; Dhome, M. The constrained
SLAM framework for non-instrumented augmented reality. Multimed. Tools Appl. 2016, 75, 9511–9547.
[CrossRef]
4. Kim, S.; Dey, A.K. Augmenting human senses to improve the user experience in cars: Applying augmented
reality and haptics approaches to reduce cognitive distances. Multimed. Tools Appl. 2016, 75, 9587–9607.
[CrossRef]
5. Sebillo, M.; Vitiello, G.; Paolino, L.; Ginige, A. Training emergency responders through augmented reality
mobile interfaces. Multimed. Tools Appl. 2016, 75, 9609–9622. [CrossRef]
6. Costanza, E.; Kunz, A.; Fjeld, M. Mixed Reality: A Survey. In Human Machine Interaction: Research Results of
the MMI Program; Lalanne, D., Kohlas, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2009; pp. 47–68.
7. Tang, A.; Owen, C.; Biocca, F.; Mou, W. Comparative effectiveness of augmented reality in object assembly.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL,
USA, 5–10 April 2003; pp. 73–80.
8. Ferrarini, L.; Dedè, A. A mixed-reality approach to test automation function for manufacturing systems.
IFAC Proc. Vol. 2009, 42, 133–138. [CrossRef]
9. Soete, N.; Claeys, A.; Hoedt, S.; Mahy, B.; Cottyn, J. Towards Mixed Reality in SCADA Applications.
IFAC-Pap. 2015, 48, 2417–2422. [CrossRef]
10. Jin, T.; Shen, X. A Mixed WLS Power System State Estimation Method Integrating a Wide-Area Measurement
System and SCADA Technology. Energies 2018, 11, 408. [CrossRef]
11. Suh, J.; Hwang, S.; Jang, G. Development of a Transmission and Distribution Integrated Monitoring and
Analysis System for High Distributed Generation Penetration. Energies 2017, 10, 1282. [CrossRef]
12. Godina, R.; Rodrigues, E.M.G.; Matias, J.C.O.; Catalão, J.P.S. Effect of Loads and Other Key Factors on
Oil-Transformer Ageing: Sustainability Benefits and Challenges. Energies 2015, 8, 12147–12186. [CrossRef]
13. Zhu, Z.; Xu, B.; Brunner, C.; Yip, T.; Chen, Y. IEC 61850 Configuration Solution to Distributed Intelligence in
Distribution Grid Automation. Energies 2017, 10, 528. [CrossRef]
14. Ong, S.K.; Nee, A.Y.C. Virtual and Augmented Reality Applications in Manufacturing; Springer Science &
Business Media: Berlin, Germany, 2013.
15. Suzuki, L.R.; Brown, K.; Pipes, S.; Ibbotson, J. Smart building management through augmented reality.
In Proceedings of the 2014 IEEE International Conference on Pervasive Computing and Communications
Workshops (PERCOM Workshops), Budapest, Hungary, 24–28 March 2014; pp. 105–110.
16. Regenbrecht, H.; Baratoff, G.; Wilke, W. Augmented reality projects in the automotive and aerospace
industries. IEEE Comput. Graph. Appl. 2005, 25, 48–56. [CrossRef] [PubMed]
17. Ribeiro, T.R.; dos Reis, P.R.J.; Júnior, G.B.; de Paiva, A.C. Agito: Virtual reality environment for power
systems substations operators training. In Proceedings of the International Conference on Augmented and
Virtual Reality, Lecce, Italy, 17–20 September 2014; pp. 113–123.
18. dos Reis, P.R.J.; Junior, D.L.G.; de Araújo, A.S.; Júnior, G.B.; Silva, A.C.; de Paiva, A.C. Visualization of Power
Systems Based on Panoramic Augmented Environments. In Proceedings of the International Conference on
Augmented and Virtual Reality, Lecce, Italy, 17–20 September 2014; pp. 175–184.
19. IEC. Communication Networks and Systems in Substations—ALL PARTS; Int. Std. IEC 61850-SER ed1.0; IEC:
Geneva, Switzerland, 2011.
20. IEC. Communication Networks and Systems for Power Utility Automation—Part 7-4: Basic Communication
Structure—Compatible Logical Node Classes and Data Object Classes; Int. Std. IEC 61850-7-4, ed2.0; IEC: Geneva,
Switzerland, 2010.
21. IEC. Wind Turbines—Part 25-2: Communications for Monitoring and Control of Wind Power Plants—Information
Models; Int. Std. IEC 61400-25-2 ed1.0; IEC: Geneva, Switzerland, 2006.
22. IEC. Communication Networks and Systems for Power Utility Automation—Part 7-420: Basic Communication
Structure—Distributed Energy Resources Logical Nodes; Int. Std. IEC 61850-7-420 ed1.0; IEC: Geneva,
Switzerland, 2009.
23. IEC. Communication Networks and Systems for Power Utility Automation—Part 7-410: Hydroelectric Power
Plants—Communication for Monitoring and Control; Int. Std. IEC 61850-7-410 ed1.0; IEC: Geneva,
Switzerland, 2007.
Energies 2018, 11, 599 17 of 17

24. IEC. Communication Networks and Systems for Power Utility Automation—Part 7-2: Basic Information and
Communication Structure—Abstract Communication Service Interface (ACSI); Int. Std. IEC 61850-7-2 ed2.0; IEC:
Geneva, Switzerland, 2010.
25. IEC. Communication Networks and Systems in Substations—Part 8-1: Specific Communication Service Mapping
(SCSM)—Mappings to MMS (ISO 9506-1 and ISO 9506-2) and to ISO/IEC 8802-3; IEC Std. IEC 61850-8-1 ed1.0;
IEC: Geneva, Switzerland, 2004.
26. Das, N.; Ma, W.; Islam, S. Comparison study of various factors affecting end-to-end delay in IEC 61850
substation communications using OPNET. In Proceedings of the 2012 22nd Australasian Universities Power
Engineering Conference (AUPEC), Bali, Indonesia, 26–29 September 2012; pp. 1–5.
27. Kumar, S.; Das, N.; Islam, S. Performance analysis of substation automation systems architecture based on
IEC 61850. In Proceedings of the 2014 Australasian Universities Power Engineering Conference (AUPEC),
Perth, WA, Australia, 28 September–1 October 2014; pp. 1–6.
28. Das, N.; Islam, S. Analysis of Power System Communication Architectures between Substations Using IEC
61850. In Proceedings of the 5th Brunei International Conference on Engineering and Technology (BICET
2014), Bandar Seri Begawan, Brunei, 1–3 November 2014.
29. IEC. Communication Networks and Systems for Power Utility Automation—Part 6: Configuration Description
Language for Communication in Electrical Substations Related to IEDs; Int. Std. 61850-6, ed2.0; IEC: Geneva,
Switzerland, 2009.
30. Toni, B.; Kalafatic, Z.; Mihajlovic, Z. Application of Augmented Reality for Supporting Instrument Service
Tasks. In Proceedings of the ACM SIGGRAPH VRCAI 2012, The 11th International Conference on Virtual
Reality Continuum and Its Applications in Industry, Singapore, 2–4 December 2012.
31. Oculus Rift. Available online: https://www.oculus.com/ (accessed on 20 November 2017).
32. Open Source Augmented Reality SDK ARToolKit.org. Available online: https://archive.artoolkit.org/
(accessed on 20 November 2017).
33. Vuforia Augmented Reality. Available online: https://www.vuforia.com/ (accessed on 20 November 2017).
34. Olwal, A.; Gustafsson, J.; Lindfors, C. Spatial augmented reality on industrial CNC-machines. In Proceedings
of the SPIE, San Jose, CA, USA, 27–31 January 2008; Volume 6804, p. 680409.
35. Soon, T.J. QR code. Synth. J. 2008, 2008, 59–78.
36. VuMark. Available online: https://library.vuforia.com/articles/Training/VuMark (accessed on
20 November 2017).
37. Ruan, K.; Jeong, H. An augmented reality system using Qr code as marker in android smartphone.
In Proceedings of the 2012 Spring Congress on Engineering and Technology (S-CET), Xi’an, China,
27–30 May 2012; pp. 1–3.
38. QRCode: QR Code Image Generator. Available online: https://github.com/lincolnloop/python-qrcode
(accessed on 7 March 2018).
39. Adobe Illustrator CC Vector Graphic Design Software. Available online: http://www.adobe.com/products/
illustrator.html (accessed on 20 November 2017).
40. Unity—Game Engine. Available online: https://unity3d.com (accessed on 20 November 2017).

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).