Working Group
D2.35
May 2015
SCALABLE COMMUNICATION
TRANSPORT SOLUTIONS
OVER OPTICAL NETWORKS
Type WG D2.35
Members
J.SANDHAM Convenor (IE), M.SCHATZ Secretary (CH), M.ACACIA (BE), F.CASTRO (ES),
J.DARNE (ES), J.FEIJOO MARTINEZ (ES), M.FLOHIL (NL), R.IRONS-MCLEAN (UK),
M.JANSSEN (NL), A.MOAINI (FR), T.V.PEDERSEN (NO), H.RIIS (DK), A.RUNESSON (SE),
S.TANNER (FI)
Corresponding Members
C.DI PALMA (AR), C.EVERITT (AU), J.MENDES (PT), P.SCHWYTER (CH), M.SEEWALD (DE),
A.SILFVERBERG (FI), V.TAN (AU)
Copyright 2015
Ownership of a CIGRE publication, whether in paper form or on electronic support only infers
right of use for personal purposes. Other uses are prohibited, except if explicitly agreed by
CIGRE, total or partial reproduction of the publication for use other than personal and transfer
to a third party; hence circulation on any intranet or other company network is forbidden.
Disclaimer notice
CIGRE gives no warranty or assurance about the contents of this publication, nor does it
accept any responsibility, as to the accuracy or exhaustiveness of the information. All implied
warranties and conditions are excluded to the maximum extent permitted by law.
ISBN : 978-2-85873-320-0
1|Page
SCALABLE COMMUNICATION
TRANSPORT SOLUTIONS OVER
OPTICAL NETWORKS
Table of Contents
1 INTRODUCTION................................................................................................................ 8
2 REQUIREMENTS FROM USERS .......................................................................................... 9
2.1 Survey Results .................................................................................................................................... 9
2.1.1 Substation applications using IP ............................................................................................ 11
2.1.2 Main operational challenges using IP as communication transport solutions ................ 12
2.1.3 Main psychological barriers with using IP protocols in Substation applications .......... 13
2.1.4 Are all applications compatible with IP at the moment? .................................................. 14
2.1.5 Prediction to migration of all operational communications into IP ................................. 14
2.1.6 How to deal with legacy protocols and equipment? ........................................................ 15
2.1.7 Requisites and concerns for the telecommunication network ........................................... 16
2.1.8 Most promising technologies to provide communications in the access network? ........ 17
2.1.9 Most promising technologies to provide communications in the core network?............ 18
2.1.10 Should the IP network be for operational service or also for corporate services? . 19
2.1.11 Survey Conclusions ................................................................................................................ 19
2.2 List of Services ................................................................................................................................ 20
3 GENERAL CONSIDERATIONS ......................................................................................... 22
3.1 Why are Utilities migrating? ........................................................................................................ 22
3.2 Migration considerations............................................................................................................... 24
3.2.1 Timing ......................................................................................................................................... 26
3.2.4 Communication Requirements Decomposition ..................................................................... 26
3.2.5 Security ...................................................................................................................................... 27
3.2.6 Communication Network Management ................................................................................ 28
3.2.7 IPv4 and IPv6 co-existence .................................................................................................... 29
3.2.8 Segmentation and Virtualization .......................................................................................... 29
3.2.9 Scalability ................................................................................................................................. 30
3.2.10 Availability ............................................................................................................................. 30
4 ASSESMENT OF TECHNOLOGIES .................................................................................... 31
4.1 Synchronous Digital Hierarchy (SDH) ......................................................................................... 32
Life Cycle ............................................................................................................................................. 32
Scalability ............................................................................................................................................ 32
Capability to support EPU Services ................................................................................................ 33
Ease of Implementation and Operation ......................................................................................... 33
Interoperability (other services/technologies)............................................................................... 33
Redundancy, Availability and Reliability ...................................................................................... 34
Quality of Service (QoS)................................................................................................................... 35
Network Management System and Overall Manageability (NMS) ......................................... 35
3|Page
4.2 Dense Wavelength Division Multiplexing & Coarse Wavelength Division Multiplexing . 36
Life Cycle ............................................................................................................................................. 36
Scalability ............................................................................................................................................ 36
Capability to support EPU Services ................................................................................................ 39
Ease of Implementation and Operation ......................................................................................... 39
Interoperability (other services/technologies)............................................................................... 40
Redundancy, Availability and Reliability ...................................................................................... 41
Quality of Service (QoS)................................................................................................................... 41
Network Management System and Overall Manageability (NMS) ......................................... 42
CWDM versus DWDM ....................................................................................................................... 42
4.3 Optical Transport Network (OTN) .............................................................................................. 43
Life Cycle ............................................................................................................................................. 43
Scalability ............................................................................................................................................ 43
Capability to support EPU Services ................................................................................................ 45
Ease of Implementation and Operation ......................................................................................... 46
Interoperability (other services/technologies)............................................................................... 46
Redundancy, Availability and Reliability ...................................................................................... 46
Quality of Service (QoS)................................................................................................................... 46
Network Management System and Overall Manageability (NMS) ......................................... 46
4.4 Dynamic Internet protocol based Multiprotocol Label Switching (IP/MPLS) ..................... 47
Life Cycle ............................................................................................................................................. 47
Scalability ............................................................................................................................................ 47
Capability to support EPU Services (as defined in chapter 2) .................................................. 47
Ease of Implementation and Operation ......................................................................................... 48
Interoperability (other services/technologies)............................................................................... 48
Redundancy, Availability and Reliability ...................................................................................... 48
Quality of Service (QoS)................................................................................................................... 49
Network Management System and Overall Manageability (NMS) ......................................... 49
4.5 Static Multiprotocol Label Switching Transport Profile (MPLS-TP) .................................... 50
Life Cycle ............................................................................................................................................. 50
Scalability ............................................................................................................................................ 50
Capability to support EPU Services (as defined in chapter 2) .................................................. 50
Ease of Implementation and Operation ......................................................................................... 51
Interoperability (other services/technologies)............................................................................... 51
Redundancy, Availability and Reliability ...................................................................................... 51
Quality of Service (QoS)................................................................................................................... 52
Network Management System and Overall Manageability (NMS) ......................................... 52
4.7 Provider Backbone Bridging (PBB) ............................................................................................. 54
4.8 Ethernet ............................................................................................................................................ 54
Life Cycle ............................................................................................................................................. 54
Scalability ............................................................................................................................................ 55
Capability to support EPU Services (as defined in chapter 2) .................................................. 55
Ease of Implementation and Operation ......................................................................................... 55
Interoperability (other services/technologies)............................................................................... 55
Redundancy, Availability and Reliability ...................................................................................... 55
Quality of Service (QoS)................................................................................................................... 55
Network Management System and Overall Manageability (NMS) ......................................... 55
4.9 Comparison matrix......................................................................................................................... 56
5 CONCLUSIONS AND PROPOSAL FOR FUTURE WORK .................................................. 58
4|Page
Survey ...................................................................................................................................................... 58
Services.................................................................................................................................................... 58
Technologies ........................................................................................................................................... 58
Future Works 1 ...................................................................................................................................... 60
Future Works 2 ...................................................................................................................................... 60
Future Works 3 ...................................................................................................................................... 61
ANNEX 1 SURVEY D2.35 ................................................................................................... 62
ANNEX 2 SURVEY D2.28 IP-BASED SUBSTATION APPLICATIONS.................................... 71
ANNEX 3 WGD2.35 SURVEY RESULTS............................................................................... 74
1 Substation applications using IP ...................................................................................................... 74
2 Substation applications eligible for use with IP ........................................................................... 76
3 Applications outside the substation eligible for use with IP ....................................................... 78
4 Main operational challenges using IP a communication transport solutions ........................... 78
5 Main psychological barriers with using IP protocols in Substation applications .................... 79
6 Are all applications compatible with IP at the moment? ............................................................ 80
7 Prediction to migration of all operational communications into IP ........................................... 81
8 Percentage of the existing applications using IP as communication transport solution? ...... 83
9 How to deal with legacy protocols and equipment? .................................................................. 83
10 Requisites and concerns for the telecommunication network................................................... 84
11 Most promising technologies are to provide communications in the access network? ........ 86
12 Most promising technologies to provide communications in the core network? ................... 86
13 Should the IP network be for operational service or also be for corporate services? ...... 87
14 What type of scalability needs to be addressed? .................................................................. 88
ANNEX 4 NETWORK TIMING ............................................................................................ 90
Synchronisation Types........................................................................................................................... 91
IRIG-B Overview.................................................................................................................................... 92
Global Positioning System (GPS) Overview .................................................................................... 92
One Pulse Per Second (1PPS) Overview .......................................................................................... 94
NTP/SNTP Overview ............................................................................................................................ 94
IEEE 1588 PTP / IEC 61588 PTP ........................................................................................................ 94
Versions of 1588 ................................................................................................................................ 95
NTP vs. SNTP vs. PTP: A Summary...................................................................................................... 95
5|Page
Figures
Figure 1 Companies represented in the survey responses ............................................................................. 10
Figure 2 Countries represented in the survey responses ................................................................................ 10
Figure 3 Trends in substation applications using IP ........................................................................................ 11
Figure 4 Trends in operational challenges ...................................................................................................... 12
Figure 5 Trends in psychological barriers ........................................................................................................ 13
Figure 6 Trends in current IP compatibility ...................................................................................................... 14
Figure 7 Trends in estimation of all operational traffic through IP ................................................................... 15
Figure 8 Trends in how to deal with legacy equipment and protocols ............................................................ 15
Figure 9 Trends in requisites and concerns .................................................................................................... 16
Figure 10 Trends in most promising access network technologies ................................................................. 17
Figure 11 Trends in most promising core network technologies ..................................................................... 18
Figure 12 Trends in network separation .......................................................................................................... 19
Figure 13 Traditional Grid Communications Architecture ................................................................................ 23
Figure 14 Tiered approach for New grid applications ...................................................................................... 23
Figure 15 Shift from TDM to packet ................................................................................................................. 24
Figure 16 Migration decision review process .................................................................................................. 25
Figure 17 Communications decomposition table for packet oriented traffic .................................................... 26
Figure 19 SDH Capacity Comparison ............................................................................................................. 32
Figure 18 SDH Add-Drop-Multiplexer .............................................................................................................. 33
Figure 20 SDH Sub-Network Connection Protection ...................................................................................... 34
Figure 22 Optical spectra in fiber ..................................................................................................................... 37
Figure 23 examples of Add-Drop-Multiplexer .................................................................................................. 37
Figure 24 Reconfigurable Optical Add Drop Multiplexer ................................................................................. 38
Figure 25 Working area for light source and receiver ..................................................................................... 38
Figure 21 WDM Multiplexing............................................................................................................................ 39
Figure 26 OTN Architecture ............................................................................................................................. 44
Figure 27 OTN Capacities ............................................................................................................................... 45
Figure 28 MPLS-TP Hierarchy ........................................................................................................................ 51
Figure 29 Protection configuration ................................................................................................................... 52
Figure 30: Substation applications using IP .................................................................................................... 75
Figure 31: Applications eligible for using IP..................................................................................................... 77
Figure 32: Operational challenges ................................................................................................................... 79
Figure 33: Psychological Barriers .................................................................................................................... 80
Figure 34: All applications IP compatible?....................................................................................................... 81
Figure 35: Predication of All operational traffic through IP .............................................................................. 82
Figure 36 Percentage of existing applications using IP ................................................................................... 83
Figure 37How to deal with legacy protocols/equipment .................................................................................. 84
Figure 38 Requisites and concerns for using only IP ...................................................................................... 85
Figure 39 Most promising access network technology .................................................................................... 86
Figure 40 Most promising core network technology ........................................................................................ 87
6|Page
Tables
Table 1: (Future) IP usage ............................................................................................................................... 78
7|Page
1 INTRODUCTION
The introduction of smart applications in the Electrical Power Utility (EPU) and consequent
dispersed intelligence result in a tremendous growth of information exchange across the power
system. This implies, in many cases, a change of scale in the requirements of the
telecommunication infrastructure and often the deployment of a core data transport network.
This may be implemented through a number of different technologies and architectures.
The present network of most power utilities is extensively composed of TDM (e.g. PDH/SDH)
technology. Packet communication and in particular Ethernet connections are growing very fast
and may bring the necessity to adapt and /or replace network technologies.
This Technical Brochure aims to identify and analyze solutions and migration plans in the light of
data network technology evolutions, new application requirements and EPUs capability to
maintain the systems operation.
During 2009 and 2010, Cigr working group D2.28 conducted a survey amongst Cigr members
to identify the current and expected future use of IP networks within Electric Power Utilities. For
this Technical Brochure a follow-up survey was conducted to identify trends in the use of networks
and network technologies
This Technical Brochure contains:
The results of the follow-up survey held for this Technical Brochure
An analysis of trends in the use of networks and network technologies based on a
comparison of the survey held for working group D2.28 and the follow-up survey held
for this Technical Brochure
General considerations for replacing, refurbishing or extending networks
An assessment of available physical technologies and transport protocols
Recommendations for future works
8|Page
The survey that was held for working group D2.28, while similar, is not exactly the same as the
survey held for this Technical Brochure. For a detailed comparison both surveys can be found as
an Annex to this TB.
The survey for D2.35 was held under all Cigr members, not just the participants in the D2.28
survey. This means that there may be differences between the results of the two different surveys
due to differing companies participating in the surveys.
Only questions that were similar or exactly the same between the two surveys are compared below.
In total, 83 filled out surveys were received from 29 different countries. Figure 1 shows which companies
from which countries have responded.
ANSWERS FROM:
TRANELSA
ENERGINET.DK
EGAT
ITAIPU BINACIONAL
Scottish Power
REN
SEAS-NVE
AXPO AG
HUAWEI
INEXUS
ABB Switzerland
AUSGRID
POWERLINK
ESSENTIAL ENERGY
ETSA Utilities
TRANSEND
ELECTRANET
AURORA ENERGY
COMMTEL
ENDAVOR
ENERGEX
ERGON
FRESNEL NETWORKS
SNOWY HYDRO
SP AUSNET
TRANSGRID
TRANSPOWER NZ
GDH
WESTERN POWER
POWER COR
ELECTROBRAS
Power Grid Corperation of Bangladesh
Bhutan Power Corporation Limited
Sikkim power
GETCO
MSETCL
PTCUL
TSECL
Nokia Siemens Networks
Nokia Siemens Networks (2)
ESKOM
DNV KEMA
ANSWERS FROM:
Argentina
Denmark
Thailand
Brazil
United Kingdom
Portugal
Denmark
Switzerland
Belgium
United Kingdom
Switzerland
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
New Zealand
Australia
Australia
Australia
Brazil
Bangladesh
India
India
India
India
India
India
Finland
Finland
South Africa
The Netherlands
IBERDROLA
FINGRID
ISREAL ELECTRIC CORP
RUGGEDCOM
STATKRAFT
STATNETT
SO-UPS
ELECTRIC RESEARCH INSTITUTE
NARI RELAYS
RTE
AGDER ENERGI
EGAT
ZTE
National Grid Saudi Aribia
HUAWEI
VERBUND
Elektroprenos BiH
Elektroprivreda BiH
Hidroelektrane na Trebisnjici
ABB CHINA(1)
ABB CHINA(2)
ABB CHINA(3)
CFE Transmission Yucatan Peninsula
CFE Transmnission north
CFE Transmission north-east
CFE Transmission central
Tata Power Co. Ltd
EEPCO
SaskPower
Altalink Management
Unknown
PSE Operator
Hokkaido Electric Power Company
Hokuriku Electric Power Company
Tokyo Electric Power Company
Chubu Electric Power Company
Chugoku Electric Power Company
Shikoku Electric Power Company
Kyushu Electric Power Company
NEC Corporation
Liandon
Spain
Finland
Israel
Germany
Norway
Norway
Russia
Mexico
China
France
Norway
Thailand
Thailand
Suadi Arabia
Thailand
Switzerland
Bosnia and Herzegovina
Bosnia and Herzegovina
Bosnia and Herzegovina
China
China
China
Mexico
Mexico
Mexico
Mexico
India
Ethiopia
Canada
Canada
China
Poland
Japan
Japan
Japan
Japan
Japan
Japan
Japan
Japan
The Netherlands
9|Page
10 | P a g e
83%
81%
2012
71%
67%
59%
2010
66%
64%
61%
64%
57%
64%
57%
52%
49% 49% 51%
56%
49%
44%
34%
24%
34%
34%
31%
28%
22%
12%
19%
12%
7%
Building control
Telephony system
Substation RTU to SCADA system
While this could mean that, since 2010, IP communication for these applications has been replaced by
non-IP communication, its more likely that the differences in these results come from the difference in
respondents.
11 | P a g e
Operational challenges:
86%
82%
2010
66%
2012
59%
58%
58% 57%
46%
41%
41%
39%
34%
27%
23%
Ruggedness
Scope of responsibility
Compared to 2010, more participants indicated that they see Quality of Service (QoS) as an
operational challenge.
12 | P a g e
Psychological barriers:
85%
77%
2010
2012
66%
57%
53%
47%
48%
32%
29%
23%
10% 11%
13 | P a g e
2010
79%
21%
2012
76%
18%
Yes
No
2010
60%
2012
36% 36%
25%
13%
8%
5%
2%
2%
0%
11%
2%
14 | P a g e
55%
43%
41%
40%
17%
12%
17%
14%
17%
10%
15 | P a g e
69%
66%
63%
54%
34%
39%
30%
23%
Existence of
Suitable
Suitable IP
Identification of
physical
transport
addressing space Applications
infrastructure technology (PDH, and scheme (IPv4
(fibre, copper, SDH, Ethernet,
/ IPv6)
wireless, etc)
etc.)
16 | P a g e
2.1.8 Most promising technologies to provide secure and reliable transport communications
in the access network?
14%
13%
17 | P a g e
2.1.9 Most promising technologies to provide secure and reliable transport com munications
in the core network?
51%
45%
39%
35%
14%
18 | P a g e
2.1.10 Should the IP network be reserved for operational service or also be used for
corporate services?
2010
2012
58%
54%
48%
47%
5%
2%
One network,
no separation
Separate
network
(physical)
Separate
network
(virtual/logical)
19 | P a g e
Teleprotection
Outage Management Systems
Network and Access management
SCADA services
Energy Management Systems
Energy metering
Event recorders
Switchover of services between control centers
Disturbance recorders
Real-time PMU
Polling telemetry operational communications and Black Start telephony
Operational Voice Services
Maintenance & Support
Private Mobile Radio
General site alarms, supervision and surveillance
Time distribution using IEEE 1588 alternative to GPS.
Video services
Physical site security using access control mechanisms.
Smart Metering communications.
IEC 61850 based communications.
Dynamic Line Rating
Weather Monitoring & Lightening Detection
Corporate IP network.
Corporate fixed telephony
Corporate mobile telephony
Corporate voicemail
Corporate Video Conferencing
In a lot of existing deployments, where adequate spare capacity exists and economically advantageous
to do so, non-operational traffic is accommodated on operational networks. Non-operational traffic is
usually a secondary consideration and in general is not a determinant of network build capacity.
20 | P a g e
Many documents from various organizations (e.g. CIGRE, IEC) already include detailed descriptions of
the communication / performance requirements for individual applications. It is not the purpose of this
document to re-write or repeat these requirements. The following is a (non-exhaustive) list of documents
that include such information:
Deriving a single mechanism to accurately classify applications based on the information above is quite
difficult. For this reason the Technical Brochure categorises differing applications into groups which can
be hereafter referred to as traffic types. Traffic types can be classified into 6 areas
Samples of possible classifications are listed below:
1. Very low latency, loss intolerant, sequence and symmetry determinant traffic:
over the same path
a. Differential teleprotection schemes
2. Very low latency, loss intolerant
a. Distance Protection
3. Very low latency, loss tolerant traffic:
a. Time distribution
b. Real-time PMU
c. Inter substation event distribution
4. Low latency sequence determinant traffic:
a. RS485
b. IEC 61870-101 IEC 61870-103
c. DNP3
5. Low latency, loss tolerant traffic
a. Voice
b. Video
c. WAMPAC
6. Latency tolerant, loss tolerant
a. Packet based SCADA protocols
b. File transfer
c. Device management
21 | P a g e
3 GENERAL CONSIDERATIONS
This section provides an overview of the requirements utilities need to consider as part of a WAN
migration strategy, and highlights key services and their performance requirements. Further details of the
technologies and services are explained in Chapter 4.
22 | P a g e
Figure 15 shows a high level overview of the state of migration. Traditionally most WANs were based on TDM
technologies. Over recent years packet technologies have emerged supporting some operational and IT
applications. These have either been a combination of packet over TDM, or TDM over packet. There is a
gradual shift in the industry towards a hybrid or all packet network architecture.
Why are we seeing this transition? The reality is that the communications network is evolving due to changing
communications requirements such as precision timing and IEC 61850, increased network functionality including
enhanced QoS features and multicast support, and the addition of multiservice capabilities like workforce
enablement and physical access solutions.
New technologies require new skillsets and engineering approaches. (one of the operational challenges
according to the survey results). Is the required skill set and mind set available at utilities for a
transition?
How does the organizational structure (with OT and IT responsibilities) influence a migration process?
Substation equipment refresh: What is the time line for refreshing substation equipment to a new
technology? Is this process in progress? Does existing equipment have physical interfaces available to
connect using a new technology such as Ethernet/IP?
Control Centre capability: Does the EPU management system/s support the new technology choice?
What is the time line for this support? Does this require a rebuild and what is the cost for the upgrade?
Is an upgrade of existing substation equipment possible, or does it need to be replaced? Cost of
substation equipment replacement/upgrade: If no refresh is taking place what is the cost of upgrading
equipment to enable adequate interfaces? Can this be justified and is it giving additional functionality?
What is the disruptive risk to the overall electrical network and overall electrical supply due to this
technology change?
24 | P a g e
Additional cost to integrate legacy applications which cannot be converted to new technology: What is
the additional cost to include TDM and traditional SCADA interfaces into the a newer network? Will this
include legacy interface support, tunneling and translation of traditional SCADA traffic? Can this cost be
justified as the additional capabilities may not be required in 5-10 years?
Cost to maintain existing TDM based infrastructure: What is the associated cost to maintain the existing
TDM/SDH infrastructure to allow the migration of the legacy devices? Is this infrastructure still
supported? What are the security and availability risks in maintaining this Infrastructure?
What is the cost of not migrating? What applications and use cases cannot be deployed? What is the
cost of supporting two or more infrastructures?
In addition there are a number of network influencing factors. The TDM infrastructure, the business and functional
requirements, existing and future communication requirements, and existing infrastructure. Reviewing the
questions and influencing factors following a process such as that outlined in Figure 16, and taking into
consideration the guidelines already documented in the Technical Brochure from D2.28, the outcome will be the
required communications infrastructure.
25 | P a g e
An architectural approach, consistent with industry and standards organisations (e.g. NIST, IEC, EPRI, CENELEC),
networking companies, and also power vendors is essential to understand the interdependencies and
communications between systems that exist, but equally important to provide a seamless platform for future use
case integration. An architectural process defines a set of artifacts required to describe a system so that it can
be reproduced and maintained over its lifespan. These artifacts provide components, structure,
interdependencies, and the guidelines determining the design and evolution over time. This is important as each
EPU may have subtly or very different requirements and WAN design will only reflect the use cases and
services an EPU wishes to deploy. All of this should be captured using relevant architecture frameworks such as
TOGAF or equivalent.
3.2.1 Timing
Substation automation and system control are mission-critical processes and electric power utilities must
synchronize applications and use cases across large-scale distributed power grid substation networks to ensure
grid stability. Precise timing is used to improve reliability, better understand operations of the power system,
predict and prevent local and system-wide faults, for testing and verification, and to reduce costs.
Refer to ANNEX 4 Network Timing for more details.
26 | P a g e
3.2.5 Security
Electrical power grids continue to evolve to meet the requirements of standardization efforts and grid
modernization in order to enhance existing use cases [1] and applications, while simultaneously enabling
new use cases driven by the concept of the smart grid. In addition there has been a surge in the number
of non-power applications introduced across the grid, as well as links between networks inside an EPU,
and links to external networks. The addition of new technologies allows an EPU to realize a more stable,
reliable, efficient and visible power grid capable of handling any type of communication flow.
Through the integration of new systems, applications, and technologies to facilitate new use cases across
network infrastructure (sometimes common), an EPUs network becomes multi-service. However with the
introduction of any new technology there is an increased risk of security threats which must also be taken
into consideration.
The highly connected nature of a smart grid has the potential to provide unauthorized users with a
greater opportunity to identify and exploit vulnerabilities to attack the electricity grid. There are many
examples of attacks on networks worldwide and constant vigilance is necessary to ensure that adequate
protective mechanisms are used to ensure the electricity grid is not compromised.
While designing and implementing operational networks it will be necessary to comprehensively address
cyber security issues. A defense in depth (security measures deployed at various physical and
conceptual layers of a deployment) approach should be adopted throughout the deployment. Security
capabilities must therefore be layered such that defense mechanisms have multiple points to detect and
mitigate breaches for all service types that an EPU wishes to deploy. These capabilities should be
integral to all segments of the grid infrastructure and address the full set of logical functional
requirements, including:
Physical security
Identity and access control policies
Hardened network devices and systems
Threat defense
Data protection for transmission, distribution and storage
Real-time monitoring, management, and correlation
Secureness of a technology. All technologies outlined within this document possess specific attributes which
at a basic level reduce or introduce additional risks to cyber security, information separation, data
privacy and data authenticity. All technologies should be assessed based upon these criterion to
evaluate suitability and to highlight any additional activities required to ensure a secure introduction of
the technology.
Cyber Security encompasses the overall security landscape from a physical level to a user and
electrical/electronic level.
Information separation is the ability for a technology to transport a specific data stream in a fashion
which can be deemed isolated from dependencies of other data streams.
Other security considerations:.
Access Control: Authentication and authorization of all personnel management tools and physical
devices
Authentication, Authorization and Accounting for data and devices
Data Confidentiality and Privacy
Securely scalable
Tools: Zones, Segmentation, Logging
27 | P a g e
Network and security management and power grid management applications must be able to
communicate over TDM, IPv4 and IPv6 network layers over the expected lifetime of the
deployment.
Role Based Access Control for administrators and operators must be able to get users
authentication from corporate directory services, thus showing interaction between IT and OT
networks.
Network and security management and power grid management solutions applicable to use
cases requiring large scalability, must be able to work in a clustering and load-balancer
networking environment. It allows growing the solution over time without reconsidering the
architecture.
Data flow and API (Application Programing Interface) must be identified, specified and
documented for communications between OT and NMS applications.
An unified Geographic Information System (GIS) solution should be shared between applications
Recommended database (DB) properties should be defined when storing all operational state,
device configuration, network event alarm, performance metric, etc.
Recommended Historian properties should be defined to store data on a long period of time as
required by regulations and operations.
Standard protocols, including public vendors extensions, should be recommended to facilitate
management data exchanges between a device and multiple management applications.
Remote Operations Management capabilities should provide utilities asset management functions
with real-time communications, improved availability of information, and enhanced visual
displays, all of which enable the asset planners and field workers to perform their work with
improved efficiency
Benefits
A powerful Network Management System enables both condition-based and predictive maintenance
strategies so the EPU does not have to rely upon reacting to failures after they happen and scheduling
inspections, which normally have to take place even if there are no equipment health issues.
In addition, asset data can be used as input for grid state determination and remote power quality
monitoring.
These benefits also contribute to improved System Reliability
28 | P a g e
Increase Service Restoration Speed: Improvements in the speed and availability of information in
the field allows technicians to diagnose and repair assets more quickly, resulting in a more rapid
restoration of service.
Increase Reliability Metrics: Increases in the speed and efficiency of delivering service, and
increased service restoration speeds will result in higher reliability metrics.
The technologies discussed within this document will be responsible for supporting critical electrical assets,
systems and services within an EPU. To ensure that adequate clarity and control of a technology,
dynamic, scalable and intuitive management systems need to be put in place. Management systems
should be able to accurately present, configure, report and alert on network state in a close to real-time
fashion while ensuring compliance with change control and management methodologies.
29 | P a g e
3.2.9 Scalability
The migration of EPU Services and applications to new communication technologies will take place over a
long period. While future requirements are captured where possible, even more demands and
requirements will be presented post deployment. For this reason it is necessary that all technologies are
assessed based on the ability to meet:
3.2.10 Availability
The availability concept is related to time. That is, availability can be defined as the time in which the
system is reliable. In Optical systems, highest availability is achieved by means of certain degree of
Redundancy. That redundancy must cover not only the electronic devices such are transmitters, receivers
or amplifiers, also must cover the fibre path implementing protected routes between nodes.
Availability is also strongly influenced by the network recovery time which depends on the failure
detection time and the availability of a protection path.
30 | P a g e
4 ASSESMENT OF TECHNOLOGIES
This section discusses a number of technologies which are currently already in use in EPU networks or
may be considered for future extensions of the network:
This is not a detailed technology tutorial, but has a focus on specific characteristics which are relevant for
EPU applications. Therefore for each technology the following is discussed:
Life Cycle
Scalability
Capability to support EPU Services
Ease of Implementation & Operation
Interoperability (other services/technologies)
Redundancy, Availability, Reliability
Quality of Service (QoS)
Network Management System and Overall Manageability (NMS)
31 | P a g e
Scalability
Huge networks have been built with SDH technology and several European Public Telecom
Operators still use SDH as the backbone network. Rates of 10 Gbps (STM-64) are commonly
installed and 40 Gbps (STM-256) rates are beginning to appear in the market. SDH is commonly
used to build extensive wide area networks. In the following table most common SDH rates are
shown. That said, SDH still provides the ability to interface and connect networks and devices at
very low data rates (64kbps) in t seemless fashion.
Synchronous
Transport Module
Hierachy
(Mbps)
Interface
electrical/optical
STM-1
155,520
G.703/G.957
STM-4
622,020
---/G.957
STM-16
16
2 488,320
---/G.957
STM-64
64
9 953,280
---/G.957
East
West
STM-1,STM-4,STM-16,STM-64
STM-1,STM-4,STM-16,STM-64
Tributaries
Electrical
intefaces
2M,34M,45M,140M
Optical
interfaces
STM-1,STM-4,STM-16
point. Especially in utilities, SDH in combination with PDH offers interfaces for legacy electrical
protection and control devices that can hardly be obtained with other technologies in a native
way. SDH provides also a deterministic behavior, which is widely appreciated by traffic
engineers.
Recent developments have been made in order to accommodate SDH with data based network,
such as Ethernet over SDH. SDH works efficiently with WDM.
DXC
ADM
SubNetwork Connection
Protection: Simultaneous
tranmission by two paths
working
protection
ADM
35 | P a g e
Scalability
There are two types of WDM architectures: CWDM, and DWDM:
A CWDM system typically provides 18 Channel-Wavelength, separated by 20nm, from 1260nm
to 1630nm.
DWDM is a technology used for a very high capacity links (ITU-T G.694.1). DWDM systems,
mainly used by Public Telecom Operators, can multiplex from 32 to more than 100 channels in
the range of 1530 1624 nm.
36 | P a g e
.
Figure 21 Optical spectra
in fiber
The number of channels, the channel selection (center frequency), and the frequency width of each
channel, as well as the channel separation, are important parameters in WDM system design.
With a channel spacing of 1nm (or below), DWDM operation requires extremely precise optical
sources (laser temperature stabilization) resulting in significantly higher complexity and cost.
WDM technologies can accommodate any bit rate providing necessary bandwidth: SDH (STM1,
STM4, ), Ethernet (1Gbit/s, 10 Gbit/s)
WDM technology has brought more than an order-of-magnitude increase in the amount of
bandwidth that can be transported over fiber. Early implementations were point-to-point only.
However, while transport networks may be thought of as the roads of the network, the
intersection points are also critical.
Natively, WDM technology provides a point to point links, however it is possible to have different
type of architectures (ring, mesh ) by mixing WDM with other protocols (Layer 2, Layer 2.5,
Layer 3).
Todays Optical Ethernet networks are much more dynamic than in the past and demand greater
flexibility. In order to provide the necessary flexibility, Reconfigurable Optical Add Drop
Multiplexers (ROADMs) were developed. They allow operators to access any wavelength at any
node at any time replicating the operational simplicity and flexibility of SONET/SDH networks
at the wavelength level.
37 | P a g e
State of the art ROADM technology and the use of tunable lasers and filters, allows operators to
not only drop any wavelength at any node and any time, but to also send any wavelengths in
any direction (directionless) using any available port on the network node (colorless).
This so-called triple-A architecture (AAA any wavelength, any node, any time) is fully flexible
and non-blocking. It requires very little technical skill from the operational staff and eliminates the
need for meticulous pre-planning.
Triple-A ROADMs are the foundation of a fully automated optical network, and they allow for
intelligent interworking using the two layer architecture concept, which despite other technologies,
have the facility to deploy a network architecture with two stacked switching layers.
Power
[dBm]
Power
[mW]
10 mW
+10
TX/Laser Diodes
1 mW
0
- 10
TX/LED
- 20
- 30
1 W
in RX
in RX
iodes
d
o
t
iodes
o
d
h
o
p
t
r
o
o
f
e ph
r limit
lanch
Lowe
r ava
o
f
it
r lim
Non detectable signal in RX
Lowe
1 nW
1M
10 M
100 M
1G
10 G
- 40
- 50
- 60
100 G
For example for 10 Gbit/s bite rate, standard photodiode sensitivity limit is higher than -35 dBm
(Red point in the figure above) because the limited energy of the incoming optical pulses.
Obviously these characteristics imply that the optical amplification becomes necessary for high
speed transmission. An alternative solution is to spread the high speed stream in some medium /
low streams and use passive WDM to transmit simultaneously those streams in a single fiber pair
channel that is independent of the working channels of the WDM product to create a standards
based data communications network that allows service providers to remotely monitor and control
system performance and use. Meeting ITU standards and utilizing a Q3 interfaces ensure that
end users retain high Operations, Administration, Maintenance, and Provisioning (OAM&P)
service. Most systems use SNMP as a standard but in-depth configurations of devices may not be
achievable with open standards.
40 | P a g e
41 | P a g e
42 | P a g e
Scalability
The basic signal architecture of OTN is shown in the FIGURE 26 below:
43 | P a g e
Client Signal
Electrical Domain
Optical Domain
Optical Channel
(Och)
44 | P a g e
Payload Rate
Supported Clients
ODU-0
1.25Gbps
ODU-1
2.5Gbps
STM-16, 2G FC
ODU-2/ODU-2e
10Gbps
STM-64, 10GE
ODU-3
40Gbps
STM-256, 40GE
ODU-4
100Gbps
100 GE
ODU-Flex
N*1.25Gbps
Transparent signal transport thanks to the service-agnostic mechanism that maps different
client services in to the same data frame level.
OTN multiplexing bandwidth granularity is one or two orders of magnitude higher than
for SDH technology, thus making it more scalable and supporting higher rate.
It uses DWDM transport system to reach line rates of 40 Gb/s and 100 Gb/s
OTN provides an integrated mechanism for forward error correction (FEC) that allows
greater reach between optical nodes and/or higher bit rates on the same fiber.
The introduction of enhanced OAM channels built on the experience gained from the
existing SDH overhead
These functionalities ensure that adequate capacity and control exist for EP~U activities however
OTN solutions do lack the ability to directly interface with legacy technologies and EPU specific
applications.
45 | P a g e
SDH/SONET
Ethernet (Electrical & optical)/ VLAN based, port based, MAC learning
IP/VPN
IP-MPLS, MPLS-TP
Ethernet SyncE
Common module redundancy: Matrix fabric, controller board, power supply, FAN
Tributary and traffic module redundancy: 1+1, 1:1 and 1:n linear protection,
Shared Protection Ring (SPRing) for HO and LO ODU-k
OTN supports enhanced OAM channels that allow the monitoring of the traffic in order to trigger
switching mechanisms for service protection.
OAM
OTN builds on the concept of Tandem Connection Monitoring (TCM) seen in SDH. SDH only allows
one tandem connection, whereas OTN supports 6. The optical channel data unit (ODU) carries
therefore TCM1-6 in the overhead section. Additionally it carries the path monitoring overhead
(PM).
TCM1 is used by the User to monitor the QoS that they see. TCM2 is used by the first operator to
monitor their end-to-end QoS. TCM3 is used by various domains for Intra domain monitoring.
Then TCM4 is used for protection monitoring by the operator.
The concept of the network management system is very much vendor specific.
Multi layer OTN/WDM networks generally require a network management for both the IP and
the transport layer. That said, some vendors have the concept of an unique network management
system for both the transport layer and the IP network.
This will be very helpful for maintenance staff that is used to the circuit switched technology
compared with other IP based technology.
Scalability
Networks consisting of thousands of nodes have been built with IP/MPLS technology. Most Public
Telecom Operators and big Enterprises use IP/MPLS within core and access networks. IP/MPLS is
not directly tied to any transport technology and thus can support any link speeds. 100Gbps links
are commonly installed and support for faster links is expected as they appear. IP/MPLS
networks can also support very slow link speeds.
Support multipoint topologies in addition to traditional point-to-point topologies makes IP/MPLS
very attractive as a potential future technology for EPU use. Many new operational
communication models/protocols/applications are built natively to run in IP networks.
IP/MPLS is supported in wide range of different products from small ruggedized models to
carrier grade multi-chassis devices. This means it can be deployed in various places, hence
addressing a wide variety of EPU use cases.
various service topologies (namely point-to-point, multipoint, tree and anycast). Point-to-point
L2VPNs are called pseudo-wires and they support most TDM services.
For a technical summary of IP-MPLS, refer to Cigr Technical Brochure from WG D2.28 Communication Architecture for IP-based Substation Applications.
A major reservation for EPUs of the use of packet-based technologies, in particular IP-MPLS, is
the ability to support the requirements of some of its critical applications, in particular
Teleprotection (see 2.2 List of Services). However, research and testing are ongoing in this area.
TDM circuits (E1/T1, SDH, etc.) can be carried by IP/MPLS but can utilise a lot of bandwidth
Support for layer 2 Ethernet (point-to-point/multi-point) is good
IP multicast over IP/MPLS is still relatively new to utilities
Layer-3 VPN (L3VPN) and Layer-2 VPN (L2VPN)
Some IP/MPLS equipment platforms to date support a substantial amount of EPU specific
hardware interfaces.
Redundancy, Availability and Reliability
IP-MPLS supports high availability through traffic engineering (TE) and Fast Reroute (FRR), with
fault detection in the order of 50ms using mechanisms such as BFD (bidirectional forwarding
detection).
48 | P a g e
The protocols surrounding MPLS-TE and FRR such as RSVP-TE have reached maturity within
enterprise and carrier deployments. Link and node protection via FRR can now be computed by
most manufacturer devices dynamically, easing deployment of the feature.
Protection in IP/MPLS consists of two complementary technologies:
Fast Reroute (FRR), which provides immediate short term recovery upon a failure. FRR has a
deterministic switchover delay.
End-to-end path protection, which provides the operator control of the path after failure, and
has non-deterministic switchover delay, due to additional signalling required. Hence end-toend path protection is used with FRR to provide both short-term recovery to minimise packet
loss immediately upon a failure, and then allow end-to-end path protection to take the time
required to calculate the new desired path.
Coarse-grained admission control using RSVP-TE, where signalling is performed between the
LSP tail end towards the head end to determine whether there are sufficient resources to set
up a path.
Per-class admission control and per-hop-behaviour using DiffServ-TE
With the per-class admission control (Diffserv-TE), although it provides more control on QoS, it is
more complex to deploy and requires that all nodes are Diffserv-TE aware.
The IP/MPLS protocol have a limited set of OAM defined. New developments to the MPLS
standard which evolves to MPLS-TP by the efforts of IETF and ITU-T, seek to define OAM features
similar to traditional transport protocols such as SONET and SDH.
The OAM toolset in IP/MPLS is not as well-defined as MPLS-TP, and in comparison, IP/MPLS
cannot provide transport-like OAM functionality. The OAM toolset within IP/MPLS are LSP ping
(to troubleshoot label switched paths) and BFD.
49 | P a g e
50 | P a g e
TDM circuits (E1/T1, SDH, etc) can be carried by MPLS but can utilise a lot of bandwidth
Support for layer 2 Ethernet (point-to-point/multi-point) is good
Layer-3 VPN (L3VPN) and Layer-2 VPN (L2VPN)
MPLS-TP equipment platforms to date support a substantial amount of EPU specific hardware
interfaces.
Some MPLS-TP equipment platforms to date support a substantial amount of EPU specific
hardware interfaces
Redundancy, Availability and Reliability
MPLS-TP is using OAM packets to trigger different protection mechanisms with the possibility to
configure 1:1, 1+1, 1:N and ring protection options. In the 1:1 protection option there is a
working and a protection path between two nodes where the protection switching can be
triggered by:
51 | P a g e
Detected defect condition (AIS-Alarm Indication Signal, LDI-Link Down Indication, and LKRLock Report messages)
Administrative action
Far end request
Server layer defect indication (LOS)
Retrieve timer (wait-to-restore)
FIGURE 29 gives an illustration of a 1:1 protection configuration and in MPLS-TP there is sub-50ms
protection switchover.
Operation, Administration, and Management (OAM) have been, and are, the key focus areas of
the MPLS-TP standardization work. The reason for this is that todays TDM transport networks like
SONET/SDH are extensively using this type of OAM functions to be able to meet the
52 | P a g e
requirements of the transport networks, and these functions also need to be implemented in
MPLS-TP. Functions that are being implemented in MPLS-TP are fault detection (connectivity check,
connectivity/path verification), fault localization and performance monitoring (delay and loss
measurement).
One of the goals of MPLS-TP is to make the establishment of the paths in a similar static way as
the TDM systems through a NMS system. Because of this the management of a circuit based and a
packet based network will be quite similar. In addition the MPLS-TP standard also has the option
of a dynamic control plane (Generalized MPLS), and this could be implemented for scaling
reasons and for establishment of more advanced protection functions. MPLS-TP bandwidth can be
reserved, and this must be configured explicitly at each hop (on the Label Switched Path), but this
bandwidth enforcement requires QoS configuration in the network
In the same way as the circuit based transport networks work today, MPLS-TP also needs to
support out-of-band management over a dedicated management network. An MPLS-TP transport
network should function without the implementation of protocols like IP and dynamic routing
protocols, and for that reason an out-of-band management network is needed, so that the NMS
can reach the MPLS-TP equipment.
53 | P a g e
4.8 Ethernet
Life Cycle
It is difficult to mark Ethernets place within its Product Life Cycle. It was first introduced in 1980
however there are no apparent successors to it being developed. For this reason it seems that it
is one of the most matured communications technologies as well as one of the most common layer
2 technology deployed today. While a roll out of Ethernet only would be one of the fastest
options for deployment. It would prove extremely difficult to manage and operate a network as
it scales to large amounts of devices. For this reason the use of Ethernet solely as a transport
mechanism would not be envisaged as an adequate method of deploying a network for
operational use. Based on this, Ethernet will not be regarded as a potential for use as a transport
mechanism
54 | P a g e
Scalability
There are three separate axis associated with the scalability of Ethernet. These are:
The tables depict the technical applicability of these technologies to deliver EPU Services however, it
should be noted that this does not necessarily reflect a practical suitability of the technology.(Example:
the availability of devices for external pole top applications)
A PPLICATION
SDH
WDM
OTN
IPMPLS E TH
MPLS -TP
* MPLS-TP is at quite an early stage of adoption within the EPU sector. Wide scale applicability is yet to
be proven
Organisational considerations (BASED on SDH legacy SDH WDM OTN IP- MPLS Eth
control)
MPLS -TP
Non-disruptive rollout
*2 MPLS implementation and operation can vary based on the method used to manage it(NMS, CLI etc)
Concerns exist with regard to the maturity of the technology within EPU environments
NMS features and functionality are not standardized or controlled therefore experience can vary
56 | P a g e
Technical appraisal
SDH WDM
OTN
Network Manageability
*1
Scalability
Life Cycle
QoS
*1 WDM has a number of flavors, some passive solutions lack any capability for vision of the
technology and rely upon layers implement above to gain manageability
*2 The current maturity of OTN is unclear within an EPU sector thus difficult to classify
Concerns exist with regard to the maturity of the technology within EPU environments
57 | P a g e
Utilities are gaining more of an appetite for new, alternative technologies, Ethernet and IP are
becoming more prevalent in operational networks
Most utilities have begun to roadmap a new technology choice within operational environments.
The survey from D2.35 however shows that the pace of transition is slower than predicted in the
D2.28 survey
Technology Life Cycle - Utilities are confronted with the phase-out of installed legacy equipment
and the risk of installing new solutions with typically much shorter life-cycles than they were used
to. It's an opportunity to look into benefits and drawbacks of either purely packet-based
technologies or hybrid networks.
Understanding of Smart Grid and IoT The communication requirements of the smart grid are
becoming more apparent, and are now beginning outline inadequacies of current solutions while
also outlining new requirements for alternative technologies
Services
To accurately assess any technology or solution within an organisation it is necessary to ensure that
requirements are clearly defined. In this topic, EPU transport solutions, requirements are defined by the
services which are carried by a technology. Issues identified by the brochure are:
No formal definition of service requirements exists within a single document Currently, not all
utilities and stakeholders share a common view to the requirements of services
There is a lot of on-going work within differing IEEE and IEC groups but there is no mechanism to
consolidate the status or results of these works
No evident attempts have been made to further classify services with common attributes into a
number of concise groups. This will be discussed further in the next paragraph
To enable the comparison of technologies outlined in the brochure a basic structure was defined
classifying different services into groups. This enables the uniform assessment of technology suitability.
This approach can account for services which are currently deployed and also future services currently
lacking concise requirements.
Technologies
58 | P a g e
Technology refresh can have substantial implications at an organisational level. It is important that
all opportunities and risks are exposed during the evaluation of suitable technologies. Implications of this
are organisation specific. The brochure assumed a control point of an EPU currently operating an SDH
solution. Opportunities and risk associated with all technologies can be noted as:
Migration to packet/frame based technologies presents new challenges and risk but also
presents new opportunity from and organisational level surrounding OT/IT crossover, skillsets and
knowledge.
Features highlighted as most relevant when assessing differing technologies are defined as :
Predictable timing for time sensitive services and the ability to distribute precision time protocols
Security and the secureness of a technology as opposed to the securability of technologies
Ability to seamlessly interact with EPU applications and protocols
Ability to segment and virtualise
Scalability potential of the technology
Natural resilience due to the fundamental concept of a technology
Each technology discussed presents unique benefits towards the specifications above. For this reason
some technologies tend to support specific applications more easily. One such case would be point to
multipoint capabilities presented by IP/MPLS which supports a huge application base. Others include the
deterministic nature of SDH to support timing constrained applications such as teleprotection.
The Matrix allows comparisons to be drawn between technologies and suitability for specific use cases.
From the matrix it is possible to affirm the conclusions of section 4
Synchronous Digital Hierarchy SDH, the current incumbent within EPU networks, has the ability
to carry all current EPU applications. During the roll out of newer distributed applications, it is
evident that SDH does not possess suitable agility and scale to suffice as the only technology in
the future.
DWDM/CWDM While the technology ensures controlled and deterministic qualities which do
apply very well to support latency sensitive applications such as Differential teleprotection,
CWDM is a robust technology, but it does not scale very well, DWDM does lack granularity,
control and vision over a network, post deployment. It could be applied as a foundation for
subsequent layers of technologies
OTN OTN presents as a technology which could support most if not all needs of a future EPU,
concerns do exist surrounding the ability to interface with OTN at a low bandwidth or with EPU
specific connections for legacy applications. OTN systems, in large seem to be scaled to support
bandwidths well beyond the operational requirements of an EPU. This may be relevant if intent
exists to commercialise bandwidth not utilized by the EPU
IP-MPLS The ability exists to support the majority of legacy, current and future applications for
utilities. The two most striking concerns relate to the ability of IP-MPLS to carry differential
teleprotection schemes over wide areas and the organizational change/skillset refresh involved
to migrate from a technology such as SDH.
MPLS-TP Available for a number of years, and set the targets IP-MPLS needs to achieve.
While achieving far more granular control during circuit provision and operation, there are
59 | P a g e
shortcomings when implemented at scale. Both technologies (MPLS-TP and IP/MPLS) will develop
further in the future. However the overlap of the two will become bigger.
Provider Backbone Bridging (PBB) This technology seems to have been overshadowed within
all industries by OTN and for that reason does not show much promise to have a substantial role
in EPU networks
Ethernet over fibre, SDH, MPLS etc. Ethernet is not a viable solution for transportation of EPU
Services. That being said it could be applied as a foundation for technologies discussed within
the document or provided itself a s service over the top of a technology
Future Works 1
Views and decisions made by utilities, at a global level, have proven invaluable to the completion of this
brochure. It has also allowed the ability to compare, contrast and trend viewpoints on technologies,
affirming the conclusions of the brochure.
The exercise of distributing D2.28s questionnaire should become an independent exercise completed on
an annual basis aiming to compile a global view into the trends, aspirations and concerns utilities are
dealing with. This would allow utilities to individually act knowing that any decision is in line with a
globally aligned movement in the same direction.
Future Works 2
No document currently exists defining the full spectrum of EPU applications and communicative
requirements associated with them. Without formal definition of requirements it is not possible to classify
applications and uniformly approach the assessment of suitable technologies. Work suggested in this
area is a working group tasked with:
Identifying all current and most future applications utilized within utilities and standardized
Creating a uniform methodology to assess applications which can be adopted by vendors and
understood by end users/network designers
Document communication constraints and requirements for each application
Categorise and group applications with common communicative needs
Define a framework and approach to assessing technologies suitability to all application groups.
A framework, similar to that outlined above would ensure that a global assessment of technologies for
EPUs would give EPUs, Telcos, vendors and solution providers a common mechanism to identify, specify,
categorise and affirm end-to-end requirements and solutions for all communications networks. The
framework would also present an opportunity for Cigr to periodically produce reports assessing the
posture of technologies including ongoing suitability, user adoptance and ranking based on
advancements and the feedback from information received in Future Works 1. This would align similar to
the magic quadrant approach to enterprise solutions.
60 | P a g e
Future Works 3
Future Works 1 and 2 highlight the need to document and identify the current standpoint of the EPU
market as well as mechanisms to approach the global classification of EPU services and define a set
criterion which manufacturers and vendors can meet. What are the services and where are we now.
A methodology toward assessment of how to migrate services and also when to migrate services could
also prove very beneficial for EPUs to plan lifecycles and utilize existing transport assets until their end
of life. Works constituting this are captured within the proposed working group Migration Planning and
Transformation Control in EPU Telecom Networks.
61 | P a g e
Survey Objective:
Dear reader,
Thank you very much for participating in this survey. Your contribution will help D2.35 to identify the
trends in the use of IP and other communication transport solutions in the EPU environment and identify the
main issues, questions and topics to be addressed in the Technical Brochure.
This survey is a continuation of the survey conducted in 2009-2010 by D2.28 and will be used to identify
trends, main issues and other topics of importance for the Technical Brochure to be prepared by D2.35
This survey is conducted among CIGR Working Group D2.35 members and CIGR members of SC B5
and D2.
Please check and detail your options.
62 | P a g e
Questionnaire:
1.
Are you a:
Consultant
Vendor
2.
Within your own company, which substation applications are using IP and IP networks? (For
vendors: Which applications do you see at customers sites?)
Metering
Telephony System
Tele protection
Time Synchronization
63 | P a g e
3.
From the list in Question 2, what substation applications (currently not running on IP) could
potentially be migrated to IP?
4.
What other applications are you using that could potentially be migrated to IP (applications
outside the substation)?
Phasor Measurements
5.
What are the main operational challenges of using IP as a communication transport solution over
optical networks?
Security
Interoperability
Ruggedness (environment)
Scope of Responsibility
64 | P a g e
Maintenance
Reliability
Additional Comments
Would the answer be different when using IP for substation specific application? If yes,
how:
6.
What are the main psychological barriers with using IP protocols in applications?
Security
Reliability
Complexity
Lack of Documentation
65 | P a g e
7.
Yes
No (please detail)
8.
What is your prediction for the migration of all operational communications into IP?
Already Done
Short Term
Within 1 Year
Within 1 2 Years
Within 2 5 Years
Within 5 10 Years
Never Why?
9.
What percentage of the existing applications within your company are already using IP as
communication transport solution?
0%
25%
50%
75%
100%
66 | P a g e
10.
Protocol Encapsulation
Emulation
Separate Network
11.
Considering that all applications will move to IP, what are the requisites and concerns for the
telecommunication network?
Organisational structure
Identification of Applications
12.
Which underlying technologies are the most promising to provide secure and reliable transport
communications in the access network?
SDH
CWDM
DWDM
OTN
Ethernet
IP / MPLS
67 | P a g e
PBB / PBT
MPLS TP
RPR
13.
Which underlying technologies are the most promising to provide secure and reliable transport
communications in the core network?
SDH
CWDM
DWDM
OTN
Ethernet
IP / MPLS
PBB / PBT
MPLS TP
RPR
14.
Should the IP network be reserved for operational service or also be used for corporate services?
Two physically/virtual separate networks or one network?
15.
Number of Users
Number of Applications
Number of Protocols
Variation of Protocols
Number of Interfaces
Number of Nodes
Number of Locations
Number of Services
Bandwidth
16.
Are you aware of any activities /documents concerning communication transport solutions over
optical networks that may be of interest for D2.35 to consider for the Technical Brochure? If yes,
can you please list them below:
Glossary of Terms
Abbreviation
Explanation
CWDM
DWDM
IP
Internet Protocol
IPv4
IPv6
LAN
MPLS
OTN
PBB
PBT
PDH
RPR
RTU
SCADA
SDH
WAN
70 | P a g e
WGD2.28 Survey
IP Based Substation Applications
Survey Objective:
Get an overview of the present use of IP-based applications within the substation environment, between
Substations and for Substation - to - Control Centre communications.
This survey is conducted among CIGR Working Group D2.28 members and CIGR members. Please
check and detail your options.
Questionnaire:
1- Are you a:
2- Within your own company, which substation applications are using IP and IP networks? (For
vendors: Which applications do you see at customers sites?)
3- What are the main operational challenges using IP into the substation environment?
Security
71 | P a g e
4- What are the main psychological barriers with using IP protocols in Substation applications?
Security
Reliability
Quality of Service
Familiarity with Ethernet/IP concepts
Rate of innovation and obsolescence
Lack of documentation
Other (please detail):
Yes
No (please detail):
Already done
Short term
6 months
1-2 yrs
5 yrs
More than 10 yrs
Never - Why?
Protocol encapsulation
Vendor
72 | P a g e
Yes
No (please detail):
10- Which underlying IP technology is the most promising to provide secure reliable IP
communications?
11- Should the IP network be reserved for operational service or also be used for corporate services?
Two physically/virtual separate networks or one network?
Virtual LAN
Virtual Private Networks (Layer 3)
Virtual Private LAN Service - VPLS (Layer 2)
Multicast filtering
Other (please detail):
73 | P a g e
74 | P a g e
75 | P a g e
Other than the applications listed in the figure above, participants indicated that they use IP for the
following applications:
-
76 | P a g e
Score
Existing
Score
Eligible
Conclusion
Equal
Equal
Higher
Lower
Lower
Higher
The totals, represented by the greens bars in can provide an indication of how suitable the participants
think IP is for certain applications. The vast majority of the participants seem to think that Tele-protection
and Current Differential protection might not be suitable for use with IP.
78 | P a g e
79 | P a g e
80 | P a g e
Revenue meters
Non availability of IP networks at some locations
(Tele) protection
Differential protection
Time synchronization
Premise access control
Power quality
Legacy systems
PABX
81 | P a g e
82 | P a g e
Currently rolling out a SDH bearer network with an MPLS network overlaid on it. Legacy
protocols will be transported on the SDH network. IP services will use the MPLS network.
Each network will have separate VPNs and VRF
Migration to IP dedicated operational equipment.
Replace legacy equipment
84 | P a g e
Reliability
Quality of Service (QoS)
traffic load and segmentation
85 | P a g e
86 | P a g e
87 | P a g e
88 | P a g e
89 | P a g e
Especially at transmission level, HV-line protection remains a critical application. Line differential protection (with
its stringent timing requirements) across Packet Switched Networks is not yet solved from the perspective of the
majority of utilities (feedback e.g. from EUTC conference 2013)
Utilities use precise timing to accurately timestamp data, measurements and events and share it between power
control, protection and measurement equipment. Timing can be used within substation environments, and also
across the WAN to exchange information between substations and control centers. Timing can be exchanged
within a single application or shared between multiple applications.
Precise timing is used for real-time processes as well as post-event analysis.
In addition to substations and grid management, timing is also essential in data center and operating center
local area networks, and throughout the EPU telecom and wide area networks. By looking at the overall network
picture, a utilities form strategies and practices to protect their timing and synchronization infrastructure.
Historically millisecond and tens of microsecond accuracy has been provided by SNTP and IRIG-B amongst other
mechanisms for clock redistribution. However, newer applications such as PMUs and protocols such as 61850
Sampled Values (SV) require microsecond accuracy. Accurate clocks enable sophisticated analysis of real-time
and post-disturbance faults and events within a short amount of time. This leads to faster and more reliable fault
isolation and resolution, and better planning of transmission resources on the electric grid.
90 | P a g e
Electric power utilities use precise GPS based timing at their power transmission and distribution facilities to time
stamp data and measurements. Time synchronised data can be analysed to assess causes of faults or disruptions
on the power grid, and is an essential component of newer real-time control and protection systems such as
System Integrity Protection Schemes (SIPS) and Centralised Remedial Action Schemes (C-RAS). Without accurate
time stamps, the collected data cannot be correlated and becomes worthless. This reliance on GPS has been
highlighted as a potential weakness of the traffics above and for this reason it would be preferable to attain
timing similar to that available within GPS streams through technologies deployed on Fibre Optics
Synchronisation Types
There are typically two types of time synchronization in EPU networks: direct and indirect over a communications
network. The internal clock of a device will be synchronized to master clock or server which is normally
connected to an accurate time source such as GPS, or quantum clock. The accuracy and variance of the time will
vary depending on a number of factors including communications network/media, distance, protocols, traffic,
interference and errors.
91 | P a g e
For direct synchronization a standalone communications network for timing redistribution is required which can
be expensive. These are typically fiber, copper or coaxial networks. This approach is normally implemented for
intra-substation timing and uses protocols such as IRIG-B, GPS and 1PPS.
Synchronization over a communications network (LAN or WAN) redistributes time over a communications media
such as Ethernet, and is usually part of a converged (timing and other data) network, reducing cost. Traditionally
NTP/SNTP have been used to synchronise devices, and more recently 1588 PTP has become an alternative or
an essential option depending on required accuracy of an application. In addition Synchronous Ethernet (SyncE)
can be used to redistribute frequency over the WAN and mitigate clock drift between WAN devices.
IRIG-B Overview
Inter-range instrumentation group time codes B (IRIG-B) and is an industry standard for GPS time
synchronization. IRIG-B can be applied in a substation for power quality and system stability monitoring,
sequence of events recording and accurate time stamping for revenue billing (1 ms). IRIG-B needs to be
transmitted via a dedicated timing network (usually of copper or coax cable) and is therefore not a lowcost solution.
IRIG-B needs an external time source such as GPS, can be modulated (over a carrier signal) or
unmodulated (no carrier signal), and can deliver accuracy in the microsecond range (as low as 10-100
microseconds). It is widely used in the electrical power industry with around 90% of intra-substation
timing use cases, and may contain time of year, year, and seconds of day information.
92 | P a g e
The GPS system is highly redundant (although there are concerns over security of the system) and is used
in the EPU industry for both direct time synchronization and as a time source for other time protocols. For
direct synchronization an antenna per IED or device is required which can be a costly solution, or GPS can
be used as a time source to be redistributed through a timing network (dedicated or shared). The GPS
system is capable of delivering timing accuracies of within ten nanoseconds,
Global Positioning Systems (GPS) are used to provide a precise time reference for power system devices
in an electrical network, ensuring time synchronization even across dispersed networks (substation or
distribution networks) and over large geographical distances.
In the distributed model a GPS antenna receives a time signal and a receiver converts it to one or more
time protocols which are distributed to power system devices which in turn set their internal clock.
The Global Positioning System provides a very accurate time reference, however when GPS is not
available the system cannot operate as intended for long, and therefore vulnerabilities of the GPS have
become a major concern.
Other available satellite based technologies using the same principles are GLONASS, GALILEO and
COMPASS
3.2.3. Impairments affecting satellite based timing systems
The reception of the signals is subjected to many error sources, which can lead to degraded system
performance. The most common sources and their effects are:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
DISTANCE ERROR
3m
7m
2m
2.5 m
12 m
100 m
1m
16,19 km/day
90 m
30 m
TIMING ERROR
10 ns
23.4 ns
6.67 ns
8.34 ns
40 ns
334 ns
3.33 ns
54 s/day
300 ns
100 ns
93 | P a g e
50 m
167 ns
The effects mentioned above may not directly effect any of the applications mentioned within this
document but could contribute as a tipping point for overall latency and delay thresholds within a system.
These can be mitigated.
NTP/SNTP Overview
NTP (Network Time Protocol, RFC 1305) is a software method to transfer time between clients and
servers over packet switched variable latency data networks, and was designed to minimize the effects
of variable latency. Although most commonly used over UDP/IP protocol, it can also be transported via
TCP/IP. It has been particularly designed to resist the effects of variable latency.
NTP provides reasonable accuracy, from a few milliseconds up to a few hundred milliseconds depending
on the nature of the connection between the NTP client and the server. NTPv4 can usually maintain time to
within 10 milliseconds over the public communications networks such as the Internet, and can achieve
accuracies of 200 microseconds or better in local area networks under ideal conditions. NTP is often used
for the time synchronization of communications network devices such as routers and switches on the EPU
communications network.
SNTP (Simple Network Time Protocol, RFC 2030) describes a simplified form of NTP. While they both use
the same network package format, but lacks some internal algorithms that are not needed for all types
of servers.
95 | P a g e