Anda di halaman 1dari 95

618

Scalable Communication Transport


Solutions Over Optical Networks

Working Group
D2.35

May 2015

SCALABLE COMMUNICATION
TRANSPORT SOLUTIONS
OVER OPTICAL NETWORKS
Type WG D2.35
Members
J.SANDHAM Convenor (IE), M.SCHATZ Secretary (CH), M.ACACIA (BE), F.CASTRO (ES),
J.DARNE (ES), J.FEIJOO MARTINEZ (ES), M.FLOHIL (NL), R.IRONS-MCLEAN (UK),
M.JANSSEN (NL), A.MOAINI (FR), T.V.PEDERSEN (NO), H.RIIS (DK), A.RUNESSON (SE),
S.TANNER (FI)

Corresponding Members
C.DI PALMA (AR), C.EVERITT (AU), J.MENDES (PT), P.SCHWYTER (CH), M.SEEWALD (DE),
A.SILFVERBERG (FI), V.TAN (AU)
Copyright 2015
Ownership of a CIGRE publication, whether in paper form or on electronic support only infers
right of use for personal purposes. Other uses are prohibited, except if explicitly agreed by
CIGRE, total or partial reproduction of the publication for use other than personal and transfer
to a third party; hence circulation on any intranet or other company network is forbidden.
Disclaimer notice
CIGRE gives no warranty or assurance about the contents of this publication, nor does it
accept any responsibility, as to the accuracy or exhaustiveness of the information. All implied
warranties and conditions are excluded to the maximum extent permitted by law.

ISBN : 978-2-85873-320-0
1|Page

SCALABLE COMMUNICATION
TRANSPORT SOLUTIONS OVER
OPTICAL NETWORKS
Table of Contents
1 INTRODUCTION................................................................................................................ 8
2 REQUIREMENTS FROM USERS .......................................................................................... 9
2.1 Survey Results .................................................................................................................................... 9
2.1.1 Substation applications using IP ............................................................................................ 11
2.1.2 Main operational challenges using IP as communication transport solutions ................ 12
2.1.3 Main psychological barriers with using IP protocols in Substation applications .......... 13
2.1.4 Are all applications compatible with IP at the moment? .................................................. 14
2.1.5 Prediction to migration of all operational communications into IP ................................. 14
2.1.6 How to deal with legacy protocols and equipment? ........................................................ 15
2.1.7 Requisites and concerns for the telecommunication network ........................................... 16
2.1.8 Most promising technologies to provide communications in the access network? ........ 17
2.1.9 Most promising technologies to provide communications in the core network?............ 18
2.1.10 Should the IP network be for operational service or also for corporate services? . 19
2.1.11 Survey Conclusions ................................................................................................................ 19
2.2 List of Services ................................................................................................................................ 20
3 GENERAL CONSIDERATIONS ......................................................................................... 22
3.1 Why are Utilities migrating? ........................................................................................................ 22
3.2 Migration considerations............................................................................................................... 24
3.2.1 Timing ......................................................................................................................................... 26
3.2.4 Communication Requirements Decomposition ..................................................................... 26
3.2.5 Security ...................................................................................................................................... 27
3.2.6 Communication Network Management ................................................................................ 28
3.2.7 IPv4 and IPv6 co-existence .................................................................................................... 29
3.2.8 Segmentation and Virtualization .......................................................................................... 29
3.2.9 Scalability ................................................................................................................................. 30
3.2.10 Availability ............................................................................................................................. 30
4 ASSESMENT OF TECHNOLOGIES .................................................................................... 31
4.1 Synchronous Digital Hierarchy (SDH) ......................................................................................... 32
Life Cycle ............................................................................................................................................. 32
Scalability ............................................................................................................................................ 32
Capability to support EPU Services ................................................................................................ 33
Ease of Implementation and Operation ......................................................................................... 33
Interoperability (other services/technologies)............................................................................... 33
Redundancy, Availability and Reliability ...................................................................................... 34
Quality of Service (QoS)................................................................................................................... 35
Network Management System and Overall Manageability (NMS) ......................................... 35
3|Page

4.2 Dense Wavelength Division Multiplexing & Coarse Wavelength Division Multiplexing . 36
Life Cycle ............................................................................................................................................. 36
Scalability ............................................................................................................................................ 36
Capability to support EPU Services ................................................................................................ 39
Ease of Implementation and Operation ......................................................................................... 39
Interoperability (other services/technologies)............................................................................... 40
Redundancy, Availability and Reliability ...................................................................................... 41
Quality of Service (QoS)................................................................................................................... 41
Network Management System and Overall Manageability (NMS) ......................................... 42
CWDM versus DWDM ....................................................................................................................... 42
4.3 Optical Transport Network (OTN) .............................................................................................. 43
Life Cycle ............................................................................................................................................. 43
Scalability ............................................................................................................................................ 43
Capability to support EPU Services ................................................................................................ 45
Ease of Implementation and Operation ......................................................................................... 46
Interoperability (other services/technologies)............................................................................... 46
Redundancy, Availability and Reliability ...................................................................................... 46
Quality of Service (QoS)................................................................................................................... 46
Network Management System and Overall Manageability (NMS) ......................................... 46
4.4 Dynamic Internet protocol based Multiprotocol Label Switching (IP/MPLS) ..................... 47
Life Cycle ............................................................................................................................................. 47
Scalability ............................................................................................................................................ 47
Capability to support EPU Services (as defined in chapter 2) .................................................. 47
Ease of Implementation and Operation ......................................................................................... 48
Interoperability (other services/technologies)............................................................................... 48
Redundancy, Availability and Reliability ...................................................................................... 48
Quality of Service (QoS)................................................................................................................... 49
Network Management System and Overall Manageability (NMS) ......................................... 49
4.5 Static Multiprotocol Label Switching Transport Profile (MPLS-TP) .................................... 50
Life Cycle ............................................................................................................................................. 50
Scalability ............................................................................................................................................ 50
Capability to support EPU Services (as defined in chapter 2) .................................................. 50
Ease of Implementation and Operation ......................................................................................... 51
Interoperability (other services/technologies)............................................................................... 51
Redundancy, Availability and Reliability ...................................................................................... 51
Quality of Service (QoS)................................................................................................................... 52
Network Management System and Overall Manageability (NMS) ......................................... 52
4.7 Provider Backbone Bridging (PBB) ............................................................................................. 54
4.8 Ethernet ............................................................................................................................................ 54
Life Cycle ............................................................................................................................................. 54
Scalability ............................................................................................................................................ 55
Capability to support EPU Services (as defined in chapter 2) .................................................. 55
Ease of Implementation and Operation ......................................................................................... 55
Interoperability (other services/technologies)............................................................................... 55
Redundancy, Availability and Reliability ...................................................................................... 55
Quality of Service (QoS)................................................................................................................... 55
Network Management System and Overall Manageability (NMS) ......................................... 55
4.9 Comparison matrix......................................................................................................................... 56
5 CONCLUSIONS AND PROPOSAL FOR FUTURE WORK .................................................. 58
4|Page

Survey ...................................................................................................................................................... 58
Services.................................................................................................................................................... 58
Technologies ........................................................................................................................................... 58
Future Works 1 ...................................................................................................................................... 60
Future Works 2 ...................................................................................................................................... 60
Future Works 3 ...................................................................................................................................... 61
ANNEX 1 SURVEY D2.35 ................................................................................................... 62
ANNEX 2 SURVEY D2.28 IP-BASED SUBSTATION APPLICATIONS.................................... 71
ANNEX 3 WGD2.35 SURVEY RESULTS............................................................................... 74
1 Substation applications using IP ...................................................................................................... 74
2 Substation applications eligible for use with IP ........................................................................... 76
3 Applications outside the substation eligible for use with IP ....................................................... 78
4 Main operational challenges using IP a communication transport solutions ........................... 78
5 Main psychological barriers with using IP protocols in Substation applications .................... 79
6 Are all applications compatible with IP at the moment? ............................................................ 80
7 Prediction to migration of all operational communications into IP ........................................... 81
8 Percentage of the existing applications using IP as communication transport solution? ...... 83
9 How to deal with legacy protocols and equipment? .................................................................. 83
10 Requisites and concerns for the telecommunication network................................................... 84
11 Most promising technologies are to provide communications in the access network? ........ 86
12 Most promising technologies to provide communications in the core network? ................... 86
13 Should the IP network be for operational service or also be for corporate services? ...... 87
14 What type of scalability needs to be addressed? .................................................................. 88
ANNEX 4 NETWORK TIMING ............................................................................................ 90
Synchronisation Types........................................................................................................................... 91
IRIG-B Overview.................................................................................................................................... 92
Global Positioning System (GPS) Overview .................................................................................... 92
One Pulse Per Second (1PPS) Overview .......................................................................................... 94
NTP/SNTP Overview ............................................................................................................................ 94
IEEE 1588 PTP / IEC 61588 PTP ........................................................................................................ 94
Versions of 1588 ................................................................................................................................ 95
NTP vs. SNTP vs. PTP: A Summary...................................................................................................... 95

5|Page

Figures
Figure 1 Companies represented in the survey responses ............................................................................. 10
Figure 2 Countries represented in the survey responses ................................................................................ 10
Figure 3 Trends in substation applications using IP ........................................................................................ 11
Figure 4 Trends in operational challenges ...................................................................................................... 12
Figure 5 Trends in psychological barriers ........................................................................................................ 13
Figure 6 Trends in current IP compatibility ...................................................................................................... 14
Figure 7 Trends in estimation of all operational traffic through IP ................................................................... 15
Figure 8 Trends in how to deal with legacy equipment and protocols ............................................................ 15
Figure 9 Trends in requisites and concerns .................................................................................................... 16
Figure 10 Trends in most promising access network technologies ................................................................. 17
Figure 11 Trends in most promising core network technologies ..................................................................... 18
Figure 12 Trends in network separation .......................................................................................................... 19
Figure 13 Traditional Grid Communications Architecture ................................................................................ 23
Figure 14 Tiered approach for New grid applications ...................................................................................... 23
Figure 15 Shift from TDM to packet ................................................................................................................. 24
Figure 16 Migration decision review process .................................................................................................. 25
Figure 17 Communications decomposition table for packet oriented traffic .................................................... 26
Figure 19 SDH Capacity Comparison ............................................................................................................. 32
Figure 18 SDH Add-Drop-Multiplexer .............................................................................................................. 33
Figure 20 SDH Sub-Network Connection Protection ...................................................................................... 34
Figure 22 Optical spectra in fiber ..................................................................................................................... 37
Figure 23 examples of Add-Drop-Multiplexer .................................................................................................. 37
Figure 24 Reconfigurable Optical Add Drop Multiplexer ................................................................................. 38
Figure 25 Working area for light source and receiver ..................................................................................... 38
Figure 21 WDM Multiplexing............................................................................................................................ 39
Figure 26 OTN Architecture ............................................................................................................................. 44
Figure 27 OTN Capacities ............................................................................................................................... 45
Figure 28 MPLS-TP Hierarchy ........................................................................................................................ 51
Figure 29 Protection configuration ................................................................................................................... 52
Figure 30: Substation applications using IP .................................................................................................... 75
Figure 31: Applications eligible for using IP..................................................................................................... 77
Figure 32: Operational challenges ................................................................................................................... 79
Figure 33: Psychological Barriers .................................................................................................................... 80
Figure 34: All applications IP compatible?....................................................................................................... 81
Figure 35: Predication of All operational traffic through IP .............................................................................. 82
Figure 36 Percentage of existing applications using IP ................................................................................... 83
Figure 37How to deal with legacy protocols/equipment .................................................................................. 84
Figure 38 Requisites and concerns for using only IP ...................................................................................... 85
Figure 39 Most promising access network technology .................................................................................... 86
Figure 40 Most promising core network technology ........................................................................................ 87

6|Page

Figure 41 Seperate operational services from corporate services? ................................................................ 88


Figure 42 What type of scalability to address? ................................................................................................ 89
Figure 43 Applications sensitive to time and frequency .................................................................................. 91
Figure 44 Time synchronisation application .................................................................................................... 92

Tables
Table 1: (Future) IP usage ............................................................................................................................... 78

7|Page

1 INTRODUCTION
The introduction of smart applications in the Electrical Power Utility (EPU) and consequent
dispersed intelligence result in a tremendous growth of information exchange across the power
system. This implies, in many cases, a change of scale in the requirements of the
telecommunication infrastructure and often the deployment of a core data transport network.
This may be implemented through a number of different technologies and architectures.
The present network of most power utilities is extensively composed of TDM (e.g. PDH/SDH)
technology. Packet communication and in particular Ethernet connections are growing very fast
and may bring the necessity to adapt and /or replace network technologies.
This Technical Brochure aims to identify and analyze solutions and migration plans in the light of
data network technology evolutions, new application requirements and EPUs capability to
maintain the systems operation.
During 2009 and 2010, Cigr working group D2.28 conducted a survey amongst Cigr members
to identify the current and expected future use of IP networks within Electric Power Utilities. For
this Technical Brochure a follow-up survey was conducted to identify trends in the use of networks
and network technologies
This Technical Brochure contains:
The results of the follow-up survey held for this Technical Brochure
An analysis of trends in the use of networks and network technologies based on a
comparison of the survey held for working group D2.28 and the follow-up survey held
for this Technical Brochure
General considerations for replacing, refurbishing or extending networks
An assessment of available physical technologies and transport protocols
Recommendations for future works

8|Page

2 REQUIREMENTS FROM USERS


To make sure the contents of this Technical Brochure are relevant Working Group D2.35 started
by identifying what kind of information would be required by the intended audience. To identify the user
requirements a survey was held amongst Cigr members. Working Group D2.28 held a similar survey a
few years before so the results of the new survey can be compared to those results to identify trends.

2.1 Survey Results


This next section describes the trends that can be identified by comparing the results from the survey held
by working group D2.28 with the survey held by working group D2.35. The complete results from the
survey held by working group D2.35 can be found in Annex 3
One of the goals of this survey was to try to identify trends in the adoption of IP networks. To that extent,
some of the questions in the survey held in 2009-2010 by D2.28 have been re-issued in the survey that
was held for this Technical Brochure.
This section provides an indication of the trends between the D2.28 survey held in 2009-2010 and the
D2.35 survey held in 2012. When comparing the results of the survey held for D2.28 and the survey
held for this technical brochure, keep in mind that:
-

The survey that was held for working group D2.28, while similar, is not exactly the same as the
survey held for this Technical Brochure. For a detailed comparison both surveys can be found as
an Annex to this TB.
The survey for D2.35 was held under all Cigr members, not just the participants in the D2.28
survey. This means that there may be differences between the results of the two different surveys
due to differing companies participating in the surveys.

Only questions that were similar or exactly the same between the two surveys are compared below.
In total, 83 filled out surveys were received from 29 different countries. Figure 1 shows which companies
from which countries have responded.
ANSWERS FROM:
TRANELSA
ENERGINET.DK
EGAT
ITAIPU BINACIONAL
Scottish Power
REN
SEAS-NVE
AXPO AG
HUAWEI
INEXUS
ABB Switzerland
AUSGRID
POWERLINK
ESSENTIAL ENERGY
ETSA Utilities
TRANSEND
ELECTRANET
AURORA ENERGY
COMMTEL
ENDAVOR
ENERGEX
ERGON
FRESNEL NETWORKS
SNOWY HYDRO
SP AUSNET
TRANSGRID
TRANSPOWER NZ
GDH
WESTERN POWER
POWER COR
ELECTROBRAS
Power Grid Corperation of Bangladesh
Bhutan Power Corporation Limited
Sikkim power
GETCO
MSETCL
PTCUL
TSECL
Nokia Siemens Networks
Nokia Siemens Networks (2)
ESKOM
DNV KEMA

ANSWERS FROM:
Argentina
Denmark
Thailand
Brazil
United Kingdom
Portugal
Denmark
Switzerland
Belgium
United Kingdom
Switzerland
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
Australia
New Zealand
Australia
Australia
Australia
Brazil
Bangladesh
India
India
India
India
India
India
Finland
Finland
South Africa
The Netherlands

IBERDROLA
FINGRID
ISREAL ELECTRIC CORP
RUGGEDCOM
STATKRAFT
STATNETT
SO-UPS
ELECTRIC RESEARCH INSTITUTE
NARI RELAYS
RTE
AGDER ENERGI
EGAT
ZTE
National Grid Saudi Aribia
HUAWEI
VERBUND
Elektroprenos BiH
Elektroprivreda BiH
Hidroelektrane na Trebisnjici
ABB CHINA(1)
ABB CHINA(2)
ABB CHINA(3)
CFE Transmission Yucatan Peninsula
CFE Transmnission north
CFE Transmission north-east
CFE Transmission central
Tata Power Co. Ltd
EEPCO
SaskPower
Altalink Management
Unknown
PSE Operator
Hokkaido Electric Power Company
Hokuriku Electric Power Company
Tokyo Electric Power Company
Chubu Electric Power Company
Chugoku Electric Power Company
Shikoku Electric Power Company
Kyushu Electric Power Company
NEC Corporation
Liandon

Spain
Finland
Israel
Germany
Norway
Norway
Russia
Mexico
China
France
Norway
Thailand
Thailand
Suadi Arabia
Thailand
Switzerland
Bosnia and Herzegovina
Bosnia and Herzegovina
Bosnia and Herzegovina
China
China
China
Mexico
Mexico
Mexico
Mexico
India
Ethiopia
Canada
Canada
China
Poland
Japan
Japan
Japan
Japan
Japan
Japan
Japan
Japan
The Netherlands

9|Page

Figure 1 Companies represented in the survey responses


Figure 2 shows the different countries from which responses were received and, per country, the
percentage that the country contributed to the total results.

Figure 2 Countries represented in the survey responses


The vast majority of the respondents work for Transmission-(49%) and/or Distribution-(28%) System
Operators.

10 | P a g e

2.1.1 Substation applications using IP

83%
81%

Substation applications using IP:

2012

71%
67%
59%

2010

66%

64%
61%
64%

57%

64%

57%

52%
49% 49% 51%

56%
49%
44%

34%

24%

34%

34%
31%

28%
22%

12%

19%
12%
7%

Figure 3 Trends in substation applications using IP


While many of the results from the two surveys are very similar, there are a few differences worth
noting. The use of Wireless LAN access has significantly increased since the last survey. There also seems
to be an increase in the use of IP for:
-

Building control
Telephony system
Substation RTU to SCADA system

The comparison indicates that there is a decreasing use of IP for:


-

Remote IP access to substation assets


Inter control center connections
Time synchronization

While this could mean that, since 2010, IP communication for these applications has been replaced by
non-IP communication, its more likely that the differences in these results come from the difference in
respondents.
11 | P a g e

2.1.2 Main operational challenges using IP as a communication transport solutions over


optical network

Operational challenges:
86%
82%
2010

66%

2012
59%

58%

58% 57%

46%
41%

41%

39%

34%
27%

23%

Figure 4 Trends in operational challenges


Again, the results of the new survey are very similar to the results of the previous survey. In the survey
held for this technical brochure, fewer of the participants indicated that they see operational challenges
in:
-

Ruggedness
Scope of responsibility

Compared to 2010, more participants indicated that they see Quality of Service (QoS) as an
operational challenge.

12 | P a g e

2.1.3 Main psychological barriers with using IP protocols in Substation applications

Psychological barriers:

85%
77%

2010
2012

66%
57%
53%
47%

48%

32%
29%
23%

10% 11%

Figure 5 Trends in psychological barriers


The results for both surveys are very similar. The most notable difference in psychological barriers is that
the participants in the 2012 survey seem to be more concerned about the Quality of Service (QoS) of IP
Protocols

13 | P a g e

2.1.4 Are all applications compatible with IP at the moment?

All applications IP compatible?

2010
79%

21%

2012
76%

18%

Yes

No

Figure 6 Trends in current IP compatibility


There do not seem to be notable differences between the results of the 2010 and the 2012 surveys.
2.1.5 Prediction to migration of all operational communications into IP

All operational traffic through IP?

2010

60%

2012

36% 36%
25%

13%
8%
5%

2%

2%
0%

11%
2%

14 | P a g e

Figure 7 Trends in estimation of all operational traffic through IP


Compared to 2010, participants seem much more convinced that they will be using IP based
communication in the foreseeable future. Only 3% of the 2012 survey participants indicated that they
believe IP networks will never host all of their operational traffic. This is a significant decrease compared
to the 11% of the 2010 survey participants.
2.1.6 How to deal with legacy protocols and equipment?

How to deal with legacy protocols and


equipment?
2010
2012
64%
58%

55%

43%
41%

40%

17%
12%

17%
14%

17%
10%

Figure 8 Trends in how to deal with legacy equipment and protocols


While the popularity for each of the different options is comparable for both surveys, participants of the
2012 survey seem to be more selective.

15 | P a g e

2.1.7 Requisites and concerns for the telecommunication network

Requisites and concerns


2010
2012

69%

66%

63%

54%

34%

39%
30%
23%

Existence of
Suitable
Suitable IP
Identification of
physical
transport
addressing space Applications
infrastructure technology (PDH, and scheme (IPv4
(fibre, copper, SDH, Ethernet,
/ IPv6)
wireless, etc)
etc.)

Figure 9 Trends in requisites and concerns


The results for the different surveys are again comparable. The percentages in Figure 9 for each option
are lower in the 2012 survey then they were in the 2010 survey, creating the impression that the
participants of the 2012 survey have fewer requisites and concerns. FIGURE 9 however does not show
option Suitable substation applications, because this option was only available in the 2012 survey.
68% of the participants of the 2012 survey chose this option.

16 | P a g e

2.1.8 Most promising technologies to provide secure and reliable transport communications
in the access network?

Most promising secure/relaible


technology access network
2010
2012
46%
41%
39%
35%

14%

13%

MPLS (IP over Ethernet/IP Ethernet/IP


fibre)
over SDH over DWDM

Figure 10 Trends in most promising access network technologies


While similar, the scope of this question differs between the two surveys. For the 2010 survey, the
question was: Which underlying IP technology is the most promising to provide secure reliable IP
communications?, while the scope for the D2.35 survey was limited to access networks. Nevertheless,
MPLS and Ethernet/IP over SDH seem to be popular in both surveys.

17 | P a g e

2.1.9 Most promising technologies to provide secure and reliable transport com munications
in the core network?

Most promising secure/relaible


technology core network
2010
2012
52%

51%
45%

39%
35%

14%

MPLS (IP over Ethernet/IP Ethernet/IP


fibre)
over SDH over DWDM

Figure 11 Trends in most promising core network technologies


Again, the scope of this question in the 2010 survey differed from the scope in the 2012 survey. For the
2010 survey, the question was: Which underlying IP technology is the most promising to provide secure
reliable IP communications?,

18 | P a g e

2.1.10 Should the IP network be reserved for operational service or also be used for
corporate services?

Seperate operational services from


corporate services?

2010
2012

58%
54%
48%

47%

5%
2%
One network,
no separation

Separate
network
(physical)

Separate
network
(virtual/logical)

Figure 12 Trends in network separation


There does not seem to be any significant difference between the results for the 2010 survey and the
results of the 2012 survey.

2.1.11 Survey Conclusions


Many utilities indicate they have concerns and are facing challenges (2.1.2 Main operational challenges
using IP as a communication transport solutions over optical network) in implementing and maintaining an
IP only network. Still, three out of four utilities indicated that they expect to migrate all their operational
traffic to IP within the next 10 years (2.1.5 Prediction to migration of all operational communications into
IP). Chapter 3 explains why utilities are migrating to IP despite their concerns and the challenges they
face and provides guidelines on what to consider when migrating to IP.

19 | P a g e

2.2 List of Services


This section outlines, at a functional level, the service requirements of a typical EPU. Over twenty
different systems are identified as supporting operational infrastructure, with many of these systems
critical to the operation of the electricity network. The integration or the support for the services listed
here will have to be considered by the utilities when they migrate to new network technologies.
Bandwidth provisioned on many EPU telecommunications networks fall into two categories - Operational
services and Group/Corporate services.
Operational services may be as follows:

Teleprotection
Outage Management Systems
Network and Access management
SCADA services
Energy Management Systems
Energy metering
Event recorders
Switchover of services between control centers
Disturbance recorders
Real-time PMU
Polling telemetry operational communications and Black Start telephony
Operational Voice Services
Maintenance & Support
Private Mobile Radio
General site alarms, supervision and surveillance
Time distribution using IEEE 1588 alternative to GPS.
Video services
Physical site security using access control mechanisms.
Smart Metering communications.
IEC 61850 based communications.
Dynamic Line Rating
Weather Monitoring & Lightening Detection

Corporate services may be as follow:

Corporate IP network.
Corporate fixed telephony
Corporate mobile telephony
Corporate voicemail
Corporate Video Conferencing

In a lot of existing deployments, where adequate spare capacity exists and economically advantageous
to do so, non-operational traffic is accommodated on operational networks. Non-operational traffic is
usually a secondary consideration and in general is not a determinant of network build capacity.
20 | P a g e

Many documents from various organizations (e.g. CIGRE, IEC) already include detailed descriptions of
the communication / performance requirements for individual applications. It is not the purpose of this
document to re-write or repeat these requirements. The following is a (non-exhaustive) list of documents
that include such information:

CIGRE D2.23: The use of Ethernet technology in the EPU environment


CIGRE JWG D2B5.30: Line and System Protection using Digital Circuit and Packet
Communications
CIGRE JWG 34/35.11- TB192: Protection using Telecommunications
IEC 61850-90-1: Use of IEC 61850 for the communication between substations
IEC 61850-90-2: Use of IEC 61850 for the communication between substations and control
centers (under preparation)
IEC 61850-90-5: Use of IEC 61850 to transmit synchrophasor information according to IEEE
C37.118
IEC 61850-90-12: Wide Area Engineering Guidelines (under preparation)
o 90-12 compiles a significant amount of information related to applications and
requirements. It is envisaged to become a document which covers a lot of requirements
surrounding IEC 61850

Deriving a single mechanism to accurately classify applications based on the information above is quite
difficult. For this reason the Technical Brochure categorises differing applications into groups which can
be hereafter referred to as traffic types. Traffic types can be classified into 6 areas
Samples of possible classifications are listed below:
1. Very low latency, loss intolerant, sequence and symmetry determinant traffic:
over the same path
a. Differential teleprotection schemes
2. Very low latency, loss intolerant
a. Distance Protection
3. Very low latency, loss tolerant traffic:
a. Time distribution
b. Real-time PMU
c. Inter substation event distribution
4. Low latency sequence determinant traffic:
a. RS485
b. IEC 61870-101 IEC 61870-103
c. DNP3
5. Low latency, loss tolerant traffic
a. Voice
b. Video
c. WAMPAC
6. Latency tolerant, loss tolerant
a. Packet based SCADA protocols
b. File transfer
c. Device management

21 | P a g e

3 GENERAL CONSIDERATIONS
This section provides an overview of the requirements utilities need to consider as part of a WAN
migration strategy, and highlights key services and their performance requirements. Further details of the
technologies and services are explained in Chapter 4.

3.1 Why are Utilities migrating?


Electrical power grids continue to evolve to meet the challenges of standardisation and grid
modernisation in order to enhance existing use cases and applications, while simultaneously enabling new
use cases driven by the concept of the smart grid. The addition of new technologies has the potential to
allow an EPU to realise a more stable, reliable, efficient and visible power grid capable of handling any
type of communication flow.
Electric utilities are among the largest users of privately owned and operated wide area network
infrastructures. These are built on a hybrid mix of fiber optics, microwave, power line carrier, a variety
of licensed and unlicensed wireless incl. Wi-Fi, PDH/SDH, plus more recently packet-based technologies.
Communication flow models have largely followed a one-way power delivery flow from generation to
consumer. As a result most communication networks consist of multiple point-to-point circuits connecting
substations to control centers for SCADA and EMS applications, and point-to-point circuits between
substations for protection applications. Many are built and operated for specific applications or solutions,
making it more challenging to integrate new use cases and operational processes.
The evolution of Smart Grids will result in a large number of intelligent devices being dispersed
throughout the electricity grid. New last mile architectures, protocols and technologies will be required to
service these devices, many of which will have modest bandwidth requirements, but may be located in
areas which traditional networks struggle to reach. New emerging protocols, such as IEC61850, require
sub systems based on IP technologies to operate on a local, regional and national level.
Telecommunications will be an essential enabler for many Smart Grid applications. The requirements of
Smart Grid telecommunications will be met by a combination of traditional networks and media
supplemented by emerging last mile technologies, many of which are as yet not fully matured. It is
nonetheless almost certain based on the distribution of physical electrical assets that last mile technologies
will be wireless based with a greater or lesser role for transmission/distribution line communications. Cost,
reliability, criticality, capability, security and performance will be the primary determinant of the
technologies selected.
The introduction of packet has mostly been restricted to IT, while OT communications have largely
remained on traditional TDM networks. The industry is experiencing a steady uptake in TDM to packet
migration due to a number of reasons including end of life of many TDM solutions, emerging standards
such as IEC 61850, interoperability and the use of IP to future-proof grid communications. Utilities are
beginning to consolidate networks on a common infrastructure as part of ongoing refresh cycles, to
minimise operational expenditure, and to introduce new use cases.
Newer, advanced multipoint use cases, for example, listed below are also driving the need for more
bandwidth, multipoint connectivity, forcing us away from the model depicted in FIGURE 13, towards that
described in FIGURE 14. This is the case for many traffics other than SCADA also.

22 | P a g e

Wide Area Measurements


Synchro-check
Adaptive relaying
Situational awareness
State Estimation and on-line security assessment
Wide Area Controls - Special Protection Schemes
Wide Area Controls - Predictive Dynamic Stability Maintaining System
Wide Area Controls - Under Voltage Load Shedding
Wide Area Controls - Phenomenon assumption type WAMPAC

Figure 13 Traditional Grid Communications Architecture

Figure 14 Tiered approach for New grid applications


The result is that new applications are driving transport models, transport models dictate the WAN design and
technology choices.
There are a number of drivers to consider as part of a grid WAN migration process. Economic drivers include
cash optimisation and margin control, cost effectiveness in grid optimization, OPEX reduction and the opportunity
to influence costs of investments for the future. Technology drivers include growing obsolescence of current
communication infrastructure due to technology constraints in current platforms and the location of equipment in
product life cycles, grid flexibility and scalability, and grid monitoring, control and protection. In addition
increasing regulation (e.g. EU 20/20/20 ad M/490 mandates and NERC-CIP for North America) and
standardization efforts such as IEC 61850 and IEC 62351, mean that utilities are looking at migration in new
ways carefully considering all implications
This drives the need for an open network architecture based on open standards, for the implementation of a
multiservice network which accommodates multi-vendors, tailored to meet the requirements for all
communications services and future ones, with the highest levels of quality, reliability, scalability and costeffectiveness.
TDM migration is one of the biggest changes in the electricity network and while many newer applications and
use cases are demanding flexibility that TDM does not have, existing applications such as Teleprotection
schemes require that packet based transport mechanisms need substantial testing over time to ensure they meet
the performance requirements of legacy, traditional and newer services.
23 | P a g e

Figure 15 shows a high level overview of the state of migration. Traditionally most WANs were based on TDM
technologies. Over recent years packet technologies have emerged supporting some operational and IT
applications. These have either been a combination of packet over TDM, or TDM over packet. There is a
gradual shift in the industry towards a hybrid or all packet network architecture.

Figure 15 Shift from TDM to packet

Why are we seeing this transition? The reality is that the communications network is evolving due to changing
communications requirements such as precision timing and IEC 61850, increased network functionality including
enhanced QoS features and multicast support, and the addition of multiservice capabilities like workforce
enablement and physical access solutions.

3.2 Migration considerations


There are a number of decision factors to consider and questions to be asked as part of a transition

New technologies require new skillsets and engineering approaches. (one of the operational challenges
according to the survey results). Is the required skill set and mind set available at utilities for a
transition?
How does the organizational structure (with OT and IT responsibilities) influence a migration process?
Substation equipment refresh: What is the time line for refreshing substation equipment to a new
technology? Is this process in progress? Does existing equipment have physical interfaces available to
connect using a new technology such as Ethernet/IP?
Control Centre capability: Does the EPU management system/s support the new technology choice?
What is the time line for this support? Does this require a rebuild and what is the cost for the upgrade?
Is an upgrade of existing substation equipment possible, or does it need to be replaced? Cost of
substation equipment replacement/upgrade: If no refresh is taking place what is the cost of upgrading
equipment to enable adequate interfaces? Can this be justified and is it giving additional functionality?
What is the disruptive risk to the overall electrical network and overall electrical supply due to this
technology change?

24 | P a g e

Additional cost to integrate legacy applications which cannot be converted to new technology: What is
the additional cost to include TDM and traditional SCADA interfaces into the a newer network? Will this
include legacy interface support, tunneling and translation of traditional SCADA traffic? Can this cost be
justified as the additional capabilities may not be required in 5-10 years?
Cost to maintain existing TDM based infrastructure: What is the associated cost to maintain the existing
TDM/SDH infrastructure to allow the migration of the legacy devices? Is this infrastructure still
supported? What are the security and availability risks in maintaining this Infrastructure?
What is the cost of not migrating? What applications and use cases cannot be deployed? What is the
cost of supporting two or more infrastructures?

In addition there are a number of network influencing factors. The TDM infrastructure, the business and functional
requirements, existing and future communication requirements, and existing infrastructure. Reviewing the
questions and influencing factors following a process such as that outlined in Figure 16, and taking into
consideration the guidelines already documented in the Technical Brochure from D2.28, the outcome will be the
required communications infrastructure.

Figure 16 Migration decision review process

25 | P a g e

An architectural approach, consistent with industry and standards organisations (e.g. NIST, IEC, EPRI, CENELEC),
networking companies, and also power vendors is essential to understand the interdependencies and
communications between systems that exist, but equally important to provide a seamless platform for future use
case integration. An architectural process defines a set of artifacts required to describe a system so that it can
be reproduced and maintained over its lifespan. These artifacts provide components, structure,
interdependencies, and the guidelines determining the design and evolution over time. This is important as each
EPU may have subtly or very different requirements and WAN design will only reflect the use cases and
services an EPU wishes to deploy. All of this should be captured using relevant architecture frameworks such as
TOGAF or equivalent.

3.2.1 Timing
Substation automation and system control are mission-critical processes and electric power utilities must
synchronize applications and use cases across large-scale distributed power grid substation networks to ensure
grid stability. Precise timing is used to improve reliability, better understand operations of the power system,
predict and prevent local and system-wide faults, for testing and verification, and to reduce costs.
Refer to ANNEX 4 Network Timing for more details.

3.2.4 Communication Requirements Decomposition


When designing any communications network it is essential to consider the various elements that make up
communications for an application or service. The communications decomposition table below gives a
good starting point for areas to consider and build into the design process.

Figure 17 Communications decomposition table for packet oriented traffic

26 | P a g e

3.2.5 Security
Electrical power grids continue to evolve to meet the requirements of standardization efforts and grid
modernization in order to enhance existing use cases [1] and applications, while simultaneously enabling
new use cases driven by the concept of the smart grid. In addition there has been a surge in the number
of non-power applications introduced across the grid, as well as links between networks inside an EPU,
and links to external networks. The addition of new technologies allows an EPU to realize a more stable,
reliable, efficient and visible power grid capable of handling any type of communication flow.
Through the integration of new systems, applications, and technologies to facilitate new use cases across
network infrastructure (sometimes common), an EPUs network becomes multi-service. However with the
introduction of any new technology there is an increased risk of security threats which must also be taken
into consideration.
The highly connected nature of a smart grid has the potential to provide unauthorized users with a
greater opportunity to identify and exploit vulnerabilities to attack the electricity grid. There are many
examples of attacks on networks worldwide and constant vigilance is necessary to ensure that adequate
protective mechanisms are used to ensure the electricity grid is not compromised.
While designing and implementing operational networks it will be necessary to comprehensively address
cyber security issues. A defense in depth (security measures deployed at various physical and
conceptual layers of a deployment) approach should be adopted throughout the deployment. Security
capabilities must therefore be layered such that defense mechanisms have multiple points to detect and
mitigate breaches for all service types that an EPU wishes to deploy. These capabilities should be
integral to all segments of the grid infrastructure and address the full set of logical functional
requirements, including:

Physical security
Identity and access control policies
Hardened network devices and systems
Threat defense
Data protection for transmission, distribution and storage
Real-time monitoring, management, and correlation

Secureness of a technology. All technologies outlined within this document possess specific attributes which
at a basic level reduce or introduce additional risks to cyber security, information separation, data
privacy and data authenticity. All technologies should be assessed based upon these criterion to
evaluate suitability and to highlight any additional activities required to ensure a secure introduction of
the technology.
Cyber Security encompasses the overall security landscape from a physical level to a user and
electrical/electronic level.
Information separation is the ability for a technology to transport a specific data stream in a fashion
which can be deemed isolated from dependencies of other data streams.
Other security considerations:.

Access Control: Authentication and authorization of all personnel management tools and physical
devices
Authentication, Authorization and Accounting for data and devices
Data Confidentiality and Privacy
Securely scalable
Tools: Zones, Segmentation, Logging
27 | P a g e

Tools: Tamper-resistant design, authenticity and integrity of hardware and software


Integrity of Platforms and Devices: Secure devices over the entire life-cycle

3.2.6 Communication Network Management


A unified security and network management architecture will be based on a set of applications
delivering the Fault, Configuration, Accounting, Performance and Security (FCAPS) functions as defined
by the OSI Network Management reference model.
A key challenge is the definition of unified management. It is unrealistic to expect that a single
application can provide all functions and services required when managing such diversity of equipment
and communications technologies. More recently applications have been able to manage multi-vendor
and multi-technology communications networks, but there is no guarantee to cover all equipment,
communication types, and future requirements. It is more realistic defining a set of requirements and
features that enables network management and power grid applications to interoperate and exchange
information, such as:

Network and security management and power grid management applications must be able to
communicate over TDM, IPv4 and IPv6 network layers over the expected lifetime of the
deployment.
Role Based Access Control for administrators and operators must be able to get users
authentication from corporate directory services, thus showing interaction between IT and OT
networks.
Network and security management and power grid management solutions applicable to use
cases requiring large scalability, must be able to work in a clustering and load-balancer
networking environment. It allows growing the solution over time without reconsidering the
architecture.
Data flow and API (Application Programing Interface) must be identified, specified and
documented for communications between OT and NMS applications.
An unified Geographic Information System (GIS) solution should be shared between applications
Recommended database (DB) properties should be defined when storing all operational state,
device configuration, network event alarm, performance metric, etc.
Recommended Historian properties should be defined to store data on a long period of time as
required by regulations and operations.
Standard protocols, including public vendors extensions, should be recommended to facilitate
management data exchanges between a device and multiple management applications.
Remote Operations Management capabilities should provide utilities asset management functions
with real-time communications, improved availability of information, and enhanced visual
displays, all of which enable the asset planners and field workers to perform their work with
improved efficiency

Benefits
A powerful Network Management System enables both condition-based and predictive maintenance
strategies so the EPU does not have to rely upon reacting to failures after they happen and scheduling
inspections, which normally have to take place even if there are no equipment health issues.
In addition, asset data can be used as input for grid state determination and remote power quality
monitoring.
These benefits also contribute to improved System Reliability
28 | P a g e

Increase Service Restoration Speed: Improvements in the speed and availability of information in
the field allows technicians to diagnose and repair assets more quickly, resulting in a more rapid
restoration of service.

Increase Reliability Metrics: Increases in the speed and efficiency of delivering service, and
increased service restoration speeds will result in higher reliability metrics.

The technologies discussed within this document will be responsible for supporting critical electrical assets,
systems and services within an EPU. To ensure that adequate clarity and control of a technology,
dynamic, scalable and intuitive management systems need to be put in place. Management systems
should be able to accurately present, configure, report and alert on network state in a close to real-time
fashion while ensuring compliance with change control and management methodologies.

3.2.7 IPv4 and IPv6 co-existence


Even if in future networks the core network devices may be running IPv4, the networks may need to offer
IPv6 support for applications.. Co-existence of these protocols is a consideration as part of any migration
or accommodation strategy involving Ethernet/IP.

3.2.8 Segmentation and Virtualization


With the growing adoption of packet based technologies to converge all data, voice and video traffic
from both an operational and IT perspective, there has been a growing need for logical segregation
rather than physical. The IEEE and IETF have standardized Layer-2 and Layer-3 mechanisms, enabling
links to be shared by all data types which have been in operation for many years.
A segmented or virtual network can be seen as decoupled from the physical network but it doesnt
change its basic characteristics. It represents an abstraction that requires proper design, deployment,
security and management to be successfully operated.
There is a broad range of mechanisms to achieve segmentation and virtualization, in TDM, and
Ethernet/IP environments. Any design should match the associated applications requirements in regards
of bandwidth, prioritization, security, latency and so on.
Example applications could be Serial SCADA over IP, Layer-2 Ethernet or Serial circuit and segregation
of operational and corporate data via MPLS VRFs or a separate VC container in SDH.
Whatever the selected mechanisms, the Virtualized Network building blocks must be designed in
considering the following topics:
Segmentation versus virtualization
Separate vs virtually separate
Network management Network management applications must be able to retrieve data from
all Virtualized Networks
Capacity estimate when designing and operating the network, capacity estimate allows proper
selection of physical and data link layers in order to provide enough bandwidth and optimized
latency particularly important for Teleprotection for all applications. Capacity estimate of the
physical infrastructure must include the network protocol overhead related to the segmentation
and virtualization mechanisms. This must include the performance and scalability impacts of the
mechanisms on the devices at the various network places.

29 | P a g e

3.2.9 Scalability
The migration of EPU Services and applications to new communication technologies will take place over a
long period. While future requirements are captured where possible, even more demands and
requirements will be presented post deployment. For this reason it is necessary that all technologies are
assessed based on the ability to meet:

future needs of applications


future deployment models
future capacities
future geographical expansion

3.2.10 Availability
The availability concept is related to time. That is, availability can be defined as the time in which the
system is reliable. In Optical systems, highest availability is achieved by means of certain degree of
Redundancy. That redundancy must cover not only the electronic devices such are transmitters, receivers
or amplifiers, also must cover the fibre path implementing protected routes between nodes.
Availability is also strongly influenced by the network recovery time which depends on the failure
detection time and the availability of a protection path.

30 | P a g e

4 ASSESMENT OF TECHNOLOGIES
This section discusses a number of technologies which are currently already in use in EPU networks or
may be considered for future extensions of the network:

Synchronous Digital Hierarchy (SDH)


Dense Wavelength Division Multiplexing/Coarse Wavelength Division Multiplexing
(CWDM/DWDM)
Optical Transport Network (OTN)
Dynamic Internet Protocol Multiprotocol Label Switching (IP/MPLS)
Static Multiprotocol Label Switching (MPLS-TP)
Ethernet

This is not a detailed technology tutorial, but has a focus on specific characteristics which are relevant for
EPU applications. Therefore for each technology the following is discussed:

Life Cycle
Scalability
Capability to support EPU Services
Ease of Implementation & Operation
Interoperability (other services/technologies)
Redundancy, Availability, Reliability
Quality of Service (QoS)
Network Management System and Overall Manageability (NMS)

31 | P a g e

4.1 Synchronous Digital Hierarchy (SDH)


Life Cycle
SDH is a mature technology. While SDH is a Legacy technology for Public Telephony, it continues
to be the predominant technology for utilities. Thus, it is most likely that SDH prolongs its existence
for years to come. This is evident within the survey.
SDH is a standardized transmission technology that allows multiple digital bit transfers over
optical and electrical links. It was established by a set of ITU-T standards in 1989. When
introduced, one of the main concerns was to assure compatibility with existing PDH services
without synchronization problems. This was the reason to incorporate standard PDH interfaces.
Right now SDH is still the transmission infrastructure of a large number of telecommunication
operators and utilities. The most suitable media for SDH is fiber optic, nevertheless electrical SDH
line interfaces as well as microwave radio are also common.
SDH is a widely adopted technology, it is possible to build SDH equipment using general
purpose components. This reduces the risk and the effect of life cycle decisions within the
vendor/manufacturing market.

Scalability
Huge networks have been built with SDH technology and several European Public Telecom
Operators still use SDH as the backbone network. Rates of 10 Gbps (STM-64) are commonly
installed and 40 Gbps (STM-256) rates are beginning to appear in the market. SDH is commonly
used to build extensive wide area networks. In the following table most common SDH rates are
shown. That said, SDH still provides the ability to interface and connect networks and devices at
very low data rates (64kbps) in t seemless fashion.

Synchronous
Transport Module

Hierachy

(Mbps)

Interface
electrical/optical

STM-1

155,520

G.703/G.957

STM-4

622,020

---/G.957

STM-16

16

2 488,320

---/G.957

STM-64

64

9 953,280

---/G.957

Figure 18 SDH Capacity Comparison


32 | P a g e

Capability to support EPU Services


SDH supports a variety of different payload types enabling transport of TDM classical services
such as E1 Circuits. SDH network can tunnel point to point Ethernet connections (Ethernet over SDH)
and SDH equipment can provide vendor specific Ethernet capabilities - even multipoint
connections as an overlay service.
A key element in SDH networks is the Add and Drop Multiplexer (ADM) which is the equipment
that provides basic capacity for traffic insert and cross-connections.

East

West
STM-1,STM-4,STM-16,STM-64

STM-1,STM-4,STM-16,STM-64

Tributaries
Electrical
intefaces

2M,34M,45M,140M

Optical
interfaces

STM-1,STM-4,STM-16

Figure 19 SDH Add-Drop-Multiplexer


This capability allows the efficient transport and support supports of legacy and TDM services
such as tele control and protections.

Ease of Implementation and Operation


From an EPU perspective, SDHs concepts are familiar and known. They are commonly regarded
as simpler than data-packet based technologies. This is evident in the questionnaire and in
section 3 skillset Moreover, with a lot of matured equipment deployed, there are a lot of
technicians familiar with the operation and maintenance of SDH and its theoretical concepts. From
this point of view it is relatively simple to establish maintenance strategies due to the fact that
know-how is widely available.
The fact that SDH is a mature technology that has been very popular ensures also that a lot of
spare parts are available.
Due to its TDM characteristics, SDH lacks modern data technologies flexibility, and traffic
engineering is hard to optimize in big networks generating the fact that some traffic capacity is
wasted in meshed networks.

Interoperability (other services/technologies)


In comparison with other newer technologies, interoperability with legacy interfaces is a strong
33 | P a g e

point. Especially in utilities, SDH in combination with PDH offers interfaces for legacy electrical
protection and control devices that can hardly be obtained with other technologies in a native
way. SDH provides also a deterministic behavior, which is widely appreciated by traffic
engineers.
Recent developments have been made in order to accommodate SDH with data based network,
such as Ethernet over SDH. SDH works efficiently with WDM.

Redundancy, Availability and Reliability


SDH incorporates native protection and resilience mechanisms such as SNCP or MS-SPRING that
can provide a 1+1 and N:1 traffic protection. Restoration times of around 50 ms can be
achieved.
Overall concept for SDH traffic redundancy is to offer an alternative path for connecting origin
and destination in a telecommunication circuit. So, in a typical ring configuration two physical
different paths are provided. In the following figure an example of SNCP protection is
provided.

DXC

ADM

SubNetwork Connection
Protection: Simultaneous
tranmission by two paths
working

protection

ADM

Figure 20 SDH Sub-Network Connection Protection


* DXC in Figure 20 refers to a Digital Cross Connect
Card protection redundancy schemes are also available. This architecture ensures that in the case
of one card failing, the remaining cards can avoid service interruption and single points of failure
in equipment. In general, if adequate traffic redundancy is provided, SDH networks have high
levels of availability of telecommunication services.
34 | P a g e

Quality of Service (QoS)


As a TDM transmission technology SDH supports only reserved bandwidth for each circuit, this
mechanism insures a guaranteed QoS especially for mission critical applications, but SDH is not
optimized in case of multi-services transport. These features are inherently given within SDH
standards. The ability to classify traffic within Ethernet circuits is not common across all vendors.

Network Management System and Overall Manageability (NMS)


SDH management technology is very well understood, providing inbuilt OAM and monitoring
functionalities. SDH management provides alarms supervision and remote provision of services.
SDH networks management platforms offer automated management functions. These platforms
provide the data that is needed in the day-to-day running of a telecom network and have the
possibility of issuing commands to the network infrastructure to activate new services and detect
and correct network faults.
One of the drawbacks is that SDH management platforms are proprietary. It is possible to
connect SDH equipment from different vendors, nevertheless is neither possible to manage a
network element from other vendor platform nor feasible to establish automatically a point to
point SDH circuit that runs on different vendors regions.

35 | P a g e

4.2 Dense Wavelength Division Multiplexing & Coarse Wavelength


Division Multiplexing (DWDM & CWDM)
Life Cycle
It comes as no surprise that traffic growth is the underlying driver for 100 Gigabit (Gbit) WDM
network transport. This was the case with the transition from 2.5G to 10G and from 10G to 40G
and now 100G optical channel rates.
WDM technology is a physical technology allowing transporting any protocol with variety of
bandwidth granularity till 100 Gbps (or more in next future), this capacity of WDM technology
will allow covering actual and future data stream needs.
Currently Wavelength-division multiplexing WDM transport technology is assuring carrier services
for both TDM technology and Ethernet technology. In essence WDM is just a support technology,
an infrastructure for the other technologies.
More recently WDM is a technology that allows the coexistence of various transmission systems
sharing the same fibre pair.
SFP optical interfaces are one of the major contributors to the popularity of the WDM because
its intrinsic easiest way to change the transmitting wavelength simply changing the plug-in
module. In fact actually the major part of equipment manufacturers offers a wide variety of SFP
possibilities on its optical interface.
In the last years the extensive use of the OADM devices, which allows some degree of optical
routing, in the Metro networks increases the popularity of that technology. In fact, at the present,
all the Optical Transmission Vendors have WDM devices on their portfolio.

Scalability
There are two types of WDM architectures: CWDM, and DWDM:
A CWDM system typically provides 18 Channel-Wavelength, separated by 20nm, from 1260nm
to 1630nm.
DWDM is a technology used for a very high capacity links (ITU-T G.694.1). DWDM systems,
mainly used by Public Telecom Operators, can multiplex from 32 to more than 100 channels in
the range of 1530 1624 nm.

36 | P a g e

.
Figure 21 Optical spectra
in fiber
The number of channels, the channel selection (center frequency), and the frequency width of each
channel, as well as the channel separation, are important parameters in WDM system design.
With a channel spacing of 1nm (or below), DWDM operation requires extremely precise optical
sources (laser temperature stabilization) resulting in significantly higher complexity and cost.
WDM technologies can accommodate any bit rate providing necessary bandwidth: SDH (STM1,
STM4, ), Ethernet (1Gbit/s, 10 Gbit/s)
WDM technology has brought more than an order-of-magnitude increase in the amount of
bandwidth that can be transported over fiber. Early implementations were point-to-point only.
However, while transport networks may be thought of as the roads of the network, the
intersection points are also critical.
Natively, WDM technology provides a point to point links, however it is possible to have different
type of architectures (ring, mesh ) by mixing WDM with other protocols (Layer 2, Layer 2.5,
Layer 3).
Todays Optical Ethernet networks are much more dynamic than in the past and demand greater
flexibility. In order to provide the necessary flexibility, Reconfigurable Optical Add Drop
Multiplexers (ROADMs) were developed. They allow operators to access any wavelength at any
node at any time replicating the operational simplicity and flexibility of SONET/SDH networks
at the wavelength level.

Figure 22 examples of Add-Drop-Multiplexer

37 | P a g e

State of the art ROADM technology and the use of tunable lasers and filters, allows operators to
not only drop any wavelength at any node and any time, but to also send any wavelengths in
any direction (directionless) using any available port on the network node (colorless).
This so-called triple-A architecture (AAA any wavelength, any node, any time) is fully flexible
and non-blocking. It requires very little technical skill from the operational staff and eliminates the
need for meticulous pre-planning.
Triple-A ROADMs are the foundation of a fully automated optical network, and they allow for
intelligent interworking using the two layer architecture concept, which despite other technologies,
have the facility to deploy a network architecture with two stacked switching layers.

Figure 23 Reconfigurable Optical Add Drop Multiplexer

LONG DISTANCE TRANSMISSION


In the transmitting side the laser diodes have its output power limited by the signaling speed and,
for the same physical reasons, in the receiver side, the quantum energy necessary to reach the
detection threshold increases following the signaling speed.

Power
[dBm]

Power
[mW]
10 mW

+10

TX/Laser Diodes

1 mW

0
- 10

TX/LED

- 20
- 30

1 W
in RX
in RX
iodes
d
o
t
iodes
o
d
h
o
p
t
r
o
o
f
e ph
r limit
lanch
Lowe
r ava
o
f
it
r lim
Non detectable signal in RX
Lowe

1 nW
1M

10 M

100 M

1G

10 G

- 40
- 50
- 60

100 G

Bit Rate [Bit/s]

Figure 24 Working area for light source and receiver


38 | P a g e

For example for 10 Gbit/s bite rate, standard photodiode sensitivity limit is higher than -35 dBm
(Red point in the figure above) because the limited energy of the incoming optical pulses.
Obviously these characteristics imply that the optical amplification becomes necessary for high
speed transmission. An alternative solution is to spread the high speed stream in some medium /
low streams and use passive WDM to transmit simultaneously those streams in a single fiber pair

Capability to support EPU Services


WDM is a method of combining multiple signals on laser beams at various infrared (IR)
wavelengths for transmission along fiber optic media. Each laser is modulated by an independent
set of signals. Wavelength-sensitive filters are used at the receiving end to separate the signals.

Figure 25 WDM Multiplexing


This is an attractive multiplexing technique that allows:

Does not directly support termination of many legacy or EPU specific


connections (RS-232/485, E1 etc)
High bit rate without high speed electronics or modulation
Mixing legacy and new network technologies on a single optical fibre
Very useful for upgrades to installed fibers
Suitable for Mission Critical Networks
Extensive use of passive components
Loss, crosstalk and non-linear effects are potential problems.

Ease of Implementation and Operation


CWDM technologies offer rapid deployments with relative ease in comparison to DWDM
solutions. However, many CWDM solutions relay upon subsequent layers of technology to enable
vision on the network. For this reason, ongoing operations and maintenance of a CWDM
solution can become laborious as a network scales.
Some CDWM and most DWDM systems include integrated, network management programs that
are designed to work in conjunction with other operations support systems (OSSs) and are
compliant with the standards the International Telecommunication Union (ITU) has established for
Telecommunications Management Network (TMN). Current systems utilize an optical service
39 | P a g e

channel that is independent of the working channels of the WDM product to create a standards
based data communications network that allows service providers to remotely monitor and control
system performance and use. Meeting ITU standards and utilizing a Q3 interfaces ensure that
end users retain high Operations, Administration, Maintenance, and Provisioning (OAM&P)
service. Most systems use SNMP as a standard but in-depth configurations of devices may not be
achievable with open standards.

Interoperability (other services/technologies)


WDM networks provide purely optical transport for different transmission technologies and
different transport protocols.
The WDM technology is designed to allow the coexistence of diverse technologies over the same
media. As a carrier system many wavelengths can be implemented inside a fibre with total
independence, carrying each one the appropriate services.
This ensures protocol and format transparency in the network. A major advantage is coexistence
with several protocols (SDH, Ethernet ), for example SDH/WDM, Ethernet/WDM, IP/WDM,
MPLS/WDM

40 | P a g e

Redundancy, Availability and Reliability


In terms of Availability and Reliability, there are some differences between the WDM
implementations.
With a channel spacing of 20 nm, CWDM systems tolerate the temperature drift over the whole
industrial temperature range of the laser emitter without leaving their allocated channel and
without use of special cooling system.
The strict frequency stability requirement to match with the DWDM channelling is achieved
keeping the laser cavity at constant temperature and limiting the emitter power output. This
implies that temperature needs to be kept constant and lowers the MTBF of a system in
comparison with CWDM
CWDM ruggedness, in addition to considerably lower cost, constitute significant advantages of
this technology in the EPU communication network where wavelength multiplexing is not often used
for attaining maximum bandwidth but rather for separating networks.
Circuit redundancy and failover are not characteristics associated with CWDM/DWDM but are
often accomplished via technologies carried on top of WDM

Quality of Service (QoS)


QoS provided by WDM technology is based on resource reservation (equivalent to Constant Bit
Rate): Each connection has reserved bandwidth (wavelength) without any overflow mechanism.
There are a lot of discussions about the suitability of the transmission of Teleprotection signals
and / or Differential protection over non dedicated transmission systems. Both Synchronous and
Packetized systems present some degree of inconvenience which are discussed in other parts of
this Brochure.
Direct transmission systems or dedicated point to point equipment. The most suitable media for
mission critical communications is fibre optic. There are methods to deploy multiple services via
separate optical channels: The use of dedicated optical fibre or the use of a dedicated
wavelength. Because optical mixer and receiver filter banks are passive elements the reliability
of the transmission path is not affected by choosing one solution or another.
WDM technology allows a deterministic end to end delay and jitter.

41 | P a g e

Network Management System and Overall Manageability (NMS)


WDM management technology varies a large amount due to the passive nature of some solutions
in comparison to the comprehensive nature of other types of solutions. CWDM devices usually
rely on subsequent technology layers above to verify connectivity and availability. DWDM can
provide extensive optical and circuit based managers providing granular control and vision of
provisioned services.
CWDM versus DWDM
The election of the optical channelling method is a decision which cannot be taken in early stages
of a design. This is mainly because both C and DWDM channels may coexist over the same fibre
and CWDM is recommended for Mission Critical applications because its highly reliable nature in
comparison with DWDM

42 | P a g e

4.3 Optical Transport Network (OTN)


Life Cycle
OTN adoption was initially slow; the first deployments of OTN were in Japan with little interest in
Europe and North American. The reason for this slow adoption is that carriers have invested huge
amounts of capital in the existing SDH/SONET and WDM networks.
Since the mid 2000s, OTN technology has been essentially proposed for point to point optical
links requiring enhanced Forward Error Correction (FEC) capability and higher data rates.
Today, OTN technology is deployed in Telco carriers for several network topologies (ring, mesh
), with the possibility to transport transparently any client traffic offering some additional
features like enhanced OAM, Redundancy & Resiliency.
OTN is supplied by carrier network vendors, mainly the same vendors as SDH. The carrier market
for OTN is envisaged to nearly double from 2012 to 2016.

Scalability
The basic signal architecture of OTN is shown in the FIGURE 26 below:

43 | P a g e

Client Signal

Optical Channel Data Unit


(ODU-k)

Electrical Domain

Optical Channel Payload Unit


(OPU-k)

Optical Channel Transport Unit


(OTU-k)

Optical Multiplex Unit


(OMU)

Optical Domain

Optical Channel
(Och)

Optical Transport Module


(OTM)
Figure 26 OTN Architecture
The client signal is passed through two main domains: Electrical domain and optical domain. The
OPU-k encapsulates the client signal and performs (if needed) the rate justification.
The ODU-k performs the same functions than the Line Overhead in SDH
The OTU-k contains the FEC and performs similar functions as the Section overhead in SDH
The OUT-k is transmitted over the Och using a wavelength; the OMU performs the
multiplexing/de multiplexing of several wavelengths.

44 | P a g e

The standardized bit rates of OTN are:


OTN container

Payload Rate

Supported Clients

ODU-0

1.25Gbps

STM-1/4, GE, FC, others

ODU-1

2.5Gbps

STM-16, 2G FC

ODU-2/ODU-2e

10Gbps

STM-64, 10GE

ODU-3

40Gbps

STM-256, 40GE

ODU-4

100Gbps

100 GE

ODU-Flex

N*1.25Gbps

Sub rate Ethernet or constant


bit rate like
EPON,GPON,CPRI

Figure 27 OTN Capacities


Source: Infonetics June 2012
OTN capacities are far beyond potential EPU OT requirements but arguably does present EPUs
with an opportunity to deploy bandwidth as a service for non-operational uses. (corporate or
commercial services)
Capability to support EPU Services
OTN has been developed by ITU-T standardization (G.709, G.879) as a new generation
technology for signal transport over optical networks. It allows the following functionalities:

Transparent signal transport thanks to the service-agnostic mechanism that maps different
client services in to the same data frame level.
OTN multiplexing bandwidth granularity is one or two orders of magnitude higher than
for SDH technology, thus making it more scalable and supporting higher rate.
It uses DWDM transport system to reach line rates of 40 Gb/s and 100 Gb/s
OTN provides an integrated mechanism for forward error correction (FEC) that allows
greater reach between optical nodes and/or higher bit rates on the same fiber.
The introduction of enhanced OAM channels built on the experience gained from the
existing SDH overhead

These functionalities ensure that adequate capacity and control exist for EP~U activities however
OTN solutions do lack the ability to directly interface with legacy technologies and EPU specific
applications.

45 | P a g e

Ease of Implementation and Operation


Introduction of OTN switching brings major benefits in terms of lower cost and greater
functionality. As networks grow and need to accommodate new types of traffic alongside existing
traffic, it becomes vitally important to fill the available transmission bandwidth efficiently. This is
the area of the market which OTN is designed to be most advantageous to deploy.

Interoperability (other services/technologies)


The OTN supports nearly all circuit switched and packet switched services as well as all constant
and variable bit rate services. The supported services are listed below:
-

SDH/SONET
Ethernet (Electrical & optical)/ VLAN based, port based, MAC learning
IP/VPN
IP-MPLS, MPLS-TP
Ethernet SyncE

Redundancy, Availability and Reliability


To achieve maximum availability in the network, OTN nodes support the following redundancy
and protection schemes:
-

Common module redundancy: Matrix fabric, controller board, power supply, FAN
Tributary and traffic module redundancy: 1+1, 1:1 and 1:n linear protection,
Shared Protection Ring (SPRing) for HO and LO ODU-k

OTN supports enhanced OAM channels that allow the monitoring of the traffic in order to trigger
switching mechanisms for service protection.

Quality of Service (QoS)


OTN QoS is deterministic (like SDH/SONET). In comparison with other technologies, where QoS
determinism is solution specific. These mechanisms insure a guaranteed QoS especially for
mission critical applications. OTN is optimized for multi-services transport.

Network Management System and Overall Manageability (NMS)


Generally, network management of the OTN technology falls under two areas:

OAM

OTN builds on the concept of Tandem Connection Monitoring (TCM) seen in SDH. SDH only allows
one tandem connection, whereas OTN supports 6. The optical channel data unit (ODU) carries
therefore TCM1-6 in the overhead section. Additionally it carries the path monitoring overhead
(PM).
TCM1 is used by the User to monitor the QoS that they see. TCM2 is used by the first operator to
monitor their end-to-end QoS. TCM3 is used by various domains for Intra domain monitoring.
Then TCM4 is used for protection monitoring by the operator.

Interoperability with different technologies (unique platform for several technologies)


46 | P a g e

The concept of the network management system is very much vendor specific.
Multi layer OTN/WDM networks generally require a network management for both the IP and
the transport layer. That said, some vendors have the concept of an unique network management
system for both the transport layer and the IP network.
This will be very helpful for maintenance staff that is used to the circuit switched technology
compared with other IP based technology.

4.4 Dynamic Internet protocol based Multiprotocol Label Switching


(IP/MPLS)
Life Cycle
IP/MPLS is now a mature technology used by many organisations and carriers worldwide.
Together, Ethernet services and IP/MPLS are widely deployed by public telecom providers. The
reason for this, is the technology's ability to reduce cost and improve operational efficiency in
organisations. For these reasons it is envisaged that IP/MPLS will be available as a technology
for a substantial amount of time.
With widely available standards-based implementations by various manufacturers, IP/MPLS
adoption by the EPU is an option for the most applications since the standards and RFCs that
constitute IP/MPLS are mature. It has been field-proven for many years by telecommunications
carriers and more recently the Enterprise, and are well supported by various equipment
manufacturers.

Scalability
Networks consisting of thousands of nodes have been built with IP/MPLS technology. Most Public
Telecom Operators and big Enterprises use IP/MPLS within core and access networks. IP/MPLS is
not directly tied to any transport technology and thus can support any link speeds. 100Gbps links
are commonly installed and support for faster links is expected as they appear. IP/MPLS
networks can also support very slow link speeds.
Support multipoint topologies in addition to traditional point-to-point topologies makes IP/MPLS
very attractive as a potential future technology for EPU use. Many new operational
communication models/protocols/applications are built natively to run in IP networks.
IP/MPLS is supported in wide range of different products from small ruggedized models to
carrier grade multi-chassis devices. This means it can be deployed in various places, hence
addressing a wide variety of EPU use cases.

Capability to support EPU Services (as defined in chapter 2)


IP/MPLS networks can support EPU Services using concepts called Virtual Private Networks
(VPNs), that provide different network services to different needs. VPNs can be partially divided
into native IP services (L3VPNs) and data link layer services (L2VPNs). IP/MPLS supports natively
47 | P a g e

various service topologies (namely point-to-point, multipoint, tree and anycast). Point-to-point
L2VPNs are called pseudo-wires and they support most TDM services.

For a technical summary of IP-MPLS, refer to Cigr Technical Brochure from WG D2.28 Communication Architecture for IP-based Substation Applications.

A major reservation for EPUs of the use of packet-based technologies, in particular IP-MPLS, is
the ability to support the requirements of some of its critical applications, in particular
Teleprotection (see 2.2 List of Services). However, research and testing are ongoing in this area.

Ease of Implementation and Operation


The use of control plane protocols in IP/MPLS comes with additional management complexity.
Additional training on the operations staff is required to troubleshoot and provision IP/MPLS. This,
however, can be alleviated by solutions which provide a Management System which is able to
abstract the complexities of the protocol.
It is therefore important for utilities which are considering IP/MPLS to ensure that the proposed
vendor solution includes a capable Management System that automates the configuration of
MPLS.
IP/MPLS is scalable in that is simple when adding new nodes to an existing MPLS network. The
required provisioning involves only configuration of the new nodes. This, of course becomes more
complex as other features are used such as MPLS-TE.
Expertise in the field of IP/MPLS exist primarily within the enterprise and within public carriers, it
should be noted that said expertise do not align with what would be expected for an EPU.

Interoperability (other services/technologies)


IP/MPLS is most commonly used as a transport protocol to carry IP traffic on Ethernet.
Due to the physical protocol-independence nature of the MPLS label, it can also be carried
directly within other physical and data link layers such as C/DWDM, SONET/SDH, and PDH.
In terms of the types of data encapsulated by IP/MPLS:

TDM circuits (E1/T1, SDH, etc.) can be carried by IP/MPLS but can utilise a lot of bandwidth
Support for layer 2 Ethernet (point-to-point/multi-point) is good
IP multicast over IP/MPLS is still relatively new to utilities
Layer-3 VPN (L3VPN) and Layer-2 VPN (L2VPN)

Some IP/MPLS equipment platforms to date support a substantial amount of EPU specific
hardware interfaces.
Redundancy, Availability and Reliability
IP-MPLS supports high availability through traffic engineering (TE) and Fast Reroute (FRR), with
fault detection in the order of 50ms using mechanisms such as BFD (bidirectional forwarding
detection).
48 | P a g e

The protocols surrounding MPLS-TE and FRR such as RSVP-TE have reached maturity within
enterprise and carrier deployments. Link and node protection via FRR can now be computed by
most manufacturer devices dynamically, easing deployment of the feature.
Protection in IP/MPLS consists of two complementary technologies:

Fast Reroute (FRR), which provides immediate short term recovery upon a failure. FRR has a
deterministic switchover delay.
End-to-end path protection, which provides the operator control of the path after failure, and
has non-deterministic switchover delay, due to additional signalling required. Hence end-toend path protection is used with FRR to provide both short-term recovery to minimise packet
loss immediately upon a failure, and then allow end-to-end path protection to take the time
required to calculate the new desired path.

Quality of Service (QoS)


IP-MPLS QoS is standardized by IETF in a group of RFCs.
QoS in IP/MPLS consists of the following major aspects:

Coarse-grained admission control using RSVP-TE, where signalling is performed between the
LSP tail end towards the head end to determine whether there are sufficient resources to set
up a path.
Per-class admission control and per-hop-behaviour using DiffServ-TE

With the per-class admission control (Diffserv-TE), although it provides more control on QoS, it is
more complex to deploy and requires that all nodes are Diffserv-TE aware.

Network Management System and Overall Manageability (NMS)


Main features supported applicable for Power Utilities

Inbuilt OAM and monitoring


Open NMS capability (e.g. northbound interface)

The IP/MPLS protocol have a limited set of OAM defined. New developments to the MPLS
standard which evolves to MPLS-TP by the efforts of IETF and ITU-T, seek to define OAM features
similar to traditional transport protocols such as SONET and SDH.
The OAM toolset in IP/MPLS is not as well-defined as MPLS-TP, and in comparison, IP/MPLS
cannot provide transport-like OAM functionality. The OAM toolset within IP/MPLS are LSP ping
(to troubleshoot label switched paths) and BFD.

49 | P a g e

4.5 Static Multiprotocol Label Switching Transport Profile (MPLS-TP)


Life Cycle
In 2008 ITU-T and IETF started a Joint Working Committee where MPLS-TP started.
The core elements of the MPLS-TP standard are completed but some work and agreement still
remains to be added. Equipment and software based on the existing standards are available in
the market from a number of vendors. MPLS-TP is in its early stage of its life cycle and the
maturity of solutions/implementations is low.
Scalability
The goals of MPLS-TP are to simplify the transport network to a total packet based transport, to
lower costs, and to improve the economics of transporting data with a simple-to-operate and
resilient end-to-end packet transport network. MPLS-TP is designed for use in very large service
provider networks and will scale for large implementations with high numbers of LSP/PW perport, per-card and per-system and supporting huge numbers of protected LSP with sub 50ms
failover.

Capability to support EPU Services (as defined in chapter 2)


Created with the objective of optimizing and simplifying MPLS for traditional transport
operational models, MPLS-TP is a subset of the MPLS protocol suite that is suitable for use in
packet-based optical transport networks. Some IP/MPLS features that are unnecessary in a
transport context are removed. Extended functionality based on the transport requirements that
are necessary to operate in a similar manner to existing transport technologies has been added.
This incorporates transport-like OAM, resilience and network management operation.
Pseudo wires are used in MPLS-TP to be able to transport layer 1 and layer 2 services like
Ethernet, TDM etc in point-to-point applications. The pseudo wire is a connection tunnel between
the PE routers that emulates a wire that is carrying L1/L2 frames.
MPLS-TP also supports point-to-multipoint and multipoint-to-multipoint services by implementing
Layer 2 VPNs.
FIGURE 28 below gives an architecture overview of MPLS-Transport Profile (MPLS-TP).

50 | P a g e

Figure 28 MPLS-TP Hierarchy


Ease of Implementation and Operation
MPLS-TP is a subset of MPLS with some OAM extensions that will make it quite similar to the
operation of SDH systems. The ease of implementation and operation is expected be similar to a
standard SDH system, but the company may still need a different kind of competency to
support(MPLS).

Interoperability (other services/technologies)


Due to the physical protocol-independence nature of the MPLS label, it can also be carried
directly within other physical and data link layers such as C/DWDM, SONET/SDH, OTN, and
PDH.
In terms of the types of data encapsulated by MPLS-TP:

TDM circuits (E1/T1, SDH, etc) can be carried by MPLS but can utilise a lot of bandwidth
Support for layer 2 Ethernet (point-to-point/multi-point) is good
Layer-3 VPN (L3VPN) and Layer-2 VPN (L2VPN)

MPLS-TP equipment platforms to date support a substantial amount of EPU specific hardware
interfaces.
Some MPLS-TP equipment platforms to date support a substantial amount of EPU specific
hardware interfaces
Redundancy, Availability and Reliability
MPLS-TP is using OAM packets to trigger different protection mechanisms with the possibility to
configure 1:1, 1+1, 1:N and ring protection options. In the 1:1 protection option there is a
working and a protection path between two nodes where the protection switching can be
triggered by:
51 | P a g e

Detected defect condition (AIS-Alarm Indication Signal, LDI-Link Down Indication, and LKRLock Report messages)
Administrative action
Far end request
Server layer defect indication (LOS)
Retrieve timer (wait-to-restore)

FIGURE 29 gives an illustration of a 1:1 protection configuration and in MPLS-TP there is sub-50ms
protection switchover.

Figure 29 Protection configuration


In a 1+1 protected configuration, the standby path will be active, but the PE1 will block the
standby LSP until switchover.
Quality of Service (QoS)
MPLS-TP supports QoS using standard MPLS Differential Services Traffic Engineering (DS-TE)
architecture, which is also used in IP-MPLS. Please refer to the IP-MPLS QoS section for a
description of DS-TE.
Network Management System and Overall Manageability (NMS)
Main features supported applicable for Power Utilities

Inbuilt OAM and monitoring functionalities such as:


o OAM Continuity Check (CC): proactive (BFD Bidirectional Forwarding Detection)
o OAM Continuity Verification (CV): reactive (LSP-Ping and Trace Route messages)
o Alarm Suppression and Fault Indication with AIS (Alarm Indication Signal), RDI
(Remote Down Indication), and Client Fault Indication
Interoperability with different technologies (unique platform for several technologies)
Open NMS capability (e.g. northbound interface)

Operation, Administration, and Management (OAM) have been, and are, the key focus areas of
the MPLS-TP standardization work. The reason for this is that todays TDM transport networks like
SONET/SDH are extensively using this type of OAM functions to be able to meet the
52 | P a g e

requirements of the transport networks, and these functions also need to be implemented in
MPLS-TP. Functions that are being implemented in MPLS-TP are fault detection (connectivity check,
connectivity/path verification), fault localization and performance monitoring (delay and loss
measurement).
One of the goals of MPLS-TP is to make the establishment of the paths in a similar static way as
the TDM systems through a NMS system. Because of this the management of a circuit based and a
packet based network will be quite similar. In addition the MPLS-TP standard also has the option
of a dynamic control plane (Generalized MPLS), and this could be implemented for scaling
reasons and for establishment of more advanced protection functions. MPLS-TP bandwidth can be
reserved, and this must be configured explicitly at each hop (on the Label Switched Path), but this
bandwidth enforcement requires QoS configuration in the network
In the same way as the circuit based transport networks work today, MPLS-TP also needs to
support out-of-band management over a dedicated management network. An MPLS-TP transport
network should function without the implementation of protocols like IP and dynamic routing
protocols, and for that reason an out-of-band management network is needed, so that the NMS
can reach the MPLS-TP equipment.

53 | P a g e

4.7 Provider Backbone Bridging (PBB)


While aware of this technology, it is apparent that PBB has not gained traction in the market as a
transport solution. For this reason, PBB is not considered as a viable option for EPU deployment.

4.8 Ethernet
Life Cycle
It is difficult to mark Ethernets place within its Product Life Cycle. It was first introduced in 1980
however there are no apparent successors to it being developed. For this reason it seems that it
is one of the most matured communications technologies as well as one of the most common layer
2 technology deployed today. While a roll out of Ethernet only would be one of the fastest
options for deployment. It would prove extremely difficult to manage and operate a network as
it scales to large amounts of devices. For this reason the use of Ethernet solely as a transport
mechanism would not be envisaged as an adequate method of deploying a network for
operational use. Based on this, Ethernet will not be regarded as a potential for use as a transport
mechanism

54 | P a g e

Scalability
There are three separate axis associated with the scalability of Ethernet. These are:

Maximum supported individual addressability


Hardware dependent but scalable into a number of thousands
Maximum throughput within a network segment
Hardware dependent as with all technologies
Maximum throughput per physical connection
Ethernet hardware come with a specific range of throughput values
10/100/1000Mbps(over optical or electrical interface) half or full duplex mode
and 1/100Gbps full duplex (over optical interface)

Capability to support EPU Services (as defined in chapter 2)


Ethernet was introduced over three decades ago and has become the main technology to
underpin consumer and industrial communications today. It is a mature physical layer technology
which is largely deployed within Local Area Networks or in a point to point fashion. Ethernet,
though successful and the foundation/end application of a lot of transport technologies discussed,
is a local and link layer technology which is not adopted on its own as a transport solution.
Simple switched Ethernet networks offer broadcast capabilities as well as logical separation of
services and applications using VLANS, this enables the ability to deploy basic security
functionality. Advanced networking features include spanning tree protocol and rapid spanning
tree protocol allows physical redundancy to be implemented and autonomous conversion to take
place. Port security and protection features such as MAC lock down, broadcast filtering and
virtual LANs keep different applications and services separate while using the same physical
infrastructure.
Ethernet equipment does not directly offer termination of legacy or EPU specific interfaces.
Ease of Implementation and Operation
For reasons highlighted in the Life Cycle section, further discussion is not necessary.

Interoperability (other services/technologies)


As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary.

Redundancy, Availability and Reliability


As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary.

Quality of Service (QoS)


As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary.

Network Management System and Overall Manageability (NMS)


As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary.
55 | P a g e

4.9 Comparison matrix


The tables below offer an overview of key features for all technologies within the brochure.
Technologies are assessed based on deployments which are widely adopted within the EPU sector.
The information gathered to populate these tables comes from two sources:

Results of the D2.28 Surveys


Expertise residing within the D2.35 WG

The tables depict the technical applicability of these technologies to deliver EPU Services however, it
should be noted that this does not necessarily reflect a practical suitability of the technology.(Example:
the availability of devices for external pole top applications)
A PPLICATION

SDH

WDM

Very low latency loss intolerant, sequence and


symmetry determinant traffic:

Very low latency loss intolerant

Very low latency, loss tolerant traffic:

Low latency sequence determinant traffic:

Low latency, loss tolerant traffic


Latency tolerant, loss tolerant

OTN

IPMPLS E TH
MPLS -TP

* MPLS-TP is at quite an early stage of adoption within the EPU sector. Wide scale applicability is yet to
be proven
Organisational considerations (BASED on SDH legacy SDH WDM OTN IP- MPLS Eth
control)
MPLS -TP

New skillset required?


Organisational structure concerns/opportunities?


IT/OT overlap?

Seamless connectivity to all existing endpoint


equipment (RTU, Relays, etc.)

Control Centre capability to interface

Non-disruptive rollout

Ease of implementation and operation (OAM)

Scaling towards an Any-to-Any solution platform


*1 Requires significantly less expertise IP-MPLS but still much more than SDH

*2 MPLS implementation and operation can vary based on the method used to manage it(NMS, CLI etc)
Concerns exist with regard to the maturity of the technology within EPU environments

NMS features and functionality are not standardized or controlled therefore experience can vary

56 | P a g e

Technical appraisal

SDH WDM

OTN

IP- MPLS Eth


MPLS -TP

Network Manageability
*1

Scalability

Life Cycle

Interoperation with other transport technologies

Redundancy, availability and reliability resilient


capabilities

QoS
*1 WDM has a number of flavors, some passive solutions lack any capability for vision of the
technology and rely upon layers implement above to gain manageability

*2 The current maturity of OTN is unclear within an EPU sector thus difficult to classify
Concerns exist with regard to the maturity of the technology within EPU environments

57 | P a g e

5 CONCLUSIONS AND PROPOSAL FOR FUTURE WORK


Survey
The brochure builds from information gathered within D2.28, comparing findings to information
gathered from reissuing the questionnaire to utilities two years on. Interesting trends identified include:

Utilities are gaining more of an appetite for new, alternative technologies, Ethernet and IP are
becoming more prevalent in operational networks

Most utilities have begun to roadmap a new technology choice within operational environments.
The survey from D2.35 however shows that the pace of transition is slower than predicted in the
D2.28 survey

Apprehension exists for a number of reasons


o Lack of knowledge within the EPU industry
o Lack of reassurance of a successful total deployment/replacement

Technology Life Cycle - Utilities are confronted with the phase-out of installed legacy equipment
and the risk of installing new solutions with typically much shorter life-cycles than they were used
to. It's an opportunity to look into benefits and drawbacks of either purely packet-based
technologies or hybrid networks.

Understanding of Smart Grid and IoT The communication requirements of the smart grid are
becoming more apparent, and are now beginning outline inadequacies of current solutions while
also outlining new requirements for alternative technologies

Services
To accurately assess any technology or solution within an organisation it is necessary to ensure that
requirements are clearly defined. In this topic, EPU transport solutions, requirements are defined by the
services which are carried by a technology. Issues identified by the brochure are:

No formal definition of service requirements exists within a single document Currently, not all
utilities and stakeholders share a common view to the requirements of services

There is a lot of on-going work within differing IEEE and IEC groups but there is no mechanism to
consolidate the status or results of these works

No evident attempts have been made to further classify services with common attributes into a
number of concise groups. This will be discussed further in the next paragraph

To enable the comparison of technologies outlined in the brochure a basic structure was defined
classifying different services into groups. This enables the uniform assessment of technology suitability.
This approach can account for services which are currently deployed and also future services currently
lacking concise requirements.

Technologies
58 | P a g e

Technology refresh can have substantial implications at an organisational level. It is important that
all opportunities and risks are exposed during the evaluation of suitable technologies. Implications of this
are organisation specific. The brochure assumed a control point of an EPU currently operating an SDH
solution. Opportunities and risk associated with all technologies can be noted as:

Migration to an alternative TDM/WDM technology does not have a large impact at an


organisational level.

Migration to packet/frame based technologies presents new challenges and risk but also
presents new opportunity from and organisational level surrounding OT/IT crossover, skillsets and
knowledge.

Features highlighted as most relevant when assessing differing technologies are defined as :

Predictable timing for time sensitive services and the ability to distribute precision time protocols
Security and the secureness of a technology as opposed to the securability of technologies
Ability to seamlessly interact with EPU applications and protocols
Ability to segment and virtualise
Scalability potential of the technology
Natural resilience due to the fundamental concept of a technology

Each technology discussed presents unique benefits towards the specifications above. For this reason
some technologies tend to support specific applications more easily. One such case would be point to
multipoint capabilities presented by IP/MPLS which supports a huge application base. Others include the
deterministic nature of SDH to support timing constrained applications such as teleprotection.
The Matrix allows comparisons to be drawn between technologies and suitability for specific use cases.
From the matrix it is possible to affirm the conclusions of section 4

Synchronous Digital Hierarchy SDH, the current incumbent within EPU networks, has the ability
to carry all current EPU applications. During the roll out of newer distributed applications, it is
evident that SDH does not possess suitable agility and scale to suffice as the only technology in
the future.
DWDM/CWDM While the technology ensures controlled and deterministic qualities which do
apply very well to support latency sensitive applications such as Differential teleprotection,
CWDM is a robust technology, but it does not scale very well, DWDM does lack granularity,
control and vision over a network, post deployment. It could be applied as a foundation for
subsequent layers of technologies
OTN OTN presents as a technology which could support most if not all needs of a future EPU,
concerns do exist surrounding the ability to interface with OTN at a low bandwidth or with EPU
specific connections for legacy applications. OTN systems, in large seem to be scaled to support
bandwidths well beyond the operational requirements of an EPU. This may be relevant if intent
exists to commercialise bandwidth not utilized by the EPU
IP-MPLS The ability exists to support the majority of legacy, current and future applications for
utilities. The two most striking concerns relate to the ability of IP-MPLS to carry differential
teleprotection schemes over wide areas and the organizational change/skillset refresh involved
to migrate from a technology such as SDH.
MPLS-TP Available for a number of years, and set the targets IP-MPLS needs to achieve.
While achieving far more granular control during circuit provision and operation, there are
59 | P a g e

shortcomings when implemented at scale. Both technologies (MPLS-TP and IP/MPLS) will develop
further in the future. However the overlap of the two will become bigger.
Provider Backbone Bridging (PBB) This technology seems to have been overshadowed within
all industries by OTN and for that reason does not show much promise to have a substantial role
in EPU networks
Ethernet over fibre, SDH, MPLS etc. Ethernet is not a viable solution for transportation of EPU
Services. That being said it could be applied as a foundation for technologies discussed within
the document or provided itself a s service over the top of a technology

Future Works 1
Views and decisions made by utilities, at a global level, have proven invaluable to the completion of this
brochure. It has also allowed the ability to compare, contrast and trend viewpoints on technologies,
affirming the conclusions of the brochure.
The exercise of distributing D2.28s questionnaire should become an independent exercise completed on
an annual basis aiming to compile a global view into the trends, aspirations and concerns utilities are
dealing with. This would allow utilities to individually act knowing that any decision is in line with a
globally aligned movement in the same direction.

Future Works 2
No document currently exists defining the full spectrum of EPU applications and communicative
requirements associated with them. Without formal definition of requirements it is not possible to classify
applications and uniformly approach the assessment of suitable technologies. Work suggested in this
area is a working group tasked with:

Identifying all current and most future applications utilized within utilities and standardized
Creating a uniform methodology to assess applications which can be adopted by vendors and
understood by end users/network designers
Document communication constraints and requirements for each application
Categorise and group applications with common communicative needs
Define a framework and approach to assessing technologies suitability to all application groups.

A framework, similar to that outlined above would ensure that a global assessment of technologies for
EPUs would give EPUs, Telcos, vendors and solution providers a common mechanism to identify, specify,
categorise and affirm end-to-end requirements and solutions for all communications networks. The
framework would also present an opportunity for Cigr to periodically produce reports assessing the
posture of technologies including ongoing suitability, user adoptance and ranking based on
advancements and the feedback from information received in Future Works 1. This would align similar to
the magic quadrant approach to enterprise solutions.

60 | P a g e

Future Works 3
Future Works 1 and 2 highlight the need to document and identify the current standpoint of the EPU
market as well as mechanisms to approach the global classification of EPU services and define a set
criterion which manufacturers and vendors can meet. What are the services and where are we now.
A methodology toward assessment of how to migrate services and also when to migrate services could
also prove very beneficial for EPUs to plan lifecycles and utilize existing transport assets until their end
of life. Works constituting this are captured within the proposed working group Migration Planning and
Transformation Control in EPU Telecom Networks.

61 | P a g e

ANNEX 1 SURVEY D2.35


WGD2.35 Survey
Scalable Communication Transport Solutions over Optical Networks

Survey Objective:
Dear reader,
Thank you very much for participating in this survey. Your contribution will help D2.35 to identify the
trends in the use of IP and other communication transport solutions in the EPU environment and identify the
main issues, questions and topics to be addressed in the Technical Brochure.
This survey is a continuation of the survey conducted in 2009-2010 by D2.28 and will be used to identify
trends, main issues and other topics of importance for the Technical Brochure to be prepared by D2.35
This survey is conducted among CIGR Working Group D2.35 members and CIGR members of SC B5
and D2.
Please check and detail your options.

62 | P a g e

Questionnaire:
1.

Are you a:

Transmission System Operator

Distribution System Operator

Consultant

Vendor

Other (please detail)

2.

Within your own company, which substation applications are using IP and IP networks? (For
vendors: Which applications do you see at customers sites?)

Substation RTU to SCADA Platform

Inter-Control Centre Connections

Condition Monitoring for HV devices

Metering

Disturbance Recorder Data

Telephony System

CCTV / Video Surveillance

Premise Access Control

Building Control (temperature, humidity, lights, etc.)

Network management System and Data Communications Network

Remote IP Access to Substation Assets

Tele protection

Current Differential Protection

Time Synchronization

Access to Office Applications

Wireless LAN Access

Other (please detail)

63 | P a g e

3.

From the list in Question 2, what substation applications (currently not running on IP) could
potentially be migrated to IP?

4.

What other applications are you using that could potentially be migrated to IP (applications
outside the substation)?

Phasor Measurements

Applications for Distribution Automation (if yes please detail)

Other (please detail)

5.

What are the main operational challenges of using IP as a communication transport solution over
optical networks?

Security

Converting Legacy Devices into IP

Interoperability

Ruggedness (environment)

Quality of Service (QoS)

Knowledge and Training

Scope of Responsibility
64 | P a g e

Maintenance

Reliability

Capital Expenditures (CAPEX)

Operational Expenditures (OPEX)

Other (please detail)

Additional Comments

Would the answer be different when using IP for substation specific application? If yes,
how:

6.

What are the main psychological barriers with using IP protocols in applications?

Security

Reliability

Quality of Service (QoS)

Familiarity with Ethernet / IP Concepts and Solutions

Rate of Innovation and Obsolescence

Complexity

Lack of Documentation

Capital Expenditures (CAPEX)

Operational Expenditures (OPEX)

Other (please detail)

65 | P a g e

7.

Are all of your applications compatible with IP at the moment?

Yes

No (please detail)

8.

What is your prediction for the migration of all operational communications into IP?

Already Done

Short Term

Within 1 Year

Within 1 2 Years

Within 2 5 Years

Within 5 10 Years

More Than 10 Years

Never Why?

9.

What percentage of the existing applications within your company are already using IP as
communication transport solution?

0%

25%

50%

75%

100%
66 | P a g e

10.

How to deal with legacy protocols and equipment?

Protocol Encapsulation

Vendor to Provide Solution

Gateway (Protocol Conversion)

Emulation

Separate Network

Other (please detail)

11.

Considering that all applications will move to IP, what are the requisites and concerns for the
telecommunication network?

Existence of Physical Infrastructure (fibre, copper, wireless, etc.)

Suitable Transport Technology (PDH, SDH, Ethernet, etc.)

Suitable IP Addressing Space and Scheme (IPv4 / IPv6)

Suitable Substation Applications (protection, control, measurements, etc.) for IP

Organisational structure

Identification of Applications

Other (please detail)

12.

Which underlying technologies are the most promising to provide secure and reliable transport
communications in the access network?

SDH

CWDM

DWDM

OTN

Ethernet

IP / MPLS
67 | P a g e

PBB / PBT

MPLS TP

RPR

Other (please detail)

13.

Which underlying technologies are the most promising to provide secure and reliable transport
communications in the core network?

SDH

CWDM

DWDM

OTN

Ethernet

IP / MPLS

PBB / PBT

MPLS TP

RPR

Other (please detail)

14.

Should the IP network be reserved for operational service or also be used for corporate services?
Two physically/virtual separate networks or one network?

Single Network, No Separation

Physical separation in the core

Logical separation in the core

Physical separation in the access/edge

Logical separation in the access/edge

15.

What type of scalability needs to be addressed?


68 | P a g e

Number of Users

Number of Applications

Number of Protocols

Variation of Protocols

Number of Interfaces

Number of Nodes

Number of Locations

Number of Services

Bandwidth

Other (please detail)

16.

Are you aware of any activities /documents concerning communication transport solutions over
optical networks that may be of interest for D2.35 to consider for the Technical Brochure? If yes,
can you please list them below:

Glossary of Terms
Abbreviation

Explanation

CWDM

Coarse Wavelength Division Multiplexing

DWDM

Dense Wavelength Division Multiplexing

IP

Internet Protocol

IPv4

Internet Protocol version 4

IPv6

Internet Protocol version 6


69 | P a g e

LAN

Local Area Network

MPLS

Multi Protocol Label Switching

OTN

Optical Transport Network

PBB

Provider Backbone Bridge

PBT

Provider backbone Transport

PDH

Plesiochronous Digital Hierarchy

RPR

Resilient Packet Ring

RTU

Remote Terminal Unit

SCADA

Supervisory Control and Data Acquisition

SDH

Synchronous Digital Hierarchy

WAN

Wide Area Network

70 | P a g e

ANNEX 2 SURVEY D2.28 IP-BASED SUBSTATION APPLICATIONS

WGD2.28 Survey
IP Based Substation Applications
Survey Objective:
Get an overview of the present use of IP-based applications within the substation environment, between
Substations and for Substation - to - Control Centre communications.
This survey is conducted among CIGR Working Group D2.28 members and CIGR members. Please
check and detail your options.

Questionnaire:
1- Are you a:

Transmission System Operator/Distribution System Operator


Vendor
Other (please detail):

2- Within your own company, which substation applications are using IP and IP networks? (For
vendors: Which applications do you see at customers sites?)

Substation RTU to SCADA platform


Inter-Control Centre Interconnections
Condition monitoring for HV devices
Metering
Disturbance recorder data
Telephony system
CCTV/Video Surveillance
Premises access control
Building control (temperature, humidity, lights, etc.)
Network Management System & Data Communication Network
Remote IP Access to substation assets
Tele-protection
Time synchronization
Access to office applications
Wireless LAN Access

Other (please detail):

3- What are the main operational challenges using IP into the substation environment?

Security

71 | P a g e

Converting legacy interfaces into IP


Interoperability
Ruggedness (environment)
Quality of Service
Knowledge and Training
Scope of Responsibility
Other (please detail):
Additional comment:

4- What are the main psychological barriers with using IP protocols in Substation applications?

Security
Reliability
Quality of Service
Familiarity with Ethernet/IP concepts
Rate of innovation and obsolescence
Lack of documentation
Other (please detail):

5- Are all of your applications compatible with IP at the moment?

Yes
No (please detail):

6- Prediction to migration of all operational communications into IP?

Already done
Short term
6 months
1-2 yrs
5 yrs
More than 10 yrs
Never - Why?

7- How to deal with legacy protocols and equipment?

Protocol encapsulation
Vendor

72 | P a g e

Gateway (protocol conversion)


Emulation
Separate network
Other (please detail):

8- Is the existing communications network ready for IP traffic?

Yes
No (please detail):

9- Requisites and concerns for the telecommunication network?

Existence of physical infrastructure (fibre, copper, wireless, etc)


Suitable transport technology (PDH, SDH, Ethernet, etc.)
Suitable IP addressing space and scheme (IPv4 / IPv6)
Identification of Applications
Other (please detail):

10- Which underlying IP technology is the most promising to provide secure reliable IP
communications?

MPLS (IP over fibre)


Ethernet/IP over SDH
Ethernet/IP over DWDM
Other (please detail):

11- Should the IP network be reserved for operational service or also be used for corporate services?
Two physically/virtual separate networks or one network?

One only network, no separation


Separate network (physical)
Separate network (virtual/logical)
12- Which technology is more promising to separate and provision different types of IP services?

Virtual LAN
Virtual Private Networks (Layer 3)
Virtual Private LAN Service - VPLS (Layer 2)
Multicast filtering
Other (please detail):

73 | P a g e

ANNEX 3 WGD2.35 SURVEY RESULTS


The following sections contain a summary of the survey results

1 Substation applications using IP


The participants were asked what substation applications are currently using IP.

74 | P a g e

Figure 30: Substation applications using IP


.

75 | P a g e

Other than the applications listed in the figure above, participants indicated that they use IP for the
following applications:
-

IEC 61850 station bus LANs


Distribution System SCADA Mesh radios to reclosers
Third party IP-WAN for low risk SCADA
Next generation service distribution

2 Substation applications eligible for use with IP


After having specified the applications that are currently using IP, the participants were asked what
applications that are currently not using IP would be eligible for using IP. To be able to interpret the
results better, the applications already using IP from the previous question have been added the figure.
For example, if only a small number of participants indicate that a certain application would be eligible
for use with IP in the future, is this because the application is already using IP or because they think the
application is not suitable for IP? The blue bars in the figure below represent the applications that are
already using IP form the previous questions, the red bars represent the applications that participants
indicated would be eligible for use with IP and the green bars represent the total (current use of IP +
eligibility for future use of IP).

76 | P a g e

Figure 31: Applications eligible for using IP


The relation between the scores for existing and eligible can provide a good indication of the current
status of IP implementation for the specified applications. Below provides an overview of this relation
and the possible conclusions.
77 | P a g e

Score
Existing

Score
Eligible

Conclusion

Equal

Equal

Migration to IP is currently being done, or will likely be done in the near


future
Example: Condition monitoring for HV devices

Higher

Lower

This application has mostly migrated to IP


Example: Network Management System & Data Communication Network

Lower

Higher

This application is expected to migrate to IP in the future


Example: Tele-protection
Table 1: (Future) IP usage

The totals, represented by the greens bars in can provide an indication of how suitable the participants
think IP is for certain applications. The vast majority of the participants seem to think that Tele-protection
and Current Differential protection might not be suitable for use with IP.

3 Applications outside the substation eligible for use with IP


When asked what applications participants are using outside the substation that could potentially
migrated to IP, 35% of all participants indicated that they are using Phasor measurements. Other
applications indicated by the participants include:
-

Operational voice radio communication


Recloser/RMU connectivity to the substation for information exchange
Wide area substation control
Climate information for Variable Line Rating and Dynamic Line Rating.
Distribution Feeder Automation Scheme
Power Quality Monitoring
High Speed Monitoring
Remote control of RMU
Metering
Sensors
Hydro meteorological
Reservoir Monitoring

4 Main operational challenges using IP a communication transport


solutions over optical networks
When asked what operation challenges participants see for using IP, most participants indicated that
they think Security, Knowledge and Training and converting legacy devices into IP are the biggest
challenges. An overview of the results can be found below.

78 | P a g e

Figure 32: Operational challenges


Other challenges mentioned by the participants are:
-

Applications with critical time requirements


Rate of innovation and Obsolescence
Latency for tele-protection
Pv6 (i.e. complex / unknown)
Network topology (i.e. Mesh / Ring / L2 / L3)
Lack of documentation
Complexity
By interoperability we are especially concerned at the protocol level (OPC, DNP3, SNMPV3,
61850) People have their favourite protocols.

5 Main psychological barriers with using IP protocols in Substation


applications
After having been asked what the participants saw as the main operational challenges for implementing
IP, they were asked what psychological barriers they have for using IP. The results can be below

79 | P a g e

Figure 33: Psychological Barriers


Beside security and familiarity with IP being the most important operational challenges, they also
represent the most important psychological barriers.

6 Are all applications compatible with IP at the moment?


To get an indication of how much work it would be to migrate all applications that utilities are currently
using to IP, participants were asked if all their applications are currently IP compatible. The results can
be seen below (note that the totals for yes and no might not be 100% since not all participants answered
this question).

80 | P a g e

Figure 34: All applications IP compatible?


The vast majority of participants indicate that not all of their current applications are IP compatible.
Applications of which participants indicated that they are not compatible with IP include:

Revenue meters
Non availability of IP networks at some locations
(Tele) protection
Differential protection
Time synchronization
Premise access control
Power quality
Legacy systems
PABX

7 Prediction to migration of all operational communications into IP


The results from the previous question clearly show that not all applications are currently using IP. The
participants were asked to provide an estimate on when they think all of their operational communication
would be IP based. The results can be seen in below.

81 | P a g e

Figure 35: Predication of All operational traffic through IP


Almost all participants expect that they will eventually have all of their operational traffic over IP. Almost
three quarters of the participants expect to have all their operational traffic migrated to IP within ten
years, while twenty-six percent expect that complete migration might take more than ten years.

Existing technology is expected to work for the next 10 years


Tele-protection is expected to be migrated later, or might even never be migrated
Near real time performance of services over IP and latency variation over redundant IP routes is
not yet mature enough
Securing an IP network for operational communications is a challenge
As assets get replaced, things will migrate to IP. This is already happening but might take more
than ten years to complete.
Robustness of the service = IP is always considered flakey.
Acceptance of the technology.
Increased Capex and OpEx for IP services.
Shorter life cycle components = higher Capex.
Increased level of vendor support for software = higher OpEx.
Maturing of Deterministic networks.
Migration is mainly driven by costs savings, equipment obsolescence and WAN availability
Still no solution for latency requirement for legacy teleprotection and Current Differential
Protection. Replacement strategies for such are based on direct optical connection and hence not
on an IP network
Migration must be a step-by-step process verifying the results after each step

82 | P a g e

8 Percentage of the existing applications using IP as communication


transport solution?
To get an indication on the current state of IP migration, the participants were asked to indicate what
percentage of their current applications is already using IP. The results can be seen below

Figure 36 Percentage of existing applications using IP


Almost all participants are already using IP, most of them even for at least half of their applications.

9 How to deal with legacy protocols and equipment?


Many utilities have legacy equipment that has been designed to last for several decades. It might not be
cost efficient to replace this equipment with IP based equipment as long as this legacy equipment is
working as it is supposed to. New IP-based equipment will have to be able to communicate (to some
extent) with this legacy equipment. Participants were asked what they thought was the best way to deal
with these legacy systems. The results can be seen below
83 | P a g e

Figure 37How to deal with legacy protocols/equipment


Most of the participants believe that a good solution is to keep the legacy networks separate were
possible, and to use gateways or protocol encapsulation where information needs to be exchanged
between different networks. Almost twenty percent of the participants think that vendors should provide
a solution.
Some of the comments made by the participants are:

Currently rolling out a SDH bearer network with an MPLS network overlaid on it. Legacy
protocols will be transported on the SDH network. IP services will use the MPLS network.
Each network will have separate VPNs and VRF
Migration to IP dedicated operational equipment.
Replace legacy equipment

10 Requisites and concerns for the telecommunication network


The survey held by Cigr working group D2.28 showed that many utilities have not yet (fully) migrated
their communication networks to IP. The participants were asked what their main requisites and concerns
were for having IP-only networks. The results can be seen below.

84 | P a g e

Figure 38 Requisites and concerns for using only IP


The most pressing concerns for only using IP that the participants indicated are the suitability of their
substation applications IP and vice versa. As can be seen, utilities are concerned about all of the
specified options. Beside the options shown in the figure below, participants indicated that they have to
following concerns:

Reliability
Quality of Service (QoS)
traffic load and segmentation
85 | P a g e

Time and economy


Maturity of applications. Experience has shown that many applications are converted to IP
rather than being developed with IP in mind. This leads to issues with security, stability, and
multiple software revisions. Product testing may not be robust enough.
Skills & knowledge
Operation & Management

11 Most promising technologies are to provide secure and reliable


transport communications in the access network?
When asked about the most promising technologies for the access network, results show a clear
preference for Ethernet over fibre, SDH and IP/ MPLS. The complete results can be seen below.

Figure 39 Most promising access network technology

12 Most promising technologies to provide secure and reliable


transport communications in the core network?
Participants believe that SDH and IP/ MPLS are not only suitable for the access network, but also for the
core network. Other promising technologies for the core network are DWDM and MPLS /TP. The
complete results can be seen below.

86 | P a g e

Figure 40 Most promising core network technology

13 Should the IP network be reserved for operational service or also


be used for corporate services?
Both operational services and corporate services can be hosted on the same network. A single IP network
hosting both corporate and operational services means that all information can potentially be accessed
from anywhere with a minimum of required equipment, making such a network cheap to install and
maintain. For security reasons however, it would be much safer to have separate networks for corporate
and operational services.
Participants were asked what they think is the best way to handle this. The results can be seen in below.

87 | P a g e

Figure 41 Seperate operational services from corporate services?

14 What type of scalability needs to be addressed?


IP networks can be very scalable, provided the required scalability was taken into account in the network
design. A scalable IP network might be more complex, more difficult to maintain and more expensive
than an IP network that is only designed to facilitate a pre-defined set of functionality. On the other
hand, if functionality would have to be added to a non-scalable IP network, it might mean that the whole
network would need to be redesigned, which could cost a lot of time and money. The most efficient
network design is one that facilitates only the required scalability. Participants were asked to indicate
what scalability they think is needed in their networks, to identify what scalability utilities require. The
results can be seen below.

88 | P a g e

Figure 42 What type of scalability to address?

89 | P a g e

ANNEX 4 NETWORK TIMING


Substation automation and system control are mission-critical processes and electric power utilities must
synchronize applications and use cases across large-scale distributed power grid substation networks to ensure
grid stability. Precise timing is used to improve reliability, better understand operations of the power system,
predict and prevent local and system-wide faults, for testing and verification, and to reduce costs.

Especially at transmission level, HV-line protection remains a critical application. Line differential protection (with
its stringent timing requirements) across Packet Switched Networks is not yet solved from the perspective of the
majority of utilities (feedback e.g. from EUTC conference 2013)
Utilities use precise timing to accurately timestamp data, measurements and events and share it between power
control, protection and measurement equipment. Timing can be used within substation environments, and also
across the WAN to exchange information between substations and control centers. Timing can be exchanged
within a single application or shared between multiple applications.
Precise timing is used for real-time processes as well as post-event analysis.
In addition to substations and grid management, timing is also essential in data center and operating center
local area networks, and throughout the EPU telecom and wide area networks. By looking at the overall network
picture, a utilities form strategies and practices to protect their timing and synchronization infrastructure.
Historically millisecond and tens of microsecond accuracy has been provided by SNTP and IRIG-B amongst other
mechanisms for clock redistribution. However, newer applications such as PMUs and protocols such as 61850
Sampled Values (SV) require microsecond accuracy. Accurate clocks enable sophisticated analysis of real-time
and post-disturbance faults and events within a short amount of time. This leads to faster and more reliable fault
isolation and resolution, and better planning of transmission resources on the electric grid.

90 | P a g e

Figure 43 Applications sensitive to time and frequency


While all have differing timing requirements, below outlines some of the applications which require accurate
timing:
Analogue trace recorders
Energy Management Systems
Lightning strike correlations
Merging unit and sample value (IEC 61850-9-2) measurement
Power quality measurement
SCADA
Sequence of event recording
Substation automation
Synchrophasor measurements
System frequency management (AGC)
TDM network synchronization
Time of use tariff metering
Wide area control and protection
Networking/communications equipment
Line Differential Protection
Teleprotection

Electric power utilities use precise GPS based timing at their power transmission and distribution facilities to time
stamp data and measurements. Time synchronised data can be analysed to assess causes of faults or disruptions
on the power grid, and is an essential component of newer real-time control and protection systems such as
System Integrity Protection Schemes (SIPS) and Centralised Remedial Action Schemes (C-RAS). Without accurate
time stamps, the collected data cannot be correlated and becomes worthless. This reliance on GPS has been
highlighted as a potential weakness of the traffics above and for this reason it would be preferable to attain
timing similar to that available within GPS streams through technologies deployed on Fibre Optics

Synchronisation Types
There are typically two types of time synchronization in EPU networks: direct and indirect over a communications
network. The internal clock of a device will be synchronized to master clock or server which is normally
connected to an accurate time source such as GPS, or quantum clock. The accuracy and variance of the time will
vary depending on a number of factors including communications network/media, distance, protocols, traffic,
interference and errors.

91 | P a g e

For direct synchronization a standalone communications network for timing redistribution is required which can
be expensive. These are typically fiber, copper or coaxial networks. This approach is normally implemented for
intra-substation timing and uses protocols such as IRIG-B, GPS and 1PPS.
Synchronization over a communications network (LAN or WAN) redistributes time over a communications media
such as Ethernet, and is usually part of a converged (timing and other data) network, reducing cost. Traditionally
NTP/SNTP have been used to synchronise devices, and more recently 1588 PTP has become an alternative or
an essential option depending on required accuracy of an application. In addition Synchronous Ethernet (SyncE)
can be used to redistribute frequency over the WAN and mitigate clock drift between WAN devices.

Figure 44 Time synchronisation application

IRIG-B Overview
Inter-range instrumentation group time codes B (IRIG-B) and is an industry standard for GPS time
synchronization. IRIG-B can be applied in a substation for power quality and system stability monitoring,
sequence of events recording and accurate time stamping for revenue billing (1 ms). IRIG-B needs to be
transmitted via a dedicated timing network (usually of copper or coax cable) and is therefore not a lowcost solution.
IRIG-B needs an external time source such as GPS, can be modulated (over a carrier signal) or
unmodulated (no carrier signal), and can deliver accuracy in the microsecond range (as low as 10-100
microseconds). It is widely used in the electrical power industry with around 90% of intra-substation
timing use cases, and may contain time of year, year, and seconds of day information.

Global Positioning System (GPS) Overview

92 | P a g e

The GPS system is highly redundant (although there are concerns over security of the system) and is used
in the EPU industry for both direct time synchronization and as a time source for other time protocols. For
direct synchronization an antenna per IED or device is required which can be a costly solution, or GPS can
be used as a time source to be redistributed through a timing network (dedicated or shared). The GPS
system is capable of delivering timing accuracies of within ten nanoseconds,
Global Positioning Systems (GPS) are used to provide a precise time reference for power system devices
in an electrical network, ensuring time synchronization even across dispersed networks (substation or
distribution networks) and over large geographical distances.
In the distributed model a GPS antenna receives a time signal and a receiver converts it to one or more
time protocols which are distributed to power system devices which in turn set their internal clock.
The Global Positioning System provides a very accurate time reference, however when GPS is not
available the system cannot operate as intended for long, and therefore vulnerabilities of the GPS have
become a major concern.
Other available satellite based technologies using the same principles are GLONASS, GALILEO and
COMPASS
3.2.3. Impairments affecting satellite based timing systems

The reception of the signals is subjected to many error sources, which can lead to degraded system
performance. The most common sources and their effects are:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

Signal arrival time measurement


Atmospheric effects
Multipath effects
Ephemeris and clock errors
Geometric dilution of precision computation (GDOP)
Selective availability
Anti spoofing
Relativity
Special and general relativity
Signal distortion
Natural sources of interference
Artificial sources of interference

Contribution of the errors on the position and timing inaccuracies


SOURCE OF ERROR
Signal arrival time
Atmospheric effects
Multipath effects
Ephemeris and clock errors
Geometric dilution of precision
Selective availability
Antenna spoofing
Relativity effects
Signal distortion
Interfering signals

DISTANCE ERROR
3m
7m
2m
2.5 m
12 m
100 m
1m
16,19 km/day
90 m
30 m

TIMING ERROR
10 ns
23.4 ns
6.67 ns
8.34 ns
40 ns
334 ns
3.33 ns
54 s/day
300 ns
100 ns

93 | P a g e

Noise in radio path

50 m

167 ns

The effects mentioned above may not directly effect any of the applications mentioned within this
document but could contribute as a tipping point for overall latency and delay thresholds within a system.
These can be mitigated.

One Pulse Per Second (1PPS) Overview


Generally a GPS receiver gives out a high precision one pulse per second to mark exact second
intervals. This signal is used for precise timing and synchronization. This is then forwarded to each
individual user over dedicated communication wiring lines, and is therefore a costly solution to implement.
The GPS receiver produces a 1PPS pulse with a defined level and pulse length, and accuracy can be in
the tens of nanosecond range (typically 40 ns).

NTP/SNTP Overview
NTP (Network Time Protocol, RFC 1305) is a software method to transfer time between clients and
servers over packet switched variable latency data networks, and was designed to minimize the effects
of variable latency. Although most commonly used over UDP/IP protocol, it can also be transported via
TCP/IP. It has been particularly designed to resist the effects of variable latency.

NTP provides reasonable accuracy, from a few milliseconds up to a few hundred milliseconds depending
on the nature of the connection between the NTP client and the server. NTPv4 can usually maintain time to
within 10 milliseconds over the public communications networks such as the Internet, and can achieve
accuracies of 200 microseconds or better in local area networks under ideal conditions. NTP is often used
for the time synchronization of communications network devices such as routers and switches on the EPU
communications network.

SNTP (Simple Network Time Protocol, RFC 2030) describes a simplified form of NTP. While they both use
the same network package format, but lacks some internal algorithms that are not needed for all types
of servers.

IEEE 1588 PTP / IEC 61588 PTP


PTP stands for Precision Time Protocol and is a standard Ethernet protocol described in the standards IEEE
1588 and IEC61588 for time synchronization. It is seen as a cost effective solution which can be
implemented over a shared Ethernet network in a substation, and across a WAN.
Software versions of 1588 PTP provided similar time accuracy as NTP on a LAN, and therefore a
hardware implementation was required to achieve microsecond consistent accuracy. This also meant that
both timing and data could exist on the same communication port on a substation device, as well as a
communication device, with sub microsecond accuracy. 1588 is capable of accuracy in the 20 to 100ns
range
This type of accuracy is essential for newer applications such as SIPS and C-RAS, and protocols like IEC
94 | P a g e

61850 GOOSE and SV.


Unfortunately some security problems have been reported concerning to this protocol and his use must be
restricted in protected areas where the intruders are kept out.
Versions of 1588
1588 (2002) Revision 1 "Standard for a Precision Clock Synchronization Protocol for Networked
Measurement and Control Systems"
IEC 61588 (2004) Revision 1
1588 (2008) Revision 2 introduces =<100ns accuracy variation, Transparent clocks, IPv6, Layer 2,
Unicast, and Profiles (Default, Power, Telecom) C37.238 (Power Profile)
C37.238 (2011) Introduces =<50ns accuracy (1 microsecond clock variation over 16 hops, equals
roughly 50ns per switch and 200ns for the grandmaster clock) for Power Profile

NTP vs. SNTP vs. PTP: A Summary


NTP should be used in computers that are synchronized by other computers on the Internet or a similar
wide-area network. The clock discipline algorithms will improve the accuracy and stability of the local
clock.
SNTP should be used for servers synchronized by a stratum-0 source, such as GPS. SNTP may be used
for clients. SNTP is optimal for synchronization where all the clients, and a single server, are located on
the same local-area network.
PTP should be used in networks built of IEEE-1588 compliant, hardware-aided devices, and which require
accuracy of better than one microsecond. Note that non-hardware-aided implementations of PTP have
performance similar to SNTP.

95 | P a g e

Anda mungkin juga menyukai