Building automated wireless

mesh networks
Attila Balogh, Magyar Telekom
 about me

• just a random IP guy

• working for Magyar Telekom

• engineering networks since ’97

• RouterOS user since 2004

6 0 G Hz

PART I.
created by Craig Thomas and Carter Bays for CBS
The spectrum (that no one wanted)

• line of sight

• rain fading

• oxygen absorption

• doppler frequency shift

• unlicensed
The spectrum (that WE need)
• minimal interference 

(high FSL, O2 absorption)

• beam-like, can be focused

• reflections - to overcome
line-of-sight problems

• unlicensed, and big

• upper 2 channels similar

to e-band
DEFINE ‘BIG’

2.4GHz Wi-Fi spectrum (92MHz)

5GHz Wi-Fi spectrum (725MHz)
Entire mobile licensed spectrum in Hungary, incl. 5G-t (1020MHz)
Entire european 26/28GHz 5G mmWave spectrum (3.25GHz)

57.24GHz - 70.20GHz = 6 x 2.16GHz = 12.96GHz free spectrum

the ‘60GHz v-band’

BUT WHAT IS IT GOOD FOR?

802.11ad 4.6Gbps / channel

BUT WHAT IS IT GOOD FOR?

802.11ad 4.6Gbps / channel

802.11ay
17.6Gbps / channel
 Once upon a time…

… in a village far, far away

Enter Terragraph
• 60 GHz single frequency mesh network

• bases on 802.11ad + GPS

synchronised TDD

• Layer 3 routed mesh with IPv6

• Self organising, Zero touch provisioning

• scales up to 100s of nodes

meanwhile in telecom industry
Broadband Internet access PPP over Ethernet 802.1q VLAN

Linear Television Multicast over IPv4

802.1q VLAN Ethernet

Video on demand Unicast over IPv4

Telephony SIP over IPv4 802.1q VLAN

 What I need What I have

Ethernet IPv6
 Affordable

Avaliable
Flexible
off-the shelf
R O U T E R O S
 idea #1 - L2TP
• Broadband Internet access

routed L2TP - IPv4 + IPv6 over PPP over L2TP

• Multicast IPTV

BCP over L2TP - Ethernet bridging over L2TP

• Session negotiation, authentication, keepalive is done by L2TP

• supports MLPPP - PPP fragmentation -> no MTU limits

• No support for ‘client-server’ tunnels over IPv6 in RouterOS

 idea #2 - EoIPv6
• Ethernet user plane -> VLANs are possible

• Works over IPv6

• Supports keep-alive and thus some link management

• Supports tunnel IDs: multiple tunnels are possible between the same
nodes
• no link negotiation: client and server needs to know each other

• pre-configuration required on both sides

 bridging the gaps

• use FQDNs to establish EoIPv6 tunnels

• use a central controller for tunnel IDs and configuration

• TR-069 is not capable enough for this task

• build a custom system that uses similar concepts

• use scripting on the routers to interact & fix bugs

 this is how we do it
1 CPE for every household

rural town/district Magyar Telekom IP Edge POP

High-speed
Internet
hAP ac2 1000BaseT Access
POP with optical junction
10GBaseLR

10GBaseLR Telekom

CRS317-1G-16S+RM
Optical

CCR1072-1G-8S+
Terragraph
Aggregation
radio mesh

10GBaseLR
Network

CCR
hAP ac2 2
hAP ac 2 Cisco
hAP ac
ASR9k
1000BaseT BNG
230VAC
CRS326-24G-2S+ 10GBaseX

multicast IPTV
10GBaseLR 10GBaseLR/WDM
city -48VDC service

1 Switch for up to 24 CPEs 1 device for up to 14 1 device for up to 2000

connected TG radios dual-play CPEs

bat, 2018
the current indoor CPE in detail
uplink
• 4-core ARMv7 (Qualcomm IPQ4018)

• 5xGE ports

• dual-band 2x2 MIMO WiFi (b/g/n/ac)

• passive POE-in supported

• features/scripting/automation
downlink
• up to 700Mbps throughput with complex tunnelling

Spring Workshop Series | Berlin

 PPPoE VLAN
Customer Ethernet EoIPv6

Terragraph
IPTV VLAN

Ethernet
IPv6
EoIPv6

EoIPv6
• 500Mbps broadband internet

• SD and HD live TV service with multicast

• Video on Demand service via unicast

RTT: Max DL: Max UL: Simultaneus OSR

12ms 956Mbps 965Mbps 35M/35M 1:15
the service architecture - protocol stacks
High-speed
Internet
Access
multicast IPTV
service

IPv4 IPv6

PPP over Ethernet IPv4

Ethernet frame Ethernet frame

802.1q VLAN 802.1q VLAN

Ethernet over IPv6 tunnel

Terragraph IPv6 only routed network

 how it is done, again?
• CPE serial numbers as identities

• DHCPv6 client to acquire address and to trigger registration

• [fixing EoIPv6 source in case of address change]

• register/update FQDN

• talk to the controller

• 1st time: stand still until new job arrives

• watchdogs, watchdogs and more custom watchdogs

I don't know who you are. What I do have are a very particular set of skills, skills I
have acquired over a very long career. I will find you, and I will manage you.
device management (reinvented?)
• ‘Internet of routers’ as in ‘Internet of Things’

• don’t talk, but listen

• CPEs post status, performance data, health data, logs (telemetry)

• CPEs look for jobs to execute:

• configuration changes

• command execution

• brought to you by the almighty /tool fetch & import

RESULTS & OUTLOOK 90%
89%
99+%

85%

• Fully automated subscriber provisioning

• network in operation for 9+ months

• 3 month network availability 99.2%

• on-line customer surveys and focus group

discussions

• customer satisfaction over 85%

AVAILABILITY
STABILITY
OVERALL

SPEED
Can I build all of this

PART II.
with Mikrotik?
Wireless Mesh network (~100 servings)

• WiGig support with at least MCS8

• L3 Routing in radio units

• Dynamic Routing protocol

• IPv6 support

• AP-AP wireless links

• synchronised network with TDD media access

 solving the issues

• Open/R >> OSPFv3 (some missing features delivered by HTTP/S)

• no AP-AP links -> dedicated radio for backhaul and for drop

• no TDD/TDMA media access -> relative low number of CPEs / radio

• link coordination -> use multiple SSIDs and ‘w60g station’ entries

• no network management system -> build one!

theterminology
network
• Fiber POP

• connected via 1Gbps SM/

MM fiber

CPE
• speaks BGP to the outside

• originates default route

• ASBR in OSPFv3

• at least 2 in each mesh

theterminology
network

• DN BH - backhaul
distribution node

• connected via MCS8+ to

CPE neighbouring DNs

• speaks OSPFv3

• wired connection to DN APs

theterminology
network

• DN AP - drop distribution
node

• CNs connect to it wirelessly

CPE
• multiple links / sector

• speaks OSPFv3

• wired connection to DN BHs

theterminology
network
• CN - client node

• connects to DN AP wirelessly

• wired connection to CPE

CPE

• starts EoIPv6 tunnels

• speaks OSPFv3

• DHCPv6 PD towards the CPE

 the network
routing

• Dynamic routing across the

entire network
BGP

OSPFv3 CPE
• node addresses

• subscriber pools
 the hardware

• wAP60G (backhaul & CPE)

• PowerBOX Pro (sector interconnect)

• wAP60Gx3 AP (access point)

• SXTsq60 Lite (low end CPE)

 the network
building blocks

wap60Gx3

wap60Gx3
wap60Gx3

wap60Gx3

wap60G

wap60G

wap60G

wap60G
wap60G

wap60G

wap60G

wap60G

wap60G

wap60G
PoE+GE PoE+GE PoE+GE PoE+GE
powerbox powerbox powerbox powerbox
pro pro pro pro
GE

48V 48V
48V 48V

mesh node with drop mesh node big mesh node with drop
 Why IPv6

• plenty of GUA available -> no need to NAT

• Link-local addresses -> link auto addressing with topology hiding

• 64 bits of address range -> flexible unique address generation

• readily supported by routerOS*

 Why OSPFv3

• complex topology, multiple routes, dynamic solution is required

• supports IPv6, works with link local addresses

• faster than RIPng

• point-to-point link support

• BFD support for faster change detection

Why you don’t use just a flat layer-2?
 mesh
N E T

W O R K S

have loops
 I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.
A tree that must be sure to span
So packets can reach every LAN.
LAYER 2 NETWORKS DON’T MIX PARTICULARLY WELL WITH LOOPS
new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:
12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55,
done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src:
00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:

Bridge port received packet with own

33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame
arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56
dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new

address as source, probably loop

frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:
12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55,
done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src:
00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:
33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame
arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56
dst: 00:2e:1a:33:44:55, done. new frame arrived, src: 00:1c:4d:12:34:56 dst: 00:2e:1a:33:44:55, done. new
 auto addressing

• links with link local addresses just work fine

• to be able to talk to the ‘outside’, GUA is needed

• let’s acquire one from the controller!

• controller is on the outside, so cannot be 


reached without a GUA
auto addressing, with some help

• what if there could be a hint…

• OSPFv3 can carry routing information of GUA prefixes in a link-local only

network

• a special suffix could indicate that this is a ‘hint’

• take the common prefix hint, add something unique -> unique GUA

• routerboard serial numbers are unique by definition

 addressing example
:local hint [/ipv6 route get [find where dst-address~"bad:babe/128"] dst-address ]

2001:db8:001:603::bad:babe/128

2001:db8:001:603::bad:babe/128 && ffff:ffff:ffff:ffff::

2001:db8:001:603::/128

:local mysn [/sys routerboard get serial-number ]

81ED08262E2B

:pick, :pick, :pick, 0000 -> ("0000:" . $part1 . ":" . $part2 . ":" . $part3)

2001:db8:001:603::81ED:0826:2E2B/128

/ipv6 address { remove [find where global and interface="lo0"]; add interface=lo0 address=$loaddr }
 addressing rules

• network infrastructure prefix: /64

• node IPv6 address: /128 (loopback)

• network UNI pool: /48

• node UNI address: /64

 external connectivity

• BGP advertises mesh aggregate network to the outside (/64-/40)

• OSPFv3 propagetes the BGP provided default into the mesh

• node addresses are not exposed as /128

• all nodes are only reachable via IPv6

• DHCPv6 PD assigned prefixes are redistributed to BGP though OSPF

 client connectivity

ethernet native
emulation routed IPv6

a.k.a legacy lean and simple

 legacy service compatibility
• EoIPv6 tunnels are started in CNs (wAP60G)

• tunnels terminated in CCR (ttt)

• tunnel creation done based on FQDNs and central

orchestrator

• 2 VLANs in each tunnel (HSI 1000, IPTV 1001)

IPoE VLAN VLAN IPoE

Ethernet
PPPoE VLAN VLAN PPPoE

Voice
C D 60GHz D
CPE
N N N
TT BNG Internet
1G backhaul 10G 10G
customer premises
IPTV
operator domain
legacy service compatibility
DHCP DHCP

VLANs

VLANs
PPPoE EoIPv6 EoIPv6 PPPoE IPTV

CPE CN DN DN TTT BNG

mikrotik 60GHz domain with Terragraph mikrotik cisco (any) Internet

DHCP DHCP
VLANs

VLANs
PPPoE EoIPv6 EoIPv6 PPPoE IPTV

CPE CN DN DN TTT BNG

any
60GHz domain with Mikrotik mikrotik cisco (any) Internet

• no crucial difference to existing setup (tunnel initiation

changes)

• less functionality in the CPE

 modern service delivery

NAT64

DNS64
XLAT DHCPv6c DHCPv6s

CPE CN DN DN

any OTT TV
60GHz domain with Mikrotik Internet

• service creation at the CN, session logging via controller

• end-to-end routed service with IPv6 only access

• NAT64/DNS64 based access to legacy IPv4 resources

• XLAT464* for IPv4 only devices behind the CPE

DNS64/NAT64 crash course
facebook.com ?
DNS64
2a03:2880:f107:83:face:b00c::25de

IPv6 IPv6

index.hu has address 81.0.120.158

index.hu ?

DNS64
2001:67c:27e4:642::5100:789e

NAT64 IPv4 IPv4

5100:789e 0x51 0x00 0x78 0x9e 81.0.120.158

try it here: https://go6lab.si/current-ipv6-tests/nat64dns64-public-test/

 automation details
• client initiated communication

• DNs and CNs push and pull data from controller using HTTPS

• node identification based on serial numbers

• non-managed DN nodes show up as TN (trans-node)

• controller to provide final settings (SSID, password, mode, frequency)

• CPE management can be done based on TR-069

 toolchain
• nginx (reverse proxy, static content)

• influxDB (time series DB)

• mySQL (config data - so far)

• grafana (data visualisation)

• perl/dancer (web application)

• /tool fetch & /system script & /system scheduler 


(routeros integration)
 room for improvement
• more responsive interaction on automation

- less complex protocol, like MQTT (pubsub) would be nice

• MCS9+ for backhaul

• support for Terragraph MAC & PHY layer

• Terragraph compatible AP-AP link support

• multi baseband units for more throughput

• N-BaseT for more capacity

thanks for the attention