Anda di halaman 1dari 4

Summary:

Phase 1 LAX VCG Software upgrade to the latest revision 3.3.0


 LA2-VCG-1.1 8.44.216.247
 LA2-VCG-1.2 8.44.216.248
 LA2-VCG-1.3 8.44.216.249
 LA2_BACKUP_VCG-1 8.44.216.251
 LA2_BACKUP_VCG-2 8.44.216.252
 LA2_BACKUP_VCG-3 8.44.216.253

 LAX-VCG-VMHOST-1 8.44.216.246
 LAX-VCG-VMHOST-2 8.44.216.250

Phase 2 DAL VCG Software upgrade to the latest revision 3.3.0


 DAL-VCG-1.13 8.76.111.243
 DAL-VCG-1.23 8.76.111.244
 DAL-VCG-1.33 8.76.111.245
 DAL-Velo-VCG1 38.76.111.246
 DAL-Velo-VCG2 38.76.111.247

 DAL-VCG-VMHOST-1 38.76.111.242
 DAL-Velo-VCG-VMHOST-1 38.76.111.249
 DAL-Velo-VCG-VMHOST-2 38.76.111.250

Phase 3 CHI VCG Software upgrade to the latest revision 3.3.0


 CHI-VCG-1.1 8.46.207.156
 CHI-VCG-1.2 8.46.207.157
 CHI-VCG-1.3 8.46.207.158
 CHI-Velo-VCG1 8.46.207.150
 CHI-Velo-VCG2 8.46.207.151

 CHI-VCG-VMHOST-1 8.46.207.155
 CHI-VELO-VMHost-01 10.17.0.78
 CHI-VELO-VMHost-02 10.17.0.79

Phase 4 NYC VCG Software upgrade to the latest revision 3.3.0


 NYC-BACKUP-VCG-1 199.195.182.11
 NYC-BACKUP-VCG-2 199.195.182.12
 NYC-BACKUP-VCG-3 199.195.182.13
 NYC-VCG-1.1 199.195.182.8
 NYC-VCG-1.2 199.195.182.9
 NYC-VCG-1.3 199.195.182.10

 NYC-Velo-VCG1 199.195.182.6
 NYC-Velo-VCG2 199.195.182.7

 NYC-VCG-VMHOST-1 199.195.182.2
 NYC-VCG-VMHOST-2 199.195.182.3
 NYC-Velo-VCG-VMHOST-1 199.195.182.4
 NYC-Velo-VCG-VMHOST-2 199.195.182.5

Date/Time:
Start: Wednesday 07/10/2019 22:00 PACIFIC
End: Wednesday 07/10/2019 23:59 PACIFIC

Engineer:
Jesus Aguilar

Service Impact:
None expected

Note:
N/A

Steps for each VCG


1. 1. Verify the number of active edges, BGP count, and number of tunnels before
the upgrade
2. 2. Perform software upgrade
3. 3. Manually restart the VCG from hypervisor if necessary
4. 4. Verify that the VCG comes back in service
5. 5. Verify that the number of active edges, BGP count, and number of tunnels
is the same after the upgrade

Fallback:
Revert VCG to previous software version.

Bridge:
https://airespring.accessionmeeting.com/j/1166542508

****checking traffic on VMHost Bridge*****


[root@nyc-vmh-01 ~]# tcpdump -nnei enp1s0f1 -vvv
tcpdump: listening on enp1s0f1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:00:12.507979 7c:25:86:af:20:20 > 52:54:00:30:d1:db, ethertype 802.1Q (0x8100), length 66: vlan
3675, p 6, ethertype IPv4, (tos 0xc0, ttl 1, id 1531, offset 0, flags [none], proto TCP (6),
length 48)
10.230.22.90.61348 > 10.230.22.89.179: Flags [S], cksum 0xa3f0 (correct), seq 777396465, win
16384, options [mss 9158,sackOK,eol], length 0
13:00:12.577506 7c:25:86:af:20:20 > 52:54:00:30:d1:db, ethertype 802.1Q (0x8100), length 66: vlan
3682, p 6, ethertype IPv4, (tos 0xc0, ttl 1, id 1783, offset 0, flags [none], proto TCP (6),
length 48)

*****Running IPERF on VCG/ VCE


velocloud 76435 - Nor-Cal Products - Oregon:~# iperf -c iperf.he.net

Client connecting to iperf.he.net, TCP port 5001


TCP window size: 45.0 KByte (default)

[ 3] local 12.88.59.202 port 48161 connected with 216.218.227.10 port 5001


[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 970 MBytes 814 Mbits/sec
velocloud 76435 - Nor-Cal Products - Oregon:~#

******Partner Gateway Troubleshooting Commands


sudo su
cd /opt/vc/bin
/opt/vc/bin/debug.py --peer : List all the enterprises and edges
/opt/vc/bin/debug.py --list 3 : List the connected VCE’s
/opt/vc/bin/debug.py --vrf : Check VRF configuration
/opt/vc/bin/debug.py --path : List all the overlay tunnels
/opt/vc/bin/debug.py --ike_sa : List all the IKE SAs between the VCEs and the VCG
/opt/vc/bin/debug.py --ike : List all the IKE SAs along with the VCE name
/opt/vc/bin/debug.py --ike_child : List all the IPSec SAs between the VCEs and the VCG
/opt/vc/bin/debug.py --bgp_view_summary : List all the BGP neighbors that the VCG peers with
/opt/vc/bin/debug.py --bgp_redis_dump : Show the VCE routes advertised to the Peering PE
/opt/vc/bin/debug.py --bgp_view : Show the learned BGP routes from peering PE
vtysh -c “show ip bgp view <enterprise logical ID>” - Show bgp routes on VCG
vtysh -c “show ip bgp view <enterprise logical ID> neighbor <neighbor ip> - show neighbor
vtysh -c “show ip bgp view <enterprise logical ID> neighbor <neighbor ip> advertised-routes -
show advertised route to neighbor
/opt/vc/sbin/gwd -v - Check the software version on VCG

*****Checking IKE messages @ VCG from VCE****


cat /var/log/ike.log | grep 166.254.206.177

*****Checking tunnels to VCG


velocloud 84998 - Fairview:~# debug.py --path
Interface VLAN PeerName PublicIpAddr PeerIpAddr TunnelingMode Version
Path State RxState TxState AvgLatencyRx AvgLatencyTx RxJitter TxJitter lossRx
lossTx MeasuredRateRx MeasuredRateTx RemoteRx HeartbeatIntervalMs MTU Dynamic Dir
Overhead
GE3 NONE vc-nyc-vcg2 166.254.206.177 199.195.182.7 DEFAULT 1837571906
ACTIVE STABLE STABLE 112 111 7.8 7.0 0.0 0.0
25435 43585 43585 500 1428 No OUT 0
GE3 NONE vc-chi-vcg1 166.254.206.177 8.46.207.150 DEFAULT 1837576816
ACTIVE STABLE STABLE 102 103 5.4 4.0 0.0 0.0
25435 43585 43585 500 1428 No OUT 0
GE3 NONE vc-nyc-vcg1 166.254.206.177 199.195.182.6 DEFAULT 1837607653
ACTIVE STABLE STABLE 115 110 1.3 6.0 0.0 0.0
25435 43585 43585 100 1428 No OUT 0
GE3 NONE 83927 - Leeds 166.254.206.177 68.186.203.136 DEFAULT 2110162918
ACTIVE STABLE STABLE 0 101 0.0 1.0 0.0 0.0
25435 43585 112502 100 1428 Yes OUT 0
GE3 NONE vc-chi-vcg2 166.254.206.177 8.46.207.151 DEFAULT 2126435468
ACTIVE INITIAL INITIAL 0 0 0.0 0.0 0.0 0.0
0 0 0 100 1428 No OUT 0

~~~~~~
*****VELO TCP DUMP (ICMP)
tcpdump -i br-network3 "host 172.16.40.2 and icmp" -c 10 -n -w /var/log/test444.pcap

to read file
tcpdmp -r /var/log/test444.pcap
~~~~~~

debug.py --bgpd_dump (show bgp db status)


debug.py --biz_pol (current business policies)
debug.py --bw_retest (retest the bw on all interfaces)
debug.py --debug_bw_test {GE1,GE2,SFP1,SFP2,USB1,USB2,USB3,USB4}
debug.py --bgpd_dump (show bgp db status)
debug.py --biz_pol_dump (dump the current business policies)
debug.py --bw_retest (retest the bw on all interfaces)
--bw_test_link logical-id (Retest the bandwidth on a link by internal logical ID)
--bw_testing_dump (dump the current bw test data)
--bwcap (dump the link bandwidth caps)
--clear_arp_cache (Clear the arp cache)
debug.py --debug_bw_test {GE1,GE2,SFP1,SFP2,USB1,USB2,USB3,USB4} (Run a bandwidth test on the
path(s) connected through an interface)
--diag_trigger (trigger diag bundle)
--dns_ip_cache (dump the dns ip cache)
--dns_ip_lookup IP ADDRESS (lookup an address in the dns ip cache)
--dns_name_cache (dump the dns domain name cache)
--dns_name_lookup HOSTNAME (lookup a hostname in the dns domain name cache)
--firewall_dump [all | segment-id] (dump the current firewall version and policies)
debug.py --flow_dump (dump the current flow table entries)
--flow_flush [flow-id] / or no flow ID (flush the current flow table entries)
--flow_route_dump (dump the current flow table entries)
--flow_stats (Dump path level flow stats)
--flush_remote_routes (flush remote routes from GW. Options)
--gateways (dump the current list of VeloCloud gateways)
--ha (dump the HA verp state of the edge or the interface mac/ip related information)
--ha_flow_dump (Dump flow info syncd from ACTIVE
--ha_switch (Switch the HA state of the edge, if its ACTIVE)
--handoffqdbg (dump handoffq related info)
--health_report (Dump health information)
--interfaces (show configured interfaces)
--ip_port_lookup [ip addr] [port num] (test lookup in ip and port)
--jitter (dump the current jitter buffer enabled flow table entries)
--link_stats (dump the current link stats)
--list_vpn_endpoints (LIST_VPN_ENDPOINTS dump the list of endpoints available for VPN Testing)
--local_routes (dump the local vc route table)
--memory_dump (Dump the current unknown allocations)
--nat_db_flush (flush the NAT database entries)
--path_stats (dump the current path stats)
--peer_routes (dump the peer route table)
--qos_dump_link (Link QoS Dump)
--qos_dump_net (Net QoS Dump)
--qos_link (Link QoS Debug interface)
--qos_net (Net QoS Debug interface)
--qos_override (override the QoS policies globally)
--reload_configs (reload configs that can be reloaded without requiring edge/gw restart)
--set_link_state_down (Set internal link state DOWN)
--set_link_state_up (Set internal link state UP)
--stale_flow_dump (dump the current flow table entries)
--static_routes (dump the current static routing table)
--uptime (Dump process uptime)
debug.py --user_flow_dump [all | seg-id] [all | src-ip] (user flush of the current flow table
entries)
--user_path_dump (Dump VeloCloud Paths for Remote Diagnostics)
--user_peer_dump (Dump VeloCloud Peers for Remote Diagnostics)
--user_route_dump (dump the current vc route table with duplicates)

***
vtysh -c 'show ip bgp neighbors 10.10.221.r advertised-routes'