Current and Future Architecture
Current state:
Desired state:
vpc001 is composed of the following subnets:
ricardotrentin@RicardontinsMBP Downloads % gcloud compute networks subnets list --network vpc001
NAME REGION NETWORK RANGE STACK_TYPE IPV6_ACCESS_TYPE INTERNAL_IPV6_PREFIX EXTERNAL_IPV6_PREFIX
network001 us-east1 vpc001 10.11.64.0/24 IPV4_ONLY
network002 us-east1 vpc001 10.11.65.0/24 IPV4_ONLY
network003 us-east1 vpc001 10.11.66.0/24 IPV4_ONLY
network010 us-east1 vpc001 100.64.0.0/21 IPV4_ONLYvpc001 routing table (filtering routes of interest):
On-prem (AS 36180) routing table:
csr1000v-3#show ip route
10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
B 10.0.0.0/16 [20/100] via 169.254.201.57, 01:42:20
B 10.11.64.0/24 [20/100] via 169.254.0.5, 01:42:20
[20/100] via 169.254.0.1, 01:42:20
B 10.11.65.0/24 [20/100] via 169.254.0.5, 01:42:20
[20/100] via 169.254.0.1, 01:42:20
B 10.11.66.0/24 [20/100] via 169.254.0.5, 01:42:20
[20/100] via 169.254.0.1, 01:42:20
100.0.0.0/21 is subnetted, 3 subnets
B 100.64.0.0 [20/100] via 169.254.0.5, 01:42:20
[20/100] via 169.254.0.1, 01:42:20
B 100.64.8.0 [20/100] via 169.254.0.5, 01:42:20
[20/100] via 169.254.0.1, 01:42:20
B 100.64.16.0 [20/100] via 169.254.0.5, 01:42:20
[20/100] via 169.254.0.1, 01:42:20Staging
On-prem route table after staging is complete (avx gateway is not attached):
csr1000v-3#show ip bgp summary
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
169.254.0.1 4 64514 100 114 34 0 0 00:32:19 6
169.254.0.5 4 64514 101 115 34 0 0 00:32:20 6
169.254.2.1 4 64514 101 113 34 0 0 00:32:20 6
169.254.2.5 4 64514 101 115 34 0 0 00:32:20 6
169.254.3.1 4 64514 100 115 34 0 0 00:32:18 3
169.254.3.5 4 64514 101 114 34 0 0 00:32:20 3
169.254.4.1 4 64514 100 115 34 0 0 00:32:19 3
169.254.4.5 4 64514 101 114 34 0 0 00:32:20 3
169.254.5.1 4 64550 17 22 34 0 0 00:13:34 0
169.254.100.45 4 64512 198 216 34 0 0 00:32:13 1
169.254.201.57 4 64512 198 213 34 0 0 00:32:14 1Attaching the gateway:
- AVX creates four (4) custom RFC1918 routes
And advertise the vpc001 subnets with a better metric (please note that RFC 6598 prefixes are not advertised from AVX by default):
csr1000v-3#show ip bgp summary
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
169.254.0.1 4 64514 81 90 29 0 0 00:25:48 6
169.254.0.5 4 64514 81 90 29 0 0 00:25:49 6
169.254.2.1 4 64514 81 90 29 0 0 00:25:49 6
169.254.2.5 4 64514 81 90 29 0 0 00:25:48 6
169.254.3.1 4 64514 81 90 29 0 0 00:25:49 3
169.254.3.5 4 64514 81 91 29 0 0 00:25:48 3
169.254.4.1 4 64514 81 91 29 0 0 00:25:49 3
169.254.4.5 4 64514 81 91 29 0 0 00:25:48 3
169.254.5.1 4 64550 30 36 29 0 0 00:25:42 5
169.254.100.45 4 64512 158 169 29 0 0 00:25:39 1
169.254.201.57 4 64512 159 168 29 0 0 00:25:48 1
csr1000v-3#show ip route
10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
B 10.0.0.0/16 [20/100] via 169.254.201.57, 00:23:00
B 10.11.64.0/24 [20/0] via 169.254.5.1, 00:01:50
B 10.11.65.0/24 [20/0] via 169.254.5.1, 00:01:50
B 10.11.66.0/24 [20/0] via 169.254.5.1, 00:01:50
B 10.12.64.0/24 [20/100] via 169.254.3.5, 00:23:00
[20/100] via 169.254.3.1, 00:23:00
B 10.12.65.0/24 [20/100] via 169.254.3.5, 00:23:00
[20/100] via 169.254.3.1, 00:23:00
B 10.12.66.0/24 [20/100] via 169.254.3.5, 00:23:00
[20/100] via 169.254.3.1, 00:23:00
100.0.0.0/21 is subnetted, 3 subnets
B 100.64.0.0 [20/100] via 169.254.0.5, 00:23:00
[20/100] via 169.254.0.1, 00:23:00
B 100.64.8.0 [20/0] via 169.254.5.1, 00:01:50
B 100.64.16.0 [20/0] via 169.254.5.1, 00:01:50To avoid traffic switching over to AVX asymmetrically during the staging we have a few options:
- advertise only vpc001 subnet prefix where the gateway is deployed. This case only work if the avx gw is deployed on its own/dedicated subnet:
csr1000v-3#show ip bgp summary
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
169.254.0.1 4 64514 367 398 33 0 0 02:01:02 6
169.254.0.5 4 64514 367 400 33 0 0 02:01:02 6
169.254.2.1 4 64514 367 399 33 0 0 02:01:02 6
169.254.2.5 4 64514 367 400 33 0 0 02:01:02 6
169.254.3.1 4 64514 367 399 33 0 0 02:01:02 3
169.254.3.5 4 64514 367 399 33 0 0 02:01:02 3
169.254.4.1 4 64514 367 401 33 0 0 02:01:02 3
169.254.4.5 4 64514 367 400 33 0 0 02:01:02 3
169.254.5.1 4 64550 126 140 33 0 0 02:00:55 1
169.254.100.45 4 64512 730 767 33 0 0 02:00:52 1
169.254.201.57 4 64512 731 768 33 0 0 02:01:01 1csr1000v-3#show ip route
10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
B 10.0.0.0/16 [20/100] via 169.254.201.57, 02:02:01
B 10.11.64.0/24 [20/0] via 169.254.5.1, 01:40:51
B 10.11.65.0/24 [20/100] via 169.254.0.5, 00:01:35
[20/100] via 169.254.0.1, 00:01:35
B 10.11.66.0/24 [20/100] via 169.254.0.5, 00:01:35
[20/100] via 169.254.0.1, 00:01:35
B 10.12.64.0/24 [20/100] via 169.254.3.5, 02:02:01
[20/100] via 169.254.3.1, 02:02:01
B 10.12.65.0/24 [20/100] via 169.254.3.5, 02:02:01
[20/100] via 169.254.3.1, 02:02:01
B 10.12.66.0/24 [20/100] via 169.254.3.5, 02:02:01
[20/100] via 169.254.3.1, 02:02:01
100.0.0.0/21 is subnetted, 3 subnets
B 100.64.0.0 [20/100] via 169.254.0.5, 02:02:01
[20/100] via 169.254.0.1, 02:02:01
B 100.64.8.0 [20/100] via 169.254.0.5, 00:01:35
[20/100] via 169.254.0.1, 00:01:35
B 100.64.16.0 [20/100] via 169.254.0.5, 00:01:35
[20/100] via 169.254.0.1, 00:01:35- AS-Prepend
csr1000v-3#show ip bgp
Network Next Hop Metric LocPrf Weight Path
* 10.0.0.0/16 169.254.100.45 200 0 64512 i
*> 169.254.201.57 100 0 64512 i
* 10.11.64.0/24 169.254.5.1 0 0 64550 64550 64550 i
*m 169.254.0.1 100 0 64514 ?
* 169.254.2.5 333 0 64514 ?
*> 169.254.0.5 100 0 64514 ?
* 169.254.2.1 333 0 64514 ?- Change GCP Cloud Router metric. This option does not work as AVX sets the advertised priority to 0.
csr1000v-3#show ip bgp
BGP table version is 46, local router ID is 169.254.201.58
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
* 10.0.0.0/16 169.254.100.45 200 0 64512 i
*> 169.254.201.57 100 0 64512 i
*> 10.11.64.0/24 169.254.5.1 0 0 64550 i
* 169.254.0.1 0 0 64514 ?- change the metric using route-map
csr1000v-3#show route-map
route-map avx-transit, permit, sequence 10
Match clauses:
Set clauses:
local-preference 1000
Policy routing matches: 0 packets, 0 bytes
route-map gcp-vpc001-havpn, permit, sequence 10
Match clauses:
Set clauses:
local-preference 2000
Policy routing matches: 0 packets, 0 bytescsr1000v-3#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 10.0.0.0/16 169.254.201.57 100 0 64512 i
* 169.254.100.45 200 0 64512 i
*> 10.11.64.0/24 169.254.0.1 100 2000 0 64514 ?
*m 169.254.0.5 100 2000 0 64514 ?
* 169.254.2.5 333 0 64514 ?
* 169.254.2.1 333 0 64514 ?
* 169.254.5.1 0 1000 0 64550 iSwitching traffic over (East-West)
- Stage avx gateways
vpc001 destination routes to vpc002:
- Remove vpc peering
When Cloud Routers learn a prefix that exactly matches the destination of an existing subnet or peering subnet route, Google Cloud does not create any custom dynamic route for the conflicting prefix.
Removing the vpc peering and its routes makes the avx static routes introduced by the controller when attaching the gateway preferred:
For cases where (routes with) more specifics (prefixes) drag the traffic to another place instead the avx gateway like the scenario below:
We can customize the avx spoke gateway vpc route table:
Switching traffic over (North-South)
While the on-prem switchover is considerably simple as we saw before, the vpc side it is not due the nature of Cloud Router. Cloud Router strips out any bgp property we could eventually use to traffic engineer traffic.
Option 1: Most specific destination
We can customize the avx spoke gateway vpc route table:
- we will replace 172.31.0.0/28 and 172.31.0.128/28 into /27s: 172.31.0.0/27, 172.31.0.32/27, 172.31.0.64/27, 172.31.0.96/27, 172.31.0.128/27, 172.31.0.160/27, 172.31.0.192/27 and 172.31.0.224/27.
Option 2: withdrawn dynamic routes
There are multiple ways to accomplish this and i believe the most simple one is to disable the bgp peer:
Once the bgp peering is tiered down, avx controller programmed static routes will be preferred:
Other approaches include on-prem router BGP manipulation as blocking/filtering prefixes out.
Spoke to Spoke Detailed Steps
The stage environment diagram is displayed below:
- Spoke gateway only advertises the network it is deployed into:
- AVX gws are deployed and attached to the transit gw
- VPCs vpc002 and vpc003 are peered
Route table:
Dealing with “Global VPC”s
This case is where subnets inside a vpc are spread across regions:
- 10.12.64-66.0/24 are on us-east1
- 10.12.67-69.0/24 are on us-central1
For a smooth migration in such case, we need to run AVX sw version 7.1 where “Global VPC” is supported:
A second transit aligned to the second region (us-central1) is deployed and also connect to on-prem. The Global VPC feature allows the deployment of multiple gateways on different regions and with proper routes to bring the traffic from the workload to the preferred region.
The migration strategy is the same but care should be take on where the import and export routes would be done to avoid cross region traffic.
References
https://cloud.google.com/vpc/docs/routes#routeselection
https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13753-25.html
