Migrating from GCP… to GCP

zebra on green grass field
Photo by Magda Ehlers on Pexels.com

Current and Future Architecture

Current state:

Desired state:

vpc001 is composed of the following subnets:

ricardotrentin@RicardontinsMBP Downloads % gcloud compute networks subnets list --network vpc001

NAME        REGION    NETWORK  RANGE          STACK_TYPE  IPV6_ACCESS_TYPE  INTERNAL_IPV6_PREFIX  EXTERNAL_IPV6_PREFIX
network001  us-east1  vpc001   10.11.64.0/24  IPV4_ONLY
network002  us-east1  vpc001   10.11.65.0/24  IPV4_ONLY
network003  us-east1  vpc001   10.11.66.0/24  IPV4_ONLY
network010  us-east1  vpc001   100.64.0.0/21  IPV4_ONLY

vpc001 routing table (filtering routes of interest):

On-prem (AS 36180) routing table:

csr1000v-3#show ip route 
      10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
B        10.0.0.0/16 [20/100] via 169.254.201.57, 01:42:20
B        10.11.64.0/24 [20/100] via 169.254.0.5, 01:42:20
                       [20/100] via 169.254.0.1, 01:42:20
B        10.11.65.0/24 [20/100] via 169.254.0.5, 01:42:20
                       [20/100] via 169.254.0.1, 01:42:20
B        10.11.66.0/24 [20/100] via 169.254.0.5, 01:42:20
                       [20/100] via 169.254.0.1, 01:42:20
      100.0.0.0/21 is subnetted, 3 subnets
B        100.64.0.0 [20/100] via 169.254.0.5, 01:42:20
                    [20/100] via 169.254.0.1, 01:42:20
B        100.64.8.0 [20/100] via 169.254.0.5, 01:42:20
                    [20/100] via 169.254.0.1, 01:42:20
B        100.64.16.0 [20/100] via 169.254.0.5, 01:42:20
                     [20/100] via 169.254.0.1, 01:42:20

Staging

On-prem route table after staging is complete (avx gateway is not attached):

csr1000v-3#show ip bgp summary 

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.0.1     4        64514     100     114       34    0    0 00:32:19        6
169.254.0.5     4        64514     101     115       34    0    0 00:32:20        6
169.254.2.1     4        64514     101     113       34    0    0 00:32:20        6
169.254.2.5     4        64514     101     115       34    0    0 00:32:20        6
169.254.3.1     4        64514     100     115       34    0    0 00:32:18        3
169.254.3.5     4        64514     101     114       34    0    0 00:32:20        3
169.254.4.1     4        64514     100     115       34    0    0 00:32:19        3
169.254.4.5     4        64514     101     114       34    0    0 00:32:20        3
169.254.5.1     4        64550      17      22       34    0    0 00:13:34        0
169.254.100.45  4        64512     198     216       34    0    0 00:32:13        1
169.254.201.57  4        64512     198     213       34    0    0 00:32:14        1

Attaching the gateway:

  • AVX creates four (4) custom RFC1918 routes

And advertise the vpc001 subnets with a better metric (please note that RFC 6598 prefixes are not advertised from AVX by default):

csr1000v-3#show ip bgp summary 

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.0.1     4        64514      81      90       29    0    0 00:25:48        6
169.254.0.5     4        64514      81      90       29    0    0 00:25:49        6
169.254.2.1     4        64514      81      90       29    0    0 00:25:49        6
169.254.2.5     4        64514      81      90       29    0    0 00:25:48        6
169.254.3.1     4        64514      81      90       29    0    0 00:25:49        3
169.254.3.5     4        64514      81      91       29    0    0 00:25:48        3
169.254.4.1     4        64514      81      91       29    0    0 00:25:49        3
169.254.4.5     4        64514      81      91       29    0    0 00:25:48        3
169.254.5.1     4        64550      30      36       29    0    0 00:25:42        5
169.254.100.45  4        64512     158     169       29    0    0 00:25:39        1
169.254.201.57  4        64512     159     168       29    0    0 00:25:48        1
csr1000v-3#show ip route       
       10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
B        10.0.0.0/16 [20/100] via 169.254.201.57, 00:23:00
B        10.11.64.0/24 [20/0] via 169.254.5.1, 00:01:50
B        10.11.65.0/24 [20/0] via 169.254.5.1, 00:01:50
B        10.11.66.0/24 [20/0] via 169.254.5.1, 00:01:50
B        10.12.64.0/24 [20/100] via 169.254.3.5, 00:23:00
                       [20/100] via 169.254.3.1, 00:23:00
B        10.12.65.0/24 [20/100] via 169.254.3.5, 00:23:00
                       [20/100] via 169.254.3.1, 00:23:00
B        10.12.66.0/24 [20/100] via 169.254.3.5, 00:23:00
                       [20/100] via 169.254.3.1, 00:23:00
      100.0.0.0/21 is subnetted, 3 subnets
B        100.64.0.0 [20/100] via 169.254.0.5, 00:23:00
                    [20/100] via 169.254.0.1, 00:23:00
B        100.64.8.0 [20/0] via 169.254.5.1, 00:01:50
B        100.64.16.0 [20/0] via 169.254.5.1, 00:01:50

To avoid traffic switching over to AVX asymmetrically during the staging we have a few options:

  • advertise only vpc001 subnet prefix where the gateway is deployed. This case only work if the avx gw is deployed on its own/dedicated subnet:
csr1000v-3#show ip bgp summary 

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.0.1     4        64514     367     398       33    0    0 02:01:02        6
169.254.0.5     4        64514     367     400       33    0    0 02:01:02        6
169.254.2.1     4        64514     367     399       33    0    0 02:01:02        6
169.254.2.5     4        64514     367     400       33    0    0 02:01:02        6
169.254.3.1     4        64514     367     399       33    0    0 02:01:02        3
169.254.3.5     4        64514     367     399       33    0    0 02:01:02        3
169.254.4.1     4        64514     367     401       33    0    0 02:01:02        3
169.254.4.5     4        64514     367     400       33    0    0 02:01:02        3
169.254.5.1     4        64550     126     140       33    0    0 02:00:55        1
169.254.100.45  4        64512     730     767       33    0    0 02:00:52        1
169.254.201.57  4        64512     731     768       33    0    0 02:01:01        1
csr1000v-3#show ip route       
       10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
B        10.0.0.0/16 [20/100] via 169.254.201.57, 02:02:01
B        10.11.64.0/24 [20/0] via 169.254.5.1, 01:40:51
B        10.11.65.0/24 [20/100] via 169.254.0.5, 00:01:35
                       [20/100] via 169.254.0.1, 00:01:35
B        10.11.66.0/24 [20/100] via 169.254.0.5, 00:01:35
                       [20/100] via 169.254.0.1, 00:01:35
B        10.12.64.0/24 [20/100] via 169.254.3.5, 02:02:01
                       [20/100] via 169.254.3.1, 02:02:01
B        10.12.65.0/24 [20/100] via 169.254.3.5, 02:02:01
                       [20/100] via 169.254.3.1, 02:02:01
B        10.12.66.0/24 [20/100] via 169.254.3.5, 02:02:01
                       [20/100] via 169.254.3.1, 02:02:01
      100.0.0.0/21 is subnetted, 3 subnets
B        100.64.0.0 [20/100] via 169.254.0.5, 02:02:01
                    [20/100] via 169.254.0.1, 02:02:01
B        100.64.8.0 [20/100] via 169.254.0.5, 00:01:35
                    [20/100] via 169.254.0.1, 00:01:35
B        100.64.16.0 [20/100] via 169.254.0.5, 00:01:35
                     [20/100] via 169.254.0.1, 00:01:35
  • AS-Prepend
csr1000v-3#show ip bgp 

     Network          Next Hop            Metric LocPrf Weight Path
 *    10.0.0.0/16      169.254.100.45         200             0 64512 i
 *>                    169.254.201.57         100             0 64512 i
 *    10.11.64.0/24    169.254.5.1              0             0 64550 64550 64550 i
 *m                    169.254.0.1            100             0 64514 ?
 *                     169.254.2.5            333             0 64514 ?
 *>                    169.254.0.5            100             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
  • Change GCP Cloud Router metric. This option does not work as AVX sets the advertised priority to 0.
csr1000v-3#show ip bgp 
BGP table version is 46, local router ID is 169.254.201.58
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *    10.0.0.0/16      169.254.100.45         200             0 64512 i
 *>                    169.254.201.57         100             0 64512 i
 *>   10.11.64.0/24    169.254.5.1              0             0 64550 i
 *                     169.254.0.1              0             0 64514 ?
  • change the metric using route-map
csr1000v-3#show route-map
 
route-map avx-transit, permit, sequence 10
  Match clauses:
  Set clauses:
    local-preference 1000
  Policy routing matches: 0 packets, 0 bytes
route-map gcp-vpc001-havpn, permit, sequence 10
  Match clauses:
  Set clauses:
    local-preference 2000
  Policy routing matches: 0 packets, 0 bytes
csr1000v-3#show ip bgp 


     Network          Next Hop            Metric LocPrf Weight Path
 *>   10.0.0.0/16      169.254.201.57         100             0 64512 i
 *                     169.254.100.45         200             0 64512 i
 *>   10.11.64.0/24    169.254.0.1            100   2000      0 64514 ?
 *m                    169.254.0.5            100   2000      0 64514 ?
 *                     169.254.2.5            333             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
 *                     169.254.5.1              0   1000      0 64550 i

Switching traffic over (East-West)

  • Stage avx gateways

vpc001 destination routes to vpc002:

  • Remove vpc peering

When Cloud Routers learn a prefix that exactly matches the destination of an existing subnet or peering subnet route, Google Cloud does not create any custom dynamic route for the conflicting prefix.

Removing the vpc peering and its routes makes the avx static routes introduced by the controller when attaching the gateway preferred:

For cases where (routes with) more specifics (prefixes) drag the traffic to another place instead the avx gateway like the scenario below:

We can customize the avx spoke gateway vpc route table:

Switching traffic over (North-South)

While the on-prem switchover is considerably simple as we saw before, the vpc side it is not due the nature of Cloud Router. Cloud Router strips out any bgp property we could eventually use to traffic engineer traffic.

Option 1: Most specific destination

We can customize the avx spoke gateway vpc route table:

  • we will replace 172.31.0.0/28 and 172.31.0.128/28 into /27s: 172.31.0.0/27, 172.31.0.32/27, 172.31.0.64/27, 172.31.0.96/27, 172.31.0.128/27, 172.31.0.160/27, 172.31.0.192/27 and 172.31.0.224/27.

Option 2: withdrawn dynamic routes

There are multiple ways to accomplish this and i believe the most simple one is to disable the bgp peer:

Once the bgp peering is tiered down, avx controller programmed static routes will be preferred:

Other approaches include on-prem router BGP manipulation as blocking/filtering prefixes out.

Spoke to Spoke Detailed Steps

The stage environment diagram is displayed below:

  • Spoke gateway only advertises the network it is deployed into:
  • AVX gws are deployed and attached to the transit gw
  • VPCs vpc002 and vpc003 are peered

Route table:

Dealing with “Global VPC”s

This case is where subnets inside a vpc are spread across regions:

  • 10.12.64-66.0/24 are on us-east1
  • 10.12.67-69.0/24 are on us-central1

For a smooth migration in such case, we need to run AVX sw version 7.1 where “Global VPC” is supported:

A second transit aligned to the second region (us-central1) is deployed and also connect to on-prem. The Global VPC feature allows the deployment of multiple gateways on different regions and with proper routes to bring the traffic from the workload to the preferred region.

The migration strategy is the same but care should be take on where the import and export routes would be done to avoid cross region traffic.

References

https://cloud.google.com/vpc/docs/routes#routeselection

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13753-25.html

Leave a Reply