multi-cloud networking

GCP Routing Without Subtitles

a person holding a map
Photo by cottonbro studio on Pexels.com

Topology 1

  • VPC001 routes:
  • CSR routes:
     Network          Next Hop            Metric LocPrf Weight Path
 *    10.0.0.0/16      169.254.100.45         200             0 64512 i
 *>                    169.254.201.57         100             0 64512 i
 *m   10.11.64.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m   10.11.65.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m   10.11.66.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *>   169.254.0.0/30   0.0.0.0                  0         32768 ?
 *>   169.254.0.4/30   0.0.0.0                  0         32768 ?
 *>   169.254.100.44/30
                      0.0.0.0                  0         32768 ?
 *>   169.254.201.56/30
                      0.0.0.0                  0         32768 ?
 *>   172.31.0.0/28    0.0.0.0                  0         32768 ?
 *>   172.31.0.128/28  172.31.0.1               0         32768 ?

Metric 100 comes from:

advertised_route_priority = 100

Topology 2

  • VPC001:

  • CSR:
     Network          Next Hop            Metric LocPrf Weight Path
 *    10.0.0.0/16      169.254.100.45         200             0 64512 i
 *>                    169.254.201.57         100             0 64512 i
 *    10.11.64.0/24    169.254.129.1          333             0 64514 ?
 *m                    169.254.0.1            100             0 64514 ?
 *>                    169.254.0.5            100             0 64514 ?
 *    10.11.65.0/24    169.254.129.1          333             0 64514 ?
 *m                    169.254.0.1            100             0 64514 ?
 *>                    169.254.0.5            100             0 64514 ?
 *    10.11.66.0/24    169.254.129.1          333             0 64514 ?
 *m                    169.254.0.1            100             0 64514 ?
 *>                    169.254.0.5            100             0 64514 ?
 *>   169.254.0.0/30   0.0.0.0                  0         32768 ?
 *>   169.254.0.4/30   0.0.0.0                  0         32768 ?
 *>   169.254.100.44/30
                      0.0.0.0                  0         32768 ?
 *>   169.254.129.0/30 0.0.0.0                  0         32768 ?
 *>   169.254.129.4/30 0.0.0.0                  0         32768 ?
 *>   169.254.201.56/30
                      0.0.0.0                  0         32768 ?
 *>   172.31.0.0/28    0.0.0.0                  0         32768 ?
 *>   172.31.0.128/28  172.31.0.1               0         32768 ?
  • CSR RIB:
S*    0.0.0.0/0 [1/0] via 172.31.0.1, GigabitEthernet1
      10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
B        10.0.0.0/16 [20/100] via 169.254.201.57, 06:57:24
B        10.11.64.0/24 [20/100] via 169.254.0.5, 00:02:15
                       [20/100] via 169.254.0.1, 00:02:15
B        10.11.65.0/24 [20/100] via 169.254.0.5, 00:02:15
                       [20/100] via 169.254.0.1, 00:02:15
B        10.11.66.0/24 [20/100] via 169.254.0.5, 00:02:15
                       [20/100] via 169.254.0.1, 00:02:15
      169.254.0.0/16 is variably subnetted, 12 subnets, 2 masks
C        169.254.0.0/30 is directly connected, Tunnel10
L        169.254.0.2/32 is directly connected, Tunnel10
C        169.254.0.4/30 is directly connected, Tunnel11
L        169.254.0.6/32 is directly connected, Tunnel11
C        169.254.100.44/30 is directly connected, Tunnel2
L        169.254.100.46/32 is directly connected, Tunnel2
C        169.254.129.0/30 is directly connected, Tunnel20
L        169.254.129.2/32 is directly connected, Tunnel20
C        169.254.129.4/30 is directly connected, Tunnel21
L        169.254.129.6/32 is directly connected, Tunnel21
C        169.254.201.56/30 is directly connected, Tunnel1
L        169.254.201.58/32 is directly connected, Tunnel1
      172.31.0.0/16 is variably subnetted, 3 subnets, 2 masks
C        172.31.0.0/28 is directly connected, GigabitEthernet1
L        172.31.0.13/32 is directly connected, GigabitEthernet1
S        172.31.0.128/28 [1/0] via 172.31.0.1

Topology 3 

routing_mode = "REGIONAL"

CSR:

 *>   10.0.0.0/16      169.254.100.45         100             0 64512 i
 *                     169.254.201.57         200             0 64512 i
 *>   10.11.64.0/24    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *>   10.11.65.0/24    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *>   10.11.66.0/24    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *>   169.254.0.0/30   0.0.0.0                  0         32768 ?
 *>   169.254.0.4/30   0.0.0.0                  0         32768 ?
 *>   169.254.100.44/30
                      0.0.0.0                  0         32768 ?
 *>   169.254.129.0/30 0.0.0.0                  0         32768 ?
 *>   169.254.129.4/30 0.0.0.0                  0         32768 ?
 *>   169.254.201.56/30
                      0.0.0.0                  0         32768 ?
 *>   172.31.0.0/28    0.0.0.0                  0         32768 ?
 *>   172.31.0.128/28  172.31.0.1               0         32768 ?

Subnetworks 10.11.64-66 are on us-east1. Adding a new subnet to vpc001 but located in us-central1:

 *>   10.0.0.0/16      169.254.100.45         100             0 64512 i
 *                     169.254.201.57         200             0 64512 i
 *>   10.11.64.0/24    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *>   10.11.65.0/24    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *>   10.11.66.0/24    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *>   100.64.0.0/24    169.254.129.1          100             0 64514 ?
 *>   169.254.0.0/30   0.0.0.0                  0         32768 ?
 *>   169.254.0.4/30   0.0.0.0                  0         32768 ?
 *>   169.254.100.44/30
                      0.0.0.0                  0         32768 ?
 *>   169.254.129.0/30 0.0.0.0                  0         32768 ?
 *>   169.254.129.4/30 0.0.0.0                  0         32768 ?
 *>   169.254.201.56/30
                      0.0.0.0                  0         32768 ?
 *>   172.31.0.0/28    0.0.0.0                  0         32768 ?
 *>   172.31.0.128/28  172.31.0.1               0         32768 ?

100.64.0.0/24 is advertised from central gateway.

Topology 4

Default Config:

  • vpc001 routes:
  • vpc002 routes:

     Network          Next Hop            Metric LocPrf Weight Path
 *    10.0.0.0/16      169.254.100.45         200             0 64512 i
 *>                    169.254.201.57         100             0 64512 i
 *m   10.11.64.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m   10.11.65.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m   10.11.66.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *>   169.254.0.0/30   0.0.0.0                  0         32768 ?
 *>   169.254.0.4/30   0.0.0.0                  0         32768 ?
 *>   169.254.100.44/30
                      0.0.0.0                  0         32768 ?
 *>   169.254.201.56/30
                      0.0.0.0                  0         32768 ?
 *>   172.31.0.0/28    0.0.0.0                  0         32768 ?
 *>   172.31.0.128/28  172.31.0.1               0         32768 ?

Import/Export:

  • vpc001:
  • vpc002
  • CSR

     Network          Next Hop            Metric LocPrf Weight Path
 *    10.0.0.0/16      169.254.100.45         200             0 64512 i
 *>                    169.254.201.57         100             0 64512 i
 *m   10.11.64.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m   10.11.65.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m   10.11.66.0/24    169.254.0.5            100             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *>   169.254.0.0/30   0.0.0.0                  0         32768 ?
 *>   169.254.0.4/30   0.0.0.0                  0         32768 ?
 *>   169.254.100.44/30
                      0.0.0.0                  0         32768 ?
 *>   169.254.201.56/30
                      0.0.0.0                  0         32768 ?
 *>   172.31.0.0/28    0.0.0.0                  0         32768 ?
 *>   172.31.0.128/28  172.31.0.1               0         32768 ?

Topology 5

 *>   10.0.0.0/16      169.254.100.45         100             0 64512 i
 *                     169.254.201.57         200             0 64512 i
 *    10.11.64.0/24    169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *    10.11.65.0/24    169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *    10.11.66.0/24    169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *m   10.12.64.0/24    169.254.3.5            100             0 64514 ?
 *>                    169.254.3.1            100             0 64514 ?
 *                     169.254.4.5            333             0 64514 ?
 *                     169.254.4.1            333             0 64514 ?
 *m   10.12.65.0/24    169.254.3.5            100             0 64514 ?
 *>                    169.254.3.1            100             0 64514 ?
 *                     169.254.4.5            333             0 64514 ?
 *                     169.254.4.1            333             0 64514 ?
 *m   10.12.66.0/24    169.254.3.5            100             0 64514 ?
 *>                    169.254.3.1            100             0 64514 ?
 *                     169.254.4.5            333             0 64514 ?

CSR1000v RIB:

S*    0.0.0.0/0 [1/0] via 172.31.0.1, GigabitEthernet1
      10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
B        10.0.0.0/16 [20/100] via 169.254.100.45, 01:27:49
B        10.11.64.0/24 [20/100] via 169.254.0.5, 01:28:10
                       [20/100] via 169.254.0.1, 01:28:10
B        10.11.65.0/24 [20/100] via 169.254.0.5, 01:28:10
                       [20/100] via 169.254.0.1, 01:28:10
B        10.11.66.0/24 [20/100] via 169.254.0.5, 01:28:10
                       [20/100] via 169.254.0.1, 01:28:10
B        10.12.64.0/24 [20/100] via 169.254.3.5, 00:12:47
                       [20/100] via 169.254.3.1, 00:12:47
B        10.12.65.0/24 [20/100] via 169.254.3.5, 00:12:47
                       [20/100] via 169.254.3.1, 00:12:47
B        10.12.66.0/24 [20/100] via 169.254.3.5, 00:12:47
                       [20/100] via 169.254.3.1, 00:12:47
      100.0.0.0/24 is subnetted, 1 subnets
B        100.64.0.0 [20/100] via 169.254.2.1, 00:03:01

Using the same AS vpc001 and vpc002 does not exchange routes. If we change vpc002 CR to a different AS (64515):

  • vpc002 rt:
  • vpc001 rt:

Topology 6

  • vpc001 rt:

Topology 7

Topology 8

  • CSR rt:
*    10.11.64.0/24    169.254.2.5            333             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *    10.11.65.0/24    169.254.2.5            333             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *    10.11.66.0/24    169.254.2.5            333             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *    10.12.64.0/24    169.254.4.1            333             0 64514 ?
 *>                    169.254.3.1            100             0 64514 ?
 *                     169.254.4.5            333             0 64514 ?
 *m                    169.254.3.5            100             0 64514 ?
 *    10.12.65.0/24    169.254.4.1            333             0 64514 ?
 *>                    169.254.3.1            100             0 64514 ?
 *                     169.254.4.5            333             0 64514 ?
 *m                    169.254.3.5            100             0 64514 ?
 *    10.12.66.0/24    169.254.4.1            333             0 64514 ?
 *>                    169.254.3.1            100             0 64514 ?
 *                     169.254.4.5            333             0 64514 ?
*m                    169.254.3.5            100             0 64514 ?
 *    100.64.0.0/24    169.254.2.5            333             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *    100.64.8.0/21    169.254.2.5            333             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
 *    100.64.16.0/21   169.254.2.5            333             0 64514 ?
 *                     169.254.2.1            333             0 64514 ?
 *>                    169.254.0.1            100             0 64514 ?
 *m                    169.254.0.5            100             0 64514 ?
  • CSR RIB:
B        10.11.64.0/24 [20/100] via 169.254.0.5, 00:13:49
                       [20/100] via 169.254.0.1, 00:13:49
B        10.11.65.0/24 [20/100] via 169.254.0.5, 00:13:49
                       [20/100] via 169.254.0.1, 00:13:49
B        10.11.66.0/24 [20/100] via 169.254.0.5, 00:13:49
                       [20/100] via 169.254.0.1, 00:13:49
B        10.12.64.0/24 [20/100] via 169.254.3.5, 00:13:49
                       [20/100] via 169.254.3.1, 00:13:49
B        10.12.65.0/24 [20/100] via 169.254.3.5, 00:13:49
                       [20/100] via 169.254.3.1, 00:13:49
B        10.12.66.0/24 [20/100] via 169.254.3.5, 00:13:49
                       [20/100] via 169.254.3.1, 00:13:49
      100.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
B        100.64.0.0/24 [20/100] via 169.254.0.5, 00:13:49
                       [20/100] via 169.254.0.1, 00:13:49
B        100.64.8.0/21 [20/100] via 169.254.0.5, 00:13:49
                       [20/100] via 169.254.0.1, 00:13:49
B        100.64.16.0/21 [20/100] via 169.254.0.5, 00:13:49
                        [20/100] via 169.254.0.1, 00:13:49
  • VPC001 rt:

References

https://cloud.google.com/vpc/docs/using-routes#gcloud

https://cloud.google.com/network-connectivity/docs/router/support/troubleshooting

https://developer.hashicorp.com/terraform/tutorials/kubernetes/gke?in=terraform%2Fkubernetes&utm_offer=ARTICLE_PAGE

https://cloud.google.com/vpc/docs/routes

Leave a Reply