Migrating a full mesh vpc deployment to Aviatrix

The diagram below shows the initial scenario:

  • vpc peering is used for inter-vpc communication
  • TGW is used for on-prem communication

This TGW is not “managed” by Aviatrix.

Private route table looks like:

Public route tables look like:

CSR config:

interface Tunnel1
 ip address 169.254.183.186 255.255.255.252
 ip tcp adjust-mss 1379
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination 34.207.45.60
 tunnel protection ipsec profile ipsec-vpn-07618b0c580da3adc-0
 ip virtual-reassembly
!
interface Tunnel2
 ip address 169.254.138.126 255.255.255.252
 ip tcp adjust-mss 1379
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination 34.238.14.11
 tunnel protection ipsec profile ipsec-vpn-07618b0c580da3adc-1
 ip virtual-reassembly
!
router bgp 36XXX
 bgp log-neighbor-changes
 bgp graceful-restart
 neighbor 169.254.138.125 remote-as 64512
 neighbor 169.254.138.125 ebgp-multihop 255
 neighbor 169.254.183.185 remote-as 64512
 neighbor 169.254.183.185 ebgp-multihop 255
 !
 address-family ipv4
  redistribute connected
  redistribute static
  neighbor 169.254.138.125 activate
  neighbor 169.254.183.185 activate
  maximum-paths 4
 exit-address-family
!   

AVX Transit Gateway connection to AWS TGW

TGW supports the following types of attachment:

  • VPC
  • VPN
  • Peering
  • Connect

An AWS TGW Connect allows you to establish connection between a transit gateway and the AVX Transit Gateway using Generic Routing Encapsulation (GRE) and Border Gateway Protocol (BGP).

You can create up to 4 Transit Gateway Connect peers per Connect attachment (up to 20 Gbps in total bandwidth per Connect attachment)

GRE is established on top of an attachment:

  • the code below creates two extra subnets in the AVX transit vpc and uses them to attach to the TGW
  • on top of the attachment, a (TGW) connect is created

data "aviatrix_transit_gateway" "avx-transit-gw" {
  gw_name = var.transit_gw
}
data "aws_vpc" "avx-transit-gw-vpc-cidr" {
  id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
}

resource "aws_subnet" "tgw-attachment-subnet" {
  for_each   = toset(var.tgw_attachment_subnets_cidrs)
  vpc_id     = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
  cidr_block = each.value
  tags = {
    Name = "tgw-attachment-subnet"
  }
}

locals {
  tgw-attachment-subnet_list = [ 
    for subnets in aws_subnet.tgw-attachment-subnet: subnets.id
  ]
}

resource "aws_ec2_transit_gateway_vpc_attachment" "tgw-attachment-avx" {
  subnet_ids = local.tgw-attachment-subnet_list
  transit_gateway_id = aws_ec2_transit_gateway.tgw.id
  vpc_id             = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
}

resource "aws_ec2_transit_gateway_connect" "tgw-connect-avx" {
  transit_gateway_id      = aws_ec2_transit_gateway.tgw.id
  transport_attachment_id = aws_ec2_transit_gateway_vpc_attachment.tgw-attachment-avx.id
}

A route towards the TGW cidr block is required on the AVX gateway route table:

  • by default AVX route table has the vpc and a default pointing to the vpc IGW
data "aws_route_table" "avx-tgw-route-table" {
  vpc_id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
  filter {
    name   = "tag:Name"
    values = ["*transit-Public-rtb"]
  }
}

resource "aws_route" "route-avx-tgw-cidr" {
  route_table_id         = data.aws_route_table.avx-tgw-route-table.route_table_id
  destination_cidr_block = element(var.transit_gateway_cidr_blocks, 0)
  transit_gateway_id     = aws_ec2_transit_gateway.tgw.id
}

The next step is to create peers:

resource "aws_ec2_transit_gateway_connect_peer" "connect-avx-primary" {
  bgp_asn                       = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
  transit_gateway_address       = "10.10.0.1"
  peer_address                  = data.aviatrix_transit_gateway.avx-transit-gw.private_ip
  inside_cidr_blocks            = ["169.254.253.0/29"]
  transit_gateway_attachment_id = aws_ec2_transit_gateway_connect.tgw-connect-avx.id
}

resource "aws_ec2_transit_gateway_connect_peer" "connect-avx-ha" {
  bgp_asn                       = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
  transit_gateway_address       = "10.10.0.2"
  peer_address                  = data.aviatrix_transit_gateway.avx-transit-gw.ha_private_ip
  inside_cidr_blocks            = ["169.254.254.0/29"]
  transit_gateway_attachment_id = aws_ec2_transit_gateway_connect.tgw-connect-avx.id
}

The IPv4 CIDR block must be /29 size and must be within 169.254.0.0/16 range, with exception of: 169.254.0.0/29, 169.254.1.0/29, 169.254.2.0/29, 169.254.3.0/29, 169.254.4.0/29, 169.254.5.0/29, 169.254.169.248/29.

The first IP from each CIDR block is assigned for customer gateway, the second and third is for Transit Gateway. An example: from range 169.254.100.0/29, .1 is assigned to customer gateway and .2 and .3 are assigned to Transit Gateway.

AVX Config

resource "aviatrix_transit_external_device_conn" "avx-aws-connect" {
  vpc_id                   = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
  connection_name          = "avx-aws-connect"
  gw_name                  = data.aviatrix_transit_gateway.avx-transit-gw.gw_name
  connection_type          = "bgp"
  tunnel_protocol          = "GRE"
  bgp_local_as_num         = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
  bgp_remote_as_num        = var.amazon_side_asn
  remote_gateway_ip        = "10.10.0.1,10.10.0.2"
  direct_connect           = true
  ha_enabled               = false
  local_tunnel_cidr        = "169.254.251.1/29,169.254.252.1/29"
  remote_tunnel_cidr       = "169.254.251.2/29,169.254.252.2/30"
  enable_edge_segmentation = false
}

Checking

The connectivity supported by Aviatrix differs from what TGW expects:

  • AVX creates a single bgp peer per gateway
  • AWS creates two bgp peers per TGW gateway

Terraform

My buddy Jun created a terraform module to automate all the steps above:

https://registry.terraform.io/modules/jye-aviatrix/bgp-over-gre-brownfield-tgw-avx-transit/aviatrix/latest

Routes Before Migration

Private:

Public:

Aviatrix Gateways Deployment

You can deploy gateways into the existing VPCs using the AVX Controller or terraform:

resource "aviatrix_spoke_gateway" "spoke_gateway" {
  for_each                          = var.deploy_spoke_gateway ? var.vpcs : {}
  cloud_type                        = 1
  account_name                      = var.account
  gw_name                           = each.value.vpc_name
  vpc_id                            = module.vpc[each.value.vpc_name].vpc_id
  vpc_reg                           = data.aws_region.aws_region-current.name
  gw_size                           = var.gw_size
  ha_gw_size                        = var.gw_size
  subnet                            = element(slice(cidrsubnets(each.value.vpc_cidr, 4, 4, 4, 4, 4, 4), 4, 5), 0)
  single_ip_snat                    = false
  manage_transit_gateway_attachment = false
  ha_subnet                         = element(slice(cidrsubnets(each.value.vpc_cidr, 4, 4, 4, 4, 4, 4), 5, 6), 0)
}

If you are using Aviatrix Cloud Services Migration Framework tool kit:

2022-11-14 17:13:45,310   subnet-060c971c19c12abd7
2022-11-14 17:13:45,310 - Discover route(s) for rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,376   ...............................................................
2022-11-14 17:13:45,376   Prefix                   Next-hop                      Origin
2022-11-14 17:13:45,376   ...............................................................
2022-11-14 17:13:45,376   10.11.0.0/23             local                         auto
2022-11-14 17:13:45,376   10.12.0.0/23             pcx-078a185d033bca1e8         manual
2022-11-14 17:13:45,376   **Alert** unexpected private IP 10.12.0.0/23 to pcx-078a185d033bca1e8 in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377   10.13.0.0/23             pcx-0014f772c8e2ba725         manual
2022-11-14 17:13:45,377   **Alert** unexpected private IP 10.13.0.0/23 to pcx-0014f772c8e2ba725 in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377   10.14.0.0/23             pcx-08d3808b36fa6cb3d         manual
2022-11-14 17:13:45,377   **Alert** unexpected private IP 10.14.0.0/23 to pcx-08d3808b36fa6cb3d in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377   0.0.0.0/0                tgw-05015cee55ad7479a         manual
2022-11-14 17:13:45,377   **Alert** route 0.0.0.0/0 to unexpected tgw-05015cee55ad7479a in rtb-0eb9d0f325efa2214

Once the gateways are deployed, the next step is to “attach” them to the transit gateways. Again, you can choose to do it using the GUI or terraform:

resource "aviatrix_spoke_transit_attachment" "spoke_attachment" {
  depends_on = [
    aviatrix_spoke_gateway.spoke_gateway
  ]
  for_each        = var.attach_spoke_gateway ? var.vpcs : {}
  spoke_gw_name   = each.value.vpc_name
  transit_gw_name = data.aviatrix_transit_gateway.avx-transit-gw.gw_name
}

Once gateways are attached, AVX will take ownership and control of VPC route tables:

  • private
  • public

East-West

An easy way to do it is simply deleting the pcx routes (10.X.0.0/23) what makes the 10.0.0.0/8 pointing to the AVX spoke gateway elastic network interface preferred. But, but, there is a gotcha: we need to remove routes in all vpcs. For example, on the vpc 10.11.0.0/23 if we remove the route to 10.12.0.0/23, we also need to remove the route towards 10.11.0.0/23 from the 10.12.0.0/23 vpc.

ACS Migration Toolkit version 0.2.39 introduced the option to delete pcx route tables.

The dm.delete_peer_route is responsible for removing peering routes across vpcs. The dm.cleanup syntax is the following:

python3.9 -m dm.delete_peer_route --ctrl_user admin --yaml_file test-lab-aviatrix-discovery.yaml 

Using the migration toolkit we have:

  • local vpc
  • remote vpc

Using the ACS migration tool kit to switch traffic to avx:

python3.9 -m dm.switch_traffic --ctrl_user admin --yaml_file test-lab-aviatrix-discovery.yaml --rm_static_route --rm_propagated_route

It is important to point out that because of the TGW attachment north-south traffic will be forced through avx and the return traffic comes through TGW. For that reason the flags –-rm_static_route and –rm_propaged_route are used to properly remove TGW attachments route propagation and use the BGPoGRE (TGW attachment type connect) for the return traffic:

Internet Egress

The way to migrate to a centralized egress is to remove the 0/0 from the private subnet route tables and enable centralized egress:

Using the migration toolkit, we can instruct the tool to ignore TGW routes during the discovery:

config:
  add_vpc_cidr: false
  filter_vgw_routes: False
  filter_tgw_routes: True
  configure_transit_gw_egress: True

After a successful migration a private subnet route table will look like:

Troubleshooting

  • The Outer IP addresses of the GRE tunnel assigned to Transit Gateway are not pingable.
  • If you have same prefix propagated into your Transit Gateway route table coming from VPN, Direct Connect, and Transit Gateway Connect attachments, we evaluate the best path in the following order:
    • Priority 1 – Direct Connect Gateway attachment
    • Priority 2 – Transit Gateway Connect attachment
    • Priority 3 – VPN attachment
  • AVX Controller creates an inbound rule for GRE

References

https://aws.amazon.com/blogs/networking-and-content-delivery/simplify-sd-wan-connectivity-with-aws-transit-gateway-connect/

https://aws.amazon.com/blogs/networking-and-content-delivery/integrate-sd-wan-devices-with-aws-transit-gateway-and-aws-direct-connect/

Leave a Reply