Migrating a full mesh vpc deployment to Aviatrix

The diagram below shows the initial scenario:

  • vpc peering is used for inter-vpc communication
  • TGW is used for on-prem communication

This TGW is not “managed” by Aviatrix.

Private route table looks like:

Public route tables look like:

CSR config:

interface Tunnel1
 ip address
 ip tcp adjust-mss 1379
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination
 tunnel protection ipsec profile ipsec-vpn-07618b0c580da3adc-0
 ip virtual-reassembly
interface Tunnel2
 ip address
 ip tcp adjust-mss 1379
 tunnel source GigabitEthernet1
 tunnel mode ipsec ipv4
 tunnel destination
 tunnel protection ipsec profile ipsec-vpn-07618b0c580da3adc-1
 ip virtual-reassembly
router bgp 36XXX
 bgp log-neighbor-changes
 bgp graceful-restart
 neighbor remote-as 64512
 neighbor ebgp-multihop 255
 neighbor remote-as 64512
 neighbor ebgp-multihop 255
 address-family ipv4
  redistribute connected
  redistribute static
  neighbor activate
  neighbor activate
  maximum-paths 4

AVX Transit Gateway connection to AWS TGW

TGW supports the following types of attachment:

  • VPC
  • VPN
  • Peering
  • Connect

An AWS TGW Connect allows you to establish connection between a transit gateway and the AVX Transit Gateway using Generic Routing Encapsulation (GRE) and Border Gateway Protocol (BGP).

You can create up to 4 Transit Gateway Connect peers per Connect attachment (up to 20 Gbps in total bandwidth per Connect attachment)

GRE is established on top of an attachment:

  • the code below creates two extra subnets in the AVX transit vpc and uses them to attach to the TGW
  • on top of the attachment, a (TGW) connect is created

data "aviatrix_transit_gateway" "avx-transit-gw" {
  gw_name = var.transit_gw
data "aws_vpc" "avx-transit-gw-vpc-cidr" {
  id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id

resource "aws_subnet" "tgw-attachment-subnet" {
  for_each   = toset(var.tgw_attachment_subnets_cidrs)
  vpc_id     = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
  cidr_block = each.value
  tags = {
    Name = "tgw-attachment-subnet"

locals {
  tgw-attachment-subnet_list = [ 
    for subnets in aws_subnet.tgw-attachment-subnet: subnets.id

resource "aws_ec2_transit_gateway_vpc_attachment" "tgw-attachment-avx" {
  subnet_ids = local.tgw-attachment-subnet_list
  transit_gateway_id = aws_ec2_transit_gateway.tgw.id
  vpc_id             = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id

resource "aws_ec2_transit_gateway_connect" "tgw-connect-avx" {
  transit_gateway_id      = aws_ec2_transit_gateway.tgw.id
  transport_attachment_id = aws_ec2_transit_gateway_vpc_attachment.tgw-attachment-avx.id

A route towards the TGW cidr block is required on the AVX gateway route table:

  • by default AVX route table has the vpc and a default pointing to the vpc IGW
data "aws_route_table" "avx-tgw-route-table" {
  vpc_id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
  filter {
    name   = "tag:Name"
    values = ["*transit-Public-rtb"]

resource "aws_route" "route-avx-tgw-cidr" {
  route_table_id         = data.aws_route_table.avx-tgw-route-table.route_table_id
  destination_cidr_block = element(var.transit_gateway_cidr_blocks, 0)
  transit_gateway_id     = aws_ec2_transit_gateway.tgw.id

The next step is to create peers:

resource "aws_ec2_transit_gateway_connect_peer" "connect-avx-primary" {
  bgp_asn                       = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
  transit_gateway_address       = ""
  peer_address                  = data.aviatrix_transit_gateway.avx-transit-gw.private_ip
  inside_cidr_blocks            = [""]
  transit_gateway_attachment_id = aws_ec2_transit_gateway_connect.tgw-connect-avx.id

resource "aws_ec2_transit_gateway_connect_peer" "connect-avx-ha" {
  bgp_asn                       = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
  transit_gateway_address       = ""
  peer_address                  = data.aviatrix_transit_gateway.avx-transit-gw.ha_private_ip
  inside_cidr_blocks            = [""]
  transit_gateway_attachment_id = aws_ec2_transit_gateway_connect.tgw-connect-avx.id

The IPv4 CIDR block must be /29 size and must be within range, with exception of:,,,,,,

The first IP from each CIDR block is assigned for customer gateway, the second and third is for Transit Gateway. An example: from range, .1 is assigned to customer gateway and .2 and .3 are assigned to Transit Gateway.

AVX Config

resource "aviatrix_transit_external_device_conn" "avx-aws-connect" {
  vpc_id                   = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
  connection_name          = "avx-aws-connect"
  gw_name                  = data.aviatrix_transit_gateway.avx-transit-gw.gw_name
  connection_type          = "bgp"
  tunnel_protocol          = "GRE"
  bgp_local_as_num         = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
  bgp_remote_as_num        = var.amazon_side_asn
  remote_gateway_ip        = ","
  direct_connect           = true
  ha_enabled               = false
  local_tunnel_cidr        = ","
  remote_tunnel_cidr       = ","
  enable_edge_segmentation = false


The connectivity supported by Aviatrix differs from what TGW expects:

  • AVX creates a single bgp peer per gateway
  • AWS creates two bgp peers per TGW gateway


My buddy Jun created a terraform module to automate all the steps above:


Routes Before Migration



Aviatrix Gateways Deployment

You can deploy gateways into the existing VPCs using the AVX Controller or terraform:

resource "aviatrix_spoke_gateway" "spoke_gateway" {
  for_each                          = var.deploy_spoke_gateway ? var.vpcs : {}
  cloud_type                        = 1
  account_name                      = var.account
  gw_name                           = each.value.vpc_name
  vpc_id                            = module.vpc[each.value.vpc_name].vpc_id
  vpc_reg                           = data.aws_region.aws_region-current.name
  gw_size                           = var.gw_size
  ha_gw_size                        = var.gw_size
  subnet                            = element(slice(cidrsubnets(each.value.vpc_cidr, 4, 4, 4, 4, 4, 4), 4, 5), 0)
  single_ip_snat                    = false
  manage_transit_gateway_attachment = false
  ha_subnet                         = element(slice(cidrsubnets(each.value.vpc_cidr, 4, 4, 4, 4, 4, 4), 5, 6), 0)

If you are using Aviatrix Cloud Services Migration Framework tool kit:

2022-11-14 17:13:45,310   subnet-060c971c19c12abd7
2022-11-14 17:13:45,310 - Discover route(s) for rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,376   ...............................................................
2022-11-14 17:13:45,376   Prefix                   Next-hop                      Origin
2022-11-14 17:13:45,376   ...............................................................
2022-11-14 17:13:45,376             local                         auto
2022-11-14 17:13:45,376             pcx-078a185d033bca1e8         manual
2022-11-14 17:13:45,376   **Alert** unexpected private IP to pcx-078a185d033bca1e8 in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377             pcx-0014f772c8e2ba725         manual
2022-11-14 17:13:45,377   **Alert** unexpected private IP to pcx-0014f772c8e2ba725 in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377             pcx-08d3808b36fa6cb3d         manual
2022-11-14 17:13:45,377   **Alert** unexpected private IP to pcx-08d3808b36fa6cb3d in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377                tgw-05015cee55ad7479a         manual
2022-11-14 17:13:45,377   **Alert** route to unexpected tgw-05015cee55ad7479a in rtb-0eb9d0f325efa2214

Once the gateways are deployed, the next step is to “attach” them to the transit gateways. Again, you can choose to do it using the GUI or terraform:

resource "aviatrix_spoke_transit_attachment" "spoke_attachment" {
  depends_on = [
  for_each        = var.attach_spoke_gateway ? var.vpcs : {}
  spoke_gw_name   = each.value.vpc_name
  transit_gw_name = data.aviatrix_transit_gateway.avx-transit-gw.gw_name

Once gateways are attached, AVX will take ownership and control of VPC route tables:

  • private
  • public


An easy way to do it is simply deleting the pcx routes (10.X.0.0/23) what makes the pointing to the AVX spoke gateway elastic network interface preferred. But, but, there is a gotcha: we need to remove routes in all vpcs. For example, on the vpc if we remove the route to, we also need to remove the route towards from the vpc.

ACS Migration Toolkit version 0.2.39 introduced the option to delete pcx route tables.

The dm.delete_peer_route is responsible for removing peering routes across vpcs. The dm.cleanup syntax is the following:

python3.9 -m dm.delete_peer_route --ctrl_user admin --yaml_file test-lab-aviatrix-discovery.yaml 

Using the migration toolkit we have:

  • local vpc
  • remote vpc

Using the ACS migration tool kit to switch traffic to avx:

python3.9 -m dm.switch_traffic --ctrl_user admin --yaml_file test-lab-aviatrix-discovery.yaml --rm_static_route --rm_propagated_route

It is important to point out that because of the TGW attachment north-south traffic will be forced through avx and the return traffic comes through TGW. For that reason the flags –-rm_static_route and –rm_propaged_route are used to properly remove TGW attachments route propagation and use the BGPoGRE (TGW attachment type connect) for the return traffic:

Internet Egress

The way to migrate to a centralized egress is to remove the 0/0 from the private subnet route tables and enable centralized egress:

Using the migration toolkit, we can instruct the tool to ignore TGW routes during the discovery:

  add_vpc_cidr: false
  filter_vgw_routes: False
  filter_tgw_routes: True
  configure_transit_gw_egress: True

After a successful migration a private subnet route table will look like:


  • The Outer IP addresses of the GRE tunnel assigned to Transit Gateway are not pingable.
  • If you have same prefix propagated into your Transit Gateway route table coming from VPN, Direct Connect, and Transit Gateway Connect attachments, we evaluate the best path in the following order:
    • Priority 1 – Direct Connect Gateway attachment
    • Priority 2 – Transit Gateway Connect attachment
    • Priority 3 – VPN attachment
  • AVX Controller creates an inbound rule for GRE




Leave a Reply