The diagram below shows the initial scenario:
- vpc peering is used for inter-vpc communication
- TGW is used for on-prem communication
This TGW is not “managed” by Aviatrix.
Private route table looks like:
Public route tables look like:
CSR config:
interface Tunnel1
ip address 169.254.183.186 255.255.255.252
ip tcp adjust-mss 1379
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination 34.207.45.60
tunnel protection ipsec profile ipsec-vpn-07618b0c580da3adc-0
ip virtual-reassembly
!
interface Tunnel2
ip address 169.254.138.126 255.255.255.252
ip tcp adjust-mss 1379
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination 34.238.14.11
tunnel protection ipsec profile ipsec-vpn-07618b0c580da3adc-1
ip virtual-reassembly
!
router bgp 36XXX
bgp log-neighbor-changes
bgp graceful-restart
neighbor 169.254.138.125 remote-as 64512
neighbor 169.254.138.125 ebgp-multihop 255
neighbor 169.254.183.185 remote-as 64512
neighbor 169.254.183.185 ebgp-multihop 255
!
address-family ipv4
redistribute connected
redistribute static
neighbor 169.254.138.125 activate
neighbor 169.254.183.185 activate
maximum-paths 4
exit-address-family
! AVX Transit Gateway connection to AWS TGW
TGW supports the following types of attachment:
- VPC
- VPN
- Peering
- Connect
An AWS TGW Connect allows you to establish connection between a transit gateway and the AVX Transit Gateway using Generic Routing Encapsulation (GRE) and Border Gateway Protocol (BGP).
You can create up to 4 Transit Gateway Connect peers per Connect attachment (up to 20 Gbps in total bandwidth per Connect attachment)
GRE is established on top of an attachment:
- the code below creates two extra subnets in the AVX transit vpc and uses them to attach to the TGW
- on top of the attachment, a (TGW) connect is created
data "aviatrix_transit_gateway" "avx-transit-gw" {
gw_name = var.transit_gw
}
data "aws_vpc" "avx-transit-gw-vpc-cidr" {
id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
}
resource "aws_subnet" "tgw-attachment-subnet" {
for_each = toset(var.tgw_attachment_subnets_cidrs)
vpc_id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
cidr_block = each.value
tags = {
Name = "tgw-attachment-subnet"
}
}
locals {
tgw-attachment-subnet_list = [
for subnets in aws_subnet.tgw-attachment-subnet: subnets.id
]
}
resource "aws_ec2_transit_gateway_vpc_attachment" "tgw-attachment-avx" {
subnet_ids = local.tgw-attachment-subnet_list
transit_gateway_id = aws_ec2_transit_gateway.tgw.id
vpc_id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
}
resource "aws_ec2_transit_gateway_connect" "tgw-connect-avx" {
transit_gateway_id = aws_ec2_transit_gateway.tgw.id
transport_attachment_id = aws_ec2_transit_gateway_vpc_attachment.tgw-attachment-avx.id
}A route towards the TGW cidr block is required on the AVX gateway route table:
- by default AVX route table has the vpc and a default pointing to the vpc IGW
data "aws_route_table" "avx-tgw-route-table" {
vpc_id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
filter {
name = "tag:Name"
values = ["*transit-Public-rtb"]
}
}
resource "aws_route" "route-avx-tgw-cidr" {
route_table_id = data.aws_route_table.avx-tgw-route-table.route_table_id
destination_cidr_block = element(var.transit_gateway_cidr_blocks, 0)
transit_gateway_id = aws_ec2_transit_gateway.tgw.id
}The next step is to create peers:
resource "aws_ec2_transit_gateway_connect_peer" "connect-avx-primary" {
bgp_asn = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
transit_gateway_address = "10.10.0.1"
peer_address = data.aviatrix_transit_gateway.avx-transit-gw.private_ip
inside_cidr_blocks = ["169.254.253.0/29"]
transit_gateway_attachment_id = aws_ec2_transit_gateway_connect.tgw-connect-avx.id
}
resource "aws_ec2_transit_gateway_connect_peer" "connect-avx-ha" {
bgp_asn = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
transit_gateway_address = "10.10.0.2"
peer_address = data.aviatrix_transit_gateway.avx-transit-gw.ha_private_ip
inside_cidr_blocks = ["169.254.254.0/29"]
transit_gateway_attachment_id = aws_ec2_transit_gateway_connect.tgw-connect-avx.id
}The IPv4 CIDR block must be /29 size and must be within 169.254.0.0/16 range, with exception of: 169.254.0.0/29, 169.254.1.0/29, 169.254.2.0/29, 169.254.3.0/29, 169.254.4.0/29, 169.254.5.0/29, 169.254.169.248/29.
The first IP from each CIDR block is assigned for customer gateway, the second and third is for Transit Gateway. An example: from range 169.254.100.0/29, .1 is assigned to customer gateway and .2 and .3 are assigned to Transit Gateway.
AVX Config
resource "aviatrix_transit_external_device_conn" "avx-aws-connect" {
vpc_id = data.aviatrix_transit_gateway.avx-transit-gw.vpc_id
connection_name = "avx-aws-connect"
gw_name = data.aviatrix_transit_gateway.avx-transit-gw.gw_name
connection_type = "bgp"
tunnel_protocol = "GRE"
bgp_local_as_num = data.aviatrix_transit_gateway.avx-transit-gw.local_as_number
bgp_remote_as_num = var.amazon_side_asn
remote_gateway_ip = "10.10.0.1,10.10.0.2"
direct_connect = true
ha_enabled = false
local_tunnel_cidr = "169.254.251.1/29,169.254.252.1/29"
remote_tunnel_cidr = "169.254.251.2/29,169.254.252.2/30"
enable_edge_segmentation = false
}Checking
The connectivity supported by Aviatrix differs from what TGW expects:
- AVX creates a single bgp peer per gateway
- AWS creates two bgp peers per TGW gateway
Terraform
My buddy Jun created a terraform module to automate all the steps above:
Routes Before Migration
Private:
Public:
Aviatrix Gateways Deployment
You can deploy gateways into the existing VPCs using the AVX Controller or terraform:
resource "aviatrix_spoke_gateway" "spoke_gateway" {
for_each = var.deploy_spoke_gateway ? var.vpcs : {}
cloud_type = 1
account_name = var.account
gw_name = each.value.vpc_name
vpc_id = module.vpc[each.value.vpc_name].vpc_id
vpc_reg = data.aws_region.aws_region-current.name
gw_size = var.gw_size
ha_gw_size = var.gw_size
subnet = element(slice(cidrsubnets(each.value.vpc_cidr, 4, 4, 4, 4, 4, 4), 4, 5), 0)
single_ip_snat = false
manage_transit_gateway_attachment = false
ha_subnet = element(slice(cidrsubnets(each.value.vpc_cidr, 4, 4, 4, 4, 4, 4), 5, 6), 0)
}If you are using Aviatrix Cloud Services Migration Framework tool kit:
2022-11-14 17:13:45,310 subnet-060c971c19c12abd7
2022-11-14 17:13:45,310 - Discover route(s) for rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,376 ...............................................................
2022-11-14 17:13:45,376 Prefix Next-hop Origin
2022-11-14 17:13:45,376 ...............................................................
2022-11-14 17:13:45,376 10.11.0.0/23 local auto
2022-11-14 17:13:45,376 10.12.0.0/23 pcx-078a185d033bca1e8 manual
2022-11-14 17:13:45,376 **Alert** unexpected private IP 10.12.0.0/23 to pcx-078a185d033bca1e8 in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377 10.13.0.0/23 pcx-0014f772c8e2ba725 manual
2022-11-14 17:13:45,377 **Alert** unexpected private IP 10.13.0.0/23 to pcx-0014f772c8e2ba725 in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377 10.14.0.0/23 pcx-08d3808b36fa6cb3d manual
2022-11-14 17:13:45,377 **Alert** unexpected private IP 10.14.0.0/23 to pcx-08d3808b36fa6cb3d in rtb-0eb9d0f325efa2214
2022-11-14 17:13:45,377 0.0.0.0/0 tgw-05015cee55ad7479a manual
2022-11-14 17:13:45,377 **Alert** route 0.0.0.0/0 to unexpected tgw-05015cee55ad7479a in rtb-0eb9d0f325efa2214Once the gateways are deployed, the next step is to “attach” them to the transit gateways. Again, you can choose to do it using the GUI or terraform:
resource "aviatrix_spoke_transit_attachment" "spoke_attachment" {
depends_on = [
aviatrix_spoke_gateway.spoke_gateway
]
for_each = var.attach_spoke_gateway ? var.vpcs : {}
spoke_gw_name = each.value.vpc_name
transit_gw_name = data.aviatrix_transit_gateway.avx-transit-gw.gw_name
}Once gateways are attached, AVX will take ownership and control of VPC route tables:
- private
- public
East-West
An easy way to do it is simply deleting the pcx routes (10.X.0.0/23) what makes the 10.0.0.0/8 pointing to the AVX spoke gateway elastic network interface preferred. But, but, there is a gotcha: we need to remove routes in all vpcs. For example, on the vpc 10.11.0.0/23 if we remove the route to 10.12.0.0/23, we also need to remove the route towards 10.11.0.0/23 from the 10.12.0.0/23 vpc.
ACS Migration Toolkit version 0.2.39 introduced the option to delete pcx route tables.
The dm.delete_peer_route is responsible for removing peering routes across vpcs. The dm.cleanup syntax is the following:
python3.9 -m dm.delete_peer_route --ctrl_user admin --yaml_file test-lab-aviatrix-discovery.yaml Using the migration toolkit we have:
- local vpc
- remote vpc
Using the ACS migration tool kit to switch traffic to avx:
python3.9 -m dm.switch_traffic --ctrl_user admin --yaml_file test-lab-aviatrix-discovery.yaml --rm_static_route --rm_propagated_routeIt is important to point out that because of the TGW attachment north-south traffic will be forced through avx and the return traffic comes through TGW. For that reason the flags –-rm_static_route and –rm_propaged_route are used to properly remove TGW attachments route propagation and use the BGPoGRE (TGW attachment type connect) for the return traffic:
Internet Egress
The way to migrate to a centralized egress is to remove the 0/0 from the private subnet route tables and enable centralized egress:
Using the migration toolkit, we can instruct the tool to ignore TGW routes during the discovery:
config:
add_vpc_cidr: false
filter_vgw_routes: False
filter_tgw_routes: True
configure_transit_gw_egress: TrueAfter a successful migration a private subnet route table will look like:
Troubleshooting
- The Outer IP addresses of the GRE tunnel assigned to Transit Gateway are not pingable.
- If you have same prefix propagated into your Transit Gateway route table coming from VPN, Direct Connect, and Transit Gateway Connect attachments, we evaluate the best path in the following order:
- Priority 1 – Direct Connect Gateway attachment
- Priority 2 – Transit Gateway Connect attachment
- Priority 3 – VPN attachment
- AVX Controller creates an inbound rule for GRE
