When I started working with AWS DirectConnect few years ago – I was a bit confused about from where to start.

During that time (few years ago & even now), articles related to Cloud Solutions are mostly focused on continuous integration, continuous deployment, configuration automation, high available RDMBS etc, etc, etc; people hardly talk about “networking” on the Cloud. These solutions mostly go to application development & maintenance which has nothings to do with a Network Engineer (well network engineer does coding as well, these are to manage the infrastructure devices and not related to business applications). People hardly talk about networking on the Cloud.

If you are new to the Cloud world – probably you will believe that Cloud infrastructures are built without the help of Network Engineers! – because nobody (most of cloud marketing articles) wants to talk about them; Cloud is all about applications – no networking is required!

Being a network engineer – I was thinking is this the end of world for a network engineer? Actually, this is the beginning. I have seen so many poorly designed VPC networking with lack of security, segregation and control. Well, why this happening? Because the guys built these infrastructures are not experienced Network Engineer; they have computing skills obviously as they are DevOps and Application Developers, it’s like asking suggestions from a skin specialist for heart issues – as they both have same medical bachelor’s degree.

So when I started working on implementing AWS DirectConnect – my mindset was I am going to learn a hell lot of coding and some “brand new” ways to implement data/IP network. I started reading AWS DirectConnect documents supplied by AWS – https://aws.amazon.com/documentation/direct-connect (also VPC networking documents http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/getting-started-ipv4.html). At the end what I found is – its the same old wine in a new bottle and also with a brand new label with few extra utensils to handle it.

Here is the summary of AWS DirectConnect network components & concepts from a regular network engineer’s perspectives and surely these are very “common knowledge” and nothing very new or unknown to worry about.

Part 1: Why use AWS DirectConnect?

Q1. Why we need AWS DirectConnect(s)?
Ans: To have an inter-connect between (a)self-managed infrastructure and (b)resources on AWS such as Virtual Private Cloud, AWS S3; Hybrid cloud is getting popular, companies these days wants to integrate self-managed cloud platform (infrastructure) with AWS or other (Azure has similar network connection offerings).

Other examples are:
-some companies have self-manage back-end systems such DB servers and they want put the front-end application servers on AWS Cloud.
-some companies use DirectConnects to send self-managed data backup to offsite location within AWS S3.
-some companies put DirectConnects to have a high-speed migration of on premise resources off to AWS Cloud during infrastructure migration to AWS.

Q2. Well, my company or client needs AWS DirectConnect. How can I get a direct connect? Where can I get this?
Ans: AWS DirectConnects are now available to many data centres across the globe these day. Check out at AWS web site for availability within your preferred data centre https://aws.amazon.com/directconnect/details/; they might be already having a presence in your data centre.

Q3. OK – I got it. Now I want to connect to AWS DirectConnect. From where to start? What are the available options?
Ans: There you go! First thing is you need to submit a DirectConnect request through AWS Console; based on your request AWS will send you a Letter of Authorization – Connecting Facility Assignment (LOA-CFA) for your cross-connect to your rack the in the data centre; AWS will also allocate a network port/interface for you on their end for this cross-connect. The data centre guys will do the physical cross-connect cable run for you. Please go through question Q4 to Q8 to get an understanding of what type of physical connection you will be ordering.

Starting from here it’s all about the “same old IP networking”.

Part 2: Lets talk about with “Physical Connectivity”

Q4. What are the available physical connectivity options?
Ans: AWS DirectConnect comes in two different physical interface capacity – 1Gbps Ethernet and 10Gbps Ethernet.

Q5. What type of network cable do I need to use?
Ans: For both the 1Gbps and 10Gbps – it is “single mode fibre optic” cable.

Q6. There are so many optical fibre weave lengths and interfaces type; which one is compatible with AWS DirectConnect?
Ans: For the 1Gbps it is 1000BASE-LX (1310nm wavelength signal) and for 10Gbps it is 10GBASE-LR (1310nm wavelength). AWS do not use 1550nm.

Q7. What else need to know about the network interfaces?
Ans: The interface must support IEEE 802.1Q VLAN tagging. Actually you will be creating “tagged” logical interfaces based on the physical interface. This is just like connectivity between two Layer3 switches/router (one is your L3 switch/router – other one is AWS managed) so that you can have many logical VLAN interfaces. If you use layer 2 switch for this connectivity, you must need a router connecting to your switch and you deliver the same VLAN tag number to the router interface (the VLAN tag id you share with AWS).

Q8. I want to have more than one DirectConnect physical interfaces – can I use LAG? What are physical connectivity options to have more than one DirectConnect?
Ans: AWS recently started supporting LAG that use LACP. You can also use L3 ECMP (equal cost multi path – routing) load balancing & link failover with BFD (Bi-Directional Forward Detection for quick link fault detection). It’s your choice and business requirements, which one to choose “L2 LACP” or “L3 ECMP”. AWS LACP LAG is active/active solution.

Key items here:
-Physical interface – 1G and 10G
-Optical cable and connectors – single mode optical fibre; 1000BASE-LX and 10GBASE-LR; optical wavelength 1310nm.
-VLAN tagging 802.1Q
-LACP or ECMP

Part 3: Lets talk about IP Routing – DirectConnect Routing

Q9. Ok – now I know about the physical connectivity; what about the traffic forwarding and routing between AWS and self-managed network?
Ans: Regarding routing – AWS does support “ONLY” BGP. Since you are connecting to AWS which is a different ASN – the BGP type here is EBGP.

Q10. Can I connect to AWS hosted public resources such as S3 and other resources via the DirectConnect?
Ans: Yes off-course; while creating a DirectConnection virtual interface – you have two options to select; either (i)private or (ii)public. Private interface will only allow you access to your private AWS resources sitting within your VPC subnets – whereas public will allow you access to AWS hosted public resources such as S3 and RDS. You need to have routes to AWS public resources via DirectConnect (to the AWS public networks) which are not directly connected to the DirectConnect router; let’s say your edge router is connected to DirectConnect, so your core or dist routers should have routes to those AWS public via the edge router.

Q11. What do I do to see my BGP advertised networks in the VPC?
Ans: You need to create virtual private gateway (VGW) on AWS web admin console and attach it to your VPC. You can have one (01) VGW per VPC; your VGW is just a network routing virtual appliance that manages external IN/OUT traffic routes to your VPC. There is an option called “route propagation” within VPC routing table – turn this on, this will show all the routes propagated via BGP.

Q12. Can I have 2 x DirectConnect working together to have load balancing and redundancy?
Ans: Yes you can. You have two options – (i)one is LAG which use LACP active/active – this is a layer 2 solution, (ii)other option is Layer3 ECMP. Regarding ECMP, by default AWS use ECMP over BGP advertised router to send traffic across all the available active virtual interfaces (from AWS >> to you). Regarding sending traffic from your end to >> AWS, create your own ECMP policies via BGP advertised routes; this can be done using routing policies telling your router/firewall to use all the active virtual interfaces for the same destination (destination is AWS) IP subnets. AWS does support BFD (bidirectional forward detection) to provide fast network fault detection and convergence.

When using ECMP, one important thing is – if your DirectConnect terminating device is non-SPI (stateful packet inspection) packet based router – then they will send/receive packets from multiple interfaces without having any issue. However, if your DirectConnect terminating device is a session based firewall (SPI/stateful firewall) it will drop packets which does not match existing session table entries for return path (as return path might be the other firewall or might be other network interface which might not belongs to the same security zone). If both the AWS DirectConnect interfaces terminates on to a same SPI firewall – then put both the interfaces (one interface is via DirectConnect X, other one is via DirectConnect Y) on to the same security zone; in this case firewall SPI return traffic will get matched in the session table and will have no packet drop; if the packet return path is a different SPI firewall – then you need to turn off SPI on both the firewalls for AWS DirectConnect traffic.

To have redundancy only (no traffic load balance), you can use BGP “AS path prepend” feature to tell AWS BGP peer to send traffic (from AWS >> to your network) via your preferred path only.

I have designed & implemented 4 x DirectConnects connected to the same VPC resources using ECMP.

Q13. I want to advertise only selected internal IP subnets to AWS VPC – can I do this?
Ans: Yes of course. Setup your BGP to advertise selected local IP subnets only. This can be done using route filtering/routing policies. Always check and make sure you are advertising the correct IP subnets and receiving the correct advertised IP subnets.

Q14. Can I send my “Internet” traffic via DirectConnects to Internet?
Ans: AWS does not allow you to route to non-AWS Internet resources via DirectConnect; in other term, you cannot use AWS as an “intermediate AS” to route traffic to Internet.

Key items here:
-BGP and ASN
-AWS DirectConnect virtual interfaces – private interfaces & public interfaces – just like any other layer3 virtual interface where you can assign an IP address and use to route traffic to destinations.
-BGP route export & import, routing policies
-ECMP – equal cost multi path; load balancing and link failover across multiple L3 links
-BFD – provide fast failure detection
-BGP AS path prepend
-AWS VPC – a virtual private network boundary which use a larger CIDR block that you divided into many smaller IP networks.
-AWS Virtual Private Gateway (VGW) – a network routing appliance which manage traffic IN/OUT, static route and BGP dynamic route.
-AWS VPC Routing Table & route propagation – just another routing table; consists of ip subnets as destinations networks and use IGW/NATGW as gateway exit path.

Part 4: Let’s talk about routing within AWS VPC

Q15. I don’t want all my VPC subnets (within AWS) to have route to self-managed network subnets?
Ans: Sure you can do this. First, create “subnets” those you don’t want to a have route to self-managed network > then create “routing table” and make sure “route propagation” is turned off – “associate” the same subnet here. You can do all these from the AWS web admin console.

Q16. I wants to have my VPC resources sitting on more than one AWS availability zones for maximum high availability?
Ans: Sure you can get this done via having distributed IP subnets across multiple availability zones. When creating a subnet, you can specify (i)which VPC the subnet is attached to (ii)set your preferred AWS “availability zone” where the subnet must reside. When creating AWS resources such as EC2, ELB; attach subnets based on availability zone thus provides greater high availability.

Q17. I want to send traffic to Internet from my VPC – how can I do that?
Ans: There are two different ways you can get this done; (i)one is attach an IGW to your VPC subnet routing table – this will enable both inbound and outbound traffic for all the subnets attached to the routing table (ii)if you only want outbound (NATed traffic) – then this can be done via a NATGW (subnet users send traffic to Internet via NAT GW), however the NAT GW needs to send traffic to Internet via an IGW. You should have NAT GW per availability zones.

Key items here:
-AWS VPC subnets
-AWS VPC routing table
-AWS Availability zones
-AWS Internet Gateway (IGW)
-AWS NAT Gateway (NATGW)

Part 5: How to administer/control AWS networking

Q18. What are the available tools to manage/administer AWS networking?
Ans: This is very simple; initially you better use the AWS Web Console, once you get a good visualisation of AWS components/products, then start using AWS command line tool (AWS CLI) and AWS APIs. As a Network Engineer probably you already know a hell lot of command syntaxes, so you will find AWS CLI much easier.

Key items here:
-AWS Web Console
-AWS CLI
-AWS APIs