logo-dark

How to inspect & manipulate network packets in AWS with Gateway Load Balancer

Table of contents

Blogs· 5min May 11, 2023

In this post, Michał walks you through a sample setup of the AWS Gateway Load Balancer. We will provision the infrastructure using Terraform, write a simple virtual appliance application and show it all in action. He demonstrates how this service can be used to route network traffic through a virtual appliance where each network packet can be inspected, modified, or dropped.

This AWS Gateway Load Balancer service is very well described in the Official getting started guide. All the code demonstrated in this post can be found in the aws-gateway-lb GitHub repository.

Note, that running the example will incur some costs. Remember, to destroy the infrastructure after playing around with it!

Infrastructure

Sample infrastructure can be provisioned using Terraform by following the readme in the code repository.

The key resources are:

  • AWS EC2 instances a, b and c. These are ones where we will run nc and execute some simple message exchanges over UDP.
    undefinedundefined
  • AWS EC2 appliance instance where our virtual appliance application will run.

All instances allow public SSH access on port 22. With some simple adjustments it can be restricted to your own public IP, but it's out of scope of this post. See the repository readme to learn how to provide the public key.

Virtual appliance

A virtual appliance is an application that supports Geneve (Generic Network Virtualisation Encapsulation) protocol and exposes a health check endpoint. It's possible to get one from the AWS Marketplace, but for the purpose of this post, we will write our own. In short, the appliance has to:

  • receive a UDP packet,
  • swap source and destination IP in the outermost IP layer,
  • optionally modify the packet’s contents and update the checksum,
  • send the packet back or drop the packet.

In this section, I will explain the steps necessary to create a virtual appliance. Full source code (along with instructions on how to run it) is accessible in the GitHub repository mentioned above.

The appliance will handle all Geneve packets, decode them using the gopacket library, and process them according to the following rules.

UDP packets where the source or destination port is 3000 will be handled as follows

  • if the payload contains the string "drop me" the packet will be dropped;
  • string "weakly typed" in the payload will be replaced with string "strongly typed".

Additionally, every 5th ICMP packet will be dropped.

The reason we've chosen ICMP and UDP is that I wanted to show how to handle various protocols and to emphasise that we're not limited to TCP or UDP only. We've chosen UDP over TCP as it's easier to show dropping packets. By design, if we drop a single TCP packet, we won't be able to process any subsequent ones. Therefore dropping a subset of packets is much easier to show with UDP.

For brevity, we will support IPv4 only and skip error handling (although the application in the repository handles errors).

Capturing packets

Packets that the virtual appliance has to handle will be encapsulated using Geneve and transferred over UDP. This means, that each received packet will begin with the following layers:

  • outer IP header,
  • outer UDP header,
  • Geneve header.

After these 3 layers, the encapsulated packet's layers will follow.

The virtual appliance has to swap source and destination IP addresses in the outer IP header and update the checksum. To properly implement this, we need to capture raw UDP packets so we get access to all the layers mentioned above.

To create a socket from which we can read raw packets we would call unix.Socket as follows.

fd, err := unix.Socket(unix.AF_INET, unix.SOCK_RAW, unix.IPPROTO_UDP)

We have to preserve other parts (except the checksum) of the outer layers. By default, when sending a packet the IP header would be generated for us. Since we will provide the outer IP header ourselves we have to set the IP_HDRINCL socket option.

err = unix.SetsockoptInt(fd, unix.IPPROTO_IP, unix.IP_HDRINCL, 1)

We also have to update the outer IP header's checksum. Fortunately, it's handled automatically.

Decoding packets

After the socket is created, we can start receiving packets from it.

buffer := make([]byte, 8500)
length, raddr, err := unix.Recvfrom(fd, buffer, 0)

We can then decode the packet using the gopacket library.

p := gopacket.NewPacket(buffer[:length], layers.LayerTypeIPv4, gopacket.Default)
packetLayers := p.Layers()

Finally, we get access to all of the packet's layers.

// access the outer IP layer
packetLayers[0].(*layers.IPv4)

// access the outer UDP layer
packetLayers[1].(*layers.UDP)

Dropping packets

Dropping packets is super simple. We just don't send anything back and stop processing the packet.

Modifying packets

Modifying packets requires a little bit more work, as we have to access inner layers. And, after a packet is modified the checksum has to be recalculated.

We will attempt to modify UDP packets only. This means, that packets interesting to us will contain the following layers:

  • outer IP,
  • outer UDP,
  • Geneve,
  • inner IP,
  • inner UDP,
  • payload.

Since the checksums of both inner UDP and IPv4 layers depend on the payload, we can't just modify the payload. We have to also recalculate the checksums. UDP checksum depends on the IPv4 header, therefore we have to explicitly set the correct IP layer in the UDP layer to the one that will be used later for the checksum calculation.

type PayloadModifyFun func([]byte) []byte

func (p *Packet) ModifyUDP(f PayloadModifyFun) {
 // get the inner layers
 ip := p.packetLayers[3].(*layers.IPv4)
 udp := p.packetLayers[4].(*layers.UDP)
 payload := p.packetLayers[5].(*gopacket.Payload)
 p.modified = true
  
 // udp checksum depends on IPv4 layer. Therefore, we need to provide a layer that will be used for checksum calculation.
 udp.SetNetworkLayerForChecksum(ip)

   // update the payload
   p.packetLayers[5] = gopacket.Payload(f(payload.Payload()))
}

Serialising packets and sending them back

Before we send the packet back, we have to swap the source and destination IP in the outer IP layer. This is quite simple.

func (p *Packet) SwapSrcDstIpv4() {
 ip, _ := p.packetLayers[0].(*layers.IPv4)
 dst := ip.DstIP
 ip.DstIP = ip.SrcIP
 ip.SrcIP = dst
}

After IP addresses have been swapped, we are ready to serialise all the layers (in reverse order). In cases where the payload has been modified, we have to additionally recompute checksums as mentioned above.

func (p *Packet) Serialize() []byte {
 buf := gopacket.NewSerializeBuffer()
 for i := len(p.packetLayers) - 1; i >= 0; i-- {
  if layer, ok := p.packetLayers[i].(gopacket.SerializableLayer); ok {
   var opts gopacket.SerializeOptions

   // recompute checksum of inner IP and UDP layers in case the packet was modified
   if p.modified && (i == p.insideUDPLayerIdx() || i == p.insideIPLayerIdx()) {
    opts = gopacket.SerializeOptions{ComputeChecksums: true, FixLengths: true}
   } else {
    opts = gopacket.SerializeOptions{FixLengths: true}
   }

   layer.SerializeTo(buf, opts)
   buf.PushLayer(layer.LayerType())
  } else if layer, ok := p.packetLayers[i].(*layers.Geneve); ok {
   bytes, _ := buf.PrependBytes(len(layer.Contents))
   copy(bytes, layer.Contents)
  } else {
   return nil
  }
 }
  return buf.Bytes()
}

Finally, we can send the packet back.

unix.Sendto(fd, response, 0, raddr)

Demo

At the very beginning, we have to provision the infrastructure.

./deploy_infra.sh
./deploy_censor.sh

After all the required resources have been created, in a new terminal, we need to install the required packages on the provisioned instances.

./init_infra.sh

Next, in two separate terminals, we can connect to instances a and b. On instance a we start to listen on UDP port 3000 and on instance b we connect to instance a (note, that in your case private IP addresses will be different so please adjust the commands below).

./ssh.sh a

[ec2-user@ip-192-168-1-209 ~]$ nc -l -u 3000
./ssh.sh b

[ec2-user@ip-192-168-2-106 ~]$ nc -u 192.168.1.209 3000
test
drop me
weakly typed programming language

In instance a we would receive the following.

test
strongly typed programming language

We expect messages containing "drop me" aren't delivered and "weakly typed" in the message is replaced with "strongly typed". The above example confirms that everything works as expected.

By pinging instance a from instance b we notice that as expected, every 5th ICMP packet is dropped.

[ec2-user@ip-192-168-2-106 ~]$ ping 192.168.1.209
PING 192.168.1.209 (192.168.1.209) 56(84) bytes of data.
64 bytes from 192.168.1.209: icmp_seq=1 ttl=253 time=3.96 ms
64 bytes from 192.168.1.209: icmp_seq=2 ttl=253 time=1.58 ms
64 bytes from 192.168.1.209: icmp_seq=3 ttl=253 time=1.51 ms
64 bytes from 192.168.1.209: icmp_seq=4 ttl=253 time=1.51 ms
64 bytes from 192.168.1.209: icmp_seq=6 ttl=253 time=1.64 ms
64 bytes from 192.168.1.209: icmp_seq=7 ttl=253 time=2.07 ms
64 bytes from 192.168.1.209: icmp_seq=8 ttl=253 time=1.87 ms
64 bytes from 192.168.1.209: icmp_seq=9 ttl=253 time=3.36 ms
64 bytes from 192.168.1.209: icmp_seq=11 ttl=253 time=1.84 ms

Eventually, when we're done we can destroy the infrastructure so we don't spend too much $.

./destroy_infra.sh

Conclusions

We've covered quite a bit in this post:

  • what AWS Gateway Load Balancer is,
  • how to capture and process raw network packets,
  • how to implement a virtual appliance,
  • presented how it all works.

While this was a pretty simple example, it's a valuable starting point for more advanced applications of this AWS service.

Further reading

Written by

Michał Szczygieł
github-icongithub-icongithub-icon
Michał Szczygieł Senior Software Engineer

Michał Szczygieł is Senior Software Engineer at Form3. His experience varies from creating native Windows applications with Delphi to developing complex backend systems with functional Scala. Recently he's started to explore the Go programming language and its ecosystem.

Further resources

Blogs · 10 min

Keeping Customers Informed During Incidents with Prometheus

Maintaining customer satisfaction during incidents is crucial for any business. In this blogpost, Piotr shares how we leverage Prometheus to expose business metrics in a secure and cost-effective way to keep customers informed and happy during those stressful situations.

May 24, 2023

Blogs · 4 min

.tech Podcast - The core components of container systems

Michael Kerrisk is a Linux expert and trainer. He joins us to explain what containers are and deep dive into the four core components of containers: namespaces, capabilities, cgroups and seccomp. He also draws parallels on how they are used by Docker to power container systems as we know them today.

May 17, 2023

Blogs · 6 min

Secrets management in build and deployment pipelines

In his previous post about bootstrapping engineering organisations, Andy Kuszyk identified secrets management as one of the key challenges that need to be tackled early on in growing organisations. In this post, he dives deeper into this topic and discusses two secrets management patterns: injecting secrets with Terraform and issuing them with a central secrets manager.

May 4, 2023