Table of contents
Blogs· 5min May 11, 2023
This AWS Gateway Load Balancer service is very well described in the Official getting started guide. All the code demonstrated in this post can be found in the aws-gateway-lb GitHub repository.
Note, that running the example will incur some costs. Remember, to destroy the infrastructure after playing around with it!
Sample infrastructure can be provisioned using Terraform by following the readme in the code repository.
The key resources are:
All instances allow public SSH access on port 22. With some simple adjustments it can be restricted to your own public IP, but it's out of scope of this post. See the repository readme to learn how to provide the public key.
A virtual appliance is an application that supports Geneve (Generic Network Virtualisation Encapsulation) protocol and exposes a health check endpoint. It's possible to get one from the AWS Marketplace, but for the purpose of this post, we will write our own. In short, the appliance has to:
In this section, I will explain the steps necessary to create a virtual appliance. Full source code (along with instructions on how to run it) is accessible in the GitHub repository mentioned above.
The appliance will handle all Geneve packets, decode them using the gopacket library, and process them according to the following rules.
UDP packets where the source or destination port is 3000 will be handled as follows
Additionally, every 5th ICMP packet will be dropped.
The reason we've chosen ICMP and UDP is that I wanted to show how to handle various protocols and to emphasise that we're not limited to TCP or UDP only. We've chosen UDP over TCP as it's easier to show dropping packets. By design, if we drop a single TCP packet, we won't be able to process any subsequent ones. Therefore dropping a subset of packets is much easier to show with UDP.
For brevity, we will support IPv4 only and skip error handling (although the application in the repository handles errors).
Packets that the virtual appliance has to handle will be encapsulated using Geneve and transferred over UDP. This means, that each received packet will begin with the following layers:
After these 3 layers, the encapsulated packet's layers will follow.
The virtual appliance has to swap source and destination IP addresses in the outer IP header and update the checksum. To properly implement this, we need to capture raw UDP packets so we get access to all the layers mentioned above.
To create a socket from which we can read raw packets we would call unix.Socket as follows.
fd, err := unix.Socket(unix.AF_INET, unix.SOCK_RAW, unix.IPPROTO_UDP)
We have to preserve other parts (except the checksum) of the outer layers. By default, when sending a packet the IP header would be generated for us. Since we will provide the outer IP header ourselves we have to set the IP_HDRINCL socket option.
err = unix.SetsockoptInt(fd, unix.IPPROTO_IP, unix.IP_HDRINCL, 1)
We also have to update the outer IP header's checksum. Fortunately, it's handled automatically.
After the socket is created, we can start receiving packets from it.
buffer := make([]byte, 8500)
length, raddr, err := unix.Recvfrom(fd, buffer, 0)
We can then decode the packet using the gopacket library.
p := gopacket.NewPacket(buffer[:length], layers.LayerTypeIPv4, gopacket.Default)
packetLayers := p.Layers()
Finally, we get access to all of the packet's layers.
// access the outer IP layer
packetLayers[0].(*layers.IPv4)
// access the outer UDP layer
packetLayers[1].(*layers.UDP)
Dropping packets is super simple. We just don't send anything back and stop processing the packet.
Modifying packets requires a little bit more work, as we have to access inner layers. And, after a packet is modified the checksum has to be recalculated.
We will attempt to modify UDP packets only. This means, that packets interesting to us will contain the following layers:
Since the checksums of both inner UDP and IPv4 layers depend on the payload, we can't just modify the payload. We have to also recalculate the checksums. UDP checksum depends on the IPv4 header, therefore we have to explicitly set the correct IP layer in the UDP layer to the one that will be used later for the checksum calculation.
type PayloadModifyFun func([]byte) []byte
func (p *Packet) ModifyUDP(f PayloadModifyFun) {
// get the inner layers
ip := p.packetLayers[3].(*layers.IPv4)
udp := p.packetLayers[4].(*layers.UDP)
payload := p.packetLayers[5].(*gopacket.Payload)
p.modified = true
// udp checksum depends on IPv4 layer. Therefore, we need to provide a layer that will be used for checksum calculation.
udp.SetNetworkLayerForChecksum(ip)
// update the payload
p.packetLayers[5] = gopacket.Payload(f(payload.Payload()))
}
Before we send the packet back, we have to swap the source and destination IP in the outer IP layer. This is quite simple.
func (p *Packet) SwapSrcDstIpv4() {
ip, _ := p.packetLayers[0].(*layers.IPv4)
dst := ip.DstIP
ip.DstIP = ip.SrcIP
ip.SrcIP = dst
}
After IP addresses have been swapped, we are ready to serialise all the layers (in reverse order). In cases where the payload has been modified, we have to additionally recompute checksums as mentioned above.
func (p *Packet) Serialize() []byte {
buf := gopacket.NewSerializeBuffer()
for i := len(p.packetLayers) - 1; i >= 0; i-- {
if layer, ok := p.packetLayers[i].(gopacket.SerializableLayer); ok {
var opts gopacket.SerializeOptions
// recompute checksum of inner IP and UDP layers in case the packet was modified
if p.modified && (i == p.insideUDPLayerIdx() || i == p.insideIPLayerIdx()) {
opts = gopacket.SerializeOptions{ComputeChecksums: true, FixLengths: true}
} else {
opts = gopacket.SerializeOptions{FixLengths: true}
}
layer.SerializeTo(buf, opts)
buf.PushLayer(layer.LayerType())
} else if layer, ok := p.packetLayers[i].(*layers.Geneve); ok {
bytes, _ := buf.PrependBytes(len(layer.Contents))
copy(bytes, layer.Contents)
} else {
return nil
}
}
return buf.Bytes()
}
Finally, we can send the packet back.
unix.Sendto(fd, response, 0, raddr)
At the very beginning, we have to provision the infrastructure.
./deploy_infra.sh
./deploy_censor.sh
After all the required resources have been created, in a new terminal, we need to install the required packages on the provisioned instances.
./init_infra.sh
Next, in two separate terminals, we can connect to instances a and b. On instance a we start to listen on UDP port 3000 and on instance b we connect to instance a (note, that in your case private IP addresses will be different so please adjust the commands below).
./ssh.sh a
[ec2-user@ip-192-168-1-209 ~]$ nc -l -u 3000
./ssh.sh b
[ec2-user@ip-192-168-2-106 ~]$ nc -u 192.168.1.209 3000
test
drop me
weakly typed programming language
In instance a we would receive the following.
test
strongly typed programming language
We expect messages containing "drop me" aren't delivered and "weakly typed" in the message is replaced with "strongly typed". The above example confirms that everything works as expected.
By pinging instance a from instance b we notice that as expected, every 5th ICMP packet is dropped.
[ec2-user@ip-192-168-2-106 ~]$ ping 192.168.1.209
PING 192.168.1.209 (192.168.1.209) 56(84) bytes of data.
64 bytes from 192.168.1.209: icmp_seq=1 ttl=253 time=3.96 ms
64 bytes from 192.168.1.209: icmp_seq=2 ttl=253 time=1.58 ms
64 bytes from 192.168.1.209: icmp_seq=3 ttl=253 time=1.51 ms
64 bytes from 192.168.1.209: icmp_seq=4 ttl=253 time=1.51 ms
64 bytes from 192.168.1.209: icmp_seq=6 ttl=253 time=1.64 ms
64 bytes from 192.168.1.209: icmp_seq=7 ttl=253 time=2.07 ms
64 bytes from 192.168.1.209: icmp_seq=8 ttl=253 time=1.87 ms
64 bytes from 192.168.1.209: icmp_seq=9 ttl=253 time=3.36 ms
64 bytes from 192.168.1.209: icmp_seq=11 ttl=253 time=1.84 ms
Eventually, when we're done we can destroy the infrastructure so we don't spend too much $.
./destroy_infra.sh
We've covered quite a bit in this post:
While this was a pretty simple example, it's a valuable starting point for more advanced applications of this AWS service.
Written by
Michał Szczygieł is Senior Software Engineer at Form3. His experience varies from creating native Windows applications with Delphi to developing complex backend systems with functional Scala. Recently he's started to explore the Go programming language and its ecosystem.
Blogs · 10 min
Maintaining customer satisfaction during incidents is crucial for any business. In this blogpost, Piotr shares how we leverage Prometheus to expose business metrics in a secure and cost-effective way to keep customers informed and happy during those stressful situations.
May 24, 2023
Blogs · 4 min
Michael Kerrisk is a Linux expert and trainer. He joins us to explain what containers are and deep dive into the four core components of containers: namespaces, capabilities, cgroups and seccomp. He also draws parallels on how they are used by Docker to power container systems as we know them today.
May 17, 2023
Blogs · 6 min
In his previous post about bootstrapping engineering organisations, Andy Kuszyk identified secrets management as one of the key challenges that need to be tackled early on in growing organisations. In this post, he dives deeper into this topic and discusses two secrets management patterns: injecting secrets with Terraform and issuing them with a central secrets manager.
May 4, 2023