a FAUCET dev blog

https://blog.cyberreboot.org/sdn-all-the-way-down-an-introduction-to-pipette-69109830cf82

How many people use FAUCET? This is a difficult question to answer; there’s no requirement that FAUCET users tell the maintainers.

However, there are a few ways we can see how many people download FAUCET – since you can install FAUCET a few different ways. These counts are known to be missing some historical data and so are on the low side (in particular, pypi’s accounting system changed, and pypi download counts are only available from October 2019).

• https://pypistats.org/packages/faucet (daily downloads over the most recent few months)
• https://github.com/faucetsdn/faucet/graphs/traffic (visitors, git clones, etc of the FAUCET code base)
• wget -q -O- https://hub.docker.com/v2/repositories/faucet/faucet/|jq .pull_count (total pull count for the faucet/faucet image)

I’ll be adding a dashboard to correlate this information ongoing. As of writing this post, there have been over 74k docker pulls, and 35k pip installs.

In part II we demonstrated faking IP services for an entire subnet. Now, let’s eliminate the need for an interactive OpenFlow control entirely, with the magic OVS learn actions. These are almost like OVS assembly language instructions, that allow OVS rewrite packets or even add flows to itself. Once new pipette has programmed OVS, once, OVS can respond to ARP requests and do the L2/L3 NAT translation entirely by itself.

This means we don’t have to rely on the controller being up to handle new connections, and connection handling is fast.

Here’s how OVS looks after it has just been programmed by pipette:

# ovs-ofctl -OOpenFlow13 dump-flows copro0
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,in_port=enx0023565c8859,vlan_tci=0x1000/0x1000 actions=pop_vlan,goto_table:1
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,tcp,in_port=enx0023565c8859 actions=goto_table:1
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,udp,in_port=enx0023565c8859 actions=goto_table:1
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,tcp,in_port=ovsfake0 actions=push_vlan:0x8100,set_field:4098->vlan_vid,goto_table:2
 cookie=0x0, duration=6.758s, table=0, n_packets=0, n_bytes=0, priority=1,udp,in_port=ovsfake0 actions=push_vlan:0x8100,set_field:4098->vlan_vid,goto_table:2
 cookie=0x0, duration=6.757s, table=0, n_packets=0, n_bytes=0, priority=1,arp,in_port=ovsfake0,dl_src=0e:00:00:00:00:66 actions=goto_table:2
 cookie=0x0, duration=6.759s, table=0, n_packets=6, n_bytes=621, priority=0 actions=drop
 cookie=0x0, duration=6.758s, table=1, n_packets=0, n_bytes=0, priority=1,tcp actions=move:NXM_OF_IP_SRC[]->NXM_NX_REG0[],load:0xa0a->NXM_NX_REG0[16..31],move:NXM_NX_REG0[0..7]->NXM_NX_REG0[8..15],move:NXM_OF_IP_DST[0..7]->NXM_NX_REG0[0..7],load:0x2->NXM_NX_REG1[0..2],load:0x1->NXM_NX_REG2[0..2],learn(table=1,hard_timeout=300,priority=2,eth_type=0x800,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],load:NXM_NX_REG0[]->NXM_OF_IP_SRC[],load:0xa0a0001->NXM_OF_IP_DST[],load:0xe0000000067->NXM_OF_ETH_SRC[],load:0xe0000000066->NXM_OF_ETH_DST[],output:NXM_NX_REG1[0..1]),learn(table=2,idle_timeout=300,priority=2,eth_type=0x800,ip_src=10.10.0.1,NXM_OF_IP_DST[]=NXM_NX_REG0[],load:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],load:NXM_OF_ETH_DST[]->NXM_OF_ETH_SRC[],load:NXM_OF_IP_SRC[]->NXM_OF_IP_DST[],load:NXM_OF_IP_DST[]->NXM_OF_IP_SRC[],output:NXM_NX_REG2[0..1]),set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,move:NXM_NX_REG0[]->NXM_OF_IP_SRC[],set_field:10.10.0.1->ip_dst,output:ovsfake0
 cookie=0x0, duration=6.758s, table=1, n_packets=0, n_bytes=0, priority=1,udp actions=move:NXM_OF_IP_SRC[]->NXM_NX_REG0[],load:0xa0a->NXM_NX_REG0[16..31],move:NXM_NX_REG0[0..7]->NXM_NX_REG0[8..15],move:NXM_OF_IP_DST[0..7]->NXM_NX_REG0[0..7],load:0x2->NXM_NX_REG1[0..2],load:0x1->NXM_NX_REG2[0..2],learn(table=1,hard_timeout=300,priority=2,eth_type=0x800,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],load:NXM_NX_REG0[]->NXM_OF_IP_SRC[],load:0xa0a0001->NXM_OF_IP_DST[],load:0xe0000000067->NXM_OF_ETH_SRC[],load:0xe0000000066->NXM_OF_ETH_DST[],output:NXM_NX_REG1[0..1]),learn(table=2,idle_timeout=300,priority=2,eth_type=0x800,ip_src=10.10.0.1,NXM_OF_IP_DST[]=NXM_NX_REG0[],load:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],load:NXM_OF_ETH_DST[]->NXM_OF_ETH_SRC[],load:NXM_OF_IP_SRC[]->NXM_OF_IP_DST[],load:NXM_OF_IP_DST[]->NXM_OF_IP_SRC[],output:NXM_NX_REG2[0..1]),set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,move:NXM_NX_REG0[]->NXM_OF_IP_SRC[],set_field:10.10.0.1->ip_dst,output:ovsfake0
 cookie=0x0, duration=6.759s, table=1, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6.758s, table=2, n_packets=0, n_bytes=0, priority=1,arp,arp_op=1 actions=move:NXM_OF_ARP_TPA[]->NXM_NX_REG0[],load:0x2->NXM_OF_ARP_OP[0..2],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],move:NXM_NX_REG0[]->NXM_OF_ARP_SPA[],set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:67->arp_sha,IN_PORT
 cookie=0x0, duration=6.758s, table=2, n_packets=0, n_bytes=0, priority=0 actions=drop

While scary looking, the flows are mostly just rearranging fields. Let’s look at the flow that responds to ARP:

 cookie=0x0, duration=6.758s, table=2, n_packets=0, n_bytes=0, priority=1,arp,arp_op=1 actions=move:NXM_OF_ARP_TPA[]->NXM_NX_REG0[],load:0x2->NXM_OF_ARP_OP[0..2],move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],move:NXM_NX_REG0[]->NXM_OF_ARP_SPA[],set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:67->arp_sha,IN_PORT

This flow matches ARP requests. Then, it turns the ARP packet into an ARP reply, rewrites source hardware addresses, and rewrites IP addresses (we temporarily store the ARP target address from the request in OVS’ software register reg0, so it doesn’t get overwritten and so we can use it at the end of the rewriting process). Finally, we say to output the packet on the same port it came in on.

Now let’s look at the really scary flow that implements learning:

cookie=0x0, duration=6.758s, table=1, n_packets=0, n_bytes=0, priority=1,tcp actions=move:NXM_OF_IP_SRC[]->NXM_NX_REG0[],load:0xa0a->NXM_NX_REG0[16..31],move:NXM_NX_REG0[0..7]->NXM_NX_REG0[8..15],move:NXM_OF_IP_DST[0..7]->NXM_NX_REG0[0..7],load:0x2->NXM_NX_REG1[0..2],load:0x1->NXM_NX_REG2[0..2],learn(table=1,hard_timeout=300,priority=2,eth_type=0x800,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],load:NXM_NX_REG0[]->NXM_OF_IP_SRC[],load:0xa0a0001->NXM_OF_IP_DST[],load:0xe0000000067->NXM_OF_ETH_SRC[],load:0xe0000000066->NXM_OF_ETH_DST[],output:NXM_NX_REG1[0..1]),learn(table=2,idle_timeout=300,priority=2,eth_type=0x800,ip_src=10.10.0.1,NXM_OF_IP_DST[]=NXM_NX_REG0[],load:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],load:NXM_OF_ETH_DST[]->NXM_OF_ETH_SRC[],load:NXM_OF_IP_SRC[]->NXM_OF_IP_DST[],load:NXM_OF_IP_DST[]->NXM_OF_IP_SRC[],output:NXM_NX_REG2[0..1]),set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,move:NXM_NX_REG0[]->NXM_OF_IP_SRC[],set_field:10.10.0.1->ip_dst,output:ovsfake0

First, we calculate the source NAT address and put it in reg0. Then we have two learn() sections – one adds a flow to table 1 to implement inbound translation, and the other adds a flow to table 2 for outbound. Finally, we have to do NAT translation on the packet we just got and send it.

If you’re like me – you live in the middle of nowhere, in New Zealand, without fiber let alone any DSL, and you develop a lot of software using docker – you might find your build/deploy cycle a bit painful, since many docker builds download packages.

Even if you are an unusual case – you live in an urban area with good espresso on every corner, scarce parking, and fiber to the couch (complete with 5G) – retrieving lots of packages from disk/SSD is still faster than fiber.

Running a docker pull through cache can help if you have a few machines pulling the same images. However, you sometimes you still need to actually build those images locally, and the build process can involve a lot of apt-get’ing or apk add’ing.

To help with package retrieval over HTTP, you can run apt-cacher-ng. That’s great if you have a way to convince your docker build to use the cache (e.g. via an environment variable). However it’d be even nicer not to have to do anything special.

The missing piece in the puzzle, is a transparent HTTP proxy (like squid), that knows how to redirect requests to apt-cacher-ng. This is what I do and it makes docker builds really fast (at least, the package retrieval part). Using a relatively old squid redirector, jesred, squid intercepts common package retrieval URLs and pass them to apt-cacher-ng.

Here’s part of my /etc/jesred.conf:

regex ^http://((.*)archive.ubuntu.com/ubuntu/(dists|pool)/.*)$    http://localho
st:3142/\1
regex ^http://(security.ubuntu.com/ubuntu/(dists|pool)/.*)$    http://localhost:
3142/\1
regex ^http://(dl-cdn.alpinelinux.org/alpine.+)$    http://localhost:3142/\1
regex ^http://(.*cdn.*.debian.org/.+)$	http://localhost:3142/\1
regex ^http://(deb.debian.org/.+)$  http://localhost:3142/\1
regex ^http://(archive.raspberrypi.org/debian/.+)$    http://localhost:3142/\1

Of course, your squid installation has to be set up to do transparent caching on port 80, and has to reference jesred in /etc/squid/squid.conf:

redirect_program /usr/lib/squid/jesred

In our last post, we used a new L2 OVS proxy to fake TCP services on an entire IP subnet. We used network namespaces for isolation. However, using namespaces can make running fake services under docker somewhat inconvenient. So this time around, we’re going to have pipette do L3 NAT for us as well, so we can just attach services as we please. We’re going to fake a webserver on port 80 (as docker based webservers are easy to come by).

We are using the same physical setup as last time. However, we are now using a slightly different FAUCET ACL. We are going to work with TCP port 80 only, and only on 192.168.2.0/24. Note that we are also instructing FAUCET to always add a VLAN VID tag when it dispatches the intercepted traffic to the coprocessor (not strictly necessary but does make things more consistent).

acls:  
  coprocessssh:
  - rule:
      dl_type: 0x800
      ip_proto: 6
      ipv4_src: 192.168.2.0/24
      ipv4_dst: 192.168.2.0/24
      tcp_dst: 80
      actions:
        output:
          vlan_vid: 2
          ports: [18]
  - rule:
      actions:
        allow: 1

We’ll also need a new version of pipette – this one is a bit smarter – it can do L3 NAT. Specifically, it will NAT 192.168.2.0/24 to 192.168.101.0/24 on the coprocessor. That way we can just run services that listen on 192.168.101.1/24, and they’ll appear in the real 192.168.2.0/24 network, magically with the right MAC addresses.

We’ll start pipette like this:

#!/bin/bash

# interface connected to FAUCET coprocessor port.
COPROINT=enx0023565c8859
# interface that will be created for fake services to run on.
FAKEINT=fake0
# Reserved MAC addresses for fake services to use to talk to clients.
FAKEHW=0e:00:00:00:00:66
FAKECLIENTHW=0e:00:00:00:00:67
# address fake services will be run on (will be proxied from real IPs)
FAKEIP=192.168.101.1/24
# OVS bridge name
BR=copro0
# pipette OF port
OF=6699

# Configure pipette's OVS switch.
# Remove all IP addresses, disable IPv6.
ip link add dev $FAKEINT type veth peer name ovs$FAKEINT
for i in $COPROINT $FAKEINT ovs$FAKEINT ovs-system ; do
  echo 1 > /proc/sys/net/ipv6/conf/$i/disable_ipv6
  ifconfig $i 0.0.0.0
done
ifconfig $COPROINT up
ifconfig ovs$FAKEINT up
ifconfig $FAKEINT hw ether $FAKEHW $FAKEIP up
ovs-vsctl del-br $BR 
ovs-vsctl add-br $BR
ovs-ofctl del-flows $BR
for i in $COPROINT ovs$FAKEINT ; do
  ovs-vsctl add-port $BR $i
done
ovs-vsctl set-controller $BR tcp:127.0.0.1:$OF

# Run pipette.
ryu-manager pipette.py --ofp-tcp-listen-port $OF  --verbose

Now, we can start a webserver on 192.168.101.1:

pi@coprocessor:~ $ docker run -d -p 192.168.101.1:80:80 hypriot/rpi-busybox-httpd
89425b1198b3eb30267f05fa10bc3691efc7b36710297e4695030705ee09c9bb
pi@coprocessor:~ $ docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                      NAMES
89425b1198b3        hypriot/rpi-busybox-httpd   "/bin/busybox httpd …"   11 seconds ago      Up 9 seconds        192.168.101.1:80->80/tcp   keen_goldberg

Now all it remains for us to do, is from outside the coprocessor, try to access the webserver “allegedly” running on 192.168.2.1:

pi@pi8021x:~ $ wget -q -O- 192.168.2.1 80
<html>
<head><title>Pi armed with Docker by Hypriot</title>
  <body style="width: 100%; background-color: black;">
    <div id="main" style="margin: 100px auto 0 auto; width: 800px;">
      <img src="pi_armed_with_docker.jpg" alt="pi armed with docker" style="width: 800px">
    </div>
  </body>
</html>

With pipette now doing L3 NAT, we can start pretty much whatever services we like listening on fake0, and provided we use the right FAUCET ACL to intercept the traffic, those services will appear in the dataplane.

In a previous post, we introduced coprocessing and faking a TCP service on a single IP. The FAUCET coprocessor feature allows an external host to inject packets to a given VLAN (or even port). FAUCET ACLs are used to select traffic to be sent to the coprocessor (by default nothing is sent for coprocessing).

In this post we will develop the idea further – we will fake a TCP service on all hosts in a subnet, and we will do away with the need to deal with ARP.

Consider this topology. The two real hosts are at the bottom. We want to cause, from 192.168.2.5’s point of view, 192.168.2.1 to have an SMTP service it doesn’t really have. Further, from 192.168.2.5’s point of view, we want any host within 192.168.2.0/24 to appear to have that service.

The first step is to have FAUCET apply a port ACL to 192.168.2.5’s port, saying we want TCP port 25 (we’re faking SMTP) for all of 192.168.2.0/24 to be coprocessed. We could also use a VLAN ACL (pipette would need a small modification to allow it to receive tagged VLAN packets in that case).

Port 13 will have the host with 192.168.2.5 – port 18 is where the coprocessor is connected.

vlans:
  trusted:
    vid: 2
interfaces:
  13:
    acls_in:
      - coprocessssh
    description: 802.1x Pi
    native_vlan: trusted
  18:
    coprocessor:
      strategy: vlan_vid
    description: coprocessor NFV
acls:
  coprocessssh:
  - rule:
      dl_type: 0x800
      ip_proto: 6
      ipv4_src: 192.168.2.0/24
      tcp_dst: 25
      actions:
        output:
          ports: [18]
  - rule:
      actions:
        allow: 1

Now, on the coprocessor, we’ll need OVS and Ryu installed, and pipette. pipette patches TCP traffic in and out of a namespace where the fake service will run while maintaining the L2 illusion that the fake service is running on the real host.

# interface connected to FAUCET switch
NFVINT=enx0023565c8859
# used privately by the fake service.
FAKEHW=0e:00:00:00:00:66
FAKECLIENTHW=0e:00:00:00:00:67
# all 192.168.2.0/24 port 25 will be redirected to this fake IP.
FAKEIP=192.168.2.1/24
NS=copro

ip netns add $NS
ifconfig $NFVINT up
ip link add dev fake0 type veth peer name fakeovs0
ifconfig fakeovs0 up
ip link set fake0 netns $NS
ip netns exec $NS ifconfig fake0 hw ether $FAKEHW $FAKEIP up
for i in $NFVINT fakeovs0 ovs-system ; do
  echo 1 > /proc/sys/net/ipv6/conf/$i/disable_ipv6
done
ifconfig ovs-system 0.0.0.0

ovs-vsctl del-br br0
ovs-vsctl add-br br0
ovs-ofctl del-flows br0

for i in $NFVINT fakeovs0 ; do
  ovs-vsctl add-port br0 $i
done

ovs-vsctl set-controller br0 tcp:127.0.0.1:6653
ryu-manager pipette.py --verbose

Now we can run the fake SMTP service in the fake namespace. Note, even though the fake service is only running on 192.168.2.1, pipette is arranging for TCP traffic for the entire subnet to be mapped to that IP address as below.

# while /bin/true ; do echo completely legit SMTP service| ip netns exec copro nc -l 192.168.2.1 25 ; done

With everything ready, let’s try to connect to the fake SMTP server, apparently on 192.168.2.1, but in reality inside the coprocessor:

pi@pi8021x:~ $ telnet 192.168.2.1 25
Trying 192.168.2.1...
Connected to 192.168.2.1.
Escape character is '^]'.
completely legit SMTP service

How odd! What about 192.168.2.2?

pi@pi8021x:~ $ telnet 192.168.2.2 25
Trying 192.168.2.2...
Connected to 192.168.2.2.
Escape character is '^]'.
completely legit SMTP service

It appears there too! What’s going on? pipette is doing some TCP MAC and IP translation and mapping that into an isolated namespace, while also proxying the real host’s MAC address dynamically.

root@coprocessor:/home/pi# ovs-ofctl -OOpenFlow13 dump-flows br0
 cookie=0x0, duration=178.489s, table=0, n_packets=6, n_bytes=428, priority=1,in_port=enx0023565c8859 actions=goto_table:1
 cookie=0x0, duration=178.489s, table=0, n_packets=9, n_bytes=600, priority=1,in_port=fakeovs0 actions=goto_table:2
 cookie=0x0, duration=178.490s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=14.819s, table=1, n_packets=1, n_bytes=70, idle_timeout=30, priority=2,tcp,dl_src=b8:27:eb:d6:31:ca,nw_dst=192.168.2.1,tp_dst=25 actions=set_field:0e:00:00:00:00:67->eth_src,set_field:0e:00:00:00:00:66->eth_dst,set_field:192.168.2.1->ip_dst,output:fakeovs0
 cookie=0x0, duration=178.490s, table=1, n_packets=2, n_bytes=146, priority=1,tcp actions=CONTROLLER:65509
 cookie=0x0, duration=178.490s, table=1, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=14.819s, table=2, n_packets=1, n_bytes=66, idle_timeout=30, priority=2,tcp,nw_dst=192.168.2.5,tp_src=25 actions=set_field:aa:56:63:87:9e:e9->eth_src,set_field:b8:27:eb:d6:31:ca->eth_dst,set_field:192.168.2.1->ip_src,push_vlan:0x8100,set_field:4098->vlan_vid,output:enx0023565c8859
 cookie=0x0, duration=178.490s, table=2, n_packets=2, n_bytes=84, priority=1,arp actions=CONTROLLER:65509
 cookie=0x0, duration=178.490s, table=2, n_packets=4, n_bytes=280, priority=0 actions=drop

We are proud to present Open Source SDN using @faucetsdn and @alliedtelesis OpenFlow gear at JRES 2019. Come and see a live demo using our stackable SDN switches and wireless APs at our booth! #networksmarter #alliedtelesis #SDN #JRES2019 pic.twitter.com/iSzIK6C38f

— Rahul Gupta (@rahul_gupta86) December 3, 2019

Great to see FAUCET stacking!

FAUCET ACLs are great for locking things down. And it is possible to add FAUCET ACL rules, that count traffic and let it through (instead of counting and denying it).

But sometimes that’s not enough – you want to deny the traffic, but you also want to see what it was. FAUCET’s ACL language lets you do this.

  - rule:
      actions:
        allow: 0
        output:
          ports: [99]

This rule, placed at the end of an ACL, will deny all traffic – but – before it denies it, it will make a copy and output it to port 99. In this way you can run a tcpdump on a host connected to port 99, and observe the denied traffic. Handy!

https://blog.cyberreboot.org/catching-up-on-all-things-faucet-ed86fd5d3a28