A Man-in-the-Middle Attack in the Wild

Last night I tried to visit one of the websites that I host on one of my dedicated servers, and to my surprise, I saw this instead of the usual content:

My first reaction was that the gzip compression had possibly broken on my server, or that it was a weird compatibility issue with Firefox 6.0 to which I had just upgraded. I enabled Firefox's Web Console to see what was actually being received (highlighting mine):

It appeared that the gzipped output was being prepended with a hidden iframe trying to load a Chinese website, but because the content was not sent with a proper Content-Encoding header, Firefox didn't see it as compressed and just displayed all of the raw contents. The iframe was loading some other junk webpages in the background with Google ads (according to the source code of them), though the two stats pages on port 81 never loaded due to Little Snitch blocking non-port-80 traffic from Firefox:

After seeing this information, I thought that maybe this was a DNS hijacking (either the DNS servers for my website or the DNS resolver on my local network). I did some quick DNS troubleshooting and determined that everything was fine, so I took out my phone, disabled the wireless, and tried loading the same site. At about this time, my network monitor sent me a message over Jabber that the content on jcs.org had changed to something it was not expecting (I monitor my sites this way to quickly get notified if they are coming up with error pages instead, or in this case, that the page has been defaced). At this point, at least two websites on that server were showing similar output. My stomach sank as my mind jumped to "oh no, the server's been compromised."

I was able to securely SSH into the server, but it was quite slow to respond (the Web Console output above shows that it took 3650 ms to download the initial page of the site). Not wanting to type in any passwords on a possibly compromised server, I looked around at the website directories to see where the hidden iframe code may have been hiding. Nothing appeared to be out of the ordinary on the server itself. I looked on my central syslog server and noticed this coming from the problematic server:

Aug 16 20:41:26 thalamus /bsd: arp info overwritten for 67.159.5.1 by 00:d0:00:ae:a4:00 on em0
Aug 16 20:41:26 thalamus /bsd: arp info overwritten for 67.159.5.1 by 90:fb:a6:34:ab:13 on em0
Aug 16 20:41:42 thalamus /bsd: arp info overwritten for 67.159.5.1 by 00:d0:00:ae:a4:00 on em0
Aug 16 20:41:42 thalamus /bsd: arp info overwritten for 67.159.5.1 by 90:fb:a6:34:ab:13 on em0
Aug 16 20:41:44 thalamus /bsd: arp info overwritten for 67.159.5.1 by 00:d0:00:ae:a4:00 on em0
Aug 16 20:41:44 thalamus /bsd: arp info overwritten for 67.159.5.1 by 90:fb:a6:34:ab:13 on em0
Aug 16 20:45:34 thalamus /bsd: arp info overwritten for 67.159.5.1 by 00:d0:00:ae:a4:00 on em0
Aug 16 20:45:34 thalamus /bsd: arp info overwritten for 67.159.5.1 by 90:fb:a6:34:ab:13 on em0

With that, it all clicked for me: there was a rogue server on the network at the dedicated hosting provider broadcasting a fake ARP entry for the router. While my server had the fake ARP entry (90:fb:a6:34:ab:13) cached, it was sending all of its output through this rogue server, which was transparently rewriting HTTP traffic to prepend its hidden iframe code and then sending it to the real router; a classic monkeyman-in-the-middle attack. The attacker's program used to rewrite traffic must not have understood gzipped content properly and was causing the garbled output that I was seeing in Firefox.

At this point, I realized my server had not been compromised (at least not in the traditional sense) and that I just needed to avoid this rogue server. I used arping (one of the first OpenBSD ports I submitted way back in 2000) to get some output to show to the network provider:

jcs@thalamus:~> sudo arping 67.159.5.1
Password:
ARPING 67.159.5.1
60 bytes from 00:d0:00:ae:a4:00 (67.159.5.1): index=0 time=795.841 usec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=1 time=184.544 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=2 time=229.707 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=3 time=229.719 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=4 time=394.402 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=5 time=448.702 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=6 time=448.730 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=7 time=594.244 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=8 time=712.000 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=9 time=712.015 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=10 time=983.803 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=11 time=983.827 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=12 time=983.837 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=13 time=994.009 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=14 time=995.243 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=15 time=995.256 msec
60 bytes from 00:d0:00:ae:a4:00 (67.159.5.1): index=16 time=46.939 msec
60 bytes from 90:fb:a6:34:ab:13 (67.159.5.1): index=17 time=189.820 msec

Clearly the rogue server (90:fb:a6:34:ab:13) is being heavily loaded while trying to process and forward the network output of the other customer servers. It's also clear that the real router (00:d0:00:ae:a4:00) is responding quickly but not as frequent as the rogue server. To avoid caching the fake ARP entry, I simply installed a hard-coded entry and made it permanent:

jcs@thalamus:~> sudo arp -F -s 67.159.5.1 00:d0:00:ae:a4:00 permanent
67.159.5.1 (67.159.5.1) deleted
jcs@thalamus:~> arp -a -n
? (67.159.5.1) at 00:d0:00:ae:a4:00 on em0 permanent static

The response times to the server were instantly much faster and my websites returned to normal. I opened a trouble ticket with the network provider and they removed the (hopefully compromised) server from the network within about 10 minutes. I've followed up with them to find out why each customer isn't on his own VLAN, which largely eliminates problems like this.

Update: I've canceled my service with this provider.