Archives: April 2017

April 29, 2017

mcblockd’s latest trick works: drop TCP connections

Evidence in the logs of mcblockd’s latest feature working. It’s successfully killing TCP connections when it adds a prefix to one of the pf tables.

Apr 29 03:42:40 ria mcblockd: [I] Dropped TCP connection from 221.144.5.116:38440
Apr 29 03:42:40 ria mcblockd: [I] Added 221.144/12 (KR) to ssh_losers for 180 days
Apr 29 05:02:02 ria mcblockd: [I] Dropped TCP connection from 46.118.248.195:40294
Apr 29 05:02:02 ria mcblockd: [I] Added 46.118/15 (UA) to ssh_losers for 180 days
Apr 29 07:07:42 ria mcblockd: [I] Dropped TCP connection from 120.132.4.45:56388
Apr 29 07:07:42 ria mcblockd: [I] Added 120.128/13 (CN) to ssh_losers for 180 days
Apr 29 10:04:23 ria mcblockd: [I] Dropped TCP connection from 95.215.2.52:50862
Apr 29 10:04:23 ria mcblockd: [I] Added 95.215.0/22 (RU) to ssh_losers for 180 days
Apr 29 11:51:34 ria mcblockd: [I] Dropped TCP connection from 110.246.84.64:32309
Apr 29 11:51:34 ria mcblockd: [I] Added 110.240/12 (CN) to ssh_losers for 180 days
Apr 29 12:22:42 ria mcblockd: [I] Dropped TCP connection from 183.184.133.58:3369
Apr 29 12:22:42 ria mcblockd: [I] Added 183.184/13 (CN) to ssh_losers for 180 days
Apr 29 13:13:54 ria mcblockd: [I] Dropped TCP connection from 120.150.231.99:50357
Apr 29 13:13:54 ria mcblockd: [I] Dropped TCP connection from 120.150.231.99:50349
Apr 29 13:13:54 ria mcblockd: [I] Added 120.144/12 (AU) to ssh_losers for 180 days
Apr 29 14:42:30 ria mcblockd: [I] Dropped TCP connection from 113.209.68.135:53280
Apr 29 14:42:30 ria mcblockd: [I] Added 113.209/16 (CN) to ssh_losers for 180 days

April 27, 2017

mcblockd’s latest tricks: kill pf state, walk PCB list and kill TCP connections

Today I added a new feature to mcblockd to kill pf state for all hosts in a prefix when the prefix is added to one of my pf tables. This isn’t exactly what I want, but it’ll do for now.

mcblockd also now walks the PCB (protocol control block) list and drops TCP connections for hosts in a prefix I’ve just added to a pf table. Fortunately there was sample code in /usr/src/usr.sbin/tcpdrop/tcpdrop.c. The trick here is that I don’t currently have a means of mapping a pf table to where it’s applied (which ports, which interfaces). In the long term I might add code to figure that out, but in the interim I can configure ports and interfaces in mcblockd’s configuration file that will allow me to drop specific connections. For this first pass, I just toast all PCBs for a prefix.

The reason I added this feature: I occasionally see simultaneous login attempts from different IP addresses in the same prefix. If I’m going to block the prefix automatically, I want to cut off all of their connections right now, not after all of their connections have ended. Blowing away their pf state works, but leaves a hanging TCP connection in the ESTABLISHED state for a while. I want the PCBs to be cleaned up.

April 26, 2017

mcblockd has been busy

The mcblockd automation has been running for roughly one week. It’s been fairly busy automatically blocking those trying to crack my ssh server. Below is some of the output from a query of the active blocked networks (the summary information for the top 10 countries by the number of addresses being blocked). Interesting to note that the automation has blocked a huge swath of addresses from China. State-sponsored cyberattacks?

% mcblockc getactive ssh_losers

...

  Addresses covered per country:
    CN 102,263,808
      /10 networks:    8 (33,554,432 addresses)
      /11 networks:   17 (35,651,584 addresses)
      /12 networks:   21 (22,020,096 addresses)
      /13 networks:   11 (5,767,168 addresses)
      /14 networks:   14 (3,670,016 addresses)
      /15 networks:    9 (1,179,648 addresses)
      /16 networks:    6 (393,216 addresses)
      /18 networks:    1 (16,384 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    1 (2,048 addresses)
      /22 networks:    1 (1,024 addresses)
    KR 7,864,320
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    1 (2,097,152 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    1 (524,288 addresses)
    IN 7,340,032
      /10 networks:    1 (4,194,304 addresses)
      /12 networks:    2 (2,097,152 addresses)
      /13 networks:    1 (524,288 addresses)
      /14 networks:    2 (524,288 addresses)
    BR 7,252,992
      /11 networks:    3 (6,291,456 addresses)
      /13 networks:    1 (524,288 addresses)
      /14 networks:    1 (262,144 addresses)
      /15 networks:    1 (131,072 addresses)
      /17 networks:    1 (32,768 addresses)
      /19 networks:    1 (8,192 addresses)
      /21 networks:    1 (2,048 addresses)
      /22 networks:    1 (1,024 addresses)
    FR 6,782,976
      /10 networks:    1 (4,194,304 addresses)
      /11 networks:    1 (2,097,152 addresses)
      /15 networks:    1 (131,072 addresses)
      /16 networks:    5 (327,680 addresses)
      /18 networks:    2 (32,768 addresses)
    AR 4,524,032
      /12 networks:    1 (1,048,576 addresses)
      /13 networks:    2 (1,048,576 addresses)
      /14 networks:    8 (2,097,152 addresses)
      /15 networks:    2 (262,144 addresses)
      /16 networks:    1 (65,536 addresses)
      /21 networks:    1 (2,048 addresses)
    JP 4,227,072
      /10 networks:    1 (4,194,304 addresses)
      /17 networks:    1 (32,768 addresses)
    RU 3,484,672
      /13 networks:    2 (1,048,576 addresses)
      /14 networks:    5 (1,310,720 addresses)
      /15 networks:    6 (786,432 addresses)
      /16 networks:    2 (131,072 addresses)
      /17 networks:    4 (131,072 addresses)
      /18 networks:    2 (32,768 addresses)
      /19 networks:    5 (40,960 addresses)
      /22 networks:    3 (3,072 addresses)
    IT 3,280,896
      /11 networks:    1 (2,097,152 addresses)
      /12 networks:    1 (1,048,576 addresses)
      /15 networks:    1 (131,072 addresses)
      /20 networks:    1 (4,096 addresses)
    TW 2,637,824
      /12 networks:    2 (2,097,152 addresses)
      /13 networks:    1 (524,288 addresses)
      /18 networks:    1 (16,384 addresses)

...

April 26, 2017

Looking at ‘Synners’ (TCP SYN data)

One of the many sets of data I collect with mcflow on my gateway is traffic counters for TCP SYN packets I receive but do not SYN ACK. I keep the source IP address, the destination port, and of course timestamps and counters. This type of data generally represents one of three things: probing for vulnerable services which I don’t run, probing for services I do run but block from offenders, or probing for botnet-controlled devices.

The table below shows the top 10 ports for the current week. In the case of ssh and http, I do run those services but mcblockd automatically blocks those who violate my configured policies. I do not run a telnet server anywhere (my IoT devices are of my own design and use ECDH, 2048-bit RSA keys and AES128). I also do not run MS SQL Server or rdp (Remote Desktop). I have no Windows hosts, and if I did, I certainly wouldn’t expose MS SQL Server or Remote Desktop.

Ports 7547 and 5358 are known to be used by Mirai and its descendants. Port 7547 is also a common port used by broadband ISPs for TR-064 services (specifically, TR-069) to manage home routers.

Port Packets Bytes
22 (ssh) 22116 1168688
23 (telnet) 3740 152784
80 (http) 1601 99216
1433 (ms-sql-s) 1279 52288
81 917 38016
7547 515 20620
3389 (rdp) 199 8792
5358 195 8148
2323 181 7384
8080 154 6700

Below is a table showing the SYNs I didn’t SYN ACK by country. This is just the top 10. Note that the top two have large swaths of their IP address space automatically blocked by mcblockd for violating my configured policies. They’re also known state sponsors of cyberattacks, and the evidence is pretty clear here. Much (but not all) of the US stuff is research scanning.

Country Packets Bytes
RU (Russian Federation) 17394 864024
CN (China) 6038 319116
US (United States) 3077 169932
NL (Netherlands) 1160 47580
TH (Thailand) 603 33480
UA (Ukraine) 467 20612
KR (Korea) 462 19380
BR (Brazil) 426 18708
FR (France) 341 17828
TR (Turkey) 281 11756

What is perhaps interesting about this data: the lines drawn during WWII and the Cold War don’t appear to have changed. I find this very sad. I’m just a tiny single user running a very modest home network, yet I’m a target of Russia and China. And my network is likely much more secure than the average home network. I assume this means that all of us are being probed all of the time, and some of us are probably regularly compromised. I think we (meaning the entire industry) need to consider completely banning telnet and doing something real about securing IoT devices.

April 20, 2017

more on mcblockd automation progress

Similar to what I have for sshd, I have real time log processing on my web server. The secure remote communication with mcblockd is very nice to have, since my web server is a separate machine from my gateway/firewall. Below you can see offending web server log entries followed immediately by an action from mcblockd. Instant blocking, without my involvement.

185.36.102.114 - [20/Apr/2017:19:08:08] "GET /blog/xmlrpc.php HTTP/1.0" 200 42
Apr 20 19:08:09 ria mcblockd: [I] Added 185.36.100/22 (CZ) to www_losers for 30 days

191.101.117.226 - [20/Apr/2017:19:57:52] "POST /blog/xmlrpc.php HTTP/1.1" 500 -
Apr 20 19:57:52 ria mcblockd: [I] Added 191.101/16 (CL) to www_losers for 90 days

5.164.231.83 - [20/Apr/2017:20:12:07] "GET /blog/xmlrpc.php HTTP/1.0" 200 42
Apr 20 20:12:08 ria mcblockd: [I] Added 5.164/14 (RU) to www_losers for 90 days

160.202.162.204 - [22/Apr/2017:21:59:24] "GET /wp-login.php HTTP/1.1" 404 210
Apr 22 21:59:24 ria mcblockd: [I] Added 160.202.160/22 (KR) to www_losers for 90 days

104.173.193.176 - [23/Apr/2017:00:58:00] "GET /wp-login.php HTTP/1.1" 404 210
Apr 23 00:58:00 ria mcblockd: [I] Added 104.173.193/24 (US) to www_losers for 30 days

191.37.7.186 - [23/Apr/2017:04:18:19] "GET /wp-login.php HTTP/1.1" 404 210
Apr 23 04:18:19 ria mcblockd: [I] Added 191.37.0/17 (BR) to www_losers for 90 days

103.229.124.123 - [23/Apr/2017:07:50:15] "GET /xmlrpc.php HTTP/1.1" 404 208
Apr 23 07:50:15 ria mcblockd: [I] Added 103.229.124/22 (TW) to www_losers for 30 days

61.77.12.200 - [23/Apr/2017:09:40:35] "GET /wp-login.php HTTP/1.1" 404 210
Apr 23 09:40:36 ria mcblockd: [I] Added 61.72/13 (KR) to www_losers for 90 days

46.161.9.14 - [23/Apr/2017:10:30:24] "GET /blog/xmlrpc.php HTTP/1.0" 405 42
Apr 23 10:30:24 ria mcblockd: [I] Added 46.161.0/18 (RU) to www_losers for 90 days

And yes, the threshold policy code works fine. Below is the result of someone trying to log in 5 times over a period of about 26 minutes. Since I have the threshold set to 5 times in 30 days, they were way above the threshold, but this would be considered a ‘slow’ attempt by some measures.

Apr 21 17:08:59 ria mcblockd: [I] Pending 69.162.73/24 (US) for ssh_losers, 1/5
Apr 21 17:08:59 ria mcblockd: [I] Pending 69.162.73/24 (US) for ssh_losers, 2/5
Apr 21 17:22:14 ria mcblockd: [I] Pending 69.162.73/24 (US) for ssh_losers, 3/5
Apr 21 17:22:14 ria mcblockd: [I] Pending 69.162.73/24 (US) for ssh_losers, 4/5
Apr 21 17:35:21 ria mcblockd: [I] Added 69.162.73/24 (US) to ssh_losers for 30 days

And another over a period of about 91 minutes:

Apr 23 01:39:43 ria mcblockd: [I] Pending 64.179.211/24 (CA) for ssh_losers, 1/5
Apr 23 01:39:43 ria mcblockd: [I] Pending 64.179.211/24 (CA) for ssh_losers, 2/5
Apr 23 02:25:48 ria mcblockd: [I] Pending 64.179.211/24 (CA) for ssh_losers, 3/5
Apr 23 02:25:48 ria mcblockd: [I] Pending 64.179.211/24 (CA) for ssh_losers, 4/5
Apr 23 03:11:15 ria mcblockd: [I] Added 64.179.211/24 (CA) to ssh_losers for 30 days

April 20, 2017

mcblockd automation progress

So far, so good. Nice to see this in the logs while I’m working on updates to mcblockd. This shows lines from my auth.log with the corresponding actions invoked in mcblockd. The key takeaway: nearly instantaneous response to login attempts from countries where I have the policy set to low tolerance, and the expected response for “US” networks where I have the tolerance set a little higher.

The way this works…

A mcblocklog process receives all auth.log entries via a pipe from syslogd. It uses a list of regular expressions (in a plain text file) to match offending lines in the log, then posts matched IP addresses to mcblockd as ‘logHit’ requests. Unlike my previous setup that periodically parsed entire logs, this happens in real time. mcblockd asks dwprdapd for prefix and country information, then applies configured policy. Depending on the policy for the network, mcblockd may instantly add an entry to its database and the pf table, or wait for the policy to be violated (number of hits over a configured time period). For foreign countries, I have the policy set to trigger from a single offending line, hence mcblockd will immediately add an entry to the pf table. For the U.S., I have the policy set to 5 hits in 7 days. These are experimental settings at the moment, it’s likely I’ll change them.

Also part of the configured policy is how long an entry will live in the pf tables, by days. For countries which have no business connecting to my network, the policy is set long versus my own country. This is a common desired feature in an IPS (Intrusion Protection System). Another part of the policy is a ‘widest mask’ setting, to allow me to avoid blocking huge swaths of address space from a given country to whom I want to grant a bit of leniency (say the U.S. and Canada in my case).

Probably worth noting that if an address is already covered in the pf tables, mcblockd does nothing.

Also worth noting that the service is secured with libDwmAuth, using ECDH and 2048-bit RSA keys during authentication, then AES128 in GCM mode after authentication.

While the log entries below are for ssh, I have a similar process for web logs and mail server logs.

Apr 19 05:33:30 ria sshd[7695]: error: maximum authentication attempts exceeded
                    for root from 81.100.183.189 port 43973 ssh2 [preauth]
Apr 19 05:33:30 ria mcblockd[1854]: [I] Added 81.96/12 (GB) to ssh_losers

Apr 19 06:09:50 ria sshd[7752]: error: maximum authentication attempts exceeded
                    for root from 36.36.254.10 port 60635 ssh2 [preauth]
Apr 19 06:09:50 ria mcblockd[1854]: [I] Added 36.36/16 (CN) to ssh_losers

Apr 19 09:22:37 ria sshd[8123]: error: maximum authentication attempts exceeded
                    for root from 123.96.0.151 port 60583 ssh2 [preauth]
Apr 19 09:22:37 ria mcblockd[1854]: [I] Added 123.96/15 (CN) to ssh_losers

Apr 19 09:29:38 ria sshd[8129]: Did not receive identification string from 34.205.143.181
Apr 19 09:29:43 ria sshd[8130]: Invalid user support from 34.205.143.181
Apr 19 09:29:43 ria sshd[8130]: Postponed keyboard-interactive for invalid user
                    support from 34.205.143.181 port 53145 ssh2 [preauth]
Apr 19 09:29:43 ria sshd[8130]: error: PAM: authentication error for illegal user
                    support from 34.205.143.181
Apr 19 09:29:43 ria sshd[8130]: Failed keyboard-interactive/pam for invalid user
                    support from 34.205.143.181 port 53145 ssh2
Apr 19 09:29:44 ria mcblockd[1854]: [I] Added 34.205.143/24 (US) to ssh_losers

Apr 19 14:11:40 ria sshd[8666]: error: maximum authentication attempts exceeded
                    for root from 200.73.205.204 port 45585 ssh2 [preauth]
Apr 19 14:11:40 ria mcblockd[1854]: [I] Added 200.73.200/21 (EC) to ssh_losers

Apr 19 14:51:48 ria sshd[9272]: Invalid user admin from 77.39.72.192
Apr 19 14:51:48 ria mcblockd[1854]: [I] Added 77.39.0/17 (RU) to ssh_losers

Apr 19 15:31:18 ria sshd[17218]: Invalid user admin from 193.105.134.184
Apr 19 15:31:18 ria mcblockd[1854]: [I] Added 193.105.134/24 (SE) to ssh_losers

Apr 19 15:34:02 ria sshd[18020]: error: maximum authentication attempts exceeded
                    for root from 85.90.198.244 port 44202 ssh2 [preauth]
Apr 19 15:34:02 ria mcblockd[31598]: [I] Added 85.90.192/19 (UA) to ssh_losers

Apr 19 15:58:13 ria sshd[23696]: error: maximum authentication attempts exceeded
                    for root from 156.213.133.233 port 58400 ssh2 [preauth]
Apr 19 15:58:13 ria mcblockd[31598]: [I] Added 156.192/11 (EG) to ssh_losers

Apr 19 16:04:49 ria sshd[23785]: error: maximum authentication attempts exceeded
                    for root from 171.50.175.114 port 46884 ssh2 [preauth]
Apr 19 16:04:49 ria mcblockd[31598]: [I] Added 171.48/12 (IN) to ssh_losers

Apr 19 16:39:23 ria sshd[23858]: Invalid user support from 181.211.93.159
Apr 19 16:39:23 ria mcblockd[31598]: [I] Added 181.211/16 (EC) to ssh_losers

Apr 19 16:59:10 ria sshd[23914]: Did not receive identification string from 
                    128.40.46.124
Apr 19 16:59:10 ria mcblockd[31598]: [I] Added 128.40/15 (GB) to ssh_losers

Apr 19 18:19:24 ria sshd[24599]: error: maximum authentication attempts exceeded
                    for root from 178.216.100.130 port 52035 ssh2 [preauth]
Apr 19 18:19:24 ria mcblockd[31598]: [I] Added 178.216.96/21 (UA) to ssh_losers

Apr 19 19:21:43 ria sshd[24873]: Invalid user admin from 200.121.233.88
Apr 19 19:21:43 ria mcblockd[31598]: [I] Added 200.121/16 (PE) to ssh_losers

Apr 19 23:12:25 ria sshd[30989]: error: maximum authentication attempts exceeded
                    for root from 131.161.55.11 port 42822 ssh2 [preauth]
Apr 19 23:12:25 ria mcblockd[31598]: [I] Added 131.161.52/22 (HN) to ssh_losers

Apr 20 00:08:10 ria sshd[31282]: error: maximum authentication attempts exceeded
                    for root from 167.250.75.214 port 4837 ssh2 [preauth]
Apr 20 00:08:10 ria mcblockd[31598]: [I] Added 167.250.72/22 (BR) to ssh_losers

Apr 20 00:22:31 ria sshd[31674]: Did not receive identification string from
                    218.93.17.146
Apr 20 00:22:31 ria mcblockd[31598]: [I] Added 218.64/11 (CN) to ssh_losers

Apr 20 00:25:41 ria sshd[31691]: Invalid user admin from 60.178.126.100
Apr 20 00:25:41 ria mcblockd[31598]: [I] Added 60.160/11 (CN) to ssh_losers

Apr 20 00:38:12 ria sshd[31715]: Invalid user ubnt from 119.191.105.117
Apr 20 00:38:12 ria mcblockd[31598]: [I] Added 119.176/12 (CN) to ssh_losers

Apr 20 00:45:53 ria sshd[31733]: Invalid user admin from 123.170.99.10
Apr 20 00:45:53 ria mcblockd[31598]: [I] Added 123.160/12 (CN) to ssh_losers

Apr 20 01:39:27 ria sshd[31845]: error: maximum authentication attempts exceeded
                    for root from 119.193.140.196 port 60716 ssh2 [preauth]
Apr 20 01:39:27 ria mcblockd[31598]: [I] Added 119.192/11 (KR) to ssh_losers

April 20, 2017

dwmrdapd nearing production-ready: RDAP cache for IDS/IPS applications

I’ve been working on a new IP to country mapping service to be used by my IDS/IPS tools. This post is about the server portion, named dwmrdapd.

dwmrdapd provides a simple service to map an IP address to its registered prefix (in one of the NICs, i.e. ARIN, RIPE, AFRINIC, LACNIC, APNIC) and its registered country. It maintains a small custom database of the mappings in order to provide a quick responses to most queries. When an entry is not found in the database, or the requested entry is more than 30 days old, dwmrdapd will make a new RDAP query to the RDAP server of the corresponding NIC (Network Information Center).

I’ve been using the service to apply policy to the networks automatically blocked by my firewall. As of this week, I can call it near production-ready.

Most of the trickery in implementing this service revolved around dealing with ARIN’s poor RDAP service. The first problem was dealing with the fact that they pad IP octets with leading zeros in startAddress and endAddress, which leads all of the standard string to address functions to interpret the numbers as octal. That was relatively easy to handle with a simple regular expression fix. The second problem is that ARIN doesn’t populate the country value. Why, I don’t know. The workaround is to parse all of the vcardArrays for a card with an adr label and then parse the label looking for a country name, then map that country name to a 2-letter country code. The latest version of dwmrdapd does this, but it’s still a bit hokey. Some ARIN RDAP responses contain many vcard entries, with different countries. There doesn’t seem to be a science to the entries, hence I prioritize non-U.S. cards and fall back to “US” as the country code as a last resort.

The service itself is secured with libDwmAuth using ECDH, RSA 2048-bit keys and AES128 in GCM mode once authentication is complete. Key management is very similar to that used by ssh, which makes it easy for me to use on my local hosts.

Inside the encryption is just simple JSON. Example output from the simple client:

% dwmrdapc 35.1.1.1
[
   {
      "country" : "US",
      "countryName" : "United States of America",
      "ipv4addr" : "35.1.1.1",
      "lastChanged" : "2014-09-23 18:00",
      "lastUpdated" : "2017-04-20 15:26",
      "prefix" : "35.1/16"
   }
]

This isn’t exactly a new kind of service. Going back to the late 1990s, we’ve had IP geolocation services. But I wanted something free, tightly secured, and automatically updated on an on-demand basis. I also wanted something small data-wise; I don’t need latitude/longitude, etc. And I also wanted to take a look at the RDAP services from the NICs.

I did look at some other freely available sources of data, one of them being ipdeny.com. While their data is useful for bootstrapping (and I have a program to bootstrap dwmrdapd’s initial database from their country ‘zone files’), I’ve found it lacking in correctness. Possibly due to no fault of their own: NIC data is messy, especially if you’re fetching it via WHOIS but even the RDAP data can be very sloppy (cough, ARIN, cough), or abysmally slow (LACNIC).

There are also RIR datasets (Routing Information Registry), but they’re not uniform and there’s less participation than some of us would like to see.

April 7, 2017

Refactoring and adding to libDwmAuth

I’ve been working on some changes and additions to libDwmAuth.

I had started a round of changes to the behind-the-scenes parts of the highest-level APIs to make managing authorized users and MitM prevention easier. However, in the end I felt like I was following the wrong course because my first solution involved too many round trips between client and server and some significant key generation overhead since I was using ephemeral 2048-bit RSA keys.

I’m now using ECDH for the first step. I have a working implementation with unit tests, using Crypto++. Unfortunately I’m still waiting for curve25519 to show up in Crypto++, but in the meantime I’m using secp256r1 despite its vulnerabilities.

I also have a rudimentary scheme for MitM prevention that is very similar to that used by OpenSSH, and client and server authentication based on RSA keys (2048 bits at the moment). I have a known_services file that’s similar to OpenSSH’s known_hosts, and an authorized_keys file that’s similar to the same for OpenSSH. This allows fairly easy management on both client and server side for my applications.

Obviously I also have a public/private key generator application.

© 2017 rfdm blog
All rights reserved