{"id":1608,"date":"2024-04-04T05:21:59","date_gmt":"2024-04-04T09:21:59","guid":{"rendered":"https:\/\/www.rfdm.com\/blog\/?p=1608"},"modified":"2024-04-29T00:38:03","modified_gmt":"2024-04-29T04:38:03","slug":"drop-garbage-web-traffic-first-manual-pass-complete-automation-and-weekly-json-file-coming-soon","status":"publish","type":"post","link":"https:\/\/www.rfdm.com\/blog\/?p=1608","title":{"rendered":"Drop garbage web traffic: first manual pass complete, automation and weekly JSON file coming soon"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">I&#8217;m basically done with my first pass at manually adding networks to block from my web server, based on my web server logs and a whole lot of queries to dwmrdapd (my secure RDAP aggregator \/ cache).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It&#8217;s kind of amazing how many HTTP requests are just garbage.  My definition of garbage: scraping (presumably to feed an LLM or other AI training) and attack probing.  While I&#8217;m still allowing Google and Apple crawler bots, my average traffic has been reduced by around 75%.  I&#8217;m not surprised or astounded.  I&#8217;m just VERY disappointed at what the modern Internet has become.  Especially since a lot of the traffic is clearly state sponsored vulnerability probing.  Not to mention attempts at mapping IP addresses to location and identity, many of those coming from networks in China, Vietnam, Singapore and Russia.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the plots below, you can see the result of the manual work I started on 3\/16.  Tedious work each night, but the results speak for themselves.  I did add more automation for things the script kiddies are trying to find.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.rfdm.com\/blog\/wp-content\/uploads\/2024\/03\/sitetraffic_20240224-20240325.png\"><img loading=\"lazy\" decoding=\"async\" width=\"785\" height=\"1024\" src=\"https:\/\/www.rfdm.com\/blog\/wp-content\/uploads\/2024\/03\/sitetraffic_20240224-20240325-785x1024.png\" alt=\"\" class=\"wp-image-1609\" srcset=\"https:\/\/www.rfdm.com\/blog\/wp-content\/uploads\/2024\/03\/sitetraffic_20240224-20240325-785x1024.png 785w, https:\/\/www.rfdm.com\/blog\/wp-content\/uploads\/2024\/03\/sitetraffic_20240224-20240325-230x300.png 230w, https:\/\/www.rfdm.com\/blog\/wp-content\/uploads\/2024\/03\/sitetraffic_20240224-20240325-768x1002.png 768w, https:\/\/www.rfdm.com\/blog\/wp-content\/uploads\/2024\/03\/sitetraffic_20240224-20240325-1177x1536.png 1177w, https:\/\/www.rfdm.com\/blog\/wp-content\/uploads\/2024\/03\/sitetraffic_20240224-20240325.png 1390w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">I&#8217;m still working on a larger effort to block all networks inside autonomous systems (ASes) that are cloud providers.  That&#8217;s because to date, 99.99% of the traffic my site sees from them is unwanted traffic.  A miniscule percentage is valid VPN exit traffic.  But I&#8217;m at the point where I&#8217;m no longer going to tolerate the monetized weaponization of cloud infrastructure at my expense.  Cloud providers aren&#8217;t going to get a free pass anymore.  Their behavior at this point is analogous to that of arms dealers.  They&#8217;re making money selling weapons used against us, and meanwhile try to sell us cloud services for &#8216;security&#8217;.   Amazon, Google, Microsoft, OVH, DigitalOcean, Hetzner, Linode, Hurricane Electric and many other smaller hosting service providers&#8230; you&#8217;re becoming deplorable.  You can keep buying up address space and using it against us, but soon I&#8217;ll have automation that keeps finding all of your address space and blocking it.  And I&#8217;ll be putting a JSON file online at least weekly so others can do the same if desired.  Parts of your business model are hostile to small websites, and I&#8217;m not the only one that&#8217;s tired of it.  If you&#8217;d bother to do DPI and police your own users, I wouldn&#8217;t have to spend my time creating this automation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the long run&#8230; I&#8217;ll go out on a limb and say this model isn&#8217;t sustainable.  I&#8217;m not interested in hosting my applications in someone else&#8217;s datacenter, co-resident with vandals (many of whom I&#8217;m sure are just as interested in side-channel attacks within the cloud infrastructure as they are in attempting to break in to my web site).  And I&#8217;m not alone.  At some point you&#8217;ll have to clean up your act, else large portions of the web just won&#8217;t be accessible from your infrastructure because we (those producing content) will just drop all of your traffic (including your search indexing crawlers).  It won&#8217;t just be the big guys like The New York Times, Reddit, et. al.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m basically done with my first pass at manually adding networks to block from my web server, based on my web server logs and a whole lot of queries to dwmrdapd (my secure RDAP aggregator \/ cache). It&#8217;s kind of amazing how many HTTP requests are just garbage. My definition of garbage: scraping (presumably to &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.rfdm.com\/blog\/?p=1608\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Drop garbage web traffic: first manual pass complete, automation and weekly JSON file coming soon&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[93],"tags":[],"class_list":["post-1608","post","type-post","status-publish","format-standard","hentry","category-cloud"],"_links":{"self":[{"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1608","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1608"}],"version-history":[{"count":4,"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1608\/revisions"}],"predecessor-version":[{"id":1719,"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1608\/revisions\/1719"}],"wp:attachment":[{"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1608"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1608"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rfdm.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1608"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}