Current adhosts script

Author: cbdevnet

.gitignore

@@ -0,0 +1,2 @@
+ad.domains
+ad.hosts

LICENSE.txt

@@ -0,0 +1,19 @@
+This program is free software. It comes without any warranty, to
+the extent permitted by applicable law. You can redistribute it
+and/or modify it under the terms of the Do What The Fuck You Want
+To Public License, Version 2, as published by Sam Hocevar and 
+reproduced below.
+
+DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
+Version 2, December 2004 
+
+Copyright (C) 2004 Sam Hocevar <sam@hocevar.net> 
+
+	Everyone is permitted to copy and distribute verbatim or modified 
+	copies of this license document, and changing it is allowed as long 
+	as the name is changed. 
+
+DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
+TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 
+
+	0. You just DO WHAT THE FUCK YOU WANT TO.

README.md

@@ -0,0 +1,57 @@
+# adhosts - Generate ad-blocking hosts file from multiple sources
+
+This minimal tool builds a composite host file, containing entries mapping
+domain names to IP addresses, from multiple sources, such as ad-blocking,
+anti-tracking and other lists.
+
+The resulting file can be used with eg. `dnsmasq` to provide ad-blocking via DNS
+for an entire network, or it can be installed on single hosts.
+
+This script condenses much of the functionality of eg. PiHole into a form that
+is easily integratable into existing systems, without introducing lots of new
+software. Thus, this tool is mostly geared towards system administrators and
+people already familiar with their infrastructure.
+
+## Setup
+
+The `config` file contains all configurable parameters, which are
+
+* `SRC_HOSTS`: A file containing lines of links to hosts-style (ip-blank-domain) blocklists.
+* `SRC_DOMAIN`: A file containing lines of links to domain blocklists.
+* `OUT_DOMAINS`: The output file containing the final list of blocked domains.
+* `OUT_HOSTS`: The output hosts-style blocklist.
+* `ADSERVER`: The host to redirect ads to. This can be used to approximately count the number of
+blocked ads (see below).
+
+To add a local blocklist, in either domain or hosts-style, use a `file://` URL.
+
+## Blocking on the local host
+
+To apply the generated blocklist on a single computer, copy it to `/etc/hosts`.
+Take care that there may be pre-existing entries in there, which may be destroyed
+by simply overwriting the existing file. A reasonable solution may be including the
+existing entries via a `file://` URL.
+
+## Blocking for the whole network
+
+To block ads for an entire network, you will need to configure that networks resolver
+to prefer entries from an external hosts file. Most resolvers will also read the local
+`/etc/hosts` and prefer that to external responses, to copying the generated hosts-style
+file to `/etc/hosts` on the resolver should work.
+
+For `dnsmasq`, the following configuration option will read an additional hosts-style file
+into the resolver:
+
+```
+addn-hosts=/path/to/ad.hosts
+```
+
+### Counting blocked ads
+
+When redirecting the blocked ads to a host on the local network via the `ADSERVER` configuration
+variable, setting up a web server such as `lighttpd` and having it count the hits will provide
+an approximation of the number of ads blocked.
+
+Note that most ad networks (sensibly) use HTTPS now, so the count may be off until you
+provide HTTPS on the diversion server (which may be complicated by having to also serve a valid
+certificate for the requested domains).

adhosts

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+source config
+
+# Clear temporary data
+: > "$OUT_DOMAINS.temp"
+
+# Fetch host-format lists, remove first column
+while read list; do
+	if [ -z "$list" ]; then
+		continue
+	fi
+	printf "Fetching host list %s\n" "$list"
+	curl "$list" | grep -v "^#" | tr '[:blank:]' ' ' | cut -d " " -f 2 >> "$OUT_DOMAINS.temp"
+done < "$SRC_HOSTS"
+
+# Fetch domain-format lists
+while read list; do
+	if [ -z "$list" ]; then
+		continue
+	fi
+	printf "Fetching domain list %s\n" "$list"
+	curl "$list" | grep -v "^#" >> "$OUT_DOMAINS.temp"
+done < "$SRC_DOMAIN"
+
+# Sort and weed out duplicates
+sort "$OUT_DOMAINS.temp" | tr -d '\r' | sed -e 's/^[[:blank:]]*//' -e 's/[[:blank:]]*$//' | uniq > "$OUT_DOMAINS"
+rm "$OUT_DOMAINS.temp"
+
+# Prepend new ad server
+sed -e "s/^/${ADSERVER} /" "$OUT_DOMAINS" > "$OUT_HOSTS"
+

adlist-sources.domain

@@ -0,0 +1,2 @@
+https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
+

adlist-sources.host

@@ -0,0 +1,3 @@
+http://sysctl.org/cameleon/hosts
+https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
+https://hosts-file.net/ad_servers.txt

config

@@ -0,0 +1,5 @@
+SRC_HOSTS="adlist-sources.host"
+SRC_DOMAIN="adlist-sources.domain"
+OUT_DOMAINS="ad.domains"
+ADSERVER="127.0.0.1"
+OUT_HOSTS="ad.hosts"