WIP domain substitution scripts and supporting files

Rename Python script for evaluating cleaning list
Add patch to disable intranet redirect detector
Remove configuration_policy=0 GYP flag setting
Update README.md

README.md

@@ -4,15 +4,16 @@
 ### Features
 
 In addition to features provided by [Iridium Browser](https://iridiumbrowser.de/) and [Inox patchset](https://github.com/gcarq/inox-patchset), the following is also included:
-* Remove additional detection of Google hosts
+* Remove additional detection of and disable specific functionality for Google hosts
 * Disabled searching in Omnibox
 * Disabled automatic formatting of URL in Omnibox
 * Disabled JavaScript dialog boxes from showing when a page closes (onbeforeunload dialog boxes)
 * Added menu item under "More tools" to clear the HTTP authentication cache on-demand
 * Disabled persistent per-site settings in Preferences file
 * Make all popups go to tabs
-* Replaced many domains in the source code with non-existant alternatives (see `generate_domain_patch.sh`)
+* Replaced many domains in the source code with non-existant alternatives (see `generate_domain_substitution_list.sh`)
 * Stripped binaries from the source code (see `generate_cleaning_list.sh`)
+* Disabled intranet redirect detector
 * Debian build scripts
   * (Debian build scripts change) Move the chrome-sandbox into a separate package
 * (Iridium Browser feature change) Prevent URLs with the `trk:` scheme from connecting to the Internet
@@ -54,27 +55,35 @@ ungoogled-chromium provides scripts to automate the downloading, patching, and b
   * `ubuntu` - This contains files to generate a dpkg debian directory for Ubuntu
 * `download_source.sh` - This script downloads the source tarball from `commondatastorage.googleapis.com` and unpacks it into the build sandbox.
   * It accepts arguments; pass in `-h` for more information.
-* `generate_domain_patch.sh` - This script generates a patch to replace multiple domain name strings with invalid domain names.
-  * It recursively works down the current working directory, and thus should be run at the root of the build sandbox.
-  * This patch should be generated when all of the other patches are applied to prevent dependencies on this patch
+* `generate_domain_substitution_list.sh` - This script generates a list of files containing domain names to be replaced
+  * It should be run at the root of the build sandbox.
+  * This script is not needed during the normal build process
+* `domain_substitution_list` - This file is generated by `generate_domain_substitution_list.sh` for a specific version of Chromium
+  * It assumes that the cleaning list has been applied
+* `domain_regex_list` - The list of regular expressions used to substitute domain names
+* `evaluate_domain_substitution_list.py` - This script evaluates the domain substitution list in the current working directory
 * `generate_cleaning_list.sh` - This script generates a file list of mostly all binary files to be stripped from the source tree.
   * It should be run at the root of the build sandbox.
+  * This script is not needed during the normal build process
+* `cleaning_list` - This file is generated by `generate_cleaning_list.sh` for a specific version of Chromium
+* `evaluate_cleaning_list.py` - This script evaluates the cleaning list in the current working directory
 * `generate_debian_scripts.sh` - This script creates a dpkg debian directory in the build sandbox for Debian
 * `generate_ubuntu_scripts.sh` - This script creates a dpkg debian directory in the build sandbox for Ubuntu
 * `build_debian.sh` - This is the main build script for Debian and derivative distributions. It handles the downloading, patching, and building of .deb packages.
   * This script will invoke the other scripts to do certain tasks.
   * It accepts arguments; pass in `-h` for more information.
-  * Currently, only Debian Stretch 64-bit and Ubuntu Wily 64-bit are tested.
 
 ### Debian and derivatives
-**NOTE:** Instructions are tested on Debian Jessie 64-bit and Stretch 64-bit, and Ubuntu Wily 64-bit
+Distributions supported: Debian Jessie 64-bit and Stretch 64-bit, and Ubuntu Xenial 64-bit
+This may work on other Debian-based distributions and 32-bit systems
 
-**Debian Jessie users**: ungoogled-chromium is configured to build against the system's [FFmpeg](https://www.ffmpeg.org/) (available in Stretch and onwards); [Libav](http://libav.org) (used in Jessie) will not work. However, FFmpeg is available in `jessie-backports`. To install it, add `jessie-backports` to the apt sources, and then install `libavutil-dev`, `libavcodec-dev`, and `libavformat-dev` from it. Note that this will replace Libav.
+**Note for Debian Jessie users**: ungoogled-chromium is configured to build against the system's [FFmpeg](https://www.ffmpeg.org/) (available in Stretch and onwards); [Libav](http://libav.org) (used in Jessie) will not work. However, FFmpeg is available in `jessie-backports`. To install it, add `jessie-backports` to the apt sources, and then install `libavutil-dev`, `libavcodec-dev`, and `libavformat-dev` from it. Note that this will replace Libav.
 
 Run these steps on the system you want to build packages for.
 
     git clone https://github.com/Eloston/ungoogled-chromium.git
     cd ungoogled-chromium
+    git checkout $(git describe --tags `git rev-list --tags --max-count=1`) # Checkout newest tag
     # Run dpkg-checkbuilddeps to find packages needed for building
     ./build_debian.sh -A
 
@@ -92,13 +101,13 @@ TODO
 
 ### Other systems, platforms, and configurations
 
-Consult the build instructions on the [Chromium homepage](http://www.chromium.org/Home) to learn how to build Chromium for your system.
+Consult the build instructions on the [Chromium homepage](http://www.chromium.org/Home) to build Chromium for your system.
 
 The patches in the `patches` directory should work for any build of Chromium. They assume a clean and unmodified Chromium source tree.
 
 These patches are also tested with the GYP flags defined in [`build_templates/debian/rules`](build_templates/debian/rules). Note that enabling some flags, such as `safe_browsing`, may cause the build to fail.
 
-Note about `generate_domain_patch.sh`: This patch will break URLs in the source tree pointing to Google servers. If your building steps requires additional downloads (such as the the PNaCl toolkit), scripts in the source tree may fail to work.
+Note about `domain_substitution_list`: Replacing the domains will break URLs in the source tree pointing to Google servers. If your building steps requires additional downloads (such as the the PNaCl toolkit), scripts in the source tree may fail to work.
 
 ## Contributing
 

build_debian.sh

@@ -170,8 +170,8 @@ fi
 cd "$SANDBOX_PATH";
 
 if [[ $EVALUATE_CLEANING_LIST -eq 1 ]]; then
-    echo "Deleting files in cleaning list..."
-    $SCRIPT_DIR/delete_files_in_list.py $SCRIPT_DIR/cleaning_list
+    echo "Evaluating cleaning list..."
+    $SCRIPT_DIR/evaluate_cleaning_list.py $SCRIPT_DIR/cleaning_list
 fi
 
 if [[ $GENERATE_BUILD_SCRIPTS -eq 1 ]]; then

build_templates/debian/rules

@@ -57,7 +57,6 @@ defines+=use_ozone=0 \
 		enable_prod_wallet_service=0 \
 		enable_supervised_users=0 \
 		enable_one_click_signin=0 \
-		configuration_policy=0 \
 		disable_newlib=1 \
 		icu_use_data_file_flag=0
 

domain_regex_list

@@ -0,0 +1,16 @@
+fonts\.googleapis\.com#about:blank
+google([A-Za-z\-]*)\.com#9oo91e\1\.qjz9zk
+gstatic([A-Za-z\-]*)\.com#95tat1c\1\.qjz9zk
+chrome([A-Za-z\-]*)\.com#ch40me\1\.qjz9zk
+chromium([A-Za-z\-]*)\.org#ch40m1um\1\.qjz9zk
+mozilla([A-Za-z\-]*)\.org#m0z111a\1\.qjz9zk
+facebook([A-Za-z\-]*)\.com#f8c3b00k\1\.qjz9zk
+appspot([A-Za-z\-]*)\.com#8pp2p8t\1\.qjz9zk
+youtube([A-Za-z\-]*)\.com#y0u1ub3\1\.qjz9zk
+ytimg([A-Za-z\-]*)\.com#yt1mg\1\.qjz9zk
+gmail([A-Za-z\-]*)\.com#9ma1l\1\.qjz9zk
+doubleclick([A-Za-z\-]*)\.net#60u613cl1c4\1\.qjz9zk
+googlezip\.net#9oo91e21p\.qjz9zk
+beacons([1-9]?)\.gvt([1-9]?)\.com#b3ac0n2\1\.9vt\2\.qjz9zk
+ggpht\.com#99pht\.qjz9zk
+microsoft\.com#m1cr050ft\.qjz9zk

evaluate_cleaning_list.py

@@ -2,7 +2,7 @@
 
 # Delete files in list generated by generate_cleaning_list.sh
 # This script is designed to run cross-platform
-# Usage: delete_files_in_list.py file_list
+# Usage: evaluate_cleaning_list.py file_list
 
 import sys
 import os

evaluate_domain_substitution_list.py

@@ -0,0 +1,29 @@
+#!/usr/bin/env python
+
+# Substitute domain names with domain_regex_list for files in domain_substitution_list
+# This script is designed to run cross-platform
+# Usage: evaluate_domain_substitution_list.py file_list
+
+import sys
+import os
+import os.path
+
+if __name__ == "__main__":
+    if not len(sys.argv) > 2:
+        print "Usage: {} domain_regex_list domain_substitution_list".format(sys.argv[0])
+        exit(1)
+    regex_list_path = sys.argv[1]
+    file_list_path = sys.argv[2]
+
+    # TODO: Parse domain_regex_list
+
+    with open(file_list_path) as file_list:
+        for line in file_list:
+            line = line.replace("\n", "")
+            if len(line) > 0:
+                line = os.path.normpath(line)
+                if os.path.isfile(line):
+                    # TODO: Checking and substitution
+                else:
+                    print "Not a file: {}".format(line)
+    print "Done evaluating {sub} with {regex}".format(sub=file_list_path, regex=regex_list_path)

generate_cleaning_list.sh

@@ -1,4 +1,5 @@
 # A script that prints files to be removed
+# This script's output is cleaning_list
 
 # Delete all binary files
 find . -path ./debian -prune \

generate_domain_patch.sh

@@ -1,52 +0,0 @@
-#!/bin/bash
-
-# A script that prints a patch for replacing domains
-
-read -r -d '' make_diff_script << EOF
-printf -- '--- a/{}\n+++ b/{}\n'
-sed -r \
--e 's/google([A-Za-z\-]*)\.com/9oo91e\1\.qjz9zk/g' \
--e 's/gstatic([A-Za-z\-]*)\.com/95tat1c\1\.qjz9zk/g' \
--e 's/chrome([A-Za-z\-]*)\.com/ch40me\1\.qjz9zk/g' \
--e 's/chromium([A-Za-z\-]*)\.org/ch40m1um\1\.qjz9zk/g' \
--e 's/mozilla([A-Za-z\-]*)\.org/m0z111a\1\.qjz9zk/g' \
--e 's/facebook([A-Za-z\-]*)\.com/f8c3b00k\1\.qjz9zk/g' \
--e 's/appspot([A-Za-z\-]*)\.com/8pp2p8t\1\.qjz9zk/g' \
--e 's/youtube([A-Za-z\-]*)\.com/y0u1ub3\1\.qjz9zk/g' \
--e 's/ytimg([A-Za-z\-]*)\.com/yt1mg\1\.qjz9zk/g' \
--e 's/gmail([A-Za-z\-]*)\.com/9ma1l\1\.qjz9zk/g' \
--e 's/doubleclick([A-Za-z\-]*)\.net/60u613cl1c4\1\.qjz9zk/g' \
--e 's/googlezip\.net/9oo91e21p\.qjz9zk/g' \
--e 's/beacons([1-9]?)\.gvt([1-9]?)\.com/b3ac0n2\1\.9vt\2\.qjz9zk/g' \
--e 's/ggpht\.com/99pht\.qjz9zk/g' \
--e 's/microsoft\.com/m1cr050ft\.qjz9zk/g' '{}' | diff -U 0 '{}' - | tail --lines=+3
-EOF
-
-
-find . -path ./debian -prune \
--o -type f \( -name "*.h" \
-    -o -name "*.hh" \
-    -o -name "*.hpp" \
-    -o -name "*.hxx" \
-    -o -name "*.cc" \
-    -o -name "*.cpp" \
-    -o -name "*.cxx" \
-    -o -name "*.c" \
-    -o -name "*.h" \
-    -o -name "*.json" \
-    -o -name "*.js" \
-    -o -name "*.html" \
-    -o -name "*.htm" \
-    -o -name "*.py*" \
-    -o -name "*.grd" \
-    -o -name "*.sql" \
-    -o -name "*.idl" \
-    -o -name "*.mk" \
-    -o -name "*.gyp*" \
-    -o -name "Makefile" \
-    -o -name "makefile" \
-    -o -name "*.txt" \
-    -o -name "*.xml" \
-    -o -name "*.mm" \
-    -o -name "*.jinja*" \) \
--printf '%P\n' | xargs -L1 -I{} sh -c "$make_diff_script"

generate_domain_substitution_list.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# A script that prints a list of files containing domains to be replaced
+# This script's output is domain_substitution_list
+
+# TODO: Construct grep command from entries in domain_regex_list
+read -r -d '' print_if_match << EOF
+if grep -qE \
+-e 'google([A-Za-z\-]*)\.com' \
+-e 'gstatic([A-Za-z\-]*)\.com' \
+-e 'chrome([A-Za-z\-]*)\.com' \
+-e 'chromium([A-Za-z\-]*)\.org' \
+-e 'mozilla([A-Za-z\-]*)\.org' \
+-e 'facebook([A-Za-z\-]*)\.com' \
+-e 'appspot([A-Za-z\-]*)\.com' \
+-e 'youtube([A-Za-z\-]*)\.com' \
+-e 'ytimg([A-Za-z\-]*)\.com' \
+-e 'gmail([A-Za-z\-]*)\.com' \
+-e 'doubleclick([A-Za-z\-]*)\.net' \
+-e 'googlezip\.net' \
+-e 'beacons([1-9]?)\.gvt([1-9]?)\.com' \
+-e 'ggpht\.com' \
+-e 'microsoft\.com' '{}'
+then
+printf '{}\n';
+fi
+EOF
+
+find . -path ./debian -prune \
+-o -path ./.pc -prune \
+-o -path './components/test/*' -prune \
+-o -type f \( -name "*.h" \
+    -o -name "*.hh" \
+    -o -name "*.hpp" \
+    -o -name "*.hxx" \
+    -o -name "*.cc" \
+    -o -name "*.cpp" \
+    -o -name "*.cxx" \
+    -o -name "*.c" \
+    -o -name "*.h" \
+    -o -name "*.json" \
+    -o -name "*.js" \
+    -o -name "*.html" \
+    -o -name "*.htm" \
+    -o -name "*.py*" \
+    -o -name "*.grd" \
+    -o -name "*.sql" \
+    -o -name "*.idl" \
+    -o -name "*.mk" \
+    -o -name "*.gyp*" \
+    -o -name "Makefile" \
+    -o -name "makefile" \
+    -o -name "*.txt" \
+    -o -name "*.xml" \
+    -o -name "*.mm" \
+    -o -name "*.jinja*" \) \
+-printf '%P\n' | xargs -L1 -I{} sh -c "$print_if_match"

patches/patch_order

@@ -80,4 +80,5 @@ ungoogled-chromium/disable-copresence.patch
 ungoogled-chromium/disable-gcm.patch
 ungoogled-chromium/disable-domain-reliability.patch
 ungoogled-chromium/intercept-all-modified-domains.patch
+ungoogled-chromium/disable-intranet-redirect-detector.patch
 

patches/ungoogled-chromium/disable-intranet-redirect-detector.patch

@@ -0,0 +1,16 @@
+# Disables the intranet redirect detector. It generates extra DNS requests and the functionality using this is disabled
+# See this page for more information: https://mikewest.org/2012/02/chrome-connects-to-three-random-domains-at-startup
+
+--- a/chrome/browser/intranet_redirect_detector.cc
++++ b/chrome/browser/intranet_redirect_detector.cc
+@@ -71,9 +71,7 @@ void IntranetRedirectDetector::FinishSleep() {
+   STLDeleteElements(&fetchers_);
+   resulting_origins_.clear();
+ 
+-  const base::CommandLine* cmd_line = base::CommandLine::ForCurrentProcess();
+-  if (cmd_line->HasSwitch(switches::kDisableBackgroundNetworking))
+-    return;
++  return;
+ 
+   DCHECK(fetchers_.empty() && resulting_origins_.empty());
+