Updates #1453. Squashed commit of the following: commit f08f68ef5493dad03d3eb120d886f2df1af28be6 Merge: b70b088af54aee2272Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Tue Aug 8 19:04:06 2023 +0300 Merge branch 'master' into 1453-stats-tests commit b70b088af0fdc7d6d048d688160048bad1fceb12 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Aug 3 19:32:04 2023 +0300 stats: imp code commit c341012ba61894c255c1868624be1cac0d26a6fa Merge: a2ac8c34e5eb3cd0f9Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Aug 3 13:36:24 2023 +0300 Merge branch 'master' into 1453-stats-tests commit a2ac8c34ee32606ca5e259c3e2a47db0dd5858de Author: Ildar Kamalov <ik@adguard.com> Date: Thu Aug 3 13:25:12 2023 +0300 client: add top upstreams and average processing time tables commit 11118947f9bf945be0b056f8475cf3b848c6e66e Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Tue Aug 1 17:24:57 2023 +0300 stats: imp docs commit 904cf81d02a1f327b9647fa7ad9e181cfabb68a4 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Mon Jul 31 17:34:06 2023 +0300 stats: imp code commit 34f0c96dd5865d1470385322a88842dd0b3d996d Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Mon Jul 31 15:43:46 2023 +0300 all: imp docs commit 2cb2d0d8bef3580f64bc25c414fe9b5ea6b9f997 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Fri Jul 28 17:24:31 2023 +0300 all: imp code commit 5251a899fecc21e50a0ba06042f96f5b404e196a Merge: b6c2b12d4300821a7fAuthor: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Jul 27 20:34:39 2023 +0300 Merge branch 'master' into 1453-stats-tests commit b6c2b12d4425012efd73549c3a426735f3a677cd Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Jul 27 20:32:18 2023 +0300 stats: imp code commit 5546b82a78326f9cc6d8c87df5083f8fc66a0178 Merge: 8a3d6b1b45f8fa006cAuthor: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Jul 27 14:24:01 2023 +0300 Merge branch 'master' into 1453-stats-tests commit 8a3d6b1b49ce189f95adfa7406a34108e885e676 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Jul 27 14:17:47 2023 +0300 all: imp code commit 2a48001e275e3cdcf70e13e1c9cebd4e502f3259 Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Tue Jul 25 18:27:20 2023 +0300 all: imp docs commit 3dd21890175af32a3368378f7e013383f6d040ec Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Tue Jul 25 16:00:39 2023 +0300 all: imp naming commit 6124456fc3149b71f6bd58d35ecf24eb6cf40d5d Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Thu Jul 20 16:15:56 2023 +0300 all: add upstreams avg processing time commit 187ad0c77a81c9fd95c24e23141355db2e83e50d Author: Stanislav Chzhen <s.chzhen@adguard.com> Date: Tue Jul 18 16:42:19 2023 +0300 all: add top upstreams
AdGuard Home Scripts
hooks/: Git Hooks
Usage
Run make init from the project root.
querylog/: Query Log Helpers
Usage
npm install: install dependencies. Run this first.npm run anonymize <source> <dst>: read the query log from the<source>and write anonymized version to<dst>.
make/: Makefile scripts
The release channels are: development (the default), edge, beta, and
release. If verbosity levels aren't documented here, there are only two: 0,
don't print anything, and 1, be verbose.
build-docker.sh: Build a multi-architecture Docker image
Required environment:
-
CHANNEL: release channel, see above. -
COMMIT: current Git revision. -
DIST_DIR: the directory where a release has previously been built. -
VERSION: release version.
Optional environment:
-
DOCKER_IMAGE_NAME: the name of the resulting Docker container. By default it'sadguardhome-dev. -
DOCKER_OUTPUT: the--outputparameters. By default they aretype=image,name=${DOCKER_IMAGE_NAME},push=false. -
SUDO: allow users to usesudoordoaswithdocker. By default none is used.
build-release.sh: Build a release for all platforms
Required environment:
-
CHANNEL: release channel, see above. -
GPG_KEYandGPG_KEY_PASSPHRASE: data forgpg. Only required ifSIGNis1.
Optional environment:
-
ARCHandOS: space-separated list of architectures and operating systems for which to build a release. For example, to build only for 64-bit ARM and AMD on Linux and Darwin:make ARCH='amd64 arm64' OS='darwin linux' … build-releaseThe default value is
'', which means build everything. -
DIST_DIR: the directory to build a release into. The default value isdist. -
GO: set an alternative name for the Go compiler. -
SIGN:0to not sign the resulting packages,1to sign. The default value is1. -
VERBOSE:1to be verbose,2to also print environment. This script callsgo-build.shwith the verbosity level one level lower, so to get verbosity level2ingo-build.sh, set this to3when callingbuild-release.sh. -
VERSION: release version. Will be set byversion.shif it is unset or if it has the defaultMakefilevalue ofv0.0.0.
clean.sh: Cleanup
Optional environment:
GO: set an alternative name for the Go compiler.
Required environment:
DIST_DIR: the directory where a release has previously been built.
go-bench.sh: Run backend benchmarks
Optional environment:
-
GO: set an alternative name for the Go compiler. -
TIMEOUT_FLAGS: set timeout flags for tests. The default value is--timeout=30s. -
VERBOSE: verbosity level.1shows every command that is run and every Go package that is processed.2also shows subcommands and environment. The default value is0, don't be verbose.
go-build.sh: Build the backend
Optional environment:
-
GOARM: ARM processor options for the Go compiler. -
GOMIPS: ARM processor options for the Go compiler. -
GO: set an alternative name for the Go compiler. -
OUT: output binary name. -
PARALLELISM: set the maximum number of concurrently run build commands (that is, compiler, linker, etc.). -
SOURCE_DATE_EPOCH: the standardized environment variable for the Unix epoch time of the latest commit in the repository. If set, overrides the default obtained from Git. Useful for reproducible builds. -
VERBOSE: verbosity level.1shows every command that is run and every Go package that is processed.2also shows subcommands and environment. The default value is0, don't be verbose. -
VERSION: release version. Will be set byversion.shif it is unset or if it has the defaultMakefilevalue ofv0.0.0.
Required environment:
CHANNEL: release channel, see above.
go-deps.sh: Install backend dependencies
Optional environment:
-
GO: set an alternative name for the Go compiler. -
VERBOSE: verbosity level.1shows every command that is run and every Go package that is processed.2also shows subcommands and environment. The default value is0, don't be verbose.
go-fuzz.sh: Run backend fuzz tests
Optional environment:
-
GO: set an alternative name for the Go compiler. -
FUZZTIME_FLAGS: set fuss flags for tests. The default value is--fuzztime=20s. -
TIMEOUT_FLAGS: set timeout flags for tests. The default value is--timeout=30s. -
VERBOSE: verbosity level.1shows every command that is run and every Go package that is processed.2also shows subcommands and environment. The default value is0, don't be verbose.
go-lint.sh: Run backend static analyzers
Don't forget to run make go-tools once first!
Optional environment:
-
EXIT_ON_ERROR: if set to0, don't exit the script after the first encountered error. The default value is1. -
GO: set an alternative name for the Go compiler. -
VERBOSE: verbosity level.1shows every command that is run.2also shows subcommands. The default value is0, don't be verbose.
go-test.sh: Run backend tests
Optional environment:
-
GO: set an alternative name for the Go compiler. -
RACE: set to0to not use the Go race detector. The default value is1, use the race detector. -
TIMEOUT_FLAGS: set timeout flags for tests. The default value is--timeout=30s. -
VERBOSE: verbosity level.1shows every command that is run and every Go package that is processed.2also shows subcommands. The default value is0, don't be verbose.
go-tools.sh: Install backend tooling
Installs the Go static analysis and other tools into ${PWD}/bin. Either add
${PWD}/bin to your $PATH before all other entries, or use the commands
directly, or use the commands through make (for example, make go-lint).
Optional environment:
GO: set an alternative name for the Go compiler.
version.sh: Generate And Print The Current Version
Required environment:
CHANNEL: release channel, see above.
snap/: Snapcraft scripts
build.sh
Builds the Snapcraft packages from the binaries created by download.sh.
download.sh
Downloads the binaries to pack them into Snapcraft packages.
Required environment:
CHANNEL: release channel, see above.
upload.sh
Uploads the Snapcraft packages created by build.sh.
Required environment:
-
SNAPCRAFT_CHANNEL: Snapcraft release channel:edge,beta, orcandidate. -
SNAPCRAFT_STORE_CREDENTIALS: Credentials for Snapcraft store.
Optional environment:
SNAPCRAFT_CMD: Overrides the Snapcraft command. Default:snapcraft.
translations/: Twosky Integration Script
Usage
-
go run ./scripts/translations help: print usage. -
go run ./scripts/translations download [-n <count>]: download and save all translations.nis optional flag where count is a number of concurrent downloads. -
go run ./scripts/translations upload: upload the baseenlocale. -
go run ./scripts/translations summary: show the current locales summary. -
go run ./scripts/translations unused: show the list of unused strings. -
go run ./scripts/translations auto-add: add locales with additions to the git and restore locales with deletions.
After the download you'll find the output locales in the client/src/__locales/
directory.
Optional environment:
-
DOWNLOAD_LANGUAGES: set a list of specific languages todownload. For examplear be bg. If it set toblockerthen script will download only those languages, which need to be fully translated (de en es fr it ja ko pt-br pt-pt ru zh-cn zh-tw). -
UPLOAD_LANGUAGE: set an alternative language forupload. -
TWOSKY_URI: set an alternative URL fordownloadorupload. -
TWOSKY_PROJECT_ID: set an alternative project ID fordownloadorupload.
companiesdb/: Whotracks.me Database Converter
A simple script that downloads and updates the companies DB in the client
code from the repo.
Usage
sh ./scripts/companiesdb/download.sh
blocked-services/: Blocked Services Updater
A simple script that downloads and updates the blocked services index from AdGuard's Hostlists Registry.
Optional environment:
URL: the URL of the index file. By default it'shttps://adguardteam.github.io/HostlistsRegistry/assets/services.json.
Usage
go run ./scripts/blocked-services/main.go
vetted-filters/: Vetted Filters Updater
Similar to the one above, a script that downloads and updates the vetted filtering list data from AdGuard's Hostlists Registry.
Optional environment:
URL: the URL of the index file. By default it'shttps://adguardteam.github.io/HostlistsRegistry/assets/filters.json.
Usage
go run ./scripts/vetted-filters/main.go