daxi20/AdGuardHome
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

all: sync with master; upd chlog

Browse Source
This commit is contained in:
Ainar Garipov
2023-07-03 14:10:40 +03:00
parent cadb765b7d
commit b22b16d98c
140 changed files with 6739 additions and 2521 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
scripts/README.md
View File

@@ -2,7 +2,7 @@
## `hooks/`: Git Hooks
### Usage
### Usage
Run `make init` from the project root.
@@ -10,7 +10,7 @@ Run `make init` from the project root.
## `querylog/`: Query Log Helpers
### Usage
### Usage
* `npm install`: install dependencies. Run this first.
* `npm run anonymize <source> <dst>`: read the query log from the `<source>`
@@ -26,157 +26,215 @@ don't print anything, and `1`, be verbose.
### `build-docker.sh`: Build A Multi-Architecture Docker Image
### `build-docker.sh`: Build A Multi-Architecture Docker Image
Required environment:
* `CHANNEL`: release channel, see above.
* `COMMIT`: current Git revision.
* `DIST_DIR`: the directory where a release has previously been built.
* `VERSION`: release version.
Optional environment:
* `DOCKER_IMAGE_NAME`: the name of the resulting Docker container. By default
it's `adguardhome-dev`.
* `DOCKER_OUTPUT`: the `--output` parameters. By default they are
`type=image,name=${DOCKER_IMAGE_NAME},push=false`.
* `SUDO`: allow users to use `sudo` or `doas` with `docker`. By default none
is used.
### `build-release.sh`: Build A Release For All Platforms
### `build-release.sh`: Build A Release For All Platforms
Required environment:
* `CHANNEL`: release channel, see above.
* `GPG_KEY` and `GPG_KEY_PASSPHRASE`: data for `gpg`. Only required if `SIGN`
is `1`.
Optional environment:
* `ARCH` and `OS`: space-separated list of architectures and operating systems
for which to build a release. For example, to build only for 64-bit ARM and
AMD on Linux and Darwin:
```sh
make ARCH='amd64 arm64' OS='darwin linux' … build-release
```
The default value is `''`, which means build everything.
* `BUILD_SNAP`: `0` to not build Snapcraft packages, `1` to build. The
default value is `1`.
* `DIST_DIR`: the directory to build a release into. The default value is
`dist`.
* `GO`: set an alternative name for the Go compiler.
* `SIGN`: `0` to not sign the resulting packages, `1` to sign. The default
value is `1`.
* `VERBOSE`: `1` to be verbose, `2` to also print environment. This script
calls `go-build.sh` with the verbosity level one level lower, so to get
verbosity level `2` in `go-build.sh`, set this to `3` when calling
`build-release.sh`.
* `VERSION`: release version. Will be set by `version.sh` if it is unset or
if it has the default `Makefile` value of `v0.0.0`.
### `clean.sh`: Cleanup
### `clean.sh`: Cleanup
Optional environment:
* `GO`: set an alternative name for the Go compiler.
Required environment:
* `DIST_DIR`: the directory where a release has previously been built.
### `go-build.sh`: Build The Backend
### `go-build.sh`: Build The Backend
Optional environment:
* `GOARM`: ARM processor options for the Go compiler.
* `GOMIPS`: ARM processor options for the Go compiler.
* `GO`: set an alternative name for the Go compiler.
* `OUT`: output binary name.
* `PARALLELISM`: set the maximum number of concurrently run build commands
(that is, compiler, linker, etc.).
* `SOURCE_DATE_EPOCH`: the [standardized][repr] environment variable for the
Unix epoch time of the latest commit in the repository. If set, overrides
the default obtained from Git. Useful for reproducible builds.
* `VERBOSE`: verbosity level. `1` shows every command that is run and every
Go package that is processed. `2` also shows subcommands and environment.
The default value is `0`, don't be verbose.
* `VERSION`: release version. Will be set by `version.sh` if it is unset or
if it has the default `Makefile` value of `v0.0.0`.
Required environment:
* `CHANNEL`: release channel, see above.
[repr]: https://reproducible-builds.org/docs/source-date-epoch/
### `go-deps.sh`: Install Backend Dependencies
### `go-deps.sh`: Install Backend Dependencies
Optional environment:
* `GO`: set an alternative name for the Go compiler.
* `VERBOSE`: verbosity level. `1` shows every command that is run and every
Go package that is processed. `2` also shows subcommands and environment.
The default value is `0`, don't be verbose.
### `go-lint.sh`: Run Backend Static Analyzers
### `go-lint.sh`: Run Backend Static Analyzers
Don't forget to run `make go-tools` once first!
Optional environment:
* `EXIT_ON_ERROR`: if set to `0`, don't exit the script after the first
encountered error. The default value is `1`.
* `GO`: set an alternative name for the Go compiler.
* `VERBOSE`: verbosity level. `1` shows every command that is run. `2` also
shows subcommands. The default value is `0`, don't be verbose.
### `go-test.sh`: Run Backend Tests
### `go-test.sh`: Run Backend Tests
Optional environment:
* `GO`: set an alternative name for the Go compiler.
* `RACE`: set to `0` to not use the Go race detector. The default value is
`1`, use the race detector.
* `TIMEOUT_FLAGS`: set timeout flags for tests. The default value is
`--timeout 30s`.
* `VERBOSE`: verbosity level. `1` shows every command that is run and every
Go package that is processed. `2` also shows subcommands. The default
value is `0`, don't be verbose.
### `go-tools.sh`: Install Backend Tooling
### `go-tools.sh`: Install Backend Tooling
Installs the Go static analysis and other tools into `${PWD}/bin`. Either add
`${PWD}/bin` to your `$PATH` before all other entries, or use the commands
directly, or use the commands through `make` (for example, `make go-lint`).
Optional environment:
* `GO`: set an alternative name for the Go compiler.
### `version.sh`: Generate And Print The Current Version
### `version.sh`: Generate And Print The Current Version
Required environment:
* `CHANNEL`: release channel, see above.
## `snap/`: Snap GUI Files
## `snap/`: Snapcraft scripts
App icons (see https://github.com/AdguardTeam/AdGuardHome/pull/1836), Snap
manifest file templates, and helper scripts.
### `build.sh`
Builds the Snapcraft packages from the binaries created by `download.sh`.
### `download.sh`
Downloads the binaries to pack them into Snapcraft packages.
Required environment:
* `CHANNEL`: release channel, see above.
### `upload.sh`
Uploads the Snapcraft packages created by `build.sh`.
Required environment:
* `SNAPCRAFT_CHANNEL`: Snapcraft release channel: `edge`, `beta`, or
`candidate`.
* `SNAPCRAFT_STORE_CREDENTIALS`: Credentials for Snapcraft store.
Optional environment:
* `SNAPCRAFT_CMD`: Overrides the Snapcraft command. Default: `snapcraft`.
## `translations/`: Twosky Integration Script
### Usage
### Usage
* `go run main.go help`: print usage.
@@ -211,7 +269,7 @@ Optional environment:
A simple script that downloads and updates the companies DB in the `client`
code from [the repo][companiesrepo].
### Usage
### Usage
```sh
sh ./scripts/companiesdb/download.sh
@@ -231,7 +289,7 @@ Optional environment:
* `URL`: the URL of the index file. By default it's
`https://adguardteam.github.io/HostlistsRegistry/assets/services.json`.
### Usage
### Usage
```sh
go run ./scripts/blocked-services/main.go
@@ -251,7 +309,7 @@ Optional environment:
* `URL`: the URL of the index file. By default it's
`https://adguardteam.github.io/HostlistsRegistry/assets/filters.json`.
### Usage
### Usage
```sh
go run ./scripts/vetted-filters/main.go

39
scripts/make/build-docker.sh
View File

@@ -5,11 +5,12 @@ verbose="${VERBOSE:-0}"
if [ "$verbose" -gt '0' ]
then
set -x
debug_flags='-D'
debug_flags='--debug=1'
else
set +x
debug_flags=''
debug_flags='--debug=0'
fi
readonly debug_flags
set -e -f -u
@@ -61,21 +62,22 @@ readonly docker_output
case "$channel"
in
('release')
docker_image_full_name="${docker_image_name}:${version}"
docker_tags="--tag ${docker_image_name}:latest"
docker_version_tag="--tag=${docker_image_name}:${version}"
docker_channel_tag="--tag=${docker_image_name}:latest"
;;
('beta')
docker_image_full_name="${docker_image_name}:${version}"
docker_tags="--tag ${docker_image_name}:beta"
docker_version_tag="--tag=${docker_image_name}:${version}"
docker_channel_tag="--tag=${docker_image_name}:beta"
;;
('edge')
# Don't set the version tag when pushing to the edge channel.
docker_image_full_name="${docker_image_name}:edge"
docker_tags=''
# Set the version tag to an empty string when pushing to the edge channel.
docker_version_tag=''
docker_channel_tag="--tag=${docker_image_name}:edge"
;;
('development')
docker_image_full_name="${docker_image_name}"
docker_tags=''
# Set both tags to an empty string for development builds.
docker_version_tag=''
docker_channel_tag=''
;;
(*)
echo "invalid channel '$channel', supported values are\
@@ -83,7 +85,7 @@ in
exit 1
;;
esac
readonly docker_image_full_name docker_tags
readonly docker_version_tag docker_channel_tag
# Copy the binaries into a new directory under new names, so that it's easier to
# COPY them later. DO NOT remove the trailing underscores. See file
@@ -117,10 +119,13 @@ cp "./docker/web-bind.awk"\
cp "./docker/healthcheck.sh"\
"${dist_docker_scripts}/healthcheck.sh"
# Don't use quotes with $docker_tags and $debug_flags because we want word
# splitting and or an empty space if tags are empty.
# Don't use quotes with $docker_version_tag and $docker_channel_tag, because we
# want word splitting and or an empty space if tags are empty.
#
# TODO(a.garipov): Once flag --tag of docker buildx build supports commas, use
# them instead.
$sudo_cmd docker\
$debug_flags\
"$debug_flags"\
buildx build\
--build-arg BUILD_DATE="$build_date"\
--build-arg DIST_DIR="$dist_dir"\
@@ -128,7 +133,7 @@ $sudo_cmd docker\
--build-arg VERSION="$version"\
--output "$docker_output"\
--platform "$docker_platforms"\
$docker_tags\
-t "$docker_image_full_name"\
$docker_version_tag\
$docker_channel_tag\
-f ./docker/Dockerfile\
.

112
scripts/make/build-release.sh
View File

@@ -78,14 +78,6 @@ else
fi
readonly oses
snap_enabled="${BUILD_SNAP:-1}"
readonly snap_enabled
if [ "$snap_enabled" -eq '0' ]
then
log 'snap: disabled'
fi
# Require the gpg key and passphrase to be set if the signing is required.
if [ "$sign" -eq '1' ]
then
@@ -106,7 +98,7 @@ log "checking tools"
# Make sure we fail gracefully if one of the tools we need is missing. Use
# alternatives when available.
use_shasum='0'
for tool in gpg gzip sed sha256sum snapcraft tar zip
for tool in gpg gzip sed sha256sum tar zip
do
if ! command -v "$tool" > /dev/null
then
@@ -128,36 +120,36 @@ readonly use_shasum
# Data section. Arrange data into space-separated tables for read -r to read.
# Use a hyphen for missing values.
# os arch arm mips snap
# os arch arm mips
platforms="\
darwin amd64 - - -
darwin arm64 - - -
freebsd 386 - - -
freebsd amd64 - - -
freebsd arm 5 - -
freebsd arm 6 - -
freebsd arm 7 - -
freebsd arm64 - - -
linux 386 - - i386
linux amd64 - - amd64
linux arm 5 - -
linux arm 6 - -
linux arm 7 - armhf
linux arm64 - - arm64
linux mips - softfloat -
linux mips64 - softfloat -
linux mips64le - softfloat -
linux mipsle - softfloat -
linux ppc64le - - -
openbsd amd64 - - -
openbsd arm64 - - -
windows 386 - - -
windows amd64 - - -
windows arm64 - - -"
darwin amd64 - -
darwin arm64 - -
freebsd 386 - -
freebsd amd64 - -
freebsd arm 5 -
freebsd arm 6 -
freebsd arm 7 -
freebsd arm64 - -
linux 386 - -
linux amd64 - -
linux arm 5 -
linux arm 6 -
linux arm 7 -
linux arm64 - -
linux mips - softfloat
linux mips64 - softfloat
linux mips64le - softfloat
linux mipsle - softfloat
linux ppc64le - -
openbsd amd64 - -
openbsd arm64 - -
windows 386 - -
windows amd64 - -
windows arm64 - -"
readonly platforms
# Function build builds the release for one platform. It builds a binary, an
# archive and, if needed, a snap package.
# Function build builds the release for one platform. It builds a binary and an
# archive.
build() {
# Get the arguments. Here and below, use the "build_" prefix for all
# variables local to function build.
@@ -167,7 +159,6 @@ build() {
build_arch="$4"\
build_arm="$5"\
build_mips="$6"\
build_snap="$7"\
;
# Use the ".exe" filename extension if we build a Windows release.
@@ -229,52 +220,13 @@ build() {
esac
log "$build_archive"
# Exit if we don't need to build the Snap package.
if [ "$build_snap" = '-' ] || [ "$snap_enabled" -eq '0' ]
then
return
fi
# Prepare the Snap build.
build_snap_output="./${dist}/AdGuardHome_${build_snap}.snap"
build_snap_dir="${build_snap_output}.dir"
# Create the meta subdirectory and copy files there.
mkdir -p "${build_snap_dir}/meta"
cp "$build_output" './scripts/snap/local/adguard-home-web.sh' "$build_snap_dir"
cp -r './scripts/snap/gui' "${build_snap_dir}/meta/"
# Create a snap.yaml file, setting the values.
sed -e 's/%VERSION%/'"$version"'/'\
-e 's/%ARCH%/'"$build_snap"'/'\
./scripts/snap/snap.tmpl.yaml\
>"${build_snap_dir}/meta/snap.yaml"
# TODO(a.garipov): The snapcraft tool will *always* write everything,
# including errors, to stdout. And there doesn't seem to be a way to change
# that. So, save the combined output, but only show it when snapcraft
# actually fails.
set +e
build_snapcraft_output="$(
snapcraft pack "$build_snap_dir" --output "$build_snap_output" 2>&1
)"
build_snapcraft_exit_code="$?"
set -e
if [ "$build_snapcraft_exit_code" -ne '0' ]
then
log "$build_snapcraft_output"
exit "$build_snapcraft_exit_code"
fi
log "$build_snap_output"
}
log "starting builds"
# Go over all platforms defined in the space-separated table above, tweak the
# values where necessary, and feed to build.
echo "$platforms" | while read -r os arch arm mips snap
echo "$platforms" | while read -r os arch arm mips
do
# See if the architecture or the OS is in the allowlist. To do so, try
# removing everything that matches the pattern (well, a prefix, but that
@@ -314,7 +266,7 @@ do
;;
esac
build "$dir" "$ar" "$os" "$arch" "$arm" "$mips" "$snap"
build "$dir" "$ar" "$os" "$arch" "$arm" "$mips"
done
log "packing frontend"
@@ -413,14 +365,14 @@ do
platform="$f"
# Remove the prefix.
platform="${platform#./${dist}/AdGuardHome_}"
platform="${platform#"./${dist}/AdGuardHome_"}"
# Remove the filename extensions.
platform="${platform%.zip}"
platform="${platform%.tar.gz}"
# Use the filename's base path.
filename="${f#./${dist}/}"
filename="${f#"./${dist}/"}"
if [ "$i" -eq "$ar_files_len" ]
then

34
scripts/make/go-lint.sh
View File

@@ -130,6 +130,7 @@ underscores() {
-e '_freebsd.go'\
-e '_linux.go'\
-e '_little.go'\
-e '_next.go'\
-e '_openbsd.go'\
-e '_others.go'\
-e '_test.go'\
@@ -169,6 +170,18 @@ run_linter govulncheck ./...
run_linter gocyclo --over 10 .
# TODO(a.garipov): Enable for all.
run_linter gocognit --over 10\
./internal/aghalg/\
./internal/aghchan/\
./internal/aghhttp/\
./internal/aghio/\
./internal/next/\
./internal/tools/\
./internal/version/\
./internal/whois/\
;
run_linter ineffassign ./...
run_linter unparam ./...
@@ -185,8 +198,25 @@ run_linter nilness ./...
run_linter -e shadow --strict ./...
# TODO(a.garipov): Enable in v0.108.0.
# run_linter gosec --quiet ./...
# TODO(a.garipov): Enable for all.
run_linter gosec --quiet\
./internal/aghalg\
./internal/aghchan\
./internal/aghhttp\
./internal/aghio\
./internal/aghnet\
./internal/aghos\
./internal/aghtest\
./internal/dhcpd\
./internal/dhcpsvc\
./internal/dnsforward\
./internal/next\
./internal/schedule\
./internal/stats\
./internal/tools\
./internal/version\
./internal/whois\
;
# TODO(a.garipov): Enable --blank?
run_linter errcheck --asserts ./...

2
scripts/make/go-tools.sh
View File

@@ -38,6 +38,7 @@ readonly go
rm -f\
bin/errcheck\
bin/fieldalignment\
bin/gocognit\
bin/gocyclo\
bin/gofumpt\
bin/gosec\
@@ -69,6 +70,7 @@ env\
github.com/kisielk/errcheck\
github.com/kyoh86/looppointer/cmd/looppointer\
github.com/securego/gosec/v2/cmd/gosec\
github.com/uudashr/gocognit/cmd/gocognit\
golang.org/x/tools/go/analysis/passes/fieldalignment/cmd/fieldalignment\
golang.org/x/tools/go/analysis/passes/nilness/cmd/nilness\
golang.org/x/tools/go/analysis/passes/shadow/cmd/shadow\

77
scripts/snap/build.sh Normal file
View File

@@ -0,0 +1,77 @@
#!/bin/sh
verbose="${VERBOSE:-0}"
if [ "$verbose" -gt '0' ]
then
set -x
fi
set -e -f -u
# Function log is an echo wrapper that writes to stderr if the caller requested
# verbosity level greater than 0. Otherwise, it does nothing.
#
# TODO(a.garipov): Add to helpers.sh and use more actively in scripts.
log() {
if [ "$verbose" -gt '0' ]
then
# Don't use quotes to get word splitting.
echo "$1" 1>&2
fi
}
version="$( ./AdGuardHome_amd64 --version | cut -d ' ' -f 4 )"
if [ "$version" = '' ]
then
log 'empty version from ./AdGuardHome_amd64'
exit 1
fi
readonly version
log "version '$version'"
for arch in\
'i386'\
'amd64'\
'armhf'\
'arm64'
do
build_output="./AdGuardHome_${arch}"
snap_output="./AdGuardHome_${arch}.snap"
snap_dir="${snap_output}.dir"
# Create the meta subdirectory and copy files there.
mkdir -p "${snap_dir}/meta"
cp "$build_output" "${snap_dir}/AdGuardHome"
cp './snap/local/adguard-home-web.sh' "$snap_dir"
cp -r './snap/gui' "${snap_dir}/meta/"
# Create a snap.yaml file, setting the values.
sed\
-e 's/%VERSION%/'"$version"'/'\
-e 's/%ARCH%/'"$arch"'/'\
./snap/snap.tmpl.yaml\
> "${snap_dir}/meta/snap.yaml"
# TODO(a.garipov): The snapcraft tool will *always* write everything,
# including errors, to stdout. And there doesn't seem to be a way to change
# that. So, save the combined output, but only show it when snapcraft
# actually fails.
set +e
snapcraft_output="$(
snapcraft pack "$snap_dir" --output "$snap_output" 2>&1
)"
snapcraft_exit_code="$?"
set -e
if [ "$snapcraft_exit_code" -ne '0' ]
then
log "$snapcraft_output"
exit "$snapcraft_exit_code"
fi
log "$snap_output"
rm -f -r "$snap_dir"
done

29
scripts/snap/download.sh Normal file
View File

@@ -0,0 +1,29 @@
#!/bin/sh
verbose="${VERBOSE:-0}"
if [ "$verbose" -gt '0' ]
then
set -x
fi
set -e -f -u
channel="${CHANNEL:?please set CHANNEL}"
readonly channel
printf '%s %s\n'\
'386' 'i386'\
'amd64' 'amd64'\
'armv7' 'armhf'\
'arm64' 'arm64' \
| while read -r arch snap_arch
do
release_url="https://static.adtidy.org/adguardhome/${channel}/AdGuardHome_linux_${arch}.tar.gz"
output="./AdGuardHome_linux_${arch}.tar.gz"
curl -o "$output" -v "$release_url"
tar -f "$output" -v -x -z
cp ./AdGuardHome/AdGuardHome "./AdGuardHome_${snap_arch}"
rm -f -r "$output" ./AdGuardHome
done

8
scripts/snap/gui/adguard-home-web.desktop
View File

@@ -1,8 +0,0 @@
[Desktop Entry]
Type=Application
Encoding=UTF-8
Name=AdGuard Home
Comment=Network-wide ads & trackers blocking DNS server
Exec=adguard-home.adguard-home-web
Icon=${SNAP}/meta/gui/adguard-home-web.png
Terminal=false

BIN
scripts/snap/gui/adguard-home-web.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.6 KiB

12
scripts/snap/local/adguard-home-web.sh
View File

@@ -1,12 +0,0 @@
#!/bin/sh
# Get the admin interface port from the configuration.
bind_port="$( grep -e 'bind_port' "${SNAP_DATA}/AdGuardHome.yaml" | awk -F ' ' '{print $2}' )"
readonly bind_port
if [ "$bind_port" = '' ]
then
xdg-open 'http://localhost:3000'
else
xdg-open "http://localhost:${bind_port}"
fi

42
scripts/snap/snap.tmpl.yaml
View File

@@ -1,42 +0,0 @@
# The %VARIABLES% are be replaced by actual values by the build script.
'name': 'adguard-home'
'base': 'core22'
'version': '%VERSION%'
'summary': Network-wide ads & trackers blocking DNS server
'description': |
AdGuard Home is a network-wide software for blocking ads & tracking. After
you set it up, it'll cover ALL your home devices, and you don't need any
client-side software for that.
It operates as a DNS server that re-routes tracking domains to a "black hole,"
thus preventing your devices from connecting to those servers. It's based
on software we use for our public AdGuard DNS servers -- both share a lot
of common code.
'grade': 'stable'
'confinement': 'strict'
'architectures':
- '%ARCH%'
'apps':
'adguard-home':
'command': 'AdGuardHome --no-check-update -w $SNAP_DATA'
'plugs':
# Add the "network-bind" plug to bind to interfaces.
- 'network-bind'
# Add the "network-observe" plug to be able to bind to ports below 1024
# (cap_net_bind_service) and also to bind to a particular interface using
# SO_BINDTODEVICE (cap_net_raw).
- 'network-observe'
# Add the "network-control" plug to be able to use raw sockets in the DHCP
# server.
#
# TODO(a.garipov): If this works, request auto-connect of this plug.
- 'network-control'
'daemon': 'simple'
'restart-condition': 'always'
'adguard-home-web':
'command': 'adguard-home-web.sh'
'plugs':
- 'desktop'

93
scripts/snap/upload.sh Normal file
View File

@@ -0,0 +1,93 @@
#!/bin/sh
verbose="${VERBOSE:-0}"
if [ "$verbose" -gt '0' ]
then
set -x
fi
set -e -f -u
# Function log is an echo wrapper that writes to stderr if the caller requested
# verbosity level greater than 0. Otherwise, it does nothing.
log() {
if [ "$verbose" -gt '0' ]
then
# Don't use quotes to get word splitting.
echo "$1" 1>&2
fi
}
# Do not set a new lowercase variable, because the snapcraft tool expects the
# uppercase form.
if [ "${SNAPCRAFT_STORE_CREDENTIALS:-}" = '' ]
then
log 'please set SNAPCRAFT_STORE_CREDENTIALS'
exit 1
fi
export SNAPCRAFT_STORE_CREDENTIALS
snapcraft_channel="${SNAPCRAFT_CHANNEL:?please set SNAPCRAFT_CHANNEL}"
readonly snapcraft_channel
# Allow developers to overwrite the command, e.g. for testing.
snapcraft_cmd="${SNAPCRAFT_CMD:-snapcraft}"
readonly snapcraft_cmd
default_timeout='90s'
kill_timeout='120s'
readonly default_timeout kill_timeout
for arch in\
'i386'\
'amd64'\
'armhf'\
'arm64'
do
snap_file="./AdGuardHome_${arch}.snap"
# Catch the exit code and the combined output to later inspect it.
set +e
snapcraft_output="$(
# Use timeout(1) to force snapcraft to quit after a certain time. There
# seems to be no environment variable or flag to force this behavior.
timeout\
--preserve-status\
-k "$kill_timeout"\
-v "$default_timeout"\
"$snapcraft_cmd" upload\
--release="${snapcraft_channel}"\
--quiet\
"${snap_file}"\
2>&1
)"
snapcraft_exit_code="$?"
set -e
if [ "$snapcraft_exit_code" -eq '0' ]
then
log "successful upload: ${snapcraft_output}"
continue
fi
# Skip the ones that were failed by a duplicate upload error.
case "$snapcraft_output"
in
(*'A file with this exact same content has already been uploaded'|\
'Error checking upload uniqueness'*)
log "warning: duplicate upload, skipping"
log "snapcraft upload error: ${snapcraft_output}"
continue
;;
(*)
echo "unexpected snapcraft upload error: ${snapcraft_output}"
return "$snapcraft_exit_code"
;;
esac
done

42
scripts/translations/main.go
View File

@@ -288,6 +288,7 @@ func downloadWorker(wg *sync.WaitGroup, client *http.Client, uriCh <-chan *url.U
data, err := getTranslation(client, uri.String())
if err != nil {
log.Error("download worker: getting translation: %s", err)
log.Info("download worker: error response:\n%s", data)
continue
}
@@ -295,6 +296,11 @@ func downloadWorker(wg *sync.WaitGroup, client *http.Client, uriCh <-chan *url.U
q := uri.Query()
code := q.Get("language")
// Fix some TwoSky weirdnesses.
//
// TODO(a.garipov): Remove when those are fixed.
code = strings.ToLower(code)
name := filepath.Join(localesDir, code+".json")
err = os.WriteFile(name, data, 0o664)
if err != nil {
@@ -307,7 +313,8 @@ func downloadWorker(wg *sync.WaitGroup, client *http.Client, uriCh <-chan *url.U
}
}
// getTranslation returns received translation data or error.
// getTranslation returns received translation data and error. If err is not
// nil, data may contain a response from server for inspection.
func getTranslation(client *http.Client, url string) (data []byte, err error) {
resp, err := client.Get(url)
if err != nil {
@@ -319,24 +326,19 @@ func getTranslation(client *http.Client, url string) (data []byte, err error) {
if resp.StatusCode != http.StatusOK {
err = fmt.Errorf("url: %q; status code: %s", url, http.StatusText(resp.StatusCode))
return nil, err
// Go on and download the body for inspection.
}
limitReader, err := aghio.LimitReader(resp.Body, readLimit)
if err != nil {
err = fmt.Errorf("limit reading: %w", err)
return nil, err
limitReader, lrErr := aghio.LimitReader(resp.Body, readLimit)
if lrErr != nil {
// Generally shouldn't happen, since the only error returned by
// [aghio.LimitReader] is an argument error.
panic(fmt.Errorf("limit reading: %w", lrErr))
}
data, err = io.ReadAll(limitReader)
if err != nil {
err = fmt.Errorf("reading all: %w", err)
data, readErr := io.ReadAll(limitReader)
return nil, err
}
return data, nil
return data, errors.WithDeferred(err, readErr)
}
// translationURL returns a new url.URL with provided query parameters.
@@ -344,6 +346,18 @@ func translationURL(oldURL *url.URL, baseFile, projectID string, lang langCode)
uri = &url.URL{}
*uri = *oldURL
// Fix some TwoSky weirdnesses.
//
// TODO(a.garipov): Remove when those are fixed.
switch lang {
case "si-lk":
lang = "si-LK"
case "zh-hk":
lang = "zh-HK"
default:
// Go on.
}
q := uri.Query()
q.Set("format", "json")
q.Set("filename", baseFile)