buchgr
Automatically rotate logs
Right now bazel-remote just logs to stdout/stderr, which means it needs to be spawn by redirecting those to a file and the log files managed externally.
Alternatively, it could use something like lumberjack and rotate the logs itself automatically. It would need some new configuration parameters to determine log location (the default would be under the cache directory), and how often to rotate the logs.
I propose the following new config options:
log_path: path.log # optional, if relative path it goes under cacheDir, defaults to bazel-remote.log
log_max_size: 500 # in megabytes
log_max_backups: 3 # number of backup files to keep around
log_max_age: 28 # how many days to keep the around for
log_compress: true # disabled by default
I think you can achieve this without modifying bazel-remote, eg by running bazel-remote as a systemd unit, or by piping bazel-remote's output to another program which sends the logs to syslog.
You could, but in our case, we run bazel-remote in a distributed fashion. That is, each host that runs bazel has its own local copy of bazel-remote that proxies to a central bazel-remote. We can easily run the central one under systemd or some other log-rotation program, but the distributed ones are a different story – and those logs grow fast.
Could you run the distributed bazel-remote instances as bazel-remote 2>&1 | logger ? I guess you would still need some syslog and/or logrotate configuration. And other platforms (win? mac?) might require something else.
This morning I stumbled across apache's rotatelogs utility, and thought of this. Sounds like it does exactly what you want without requiring syslog: https://httpd.apache.org/docs/2.4/programs/rotatelogs.html
And there's also multilog from daemontools (and various forks) which work similarly: http://cr.yp.to/daemontools/multilog.html
Hi. Faced the same problem.
The straightforward solution was StandardOutput=append:<your access log> in service description. But I can't find a way to force systemd to reopen the log. So all the sources recommend to restart the service.
For the solution recommended here: as I see they rotation logs by thirselves, but I'd like to use logrotate to have common rotation configuration. Usual way to do that is to add postrotate script which sends the signal to process so it reopens the log file in the moment it likes. Here is the lines of nginx logrotate config they provide:
postrotate
invoke-rc.d nginx rotate >/dev/null 2>&1
endscript
Since bazel-remote doesn't have that implemented I guess I'll write a bash script to manage access log. Will look like
[Service]
...
ExecStart=bazel-remote <...> | bazel-remote-logger <access_log_path>
postrotate
invoke-rc.d bazel-remote-logger rotate >/dev/null 2>&1
endscript
But I belive it would be much better to have that code in this repo )
I think systemd has options to do this by itself, see the answer here for some options: https://superuser.com/questions/1202771/how-to-logrotate-with-systemd
If you find something that works well for your needs, maybe we can add it to the examples/bazel-remote.service file.
Well, yes. journalctl rotates itself just fine. But it's really wrong to have access log there. Becomes too huge and skipping some entries
systemd-journald[2074625]: Suppressed 1207 messages from bazel-remote.service
Also I run into another problem: systemctl restart never report failure. Because it takes service as running once it's started.
So I end up with notifying wrapper. This means in .service file you have something like
[Service]
...
Type=notify
# For audit service
NotifyAccess=all
ExecStart= wrapper.sh -a <access_log> -- bazel-remote <args>
Script notifies systemd that service stared when receiving both Starting gRPC server on address and Starting HTTPS server on address messages from bazel-remote. So systemctl restart will wait for that notification and report failure if notification is not received.
Logrotate config looks like:
<access_log> {
...
postrotate
systemctl kill -s USR1 --kill-who=main <service_name>
endscript
To kick log rotation: kill -s USR1 <pid> or systemctl kill -s USR1 --kill-who=main <service_name>
wrapper.sh
#!/bin/bash
function usage() {
cat <<HEREDOC
Usage:
$0 [options] -- BAZEL_REMOTE_COMMAND
Redirects stout of BAZEL_REMOTE_COMMAND to ACCESS_LOG file.
On USR1 signal reopens ACCESS_LOG.
Notifies systemd when bazel-remote started.
Options:
-a,--access-log ACCESS_LOG Access log file
BAZEL_REMOTE_COMMAND Command to execute
-h,--help Show this message and exit
HEREDOC
}
access_log=
OPTS=$(getopt --options=a:h --long=access-log:,help -n '$0' -- "$@")
if [ $? -ne 0 ]; then
echo 'Terminating...' >&2
exit 1
fi
eval set -- "$OPTS"
unset OPTS
while true; do
case "$1" in
-a | --access-log)
access_log=$2
shift 2
continue
;;
-h | --help)
usage
exit 1
;;
--)
shift
break
;;
*)
echo >2 "Internal error"
exit 1
;;
esac
done
if [ $# -eq 0 ]; then
echo >&2 Command is not specified
exit 1
fi
main_pid=$$
echo >&2 "Main pid $main_pid"
function notify() {
systemd-notify --pid=$main_pid "$@" >/dev/null 2>/dev/null
}
function log_pump() {
echo "Log pump started"
local https_started=
local grpc_started=
local notified=
while read -r err_line; do
echo "$err_line"
case "$err_line" in
*"Starting gRPC server on address"*)
grpc_started=1
;;
*"Starting HTTPS server on address"*)
https_started=1
;;
esac
if [ -z "$notified" ] && [ ! -z "$https_started" ] && [ ! -z "$grpc_started" ]; then
notify --ready && notified=1
fi
done
echo "Log pump stopped"
}
# start log pump and redirect stream 5 there. It will stop itself when stream 5 closed
exec 5> >(log_pump >&2)
# this is not documented, but works. If it doesn't we will just wait for invalid pid
log_pump_pid=$!
echo >&2 "Log pump pid $log_pump_pid"
fifo=$(mktemp -u bazel-remote-fifo-XXXXXXXX)
mkfifo -m 700 "$fifo"
# There is a way to not use fifo: `exec 6< <("$@" 2>&5)`
# So it will redirect the output to stream 6,
# but taking its pid with $! is not documented feature.
# Since we have to send SIGTERM there we have to be sure pid is valid.
"$@" 2>&5 1>"$fifo" &
worker_pid=$!
echo >&2 "Worker pid $worker_pid"
function reopen_log() {
if [ ! -z "$access_log" ]; then
exec 4>&- 4>>"$access_log"
echo >&2 Access log reopened
fi
}
stopping_notified=
trap reopen_log USR1
function terminate() {
echo >&2 "Terminating worker"
kill -s SIGTERM "$worker_pid"
notify --stopping && stopping_notified=1
}
trap terminate SIGTERM
if [ -z "$access_log" ]; then
echo >&2 "Access log is not redirected"
exec 4>&1
else
echo "Using access log $access_log"
exec >&2 4>>"$access_log"
fi
echo >&2 "Access pump started"
while read -r log_line; do
echo >&4 "$log_line"
done >&2 <"$fifo"
echo >&2 "Access pump stopped"
# If stop by signal this notification already sent from trap
if [ ! -z "$stopping_notified" ]; then
notify --stopping
fi
echo >&2 "Waiting for worker"
wait "$worker_pid"
# close log pump input to make it exit
exec 5>&-
echo >&2 "Waiting for log pump"
wait "$log_pump_pid" || true # in case pid is invalid
rm -rf "$fifo"