beetbox
Date queries off by one hour (after DST shift)
Problem
When I search for files added in the last hour, I get no results, despite having added files minutes ago. If I subtract one hour from the query, though, the file I just added is found:
pi@media-pi:~ $ date
Thu Apr 1 11:26:16 PDT 2021
pi@media-pi:~ $ beet ls -f '${id} - ${added}' added:'2021-04-01T10:00:00..' id:40427
40427 - 2021-04-01 11:23:32
pi@media-pi:~ $ beet ls -f '${id} - ${added}' added:'2021-04-01T11:00:00..' id:40427
pi@media-pi:~ $ beet ls -f '${id} - ${added}' added:'2021-04-01T10:30:00..' id:40427
pi@media-pi:~ $ beet ls -f '${id} - ${added}' added:'2021-04-01T10:20:00..' id:40427
40427 - 2021-04-01 11:23:32
pi@media-pi:~ $
Expected behavior is that a query for files added after 11:00:00 would find a file added at 11:23:32; actual behavior is that I have to search for files added after 10:00:00, indicating that maybe the query is happening in the wrong time zone context. I'm sure this worked before the daylight saving time shift.
Running this command in verbose (-vv) mode:
pi@media-pi:~ $ beet -vv ls -f '${id} - ${added}' added:'2021-04-01T10:30:00..' id:40427
user configuration: /home/pi/.config/beets/config.yaml
data directory: /home/pi/.config/beets
plugin paths:
Sending event: pluginload
library database: /home/pi/musiclibrary.db
library directory: /mnt/Wurlitzer/Shared Music/Music
Sending event: library_opened
Sending event: cli_exit
Setup
- OS: Raspbian GNU/Linux 10 (buster)
- Python version: 2.7.16
- beets version: 1.4.9
- Turning off plugins made problem go away (yes/no): no
My configuration (output of beet config) is:
play:
command: mocp -O ALSAMixer1=Headphone
warning_threshold: 100
use_folders: no
raw: no
bom: no
relative_to:
asciify_paths: yes
smartplaylist:
playlist_dir: /mnt/Wurlitzer/Playlists
playlists: [{name: Ida Red Tune Of The Day.m3u, query: 'usertags:IdaRed'}, {name: Free Little Birds.m3u, query: 'usertags:FreeLittleBirds'}]
auto: yes
relative_to:
chroma:
auto: yes
convert:
command: /home/pi/.local/bin/anytomp3.sh $source $dest
extension: mp3
never_convert_lossy_files: no
paths: {}
album_art_maxwidth: 0
format: mp3
dest:
auto: no
pretend: no
quiet: no
id3v23: inherit
copy_album_art: no
threads: 4
formats:
alac:
command: ffmpeg -i $source -y -vn -acodec alac $dest
extension: m4a
aac:
command: ffmpeg -i $source -y -vn -acodec aac -aq 1 $dest
extension: m4a
opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
mp3: ffmpeg -i $source -y -vn -aq 2 $dest
flac: ffmpeg -i $source -y -vn -acodec flac $dest
ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest
wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
embed: yes
no_convert: ''
tmpdir:
max_bitrate: 500
library: ~/musiclibrary.db
edit:
itemfields: track title artist album comments composer year genres albumartist
albumfields: album albumartist
ignore_fields: id path
ftintitle:
format: (feat. {0})
auto: yes
drop: no
set_fields:
plugins: 'info playlist duplicates chroma acousticbrainz mbsubmit discogs fetchart edit fuzzy convert play usertag smartplaylist ftintitle
'
directory: /mnt/Wurlitzer/Shared Music/Music
extrafiles:
patterns:
pdf: '*.pdf'
playlist:
auto: yes
playlist_dir: /mnt/Wurlitzer/Playlists
relative_to: library
acoustid:
apikey: REDACTED
duplicates:
count: no
full: no
format: ''
keys: []
move: ''
tag: ''
path: no
copy: ''
tiebreak: {}
album: no
strict: no
checksum: ''
merge: no
delete: no
fuzzy:
threshold: 0.7
prefix: '~'
fetchart:
auto: yes
minwidth: 0
sources:
- filesystem
- coverart
- itunes
- amazon
- albumart
google_engine: 001442825323518660753:hrh5ch1gjzm
enforce_ratio: no
cautious: no
maxwidth: 0
store_source: no
google_key: REDACTED
fanarttv_key: REDACTED
cover_names:
- cover
- front
- art
- album
- folder
mbsubmit:
threshold: medium
format: $track. $title - $artist ($length)
discogs:
tokenfile: discogs_token.json
user_token: REDACTED
apikey: REDACTED
apisecret: REDACTED
source_weight: 0.5
acousticbrainz:
auto: yes
force: no
tags: []
Huh, that does indeed look like a problem! Since time zone issues are so system-dependent, this is going to be unfortunately hard to track down… perhaps someone who is experiencing this issue (perhaps you) can try digging into the DateQuery matching code with some print-debugging to try to nail down exactly what's not being shifted correctly?
It looks like the issue is with the _to_epoch_time function in dbcore/query.py. I've pulled that function out into a minimal test script, and the results of running it under python3 and python2 are interesting:
import datetime
import time
def _to_epoch_time(date):
"""Convert a `datetime` object to an integer number of seconds since
the (local) Unix epoch.
"""
if hasattr(date, 'timestamp'):
# The `timestamp` method exists on Python 3.3+.
return int(date.timestamp())
else:
epoch = datetime.datetime.fromtimestamp(0)
delta = date - epoch
return int(delta.total_seconds())
dateFromCurrentTime = datetime.datetime.fromtimestamp(time.time());
print(dateFromCurrentTime)
print(_to_epoch_time(dateFromCurrentTime))
print(datetime.datetime.fromtimestamp(_to_epoch_time(dateFromCurrentTime)))
When run under python 2.7, the conversion is off by an hour:
pi@media-pi:~ $ python2.7 timetest.py
2021-04-03 16:39:10.550180
1617496750
2021-04-03 17:39:10
pi@media-pi:~ $ python3.7 timetest.py
2021-04-03 16:39:19.487139
1617493159
2021-04-03 16:39:19
I'll try that on another system to see if it's something weird that's just happening on this one.
Update: the same thing happens on a second system running stock Ubuntu 20.04.
Wow; thank you so much for digging into this!! You certainly nailed the problem. That's quite subtle!
The Python documentation has a suggestion for how to implement .timestamp(). Fixing this on 2.7 might be as simple as just using the expression listed here:
https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp
Because we are on the verge of dropping Python 2 support, this will also resolve itself, in an indirect kind of way…
