Question/Feature request: force refetching all new

Open yarikoptic opened this issue 3 years ago • 0 comments

General informations

system/distribution (with version):

$> apt-cache policy offlineimap
offlineimap:
  Installed: 7.3.3+dfsg1-1+0.0~git20211018.e64c254+dfsg-1
  Candidate: 7.3.3+dfsg1-1+0.0~git20211018.e64c254+dfsg-1
  Version table:
 *** 7.3.3+dfsg1-1+0.0~git20211018.e64c254+dfsg-1 900
        900 http://deb.debian.org/debian bookworm/main amd64 Packages
        900 http://deb.debian.org/debian bookworm/main i386 Packages
        600 http://http.debian.net/debian sid/main amd64 Packages
        100 /var/lib/dpkg/status

offlineimap version (offlineimap -V):

$> offlineimap -V
offlineimap v7.3.0, imaplib2 v3.05, Python v3.10.5, OpenSSL 3.0.3 3 May 2022

Python version: 3.10.5
server name or domain: N/A
CLI options:

$> cat ~/bin/offlineimap                  
#!/bin/bash

# Just runs offlineimap with logging into a file (which is not
# configurable in config file for some reason according to docs)

odir=$HOME/.var/offlineimap/logs
logfile=$odir/$(date +"20%y%m%d:%H%M%s")
mkdir -p $odir
/usr/bin/offlineimap -l $logfile "$@"

apparently I have collected over 29k of logs! need to clean up ;) but also can now provide details

Configuration file offlineimaprc

$> grep -v '^#' ~/.offlineimaprc
[general]
ui = TTY.TTYUI
metadata = ~/.var/offlineimap
accounts = 1rRecv
maxsyncaccounts = 8
maxconnections = 1
fsync = false

[Account 1rRecv]
localrepository = Local
remoterepository = 1r
postsynchook = notmuch new

[Repository Local]
type = Maildir
localfolders = ~/Maildir
sep = .

[Repository 1r]
type = IMAP
preauthtunnel = setsid env -u DISPLAY ssh sensored.example.com \
    bin/offlineimaptunnel
sep = .
folderfilter = re.compile(r'^INBOX\.(Inbox(|\.(Admin|MC|test|signed-keys.*|ccards|Shopping|Employment|Air))|Drafts|Sent|Lists\.Debian\.(devel|private|security|vote))$').search
nametrans = lambda foldername: re.sub('^INBOX', '', foldername)

pythonfile (if any)

REMOVE PRIVATE DATA.

Logs, error

REMOVE PRIVATE DATA.

Steps to reproduce the error

My BTRFS partition ran out of space today
invocation of offlineimap started to crash with

sqlite3 traceback etc

2022-08-12 09:14:44 INFO: Copy message UID 146736 (4/5) 1r:INBOX.Lists.Debian.devel -> Local:.Lists.Debian.devel
2022-08-12 09:14:46 ERROR: ERROR: Copying message 1151350 [acc: 1rRecv]
  ('disk I/O error while inserting UID 1151350', <traceback object at 0x7f6211a9b840>)
2022-08-12 09:14:46 WARNING: Thread 'Copy message from 1r:INBOX.Inbox' terminated with exception:
Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/folder/LocalStatusSQLite.py", line 349, in savemessage
    self.__sql_write('INSERT INTO status (id,flags,mtime,labels) VALUES (?,?,?,?)',
  File "/usr/share/offlineimap3/offlineimap/folder/LocalStatusSQLite.py", line 181, in __sql_write
    self.connection.commit()
sqlite3.OperationalError: disk I/O error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/threadutil.py", line 146, in run
    Thread.run(self)
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 815, in copymessageto
    statusfolder.savemessage(new_uid, message, flags, rtime)
  File "/usr/share/offlineimap3/offlineimap/folder/LocalStatusSQLite.py", line 352, in savemessage
    raise UserWarning("%s while inserting UID %s" %
UserWarning: ('disk I/O error while inserting UID 1151350', <traceback object at 0x7f6211a9b840>)


Last 3 debug messages logged for Copy message from 1r:INBOX.Inbox prior to exception:
thread: Register new thread 'Copy message from 1r:INBOX.Inbox' (account '1rRecv')
maildir: Write mail '.Inbox:1151350' with flags set()
maildir: savemessage: returning uid 1151350
2022-08-12 09:14:46 INFO: Copy message UID 1151352 (31/111) 1r:INBOX.Inbox -> Local:.Inbox
2022-08-12 09:14:46 WARNING: ERROR: Exceptions occurred during the run!
2022-08-12 09:14:46 WARNING: ERROR: Copying message 1151350 [acc: 1rRecv]
  ('disk I/O error while inserting UID 1151350', <traceback object at 0x7f6211a9b840>)
2022-08-12 09:14:46 WARNING: 
Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 815, in copymessageto
    statusfolder.savemessage(new_uid, message, flags, rtime)
  File "/usr/share/offlineimap3/offlineimap/folder/LocalStatusSQLite.py", line 352, in savemessage
    raise UserWarning("%s while inserting UID %s" %

but I was "blind" -- I simply have not mentioned it, reran again (may be have gained some free space since then), and then mentioned that some emails lack any body and for some offlineimap invocation, or seems more exactly - notmuch indexer, says smth like

2022-08-12 09:34:35 INFO: Calling hook: notmuch new
2022-08-12 09:35:11 INFO: Hook stdout: Processed 121 total files in 7s (15 files/sec.).
Added 86 new messages to the database. Removed 22 messages. Detected 12 file renames.

Hook stderr:Note: Ignoring non-mail file: /home/yoh/Maildir/.Inbox/cur/1534341673_1.10143.hopa,U=713318,FMD5=719bb12f07d1f7b3327f0c273dc767b8:2,S
Note: Ignoring non-mail file: /home/yoh/Maildir/.Inbox/cur/1567631085_2.30567.hopa,U=816695,FMD5=719bb12f07d1f7b3327f0c273dc767b8:2,
Note: Ignoring non-mail file: /home/yoh/Maildir/.Inbox/cur/1568896257_5.20836.hopa,U=821340,FMD5=719bb12f07d1f7b3327f0c273dc767b8:2,
Note: Ignoring non-mail file: /home/yoh/Maildir/.Inbox/cur/1601259280_1.543106.lena,U=939918,FMD5=719bb12f07d1f7b3327f0c273dc767b8:2,S
Note: Ignoring non-mail file: /home/yoh/Maildir/.Inbox/cur/1660310079_5.599928.lena,U=1151322,FMD5=719bb12f07d1f7b3327f0c273dc767b8:2,S

the question really is -- would it be possible to force refetch of all /new mail somehow?? I think that should address the issue at large (get me the mails body etc)

edit 1: or some other way to refetch some mail, e.g. by providing Message-Ids or smth like that... I would need to figure out though which emails are "bad" locally first :-/

Aug 12 '22 14:08 yarikoptic