offlineimap icon indicating copy to clipboard operation
offlineimap copied to clipboard
verified publisher icon OfflineIMAP

KeyError in `folder/Gmail.py`

Open nicolasbock opened this issue 6 years ago • 17 comments

General informations

  • system/distribution (with version): Gentoo
  • offlineimap version (offlineimap -V): offlineimap v7.2.2, imaplib2 v2.57 (bundled), Python v2.7.15, OpenSSL 1.0.2r 26 Feb 2019
  • server name or domain: imap.gmail.com
  • CLI options: -o

Running offlineimap I am getting the following:

 ERROR: ERROR in syncfolder for nicolasbock.gmail folder [Gmail]/Trash: Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/offlineimap/accounts.py", line 652, in syncfolder
    localfolder.syncmessagesto(remotefolder, statusfolder)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/Base.py", line 1116, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/Base.py", line 944, in __syncmessagesto_copy
    self.copymessageto(uid, dstfolder, statusfolder, register=0)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/GmailMaildir.py", line 227, in copymessageto
    labels = dstfolder.getmessagelabels(uid)
  File "/usr/lib64/python2.7/site-packages/offlineimap/folder/Gmail.py", line 105, in getmessagelabels
    if 'labels' in self.messagelist[uid]:
KeyError: 690815

Is this a known issue? Any tips for debugging it further?

Thanks!

nicolasbock avatar Apr 25 '19 20:04 nicolasbock

I think this should not happen. It looks like there's a race condition. Is this reproducible? If so, with the same key number?

nicolas33 avatar Apr 25 '19 22:04 nicolas33

The UUID changes every time I run the command. And I tried this now with -1 and master and it's still happening. I'll see whether I can get a more precise location in the code.

nicolasbock avatar Apr 26 '19 11:04 nicolasbock

folder/GmailMaildir.py didn't change for a long time. I think it would be wise to check v7.2.0 (and v7.1.1 if v7.2.0 fails the same way). If this appear to be a regression, we could git bissect this issue.

nicolas33 avatar Apr 26 '19 11:04 nicolas33

This problem is present in v7.0.0 and in v6.x the process seems to hang at syncing the All Mail folder.

nicolasbock avatar Apr 26 '19 11:04 nicolasbock

@nicolas33 I think the problem is here. It seems that the new UUID is not communicated and subsequent code assumes that the old UUID is still there. I might be wrong of course :wink:

nicolasbock avatar Apr 26 '19 12:04 nicolasbock

Maybe it is worth adding print() debug statements to track how the uid lists change during the sync.

nicolas33 avatar Apr 26 '19 12:04 nicolas33

Yes, that's what I did. And right before the call to change_message_uid the old uid is still in messagelist but right after it's not anymore.

nicolasbock avatar Apr 26 '19 12:04 nicolasbock

I suspect that the problem is here

nicolasbock avatar Apr 26 '19 12:04 nicolasbock

This is a line of code where we delete the uid after the mail is stored in the maildir. This statement is legal.

I'd say all the current lines of code where we delete uids from the message list are legal. The issue might be more subtle like calling any upper method while it should not in this context or not correctly locking some chunk of code, etc.

Could you share your config file? Could you share more detailed logs (with -d imap,maildir CLI option)?

nicolas33 avatar Apr 26 '19 13:04 nicolas33

I don't quite understand how this is supposed to work. In https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L218 copymessageto is called which removes the uid. Then in https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L227 getmessagelages is called with the (now deleted) uid. Why would that work?

nicolasbock avatar Apr 26 '19 13:04 nicolasbock 

[Repository nicolasbock.gmail-remote]
type = Gmail
createfolders = false
folderfilter = lambda folder: folder in ['INBOX',
                                         '[Gmail]/All Mail',
                                         '[Gmail]/Chats',
                                         '[Gmail]/Drafts',
                                         '[Gmail]/Sent Mail',
                                         '[Gmail]/Spam',
                                         '[Gmail]/Starred',
                                         '[Gmail]/Trash']
auth_mechanisms = LOGIN
maxconnections = 5

nicolasbock avatar Apr 26 '19 13:04 nicolasbock 

 Copy message UID 690892 (1/296) nicolasbock.gmail-local:[Gmail].Trash -> nicolasbock.gmail-remote:[Gmail]/Trash
 nbock 3: starting copymessageto
 nbock 1: 690892 in messagelist: True
 nbock 5: 690892 in messagelist: True
 nbock 6: 690892 in messagelist: True
 []: getmessageheaderlist: called to get X-Keywords
 []: getmessageheaderlist: eoh = 4598
 []: getmessageheaderlist: headers = 'Delivered-To: [email protected]\nReceived: by 2002:a9d:21:0:0:0:0:0 with SMTP id 30csp5503157ota;\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\nX-Google-Smtp-Source: APXvYqxX3NZJHrFPIj+0rcb8AkvWZ0ZkJL6+w+B8gJcqTKEGC0ZJTh3gjs0PQFFNuYkswiCvza4j\nX-Received: by 2002:a37:59c5:: with SMTP id n188mr64646686qkb.208.1555519348036;\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\nARC-Seal: i=1; a=rsa-sha256; t=1555519348; cv=none;\n        d=google.com; s=arc-20160816;\n        b=VvUdCI2cMCNbRW4UzrrFxTDVxdHwcQQaRaZsVTqU+mWjUqMj6fUpGEj7rvFG8JyCJS\n         sPjkxXPau4K4HzFeNh8tOnJ+4AL3qy7EKfEL3fp9R2ktMjxDqs4osl8y8Nj6aUwKaz8f\n         z4DJoOJWb9VzBkK1Mci8KUm4d/DKK6z8BlzutCarPS6+lH6w10JZ+WJiVW1JF32goRs5\n         0JlY1ITXbE2h2Cr+E5PaeiA90x2Ty1/nA2WcR8Kl8T0AifZ7g+1D0gzwqw4aTKFbNrTZ\n         pbI+A63SS2sM6TS/3JPRjHLkB7XmiyH6Uj7CJK1/yjUGykfncttZELcA6TVV0QqbVGJu\n         5DNQ==\nARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;\n        h=list-unsubscribe:list-post:list-archive:list-id:precedence\n         :content-transfer-encoding:mime-version:subject:references\n         :in-reply-to:message-id:cc:to:reply-to:from:dkim-signature:date;\n        bh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\n        b=jaiVVDM3cO5pC3HNUf2Y3XWQuhurPVf20I6ehaN1srsWLEOg43dZkbnbvMJ0riXWQM\n         2YHEc1BAJsGrEnHbRSjjlQNbKIOZ8/POyYmxIJdSgDE3ta3QuXQmf3qA3SK1T+AfbRQa\n         2uJ4c0GrhjjP2LVtTsvCJq7oUWufwnbIUEP8MNnqPqhzX24Phe/xhO+t9A3nwo7AXDbi\n         AOLVOpTt/IocY+QtTkcn+CJVCn8JIqiyV41rE/BlpnR04qRBQNwPYxwSwP0Zrk7HMEM/\n         UOBfZfzJeCAJL716MxbSOjKjMsEBNixiIvzQXG18U5J0QdDPHg1jgvGxch4LN7miGPlb\n         gsRw==\nARC-Authentication-Results: i=1; mx.google.com;\n       dkim=pass (test mode) [email protected] header.s=pf2014 header.b=ZA2IhJG2;\n       spf=pass (google.com: domain of [email protected] designates 192.30.252.198 as permitted sender) [email protected];\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\nReturn-Path: <[email protected]>\nReceived: from out-7.smtp.github.com (out-7.smtp.github.com. [192.30.252.198])\n        by mx.google.com with ESMTPS id 50si5749913qtv.225.2019.04.17.09.42.27\n        for <[email protected]>\n        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\nReceived-SPF: pass (google.com: domain of [email protected] designates 192.30.252.198 as permitted sender) client-ip=192.30.252.198;\nAuthentication-Results: mx.google.com;\n       dkim=pass (test mode) [email protected] header.s=pf2014 header.b=ZA2IhJG2;\n       spf=pass (google.com: domain of [email protected] designates 192.30.252.198 as permitted sender) [email protected];\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\nDate: Wed, 17 Apr 2019 09:42:27 -0700\nDKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=github.com;\n\ts=pf2014; t=1555519347;\n\tbh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\n\th=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:List-ID:\n\t List-Archive:List-Post:List-Unsubscribe:From;\n\tb=ZA2IhJG2p7mafWxB7+Q31vXHkG8lrtSCr2bklhD9/Ir5gmflqZgFp8PKCfz4jc9Rd\n\t BdAFxWA0QSzMYucse3OihFjrXanxcdo92njY95dCBE6K7xI5J2fhTuBDmHJlgbF5q2\n\t BEeBoEgnYM05w4G7tWzCR53NU67CAOTRzVOudfw0=\nFrom: =?UTF-8?B?VGlsbCBTY2jDpGZlcg==?= <[email protected]>\nReply-To: gentoo/gentoo <reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\nTo: gentoo/gentoo <[email protected]>\nCc: Subscribed <[email protected]>\nMessage-ID: <gentoo/gentoo/pull/11463/[email protected]>\nIn-Reply-To: <gentoo/gentoo/pull/[email protected]>\nReferences: <gentoo/gentoo/pull/[email protected]>\nSubject: Re: [gentoo/gentoo] Davical 1.1.8 version bump (#11463)\nMime-Version: 1.0\nContent-Type: multipart/alternative;\n boundary="--==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf";\n charset=UTF-8\nContent-Transfer-Encoding: 7bit\nPrecedence: list\nX-GitHub-Sender: tillschaefer\nX-GitHub-Recipient: nicolasbock\nX-GitHub-Reason: subscribed\nList-ID: gentoo/gentoo <gentoo.gentoo.github.com>\nList-Archive: https://github.com/gentoo/gentoo\nList-Post: <mailto:reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\nList-Unsubscribe: <mailto:unsub+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>,\n <https://github.com/notifications/unsubscribe/AAzWqWI1bEAMqePWre2mTLiRMBwRTkSjks5vh07zgaJpZM4cEGa_>\nX-Auto-Response-Suppress: All\nX-GitHub-Recipient-Address: [email protected]\nX-Keywords: gentoo/github'
 [imap]: Write mail '[Gmail]/Trash:690892' with flags set([])
 []: deletemessageheaders: called to delete []
 []: getmessageheader: called to get message-id
 []: getmessageheader: eoh = 7157
 []: getmessageheader: headers = 'Delivered-To: [email protected]\r\nReceived: by 2002:a9d:21:0:0:0:0:0 with SMTP id 30csp5503157ota;\r\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\r\nX-Google-Smtp-Source: APXvYqxX3NZJHrFPIj+0rcb8AkvWZ0ZkJL6+w+B8gJcqTKEGC0ZJTh3gjs0PQFFNuYkswiCvza4j\r\nX-Received: by 2002:a37:59c5:: with SMTP id n188mr64646686qkb.208.1555519348036;\r\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\r\nARC-Seal: i=1; a=rsa-sha256; t=1555519348; cv=none;\r\n        d=google.com; s=arc-20160816;\r\n        b=VvUdCI2cMCNbRW4UzrrFxTDVxdHwcQQaRaZsVTqU+mWjUqMj6fUpGEj7rvFG8JyCJS\r\n         sPjkxXPau4K4HzFeNh8tOnJ+4AL3qy7EKfEL3fp9R2ktMjxDqs4osl8y8Nj6aUwKaz8f\r\n         z4DJoOJWb9VzBkK1Mci8KUm4d/DKK6z8BlzutCarPS6+lH6w10JZ+WJiVW1JF32goRs5\r\n         0JlY1ITXbE2h2Cr+E5PaeiA90x2Ty1/nA2WcR8Kl8T0AifZ7g+1D0gzwqw4aTKFbNrTZ\r\n         pbI+A63SS2sM6TS/3JPRjHLkB7XmiyH6Uj7CJK1/yjUGykfncttZELcA6TVV0QqbVGJu\r\n         5DNQ==\r\nARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;\r\n        h=list-unsubscribe:list-post:list-archive:list-id:precedence\r\n         :content-transfer-encoding:mime-version:subject:references\r\n         :in-reply-to:message-id:cc:to:reply-to:from:dkim-signature:date;\r\n        bh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\r\n        b=jaiVVDM3cO5pC3HNUf2Y3XWQuhurPVf20I6ehaN1srsWLEOg43dZkbnbvMJ0riXWQM\r\n         2YHEc1BAJsGrEnHbRSjjlQNbKIOZ8/POyYmxIJdSgDE3ta3QuXQmf3qA3SK1T+AfbRQa\r\n         2uJ4c0GrhjjP2LVtTsvCJq7oUWufwnbIUEP8MNnqPqhzX24Phe/xhO+t9A3nwo7AXDbi\r\n         AOLVOpTt/IocY+QtTkcn+CJVCn8JIqiyV41rE/BlpnR04qRBQNwPYxwSwP0Zrk7HMEM/\r\n         UOBfZfzJeCAJL716MxbSOjKjMsEBNixiIvzQXG18U5J0QdDPHg1jgvGxch4LN7miGPlb\r\n         gsRw==\r\nARC-Authentication-Results: i=1; mx.google.com;\r\n       dkim=pass (test mode) [email protected] header.s=pf2014 header.b=ZA2IhJG2;\r\n       spf=pass (google.com: domain of [email protected] designates 192.30.252.198 as permitted sender) [email protected];\r\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\r\nReturn-Path: <[email protected]>\r\nReceived: from out-7.smtp.github.com (out-7.smtp.github.com. [192.30.252.198])\r\n        by mx.google.com with ESMTPS id 50si5749913qtv.225.2019.04.17.09.42.27\r\n        for <[email protected]>\r\n        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\r\n        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)\r\nReceived-SPF: pass (google.com: domain of [email protected] designates 192.30.252.198 as permitted sender) client-ip=192.30.252.198;\r\nAuthentication-Results: mx.google.com;\r\n       dkim=pass (test mode) [email protected] header.s=pf2014 header.b=ZA2IhJG2;\r\n       spf=pass (google.com: domain of [email protected] designates 192.30.252.198 as permitted sender) [email protected];\r\n       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=github.com\r\nDate: Wed, 17 Apr 2019 09:42:27 -0700\r\nDKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=github.com;\r\n\ts=pf2014; t=1555519347;\r\n\tbh=nc9CK8jr8JqtXdLPxeBkBOKkjFaJmXfGwfnpxlP9+gE=;\r\n\th=Date:From:Reply-To:To:Cc:In-Reply-To:References:Subject:List-ID:\r\n\t List-Archive:List-Post:List-Unsubscribe:From;\r\n\tb=ZA2IhJG2p7mafWxB7+Q31vXHkG8lrtSCr2bklhD9/Ir5gmflqZgFp8PKCfz4jc9Rd\r\n\t BdAFxWA0QSzMYucse3OihFjrXanxcdo92njY95dCBE6K7xI5J2fhTuBDmHJlgbF5q2\r\n\t BEeBoEgnYM05w4G7tWzCR53NU67CAOTRzVOudfw0=\r\nFrom: =?UTF-8?B?VGlsbCBTY2jDpGZlcg==?= <[email protected]>\r\nReply-To: gentoo/gentoo <reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\r\nTo: gentoo/gentoo <[email protected]>\r\nCc: Subscribed <[email protected]>\r\nMessage-ID: <gentoo/gentoo/pull/11463/[email protected]>\r\nIn-Reply-To: <gentoo/gentoo/pull/[email protected]>\r\nReferences: <gentoo/gentoo/pull/[email protected]>\r\nSubject: Re: [gentoo/gentoo] Davical 1.1.8 version bump (#11463)\r\nMime-Version: 1.0\r\nContent-Type: multipart/alternative;\r\n boundary="--==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf";\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit\r\nPrecedence: list\r\nX-GitHub-Sender: tillschaefer\r\nX-GitHub-Recipient: nicolasbock\r\nX-GitHub-Reason: subscribed\r\nList-ID: gentoo/gentoo <gentoo.gentoo.github.com>\r\nList-Archive: https://github.com/gentoo/gentoo\r\nList-Post: <mailto:reply+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>\r\nList-Unsubscribe: <mailto:unsub+000cd6a9c12b39e1a7644b332064b13547f6beb1a1bc3f6e92cebac489f392a169ce194abc8b@reply.github.com>,\r\n <https://github.com/notifications/unsubscribe/AAzWqWI1bEAMqePWre2mTLiRMBwRTkSjks5vh07zgaJpZM4cEGa_>\r\nX-Auto-Response-Suppress: All\r\nX-GitHub-Recipient-Address: [email protected]\r\nX-Keywords: gentoo/github\r\n\r\n\r\n----==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf\r\nContent-Type: text/plain;\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit\r\n\r\nAny progress here? Anything else, that blocks the request?\r\n\r\n-- \r\nYou are receiving this because you are subscribed to this thread.\r\nReply to this email directly or view it on GitHub:\r\nhttps://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036\r\n----==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf\r\nContent-Type: text/html;\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit\r\n\r\n<p>Any progress here? Anything else, that blocks the request?</p>\r\n\r\n<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">&mdash;<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AAzWqc02QcRbaAIXSE3xmcUi_hs0hS9tks5vh07zgaJpZM4cEGa_">mute the thread</a>.<img src="https://github.com/notifications/beacon/AAzWqVGhYniRbECnUnJkzadTEK4FmdyKks5vh07zgaJpZM4cEGa_.gif" height="1" width="1" alt="" /></p>\r\n<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/gentoo/gentoo","title":"gentoo/gentoo","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/gentoo/gentoo"}},"updates":{"snippets":[{"icon":"PERSON","message":"@tillschaefer in #11463: Any progress here? Anything else, that blocks the request?"}],"action":{"name":"View Pull Request","url":"https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036"}}}</script>\r\n<script type="application/ld+json">[\r\n{\r\n"@context": "http://schema.org",\r\n"@type": "EmailMessage",\r\n"potentialAction": {\r\n"@type": "ViewAction",\r\n"target": "https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036",\r\n"url": "https://github.com/gentoo/gentoo/pull/11463#issuecomment-484169036",\r\n"name": "View Pull Request"\r\n},\r\n"description": "View this Pull Request on GitHub",\r\n"publisher": {\r\n"@type": "Organization",\r\n"name": "GitHub",\r\n"url": "https://github.com"\r\n}\r\n}\r\n]</script>\r\n----==_mimepart_5cb7577399b16_19493fecaf6cd9601666cf--\r\n'
 [imap]: savemessage: date: "22-Apr-2019 16:09:18 -0600", content: 'Delivered-To: [email protected]
Received: by 2002:a9d:21:0:0:0:0:0 with SMTP id 30csp5503157ota;
        Wed, 17 Apr 2019 09:42:28 -0700 (PDT)
..._mimepart_5cb7577399b16_19493fecaf6cd9601666cf--
'
 [imap]: savemessage: returning new UID 706419
 [imap]: __options2hash returning: {'X-GM-LABELS': '(gentoo/github)', 'UID': '706419'}
 nbock 7: 690892 in messagelist: True
 nbock 8: new_uid 706419
 nbock 9: 690892 in messagelist: False
 nbock 10: 690892 in messagelist: False
 nbock 2: 690892 in messagelist: False
 ERROR: while syncing [Gmail].Trash [account nicolasbock.gmail]
  690892
 ['  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 1127, in syncmessagesto\n    action(dstfolder, statusfolder)\n', '  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 954, in __syncmessagesto_copy\n    uid, dstfolder, statusfolder, register=0)\n', '  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/GmailMaildir.py", line 233, in copymessageto\n    labels = dstfolder.getmessagelabels(uid)\n', '  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Gmail.py", line 105, in getmessagelabels\n    if \'labels\' in self.messagelist[uid]:\n']
 ERROR: ERROR in syncfolder for nicolasbock.gmail folder [Gmail]/Trash: Traceback (most recent call last):
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/accounts.py", line 652, in syncfolder
    localfolder.syncmessagesto(remotefolder, statusfolder)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 1127, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Base.py", line 954, in __syncmessagesto_copy
    uid, dstfolder, statusfolder, register=0)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/GmailMaildir.py", line 233, in copymessageto
    labels = dstfolder.getmessagelabels(uid)
  File "/home/nbock/venv-offlineimap/lib/python2.7/site-packages/offlineimap/folder/Gmail.py", line 105, in getmessagelabels
    if 'labels' in self.messagelist[uid]:
KeyError: 690892

  690892

nicolasbock avatar Apr 26 '19 14:04 nicolasbock

I don't quite understand how this is supposed to work. In https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L218 copymessageto is called which removes the uid. Then in https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap/folder/GmailMaildir.py#L227 getmessagelages is called with the (now deleted) uid. Why would that work?

GmailMaildir#L212 doesn't remove the uid from the messagelist. It changes the local uid once the email is uploaded and assigned a new uid by the server.

Email added in the local maildir does not have uid, yet. They are assigned uid = -1 and are uploaded to the server. In the code, the variable realcopy should be False. I don't get why offlineimap wants to upload emails with already assigned uid in the first place.

I wonder you're moving email locally to the trash folder without renaming the filename of the email. I this something possible in your workflow?

NOTE: mutt, for example, renames the files like this when moving/copying emails to another mail folder: 1556345324.31940_5.<hostname>:2,S.

nicolas33 avatar Apr 27 '19 06:04 nicolas33

Thanks for the logs. This makes things much more clear.

nicolas33 avatar Apr 27 '19 06:04 nicolas33

https://github.com/OfflineIMAP/offlineimap/issues/613#issuecomment-487259011 updated.

nicolas33 avatar Apr 27 '19 06:04 nicolas33

@nicolas33 Thanks for the reply! I use afew to move emails around and notmuch to read them. I haven't had any issues with this setup so far.

Do I understand you correctly that the problem is that the local mail has a UID (based on its filename) but there is no copy of it on the remote? Sorry my understanding of how IMAP works is pretty basic.

nicolasbock avatar Apr 29 '19 17:04 nicolasbock

Do I understand you correctly that the problem is that the local mail has a UID (based on its filename) but there is no copy of it on the remote? Sorry my understanding of how IMAP works is pretty basic.

Yes, I don't get why offlineimap tries to upload new emails with UID > 0. New emails in the local maildir are assigned the UID = -1.

However, I was wrong about the move over maildirs as the root cause. We have the folder MD5 to protect against this.

To dig into this issue, you might like to debug why these messages (= emails) are assigned a positive UID (likely by GmailMaildir or a parent object). In the offlineimap workflow, positive UIDs are stored in the filename and the status cache once they (the UIDs) are fetched from the IMAP server (at message download time or after a sucessful upload).

nicolas33 avatar Apr 29 '19 17:04 nicolas33