forge notifications errors when pulling notifications

[luigy]

from magit status "F Y"

...
Pulling notifications...
Pulling notifications (page 1/6)...
Pulling notifications (page 2/6)...
Pulling notifications (page 3/6)...
error in process filter: ghub--signal-error: peculiar error: ((type . "NOT_FOUND") (path "_Z2l0aHViLmNvbTowMTA6UmVwb3NpdG9yeTEyODU2Mzk6MjUyMzgyODIx" "issue") (locations ((line . 68) (column . 1))) (message . "Could not resolve to an Issue with the number of 151."))
error in process filter: peculiar error: ((type . "NOT_FOUND") (path "_Z2l0aHViLmNvbTowMTA6UmVwb3NpdG9yeTEyODU2Mzk6MjUyMzgyODIx" "issue") (locations ((line . 68) (column . 1))) (message . "Could not resolve to an Issue with the number of 151."))

[luigy]

After clearing some notifications from https://github.com/notifications this is now running without failures

[vermiculus]

It sounds like we don't handle the 'lots of unread notifications' well, though. It might be worthwhile leaving this open – even if just remains as 'this is a known issue'.

[tarsius]

"Could not resolve to an Issue with the number of 151."

Could be that that issue was deleted just as you were fetching.

Use this code to determine the repository and then check whether that issue still exists:

(forge-sql
 [:select [owner name]
  :from repository
  :where (= id $s1)]
 (base64-encode-string
  (mapconcat
   #'identity
   (butlast
    (split-string
     (base64-decode-string
      (substring
       ;; this was taken from the error message
       "_Z2l0aHViLmNvbTowMTA6UmVwb3NpdG9yeTEyODU2Mzk6MjUyMzgyODIx"
       1))
     ":"))
   ":")
  t))

[luigy]

@tarsius good call that issue looks to be deleted. I ran that query and got (("dataarts" "dat.gui")) which doesn't exist at https://github.com/dataarts/dat.gui/issues/151

[tarsius]

This won't happen more than once in a blue moon and so won't attempt to come up with a work-around. Notification fetching is already very complicated because of limitations of the Github API in this area and handling this issue properly would make the respective code completely unreadable.

[jwiegley]

@tarsius This is now happening to me every time, in every one of my repositories. I only started noticing it recently.

[tarsius]

Please post the error message. Use the above code-snippet to determine which issue from which repository this is about. Is there anything special about that issue?

[jwiegley]

The issue is indeed not there: https://github.com/jwiegley/use-package/issues/782, but I don't see in my notifications on GitHub either.

[tarsius]

In the short-run the only workaround is to not fetch notifications (by setting forge-pull-notifications to nil) until Github forgets about the offending notification.

[tarsius]

I am unable to reproduce this. For me the list of notifications does not appear to include any entries about topics that no longer exist. (I created two issue using another user account and deleted one of them again. That issue did not show up in my notifications while the other one did. After I deleted the the second issue also it disappeared from my notifications.)

For you this appears to be different and if you forward me the respective data I might be able to come up with heuristics to distinguish invalid notifications from valid ones.

Fetching notifications essentially works like this:

1. Use the REST API to fetch the list of notifications. We use the REST API because the GraphQL API does not support this. The returned data does not contain all the information that we need.
2. Use the REST response to construct a GraphQL request.

If the GraphQL did support notifications then we would tell it "for each notifications return all issue/pull-request data". Because that is not supported we have to tell it "here is a list of notifications, for each one of them return the issue/pull-request data".

If that list of notifications contains a notification that some parts of Github software consider to be valid (the parts that are used to generate the REST response) but other parts of the Github software consider to be invalid (the parts that are used when responding to GraphQL requests), then the later parts of the software fail.

We therefore have to filter out the invalid data.

1. Repeat what you have done at my request before; try to fetch the notifications as usual. And then determine which topic causes that to fail. You will probably get the same topic as before, but repeat this just in case.

2. Make sure default-directory is inside some repository that is fully configured for Forge.

3. Run this REST request:

   (forge--ghub-get nil "/notifications" '((all . "true")) :unpaginate t)
   

4. Extract the part that is about the topic that you have identified in step 1. You can also post the complete data here or (if you think it might contain sensitive data) mail it to me.

[tarsius]

Can you provide more information about the issue "that is not there"? Do you know that the issue by that number existed at one point? Did you delete it or did someone else do that?

[jwiegley]

I'm unaware of that issue, actually, so I don't know if it was ever there before.

I'm traveling for work now, but will check this out once I'm back home next week. In the meantime, I've disabled updating of notifications, and everything is back to working smoothly. I love Forge btw! It is by far the best way to create PRs, which I make very many of...

[tarsius]

I'm unaware of that issue, actually, so I don't know if it was ever there before.

Well there is a 783 so there must have been a 782 at some point.

I love Forge btw! It is by far the best way to create PRs, which I make very many of...

That's nice to hear!

[Erk-]

This is happening to me now, when I run the code you posted above, (forge--ghub-get nil "/notifications" '((all . "true")) :unpaginate t), I get an error. The notifications seem like for the others here to point to deleted issues.

[tarsius]

I've re-implemented notification support. There are big limitations due to the abysmal API, see https://github.com/magit/forge/issues/367#issuecomment-1848723122.

Notifications that reference topics that no longer exist are now ignored, so I am closing this.