robert7
API Issues with initial sync (RateLimitExceeded)
Expected vs. actual behavior
After installation and having granted a fresh token I expect to be able able to download and sync my evernote database (around 17k notes) locally on nixnote2.
Unfortunately the sync get interrupted for an API error. I installed nixnote2 on this machine just yesterday, the token is from yesterdays, to my knowledge the API limits should be initially lifted, instead this is what I find in the log:
ERROR 2024-03-27 16:10:56.596 src/communication/communicationmanager.cpp:546 Exception stacktrace:
ERROR 2024-03-27 16:10:56.601 src/communication/communicationerror.cpp:88 "RateLimitExceeded[code=19]: API rate limit exceeded. Enable 'Sync' > 'Restart sync on API limit' or try again in 53 minutes ## >
ERROR 2024-03-27 16:10:56.601 src/threads/syncrunner.cpp:278 Error retrieving chunk
ERROR 2024-03-27 16:12:57.723 src/communication/communicationmanager.cpp:546 Exception stacktrace:
ERROR 2024-03-27 16:12:57.726 src/communication/communicationerror.cpp:88 "RateLimitExceeded[code=19]: API rate limit exceeded. Application will continue to sync in 48 minutes ## EDAMSystemException: EDAMErrorCode::RATE_LIMIT_REACHED rateLimitDuration= 2868 sec. # rateLimitDuration=\u0B34"
Restart sync on API limit (experimental) is flagged in my preferences.
Do you have any suggestion? Thank you so much!
Specifications
- Version: 202310040716~2.1.10~ubuntu22.04.1
This limit is from server side. I am afraid we have to wait until the time is out.
Alright, but I have more than 20k notes to download locally in order to finish the sync I had to restart the client at least 20 times in order to be able to download 1k of them (at the beginning I was not receiving the ratelimitexceed error above but the synchronization was getting stuck after about 10 minutes) It would be next to impossible to complete the first sync with these limits.
I have read that with a fresh token the API limits should be initially lifted, then why am I facing all this issues? Do you guys have any suggestion about that? Thank you very much again!
I have read that with a fresh token the API limits should be initially lifted, then why am I facing all this issues?
Do you mean this link? https://dev.evernote.com/doc/articles/rate_limits.php This page says that the rate limit of initial sync for a new token can be temporarily increased within 24 hours only, but it may be not large enough for your case, I think. This may be the reason.
If you still have Evernote installed, and your Evernote has most of your notes synced, you can transfer locally.
If your notes consist of text mainly, and not too many multimedia files, you could consider to sync with Evernote and then transfer locally. I remember that Evernote client has a limit for data amount every month for free user, I am not sure if it is enough for your notes.
I don't have so many notes, and I don't know whether the option of 'Restart sync on API limit (experimental)' works or not in reality. But if nixnote doesn't restart the sync automatically, what about restarting nixnote every one hour with an outside shell commands or something like that, with the option of auto 'sync on start up' enabled? This method may take a long time, but it is available.
I have read that with a fresh token the API limits should be initially lifted, then why am I facing all this issues?
Do you mean this link? https://dev.evernote.com/doc/articles/rate_limits.php This page says that the rate limit of initial sync for a new token can be temporarily increased within 24 hours only, but it may be not large enough for your case, I think. This may be the reason.
If you still have Evernote installed, and your Evernote has most of your notes synced, you can transfer locally.
If your notes consist of text mainly, and not too many multimedia files, you could consider to sync with Evernote and then transfer locally. I remember that Evernote client has a limit for data amount every month for free user, I am not sure if it is enough for your notes.
I don't have so many notes, and I don't know whether the option of 'Restart sync on API limit (experimental)' works or not in reality. But if nixnote doesn't restart the sync automatically, what about restarting nixnote every one hour with an outside shell commands or something like that, with the option of auto 'sync on start up' enabled? This method may take a long time, but it is available.
Thank you very much for your reply. I' m on linux and evernote for linux has been disabled months ago. Until a couple of days ago I was using evernote legacy through wine and a repackage windows version that a guy was releasing on github here , and evernote disabled the API endpoint for both of them a couple of days ago. That's how I landed here. I still have both of them installed and I also have some recent backups in enex format. I'm a subscribed user (not free user). Should I import the backup notes on a fresh installation of nixnote and then pull down the new notes through a server sync? Frankly I'm very afraid of doing a mess with the notes on the server, I keep very important information to me there.
The idea of a script killing and restarting nixnote every 20 minutes is also a good idea and I could try that. But, as I' m not using nixnote since many many years, would be all this syncing/importing work worth it? Is the client completely functional at this point? It would be a great way to be able to keep using evernote on linux (other than migrating to a different platform). Thank you again
Until a couple of days ago I was using evernote legacy through wine and a repackage windows version that a guy was releasing on github here
You were using evernote legacy through wine, so the latest evernote client can't work with wine, right?
I still have both of them installed and I also have some recent backups in enex format.
If there are only a few unsynced notes, how about using the web evernote and backing them up manually?
But, as I' m not using nixnote since many many years, would be all this syncing/importing work worth it?
You have backup files, so you don't need to sync all of them, backing up the unsynced notes is enough. Once you get all the necessary notes you want, you could not only import them to nixnote but also other apps you like.
Should I import the backup notes on a fresh installation of nixnote and then pull down the new notes through a server sync? Frankly I'm very afraid of doing a mess with the notes on the server, I keep very important information to me there. The idea of a script killing and restarting nixnote every 20 minutes is also a good idea and I could try that.
If you decide to use nixnote or other apps which are integrated with Evernote cloud, then this is a safe way, it just takes a long time.
Is the client completely functional at this point?
Nixnote is functional in most of the time. Some bugs may appear, but not often. The ui may be a little legacy. and don't offer some modern, fancy functions just as other newer note apps such as obsidian.
Thank you very much for your reply.
Until a couple of days ago I was using evernote legacy through wine and a repackage windows version that a guy was releasing on github here
You were using evernote legacy through wine, so the latest evernote client can't work with wine, right?
Yes, the new evernote version doesn't run under wine.
I still have both of them installed and I also have some recent backups in enex format.
If there are only a few unsynced notes, how about using the web evernote and backing them up manually?
The backup is from about two weeks ago, and in the meanwhile I might have probably edited many old notes other than adding new ones. Even if I'm sure that it's a viable option, I'm afraid of messing with imports/exports of data that is not updated.
But, as I' m not using nixnote since many many years, would be all this syncing/importing work worth it?
You have backup files, so you don't need to sync all of them, backing up the unsynced notes is enough. Once you get all the necessary notes you want, you could not only import them to nixnote but also other apps you like.
So my last backup is not very fresh, but it looks like that tools like this are still working. So maybe I could pull down a fresh enex backup with that tool and then import the notes in nixnotes (but, by the way, why this tool is able to download and export the notes and nixnotes can't do the same?)
Should I import the backup notes on a fresh installation of nixnote and then pull down the new notes through a server sync? Frankly I'm very afraid of doing a mess with the notes on the server, I keep very important information to me there. The idea of a script killing and restarting nixnote every 20 minutes is also a good idea and I could try that.
If you decide to use nixnote or other apps which are integrated with Evernote cloud, then this is a safe way, it just takes a long time.
Yes, I agree, I would much prefer to do a clean sync from the client itself. Scripting a recurrent shutdown/restart of nixnote like you were suggesting could be a good option.
Is the client completely functional at this point?
Nixnote is functional in most of the time. Some bugs may appear, but not often. The ui may be a little legacy. and don't offer some modern, fancy functions just as other newer note apps such as obsidian.
I'm not concerned about the look of the UI (and as I said, I was using the legacy version of evernote which was really looking oldshool, but it was quite snappy and functional). What I'm afraid of is inconsistencies between notes written in nixnote and in the new evernote official client. Recently I was using a mix a evernote legacy on one machine and evernote 10 on another machines and very often I was stepping into notes that were missing information in one client or in the other (it was happening especially with image attachments). I guess that overtime they changed things in their architecture and this is one of the reasons why they also dismissed the possibility to use the old clients.
So my last backup is not very fresh, but it looks like that tools like this are still working. So maybe I could pull down a fresh enex backup with that tool and then import the notes in nixnotes
If this tool works and has no rate limit, it will be easy to solve the problem.
(but, by the way, why this tool is able to download and export the notes and nixnotes can't do the same?)
I have not studied the code, I just read the README page, and I guess that it syncs the notes by calling the offical Evernote client APIs. The offical Evernote client may call another group of APIs, which are different from the third-party ones called by nixnote, and have no rate limit, I guess.
What I'm afraid of is inconsistencies between notes written in nixnote and in the new evernote official client. Recently I was using a mix a evernote legacy on one machine and evernote 10 on another machines and very often I was stepping into notes that were missing information in one client or in the other (it was happening especially with image attachments). I guess that overtime they changed things in their architecture and this is one of the reasons why they also dismissed the possibility to use the old clients.
Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.
If this tool works and has no rate limit, it will be easy to solve the problem.
I pulled a backup with that tool yesterday night and it correctly updated the backup I previously did last month (he even just downloaded the new and changed notes). Just great
I have not studied the code, I just read the README page, and I guess that it syncs the notes by calling the offical Evernote client APIs. The offical Evernote client may call another group of APIs, which are different from the third-party ones called by nixnote, and have no rate limit, I guess.
I guess that integrating their syncing method or the tricks they are using would benefit this project a lot. At this stage the barrier of entry for someone who is using evernote since a long time and has to import a lot of notes (hitting the API limit) is too high
Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.
That makes me think that issues and inconsistencies might appear if someone is using nixnote and at the same time evernote 10 on other machines or on mobile clients
Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.
Any source? I haven't heard of them changing anything. Considering that they still have lots of business API integrations, it seems unlikely such moves would be done, especially silently. Here's the specification for their ENEX format: http://xml.evernote.com/pub/evernote-export3.dtd.
What they could have actually changed is many other things which are present in note metadata but not in the note content (i.e. ENEX) itself - like all these reminder things and such.
Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.
Any source? I haven't heard of them changing anything. Considering that they still have lots of business API integrations, it seems unlikely such moves would be done, especially silently. Here's the specification for their ENEX format: http://xml.evernote.com/pub/evernote-export3.dtd.
What they could have actually changed is many other things which are present in note metadata but not in the note content (i.e. ENEX) itself - like all these reminder things and such.
I have no source but I have written what is my direct experience.
After many years of exclusively using the legacy version of evernote (under wine on linux), in the last 6 months I started using it alongside evernote 10 (on other machines), evernote android client, and evernote web. Over the course of these months I ran into a lot of problems with synchronisation (especially on the evernote legacy side), problems with attachments (especially images which could no more be displayed natively in the client but could be downloaded) and even situations where whole chunks of text were missing.
This didn't happen when I was using legacy evernote exclusively. My experience in these recent months has been that the two versions of evernote were less and less interoperable. If the enex protocol will certainly have remained the same other things may have probably changed.
I pulled a backup with that tool yesterday night and it correctly updated the backup I previously did last month (he even just downloaded the new and changed notes). Just great
It is nice.
I guess that integrating their syncing method or the tricks they are using would benefit this project a lot. At this stage the barrier of entry for someone who is using evernote since a long time and has to import a lot of notes (hitting the API limit) is too high
I see that the API rate limit was often talked about on github before, if there are not too many notes, just waiting and restarting the sync can solve it. In recent years, I feel that the users are not as many as before, so I think the improvement on syncing will not be needed by too many people.
That makes me think that issues and inconsistencies might appear if someone is using nixnote and at the same time evernote 10 on other machines or on mobile clients
I am afraid so.
@d1vanov
Any source? I haven't heard of them changing anything. Considering that they still have lots of business API integrations, it seems unlikely such moves would be done, especially silently. Here's the specification for their ENEX format: http://xml.evernote.com/pub/evernote-export3.dtd.
What they could have actually changed is many other things which are present in note metadata but not in the note content (i.e. ENEX) itself - like all these reminder things and such.
What you said is reasonable, Evernote should be very serious about the format. But there are some posts about the note format on the evernote forum. For example, this link: https://discussion.evernote.com/forums/topic/148763-old-notes-bulleted-and-numbered-lists-messed-up/. To this kind of issues, I just guessed it was because of the format changes, because except the note format changes, I cannot think out other reasons. To find the source Evernote released, I searched just now, and found this link: https://evernote.com/blog/legacy-decommissioning#:~:text=Legacy%20versions%20of%20Evernote%20use,and%20update%20at%20lightning%20speed. This link says: Expect a slower experience the first time you open a note in v10. Legacy versions of Evernote use a different note format than v10. As such, each note must be converted to the new format. This action takes place when the note is opened for the first time in v10 and can last a couple of seconds. Going forward, it’ll load and update at lightning speed.
Maybe the word I used and my understanding are not precise enough, but it seems that there is incompatibility between the legacy client and the latest one.
I see that the API rate limit was often talked about on github before, if there are not too many notes, just waiting and restarting the sync can solve it. In recent years, I feel that the users are not as many as before, so I think the improvement on syncing will not be needed by too many people.
the situation with the API syncing is making it very very difficult to switch to nixnote for any user that has more than 2k notes and above stored on evernote servers. That's a pity, and it is certainly a reason that keeps nixnote's user base low, even on a platform like linux where at the moment there is a desperate need for an evernote local client
To this kind of issues, I just guessed it was because of the format changes, because except the note format changes, I cannot think out other reasons
Well, I don't know for certain either but I believe what has actually changed is not the ENEX format spec itself but the note editor implementation between legacy Evernote client and v10 and in how each of them interprets the format. ENEX in its core is just a stripped down XHTML which in turn is just a little more strict and XML-like HTML. So it all kinda boils down to how a web browser implements a rich text editor. V10 might use different HTML tags for lists than the legacy client used. But I don't think a change in ENEX spec was necessary for that to happen.
To this kind of issues, I just guessed it was because of the format changes, because except the note format changes, I cannot think out other reasons
Well, I don't know for certain either but I believe what has actually changed is not the ENEX format spec itself but the note editor implementation between legacy Evernote client and v10 and in how each of them interprets the format. ENEX in its core is just a stipped down XHTML which in turn is just a little more strict and XML-like HTML. So it all kinda boils down to how a web browser implements a rich text editor. V10 might use different HTML tags for lists than the legacy client used. But I don't think a change in ENEX spec was necessary for that to happen.
Yes, it makes sense
the situation with the API syncing is making it very very difficult to switch to nixnote for any user that has more than 2k notes and above stored on evernote servers. That's a pity, and it is certainly a reason that keeps nixnote's user base low, even on a platform like linux where at the moment there is a desperate need for an evernote local client
Yes, it is one of the reasons. On many sides, nixnote is not close enough to perfect. On the other hand, it is not an easy thing to make it a perfect software through working on it just in spare time.
the situation with the API syncing is making it very very difficult to switch to nixnote for any user that has more than 2k notes and above stored on evernote servers. That's a pity, and it is certainly a reason that keeps nixnote's user base low, even on a platform like linux where at the moment there is a desperate need for an evernote local client
Yes, it is one of the reasons. On many sides, nixnote is not close enough to perfect. On the other hand, it is not an easy thing to make it a perfect software through working on it just in spare time.
Of course, I was making a general point because I think it's a shame that a ready-made, working project is not easily accessible to a potentially very large group of users who are in need of an evernote client on linux.
In all cases investing in a free project available to all is always a great act of generosity that should be appreciated
the situation with the API syncing is making it very very difficult to switch to nixnote for any user that has more than 2k notes and above stored on evernote servers. That's a pity, and it is certainly a reason that keeps nixnote's user base low, even on a platform like linux where at the moment there is a desperate need for an evernote local client
Yes, it is one of the reasons. On many sides, nixnote is not close enough to perfect. On the other hand, it is not an easy thing to make it a perfect software through working on it just in spare time.
Of course, I was making a general point because I think it's a shame that a ready-made, working project is not easily accessible to a potentially very large group of users who are in need of an evernote client on linux.
In all cases investing in a free project available to all is always a great act of generosity that should be appreciated
I studied evernote-backup code, and I think we could have an easy solution for this problem. I'll try it, but I can't promise.
Hi, wokawoka. I hope this reply is not too late. I made a PR for this issue and it may probably solve it, since I compared the logic of evernote-backup and nixnote, and removed tedious part of nixnote. For the details you can visit https://github.com/robert7/nixnote2/pull/223. It works fine for me, but I am not sure whether the rate limit exception will appear when there are thousands of notes with resource files. Could you please help to test it?
If you'd like to, I can offer you a binary under Windows here: https://github.com/boo-yee/nixnote2/releases/download/v2.1.10/nixnote.zip.
Update: And also for Linux there is appimage now. https://github.com/robert7/nixnote2/releases/download/continuous-develop/NixNote2-x86_64.AppImage
this is very good news I am going to test the appimage later tonight when at home Thank you very much again!
I feel very glad to receive your reply and appreciate your help. I think, if syncing all the notes takes too long time, syncing only part of your notes is enough, as long as there are no exception reported, for a longer period of time than previously. Syncing 2-3 thousand notes is enough I think. I created one thousand notes and synced them without exception. The only thing I am not sure is if there is any side effect after my changes.
And a reminder for you for the sake of data safety: please do not let the new nixnote appimage mess up your old local database.
yesterday night I tried to pull a full sync through the appimage you posted above but unfortunately I failed one again. The first time the sync stopped at 800 notes. The second one stopped at 1790. And the third one at 2794. Do you have any suggestion? Is there something that I could try in order to make it work? Thank you
Do you have any suggestion? Is there something that I could try in order to make it work?
It should be the code, not your mkstake. I think I have to ask you several questions. You said it stopped, did it report the rate limit exceeded exception? Or no exception reported, just the note list not updated? And do you know how long time around does it take to sync the first 800 notes? (I think the network speed may also determine the api calling frequency.) Do you think the new appimage peformed better than previous version?
I never stepped into a "rate limit exceeded" error, the syncing each time just stopped pulling the notes (even if the syncing icon was still showing as spinning). Regarding the speed, I can't really tell, I was on wifi and the connection was quite stable, I just left the program downloading in the background and I was monitoring from time to time the note count through the "tools -> database status" menu option. If you need, just let me know how I could monitor the syncing rate in a more specific way and I'll do more test. Thank you very much again
If the rate limit exceeded exception does not appear, then the issue about the api should be fixed. As I was not sure if my changes would bring any side effects, I just focused on the rate limit issue, and left the other things alone, hoping to fix all the issues one by one.
the syncing each time just stopped pulling the notes (even if the syncing icon was still showing as spinning)
For this issue, I think I have to continue to inspect the code to find the reason. (This issue is probably nothing about the rate limit exception. I think it is something about optimization. And the thread number is also another reason that nixnote is not as fast as evernote-backup, which uses multi-thread. So nixnote can be still improved.)
Regarding the speed, I can't really tell, I was on wifi and the connection was quite stable, I just left the program downloading in the background and I was monitoring from time to time the note count through the "tools -> database status" menu option. If you need, just let me know how I could monitor the syncing rate in a more specific way and I'll do more test.
As the rate limit exception is fixed, we do not need to focus on the api and the network, I think. When I asked that question about the network speed, I thought the rate limit exception appeared again. However it did not in fact.
The resources such as the images in the newly synced notes is correctly synced together, right?
The resources such as the images in the newly synced notes is correctly synced together, right?
I actually didn´t inspect any note because I didn't want to interfere in any way with the syncing. Since we are here, do you think that it's safe for me doing these tests with you on my (very much important) notes? Should I pull a backup before going on with these tests or the syncing is (for now) just a download of the notes through the API from the server to client? Thank you
Since we are here, do you think that it's safe for me doing these tests with you on my (very much important) notes? Should I pull a backup before going on with these tests or the syncing is (for now) just a download of the notes through the API from the server to client?
Yes, it is just a download. Only syncing the notes from a fresh nixnote client will not affect the notes on the cloud server.
As you synced the notes with evernote-backup earlier, just importing and using them is OK. Nixnote syncing is still slower, full sync will take too much time.
For the very important notes, I think it's best to backup them well and carefully, as Nixnote is not reliable sometimes, one of my note's content once went out of order when I was using it. It is about charater encoding. It happend only once, so I didn't put it at the first priority. I also saw similar issues from another user: https://github.com/baumgarr/nixnote2/issues/446#issuecomment-1730526809 (I guess that it was because of the thread mutex not set correctly and not write back to the disk when exiting abnormally. And I'll work on this later.)
'Restart sync on API limit (experimental)' works. The only issue I have and is likely an Evernote API bug is that when the limit is reached the evernote API returns the exception and nixnote waits for the returned timeout. But once this timeout is reached, the sync restart only for around 5 minutes then an API limit exceed is triggered anew and waits for an hour. The next attempt works fine (it syncs for half an hour or more before the API limit triggers), then the next is 5 minutes ... and so forth. So it takes twice the time of the theoretical API limit. But I don't think this is a nixnote/qevernote issue. Either way, I cannot devote time to document this issue properly in a bug report for now. But I compile notes about it.
