brumar
Working update
Summary of changes:
- Note validation (via external script)
- Find deleted notes
- Updated see also processing
- Confirmed some old references to deleted functions were removed
- README updated
Hey still getting import errors i could fix the most of them but there are a few NoneType and so on
On 23.09.2015 08:11, Avi Puchalapalli wrote:
Summary of changes: 1. Note validation (via external script) 2. Find deleted notes 3. Updated see also processing 4. Confirmed some old references to deleted functions were removed 5. README updated
You can view, comment on, or merge this pull request online at:
https://github.com/brumar/anknotes/pull/16
Commit Summary
- Added DTD Validation to makeNote/updateNote functions. These are for uploading content to Evernote's servers * 1. Move lxml code out of addon and into separate command-line script since unable to bundle lxml * Minor changes to logging * Added Command Line Tools such as Find Deleted Notes * Finalized Note validation and Note maintenance ('Find deleted notes') * Class-based processing of See Also notes. This is to detect changes that need to be uploaded to Evernote * Confirmed no references to generateTocTitle, all are to EvernoteNoteTitle.generateTOCTitle * Squashed some bugs * Long-overdue update for README, moved README to root, and added my /extra/dev folder to the repo
File Changes
- M .gitignore https://github.com/brumar/anknotes/pull/16/files#diff-0 (4) * A README.md https://github.com/brumar/anknotes/pull/16/files#diff-1 (219) * M anknotes/Anki.py https://github.com/brumar/anknotes/pull/16/files#diff-2 (112) * M anknotes/AnkiNotePrototype.py https://github.com/brumar/anknotes/pull/16/files#diff-3 (296) * M anknotes/Controller.py https://github.com/brumar/anknotes/pull/16/files#diff-4 (333) * M anknotes/EvernoteImporter.py https://github.com/brumar/anknotes/pull/16/files#diff-5 (186) * M anknotes/EvernoteNoteFetcher.py https://github.com/brumar/anknotes/pull/16/files#diff-6 (75) * M anknotes/EvernoteNotePrototype.py https://github.com/brumar/anknotes/pull/16/files#diff-7 (30) * M anknotes/EvernoteNoteTitle.py https://github.com/brumar/anknotes/pull/16/files#diff-8 (100) * M anknotes/EvernoteNotes.py https://github.com/brumar/anknotes/pull/16/files#diff-9 (99) * D anknotes/README.md https://github.com/brumar/anknotes/pull/16/files#diff-10 (23) * M anknotes/main.py https://github.com/brumar/anknotes/pull/16/files#diff-11 (66) * A anknotes/_re.py https://github.com/brumar/anknotes/pull/16/files#diff-12 (277) * M anknotes/ankEvernote.py https://github.com/brumar/anknotes/pull/16/files#diff-13 (235) * M anknotes/bare.py https://github.com/brumar/anknotes/pull/16/files#diff-14 (347) * M anknotes/constants.py https://github.com/brumar/anknotes/pull/16/files#diff-15 (50) * M anknotes/db.py https://github.com/brumar/anknotes/pull/16/files#diff-16 (15) * A anknotes/enums.py https://github.com/brumar/anknotes/pull/16/files#diff-17 (108) * M anknotes/error.py https://github.com/brumar/anknotes/pull/16/files#diff-18 (4) * A anknotes/extra/ancillary/QMessageBox.css https://github.com/brumar/anknotes/pull/16/files#diff-19 (15) * R anknotes/extra/ancillary/enml2.dtd https://github.com/brumar/anknotes/pull/16/files#diff-20 (0) * M anknotes/extra/ancillary/regex-see_also.txt https://github.com/brumar/anknotes/pull/16/files#diff-21 (11) * M anknotes/extra/ancillary/regex.txt https://github.com/brumar/anknotes/pull/16/files#diff-22 (10) * A anknotes/extra/dev/Restart Anki.lnk https://github.com/brumar/anknotes/pull/16/files#diff-23 (0) * A anknotes/extra/dev/anknotes.developer https://github.com/brumar/anknotes/pull/16/files#diff-24 (0) * A anknotes/extra/dev/anknotes.developer.automate https://github.com/brumar/anknotes/pull/16/files#diff-25 (0) * A anknotes/extra/dev/anknotes_standAlone_template.py https://github.com/brumar/anknotes/pull/16/files#diff-26 (112) * A anknotes/extra/dev/anknotes_test.py https://github.com/brumar/anknotes/pull/16/files#diff-27 (209) * A anknotes/extra/dev/restart_anki.bat https://github.com/brumar/anknotes/pull/16/files#diff-28 (2) * A anknotes/extra/graphics/Tomato-icon.ico https://github.com/brumar/anknotes/pull/16/files#diff-29 (0) * A anknotes/extra/graphics/Tomato-icon.png https://github.com/brumar/anknotes/pull/16/files#diff-30 (0) * M anknotes/find_deleted_notes.py https://github.com/brumar/anknotes/pull/16/files#diff-31 (162) * M anknotes/graphics.py https://github.com/brumar/anknotes/pull/16/files#diff-32 (3) * M anknotes/html.py https://github.com/brumar/anknotes/pull/16/files#diff-33 (78) * M anknotes/logging.py https://github.com/brumar/anknotes/pull/16/files#diff-34 (459) * M anknotes/menu.py https://github.com/brumar/anknotes/pull/16/files#diff-35 (209) * M anknotes/settings.py https://github.com/brumar/anknotes/pull/16/files#diff-36 (12) * M anknotes/shared.py https://github.com/brumar/anknotes/pull/16/files#diff-37 (60) * A anknotes/stopwatch/init.py https://github.com/brumar/anknotes/pull/16/files#diff-38 (278) * A anknotes/stopwatch/tests/init.py https://github.com/brumar/anknotes/pull/16/files#diff-39 (71) * M anknotes/structs.py https://github.com/brumar/anknotes/pull/16/files#diff-40 (645) * M anknotes/toc.py https://github.com/brumar/anknotes/pull/16/files#diff-41 (76) * A anknotes_start_bare.py https://github.com/brumar/anknotes/pull/16/files#diff-42 (9) * A anknotes_start_find_deleted_notes.py https://github.com/brumar/anknotes/pull/16/files#diff-43 (11) * A anknotes_start_note_validation.py https://github.com/brumar/anknotes/pull/16/files#diff-44 (120)
Patch Links:
- https://github.com/brumar/anknotes/pull/16.patch * https://github.com/brumar/anknotes/pull/16.diff
— Reply to this email directly or view it on GitHub https://github.com/brumar/anknotes/pull/16.
Okay, thanks for looking into this. I'm running a code inspection with PyCharm now, and hopefully that will catch everything
@Scriptkiddi Unfortunately PyCharm couldn't find anything that would cause an import error. Could you screenshot the errors and paste them here?
An error occurred in an add-on. Please post on the add-on forum: https://anki.tenderapp.com/discussions/add-ons
Traceback (most recent call last): File "/usr/share/anki/aqt/main.py", line 165, in onOpenProfile self.loadProfile() File "/usr/share/anki/aqt/main.py", line 244, in loadProfile runHook("profileLoaded") File "/usr/share/anki/anki/hooks.py", line 26, in runHook func(*args) File "/home/fritz/Anki/addons/anknotes/main.py", line 83, in anknotes_profile_loaded menu.upload_validated_notes(True) File "/home/fritz/Anki/addons/anknotes/menu.py", line 140, in upload_validated_notes controller = Controller() File "/home/fritz/Anki/addons/anknotes/Controller.py", line 50, in init self.evernote = Evernote() File "/home/fritz/Anki/addons/anknotes/ankEvernote.py", line 87, in init self.client = EvernoteClient(token=auth_token, sandbox=ANKNOTES.EVERNOTE_IS_SANDBOXED) NameError: global name 'EvernoteClient' is not defined
i think i fixed this by moving the evernote client import out of your if statment
Could you try this: http://pastebin.com/8111hStH
I added an assert statement at line 19. I also changed the conditional to test if Anki module is importable instead of lxml. It's possible that lxml is importable in your Anki config
Unfortunately having it outside of the conditional will break the script I'm using to run note validation. I couldn't get lxml imported into my Anki config, so I made a script that runs externally via command line as opposed to within the Anki environment as an addon
I fixed the issue with templates that was probably causing a string formatting error when you first loaded Anknotes. After your first import, Anknotes will now detect the uid and shard, and after next launch, will rebuild the templates. This is required for the template to be able to generate Evernote Note links from guids
so the import errors are gone good job now I tried to continue the test but when i open the settings i get An error occurred. It may have been caused by a harmless bug, or your deck may have a problem. To confirm it's not a problem with your deck, please run Tools > Check Database. If that doesn't fix the problem, please copy the following into a bug report: Traceback (most recent call last): File "/usr/share/anki/aqt/preferences.py", line 41, in reject self.accept() File "/usr/share/anki/aqt/preferences.py", line 37, in accept self.mw.reset() File "/usr/share/anki/aqt/main.py", line 433, in reset self.col.reset() File "/usr/share/anki/anki/collection.py", line 228, in reset self.sched.reset() File "/usr/share/anki/anki/sched.py", line 53, in reset self._resetLrn() File "/usr/share/anki/anki/sched.py", line 459, in _resetLrn self._resetLrnCount() File "/usr/share/anki/anki/sched.py", line 451, in _resetLrnCount self.dayCutoff) or 0 File "/usr/share/anki/anki/db.py", line 72, in scalar res = self.execute(_a, *_kw).fetchone() TypeError: instance of cursor required for first argument
are you using the database cursor anywhere?
Hmm... I think I made a function that wraps around the cursor for easy access ... ankDB() in db.py. Does it still occur if you remove the addon from Anki? I don't think I do anything to the cursor. There are a few places where I have to call an external script so I close the collection entirely... But I don't think that's happening here.
I'm not sure honestly.... This happens when you open the preferences? can you add a log statement to setup_evernote() method in line 69 of settings.py to debug the state of the cursor?
log_dump({'DB CON': mw.col.db._db, 'DB CUR': mw.col.db_db.cursor(), 'CUR DESC': mw.col.db._db.cursor().description})
This should dump data to extra\logs\dump.log
Also try commenting out lines of anknotes_onload() method at line 102 of main.py and seeing if that resolves.
so if i take out anknotes it disappears also if i comment the lines in anknotes_onload() ok so i traced it back to upload_validated_notes
[2015-09-23 19:57:25]: **** Dumping Object
{ 'CUR DESC': None,
'DB CON': <anki.db.DB object at 0x7fd03016df10>,
'DB CUR': <pysqlite2.dbapi2.Cursor object at 0x7fd02275d6c0>}
oh and also could you please use explicit importing otherwise it is almost impossible to trace through your code
By explicit importing do you mean not using the * in my import statements? If so sure I can do that.
It looks like the cursor is valid... I don't think the cursor description has to be set. The only modification happens at the end of that method - two executions and then a commit. I don't see how that would invalidate the cursor, however.
Can you execute a query with the DB connection in settings.py?
Also maybe try this change so the connection doesn't try to commit unless there is a query executed: https://gist.github.com/holycrepe/27bd14592e908eb6a776
If you're unable to resolve the error by the end of the day, I'll set up a new Anki profile and see if that has something to do with the error
so the move of the commit statment did not fix it so i look a little bit deeper and found on line 130 in db.py (all) is calling self.execute with out a sql parameter i guess thats the missing cursor anki is complaning about
Hmm good catch... But where is all() being called without a sql parameter? And why would the call to all() in upload_validated_notes result in a missing cursor?
The call is: dbRows = ankDB().all("SELECT * FROM %s WHERE validation_status = 1 " % TABLES.MAKE_NOTE_QUEUE)
Still a bit confused
If you force a sql parameter for all() with the following code, do you get an Argument Error instead of the cursor error?
def all(self, sql, *a, **kw):
return self.execute(sql, *a, **kw).fetchall()
Also the error you referenced is using the scalar method. Unless you think the call to .all() somehow messed up the cursor for the subsequent call to scalar by Anki. To save you some time, I changed all the methods to require a sql parameter:
https://gist.github.com/holycrepe/6d0456aaa4d4e57f6b41
Unfortunately I couldn't reproduce the error, even with a new profile. Must have something to do with your particular setup
Another step to try - Override the anki functions that are giving you an error so you can trace what's going on.
So if the scalar function is what's failing, you can try inserting the following code at the beginning of anknotes_onload: (I would expect that the self object is None, since the first argument to execute should be self. I don't think the failure is the sql argument, which would be second)
def my_scalar(self, *a, **kw):
print [self, type(self)]
print [a, kw]
# more debug code here
mw.col.db.scalar = wrap(mw.col.db.scalar, my_scalar, "before")
Also add the following to the wrap function so you can double check the path:
print self._db.path
There is a bug I just fixed where I hard coded the path for local db instances, but I don't think that's happening. To confirm, add an assert False or raise to line 29 of db.py under if dbLocal:
I just went through every single line of code that affects the db, and I couldn't find any errors. Can you add assertions or raises to the following lines of menu.py, to ensure the profile is not being loaded or unloaded:
207 above mw.unloadCollection(),
245 above mw.loadCollection()
@Scriptkiddi any luck with the errors? Can you try disabling the automated upload on start (set ANKNOTES.AUTOMATE_VALIDATION to False in constants.py or comment out the code in __main__.py?
Then, try an import or other database activity, then, try loading the preferences to see if any database activity is causing the error, or only the database activity associated with upload_validated_notes()?
If you continue having issues I will set up Anki on a friends computer, and download a fresh copy of anknotes from my branch, and see if the issue is reproducible that way.
On a side note I completed, but did not push, updates to the 'See Also' links... They are fully functional now. Anknotes successfully uploaded notes with modified see also links to my Evernote account. So I think pretty much updates on my TODO list are done. It's a shame that we can't get it working on your setup, though.
@brumar, can you test and see if you are getting @Scriptkiddi's error?
hey i just spend the weekend in munich so i will get to it maybe tomorrow
Thanks guys. I setup a new account and debugged everything from scratch so the core functionality should be good to go. There are some issues with automation (e.g. using the "Complete All Steps" function in Process See Also Footer Links menu. I think I need to put more checks to ensure the clients are properly loaded and db is not locked since we have to close out the Anki profile and run an external script.
So, feel free to ignore that functionality when doing your testing.
Also - dev functions: there's some info in the readme, but in addition, I added the ability to automatically reload the main anknotes.Controller module every time you click a menu item, which also forces reload of the anknotes.Evernote and anknotes.Anki classes. This is purely for testing, when making multiple changes to these classes, so you don't have to keep quitting and restarting Anki to test your changes. (Side note: I included a batch file that silently reloads Anki in the /extra/dev/ folder). Finally, for testing purposes, you can add code to main.py that automatically executes every time the Anki profile is loaded, which is useful if you're updating something other than the Controller, Evernote, or Anki classes (since those are the only ones that automatically reload when you click a menu item).
For these dev functions, you'll have to create empty files anknotes.developer and anknotes.developer.automate in the /extra/dev/ folder.
so first of please follow https://www.python.org/dev/peps/pep-0008/ makes the code a lot more readable and please do not use one line if statments they are really hard to read when going over the code real quick
also my database error is still there An error occurred. It may have been caused by a harmless bug, or your deck may have a problem. To confirm it's not a problem with your deck, please run Tools > Check Database. If that doesn't fix the problem, please copy the following into a bug report: Traceback (most recent call last): File "/usr/share/anki/aqt/preferences.py", line 41, in reject self.accept() File "/usr/share/anki/aqt/preferences.py", line 37, in accept self.mw.reset() File "/usr/share/anki/aqt/main.py", line 433, in reset self.col.reset() File "/usr/share/anki/anki/collection.py", line 228, in reset self.sched.reset() File "/usr/share/anki/anki/sched.py", line 53, in reset self._resetLrn() File "/usr/share/anki/anki/sched.py", line 459, in _resetLrn self._resetLrnCount() File "/usr/share/anki/anki/sched.py", line 451, in _resetLrnCount self.dayCutoff) or 0 File "/usr/share/anki/anki/db.py", line 72, in scalar res = self.execute(_a, *_kw).fetchone() TypeError: instance of cursor required for first argument
Sorry for the late response, I didn't see a notification from github. I'll make the formatting changes you requested, and avoid the one line statements. I'll run a code formatting tool to fix this
Were you able to further debug the database error? E.g. my suggestion to override Anki's db method:
def my_scalar(self, *a, **kw):
print [self, type(self)]
print [a, kw]
# more debug code here
mw.col.db.scalar = wrap(mw.col.db.scalar, my_scalar, "before")
Does disabling script validation help?