Too many files make SQLite choke.

[michaelcadilhac]

I have a few thousand ebooks on an old Kindle Touch, that are basically split into three collections. LibrarianSync fails with the POST request returning:

{ "Error": "Failed to execute handler: /usr/lib/ccat/change.lua:1371: /usr/lib/ccat/change.lua:1277: {"http_status_code":400,"sqlite3_msg":"too many SQL variables","sqlite3_code":4}" }

Also, instead of pretty failing, it blurts out:

  File "./generate_collections.py", line 308, in <module>
    source="librarian")
  File "./generate_collections.py", line 233, in update_cc_db
    cc.execute()
  File "/mnt/us/extensions/librariansync/cc_update.py", line 107, in execute
    if r.json()[u"ok"]:
  File "/mnt/us/python/lib/python2.7/site-packages/requests/models.py", line 818, in json
    self.content.decode(encoding), **kwargs
  File "/mnt/us/python/lib/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/mnt/us/python/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/mnt/us/python/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting ',' delimiter or '}': line 1 column 104 (char 103)

and ends with a "Something went very wrong".

Cheers!

[thaelina]

I realize this is old, but this repo is old and I am new to this issue. Anyone find a fix to this or am I just limited to fewer books because of this?

[NiLuJe]

I'm afraid that it's a limitation of the (undocumented) backend LibrarianSync relies on...

The best we could do is handle the error gracefully (probably by checking the HTTP status code before trying to parse the result as JSON).

EDIT: Or maybe we'd need to chunk the requests instead of sending one giant POST? Meh.

[thaelina]

That's actually not a bad idea, just sending the commands in chunks. I can wade my way through python and see if I can manage something.