Unicode on terminal

[quadrater]

There is a unicode problem in libgsync/output.py which crashes terminal output with unicode characters like /xe4.

DEBUG: UnicodeEncodeError('ascii', u'Ekonomi/L\xe5n R\xe4nta Amortering\n', 9, 10, 'ordinal not in range(128)'):  
...
  File "libgsync/output.py", line 25, in _print
    sys.stdout.write("%s\n" % msg)

Please change to something like:

    sys.stdout.write("%s\n" % msg.encode("utf-8"))

[iwonbigbro]

I have implemented a similar change, but one that will work with non-unicode objects. Can you give it a try?

[iwonbigbro]

Duplicate of #46

[quadrater]

The change is probably good in it self but does not address the underlying issue, which seems to be that sys.stderr.write does not know the encoding of the terminal and thus tries to print the string as ascii which fails due to the unicode characters.

DEBUG: UnicodeEncodeError('ascii', u'gsync-test/\xe5\xe4\xf6/\n', 11, 14, 'ordinal not in range(128)'): 'ascii' codec can't encode characters in position 11-13: ordinal not in range(128)
DEBUG: UnicodeEncodeError('ascii', u'gsync-test/\xe5\xe4\xf6/\n', 11, 14, 'ordinal not in range(128)'):   File "lib/python2.7/site-packages/gsync-0.1.14-py2.7.egg/libgsync/crawler.py", line 193, in run
    self._walk(srcpath, self._walk_callback, self._dev)
  File "lib/python2.7/site-packages/gsync-0.1.14-py2.7.egg/libgsync/crawler.py", line 152, in _walk
    self._sync(dirpath)
  File "lib/python2.7/site-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/__init__.py", line 263, in __call__
    self._sync(path)
  File "lib/python2.7/site-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/__init__.py", line 311, in _sync
    verbose(rel_path)
  File "lib/python2.7/site-packages/gsync-0.1.14-py2.7.egg/libgsync/output.py", line 39, in __call__
    self.write(msg, priority)
  File "lib/python2.7/site-packages/gsync-0.1.14-py2.7.egg/libgsync/output.py", line 45, in write
    sys.stdout.write(u"%s\n" % unicode(msg))

Error: UnicodeEncodeError('ascii', u'gsync-test/\xe5\xe4\xf6/\n', 11, 14, 'ordinal not in range(128)')

After spending a while looking at the code, I found the culprit. The sys.stdout object is overwritten to make it unbuffered, setting sys.stdout.encoding to None in the process which breaks the output. The solution is to explicitly call sys.stdout.flush() instead. Implemented in pull request.

[mthssdrbrg]

Is this supposed to be resolved, or is it still an open issue? I've been playing around with gsync today (and rather than opening a new issue I figured I'd use this one), and I've been running into the following:

DEBUG: UnicodeDecodeError('ascii', '/music/SINGLES/Roffe Ruff - B\xc3\xa4ttre P\xc3\xa5 Botten Blues.mp3', 29, 30, 'ordinal not in range(128)'): 'ascii' codec can't decode byte 0xc3 in position 29: ordinal not in range(128)
DEBUG: UnicodeDecodeError('ascii', '/music/SINGLES/Roffe Ruff - B\xc3\xa4ttre P\xc3\xa5 Botten Blues.mp3', 29, 30, 'ordinal not in range(128)'):   File "/usr/lib/python2.7/site-packages/libgsync/crawler.py", line 193, in run
    self._walk(srcpath, self._walk_callback, self._dev)
  File "/usr/lib/python2.7/site-packages/libgsync/crawler.py", line 163, in _walk
    self._sync(absfile)
  File "/usr/lib/python2.7/site-packages/libgsync/sync/__init__.py", line 263, in __call__
    self._sync(path)
  File "/usr/lib/python2.7/site-packages/libgsync/sync/__init__.py", line 311, in _sync
    verbose(rel_path)
  File "/usr/lib/python2.7/site-packages/libgsync/output.py", line 40, in __call__
    self.write(msg, priority)
  File "/usr/lib/python2.7/site-packages/libgsync/output.py", line 46, in write
    sys.stdout.write(u"%s\n" % unicode(msg))

Error: UnicodeDecodeError('ascii', '/music/SINGLES/Roffe Ruff - B\xc3\xa4ttre P\xc3\xa5 Botten Blues.mp3', 29, 30, 'ordinal not in range(128)')

Unfortunately my python-fu is essentially nonexistent, so I'm not sure what's going on, but I'd be happy to attempt to dig into this, just not sure what's actually failing...

[plijnzaad]

I can confirm that this is still an issue, and a bit of a show stopper for me to be honest. This is gsync version 0.1.14 on python version 2.7.6

Error: UnicodeEncodeError('ascii', u'misc_prive/archief/03 - La boh\xe8me.m4a\n', 43, 44, 'ordinal not in range(128)')
sent 0 bytes  received 0 bytes  0.00 bytes/sec

[rmkjr]

Same here, from cloned master

[tloudev]

perhaps http://stackoverflow.com/questions/21129020/how-to-fix-unicodedecodeerror-ascii-codec-cant-decode-byte can fix it? I'm trying to fix it this way, that means in /usr/lib/python2.7/site-packages/libgsync/output.py change unicode(msg) to unicode(msg,'utf-8') - now I'm just waiting to one really huge file to transfer :-( and after few hours I'll paste the result. BTW I tried to grep unicode() in the source and I really don't know, if the 'ascii' default behavior can't affect the result in other functions/files. I'm afraid of pythons ;-)

[tloudev]

BTW "duplicate of #46 probably means that none slves this issue, because #46 has a tag 'duplicate of #47'... and it's more than year old both.

[sgosnell]

It probably isn't in good taste to post this, but I solved all the problems by using rclone. It also uses rsync to sync google drive, as well as almost anything else you need to sync, including dropbox and local directories. I've abandoned gsync, and use rclone to sync my dropbox and google drive with each other, as well as with a local directory.

[ntim]

Fast fix is to omit the verbosity option "-v" since the error occurs when printing to the terminal.