"ValueError: end is out of bounds" and "ValueError: Unterminated string..." when run in subprocess.call, but NOT subprocess.Popen

[Michael-B-G]

Hi,

So, I have a large Python program I'm writing that calls both gen_outline.py and json2csv.py as initial steps. Following this, I call Rufus Pollock's "csv2sqlite.py" (https://github.com/rgrp/csv2sqlite) to get the data in SQLITE format. In order to invoke these Python module from within my larger program I use the "subprocess" Python library. Up until now I was using Popen, but soon found I was running into race conditions (https://github.com/rgrp/csv2sqlite/issues/20), After doing some research, I decided to either stick to subprocess.call(), or at least invoke Popen.wait(). While these solutions appear to solve the race condition, they unfortunately present another error, albeit a non-fatal one.

Basically, my Python program calls gen_outline.py and json2csv.py each four times, each one corresponding to a separate JSON file. The resulting stack trace is as follows, which again only manifests when I invoke Popen.wait() (or use the call() command instead), and did not occur before when I simply spawned a new process (but risked a race condition). I am not sure what the problem is, but I would very much appreciate help. Thanks in advance!

Traceback (most recent call last): File "gen_outline.py", line 86, in main() File "gen_outline.py", line 75, in main outline = make_outline(args.json_file, args.each_line, args.collection) File "gen_outline.py", line 50, in make_outline key_map = gather_key_map(iterator) File "gen_outline.py", line 31, in gather_key_map for d in iterator: File "gen_outline.py", line 25, in coll_iter data = json.load(f) File "C:\Python278\lib\json__init__.py", line 290, in load *kw) File "C:\Python278\lib\json__init_.py", line 338, in loads return default_decoder.decode(s) File "C:\Python278\lib\json\decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=w(s, 0).end()) File "C:\Python278\lib\json\decoder.py", line 382, in raw_decode obj, end = self.scan_once(s, idx) ValueError: end is out of bounds Traceback (most recent call last): File "gen_outline.py", line 86, in main() File "gen_outline.py", line 75, in main outline = make_outline(args.json_file, args.each_line, args.collection) File "gen_outline.py", line 50, in make_outline key_map = gather_key_map(iterator) File "gen_outline.py", line 31, in gather_key_map for d in iterator: File "gen_outline.py", line 25, in coll_iter data = json.load(f) File "C:\Python278\lib\json__init.py", line 290, in load *kw) File "C:\Python278\lib\json__init_.py", line 338, in loads return default_decoder.decode(s) File "C:\Python278\lib\json\decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=w(s, 0).end()) File "C:\Python278\lib\json\decoder.py", line 382, in raw_decode obj, end = self.scan_once(s, idx) ValueError: Unterminated string starting at: line 1 column 200693 (char 200692) Traceback (most recent call last): File "gen_outline.py", line 86, in main() File "gen_outline.py", line 75, in main outline = make_outline(args.json_file, args.each_line, args.collection) File "gen_outline.py", line 50, in make_outline key_map = gather_key_map(iterator) File "gen_outline.py", line 31, in gather_key_map for d in iterator: File "gen_outline.py", line 25, in coll_iter data = json.load(f) File "C:\Python278\lib\json__init.py", line 290, in load *kw) File "C:\Python278\lib\json__init_.py", line 338, in loads return default_decoder.decode(s) File "C:\Python278\lib\json\decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=w(s, 0).end()) File "C:\Python278\lib\json\decoder.py", line 382, in raw_decode obj, end = self.scan_once(s, idx) ValueError: Unterminated string starting at: line 1 column 57339 (char 57338) Traceback (most recent call last): File "gen_outline.py", line 86, in main() File "gen_outline.py", line 75, in main outline = make_outline(args.json_file, args.each_line, args.collection) File "gen_outline.py", line 50, in make_outline key_map = gather_key_map(iterator) File "gen_outline.py", line 31, in gather_key_map for d in iterator: File "gen_outline.py", line 25, in coll_iter data = json.load(f) File "C:\Python278\lib\json__init.py", line 290, in load *kw) File "C:\Python278\lib\json__init_.py", line 338, in loads return _default_decoder.decode(s) File "C:\Python278\lib\json\decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Python278\lib\json\decoder.py", line 382, in raw_decode obj, end = self.scan_once(s, idx) ValueError: end is out of bounds

[Michael-B-G]

I think I may have solved this. Not all instances of invoking "popen" within my code were configured properly in terms of their stdout and stderr parameters. I have corrected this, and it appears to work well, though I will update this if anything goes wrong again.

Thanks!