pysimdjson icon indicating copy to clipboard operation
pysimdjson copied to clipboard

troubleshooting dangling proxy objects

Open memetb opened this issue 10 months ago • 1 comments

Hello, I was just troubleshooting this following strange situation:

async def reaper():
	reaper_simdjson_parser = simdjson.Parser()
	while True:
		await asyncio.sleep(1)

		text = await get_json_payload()
		doc = reaper_simdjson_parser.parse(text)
		*results, status = doc

		for x in results:
			# do something

		# del x # forgetting to explicitly delete x would cause intermittent problems
		del status
		del results
		del doc

the error is this:

    doc = reaper_simdjson_parser.parse(text) 
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "simdjson/csimdjson.pyx", line 435, in csimdjson.Parser.parse
RuntimeError: Tried to re-use a parser while simdjson.Object and/or simdjson.Array objects still exist referencing the old parser.

Perhaps specifying which reference is dangling may be helpful? At the very least putting it in the documentation as a potential gotcha might be helpful.

Thank you for pysimdjson and simdjson in general. Huge fan.

memetb avatar Mar 25 '25 14:03 memetb

Unfortunately we only keep a reference count, since tracking with more detail would be a significant performance and potentially significant memory hit.

Version 7.0.0 (#124) switches to using the new(ish) document object in simdjson, which means the lifecycle of the objects and the parser can be disconnected without issue.

TkTech avatar Mar 25 '25 15:03 TkTech