gpoore
Labeling chunks and show progress bar increment on processing each chunk
I would like to know on compilation what particular chunk is executing in a particular time moment. That would make things much more clear. Ideally, it would be nice to see a progress bar like n chunk of N have been executed and so on. How to make pythontex talk more?
That sounds like a useful feature. I am rewriting the core of PythonTeX, and will look into adding this as part of that process. The rewriting process is going slowly, so this isn't something I will be able to add soon.
I also suggest providing an opportunity to print messages from code chunk. Now pythontex grabs stderr content, wraps it and print as warning or error unlike knitr, where one can print to stderr e.g. progress bars showing the progress within the chunk. This is useful when running long time procedures such as Monte-Carlo simulation and so on.
Could you, please, make a quick patch to master that would at least unwrap output to stderr so users can print custom run-time messages (i.e. progress bars, etc.) indicating what is running?
I'd like to help, but I think this is more complex than a quick patch. Getting stderr in real time would require accessing stderr from a subprocess.Popen(), which is itself used inside of a multiprocessing.Pool(). Getting that to work seems like it could require a combination of pipes, queues, and possibly threads. That's more than I have time to dig into at the moment.
But what if consider disabling multiprocessing? Will it make things easier or stay the same?
As a temporary quick workaround I can suggest set max jobs number to 1 and, thus, disable multiprocessing, then I suggest subclassing file-like object (let's call it Logger) so that it would write both to file and to terminal.
Then one just should create an instance of Logger and pass it to
subprocess.Popen(..., stderr=logger)
like that.
The only misconception that I see is that we will get the content of stderr one more time on finish. Maybe I am missing something else. I can do this in my fork or you can do this in a separate branch or something else. What do you think?
See what you can do in your fork, and let me know how it goes. Not needing multiprocessing should make things easier.