pmix
IO Forwarding for Tools/Debuggers
Brief Description
Discusses support for all tools by adding the ability for the RM to forward output from applications to a tool, and to forward stdin from the tool to one or more application processes.
Use Case Details
Support for tools and debuggers can be extended by adding the ability for the RM to forward output from applications to a tool, and to forward stdin from the tool to one or more application processes.
- Forwarding Stdout/Stderr Once the tool has connected to the target server, it can request that output from a specified set of processes in a given executing application be forwarded to it. In addition, requests to forward output from processes being spawned by the tool can be included in calls to the PMIx_Spawn API. When using PMIx_Spawn API two modes are envisioned:
- PMIX_IOF_COPY // deliver a copy of the output to the tool
- PMIX_IOF_REDIRECT //intercept the output stream and deliver to the requesting tool instead of its current final destination
If PMIx is used to forward stdout/stderr for an already running application PMIx passes a request to the host environment via PMIx_IOF_pull asking that it collect the stdout/stderr from the specified process and forward it to the requestor's specified callback function. The success of this operation depends on the host environment. Typically, the host environment has fork/exec'd the target procs and therefore is already collecting the stdout/stderr from them (e.g., for forwarding to the user).
Interfaces
PMIx_IOF_pull // Register to receive output forwarded from a remote process
PMIx_IOF_deregister // Deregister from output forwarded from a remote process
PMIx_server_IOF_deliver // Provide a function by which the host RM can pass forwarded IO to the local PMIx server
PMIx_IOF_push // Push data collected locally (typically from stdin) to target recipients.
Macros
PMIX_FWD_NO_CHANNELS 0x0000
PMIX_FWD_STDIN_CHANNEL 0x0001
PMIX_FWD_STDOUT_CHANNEL 0x0002
PMIX_FWD_STDERR_CHANNEL 0x0004
PMIX_FWD_STDDIAG_CHANNEL 0x0008
PMIX_FWD_ALL_CHANNELS 0x00ff
Attributes/Directives
PMIX_IOF_TAG – output is prefixed with the nspace,rank of the source and a string identifying the channel (stdout, stderr, etc.)
PMIX_IOF_TIMESTAMP – output is marked with the time at which the data was received by the tool (note that this will differ from the time at which it was actually output by the source)
PMIX_IOF_XML_OUTPUT – output is to be formatted in XML
PMIX_IOF_COPY_STREAM - to get a copy of output streams
PMIX_IOF_INTERCEPT_STREAM - to intercept streams
- Forwarding Stdin The tool is not necessarily a child of the RM as it may have been started directly from the command line. Thus, provision must be made for the tool to collect its stdin and pass it to the host RM (via the PMIx server) for forwarding. Two methods of support for forwarding of stdin are defined:
- inclusion of PMIX_IOF_PUSH_STDIN in a call to PMIx_Spawn indicating that your stdin is to be forwarded to the specified proc started by the spawn
- external collection directly by the PMIx tool code. The tool can directly control the targets for the data on a per-call basis through PMIx_IOF_push to begin pushing your stdin to the specified targets
Attributes/Directives
/*from RFC*/
PMIX_IOF_CACHE_SIZE
"Pmix.iof.csize"
(uint32_t) requested size of the server cache in bytes for each specified channel. By default, the server is allowed (but not required) to drop all bytes received beyond the max size
PMIX_IOF_DROP_OLDEST
"Pmix.iof.old"
(bool) in an overflow situation, drop the oldest bytes to make room in the cache
PMIX_IOF_DROP_NEWEST
"Pmix.iof.new"
(bool) in an overflow situation, drop any new bytes received until room becomes available in the cache (default)
PMIX_IOF_BUFFERING_SIZE
"Pmix.iof.bsize"
(uint32_t) basically controls grouping of IO on the specified channel(s) to avoid being called every time a bit of IO arrives. The library will execute the callback whenever the specified number of bytes becomes available. Any remaining buffered data will be "flushed" upon call to deregister the respective channel
PMIX_IOF_BUFFERING_TIME
"Pmix.iof.btime"
(uint32_t) max time in seconds to buffer IO before delivering it. Used in conjunction with buffering size, this prevents IO from being held indefinitely while waiting for another payload to arrive
PMIX_IOF_COMPLETE
"Pmix.iof.cmp"
(bool) indicates whether or not the specified IO channel has been closed by the source
PMIX_IOF_PUSH_STDIN
"Pmix.iof.stdin"
(pmixproc_t*) Used by a tool to request that the PMIx library collect the tool's stdin and forward it to the indicated proc - the PMIX_RANK_WILDCARD value indicates that all procs in the provided nspace are to receive the data
Error Codes
PMIX_ERR_NOT_SUPPORTED
References
- https://github.com/pmix/RFCs/blob/master/RFC0025.md
- https://pmix.org/pmix-standard/input-output-forwarding-for-tools/
I don't think I understand this - it seems to mix several completely separate things:
-
envars - this has nothing to do with IO forwarding. We allow specification of envars to be forwarded and/or modified. This can be done by anyone (client, tool, etc), but is also used by debuggers to modify key system parameters when launching an application under a debugger.
-
event notification - again, nothing to do with IO forwarding. We forward events for lots of reasons, but actually never for IO forwarding.
-
IO forwarding for tools. The channel definitions and APIs relate to that area and conform to the title of this use-case
-
Log - which has nothing to do with IO forwarding or any of the other subjects. The paragraph on log captures much of that one.
Perhaps I am missing something?
@rhc54 This is the first step to addressing the discussion at the Quarterly where you mentioned that I/O Forarding was a completely different use-case and not limited to tools/debuggers.
What I have here is from the RFCs. I may have mixed or misunderstood something there. Thanks for your comments, I will use your them to clean this use case.
Also looking at https://pmix.org/pmix-standard/input-output-forwarding-for-tools/ for guidance.
Thanks @spophale for the latest update. I think this is looking good! Few comments below:
covering the cases where an application is being spawned using the PMIx_Spawn API versus where the tool is connecting to an already-running application.
I really like this breakdown of "PMIx_Spawn vs already running" along with "stdout vs stdin". I think the differentiation between these two modes (PMIx_spawn vs already running) should be explicitly called out within the two current sections ("stdout vs stdin"). For example, in the stdout/stderr section, the PMIx_IOF directives can be passed to PMIx_Spawn, that is clearly called out. What isn't as clear is how one would use PMIx to forward stdout/stderr for an already running application (i.e., via the PMIx_IOF_pull function).
It may be worth noting in the text that a tool can specify several formatting options when using PMIx_IOF_pull (the PMIX_IOF_TAG, PMIX_IOF_TIMESTAMP, and PMIX_IOF_XML_OUTPUT attributes) and to find their definitions in the Attributes/Directives section further down.
Two methods of support for forwarding of stdin are defined:
Only one method is defined currently, so I think this should be reworded, or the second method defined. Based on the breakdown in the previous section, I would expect the second method to involve PMIx_Spawn (i.e., passing PMIX_FWD_STDIN and PMIX_STDIN_TGT to PMIx_Spawn).
What isn't as clear is how one would use PMIx to forward stdout/stderr for an already running application (i.e., via the PMIx_IOF_pull function).
I'm assuming you are just asking for more text describing that call? Or is there a question as to how that call works?
The latter is rather simple as PMIx just passes a request to the host environment asking that it collect the stdout/stderr from the specified procs and forward it to the requestor's specified callback function. Completely up to the host environment as to if it can do so, and how it does it. Typically, the host environment has fork/exec'd the target procs and therefore is already collecting the stdout/stderr from them (e.g., for forwarding to the user).
There are attributes (though I don't see them in the above list) for specifying whether you want to get a copy of those streams (PMIX_IOF_COPY_STREAM), or intercept those streams (PMIX_IOF_INTERCEPT_STREAM) - e.g., to execute a debugging operation without polluting the app's stdout/stderr being collected in a file.
Two methods of support for forwarding of stdin are defined:
Only one method is defined currently, so I think this should be reworded, or the second method defined.
The two methods indeed are:
- separate call to
PMIx_IOF_pushto begin pushing your stdin to the specified targets - inclusion of
PMIX_IOF_PUSH_STDINin a call toPMIx_Spawnindicating that your stdin is to be forwarded to the specified proc started by the spawn
The actual result (FWIW) is the same in that PMIx will collect your stdin and pass it to the host environment for forwarding and delivery to the target proc(s).