Allow Funnel to run tasks against custom domain socket

[lbeckman314]

Overview

General Use Case: user launches funnel and then launches another process to issue the jobs, all while using named sockets to handle communication between the funnel server and the job issuer process. This would allow the job issuer process to use existing file permissions as the socket network by inheriting permissions from the containing directory.

Specific Use Case: user launches funnel on an Exacloud head node with jobs scheduled by Slurm.

This feature should be configurable from the funnel command line:

Funnel server:

funnel server run --socket ./my_socket

Job issuer:

funnel task create my_task.json --socket ./my_socket

Testing

Minimal testing environment:

• Funnel server configured to use custom Unix socket
• One working directory
• Using Slurm (in Exacloud)
• Introspect server address to determine whether Funnel server is running over HTTP or over Unix socket
• Submit tasks with py-tes using Unix socket
• Submit tasks with Funnel client using Unix socket

Additional Resources

• https://www.brainboxes.com/faq/what-is-unix-domain-socket
• https://dev.to/douglasmakey/understanding-unix-domain-sockets-in-golang-32n8
• https://www.baeldung.com/linux/python-unix-sockets#1-server-code

Python Implementation

# Create the Unix socket server
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

# Bind the socket to the path
server.bind(socket_path)

UNIX Domain Sockets Zine by @jvns

https://wizardzines.com/comics/unix-domain-sockets/

[lbeckman314]

Overview

Diagrams for designing and implementing Funnel on Exacloud Google Docs Link

Here we are documenting the required connections between the three different entities involved with Funnel's task execution:

1. The Funnel Server
2. The Job Submitter
3. The Task Executor

⚠️ Note: these diagrams outline features under active development and may not accurately reflect Funnel's default behavior.

Funnel on Generic Platform

Google Diagram Link

Funnel Exacloud

Google Diagram Link