aurae-runtime
Create "Container Service"
This will sound a bit confusing at first, but just go with me on this.
We need a new "top level" service that sits alongside the existing "Cell Service".
This should be an extremely lightweight container service that runs containers without the concept of CRI or a Pod Sandbox. The new "container service" will be called from a host auraed to schedule a container in a new pod sandbox.
Example
Here is an example "model" to help me communicate what I am thinking.
Imagine a new host called "alice" which is effectively a bare metal server running in a rack.
Alice will run a Linux operating system with auraed as pid 1 and we will call this initial instance of auraed the Host Auraed.
A user remotely schedules 2 pods on Alice (auraed-guest-1 and auraed-guest-2) each has their own nested instance of auraed.
The Host Auraed first creates the guest for each pod, and then calls out to the cell service being suggested in this GitHub issue to schedule containers on each guest. The containers in each guest are free to communicate with each other using the guest filesystem, and the guest namespaces where applicable.
alice/
├── auraed # Host Auraed
├── auraed-guest-1 # Guest Pod 1
│ ├── auraed # Guest Auraed 1
│ └── container-service.rpc # New unwritten RPC service
├── auraed-guest-2 # Guest Pod 2
│ ├── auraed # Guest Auraed 2
│ └── container-service.rpc # New unwritten RPC service
├── cell-service.rpc # Host cell service; sits alongside other RPCs, health, etc
└── cri.rpc # Host CRI where we user calls "RunPodSandbox"
i never quite got my head around how this sits alongside the CRI or Pod sandbox concept, but if we're moving to a place where a guest can run "bare-metal" Cells, or VMs (cloud-hypervisor), or Containers (this service) then i'm happy with that as a direction.
It would be nice to (eventually) have CRI support to allow for something like an auraed-managed kubelet spawning pods in a nested cell or in a VM (using aurae as its container runtime). I'm not sure, though, that this is something we'd want to prioritize for the short term goals of the project?
if we're moving to a place where a guest can run "bare-metal" Cells, or VMs (cloud-hypervisor), or Containers (this service) then i'm happy with that as a direction.
To this point, I think having support for pulling/running container images normally would be a great start (similar to our approach with the VM service). Once we have that functionality, we can iron out how/when we'd want to implement CRI, whether with cells or VMs (or both!) as the Pod boundary.
i never quite got my head around how this sits alongside the CRI or Pod sandbox concept, but if we're moving to a place where a guest can run "bare-metal" Cells, or VMs (cloud-hypervisor), or Containers (this service) then i'm happy with that as a direction.
doing some more research on this i think "containers" should be an easy to specify configuration for cells vs its own thing. I believe aurae should only have cells & vm's
what's the difference from a user perspective between a cell-with-container service and a container service?
I think the latter can be implemented as the former, but having a clean separation in the API makes it easier to use, and more obvious which mode the user code is working in.
I suspect merging the implementations might also make cells serviced harder to maintain, as there's going to be a bunch of conditionals everywhere, but I'd be happy to be shown that it actually makes it easier to maintain than two services.
I mostly agree with this. people have a specific expectation of what "container" means and cells allow us to explore other things, but functionally yea an aurae container is absolutely a cell