ceph
Add Support for Docker Swarm / Repackage as Docker Plugin
Describe the feature you'd like to have
Recently published Docker 23.0.0 added CSI support for Swarm mode so it is possible to use CSI drivers with it but they have to be packaged as Docker plugin.
What is the value to the end user? (why is it a priority?)
Docker Swarm users would gain the ability to use ceph in Docker Swarm via CSI instead of relying on bind mounting volumes.
How will we know we have a good solution? (acceptance criteria)
- Packaged the CSI driver for Docker.
- Able to create a cluster volume via the docker cli
Additional context
You can find more information here. Hetzner CSI has already merged a PR that adds support
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.
Contributions from someone that has tested it on Docker Swarm are still welcome!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.
There seems some interest in this feature. Most of the regular Ceph-CSI contributors are not familiar with Docker Swarm, so it is not something that they easily can do.
Contributions from others that would like to see this feature made available are most welcome!
@nixpanic I would love to help out here at least with guidance, but sadly I don't have a working CEPH cluster to test this against. Any hints how to solve this (for dev at least)?
Hmm, I always use Rook to deploy my Ceph cluster. But that obviously won't work if you want to use Docker Swarm instead of Kubernetes.
The go-ceph project (https://github.com/ceph/go-ceph) runs CI with a small Ceph cluster that is deployed by some scripts. You might be able to use that to setup a small environment to test against.
Ah, you are proposing adapting this https://github.com/ceph/go-ceph/blob/9408dd58842306bca172192d320ea9cfab08065a/Makefile#L182 but leaving out the final testing step. Good idea, should work.
Even though this is closed as an issue, I guess we can keep the discussion here? I plan to work on this when I have time, but I don't know exactly when that will be.
I do not see why this issue closed when its clearly should not be. Definitely keep us posted with you progress, or if there anything what we can help with.
There seems some interest in this feature. Most of the regular Ceph-CSI contributors are not familiar with Docker Swarm, so it is not something that they easily can do.
Contributions from others that would like to see this feature made available are most welcome!
I'd like to second what @nixpanic mentioned in a comment above.
I've reopened the issue and added keep alive label to avoid it from closing again.
I 100% understand. That's why I am offering to help out. But as you know, time is a precious resource :). I will try to get some prototyping in, but that might not happen for a long time or it might happen over the weekend. I just don't know.
So to anyone of the Swarm fans - if you want to work on this (maybe with my help) - reach out to me or join our Swarm Fans discord over at https://devops.fan . Then we can join efforts :).
In general, I think that having a CEPH CSI provider for Swarm would be huge for anyone that is not on a cloud provider.
Leaving some notes of prelimiary investigation here:
The Docker volume plugin will require its own entrypoint as configuration needs to happen via environment variables with docker plugins. I have only found cephcsi.go which uses flags.
To anyone wanting to pick this up, seaweed has also been repackaged to be compatible with Docker Swarm now: https://github.com/seaweedfs/seaweedfs-csi-driver/tree/master/cmd/seaweedfs-csi-driver/swarm
Maybe this helps.
I ventured into trying this myself and have some results, although nothing working yet. (Screenshots are barebones for illustration so as to keep this short)
The primary hurdle is to package the driver into a docker managed plugin and provide some wrapper that parses the docker plugin settings into cephcsi startup arguments, but that is manageable. I did this with a simple shell script, eg:
I then configured the docker plugin config.json accordingly, eg:
Then I constructed a Gitlab Pipeline to build the thing:
..and installed the plugin on a suitable swarm node with all ceph config present.
All goes well, and there are no obvious errors, even creating volumes goes successfully, but they stay stuck in "pending creation":
Also when running docker volume list --cluster, there is a "invalid character '/' after object key:value pair" in the daemon log.
If anyone wants to pick this up I will gladly share my stuff, as I'm scraping at the outer edges of my wits.
If you share your code somewhere (here?) we can take a deeper look to see if we spot the issue. I would even give it a test run on a Proxmox installed Ceph with some Swarm VMs
Besides Ceph RBD did you tried CephFS?
Thank you for the swift response, @miberecz . I have uploaded my code here: https://github.com/monoamin/ceph-csi-docker
And yes, I did try setting the driver to cephfs, unfortunately it resulted in the same behaviour with no noticeable difference.
The main cephcsi executable should provide many logs if it received the request to provision a volume. I am not familiar with Docker Swarm, but if you have error messages I might be able to give a hint on what you could check/try.