ServiceWeaver
ssh deployer errors with "unknown keys [location_file]"
I've tried both setting up my own test weaver application and using the collatz example and both times when running
❯ weaver ssh deploy weaver.toml
I get the following error message
unable to parse ssh config: section "ssh" has unknown keys [locations_file]
I've tried looking at the code but nothing seems to stand out to me at a quick glance as to why this is erroring. I don't fully understand how all the code works just yet but thought I'd raise the issue and continue debugging
Nice bug report! I'm just learning the code as well, but I may have an idea about this problem.
https://github.com/ServiceWeaver/weaver/blob/35727ae307c392d27cadfbf5302d56a3e76977ab/internal/tool/ssh/deploy.go#L181-L186
I believe the serverweaver_metric struct tag on line 182 is wrong. It doesn't appear anywhere else in the code, and ParseConfigSection (which receives that struct) seems to use https://pkg.go.dev/github.com/burntsushi/toml#Decode to decode the config. That function expects a toml struct tag.
That leaves the locations_file field from the config as "undecoded", which causes the error here:
https://github.com/ServiceWeaver/weaver/blob/35727ae307c392d27cadfbf5302d56a3e76977ab/runtime/config.go#L87-L88
I suspect this diff should fix it, but I'm not familiar enough with the testing of this package to add test coverage for the change:
diff --git a/internal/tool/ssh/deploy.go b/internal/tool/ssh/deploy.go
index 6a6e276..9be6534 100644
--- a/internal/tool/ssh/deploy.go
+++ b/internal/tool/ssh/deploy.go
@@ -179,7 +179,7 @@ func getLocations(app *protos.AppConfig) ([]string, error) {
const shortSSHKey = "ssh"
type sshConfigSchema struct {
- LocationsFile string `serverweaver_metric:"locations_file"`
+ LocationsFile string `toml:"locations_file"`
}
parsed := &sshConfigSchema{}
if err := runtime.ParseConfigSection(sshKey, shortSSHKey, app, parsed); err != nil {
Hi @aranw. Thanks for pointing out the issue. @dnephin this is one of the problem, great find :). There is another small issue we introduced while renaming the project. I am creating a fix.
Ah thanks for finding this so quickly. I didn't get too deep into identifying the issue before having to go out. I'll try this diff either later tonight or tomorrow at some point
Sorry for the inconvenience. Please try the diff and let us know if you have any issues running the SSH deployer.
Yeah that looks like it has progressed a little further now onto a new error relating to creating deployment directory
Thanks for the quick turn around with a fix!
Hmm, can you share the error? I tried on my local machine and it works fine. Are you testing on your local machine or on multiple machines?
It is a remote server
unable to create deployment directory at location #####: exit status 1
##### is the name of my host alias in my ssh config file
Got it. We create a deployment directory on each remote machine where Service Weaver will be deployed. Right now we try to create the directory using ssh and the location is os.TempDir()/deploymentId. https://github.com/ServiceWeaver/weaver/blob/main/internal/tool/ssh/deploy.go#L147
Can you try if simply creating the remote directory over SSH works? e.g., ssh #### "mkdir -p /tmp/bla". I am wondering whether you have some permission issues. Are you running linux machines?
I added some output to the command to see what the tool was doing and this seems to be the error
mkdir: cannot create directory ‘/var/folders’: Permission denied
Not sure where it is getting /var/folders yet need to trace the code a bit more
This is the command it is running
I'm guessing this is a folder on my local machine
/usr/bin/ssh -v ### mkdir -p /var/folders/sq/3sdsh9n14p790p7xwhdlvdzc0000gn/T/915574ab-82d5-47f1-8efa-47f5062f03cf
Hmm, this is interesting. On linux, the temp dir is /tmp. What is your setup and which OS are you running? It feels like macOS?
It seems like your ssh client doesn't have privileges to create a directory in /var/folders, maybe? Alternatively you can try to change the os.tempDir() to a custom directory where you can do writes over SSH and check if it works for now.
I have a macOS laptop and I will give it a try as well. To be fair, we tested the SSH deployer only on linux machines.
Ahh so I understand what is happening now
https://github.com/ServiceWeaver/weaver/blob/35727ae307c392d27cadfbf5302d56a3e76977ab/internal/tool/ssh/deploy.go#L143
On this line you call os.TempDir() but because I am running this from my laptop (macOS) that call to os.TempDir() returns a temp directory path for my local machine not for the remote ssh host that I am connecting too
Edit: I can create a separate issue for this if you want?
That's a good point. Yeah, we didn't try to deploy it using SSH from our laptop on a cluster. Thanks for the finding. Sure, send us a fix. Thanks for fixing this :).
@rgrandl I'll write up a new issue and little summary of the problem later today. I did some digging into the issue last night but was late for me (GMT timezone) so ended up not working on a fix. I have an idea for a possible fix but will discuss in more detail a possible solution in the issue later