Issues installing persistent binderhub chart

[sgibson91]

Hi folks,

I was trying to have a play with the persistent binderhub today but I couldn't get it installed (following instructions in the readme).

Here's the install command and error:

$ helm install persistent_binderhub/persistent_binderhub --version=0.2.0-n181 --name pbhub --namespace pbhub -f config.yaml --debug
[debug] Created tunnel using local port: '65528'

[debug] SERVER: "127.0.0.1:65528"

[debug] Original chart version: "0.2.0-n181"
[debug] Fetched persistent_binderhub/persistent_binderhub to /Users/sgibson/.helm/cache/archive/persistent_binderhub-0.2.0-n181.tgz

[debug] CHART PATH: /Users/sgibson/.helm/cache/archive/persistent_binderhub-0.2.0-n181.tgz

Error: render error in "persistent_binderhub/charts/binderhub/templates/deployment.yaml": template: persistent_binderhub/charts/binderhub/templates/deployment.yaml:98:74: executing "persistent_binderhub/charts/binderhub/templates/deployment.yaml" at <"/">: invalid value; expected string

And my config file looks like this:

binderhub:
  jupyterhub:
    hub:
      services:
        binder:
          apiToken: "redacted"
    proxy:
      secretToken: "redacted"
    singleuser:
      storage:
        dynamic:
          storageClass: azurefile
  registry:
    username: sgibson91
    password: "redacted"

Looks to me like a templating error somewhere into the binderhub chart, but I can deploy the standard binderhub chart without issue. Also a / character does not exist anywhere in the redacted secrets, so I'm not quite sure where that error comes from.

Any help would be greatly appreciated, thanks!

[bitnik]

Hi Sarah! I am very glad that you are try this out. And I would really appreciate any feedback for the documentation (I know it has a lot to improve).

Error: render error in "persistent_binderhub/charts/binderhub/templates/deployment.yaml": template: persistent_binderhub/charts/binderhub/templates/deployment.yaml:98:74: executing "persistent_binderhub/charts/binderhub/templates/deployment.yaml" at <"/">: invalid value; expected string says that problem is at 98:74 (line 98, column 74) which is https://github.com/jupyterhub/binderhub/blob/23ae2061decdf8e9f7b94829cabe602b6ed36519/helm-chart/binderhub/templates/deployment.yaml#L98 - I think you are missing binderhub.config.BinderHub.hub_url in your config.

And my config file looks like this:

Is that all your config file? If yes, you are also missing the auth config (https://github.com/gesiscss/persistent_binderhub#authentication).

[sgibson91]

I haven't put the hub_url or auth config in yet because I don't have the IP addresses of the hub/binder pods (or a domain name) to fill in those config values because I can't deploy the chart yet, hence no IP addresses. This is a circular problem if those values are required for initial deployment.

[manics]

Can you put a dummy IP in so that the config validates, then redeploy with the actual IP when you know it?

[sgibson91]

I mean I could, but the BinderHub chart doesn't require this. And I think for newcomers to helm, I think that's a bit of an intuition leap.

[bitnik]

I haven't put the hub_url or auth config in yet because I don't have the IP addresses of the hub/binder pods (or a domain name) to fill in those config values because I can't deploy the chart yet, hence no IP addresses.

Ah I understand, I never had this issue because we work on bare metal, we always know the IP beforehand. But @manics 's suggestion sounds very good, did that error disappeared after putting a dummy IP?

There is another thing that I am curious about: that part in the binderhub chart is only executed when authentication is enabled. As far as I remember you also already installed a BinderHub with authentication, did BinderHub chart also complained about it when you installed a BinderHub with authentication? That would be very useful for me to know.

[sgibson91]

Ah I understand, I never had this issue because we work on bare metal, we always know the IP beforehand. But @manics 's suggestion sounds very good, did that error disappeared after putting a dummy IP?

I haven't tried yet, working on project stuffs today :) Will let you know

There is another thing that I am curious about: that part in the binderhub chart is only executed when authentication is enabled. As far as I remember you also already installed a BinderHub with authentication, did BinderHub chart also complained about it when you installed a BinderHub with authentication? That would be very useful for me to know.

No, the BinderHub chart doesn't complain at all if I don't provide hub_url straight away or when I upgraded to include authentication. I wonder if it's because I already had a functional BinderHub and only applied the authentication config as an upgrade later? As opposed to the approach here which is to define it all at the installation point

[bitnik]

I wonder if it's because I already had a functional BinderHub and only applied the authentication config as an upgrade later? As opposed to the approach here which is to define it all at the installation point

That could be the reason. Today I will test this with BinderHub chart.

[sgibson91]

Another potential reason is that I used GitHub auth and if you used a custom authenticator, I wonder if that's less rigorously tested in the source code? Because finding edges cases in the realm of all possible authenticators is hard.

[bitnik]

I wonder if it's because I already had a functional BinderHub and only applied the authentication config as an upgrade later? As opposed to the approach here which is to define it all at the installation point

I just tested locally and got the same error (Error: render error in "binderhub/templates/deployment.yaml": template: binderhub/templates/deployment.yaml:98:74: executing "binderhub/templates/deployment.yaml" at <"/">: invalid value; expected string) when I tried to install the latets BinderHub chart directly with auth enabled and with missing hub_url in the config. I will open an issue on binderhub repo about this. https://github.com/gesiscss/persistent_binderhub/issues/5#issuecomment-669821090 is the workaround for now.

Another potential reason is that I used GitHub auth and if you used a custom authenticator, I wonder if that's less rigorously tested in the source code? Because finding edges cases in the realm of all possible authenticators is hard.

Before I installed persistent binderhub chart with GitHub authenticator as well. But I think using a different authenticator shouldn't be a problem anyways.

[sgibson91]

Ok, adding in dummy values got it to deploy successfully, then I updated with the IP addresses from the binder and proxy-public services and enabled GitHub OAuth. The problem is now, that when I visit the Binder IP page, I get a 404 error? Config is below.

binderhub:
  config:
    BinderHub:
      use_registry: true
      image_prefix: "sgibson91/pbhub-dev-"
      hub_url: "http://20.50.231.204"

  jupyterhub:
    hub:
      services:
        binder:
          apiToken: "redacted"
          oauth_redirect_uri: "http://20.50.231.247/oauth_callback"
          oauth_client_id: "binder-oauth-client-test"

    proxy:
      secretToken: "redacted"

    singleuser:
      storage:
        dynamic:
          storageClass: azurefile

    auth:
      type: github
      github:
        clientId: "redacted"
        clientSecret: "redacted"
        callbackUrl: "http://20.50.231.204/hub/oauth_callback"

  registry:
    username: sgibson91
    password: "redacted"

where 20.50.231.247 is the IP address of the binder service and 20.50.231.204 is the IP address of the proxy-public service.

[bitnik]

hmm can you add this config:

binderhub:
  jupyterhub:
    hub:
      services:
        binder:
          url: http://20.50.231.247

and then change value of oauth_redirect_uri from "http://20.50.231.247/oauth_callback" to "http://20.50.231.204/services/binder/oauth_callback"?

Then binder must be available at http://20.50.231.204/services/binder/ as a service but jupyterhub home page (http://20.50.231.204/hub/home) is where you will find the interface for persistent bhub.

[sgibson91]

Ok, I got it to work! 🎉 Here are my suggestions for improving the docs:

• Document the gotcha about dummy IPs. Folks like me coming from cloud-based BHub deployments would probably expect the chart to "just work" and we update it with the IP addresses when they're created. Explicitly saying that this chart won't work that way will save some confusion.
• Document how and why the hub/binder URLs are different from the standard BHub. Again coming from the standard chart, I would never have assumed that these values needed to be different. Also a little bit of explanation as to why it's set up that way might be nice for the curious, and also admins who may need to debug at some point.
• The URLs provided to the OAuth Provider are different than those described in the z2jh auth docs. With the explanation about the URLs from the previous point, this need be nothing more than "Here's the z2jh auth guide. Remember, the URLs you will be providing are the following..."

Hope this is helpful! ✨

[bitnik]

thank, it is very helpful! I will update the docs, according to your suggestions.

[sgibson91]

Fab, I'm happy to review too :)

[FasihZafar]

Hi @sgibson91, hope you are doing well can you please share a config file that worked for your install. I'm struggling to deploy it would be helpful to have something for reference. Thanks