gitpod-io
Support stable URLs for endpoints
Hey gitpod team,
I'm on the premium plan and using one of my workspaces for development work on a repo. I've proxied a url e.g. dev.my-url.com to the url of the exposed port when I'm running the site so I can preview the work on my domain url (because some services I use the api keys are tied to the url) and also I can remember it then etc.
Anyway, this has been working fine, but today I opened the workspace and ran the site on the port, but it seems the workplace url has changed. Is this something that is expected? As in the same workspace may have a different url? I've not created a new workspace from the same repo or anything like that. I can reset up the proxy to the new url, but I would rather not have to keep doing this if it can be avoided. I expect to be working on this project for a few months.
Thanks 😄
Workspace urls are not meant to be stable, as they depend on the cluster they are running in. You can always look up the URL by running
gp url <port>
Would it be possible to have your proxy accepting this information on some endpoint? That way you could update the proxy whenever you start a workspace and you could even easily start fresh workspaces on your repo all the time, which is how Gitpod is intended to be used.
Thanks @codemzy for sharing your usecase! I think this is an excellent point for adding a "URL" feature to Gitpod. :slightly_smiling_face:
I use Auth0 to handle logins. Authorized login urls need to be defined in my Auth0 dashboard or via an admin api.
Using the admin api just heaps extra complexity on my dev environment (and means I have to strip this code out in production) - the management of live and unused urls via the api would not be trivial either.
I've learned to double check my workspace url whenever I get a service error page from Auth0 now and update my live urls in Auth0 dashboard.
However I can see this issue become more critical as I use more 3rd party services with webhooks/authorized urls etc.
Workspace urls are not meant to be stable, as they depend on the cluster they are running in. You can always look up the URL by running
gp url <port>Would it be possible to have your proxy accepting this information on some endpoint? That way you could update the proxy whenever you start a workspace and you could even easily start fresh workspaces on your repo all the time, which is how Gitpod is intended to be used.
I don't think this would be possible, I set up the proxy from my netlify production deploy in the _redirects configuration, so while it is something I can easily update, I don't think it's something I could easily automate.
Maybe I'm not using gitpod how it was intended, by I usually need my dev environment for at least a few weeks at a time while I work on adding a feature or working on a new project, so it would really handy if each workspace could have a static url that doesn't change for the lifetime of the workspace. Especially useful when you have to authorise your api keys against a URL on other services you are using during development.
Please consider this! 🙏 🌟
Please consider this! 🙏 🌟
Sure, we are thinking about a solution. Thank you for sharing your use cases.
I also have the same Auth0 challenge as @devops-at-alinea - a stable endpoint would be very helpful so I don't have to update the Auth0 dashboard all the time.
I think that wildcards can be specified for subdomains in the Auth0 dashboard, so it there was a way to have the project name in the subdomain it might solve the need.
For example: https://xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxx.myprojectidentifier.ws-eu03.gitpod.io/ OR with a hyphen https://xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxx-myprojectidentifier.ws-eu03.gitpod.io/
Where myprojectidentifier is some combination of project/username (would need some thinking about).
Within Auth0 I could allow the callback url https://*.myprojectidentifier.ws-eu03.gitpod.ioor https://*-myprojectidentifier.ws-eu03.gitpod.io respectively for the two cases above.
According to their docs I think that would work: https://auth0.com/docs/applications/wildcards-for-subdomains. As a guess maybe the hyphen option would be easier to implement because no additional sub domain level would be required?
I have created an example for Auth0 https://github.com/gitpod-io/auth0-express-webapp-sample
Thanks @svenefftinge!
That's what I've done - the other suggestion was to be slightly more secure by limiting to the project identifier but actually it doesn't really matter for the dev environment anyway.
Another use case, @svenefftinge, at GitLab we want to use it to develop integrations with the Jira Cloud Platform. A changing URL means updating the URLs inside of Jira every time you restart a pod. The linking happens via an iframe.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
We just announced preview version of Gitpod local companion which allows to tunnel any tcp port: https://www.gitpod.io/blog/local-app particularly it allows you access everything on localhost on proper port, similarly how you will do it in the local env, please try and give us feedback 🙏🏻
One key question to answer here is how that stable URL gets assigned to a workspace? What if I have two workspaces open?
From Slack:
@geropl Some thoughts on the actual "redirector":
- |name|.|userId|.gitpod.io won't work because we need to limit ourselves to one level of wildcards (because of certificates)
- using meta/"proxy" for this is not nice for tying workspace-bound connections to it (again)
- the only reason we need "meta" is for the "name" -> "workspace" mapping
- (authentication is done by ws-proxy anyway)
- one could think about:
- have an endpoint on /api/ws-name-mapping that on GET stores a cookie containing the current mappings
- have a completely decoupled "redirector" component that could be deployed anywhere; for instance under
@csweichel
|name|.|userId|.gitpod.io won’t work because we need to limit ourselves to one level of wildcards (because of certificates)
:100: how about
using meta/“proxy” for this is not nice for tying workspace-bound connections to it (again)
indeed, this would introduce some level of connection. However, I don’t think it would be stronger than what we have right now, because it would very much be based on the IDE URL that we already rely on from the meta side. It would introduce a “failure coupling” though where if meta goes down the redirector fails. the only reason we need “meta” is for the “name” -> “workspace” mapping We could also consider transferring this to the workspace side and remove meta from the equation altogether.
have an endpoint on /api/ws-name-mapping that on GET stores a cookie containing the current mappings
Downside of this approach is that you need cookies available/set for the redirect to work. For API calls/curl that won’t be the case.
have a completely decoupled “redirector” component that could be deployed anywhere; for instance under |name|.names.gitpod.io
Could certainly be its own component, but it would still need to run in either meta or workspace. I don’t think the operational overheard of introducing something outside of those two structures is worth it right now.
:100: how about
.ws.gitpod.io?
We could also consider transferring this to the workspace side and remove meta from the equation altogether.
Both contradict each other, no? I thought getting rid of the ws part was one of the requirements (cluster independence).
However, I don’t think it would be stronger than what we have right now, because it would very much be based on the IDE URL that we already rely on from the meta side.
Not sure what you mean with this. What I implied above is that we want to have an "internal redirect" which is transparent to browsers - hence the strong connection. The reasons I thought this is a requirement was to support all kind of clients scenarios.
But it might make sense to go through those use-cases one-by-one and make sure they are important enough to warrant that complexity. If we would have a "simple redirect" (302) the implementation is far easier indeed.
💯 how about .ws.gitpod.io?
We could also consider transferring this to the workspace side and remove meta from the equation altogether.
Both contradict each other, no? I thought getting rid of the
wspart was one of the requirements (cluster independence).
It would just be ws, not ws-eu and hence independent of the region. Basically I wasn't too fond of the names segment and figured that ws is closer to home.
However, I don’t think it would be stronger than what we have right now, because it would very much be based on the IDE URL that we already rely on from the meta side.
Not sure what you mean with this. What I implied above is that we want to have an "internal redirect" which is transparent to browsers - hence the strong connection. The reasons I thought this is a requirement was to support all kind of clients scenarios.
But it might make sense to go through those use-cases one-by-one and make sure they are important enough to warrant that complexity. If we would have a "simple redirect" (302) the implementation is far easier indeed.
I reckon we should avoid to actually tunnel traffic through the "redirector". That would
- incur considerable network load,
- increase cross-cluster egress cost,
- too tightly couple meta/workspace.
If we combined the simple redirect with CNAME DNS entries, we'd also make this work for clients that don't cope with redirects. We would also open the door for weird DNS-caused failure modes (e.g. cached DNS entries pointing to the wrong workspace).
It would just be ws, not ws-eu and hence independent of the region. Basically I wasn't too fond of the names segment and figured that ws is closer to home.
Got it, completely d'accord.
I reckon we should avoid to actually tunnel traffic through the "redirector"
:+1:
If we combined the simple redirect with CNAME DNS entries
That would be nice indeed! And would nicely match the semantics as we're offering a "name service" :slightly_smiling_face:
Awesome - happy to find consensus on the technical side :)
We still need to answer a bunch of questions around the stable name itself:
- how does that name come to be? Is it computed based on the repo/context URL - and/or can users choose it?
- what happens if you start multiple workspaces that would want the same URL?
- how do we handle name clashes with other workspaces that might end up on the same "stable URL", esp. if users can influence that URL?
Could it have the username or a unique string assigned to each user in the url? E.g. |userId|-|custom-url|.ws.gitpod.io and assuming stable urls will be a premium feature (e.g. on the paid plans) you get a random url how it is now as standard, and maybe the user could optionally assign the stable url (e.g. the custom url part of it). If they try to assign a stable url that they are already using, it blocks it. (Like on github if you try to assign a repo with the same name it tells you that you can't).
Although to be honest, for my original use case I think how things are good now. The workspace name carries over the url, e.g. the workspace name red-penguin-random becomes https://[port]-red-penguin-random.ws-eu11.gitpod.io/ so I can proxy a custom domain (e.g. dev.domain.com) to that. If I get rid of a workspace and create a new one from the same repo, I don't expect the url to stay the same. The issue I had before was that the workplace url might change day to day. I don't think this happens anymore (although I may be wrong and I can't speak for other use cases!).
So far I thought about this as a way to expose a workspace, so there's always a 1-1 connection between "stable name/URL" and "workspace". Following this |workspaceId|.ws.gitpod.io is the simplest solution that avoids most of the mention problems (name clashes).
Everything else - especially custom names (with possible per-user namespace) - would be an addon and could be handled atop of this.
So far I thought about this as a way to expose a workspace, so there's always a 1-1 connection between "stable name/URL" and "workspace". Following this
|workspaceId|.ws.gitpod.iois the simplest solution that avoids most of the mention problems (name clashes).Everything else - especially custom names (with possible per-user namespace) - would be an addon and could be handled atop of this.
That would work great for my needs. Even if you still include the port in the url |workspaceId|-|port|.ws.gitpod.io to allow for different ports to be used.
how does that name come to be? Is it computed based on the repo/context URL - and/or can users choose it?
We should not include workspace names, because they are ephemeral. Instead, we should use a combination of project and user.
Something like |team|-|project|-|port|-|username|.ws.gitpod.io should work.
what happens if you start multiple workspaces that would want the same URL?
We should surface the public URLs in the ports view. When not available we can assign a different one (e.g. add a number to the end) or indicate that it is taken already.
how do we handle name clashes with other workspaces that might end up on the same "stable URL", esp. if users can influence that URL?
This would not be possible with the proposal above. Do we know a use case for "user influenced" URLs?
I've been following along with this thread as I came to Gitpod looking for something similar. I've actually switched to GH Codespaces now but thought I'd share my setup on how I got a custom domain / stable URL working with Gitpod and Cloudflare. Hopefully it helps others.
I created node script called dev-server-redirect.js which contains the following...
const { exec } = require("child_process");
const CloudflarePageRules = require('cloudflare-page-rules');
// Setup CloudFlare SDK...
const cloudflare = new CloudflarePageRules(process.env.CLOUDFLARE_TOKEN);
const devServerPort = 8081;
// Get our workspaces url...
exec("gp url", async (error, stdout, stderr) => {
if (error) {
console.log(`error: ${error.message}`);
return;
}
if (stderr) {
console.log(`stderr: ${stderr}`);
return;
}
// Get the dev environment url and remove hidden line breaks...
let url = stdout.replace(/(\r\n|\n|\r)/gm, "");
// Prepend the dev server's port...
url = url.replace('https://', `https://${devServerPort}-`);
// Append the url params so CloudFlare passes them through correctly...
url = url + "/$1";
try {
const result = await cloudflare.edit(process.env.CLOUDFLARE_PAGE_RULE_ID, {
"targets": [
{
"target": "url",
"constraint": {
"operator": "matches",
"value": `${process.env.CLOUDFLARE_CUSTOM_DOMAIN}\/*`
}
}
],
"actions": [
{
"id": "forwarding_url",
"value":
{
url,
"status_code": 302
}
}
]
}, process.env.CLOUDFLARE_ZONE_ID);
if (result.success) console.log(`CloudFlare Page Rule updated with ${url}`);
} catch (error) {
console.log(error);
}
});
It essentially grabs the dev environment's current url and updates a Page Rule on Cloudflare to manage the redirection for the custom domain / stable url.
I ran this script each time the environment started so it auto updated for me.
It worked well for what I needed so hopefully it can be an interim solution for others.
Thanks
We should also consider if static IPs are important to support in addition or alternatively. It has been discussed here, but I'd like to discuss this in this context. So I'm closing it.
Something like |team|-|project|-|port|-|username|.ws.gitpod.io should work.
That seems reasonable. I'm just concerned with two things:
- the edge case of having multiple workspaces of the same project, as the requirement is not met in those cases (i.e. it breaks), but can't really think of a better option.
- the URL is stable for the individual and not for the whole team which requires additional steps from users (e.g. configure env. vars with their API keys). The only way I see to mitigate that is by defining local urls: the local companion app could automatically define an entry for
|team|-|project|-|port|.gitpod.local.
We should also consider if static IPs
The problem we are solving with the static IPs seems somewhat different: We want to avoid having to reconfigure some firewalls with whatever IP is being used by the workspace(s).
To solve that, and thinking that this might be relevant to companies using Gitpod SaaS, we could add a "Static External IPs" option for teams. When selected, one or multiple IP addresses would be shown and those addresses would be exclusively and always used by the prebuilds and workspaces of that team.
Another place where I think stable URLs would help with a conjunction of own DNS server on client side: https://github.com/gitpod-io/gitpod/issues/5306
We discussed this internally and we constantly go back to the fact that this problem is not entirely solvable as long as the URL is globally defined (because of conflicts of multiple running workspaces).
We could for example assign a pool of URLs to a given project to be used to configure third-party callback URLs settings that don't support wildcards, but it would be a complex flow and wouldn't solve the iframe/JIRA problem @leipert is having.
💡 Would it be possible to use the browser extension as a proxy, and tightly integrate with the Gitpod app?
I'm thinking something like: (1) having an option in the workspaces list and IDE to "Set local URL...", (2) user inputs string into a textbox, (3) browser extension would proxy {chosen-string}-{port}.gitpod.local to the usual workspace url. If the browser extension is not installed the user is invited to install it.
Locally the user could do something similar (functionally-wise) via VS code extension or CLI.
Would it be possible to use the browser extension as a proxy, and tightly integrate with the Gitpod app?
I don't think so, we are talking about integrating in 3rd party services which don't use the browser or cannot install any apps in own infrastructure. If it would be the case one can just use localhost via the local companion. An idea that it should work on Internet without extra.
@akosyakov maybe we are thinking about different issues. I'm assuming these URLs don't need to be publicly available and they can be defined in any domain (i.e. gitpod.local) as long as they are stable. Are those assumptions incorrect?
In order to move this forward faster, @akosyakov and I jumped on a call to talk about this. Notes:
- The local companion app should solve the issue of non-stable URLs since ports become accessible at
localhost:{port}as it's normal in local development. - Currently, we are still improving and stabilizing the local companion app. Documentation needs to be written also.
- Concluded that probably the best path forward is investing in making the process of using the local companion app for this use case as seamless as possible. For example, by making it run as a daemon—which in fact is already planned to happen.
Although, if we understood correctly, @gilesbutler's use case might not be solved by this. @gilesbutler could you please share with us more details? Particularly whether you were looking to have a stable URL that is globally available on the internet and, if that's the case, why?
In the meantime, anyone feeling adventurous may try our preview release of the local companion app following the steps detailed here. :)