EPIC: Continuous Deployment of Phoenix Apps to Any VPS

[nelsonic]

At present we don't have a reliable way of doing Continuous Deployment to Linode for our Hits App. https://github.com/dwyl/hits ... Because it's been infrequent, I have done the deploy manually. 🤦

We have setup CI/CD for Client Phoenix Apps in the past e.g: https://github.com/dwyl/learn-microsoft-azure#how But so far we have stuck to using Heroku for @dwyl Apps because it's so easy and cheap (Free!)

Now we need to do it with a fresh pair of eyes and with the benefit of all the enhancements that have occurred in the Phoenix/Elixir/Erlang community over the past couple of years.

Despite running Cowboy (Erlang) under the hood, Heroku still places limits on concurrency and connections to preserve service quality for all users. Also, when you factor in a PostgreSQL database (min $9/month), Heroku gets quite expensive for "hobby" or "side project" apps like Hits. And given that Hits has way more than 10M rows (the limit for the $9/month DB) at this point, we would probably need to pay the $200/month for the "standard" Postgres instance ... 💸 see: https://elements.heroku.com/addons/heroku-postgresql

Todo

• [ ] Create a new file: phoenix-continuous-delivery.md
• [ ] Research and document how to deploy a Phoenix Application the "right" (official) way.
• [ ] Determine if that is suitable for our needs.
  • [ ] We need WebSockets (Channels)
• [ ] Ideally we would like to be able to deploy multiple basic apps to the same VPS. But this is not essential as we can afford to pay $3.50/month for a basic VPS. e.g ovh.com

Related:

• [ ] Logging! #60
• [ ] How to run PostgreSQL on the Same instance or a Cluster of independent machines? see: https://github.com/dwyl/learn-devops/issues/58
• [ ] Should we Use OpenBSD the OS: https://github.com/dwyl/learn-security/issues/73

@SimonLab given that you prefer to focus on "back end" part of your "full stack" skills, I feel like your name is missing from this list: https://github.com/dwyl/learn-devops/graphs/contributors 😉 I've added you to the Linode account using your gmail address. Please add 2FA to enhance security: https://www.linode.com/docs/security/authentication/two-factor-authentication/linode-manager-security-controls And feel free to play with a Linode VPS on my account.

P.S: I'm not married to Linode by any means. I just prefer to use an independent service that is not VC funded or pouring money into the pocket of the world's richest person 🙄 The reason the title of this issue is "Any VPS" is precisely because we want a generic solution to deploying our Phoenix Apps.

[nelsonic]

@SimonLab please give a ballpark estimate for how long you think this will take. ⏳ If the issue description does not have enough detail, please let me know what is unclear. 💭 Thanks! ☀️

[SimonLab]

• [x] Updating the hits dependencies: https://github.com/dwyl/hits/issues/97
• [ ] Learn how mix realease works:
  1. https://hexdocs.pm/mix/Mix.Tasks.Release.html
  2. https://hexdocs.pm/phoenix/releases.html

[SimonLab]

• Best practices for deploying Elixir apps: https://www.cogini.com/blog/best-practices-for-deploying-elixir-apps
• Deploying an Elixir app to Digital Ocean with mix_deploy: https://www.cogini.com/blog/deploying-an-elixir-app-to-digital-ocean-with-mix_deploy

[SimonLab]

My goal is to see how easy it is to run a small Phoenix application with Linode. I'll try to install the following app: https://github.com/simonlab/phx . It doesn't contain Ecto so I don't have to worry with setting up Postgres at the moment (see https://github.com/dwyl/learn-devops/issues/58 Linode provide the Postgres as application which means that it will create a Debian which will host the application. However this means having two linodes, one for the app and one for Postgres, ie paying twice).

The steps I'm going to try is to

• clone the git repository application on the ubuntu server
• Install asdf to then install Erlang and Elixir
• compile the application with mix release

So far I've:

• created an Ubuntu nanode (small linode)

• Install asdf to manage erlang and elixir version

  • git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.7.8

  • Edit ~/.bashrc file and add . $HOME/.asdf/asdf.sh

  • reload terminal to be able to access asdf command. I've rebooted the Linode but I think a simple source ~/.bashrc might have suffice

  • install tools needed for Erlang otherwise you'll get the following warning:

  • sudo apt install libssl-dev make automake autoconf libncurses5-dev gcc

  • add erlang plugin to asdf: asdf plugin-add erlang

  • install erlang: asdf install erlang latest. If I remember correctly this can take a bit of time

  • isntall elixir: asdf install elixir latest

  • define as default the elang and elixir version: asdf global elxir <version> (run same command for erlang

  • run erl or iex to check erlang/elixir is isntalled:

  • clone application: git clone https://github.com/SimonLab/phx.git

  • create a secret key base with mix phx.gen.secret and save it in a .env file as export SECRET_KEY_BASE=<secret> then run source .env

  • create a release: MIX_ENV=prod mix release --path ../phx_release

  • install node via nvm: wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash then nvm install node

  • install depencies and create the release: MIX_ENV=prod mix release --path ../phx_release

  • run the release ../phx_release/bin/phx start

got the following error:

15:17:52.358 [error] Could not find static manifest at "/root/phx_release/lib/phx-0.1.0/priv/static/cache_manifest.json". Run "mix phx.digest" after building your static files or remove the configuration from "config/prod.exs

• See https://hexdocs.pm/phoenix/deployment.html#compiling-your-application-assets to create the assets for the application

• running mix phx.server (without using the release) will start the server and it can be accessible at http://151.236.220.56:4001/

• Start the server with release:

[SimonLab]

I've installed posstgres on the Ubuntu server and at the moment running mix phx.server (without creating a realease):

[nelsonic]

@SimonLab have you had a chance to read Chapter 11 "Deploy Your Application to Production" of the Real-Time Phoenix book?

I think it might be highly relevant to this Epic/Quest as we 100% want to have WebSockets/Channels in our App (that's the biggest selling point of using Elixir/Phoenix! 😉 )

[SimonLab]

I just rapidly read the deployment chapter:

A production server can be run with mix phx.server or by using releases.

Releases package BEAM (Erlang virtual machine), provide scripts (for example to monitor the running application), allow BEAM customisation with flags and preload some code to make the initial responses to client requests faster.

You can decide to deploy the application on a "Platform as a Service" PaaS (e.g. Heroku, Gigalixir, Render) or on a virtual private server VPS (e.g. Linode). Make sure the server/platform can handle a large amount of concurrent connection and can use websocket.

To manage the requests load you can use a load balancer which manage where the requests are sent to (e.g. HAProxy, nginx)

Redeploying an application can be done with rolling deployment where servers are restarted after each others. The blue-green deployment switch between an old cluster (running current applications) to a new cluster (running the new applications). When deploying the live connection will be closed however the load balancer should manage the reconnection to the new servers.

[nelsonic]

@SimonLab thanks for sharing this good summary of your reading. 👍 As discussed on our Standup call on Monday, please add a lot more detail for everything you have researched and links you have followed in your quest to do Continuous Deployment (CD) on a VPS.

Ideally we would like to have a "how to guide" that gives the exact steps to setup CD for Elixir/Phoenix Apps (as per the issue description). And a set of (version controlled) scripts for actually executing the DevOps. Have you started creating a markdown file with your notes of what you have learned? (please push to GitHub)

OpenBSD is a (really) "nice to have" at this point, definitely not a "requirement" right now. Given that we cannot use backups on Linode with OpenBSD, I'd say we should "park" the idea of using BSD on Linode (which is targeted at Linux, the clue is in the name). The goal with considering OpenBSD is "high security" as described in https://github.com/dwyl/learn-security/issues/73 🔐 (please leave a comment on that issue with your experience so far of using BSD... 🙏)

Just the fact that there are considerably fewer people using OpenBSD than Linux means that there are fewer people who understand how to attack/hack it. From a security perspective that's a major plus. It's something Mac had a few years ago that Windows never did. (fewer users means hackers don't bother trying!) Now that Mac is 9% of the desktop market and Hackers know that higher value users are on Mac (e.g. Developers who need to build iOS Apps and pretentious rich brats who buy $2k laptops to do their word processing and web surfing! 🙄 ), Macs are increasingly targeted with malware.

To clarify: I would much rather figure out CD fast using Ubuntu (which will have many more answers on StackExchange, tutorials, blog posts, etc.) than spend days learning BSD so we can deploy the "Hits" (Phoenix) App ASAP.

Linux is "fine" for deploying "Hits" to Linode with backups enabled. The only data that can be considered PII that we store is IP Address and we can easily add Fields.IpAddressEncrypted to encrypt the data at rest thus minimising the effect of a potential breach.

For the purposes of deploying the DWYL APP, we need to make a time-value tradeoff assessment and determine how much more effort is needed to figure out how to get Zero-downtime CD on Linode. The DWYL APP is our "crown jewel" and we cannot afford to have a "hacky" deployment pipeline. We need something with an SLA and at least "five nines" of uptime. If the amount of effort required to set this up now is another 5 days of your time, that's enough cash to pay for Heroku or Gigalixir for a Year. By which point we will either have a few thousand paying customers or have run out of funds to continue working on the App ... 💸

Notes

We still need to get better at Estimating Tasks before starting the work. https://github.com/dwyl/learn-devops/issues/59#issuecomment-632678545 ⬆️ We need to get better at holding ourselves accountable to those estimates to avoid spending time on activities that are not building "features" people using the App want/need. I hope that our App will help with this. 🤞

To be clear: I consider data security to be a "feature" but I don't know how many customers will use our product because of the security. It's more of a "hygiene" factor than a "motivator".

As much as I want to have our own Continuous Delivery with Zero-downtime Deploys to a VPS, I feel that investing more time into this quest is not wise right now. The reason I asked for an estimate above was to help with prioritising the task.

Deploying our App to an existing (PaaS) provider is "enough" security during our MVP. Once we have 1000 paying customers we can re-visit deploying to a VPS with BSD. 👍

[SimonLab]

Thanks for your comment above @nelsonic I've just created a PR (https://github.com/dwyl/learn-devops/pull/61) where I try to recap what I've learn so far while searching how to deploy Elixir/Phoenix. I will need to add more detailed steps later on

I feel confident that for a simple Phoenix application a VPS could be a nice solution, however I still have some aspect that I will need to research (backup, deployment without any downtime, elixir cluster) which can take a at least a few days of reading/testing. With that in mind I also think at the moment a PaaS is a good solution while the application is getting build and user tested.

As much as I want to have our own Continuous Delivery with Zero-downtime Deploys to a VPS, I feel that investing more time into this quest is not wise right now. :+1:

[SimonLab]

I'm currently reviewing the changes that need to be made on #61 As mention in the PR the idea is to use Travis and mix release to test and build the application, once the build is done to store it on S3 to keep an history of the different build versions then to deploy the build on Linode.

I'm reading/searching the following

• [x] deploy-your-phoenix-web-app-using-edeliver
• [x] Read learn-travis
• [ ] Read Travis documentation to understand how the application build is created and how to retrieve it
  • [ ] Read Deployment section on Travis documentation

• [ ] Understand mix release configuration to define the environment variables

Other questions I want to also clarify:

• Difference between Continuous Integration, Continuous Deployment, Continuous Delivery
  • [] read https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html
• Can we use Linode to store build version instead of S3, are there any cost differences?
• How to manage database on Linode
  • Install Postgres on the Linode server directly, or create a "one click" Postgres Linode install?
    • [ ] Read Linode Postgres documentations: https://www.linode.com/docs/databases/postgresql/

[nelsonic]

@SimonLab we can safely skip the S3 step for the Hits Application. Just deploy it directly from Travis-CI to Linode. 🚀

I had to create specific deployment keys to add to Travis. Obvs documented it: https://github.com/dwyl/learn-travis/blob/master/encrypted-ssh-keys-deployment.md If anything unclear, please open an issue on learn-travis/issues and link to it here.

[SimonLab]

Testing the script step on a simple Phoenix application, where the .travis.yml file is:

language: elixir
elixir:
  - 1.10.3
otp_release:
  - 22.1.8
env:
  - MIX_ENV=test
script:
  - mix test
cache:
  directories:
  - _build
  - deps

deploy:
  provider: script
  script: bash deploy.sh

And a dummy deploy.sh file:

#!/bin/bash
echo "#########################"
echo "DEPLOYMENT SCRIPT RUNNING"
echo "#########################"
exit 0

~The script is not run as:~

expanding information of the deployment section, we can see the script is run:

[SimonLab]

Using Distillery and Edeliver we can use hot code upgrade (updating the code to a new version while the server is still running). However mix release doesn't provide hot code upgrade, from https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-hot-code-upgrades

The following thread is the current problem I'm trying to solve: https://elixirforum.com/t/graceful-restart-of-an-elixir-v1-9-release/24211/5

The idea is to have the current and new application running at the same time and to use a load balancer to transfer the requests to the new application. There are two methods, the rolling and blue-green deployments which help to switch to the newest application version:

• blue/green deployment with nginx: https://medium.com/@miket969/blue-green-deployments-with-nginx-cbaa9938bcf8

[nelsonic]

@SimonLab for the Hits App a few seconds of downtime once a month when there is a new release is perfectly acceptable. Please don’t worry about that. As long as we have a deployment we are fine. We can revisit the “zero downtime” deploys if/when we use the script for our “real” app (later).

[nelsonic]

Continue:

• https://hexdocs.pm/phoenix/releases.html
• https://alchemist.camp/articles/elixir-releases-deployment-render
• https://www.cogini.com/blog/best-practices-for-deploying-elixir-apps
• https://elixirforum.com/t/whats-the-best-way-to-deploy-a-phoenix-app-in-production/26420
• https://www.digitalocean.com/community/tutorials/how-to-automate-elixir-phoenix-deployment-with-distillery-and-edeliver-on-ubuntu-16-04

[t0mhaines]

http://dokku.viewdocs.io/dokku/ is an excellent open-source Heroku-style PaaS. - It uses Heroku build scripts under the hood so porting apps from Heroku to Dokku should be reasonably painless.

You host it on your own VPS so it is a bit more involved than Heroku but you have control over your own infrastructure and allows for a transition to container-based solution in the future

For running small applications with CD pipelines its probably one of the best options, only needing a git push in .travis.yml to deploy your application.

[nelsonic]

@th0mas yeah, we used Dokku a couple of years ago to deploy Node.js Apps: See: https://github.com/dwyl/learn-devops/blob/master/nodejs-digital-ocean-centos-dokku.md Agree 100% that it's very good in most usecases. 👍

The reason we decided not to use Dokku at the time for our Phoenix Apps is because WebSockets are not supported: https://github.com/dokku/dokku/issues/3480 😞 And WebSockets (Phoenix Channels) is one of the biggest reasons we addopted Elixir https://github.com/dwyl/learn-elixir/issues/102 If WebSocket support is available now in 2020, happy to reconsider. 💡