6 min read

2 weeks of Elixir/Phoenix/Docker Hell

Hole
Photo by Valentin Lacoste / Unsplash

I'll let you in on a secret – it all ends well :)

Some time ago my team decided (well let's be honest, it's a very small team, and I was a strong advocate) to start preparing for a new product launch – some time next summer probably.

We anticipate fairly high traffic volume and above all we are designing the product for providing updates to the client UI in realtime (or as close to realtime as everything else will allow us to). I have longed for a project to wet my feet in Elixir/Phoenix for quite some time; now was the chance!

Docker is a wonderful concept and coming from Ruby/Rails with Capistrano with no immediate swap-in in sight I reckoned Docker (and perhaps later on, if everything will take off like in all the fairy-tails, Kubernetes) would be a manageable solution. I knew Dokku from a previous project and decided to let that tool aid us taking our first baby-steps.

Enough name-dropping – why "Hell"?

Ok – I'll skip the rest of the introduction and cut to the chase;

Project Initialization

I will not dwell on this - it's been said at least a hundred times by now:

mix phx.new fish --database=mysql

I've been an almost evangelical MaterializeCSS fanboy for at least 6 years now but watching Nicole & Mike's excellent tutorials made me decide to go with TailwindCSS for this one! Praying that I will not regret it!

Mike has a great tutorial on adding TailwindCSS so go there but be careful! It's quality content and you just might hang around for a while ;)

I've grown used to MySQL and MariaDB over the past 20 odd years so I really needed to go down that road – as it turned out two great resources (if you will start off easy with a DB in a container, see Chris Chuck's post on Medium, and if you like me have a dedicated DB server straighten your back against Etel Sverdlov's post on the DigitalOcean Community)  had my back on that!

The product will be for a wide variety of users and thus we probably will need to support a number of ways to authorize guessing that going with OAuth will hurt the least?! To that end Pow and Pow Assent chips in!

And all of a sudden 16 days ago (according to my git log) I was able to do my first meaningful

mix release

Deployment

Up until this moment most of the work were "reruns" – I'd played around with Elixir and Phoenix a number of times albeit just for the fun it. So the entire drill was down to what came next!

Local

One of the (many) great things about Docker is that you can setup almost everything locally - building an entire systems center on your personal computer (I'm told Windows actually are getting it too).

I added a Dockerfile (wait for it - this is part of Hell) and a build script to cut down on the typing

#!/bin/sh
# ./build.sh

# Remove old releases
rm -rf _build/prod/rel/*

# Build the image
docker build --rm -t fish -f Dockerfile .

With that one completed I hurried on with 

chmod +x ./build.sh
$ ./build.sh
...
Removing intermediate container 86c58a1d3277
 ---> 527f66f825ed
Successfully built 527f66f825ed
Successfully tagged fish:latest

and found myself moving on to adding the docker-compose.yml file.

# ./docker-compose.yml
#
version: "3"
services:
  web:
    image: fish
    environment:
      - MIX_ENV=prod
      - APP_PORT=4000
      - POOL_SIZE=10
      - COOL_TEXT='HELLO'
      - SECRET_KEY_BASE="K8(....+WGYG9Hy5CGRBxUZR9gUdycueITq+BZOwv20czu"
      - DBUSER=mysql
      - DBPWRD=aDifficultpASSword2Guess!
      - DBNAME=fish_prod
      - DBHOST=1.2.3.4
      - URLHOST=http://localhost
    volumes:
      - .:/opt/app
    ports:
      - "4000:4000"

I turned up the volume on Jung rested my feet on the table and hit

docker-compose up

When is Hell coming already? You say?

Right about now in fact :)

Server

Bathing in my success I quickly ran through the recipe to installing dokku and added an app on my docker enabled server with

dokku apps:create stage2.speicher.ltd

then back on my mac I continued with

git remote add dokku dokku@docker1.alco.dk:stage2.speicher.ltd
git push dokku master

and I finished the recipe on the server with

dokku config:set --no-restart \
      stage2.speicher.ltd DOKKU_LETSENCRYPT_EMAIL=walther@alco.dk
dokku letsencrypt stage2.speicher.ltd

only to quickly having my feet swept of not only the table but the entire office floor when I browsed https://stage2.speicher.ltd - dang, something was way off!

Where are all "my" css rules? Looking at the app-5678987654rthuytr34567.css I was quickly able to realize that right about 50% of the file was missing! Where did it go?

What followed next was 2 weeks of fiddling, hacking, googling, and asking all sorts of stupid questions on Slack Channels, ElixirForum and whom else I was able to identify as "in-the-know".

From Heaven to Hell – and back, luckily!

With my options close to exhausted this very morning I got up to a message from @savant on the GliderLabs#dokku channel which steered me in a new direction!

My Dockerfile was working locally – that was my conviction at least! So, I uploaded it to hub.docker.com after having created a new repo there following the recipe for doing so, and pulled it down on my server with a somewhat slightly different Dockerfile and docker-compose.yml combo

# Dockerfile on docker1.alco.dk
FROM wdiechmann/fish:fish2


ENV NODE_ENV=prod \
  MIX_ENV=prod \
  APP_PORT=5000 \
  POOL_SIZE=10 \
  COOL_TEXT='tjuhej' \
  SECRET_KEY_BASE=æj...gjkhndmnfalkjhdsfgoiusdr98765dfdde2sgdx \
  DBUSER='mysql' \
  DBPWRD='pASSwordsRaMess!' \
  DBNAME='fish_prod' \
  DBHOST="10.11.12.13" \
  URLHOST='localhost' 

WORKDIR /app

CMD ["/app/_build/prod/rel/fish/bin/fish", "start"]


# docker-compose.yml
version: "3"
services:
  web:
    image: fish2
    # command: mix phx.server
    environment:
      - MIX_ENV=prod
      - APP_PORT=4000
      - POOL_SIZE=10
      - COOL_TEXT='HELLO'
      - SECRET_KEY_BASE="y...+WGYG9Hy5CGRBxUZR9gUdycueITq+BZOwv20czu"
      - DBUSER=mysql
      - DBPWRD=Really?WhatDidYouThink=)
      - DBNAME=fish_prod
      - DBHOST=0.0.0.7
      - URLHOST=http://localhost
    volumes:
      - .:/opt/app
    ports:
      - "5000:4000"

Two things happened!

1) My stage2.speicher.ltd did not improve - not one single bit!

2) I did a recheck locally - following advice from emoragraf on the Elixir Forum - and that shattered my entire world! My local build was somehow wrong too!

That forced me back to the drawing table and made me come up with this recipe for deploying with dokku to one-server-setups outside the mainstream cloud providers where you're left with a "ssh-shell"

  1. build your app locally
  2. create a Dockerfile and docker-compose.yml file (see below)
  3. add your app with dokku
  4. push your master to dokku

Dokku will switch to your new deployment on-the-fly meaning that you should see next to no downtime – at least not pertaining to the deployment process itself, but sadly dokku may not relieve you of your responsibility to delivering functioning apps - I'd take that product any given Sunday ;) 

# .dockerignore
.git
_build
deps
assets/node_modules
test
priv/static


# Dockerfile
FROM bitwalker/alpine-elixir-phoenix:latest
ARG mix_env=prod
ARG hex_mirror_url=https://repo.hex.pm

# Set envs
ENV NODE_ENV=${mix_env} \
  MIX_ENV=${mix_env} \
  APP_PORT=5000 \
  POOL_SIZE=10 \
  COOL_TEXT='tjuhej' \
  SECRET_KEY_BASE=dyhnmj...æ.æ-æp098uuikuygyhgfdw2123esdr5tghy78uhjki \
  DBUSER='xxx' \
  DBPWRD='xxx' \
  DBNAME='fish_prod' \
  DBHOST="xxx" \
  URLHOST='xxx' \
  HEX_MIRROR_URL=${hex_mirror_url} 

WORKDIR /app
COPY . .
RUN mix do deps.get, deps.compile
RUN npm install --prefix assets && \
  npm rebuild node-sass --prefix assets && \
  npm --prefix ./assets ci --progress=false --no-audit --loglevel=error

RUN npm run --prefix ./assets deploy
RUN mix phx.digest
RUN mix do compile, release
RUN chown -R nobody:nobody /app
USER nobody:nobody
CMD ["/app/_build/prod/rel/fish/bin/fish", "start"]


# docker-compose.yml
version: "3"
services:
  web:
    image: fish
    # command: mix phx.server
    environment:
      - MIX_ENV=prod
      - APP_PORT=4000
      - POOL_SIZE=10
      - COOL_TEXT='HELLO'
      - SECRET_KEY_BASE="y9Tb...Hy5CGRBxUZR9gUdycueITq+BZOwv20czu"
      - DBUSER=yourUser
      - DBPWRD=andYourPassWord!
      - DBNAME=fish_prod
      - DBHOST=1.3.5.7
      - URLHOST=http://localhost
    volumes:
      - .:/opt/app
    ports:
      - "4000:4000"

Conclusion

My respect for Chris McCord's work on Phoenix is way to big for me to point any fingers at his Dockerfile example suffice to say it did not cut my mustard – or even more specifically: if you clone this repo and deploy to a 'private' VM appliance infused with Docker and Dokku, like you can shop with a number of hosting companies I'll wage it just works :)

Epilog

Just got a rather lengthy rebut from https://elixirforum.com/u/otijhuis and then https://elixirforum.com/u/praveenperera chipped in too, with a very detailed and thoroughly built Dockerfile: https://gist.github.com/praveenperera/ff0feb71c6804f66aeef7b37e8b8fbef

But in any case - and I have to point my finger, apparently not at Chris McCord but Okke Tijhuis - the Dockerfile in the Phoenix documentation is perfectly valid, and a lot better and more concise than mine, only it did not help me - sorry Okke ...

Walther Højgaard Diechmann

Denmark