Code Tutorial #1: Caliscope (3/3)

Today, it’s the third and final part of my coding tutorial on “The Cube App”. We’ve now finished programming the app itself, so let’s see how we can put our app in production and deploy it easily using containerization.

This article is part of a series of three:

  • Part 1: setting up the app in a local setup and preparing the main elements
  • Part 2: loading images, adding a moves history and an “auto play mode”
  • Part 3 (this article): serving the app in production mode (with a Golang server) and deploying it using Docker

Last time, we finished our Cube App by adding nice images, an undo/redo mechanism and even an auto-play mode that triggers when the user doesn’t touch the screen for a few seconds. We ended up with something like that:

Now that our app is ready, it is time to step up from development mode and prepare it for an actual deployment. At the end of this post, we will have packaged our app so that it can be run on any computer, assuming it has the Docker tool installed. More precisely, in this article, I’ll show how to wrap up the project for production by implementing a small Golang server and creating a docker container for the project.

Setting up a lightweight but production-ready server

Switching to production mode with Go

So far, we’ve used a basic Python development server. But as is written in the http.server documentation, it is “not recommended for production”, mostly for security reasons. There are plenty of ways to code production-compliant servers, and Python is a really nice programming language for web programming:

  • if you’re looking for a scalable and robust framework (for bigger projects than this example), Django might be a good choice: even though it has a somewhat steep learning curve for beginners, it is a very powerful and customizable tool
  • for lots of projects, I use Flask: it is a lightweight and user friendly web lib that lets you build a server in the matter of minutes – however, when you want to step up your game and get an app valid for production, you’ll need to combine it with other technologies such as Gunicorn, uWSGI, Twisted…
  • Bottle is also interesting, but it’s really for super-simple apps (and similarly requires additional steps for production deployment)

I’m a big fan of Python and it’s usually my go-to when I want to throw some things together and stir up the thoughts. However, this time, I’ve decided to switch gears a bit and write the production server in Golang for a change 😉

Just as a quick reminder, Go – or Golang – is a language by Google that has been around since 2009, although it only really started to get noticed in 2015. It is a statically typed and compiled language that resembles C yet provides plenty of “more recent” features other competitors have such as garbage collection, nil values, structural typing… It also focuses on the writing and highly efficient execution of concurrent code which makes it a great choice for intense computing, web development or any kind of high-performance multiprocessing. Plus, the ecosystem around Go has gradually evolved and improved, so there is now, for example, a package manager, more and more libraries to share among coders and an ever-growing list of tutorials. All in all, Golang is worth taking a look at and is nowadays in the stack of many big players (Google, of course, but also Docker, Kubernetes, Netflix, Uber, MongoDB…).

Note: the interest in this tool keeps on increasing and people have written a variety of articles to compare to other programming languages: “Nine reasons to switch from Python to Go” (S. Bajpai), “9 Reasons to Choose Golang for your Next Web Application” (T. Svityk), “How a Company Reduced its number of Server from 30 to 2 Using Golang” (R. Shirsath)…

I already wrote a series of tutorials about the basics of Golang a while ago, so I’d recommend you check it out if you’re new to this programming language – especially since I won’t dive into all the nitty-gritty details here.

Preparing the server

When working in Go, a very basic way to render HTML pages is to use the built-in Go packages and tools to render HTML templates. This is the main idea behind well-established web-templating-centric languages like PHP but also behind more recent tools such as Jinja2 or Django. Here, the templating technology is Mustache.

This combo allows us to write templates in the following form:

[snippet slug=072019_goweb-template2 lang=html]

This template will then be filled by the Go server to replace the {{.}} by an actual variable value (in this case, the entire data that is fed to the template because we used the “base” accessor .).

Our Go web server will thus have 2 objectives:

  • reading data folders to extract and load up assets (images and HTML layouts)
  • handling the serving of these assets to render our HTML filled with variables

Let’s start by setting up a basic HTTP server; thanks to Go, this just requires a few imports of built-in libs and a dozen lines of code:

[snippet slug=072019_goweb-server lang=golang]

You can now save this content at the root of our project as server.go for example. Then, run this code in a terminal (with go run server.go) and go to http://localhost:8000/, you should see a page with “Hello server!” displayed at the top left.

We can make this code a bit more robust AND customizable by using environment variables and error handling:

  • environment variables are global values that are defined for the entire work environment and can be easily accessed in any of your processes – since we’re ultimately going to isolate and finely tweak our environment using containerization (see the second part on Docker), it makes sense to make the most out of this perfectly crafted work environment, right ? 😉
  • error handling is one of the difficulties in Go, but still essential: whenever something goes bad, you should be able to track where the error originated from so you can find and correct it (as quickly as possible… which is sometimes quite a long time!)

We’ll create a custom getEnv function that uses error handling and fallbacks to have a default value for the given environment variable if no real value is passed:

[snippet slug=201205_caliscope-tuto_17 lang=golang]

Now, let’s use it in our web server to load up the port to run the server on.

[snippet slug=201205_caliscope-tuto_17b lang=golang]

There is not much change but now, if we define an environment variable called PORT, its value will automatically be loaded by the Go script to set up the server. If none is defined, thanks to the fallback value, the port will be set to 8000. This is the skeleton of our web server.

Loading up our assets (scripts, stylesheets and images)

To begin with, let’s define our data structure so we can store all the necessary info for our HTML templating. This data will contain two items: first, the name of the static directory we load our images from (so the Javascript file knows where to look); second, the list of all image files in this said directory (so the Javascript file knows what to look for).

So we end up with something like:

type PageData struct {
  StaticDir   string
  Images      []string
}

Let’s also use an environment variable to get the path to our static folder – we’ll define a global var in our Go script to store this value:

var staticPath = getEnv("STATIC_PATH", "static")

By default, the static path is a subdirectory static/ at the root of our project. This directory will contain our current img/ folder but also the style.css and script.js files.

And to wrap things up, here is our little function to browse the static directory and fetch our image file names (note that this method returns an object of the PageData type):

[snippet slug=201205_caliscope-tuto_17c lang=golang]

Adding HTML templating and feeding them our data

First things first, we need to decide how we are going to cut down our HTML file (the index.html file we’ve been cooking up so far) into logical pieces in order to get smaller well-organized HTML templates. On this project, I’ve done the following:

  • layout.html: the entrypoint, i.e. the template that holds all the others and will be the one loaded directly by the server
  • header.html: the content of the header of the app (the title bar)
  • main.html: the 3D scene at the center of the app, with our cube
  • footer.html: a simple block with some credits

All files are stored in a subdirectory called templates/ that is referenced by the TEMPLATES_DIR environment variable. Here are the files after re-templating:

layout.html

[snippet slug=201205_caliscope-tuto_18 lang=html]

header.html

[snippet slug=201205_caliscope-tuto_18b lang=html]

main.html

[snippet slug=201205_caliscope-tuto_18c lang=html]

footer.html

[snippet slug=201205_caliscope-tuto_18d lang=html]

You’ll notice that:

  • the layout.html contains the core of our previous index.html; in particular, it has the stylesheets and scripts imports in its <head> tag and it calls up our Javascript file in its body, with the window.onload function
  • the other templates are very light and only contain HTML
  • we are now using a local version of the hammer.js lib: instead of calling the remote version on the CDN, I’ve downloaded the minified Javascript file and I’ve added it to my static folder, in a vendors/ subfolder
  • we use the PageData object that will be fed to this template in various places: in the imports (to properly reference our static directory) and in the Javascript initialization script
  • we add a super-light wrapper around the start() function and we first called another method initialize() to store the variables sent by the Go server into Javascript variables

This new initialize() function is very basic:

[snippet slug=201205_caliscope-tuto_18e lang=javascript]

This lets us simplify our start() function a bit, while making sure we keep only the right file names (because the way we read the static directory is somewhat crude, so it might also read some non-images):

[snippet slug=201205_caliscope-tuto_18f lang=javascript]

And the final fix to do in our script.js file is to prefix our image paths with the static directory reference:

[snippet slug=201205_caliscope-tuto_18g lang=javascript]

Note: once again, for a more in-depth explanation of Go variables injection in Mustache templated HTML, feel free to take a look at my previous Go tutorial 🙂

Alright! At this point, we’re finally ready to load and feed our HTML templates from our Go server. This is done in the serveIndex() function and replaces our previous super-simple “Hello server!” print:

[snippet slug=201205_caliscope-tuto_19 lang=golang]

Also, we need to tell our Go server to handle our static assets separately: images, stylesheets and images should be sent directly to the client without any additional parsing or templating, so we can use a simple FileServer for that:

[snippet slug=201205_caliscope-tuto_20 lang=golang]

And tadaa! We’re done. If you restart your Go server and go to http://localhost:8000, you should find back our cube app!

The final step of this tutorial is to show you how Docker can help with app deployment by providing a portable, lightweight and yet powerful environment to run the production-ready app in.

Dockerization!

What is Docker and why use it?

Docker is a technology that allows you to package your apps into “containers” to deploy them in a flexible, lightweight, portable and yet scalable way. As is stated in the Docker’s docs, “containers are not new, but their use for easily deploying applications is”. The idea is to create a specific work environment to run your app in by simulating a “lightweight virtual machine”.

Note: specialists might scream at me for saying that Docker creates virtual machines because there is actually a core distinction to make between containers and virtual machines. However, for the level of details of this article, thinking of Docker containers as “small isolated OS-specific environments” looks to me to be valid enough 😉

The two basic notions in Docker are “images” and “containers”. For a real explanation, I recommend you take a peek at Docker’s getting started guide – but in the meantime here is a short summary:

  • containers are processes mostly isolated from the host system that run in a specifically tuned work environment
  • images are “blueprints” for containers: they determine what your environment contains and what tools the containers spawned from this image will have access to

Images are the bare bones of your container’s environment. Since Docker aims at making it easier to deploy and run your apps, it makes sense that they provide a wide, free and well-furnished list of images to start from. To discover the public registry of official Docker images, simply go to the Docker hub and start browsing!

To be honest, Docker is on my top list of “best discoveries” from these past years. I find it just amazing to be able to spawn up a completely custom and yet super precisely prepared environment in only a few seconds! I believe it’s an incredible tool to have in your toolbox as a developer because it really helps you finish up your project and get to the final “it’s running in production” stage, all on your own. (To me, dockerization is a good first step in the DevOps universe because it truly is a glue between the app development and app deployment worlds.)

Building our Docker image

But, of course, using only the images already built on the Docker hub is not enough – to put our app in production, we need to create our own Docker image that will be based on these reference Docker images but will contain more specific files and environment variables than those.

Docker images rely on sequences of “layers”: those are sequential steps that are performed after downloading a base image to gradually tune it to your own needs. Those steps are written in a special file, the Dockerfile: this file can then be given to the docker command-line tool to build our image for the sequence of layers.

In our case, the core technology we require is Golang. Our HTML, CSS and Javascript only need to be copied and served, there is no complex logic with those languages – remember they are directly interpreted by browsers. Therefore only the server truly needs a specific work environment. If we head over to the Docker hub, we will easily find that there is indeed a Docker image for Go. This will be our “base image”.

Let’s then recap the steps that we need to build our Docker image:

  • pull the base Golang image
  • define our environment variables
  • copy all of our project files
  • tell the image that whenever it is “instantiated” as a container, it should start serving the Go server by using our usual command: go run server.go

You’ll see that it’s actually pretty close to the Dockerfile itself (this file is put at the root of the project, next to the server.go):

[snippet slug=201205_caliscope-tuto_21 lang=dockerfile]

Note: since we’re essentially entering “a brand new computer” when we start up our image, we need to create some directories to copy our files in. Similarly, we need to make sure we are in the right folder before executing our final command to run the server, otherwise the Go CLI tool won’t be able to start any server.go file.

To actually build the image, make sure you’re in your project folder in the shell and then run this command:

docker build -t my-cube-app .

This will build the image using the Dockerfile in the current folder – because of the . character at the end – (by default it takes any file named Dockerfile in the given directory) and name this image my-cube-app. Our image is quite small and easy to create so this shouldn’t take too long. You’ll see that Docker will show you its process step by step… each step being one of the layers we defined earlier! When it’s done, to check the image indeed exists, you can use:


gt; docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
my-cube-app                         latest              7ee9c43c642a        56 seconds ago      373MB
golang                              1.14-alpine         3289bf11c284        6 months ago        370MB
Great! We’ve just built a Docker image for our cube app: it has a unique ID, a “tag” (we’ll talk about these in a second) and we see that it’s size is a few MB, just a bit more than the initial “golang” base image we started from.

We can actually run this image to create a Docker container from it with the following command:

docker run -p 8000:8000 my-cube-app

The -p 8000:8000 section in the middle is called a port forward: it allows us to tell the Docker container (which is somewhat isolated from the host system if you recall) that it should “expose” its port 8000; this basically maps the inner port to the port on the external host system, so now going to http://localhost:8000/ will in fact hit the server running inside the Docker container. Hitting the port 8000 on the local system directly forwards the requests to the internal port 8000 in the container we decided to start our server on. Pretty cool, right?

To wrap things up for a basic usage of Docker, lets talk a bit about image sizes and optimization.

Optimizing the image

If you start reading some posts on dockerization, you might quickly see that having the smallest possible image is often considered better. It takes less space in storage, it’s easier to spawn into a container… so the question is: how can you reduce the size of your Docker image?

When you create a new Dockerfile, an important choice is which your base image you’re going to start from. By that, I’m actually talking about what version and what “tag” of the base image you should use.

The version is the standard major.minor.patch format string you get on most libraries – it can be more or less “detailed”; for example, this lets you choose between a Docker image with Python 2 (python:2), Python 3.6 (python:3.6), Python 3.8.6 (python:3.8.6)… Setting a detailed fixed version can be a good way of insuring your image is always valid; otherwise some update in a “global version” like “python:3” might impact your own app later on. To be even more specific, you can even reference the exact SHA256 code of the image you want: this way, you’re absolutely sure it will never change! This is the technique we’ll use here.

The “tag” is a bit more subtle. It is a little suffix that may be added at the end of your image name after dash -. There are plenty articles (for example this one by J. P. Garcia) that teach you the difference between those varieties of the same image but you can remember that:

  • the full official image, with no special tag, is the latest stable release
  • the “stretch”, “buster” and “jessie” images are for different Debian distributions
  • the “slim” image has a reduced size because it doesn’t include some secondary less-used dependencies
  • the “alpine” is even lighter by only bringing in the bare minimum and letting you handle the install of any additional tools

To get a really lightweight image, we can therefore use an “alpine” version of the Golang base image, for example the 1.14-alpine. But as I mentioned before, to make sure it doesn’t go out of date, let’s actually get its SHA256 reference. To do this, first pull the image – yes, we still need to pull it once! – with the following command in a shell:

docker pull golang 1.14-alpine

Now, you can inspect this image’s digest to get its SHA256:

docker images --digests | grep golang

This big string starting with “sha256:” is the one we are going to use in our Dockerfile to reference this exact image ad vitam aeternam. So, we can now update the first line of our Dockerfile:

[snippet slug=201205_caliscope-tuto_21b lang=dockerfile]

And now, here’s the real trick (and a big advantage of using Go over Python for our project): since Go is a compiled language, we don’t truly need to keep all of source files and the Go tool from the base image! Instead, we can benefit from Docker multi-stages build:

  • first, we will use the Golang base image, import our project files and compile the server (at this point, we do need Go, so we do need the Golang base image)
  • then, we will start back from scratch and only copy the compiled server and asset files

Here is the updated Dockerfile with 4 modifications:

  • we name the first stage “builder” so we can refer to it later on
  • we compile the Golang server just before exiting the first stage
  • we add a second stage based on the Docker “scratch” image: a minimal environment with no libs, no tools… (“an explicitly empty image, especially for building images “FROM scratch” “)
  • we copy back all the files we add in our project folder, plus the server we compiled as a binary
  • we set the entrypoint of the second stage (i.e. the final entrypoint of the image) to be this binary – it will automatically be executed when the image is instantiated as a container

[snippet slug=201205_caliscope-tuto_21c lang=dockerfile]

If you rebuild the image now and re-examine our images list, we’ll see that our 363MB image has reduced down to… only 10.4MB!

REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
my-cube-app                         latest              153070db4f0e        16 seconds ago      10.4MB

This already has some impact on a small app like ours, so imagine what it can do for more complex projects 🙂 In my experience, devoting some time to optimizing your Docker images is worthwhile because it makes them more efficient and it teaches you a lot about the inner workings of Docker.

Conclusion & possible improvements

I’ve already mentioned some possible improvements for our app itself last time (like a shadow under the cube or a small fade in at the beginning). Another cool bonus feature would be to add a small “middleware” function to print request and response statuses in our Go server; this way, we could easily spot the correct responses (with 200 HTTP codes) and the incorrect responses (with 4xx or 5xx HTTP codes, as is usual with HTTP status codes). For the ones interested, this can be achieved quite quickly using a Mux Golang’s HTTP server and a set of structs and logging function:

[snippet slug=201205_caliscope-tuto_22 lang=golang]

After this update, the server will log all incoming and outgoing transactions (with the time spent to process requests):

2020/12/06 15:14:28 ==> [GET] /
2020/12/06 15:14:28 <== [GET] / 200 (713.046µs)
2020/12/06 15:14:28 ==> [GET] /static/style.css
2020/12/06 15:14:28 ==> [GET] /static/script.js
2020/12/06 15:14:28 ==> [GET] /static/vendors/hammer.min.js
2020/12/06 15:14:28 <== [GET] /static/style.css 200 (7.777684ms)
2020/12/06 15:14:28 <== [GET] /static/script.js 200 (3.759455ms)
2020/12/06 15:14:28 <== [GET] /static/vendors/hammer.min.js 200 (3.531568ms)
2020/12/06 15:14:28 ==> [GET] /static/vendors/hammer.min.js.map
2020/12/06 15:14:28 <== [GET] /static/vendors/hammer.min.js.map 404 (59.183µs)
2020/12/06 15:14:28 ==> [GET] /static/img/aigle.jpg
2020/12/06 15:14:28 <== [GET] /static/img/aigle.jpg 200 (1.027094ms)
2020/12/06 15:14:28 ==> [GET] /static/img/chien.jpg

There are probably plenty more ideas of improvements on this app: I’d be happy to read yours in the comments section below or on my LinkedIn! I hope you’ve enjoyed this tutorial and found it useful. And if you have any other suggestions of code tutorials I could do, feel free to reach out 🙂

References
  1. Python HTTP server’s documentation: https://docs.python.org/3/library/http.server.html#module-http.server
  2. Django’s documentation: https://www.djangoproject.com/
  3. Flask’s documentation: https://flask.palletsprojects.com/en/1.1.x/
  4. Bottle’s documentation: http://bottlepy.org/docs/dev/
  5. The Go Programming Language: https://golang.org/
  6. Docker’s documentation: https://www.docker.com/
  7. Docker Hub’s website: https://hub.docker.com/
  8. S. Bajpai, “Nine reasons to switch from Python to Go” (https://medium.com/datadriveninvestor/nine-reasons-to-switch-from-python-to-go-f1b0cd746974), February 2020. [Online; last access 07-December-2020].
  9. T. Svityk, “9 Reasons to Choose Golang for your Next Web Application” (https://letzgro.net/9-reasons-to-choose-golang-for-your-next-web-application/), January 2016. [Online; last access 07-December-2020].
  10. R. Shirsath, “How a Company Reduced its number of Server from 30 to 2 Using Golang” (https://reemishirsath.medium.com/how-a-company-reduced-its-number-of-server-from-30-to-2-using-golang-16ef84fd2ddb), July 2020. [Online; last access 07-December-2020].
  11. J. P. Garcia, “Alpine, Slim, Stretch, Buster, Jessie, Bullseye — What are the Differences in Docker Images?” (https://medium.com/swlh/alpine-slim-stretch-buster-jessie-bullseye-bookworm-what-are-the-differences-in-docker-62171ed4531d), July 2020. [Online; last access 07-December-2020].
  12. Golang By Example, “Basic HTTP Server Implementation Using Go (Golang)” (https://golangbyexample.com/basic-http-server-go/#Mux), July 2020. [Online; last access 07-December-2020].
  13. I. Wikimedia Foundation, “Web template system” (https://en.wikipedia.org/wiki/Web_template_system), November 2020. [Online; last access 07-December-2020].
  14. I. Wikimedia Foundation, “Mustache (template system)” (https://en.wikipedia.org/wiki/Mustache_(template_system)), July 2020. [Online; last access 07-December-2020].
  15. I. Wikimedia Foundation, “List of HTTP status codes” (https://en.wikipedia.org/wiki/List_of_HTTP_status_codes), November 2020. [Online; last access 07-December-2020].

Leave a Reply

Your email address will not be published. Required fields are marked *