Golang & Web programming: “The World Aggregator” (0)

Today, I start a little series of 3 articles on my first steps in the world of Golang, the programming language developed by Google. I’ve recently started to look into this and decided to apply it to a small web application project: “The World Aggregator”.

What is Go?

Golang, or simply “Go”, is a programming language developed by Google that has had ups and downs these past few years (it was named Programming Language of the Year by the TIOBE Programming Community Index when it was first created, then dropped in the rankings until 2015 but has been ranked amongst the best languages since then). It was originally designed by Robert Griesemer, Rob Pike, and Ken Thompson in 2009. Its goal was to help Google deal with their large codebase and all the multiprocessing requirements we have today.

Go’s mascot: the Gopher!

At first glance, it is a very interesting although surprising language. From a syntax point of view, it resembles C or C++ a lot, for example with static typing and curly braces. Also, it is very efficient, like those languages. On the other hand, it has garbage collection and is specifically meant for concurrent programming. While the ecosystem is not yet as developed as for other languages (in particular Python, which now has thousands and thousands of available libraries thanks to the community), the Go team has recently introduced a package manager that facilitates the import and usage of external packages that extend the initial functionalities of the language.

Many people have stated that Go is quite easy to learn, even if it still in development and a bit “rough around the edges” (Dave Astels). Go is often complimented for simplifying C++ syntax and handling concurrency better than most languages; but it has also been criticized for how it handles errors, its “nil” value that can lead to harder base cases management, some unsafe type conversions, its lack of function overloading and some syntactic strangeness (such as not allowing an open brace alone on its line).

I’ll dive into more details about the language and how it compares to C or Python in the next article; but to give you an idea of how powerful it can be, just remember that it is used by big players like Google (of course!), Docker, Kubernetes, Netflix, Uber, MongoDB, Ethereum…

“The World Aggregator” project

All in all, it was a new adventure for me and I was keen on discovering a bit more about this new language. But the question remained: what project could I tackle with this? What is Go really good at?

A typical Go starting project is a small web scraping application, in which a server written in Go both scraps the Internet for relevant information, processes it and feeds it to a front-end. (Note: if you want a little reminder on web-scraping, I wrote an article a while ago about web-scraping with the Python Scrapy library). Why is that? Mostly because:

  • the core strength of Go is concurrency which makes it ideal for web-scraping (where, by definition, you’re going to crawl through various web pages in parallel and extracts information from all of them to gather them all in one place)
  • the basic libraries offered with the Go installation include some HTTP serving and HTML templating stuff that allows us to very easily create a web server and an HTML/CSS front-end
  • there now exists some Go packages for web content parsing (e.g.: here, we will use the XPath tool to extract data from the web pages we crawl)
  • you get to manipulate Go’s data structures like maps and slices (that more or less correspond to dictionaries and lists in Python)

So, I decided to create a small and simple application called “The World Aggregator”: this website will visit some arbitrary urls, get world-wide metrics and then display them either as tables or as maps.

Note: this project is inspired by this YouTube tutorial by sentdex and this brief article by Alex Edwards.

Here is a small screenshot of a page of the app displaying the world GDP per country with a heatmap:

A screenshot of “The World Aggregator”, the small web application we will talk about in this series of articles

As you can see, it is nothing fancy: the style is basic and there is not a lot of pages to browse. Still, this will be a nice starting example to discuss the base concepts of Go, and also to talk about how to render tables or maps in HTML and JavaScript.

Project’s goals and organisation

My goals with this project were:

  • to learn and discover Go with a hands-on project
  • to do some easy web-scraping and data parsing/processing
  • to do some data visualization with maps in HTML and JavaScript

The series will contain 3 articles:

  1. An overview: this is the article you are reading that just gives an introduction to the project and presents the main goals
  2. Writing the Go server: in the second article I will talk about my basic understanding of Go and how I coded up the server side of this application to scrap the web, process the information and handle the HTTP serving
  3. Showing tables and maps in HTML and JavaScript: in the third and last article, I will talk a bit about nice visualization tools in JavaScript and HTML like d3.js that lets us plot and color maps without too much work

I hope you’ll like this series – if you’re interested in the project, make sure to check out the article I’ll post next week on writing the server side in Go!

  1. Golang’s website: https://golang.org/
  2. d3.js’ website: https://d3js.org/
  3. sentdex, “Go Language Programming Practical Basics Tutorial” (https://www.youtube.com/playlist?list=PLQVvvaa0QuDeF3hP0wQoSxpkqgRcgxMqX), Nov. 2017
  4. A. Edwards, “Serving Static Sites with Go” (https://www.alexedwards.net/blog/serving-static-sites-with-go), Jan. 2017

Leave a Reply

Your email address will not be published.