Golang & Web programming: “The World Aggregator” (2)

To finish this mini-series on “The World Aggregator”, let’s see how to create some nice pages to display our data. We will use some user-friendly JS libraries like D3.js and DataTables to easily render the information we gathered.


Last time, we created our first HTML templating file and managed to show the data we had fetched in it with tables. While this is an ok first step (and we will keep this page in the final version, by the way), it’s clearly not the only way to display our datasets: given we are working with geographical data, wouldn’t it be nicer to use maps?

In this article, we are going to create a new page to show our datasets with a world map and some additional “home” and “about” pages.

Warming up: using static files, preparing more HTML templates and making a layout

Since we are going to have multiple pages, we also want to add a little navigation bar at the top that will be shared between all of them. To do that, let’s create a layout template: this new file will be a template for our templates!

The idea is to have some global HTML with small blocks that can be further filled with a HTML template, that will in its turn be filled with our data. Here is a little diagram to sum up what pages we will create and how they relate to the rest:

We will store all these HTML pages in a templates/ subfolder in our project folder.

First, let’s make our layout.html. Adding a block in an HTML template with Go’s templating syntax is quite easy: we just need to use the template keyword (once again, if you want more info, check out Go’s templating package’s doc):

[snippet slug=072918_goweb-layout lang=html]

Note that:

  • we offer a flexible layout where you can add both content in the body and in the head of your pages, because we will need to import some libraries depending on the page, so it’s better to leave it to the page themselves than to the overall layout template
  • we pass down our data from the template to the body block with the . character

Let’s also name our layout so that we can load it easily on the server side afterwards, with the define keyword enclosing our entire layout.html code (first and last lines):

[snippet slug=072918_goweb-layout2 lang=html]

To finish off this warming up on the front-end, we can

  • simplify our previous template.html and rename it to tables.html:

[snippet slug=072918_goweb-layout3 lang=html]

  • create a basic maps.html page that makes use of the layout too:

[snippet slug=072918_goweb-layout4 lang=html]

On the server side, we only need to change a few things in our imports and at the end of our serverIndex() function to load our new layout:

[snippet slug=072918_goweb-layout5 lang=golang]

Now, if you go to the http://localhost:8000/tables.html or http://localhost:8000/maps.html URLs, you will see two pages that share the same title but have different contents. But if you try to go to a URL that we haven’t create a page for (like the root http://localhost:8000/ or a random page such as http://localhost:8000/myPage.html) then you’re gonna have an error. And not a pretty error, as in a “page not found” display, but a crash of your browser because the server doesn’t know what to return for those URLs.

To handle this, we can a few more things to our server code. These new lines will deal with errors upon HTML templating file opening or incorrect URLs loading (you will still get an error, but at least it will be “gracefully” handled by the browser and be an actual 404 not found error):

[snippet slug=072918_goweb-layout6 lang=golang]

And finally, let’s modify our main() function a bit so that we can load static files (such as interesting JS or CSS files):

[snippet slug=072918_goweb-layout7 lang=golang]

This way, whatever files we put in the static/ subfolder of our project folder, we will be able to access them with usual HTML imports. For example, we can import the styles.css file (that is empty for now) in our layout.html so that it is used in all the pages:

[snippet slug=072918_goweb-layout8 lang=html]

D3.js: a JS library to show your data!

D3.js (for Data-Driven Documents) is a JS library meant to create neat data visualizations in HTML, CSS and SVG. Its focus is on the data: the data you give determine the HTML elements that are created and any change in the data triggers a transformation of the page. D3’s core scripts was written by Mike Bostock but, since then, lots of modules and addons have been offered by the community and integrate seamlessly with the base.

The library is optimized for big data; it only updates what needs to be updated, allows us to easily select elements with specific queries, has user-friendly interfaces for CSS transitions… I encourage you to check out the gallery on the website to see some really cool stuff that was made with D3!

Here is an example of an interactive rendering made with the library (the code is taken from here):

“GeoJSON”, the file type for geographical data

When you work with geographical data, one cool file format is the GeoJSON format. It is an open standard format derived from the common JSON that is specifically designed to link various datasets to given locations.

Here is an example of GeoJSON file (from the Wikipedia page):

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [102.0, 0.5]
      },
      "properties": {
        "prop0": "value0"
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [102.0, 0.0], [103.0, 1.0], [104.0, 0.0], [105.0, 1.0]
        ]
      },
      "properties": {
        "prop0": "value0",
        "prop1": 0.0
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [100.0, 0.0], [101.0, 0.0], [101.0, 1.0],
            [100.0, 1.0], [100.0, 0.0]
          ]
        ]
      },
      "properties": {
        "prop0": "value0",
        "prop1": { "this": "that" }
      }
    }
  ]
}

As you can see, you can put in various types of geometries (points, lines, polygons…) and associate values with each.

The, the TopoJSON format is an extension that also encodes some topology.

And finally, this nice Github repository by Mark Markoh called Datamaps uses D3.js to make use of TopoJSON files and display great maps easily with our own dataset. It is basically a user-friendly wrapper around D3.js plots and TopoJSON downloads with which you just need to feed in:

  • the reference of JS file from the repo depending on the map you want to show
  • the specific values you want to map if you have some
  • some settings and display options to tell D3 how to actually show your data on the map

If we look at the “Getting Started” example in the repo, we can easily create an element in our page that displays a world map:

The TopoJSON file used by the Datamaps repo links every country to the right visual element by referencing its 3-letters ISO code. So, if we grab the countries ISO3 codes of some countries, the datamaps.world.min.js file from the Github repo, that we assign random data to our map and that we also add some coloring options to our object, we start to get some visual cues that are directly dependent on our data!

[snippet slug=072918_goweb-datamaps-example lang=html]

Be careful: you’ll need to change the $YOUR_PATH_TO_THE_JS to the right thing so that it points to the location where you stored the datamaps.world.min.js file downloaded from the Github repo.

This small code generates the following map:

Pretty nice, right? Now, let’s see how we can use that in our previous HTML maps.html template file to display our world datasets a bit more prettily than just with tables…

Preparing the maps.html page

For now, let’s just get back some info from our server and display in basic divs, without any map. Just like we did before, we simply need to add some ranges and other variables in our body block. We will also add a little dropdown menu to select the dataset we want to display and use a JS function to update the UI accordingly:

[snippet slug=072918_goweb-ui lang=html]

This code does several things:

  • first we create some HTML elements for each of the datasets in the data we get from the server: an option in the selection menu and a div with just the name of the dataset
  • then, we add a script tag to execute after the HTML page has loaded (this is why it is at the end of the body and not in the headers) that stores the name of all the datasets, hides all but one at first and then handles the UI change whenever the user clicks on another option in the selection menu

Note: once again, this article is not an HTML or JS tutorial so I won’t dive in the details of how to program in these languages. But there are plenty of tutorials around if want to learn more!

Here is a piece of CSS to go with it and to put in the styles.css file we included earlier directly in the layout:

[snippet slug=072918_goweb-ui2 lang=css]

Now, if you refresh the maps.html page, you should see a super basic page with the website title (automatically created by the overall layout), the little dropdown menu with entries for each of the datasets your server loads and the name of the first dataset in the middle of the screen. Then, if you select another dataset, the name should update.

This is not very impressive yet, but we actually already have most of the logic in place for our final version…

Using D3.js and Datamaps in our HTML page

Alright, let’s see how we can use our little knowledge of D3 and Datamaps to fill our maps.html page!

This will require us to do a few things:

  • import some JS files (mostly D3 and some of its packages) in the headers
  • add containers in the HTML in which to display maps
  • add some JS code to truly create the Datamap objects and fill them the data from the server

We will do that one by one. First, the imports:

[snippet slug=072918_goweb-ui3 lang=html]

You notice that we load the D3.js library, some other util packages for color maps, the Datamap worldmap JS reference and a custom file called iso3_codes.js that holds the “name to ISO3 code” mapping we discussed before. If you want to download it, it is available over here (I’ve added some extra pairs because some of my data sources have different country names, so if you want to use the same data URLs as me, I suggest you use this file too).

Now, we can prepare the containers for the maps; it’s just one more line, we will create one for each dataset – our previous handleSelect() function will take care of hiding the ones we don’t want to see:

[snippet slug=072918_goweb-ui4 lang=html]

And to finish setting everything up, here is the updated JS code to put in the script tag at the end of our body:

[snippet slug=072918_goweb-ui5 lang=html]

This is a bit more complex, so let me briefly talk you through it:

  • at the top, we declare a new variable called COLORMAPS: this object will associate each dataset name to a given D3 colormap so that, depending on the dataset we show, the plot uses a different heatmap (to get a full list of available colormaps, take a look here)
  • then when we load the window, we go through all our datasets and create a Datamap for each; we prepare a data object beforehand by going through each entry in the dataset we are working on

If you refresh the page, you should see a world map with colors… but everything is colored the same! What’s going on?!

Well, the issue is that like lots of color scales, D3’s is normalized so it expects a floating input value between 0 and 1. We are gonna need to do some additional preprocess on the server side to send this normalized value as well. And, while we’re at it, let’s add some information to all our structs to record things like the unit that is used for the values of a particular dataset or a specific postprocessing function to apply to the values (to deal with really large value intervals).

Refining the data processing (server-side)

We know that adding the unit for the values is quite straight-forward: basically, we just need to scrape a bit more data from the web page and to have some additional fields in our structs.

Postprocessing functions, on the other hand, require a bit more preparation. The idea of those is that, for some datasets, like the GDP or the population per country, values vary a lot. And if we plot our heatmaps for these wide value intervals, they will be hard to interpret because colors will all be crunched to one side. For those datasets, it would be better to use the logarithmic function for example to make up for those large differences in order. So, let’s define a PostprocessingFunction type and two basic postprocessing functions:

[snippet slug=072918_goweb-refine lang=golang]

We are now going to update our Go structures with various fields (as you can see, quite repetitive from one struct to another but this is so that the data can flow down from the parsed content up to the CountryDatasets items we send to the client):

[snippet slug=072918_goweb-refine2 lang=abap]

As we have some new fields in our SearchContext struct, we’ll also need to update our map of data-fetching URLs:

[snippet slug=072918_goweb-refine3 lang=golang]

In our getData() function, we just need to change 2 lines: first, when we create a CountryData object, we need to initialize its NormValue field (for now, it’s just set to zero; we’ll compute the value later on), and then in the return statement we must add some values to our GatheredData object:

[snippet slug=072918_goweb-refine4 lang=golang]

And finally in the serveIndex() function, we’ll upgrade our data processing phase to apply our new postprocessing functions, get a normalized value in addition to the real one and further propagate the dataset unit:

[snippet slug=072918_goweb-refine5 lang=golang]

Now, on the client side, in our JS, when we compute the color by passing the value to the D3 colormap, we simply have to replace the absolute value by the normalized one:

[snippet slug=072918_goweb-refine6 lang=javascript]

And, in the title above the graph, we can add the dataset unit:

[snippet slug=072918_goweb-refine7 lang=html]

We can put a super simple style on our map class:

[snippet slug=072918_goweb-refine8 lang=css]

Voilà! We’re getting a worldmap with our datasets displayed as heatmaps and we can switch from one to another just by selecting a dataset name in the dropdown menu on the left.

Last improvements

Adding colorbars to better read the values of the dataset

Even though we have a nice color mapping of our values, it’s kind of hard to know what interval the values live in. To make this more apparent, let’s add a colorbar on the right of the map with an indication of the minimal and maximal values in the displayed dataset.

To do this, we’re gonna use a basic CSS gradient in a div and set its extreme colors to the two extreme values of the colormap used by the dataset. First, let’s create the HTML elements in our inner range loop (and rearrange the CSS classes a bit so that it’s easier to apply styling afterwards):

[snippet slug=072918_goweb-improvements lang=html]

Now, we can update our JS tag in the file with some additional lines:

[snippet slug=072918_goweb-improvements2 lang=html]

And last but not least: the CSS! We have a few new classes to style:

[snippet slug=072918_goweb-improvements3 lang=css]

Adding an App Bar to navigate the various pages

To better switch between pages, we’re going to replace our plain title by an app bar. This navigation bar will display a small icon for our app, the title and four buttons to the homepage, the “tables” page, the “maps” page and the about page.

Note: of course, if you don’t want to make a homepage and/or about page, you don’t need to have all these links and can just keep the ones for the “tables” and “maps” pages.

In terms of HTML, it is quite straight-forward:

[snippet slug=072918_goweb-improvements4 lang=html]

Actually, most of it happens in the CSS styling:

[snippet slug=072918_goweb-improvements5 lang=css]

This makes a nice and simple app bar like this:

Tuning the “tables” page

For now, our tables.html is pretty simple: we just create a big table with each of our datasets. The country are not even sorted alphabetically and you have to scroll a lot to read everything.

To make something a bit more easy to eyeball, we can use a nice jQuery plug-in called DataTables that transforms our tables to add pagination, filtering, search fields…

To use it, like before, we need to import a few things and add a script tag at the end of our body to grab our tables and apply the DataTables’ scripts on them:

[snippet slug=072918_goweb-improvements6 lang=html]

That is way better! Now, we can actually search through this and see all of our datasets in a glance. Of course, this page could be further refined with some formatting of the big numbers for example; but this I leave to you and we’re going to stop there for the tables.html file.

Adding some URLs and boosting the data parsing

In the last article, I mentioned that I had some other URLs than the two shown in the code examples. If you want to use those, feel free to take a look at the downloadable code given at the end of this post.

The Go server also contains a few new extraction functions and the front-end some additional colormaps for these new URLs.

Adding other pages (a homepage and an about page)

Other than that, you can add a homepage and an about page with some static info if you want: just create new classic HTML files in the templates/ folder and add in whatever you want!

Here are some examples of very basic page screenshots:

Note: if you want, the HTML code for those pages (and the required assets) are all provided in the downloadable archive at the end of this article.

Further developments

On the front-end side, the app bar could be improved by indicating which page is currently active, overall style could be refined by playing around with the colors and of course we should put more content in the homepage and about page.

On the server side, the data gathering is not yet perfect: some URLs have specific country names that are not properly remapped to the D3.js ISO3 codes and therefore the information is not correctly propagated to the actual map render. We can also notice that the tables don’t all have the same number of entries, which indicates that it could be interesting to do some kind of renormalization on the data before sending it to the client.

Finally, something that is really not ideal with this current version is the fact that whenever you load either the “maps” or the “tables” page, the entire processing chain is triggered and the server goes back to scrapping the Internet to fetch data from all of our sources. To improve this, we could implement a caching system that would keep the fetched data stored for a while so that it can be accessed instantaneously. However, this would lead to other questions and issues such as: how long should the cache be? How massive is the amount of data we need to store? Where and how should we store it?

In conclusion…

To conclude, there are plenty of things we could improve or add but we still have an okay, working version of our app! We’ve written an HTTP server, various data structures and a concurrent data fetching process in Golang and then sent these datasets to a front-end that makes use of HTML, CSS and JavaScript (in particular the D3.js library and the DataTables plugin) to display the info in a nice and pretty way.

This small series was really short (even if the bits of code were longer than usual!) and to be honest, I’ve just mostly scratched the surface of what we can do with those tools. Go is clearly a very interesting language (and I’m particularly fond of this desire its creators have of keeping the language unique and special by refusing to implement some features found in other programming languages); there is many projects that could benefit from core concepts of Golang (like the way it handles concurrency) and it is still easy enough to write when you’ve gotten used to the syntax (conversely, I don’t think anyone has ever managed to write assembly code fluently even after years of training…). D3.js has lots of cool visualization ideas in its gallery and DataTables is a really neat quick tool for table rendering (its example page shows you a palette of useful properties).

Here is the archive with the entire code for this project.

Final note: I hope that you enjoyed this series and that you discovered interesting basics on those technologies if you didn’t know about them before. By the way, don’t hesitate to leave a comment to tell me if you’d like more “hands-on tutorial” series like this one, or if you prefer the usual articles where I just broadly introduce a topic with a few examples!

REFERENCES
  1. Golang’s website: https://golang.org/
  2. D3.js’ website: https://d3js.org/
  3. D3’s “d3-scale-chromatic” addon: https://github.com/d3/d3-scale-chromatic
  4. Datamaps’ Github: https://github.com/markmarkoh/datamaps
  5. DataTables’ website: https://datatables.net/
  6. sentdex, “Go Language Programming Practical Basics Tutorial” (https://www.youtube.com/playlist?list=PLQVvvaa0QuDeF3hP0wQoSxpkqgRcgxMqX), Nov. 2017
  7. A. Edwards, “Serving Static Sites with Go” (https://www.alexedwards.net/blog/serving-static-sites-with-go), Jan. 2017
  8. I. Wikimedia Foundation, “GeoJSON” (https://en.wikipedia.org/wiki/GeoJSON), July 2019. [Online; last access 5-August-2019].

[sc name=”d3-lib”]
[sc name=”topojson-lib”]
[sc name=”datamaps-worldmap”]
[sc name=”d3-example”]
[sc name=”datamaps-getting-started”]
[sc name=”datamaps-data-example”]

Leave a Reply

Your email address will not be published. Required fields are marked *