What’s coming in TeleIRC v2.0.0? 🔗

TeleIRC v2.0.0 is a complete rewrite of TeleIRC. The team is migrating the code base from NodeJS to Go. In September 2019, the team began scoping the requirements and how to approach this large task. TeleIRC v2.0.0 does not add new features, but aims to have feature parity with the v1.x.x version of TeleIRC.

You might be asking, why bother with a total rewrite? What does this actually accomplish for the project? To answer this question, some historical context is needed!

TeleIRC v1.0.0 was an experiment. 🔗

TeleIRC v1.0.0 was originally created and released in September 2016 by RIT alum Mark Repka. Mark created TeleIRC as a cool project for the RIT Linux Users Group (RITlug) when he was a student and vice president of RITlug. The project was written in hackathon spirit: to prove that something that was not yet common wasn’t that hard to do.

Fast forward to today: TeleIRC ends up being pretty popular! As do chat bridges (Matterbridge, Matrix/Riot, etc.) as a whole. The Fedora Project is one of our largest users, with a dedicated Special Interest Group to manage the bots. The LibreOffice community is another one of our biggest users. Several international communities also adopted TeleIRC to make their chat rooms more accessible to a new generation of open source fans. Some example users are Linux and BSD user groups and hackerspaces in Argentina, Albania, and across Asia. You can see the full list of TeleIRC users for yourself.

TeleIRC has grown in a way we never thought it would. Which is awesome! But the project was not originally designed to grow or scale the way it has. Additionally, by being at a university, contributors come and go as students graduate and move on to industry. We also have to think about how to maintain TeleIRC beyond the typical student life-cycle common in the academic world.

Let’s approach TeleIRC v2.0.0 as engineers. 🔗

A full rewrite allows us to fully leverage our knowledge as software engineers. In 2020, we know TeleIRC has a large user community and is an important part of how many open source communities communicate. We also know that breaking code into smaller, more modular pieces makes it easier to maintain and bring in new contributors. A full rewrite allows us to apply the lessons the team has learned over the years, in a way that incremental feature releases does not allow.

A few areas are in clear focus for the TeleIRC v2.0.0 rewrite:

1. Write clean, simple code that is easy to understand
2. Test the code so it is easy to tell when things are working and when they aren’t
3. Think about how to bring in new contributors to continue the project in the future

But maybe you are also asking, why the jump to Go?

A Go rewrite distinguishes our project. 🔗

When Mark and I launched the project in 2016, we didn’t look around to see if anything else like RITlug’s TeleIRC already existed. Turns out, there was another NodeJS project with the same name. Skip forward a few years, and there are also projects like Matterbridge, pytgbridge, and other implementations. So, with all this commotion out there these days, why bother with our version of yet another chat bridge?

First, there is one design principle guiding our project from others like us: to do one thing and to do it well. Matterbridge is an excellent tool, and we even use it in conjunction with TeleIRC at our university. However, it is a complex tool with many features and options. For some people, this is a non-issue. But the TeleIRC team likes to think there is beauty in simplicity. Instead of offering a tool with the most features and configuration options, we aspire to do a single thing and to do it really well: connect Telegram groups and IRC channels together.

Second, although the FruitieX/teleirc project is archived today, it was once the biggest alternative to our project, also written in NodeJS. When we decided to launch TeleIRC v2.0.0 development, it had a larger community and user base then ours. So instead of offering a “similar but different” NodeJS project, we would be the first Telegram-IRC bridge written in Go. (Yes, Matterbridge is also written in Go, but see the above paragraph.)

Third… many of the existing maintainers of TeleIRC simply wanted an excuse to learn Go. It is an opportunity to expand our knowledge, experience, and skills, especially since we are students preparing to enter the industry.

Go has a better story for Kubernetes / OpenShift. 🔗

Finally, we are carefully considering the needs of one of our biggest downstream users: the Fedora Project. Several TeleIRC developers also support Fedora’s TeleIRC SIG. Recently, the Fedora Infrastructure team launched an OpenShift instance for the Fedora community, called Communishift. All existing infrastructure in Fedora is gradually moving from virtual machines or OpenStack to OpenShift. To support this migration, we want to make a Go-based TeleIRC as easy to deploy in OpenShift as possible.

And fortunately, Go has a great story in the container orchestration world. Kubernetes and OpenShift are also Go-based projects. Go is the dominant language of this ecosystem. Its excellent performance in the niche of networking makes it a great choice for what TeleIRC does.

Now that you know more about the “why is this happening,” let’s talk on where things are and what you can expect!

TeleIRC v2.0.0: Progress so far 🔗

TeleIRC v2.0.0 is approximately 76% complete. All progress is tracked in the v2.0.0 milestone on GitHub. 46 issues and pull requests were closed since we began in September 2019. At publishing time, about 16 more issues and pull requests are left before we cut the v2.0.0 release.

Earlier in 2019, the maintainer team consisted of Justin Wheeler, Tim Zabel, Seth Hendrick, Nate Levesque, Nic Hartley, and Robby O’Connor. Now joining the committer group, we are happy to welcome Nicholas Jones, Kevin Assogba, and Kennedy Kong to the team. The current core group of maintainers for v2.0.0 are Justin, Tim, Nicholas, Kevin, and Kennedy.

When to expect TeleIRC v2.0.0 🔗

TeleIRC v2.0.0 is targeted for a release date of Friday, May 15th, 2020. At this point, we expect to have full feature parity with the v1.x.x version. We will recommend all existing users to upgrade to the latest release then.

In the meanwhile, the team is getting ready to cut a v2.0.0-pre1 release, our first “pre-release” of the Go port. We expect this release to be available on our Releases by Saturday, March 28th. Along with the v2.0.0-pre1 release, there are a few other details to note:

1. TeleIRC v1.5.0, the final version of the NodeJS version, will be released.
2. No future contributions will be accepted to the NodeJS version.
3. master branch in git will reflect the latest Go version of TeleIRC.

Once the v2.0.0-pre1 release is available, we want help to take it for a test drive! If TeleIRC is critical for you, we do not recommend using it yet, as it does not have full feature parity yet. But your early feedback can help improve the future of the next release while we are in active development.

Get involved with TeleIRC! 🔗

You can be a part of the upcoming TeleIRC v2.0.0 release. We’d love your help! There is no formal commitment to contributing, although we ask for participation through a single sprint cycle.

Read our Contributing guidelines on how to get started with TeleIRC. Virtual developer meetings take place every Saturday at 15:00 US EDT, so anyone can join and participate.

Come say hello in our developer chat rooms, either on IRC or in Telegram!

Background photo by Daria Nepriakhina on Unsplash.

However, the times are a-changing! Containers are entering as real players in the HPC space. Previously, containers were brushed off as incompatible with most HPC workflows. Now, several open source projects are emerging with unique approaches to enabling containers for HPC workloads. This blog post evaluates four container run-times in an HPC context, as they stand in July 2019:

• Charliecloud
• Shifter
• Singularity
• Podman

Research requirements 🔗

My research focused around a specific set of requirements. To receive a favorable review, a container run-time needed to meet three basic requirements:

• Support CentOS/RHEL 7.5+
• Compatibility with Univa GridEngine
• Support for very large numbers of users

Obviously there are security concerns with the third requirement. This is one reason containers have not made a strong showing in the HPC world yet. With the Docker security model, root access is a requirement to build and run containers. In a production HPC environment where users do not trust other users, this is a hard blocker.

Other HPC environments may differ. If you are an HPC administrator and also considering containers in your environment, consider my requirements. My research was exclusively framed through these three requirements.

Charliecloud 🔗

Charliecloud is an open source project based on a user-defined software stack (UDSS). Like most container implementations, it uses Linux user namespaces to run unprivileged containers. It is designed to be as minimal and lightweight as possible, to the point of not adding features that could conflict with any specific use cases. This can be a positive or a negative, depending on how complex your environment is.

However, I abandoned my research on Charliecloud early on after reading this PLOS research paper:

The software makes use of kernel namespaces that are not deemed stable by multiple prominent distributions of Linux (e.g. no versions of Red Hat Enterprise Linux or compatibles support it), and may not be included in these distributions for the foreseeable future.

The software is emphasized for its simplicity and being less than 500 lines of code, and this is an indication of having a lack of user-driven features. The containers are not truly portable because they must be extracted from Docker and configured by an external C executable before running, and even after this step, all file ownership and permissions are dependent on the user running the workflow.

Singularity: Scientific containers for mobility of compute, May 2017 (Gregory M. Kurtzer, Vanessa Sochat, Michael W. Bauer)

However, it is worth noting this paper was written in support of Singularity. It was also written by the Singularity project lead and others from the Singularity open source community. If you are conducting your own independent research, consider looking closer at Charliecloud, since at the time of writing it is still actively developed. The research paper was written in May 2017.

Edit: This situation already changed and Charliecloud is probably worth a deeper look:

This is a fantastic write up, one thing to mention is that abandoning the CharlieCloud research based solely on lack of support of the user kernel namespace is no longer a blocker. For example, PodMan now uses the same technology and it was released in RHEL8.

— Apptainer (formerly Singularity) (@SingularityApp) August 20, 2019

Shifter 🔗

Shifter is another container run-time implementation focused on HPC users. At time of writing, it is almost exclusively backed by the National Energy Research Scientific Computing Center and Cray. Most documented use cases use Slurm for cluster management / job scheduling. Instead of a Docker/OCI format, it uses its own Shifter-specific format, but this is reverse-compatible with Docker container images. It requires hosting a registry service and a Shifter Image Gateway.

The Shifter Image Gateway is a REST interface implemented with Python Flask. It pulls images from the registry service and converts them to the Shifter image format. MPI integration is supported but its implementation is MPICH-centric.

The downside to Shifter is lack of community. There are not many other organizations other than NERSC and Cray that appear to support Shifter. Documentation exists, but at writing time (July 2019), the last significant contribution was April 2018. Some bugs and feature requests are triaged, but there is not much of a maintainer presence in these issues. Most follow-up discussion to new issues are from a handful of outside contributors without commit access.

Additionally, there are several signs of stagnant development, such as NERSC/shifter#172 to add better MPI integration. However, the PR stalled out since it was first opened in April 2017. Furthermore, there is a high bus factor: most contributions and pull requests come from the same two developers, indicating low engagement from the wider HPC community. Code is regularly tested, but integration tests only exist for Slurm. For more details, check out the GitHub project pulse.

A detail worth noting is Shifter was one of the first real container run-times for HPC. A former Shifter collaborator branched off from Shifter to start Singularity (and eventually, a for-profit company to support it, Sylabs). It invites room for personal biases when evaluating Shifter and Singularity, specifically if you are not a newcomer in the HPC community.

Singularity 🔗

Singularity is the third and last HPC-specific player in the container run-time world. The vendor is Sylabs Inc. There are a few different factors that make Singularity interesting, and in my opinion, the most promising HPC container implementation.

General overview 🔗

Singularity v3.x.x is written almost entirely in Golang. It supports two image formats: Docker/OCI and Singularity’s native Single Image Format (SIF). As of September 2018, there are an estimated 25,000+ systems running Singularity, including users like TACC, San Diego Supercomputer Center, and Oak Ridge National Laboratory. Additionally, Univa announced a partnership with Sylabs in July 2018 to bring Singularity workflows to Univa GridEngine.

Sylabs offers Singularity (free and open source) and SingularityPRO (paid and proprietary). The commercial version comes with a support contract and long-term support for some releases (among other things).

Admin/root access is not required to run Singularity containers and it requires no additional configuration to do this out of the box. Containers are run under the Linux user ID that launches them (see Security and privilege escalation).

At a quick glance, Sylabs developers appear to be actively engaged in the Kubernetes development community, particularly around Red Hat technology. They also seem to keep their promises: in early 2018, blog posts show ambitious feature promises for the then-upcoming v3.0.0 release at the end of the year. Near the end of 2018, the release was delivered on-time with most/all of the promised functionality.

Image formats 🔗

The Singularity Image Format (SIF) is a single-image format (i.e. no layers involved). This was a design decision specifically for HPC workloads. SIFs are treated like a binary executable by a Linux user. Additionally, it is possible to create SIFs using the Definition File spec.

However, Singularity is also compatible with Docker/OCI images and OCI is given active development focus by upstream Singularity. Docker/OCI images are converted on-the-fly to a SIF. Docker/OCI images can be used locally or pulled from a remote registry like Docker Hub or Quay. To the user, if using a Docker/OCI image, the conversion is seamless and does not require additional configuration to use.

See this Sylabs blog post for a deeper dive on how SIFs were designed.

Flexible configuration 🔗

Singularity (uniquely?) offers advanced configuration options for HPC administrators. Some highlights are detailed here:

• Controlling bind mounts:
  • mount dev = minimal: Only binds null, zero, random, urandom, and shm into container
  • mount home = {yes,no}, mount tmp = {yes,no}: Choose to enable or disable these bind mounts globally
  • bind path = "": Bind specific paths into containers by default
  • user bind control = {yes,no}: Allow users to include their own bind mount paths or limit it to an admin-approved set of paths (above)
• Controlling containers:
  • limit container paths =: Possible to limit SIFs provided at a specific path and nowhere else

HPC community engagement 🔗

These notes only apply to Singularity free, not the proprietary SingularityPRO product.

The signals from their open source community engagement are positive and strong. They appear authentic and genuine to an open source commitment (i.e. not open-core business model). This is demonstrated in a few ways:

First, they have thorough user documentation, intended for end-users in HPC environments using Singularity. They have a less thorough but still useful admin documentation.

Second, all issues are triaged quickly and get feedback from core developers or outside contributors at a consistent pace. Pull requests don’t stagnate either: the oldest PR is less than six months old.

Third, code is regularly tested (1, 2). The code generally follows best practices (i.e. it is not atrocious to work with).

Fourth, there are also a handful of active contributors (both developers and in the community support channels) who come from outside of Sylabs, which indicates more engagement by a wider audience of people.

For more statistics, check out the GitHub project pulse.

Podman 🔗

tl;dr: Podman is an underdog that shows promise, but likely needs another one or two years of time for most HPC use cases.

Podman is a container run-time developed by Red Hat. Its primary goal is to be a drop-in replacement for Docker. While it is not explicitly designed with HPC use cases in mind, it intends to be a lightweight “wrapper” to run containers without the overhead of the full Docker daemon. Furthermore, the Podman development team is recently looking into better support for HPC use cases.

Podman is currently lacking for a HPC use case for some of these reasons:

1. Missing support for parallel filesystems (e.g. IBM Spectrum Scale)
2. Rootless Podman was designed to use kernel user namespaces which is not compatible with most parallel filesystems (might change in a year or two)
3. Not yet possible to set system site policy defaults
4. Pulling Docker/OCI images requires multiple subuids/subgids (might change in a year or two)

Where Podman does shine is providing a way to run and build containers without root access or setuid.

The same challenges and problems required for Podman to run OCI containers in an HPC environment are the same problems faced by Singularity to build SIF images without root in the HPC environment: mapping UIDs to subuids/subgids on the compute nodes. More interestingly, Buildah offers a promising way to enable users to build container images as Docker/OCI images all without root. It is plausible to use Buildah as the container image delivery mechanism and swap out the container run-time implementation (Podman vs. Singularity) depending on specific needs and requirements.

What do you think? 🔗

I hope other folks out there in the HPC world find this preliminary research useful. Do you agree or disagree with any parts of this write-up? Is something out-of-date? Drop a comment down below.

This article covers Horizontal Pod Autoscaling, what it is, and how to try it out with the Kubernetes guestbook example. By the end of this article, you will…

• Understand what Horizontal Pod Autoscaling (HPA) is
• Be able to create an HPA in Kubernetes
• Create an HPA for the Guestbook and watch it work with Siege

What is Horizontal Pod Autoscaling? 🔗

Horizontal Pod Autoscaling (HPA) is a Kubernetes API resource to dynamically grow an environment. To help simplify things, consider it in three pieces:

• Horizontal: Think of horizontal growth, i.e. adding more nodes to your available pool (unlike vertical, which would be adding more memory / CPU to your existing nodes)
• Pod: Your deployable units in Kubernetes
• Autoscaling: Automatically scaling out when needed

To help visualize it, imagine you have a Python Flask web server that reads and writes data to a Redis back-end. Your web server is the front-end for all of your incoming traffic. You run it with three pods in Kubernetes, with 512MB of RAM and 50m of CPU. Now, suddenly, BuzzFeed writes an article about your app, Kanye West name drops the app in a TV interview, and the president of the United States retweets a link to your site.

Oops.

Now you have a serious problem on your hands, where your tiny application is overloaded. Three pods aren’t cutting it anymore. You get woken up at 3:00am to hastily adjust the number of replicas and rapidly scale your infrastructure. While you’re wondering how this happened, you also wonder… isn’t there an easier way? Could I have avoided this panicked, pre-dawn scaling crisis? Yes, there is! At least, somewhat.

Building to scale 🔗

By creating and managing your deployments with HPAs, your application grows horizontally to handle the load. As the CPU utilization rises, HPAs trigger the addition of more pods to scale automatically. Previously, you could create a Horizontal Pod Autoscaler that would begin scaling when cumulative CPU utilization was at 60%. You could also tell it to scale to a maximum of 500 pods, but no less than three. So then, when the Apocalypse of Viral Sharing happened to your web application, it could have grown dynamically.

If you want to dive deeper in the technical implementation of HPAs, you can read more in the Kubernetes documentation.

Create a Horizontal Pod Autoscaler 🔗

Now that you understand how a Horizontal Pod Autoscaler (HPA) is helpful, how do you create one? Like any other resource in Kubernetes, define HPAs in a YAML definition file. Here’s a template for getting started.

---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
  namespace: my-app-space
  labels:
    app: my-app
    tier: frontend
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: my-app-deployment
  minReplicas: 2
  maxReplicas: 20
  targetCPUUtilizationPercentage: 60

This is the minimal spec you need to deploy an HPA. It’s not that different from other Kubernetes resources you may have seen.

Explaining the configuration 🔗

Let’s look at what some of the specific lines are.

• spec.scaleTargetRef.name: Name of resource to scale (e.g. name of a deployment)
• spec.minReplicas: Minimum number of replicas running when CPU use is minimal
• spec.maxReplicas: Maximum number of replicas running when CPU use peaks
• spec.targetCPUUtilizationPercentage: Percentage threshold when HPA begins scaling out pods

When starting out for the first time, tweak these values based on the amount of traffic you expect to receive or what your budget is. Load testing your application is one way to see the HPAs do their job.

Obliterating the Guestbook 🔗

But this guide wouldn’t be complete without a live demo to try. You can create one with an existing application and put it to the test. This section assumes you have a running Guestbook application in your Kubernetes environment. As a quick refresh, the Guestbook is a three-part application:

• PHP web application for writing messages into a virtual guestbook
• Primary Redis node for writing new messages from web page
• Replica Redis nodes for reading the data into web page

We’ll add an HPA as a fourth part to scale the PHP web application for new traffic.

Create the HPA for Guestbook 🔗

Now, create a new HPA spec file for the guestbook.

---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: guestbook-frontend
  namespace: guestbook
  labels:
    app: guestbook
    env: production
    tier: frontend
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: guestbook-frontend
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 75

Put this into a file and create the HPA with kubectl.

$ kubectl apply --record -f guestbook-frontend-hpa.yaml

Now, the Horizontal Pod Autoscaler is operational and monitoring the CPU utilization of your deployment.

Load test with Siege 🔗

To force the HPA into action, we’ll use Siege, an HTTP load testing and benchmark utility. Siege is a multi-threaded load testing tool and has a few other capabilities included to make it a good option for putting some force onto a simple web app.

First, put various permutations of the URL in a plaintext file. By doing this, Siege can randomly scan the URLs in he text file and ping them in “Internet mode” by randomly selecting a URL from the list for each request. This could look like the following…

http://my-guestbook.example.com/
http://my-guestbook.example.com/index.html
http://my-guestbook.example.com/guestbook.php
http://my-guestbook.example.com/guestbook.php?cmd=get&key=messages

Once this is done, you can fire up Siege to begin load testing. In this case, to get fast results, we’ll use 255 concurrent users for five minutes, using Internet and benchmark modes.

$ siege --verbose --benchmark --internet --concurrent 255 --time 10M --file siege-urls.txt

You should see Siege begin to rapidly send requests to your Guestbook application. Now that the action is in progress, you can slowly observe your CPU utilization begin to climb. Watch it slowly change by using watch.

$ watch -d -n 2 -b -c kubectl get hpa -l app=guestbook

During the five minute load test, you should notice CPU usage rise and then new replicas will appear. Depending on what your original requests and limits are for the deployment, you will see different results. Next, try setting the deployment’s requests / limits to lower values if nothing seems to happen while testing.

Learn more about Horizontal Pod Autoscaler 🔗

Horizontal Pod Autoscalers are a stable resource in Kubernetes and are available for you to begin playing around with now. To learn more, read the documentation or see another example in the official walkthrough.

My progress with ListenBrainz slowed, but I am resuming the pace of contributing and advancing on my independent study timeline. This past week, I finished out assigned tasks to discuss contributor-related documentation, like a Code of Conduct, contributor guidelines, and a pull request template. I began research on user statistics and found some already created. I wrote one of my own, but need to learn more about Google BigQuery to advance further.

Paving the contributor pathway 🔗

Earlier, I identified weaknesses for the ListenBrainz contributor pathway and found ways we could improve the pathway. This started with the development environment documentation. Now, I helped draft first revisions of our contributor guidelines, Code of Conduct reference, and pull request templates. Together, these three documents have two goals.

1. Make it easier to contribute to ListenBrainz
2. Have a better experience and have fun contributing!

Adding these documents addresses these goals. Additionally, the GitHub community profile also highlights these deliverables as ways to meet these goals. After getting feedback and seeing what others think, we make more revisions later (with some trial runs).

Back to SELinux context flags 🔗

Recently, I set my desktop back up and installed Docker for the first time on this machine; however, the development environment still failed to start. When I ran the script, it would eventually error out because of a permission denial. The web server image for ListenBrainz was failing.

After debugging, I noticed that I missed the SELinux volume tags for the ListenBrainz web server images in my original pull request, #257. When I created the pull request, I might have had cached data that let my laptop run the development environment without a problem. In either case, it was an easy fix and I knew what the issue was when it happened. Therefore, I submitted a new fix in #290.

Writing new user statistics 🔗

The most interesting part of my independent study is working with the music data to build and generate interesting statistics. I finally began exploring the existing statistics in ListenBrainz. The statistic queries use BigQuery standard SQL. BigQuery helps rapidly scan and scale data queries to help with performance (I have a lot to learn about BigQuery).

Two types of statistics 🔗

Additionally, ListenBrainz generates two types of statistics:

1. Site-wide statistics
2. User statistics

Site-wide statistics are metrics non-specific to a single user. There is only one site-wide query now. It counts how many artists were ever submitted to this ListenBrainz instance and returns an integer. There’s room for expansion in site-wide statistics.

On the other hand, user statistics are metrics specific to a single user. There’s a fair number already, like the top artists and songs in a time period and the number of artists you’ve listened to. These are a little more complete and offer more expansion for doing cool front-end work with something like D3.js.

Writing user statistics 🔗

Of course, I had to try writing my own. One helpful query I thought of was getting a count of the songs you listened to over a time period (e.g. “you listened to 500 songs this week!”). I haven’t tested it yet, but I have this in a local branch and hope to test it with real data soon.

def get_play_count(musicbrainz_id, time_interval=None): 
 
 filter_clause = "" 
 if time_interval: 
     filter_clause = "AND listened_at >=
     TIMESTAMP_SUB(CURRENT_TIME(), 
     INTERVAL {})".format(time_interval) 
 
 query = """SELECT COUNT(release_msid) as listen_count 
            FROM {dataset_id}.{table_id} 
            WHERE user_name = @musicbrainz_id 
            {time_filter_clause} 
            LIMIT {limit} 
         """.format( 
                 dataset_id=config.BIGQUERY_DATASET_ID, 
                 table_id=config.BIGQUERY_TABLE_ID, 
                 time_filter_clause=filter_clause, 
                 limit=config.STATS_ENTITY_LIMIT, 
            ) 
 
 parameters = [ 
     { 
         'type': 'STRING', 
         'name': 'musicbrainz_id', 
         'value': musicbrainz_id 
     } 
 ] 
 
 return stats.run_query(query, parameters)

Researching Google BigQuery 🔗

My next steps for the independent study are researching Google BigQuery. After going through the existing statistics and understanding how ListenBrainz generates them, an understanding of Google BigQuery is essential to writing effective queries. When I become more comfortable with the tooling and how it works, I want to map out a plan of statistics to generate and measure.

Until then, the hacking continues! As always, keep the FOSS flag high…

Last week moved quickly for me in ListenBrainz. I submitted multiple pull requests and participated in the weekly developer’s meeting on Monday. I was also invited to take part as a mentor for ListenBrainz for the upcoming round of Google Code-In! In addition to my changes and new role as a mentor, I’m researching libraries like D3.js to help build visualizations for music data.  Suddenly, everything started moving fast!

Last week: Recap 🔗

The ListenBrainz team accepted my development environment improvements and documentation. This gave me an opportunity to better explore project documentation tools. I experimented with Sphinx and Read the Docs. Sphinx introduced me to reStructuredText for documentation formats. I’ve avoided it in favor of Markdown for a long time, but I see where reStructuredText is stronger for advanced documentation.

Since ListenBrainz is a new project, I plan to contribute documentation for any of my work and improve documentation for pre-existing work. One of the goals for this independent study is to make ListenBrainz a viable candidate for a future data analysis course. To make it easy to use and understand, ListenBrainz needs excellent documentation. Since one of my strengths is technical writing, I plan to contribute more documentation this semester.

You can see some of the new documentation already!

Google Code-In mentor 🔗

The MetaBrainz community manager, Freso Olesen, approached me to mentor for Google Code-In. Google Code-In is an opportunity for teenagers to meaningfully contribute to open source projects. Google describes Google Code-In as…

Pre-university students ages 13 to 17 are invited to take part in Google Code-in: Our global, online contest introducing teenagers to the world of open source development. With a wide variety of bite-sized tasks, it’s easy for beginners to jump in and get started no matter what skills they have.

Mentors from our participating organizations lend a helping hand as participants learn what it’s like to work on an open source project. Participants get to work on real software and win prizes from t-shirts to a trip to Google HQ!

MetaBrainz is a participating organization of Google Code-In this cycle. Because of my work with ListenBrainz, I will contribute a few hours a week to help mentor participating students with ListenBrainz. Beginner problems should be easy to help with since I’m still beginning too, and as I spend more time with ListenBrainz, I can help with harder problems.

I’m excited to give back to one of my favorite open source projects in this way! I’m grateful to have this chance to help out during Google Code-In.

Choosing easyfix bugs 🔗

After I figured out the development environment issues, I went through open tickets filed against ListenBrainz to find some to work on. I made a preliminary pass through all open tickets and left some comments for more information, when needed. The tickets I highlighted to look into next were

• LB-85: Username in the profile URL should be case insensitive
• LB-124: Install messybrainz as a a python library from requirements
• LB-176: Add stats module and begin calculating some user stats from BigQuery
• LB-206: “playing_now” submissions not showing on profile
• LB-212: Show the MetaBrainz logo on the listenbrainz footer.

Of these five, LB-124 and LB-212 are already closed. While drafting this article, I completed LB-124 in PR #266. This was part of a test to get the documentation building again because of odd import errors. Later, a new student also learning the project for the first time asked to work on LB-212. Since it was a good first task to explore the project code, I passed the ticket to him.

I want to do one more “easyfix” bug before going into the main part of my independent study timeline. I don’t yet feel comfortable with the code and one more bug solved will help. After this, I plan to pursue the heavier lifting of the independent study to explore data operations and queries to make.

Researching D3.js 🔗

Prof. Roberts introduced D3.js as a library to build interactive, dynamic charts and visual representations of data. I haven’t yet looked into much front-end work, but this was a cool project that I wanted to highlight in my weekly report. This feels like it could be a powerful match for ListenBrainz, especially since the data has high detail.

Upcoming activity 🔗

This next week, I won’t have as much time to contribute to ListenBrainz. On October 21, I’m traveling to Raleigh, NC for All Things Open. On October 24, I present my talk, “What open source and J.K. Rowling have in common”. Since I’ll be out of Rochester and missing other classwork, I expect less time on my ListenBrainz work.

This next week will be slower than the last two weeks. Hopefully I’ll learn something at the conference too to bring back for ListenBrainz.

Until then… keep the FOSS flag high.

One of the first rites of passage when working on a new project is creating your development environment. It always seems simple, but sometimes there are bumps along the way. The first activity I did to begin contributing to ListenBrainz was create my development environment. I wasn’t successful with the documentation in the README, so I had to play around and work with the project before I was even running it.

The first part of this post details how to set up your own development environment. Then, the second half talks about the solution I came up with and my first contribution back to the project.

Install dependencies: Docker 🔗

This tutorial assumes you are using a Linux distribution. If you’re using a different operating system, install the necessary dependencies or packages with your preferred method.

ListenBrainz ships in Docker containers, which helps create your development environment and later deploy the application. Therefore, to work on the project, you need to install Docker and use containers for building the project. Containers save you from installing all of this on your own workstation! Since I’m using Fedora, I run this command.

sudo dnf install docker docker-compose

Register a MusicBrainz application 🔗

Next, you need to register your application and get a OAuth token from MusicBrainz. Using the OAuth token lets you sign into your development environment with your MusicBrainz account. Then, you can import your plays from somewhere else.

To register, visit the MusicBrainz applications page. There, look for the option to register your application. Fill out the form with these three options.

• Name: (any name you want and will recognize, I used listenbrainz-server-devel)
• Type: Web Application
• Callback URL: http://localhost/login/musicbrainz/post

After entering this information, you’ll have a OAuth client ID and OAuth client secret. You’ll use these for configuring ListenBrainz.

Update config.py 🔗

With your new client ID and secret, update the ListenBrainz configuration file. If this is your first time configuring ListenBrainz, copy the sample to a live configuration.

cp listenbrainz/config.py.sample listenbrainz/config.py

Next, open the file with your favorite text editor and look for this section.

# MusicBrainz OAuth
MUSICBRAINZ_CLIENT_ID = "CLIENT_ID"
MUSICBRAINZ_CLIENT_SECRET = "CLIENT_SECRET"

Update the strings with your client ID and secret. After doing this, your ListenBrainz development environment is able to authenticate and log in from your MusicBrainz login.

Initialize ListenBrainz databases 🔗

Your development environment needs some databases present to work. Before proceeding, run these three commands to initialize the databases.

docker-compose -f docker/docker-compose.yml -p listenbrainz run --rm web python3 manage.py init_db --create-db
docker-compose -f docker/docker-compose.yml -p listenbrainz run --rm web python3 manage.py init_msb_db --create-db
docker-compose -f docker/docker-compose.yml -p listenbrainz run --rm web python3 manage.py init_influx

Your development environment is now ready. Now, let’s actually see ListenBrainz load locally!

Run the magic script 🔗

Once you have done this, run the develop.sh script in the root of the repository. Using docker-compose, the script creates multiple Docker containers for the different services and parts of the ListenBrainz server. Running this script will start Redis, PostgreSQL, InfluxDB, and web server containers, to name a few. But this also makes it easy to stop them all later.

./develop.sh

You will see the containers build and eventually run. Leave the script running to see your development environment. Later, you can shut it down by pressing CTRL^C. Once everything is running, visit your new site from your browser!

http://localhost/

Now, you are all set to begin making changes and testing them in your development environment!

Making my first pull request 🔗

As mentioned earlier, my first attempt at a development environment was unsuccessful. My system kept denying permission to the processes in the containers. After looking at system audit logs and running a temporary setenforce 0, I tried the script one more time. Everything suddenly worked! So the issue was mostly with SELinux.

With my goal to get my environment set up, I figured out a few issues with the configuration offered by the project developers. I eventually made PR #257 against listenbrainz-server with my improvements.

Labeling SELinux volume mounts 🔗

To diagnose the issue, I started with a quick search and found a StackOverflow question with my same problem. There, the question was about Docker containers and denied permissions in the container. The answers explained it was an SELinux error and the context for the containers was not set. However, temporarily changing context for a directory didn’t seem too effective and doesn’t persist across reboots.

Continuing the search, I found an issue filed against docker-compose about the :z and :Z flags for volume mounts. These flags set SELinux context for containers, with the best explanation I found coming from this StackOverflow answer.

Two suffixes :z or :Z can be added to the volume mount. These suffixes tell Docker to relabel file objects on the shared volumes. The ‘z’ option tells Docker that the volume content will be shared between containers. Docker will label the content with a shared content label. Shared volumes labels allow all containers to read/write content. The ‘Z’ option tells Docker to label the content with a private unshared label.

Therefore, I added the :z flag to all the volume mounts in the docker-compose.yml file. I submitted a fix upstream for this in listenbrainz-server#257!

Correct the startup port 🔗

In the README, it says the server will start on port 8000, but the docker-compose.yml file actually started the server on port 80. I included a fix for this in my pull request as well.

git push! 🔗

This post makes a debugging experience that actually took hours look like it happened in minutes. But after getting over this hurdle, it was awesome to finally see ListenBrainz running locally on my workstation. It was an even better feeling when I could take my improvements and send them back in a pull request to ListenBrainz. Hopefully this will make it easier for others to create their own development environments and start hacking!

Welcome back to the Kubernetes and Fedora series. Each week, we build on the previous articles in the series to help introduce you to using Kubernetes. This article picks up from where we left off last when you installed Tectonic to Amazon Web Services (AWS). By the end of this article, you will…

• Start up Redis master and slave pods
• Start a front-end pod that interacts with the Redis pods
• Deploy a simple web app for all of your friends to leave you messages

Compared to previous articles, this article will be a little more hands-on. Also like before, this is based off an excellent tutorial in the upstream Kubernetes documentation. Let’s get started!

Pre-requisites 🔗

This tutorial assumes you followed the Minikube how-to earlier in this series and that you already have a Tectonic installation running (doesn’t have to be on AWS). In case you’re jumping in now, make sure you have the Kubernetes client tools installed on your Fedora system, like kubectl. If not, you can install them now.

$ sudo dnf install kubernetes-client

Configure kubectl for Tectonic 🔗

To use kubectl with your Tectonic installation, you need to have a valid configuration in ~/.kube/config for your cluster. This is how kubectl knows where and how to talk to Tectonic. To get these values, first log into the Tectonic Console you installed.

1. Click username (usually admin) > My Account on the bottom left.
2. Click Download Configuration.
3. When the Set Up kubectl window opens, click Verify Identity.
4. Enter your username and password, and click Login.
5. From the Login Successful screen, copy the provided code.
6. Switch back to Tectonic and enter the code in the field.

Now you will be able to download kubectl-config from Tectonic. There’s two ways to proceed from here.

Add a new configuration 🔗

If this is your first time using kubectl, your configuration is likely empty. If it’s empty or you don’t care about overwriting an old configuration, you can run the following commands to add the configuration.

$ mkdir ~/.kube/
$ mv ~/Downloads/minikube-config ~/.kube/config
$ chmod 600 ~/.kube/config

Append to an existing configuration 🔗

If you already have a configuration, like from Minikube, you might not want to wipe it all out. In this case, you can merge the files manually together. You’ll need to copy the clusters, users, and contexts from the Tectonic configuration into your existing one. The benefit of doing this is that you’ll be able to change contexts to switch from one cluster to another.

Test your configuration 🔗

Once you finished your configuration, test to see if it works.

$ kubectl config use-context tectonic       # if you have multiple contexts in config
$ kubectl get nodes
NAME                                        STATUS    AGE
ip-10-0-0-59.us-east-2.compute.internal     Ready     1d
ip-10-0-23-239.us-east-2.compute.internal   Ready     1d
ip-10-0-44-211.us-east-2.compute.internal   Ready     1d
ip-10-0-61-218.us-east-2.compute.internal   Ready     1d
ip-10-0-67-239.us-east-2.compute.internal   Ready     1d
ip-10-0-95-51.us-east-2.compute.internal    Ready     1d

Huzzah! Now we’re ready to get to work.

Getting the deployment and service files 🔗

All of the example files come from the official Kubernetes GitHub repo. You can find them in the Guestbook example. To get started, create a new directory and download all of the files.

$ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/examples/guestbook/redis-{master,slave}-{deployment,service}.yaml \
       https://raw.githubusercontent.com/kubernetes/kubernetes/master/examples/guestbook/frontend-{deployment,service}.yaml

We’ll explain what all of these do in next steps. All of these next steps will start with the command to run, followed by a short explanation of what’s actually happening.

Start the Redis master 🔗

$ kubectl create -f redis-master-service.yaml
service "redis-master" created
$ kubectl create -f redis-master-deployment.yaml
deployment "redis-master" created

Define the deployment 🔗

The redis-master-deployment.yaml file downloaded earlier defines the deployment and its characteristics. In this case, we have one pod that runs the Redis master in a container. Since we’re using a deployment, that means if our pod goes down, Kubernetes will spin up a new pod to replace it. Worth noting in this example, if the pod did go down, there would be a potential for data loss until the new one replaces the old one (since the Redis master is not highly available, i.e. there are multiple).

Define the service 🔗

Our service in this example is a named load balancer that proxies traffic across one or many containers. Even though we only have one Redis master pod, we still want to use a service. This is a deterministic way of making the route to the master with a dynamic (or elastic) IP address.

Labeling the pods is important in this case, as Kubernetes will use the pods’ labels to determine which pods receive the traffic sent to the service, and load balance it accordingly.

Create the service 🔗

The next important step is to create the service. Note that we’re doing this before we create the deployment. It’s best practice to create the service first. This allows the scheduler to later spread the service across the deployments you create to support your application.

After creating the service, you can check its status by running this command. You should see similar output.

$ kubectl get services
NAME              CLUSTER-IP       EXTERNAL-IP       PORT(S)       AGE
redis-master      10.0.76.248      <none>            6379/TCP      1s

Now your Redis master serivce is up and running! The next step will be to create the Redis master deployment.

If you look at the service configuration file, you’ll notice port and targetPort are two defined variables. Once everything is up and running, these will be important for determining how the traffic from the slaves to the masters is routed.

1. Redis slave connects to port on Redis master service
2. Traffic forwarded from service’s port to targetPort on pod the service listens to

Create the deployment 🔗

Next, we created the Redis master pod in the cluster. To see our deployment and pods, we can run the following commands to see what was created.

$ kubectl get deployments
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
redis-master   1         1         1            1           27s

$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
redis-master-2353460263-1ecey   1/1       Running   0          1m
...

You should see all of the pods in your cluster so far. For now, that’s just the Redis master. Let’s give it some friends!

Start the Redis slaves 🔗

$ kubectl create -f redis-slave-service.yaml
service "redis-slave" created
$ kubectl create -f redis-slave-deployment.yaml
deployment "redis-slave" created

Defining the deployment 🔗

In the configuration file, we defined two replicas, unlike the master. By doing this, it tells Kubernetes that the minimum number of pods that should always be running is two. If one of your pods goes down, Kubernetes automatically creates a new one to support the application. If you want, you can try killing the Docker process for one of your pods to see it happen in real time.

Start the guestbook front-end 🔗

$ kubectl create -f frontend-service.yaml
service "frontend" created
$ kubectl create -f frontend-deployment.yaml
deployment "frontend" created

The front-end is a PHP application with an AJAX interface and Angular-based UI. When using the form on the front-end application, it talks to the Redis master or slave, depending on if it’s reading or writing to Redis. Again, we’re deploying the front-end with multiple replicas. In this case, there will be three pods to support the front-end.

Say hello! 🔗

Once you’ve finished deploying everything, your web app should now be accessible! To get the full domain from AWS, run this command to figure out where to look.

$ kubectl get deploy/frontend svc/frontend -o wide
NAME           CLUSTER-IP   EXTERNAL-IP                                                             PORT(S)        AGE       SELECTOR
svc/frontend   10.3.0.175   aaebd8247ef2311e6a045021d1620193-54019671.us-east-2.elb.amazonaws.com   80:31020/TCP   1m        k8s-app=guestbook,tier=frontend

NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/frontend   3         3         3            3           1m

Congratulations, we’re all finished!

Cleaning up 🔗

Once you’re finished or when you want to stop running the guestbook, it’s easy to get rid of the deployments and services we created. Using labels, all the deployments and services can be deleted with one command.

$ kubectl delete deployments,services -l "app in (redis, guestbook)"

And now your guestbook application is offline. (It was nice while it lasted!)

Learn more about Kubernetes and Tectonic 🔗

If you want to explore more about Kubernetes, you can read some of the earlier articles in this series. You can also read the original tutorial published by Kubernetes on GitHub. Additionally, the upstream documentation for Kubernetes and Tectonic is thorough and can help answer more advanced questions.

Questions, Tectonic stories, or tips for beginners? Add your comments below.

Welcome back to the Kubernetes and Fedora series. Each week, we build on the previous articles in the series to help introduce you to using Kubernetes. This article takes off from running Kubernetes on your own hardware and moves us one step closer to the cloud. By the end of this article, you will…

• Understand what CoreOS Tectonic is
• Set up Amazon Web Services (AWS) for Tectonic
• Deploy Tectonic to AWS

This article is also based off of the excellent tutorial provided in the CoreOS documentation. Let’s get started!

What is Tectonic? 🔗

In the first article, some of the key concepts of Kubernetes and why it’s useful were explained. Kubernetes automates the deployment and setting up of your infrastructure across the three layers (users, masters, nodes). If you’re working on your own at a small scale, Kubernetes itself can be plenty to meet your needs. However, there is still a decent amount of human involvement in managing the different pieces of Kubernetes. If you’re working with multiple people in a team and across different environments, vanilla Kubernetes can be a lot to manage. For an enterprise environment, there’s still some unmet needs. This is where Tectonic steps in.

Tectonic is a commercial product offered by CoreOS, the providers of Container Linux and the original developers of etcd, now one of the core components of Kubernetes. Tectonic takes all of the open source components and pre-packages them. The self-proclaimed goal of doing this is to let anyone build a Google-style infrastructure into a cloud or on-premise environment. The outcome for the user is that it’s easy to install a Kubernetes infrastructure across many different environments. In addition to simplifying the installation of the various components of a Kubernetes stack, Tectonic also provides a management console, a container registry for building and sharing containers, additional tools for deployment, and a few other nice features.

If we think about Kubernetes as a cake like we did before with three layers, Tectonic is like the box you set it in. Now, you can take your cake anywhere, move it around, and stack it with other cakes-in-a-box. All of your cakes are in their own boxes and you don’t have to worry about them accidentally being damaged. If you’re still a little confused, this diagram might help make more sense of it.

Fortunately, Tectonic has a free license that lets you use it for ten nodes. In this example, we’ll register, get a free license, and deploy it into AWS.

(Note: If you want to revert anything we do in this example, there’s an easy way to dismantle it across AWS and bring your bill to $0.00.)

Pre-requisites 🔗

In order to successfully run this guide, there’s a few things you’ll need first.

• Amazon Web Services (AWS) account (free)
  • Register here
• CoreOS Tectonic account and license (free)
  • Register here
• A root-level or sub-domain (e.g. example.com or k8s.example.com)
  • If you look around, you can probably find some for less than USD$1 a year if you need one
• Curiosity!

Setting up DNS with Route 53 🔗

The first things we’ll do is set up our domain with Route 53 in AWS. Route 53 can do a lot of things, like DNS management, traffic management, availability monitoring, domain registration, and more. However, we’re only going to be using it for DNS management. Tectonic will use this to automatically provision DNS records for internal and external use.

Add your domain 🔗

To add your domain to Route 53, follow these steps from AWS.

1. From Services, select Networking & Content Delivery > Route 53.
2. Select Hosted zones from the left pane and click Create Hosted Zone.
3. Enter your domain or sub-domain, add a comment if you want, and choose a Public Zone for the type.

Once you’ve done this, you can go ahead and click “Create”.

Change the nameservers 🔗

After adding the hosted zone to Route 53, you’ll need to change the nameservers for your domain via the domain registrar (whoever you bought the domain from). Usually it should be easy to find this, but it varies among registrars. If you’re having a hard figuring out how to do this, try searching for a how-to or contacting your registrar’s support.

After you added the hosted zone, you should see the nameservers in Route 53. There will be four nameservers provided there. You can copy and paste them from Route 53 to your domain registrar. Also note that if you’re using a subdomain, the instructions might be a little different. You can read how to do this in the Route 53 documentation.

The nameservers could take minutes or hours to update, depending on how lucky you are. If you’re impatient and want to check, open up a terminal and run this command. If you see the AWS nameservers in the output, then your domain has propagated and is now usable by Route 53.

dig -t ns <example.com>

Configuring EC2 with SSH key pair 🔗

This guide assumes you already have an SSH key pair created on your system. If you don’t have one generated, you can read how to generate one here.

The next step for us is to add an SSH key pair to EC2, one of the compute engine products offered by AWS. We’ll import an existing key on your system into EC2.

1. From AWS, go to Services > Compute > EC2.
2. Confirm that you are in the correct EC2 region by checking the location next to your name in the menu bar.
3. Under Network & Security, click Key Pairs.
4. Click Import Key Pair.
5. Either upload your public key file (~/.ssh/id_rsa.pub) or paste it into the text field. Don’t forget to give it a name.

And that’s all you need to do!

Assigning AWS user privileges 🔗

Tectonic does the magic of setting up AWS for you, so you don’t have to manually add and create the services from the web interface. In order to do this, you need to add a user account that Tectonic can use to do all of the provisioning it needs. To do this, you’ll need to create a new Access ID and Secret key pair from AWS.

1. Select Services > Security, Identity & Compliance > IAM.
2. From the left hand pane, click Users, then click Add user.
3. Set the user details:
   1. User name can be anything you like (I used tectonic-mydomain.com)
   2. Access type only needs to be Programmatic access
4. For permissions, click Add user to group and create a new group for your user.
5. When creating a new group, attach only the policies needed by Tectonic to operate correctly:
   1. AmazonEC2FullAccess
   2. IAMFullAccess
   3. AmazonS3FullAccess
   4. AmazonVPCFullAccess
   5. AmazonRoute53FullAccess
6. Finish creating the user. You’ll then see the Access key ID and Secret access key. Hold onto these, you’ll need them later. You won’t get to see the secret key again!

Now we’re ready to install Tectonic! Let’s grab your credentials next.

Download Tectonic credentials 🔗

Jump back over to the CoreOS accounts page. When you’re logged in, you’ll see the Account Assets area. Download the CoreOS license file and pull secret. Later on in the installer, you’ll need to insert these to finish the installation.

Running the installer 🔗

Now things get interesting! We finally get to install and deploy Tectonic into AWS. The installer takes the form of a graphical installer in your web browser. To use the installer, you need to download the binary and run it. If you’re curious, you can find the installer source code on GitHub.

Download and run installer 🔗

First, open up a new terminal window and navigate to a directory you want to download the installer to. Even though you likely won’t need to run the installer again, you will want to hang on to this if you ever want to easily dismantle everything in AWS later.

curl -O https://releases.tectonic.com/tectonic-1.6.4-tectonic.1.tar.gz

Next, extract the tarball and navigate into the directory.

tar -xzvf tectonic-1.6.4-tectonic.1.tar.gz
cd tectonic/tectonic-installer

Now execute the installer binary. After running this, a new browser window will open that features the graphical installer.

./linux/installer

Running the installer 🔗

The installer is thorough and assumes safe defaults for most of the steps. Be sure to have your AWS Access and Secret ID keys on hand. You should be able to run through the installer without issue. If you’re confused about what any of the values mean or want to make custom changes, you can read more in the upstream documentation.

Once you’re finished, congrats! You’ve successfully installed Tectonic!

Check out your Tectonic install 🔗

Once you finish the installation successfully, your Tectonic installation will be accessible within AWS. You can navigate to the domain you specified during the install to find it. Unless you added a CA authority and certificates, your browser will probably complain about invalid SSL certificates, but you can ignore the warning safely. It might also take a few minutes before the URL is accessible, so if you were looking for a coffee or tea break, now would be a good time!

Once you’re logged in, you should see something like this.

Blow it all away! 🔗

If you’re like me, you might be frustrated by guides that tell you how to install things but not how to take it all apart. Fortunately, this guide not only tells you how to do that, but the Tectonic installer also makes it super easy to do. If you’re sure that you’re done with Tectonic and don’t want any leftovers to remain in AWS, this is the best way to do it, instead of deleting everything manually from the AWS Console.

Every installation has a time-stamped folder in the tectonic directory we used earlier. First, you need to navigate into the specific folder for the cluster you installed. It’s important to be inside of this directory first.

cd tectonic/tectonic-installer/linux/clusters/<CLUSTERNAME>

<CLUSTERNAME> will be the time-stamped directory. Once you’re in the folder, run this command to trigger the uninstaller. After running this, you’ll see the installer slowly dismantle everything and delete any leftovers in AWS.

../../terraform destroy

Once it finishes, you should see an output message confirming how many AWS resources were destroyed. And now you’re back to where you started.

Learn more about Tectonic 🔗

If you thought this was exciting and want to learn more, there is no shortage of resources for you to read. You can learn more about Tectonic from the CoreOS website or the original release announcement. You can also dig into the installer’s source code on GitHub. If you’re still trying to wrap your head around Tectonic, there’s a good write-up on virtualizationreview.com.

Next week, we’ll install a simple guestbook application to our Tectonic installation to see how it all works and what you can do with it. Stay tuned!

Questions, Tectonic stories, or tips for beginners? Add your comments below.

This is a short series to introduce Kubernetes, what it does, and how to experiment with it on Fedora. This is a beginner-oriented series to help introduce some higher level concepts and give examples of using it on Fedora. In the first post, we covered key concepts in Kubernetes. This second post shows you how to build a single-node Kubernetes deployment on your own computer.

Once you have a better understanding of what the key concepts and terminology in Kubernetes are, getting started is easier. Like many programming tutorials, this tutorial shows you how to build a “Hello World” application and deploy it locally on your computer using Kubernetes. This is a simple tutorial because there aren’t multiple nodes to work with. Instead, the only device we’re using is a single node (a.k.a. your computer). By the end, you’ll see how to deploy a Node.js application into a Kubernetes pod and manage it with a deployment on Fedora.

This tutorial isn’t made from scratch. You can find the original tutorial in the official Kubernetes documentation. This article adds some changes that will let you do the same thing on your own Fedora computer.

Introducing Minikube 🔗

Minikube is an official tool developed by the Kubernetes team to help make testing it out easier. It lets you run a single-node Kubernetes cluster through a virtual machine on your own hardware. Beyond using it to play around with or experiment for the first time, it’s also useful as a testing tool if you’re working with Kubernetes daily. It does support many of the features you’d want in a production Kubernetes environment, like DNS, NodePorts, and container run-times.

Installation 🔗

This tutorial requires virtual machine and container software. There are many options you can use. Minikube supports virtualbox, vmwarefusion, kvm, and xhyve drivers for virtualization. However, this guide will use KVM since it’s already packaged and available in Fedora. We’ll also use Node.js for building the application and Docker for putting it in a container.

Pre-requirements 🔗

You can install the prerequisites with this command.

$ sudo dnf install kubernetes libvirt-daemon-kvm kvm nodejs docker

After installing these packages, you’ll need to add your user to the right group to let you use KVM. The following commands will add your user to the group and then update your current session for the group change to take effect.

$ sudo usermod -a -G libvirt $(whoami)
$ newgrp libvirt

Docker KVM drivers 🔗

If using KVM, you will also need to install the KVM drivers to work with Docker. You need to add Docker Machine and the Docker Machine KVM Driver to your local path. You can check their pages on GitHub for the latest versions, or you can run the following commands for specific versions. These were tested on a Fedora 25 installation.

Docker Machine 🔗

$ curl -L https://github.com/docker/machine/releases/download/v0.12.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine
$ chmod +x /tmp/docker-machine
$ sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

Docker Machine KVM Driver 🔗

This installs the CentOS 7 driver, but it also works with Fedora.

$ curl -L https://github.com/dhiltgen/docker-machine-kvm/releases/download/v0.10.0/docker-machine-driver-kvm-centos7 >/tmp/docker-machine-driver-kvm
$ chmod +x /tmp/docker-machine-driver-kvm
$ sudo cp /tmp/docker-machine-driver-kvm /usr/local/bin/docker-machine-driver-kvm

Installing Minikube 🔗

The final step for installation is getting Minikube itself. Currently, there is no package in Fedora available, and official documentation recommends grabbing the binary and moving it your local path. To download the binary, make it executable, and move it to your path, run the following.

$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
$ chmod +x minikube
$ sudo mv minikube /usr/local/bin/

Now you’re ready to build your cluster.

Create the Minikube cluster 🔗

Now that you have everything installed and in the right place, you can create your Minikube cluster and get started. To start Minikube, run this command.

$ minikube start --vm-driver=kvm

Next, you’ll need to set the context. Context is how kubectl (the command-line interface for Kubernetes) knows what it’s dealing with. To set the context for Minikube, run this command.

$ kubectl config use-context minikube

As a check, make sure that kubectl can communicate with your cluster by running this command.

$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Build your application 🔗

Now that Kubernetes is ready, we need to have an application to deploy in it. This article uses the same Node.js application as the official tutorial in the Kubernetes documentation. Create a folder called hellonode and create a new file called server.js with your favorite text editor.

var http = require('http');

var handleRequest = function(request, response) {
 console.log('Received request for URL: ' + request.url);
 response.writeHead(200);
 response.end('Hello world!');
};
var www = http.createServer(handleRequest);
www.listen(8080);

Now try running your application and running it.

$ node server.js

While it’s running, you should be able to access it on localhost:8080. Once you verify it’s working, hit Ctrl+C to kill the process.

Create Docker container 🔗

Now you have an application to deploy! The next step is to get it packaged into a Docker container (that you’ll pass to Kubernetes later). You’ll need to create a Dockerfile in the same folder as your server.js file. This guide uses an existing Node.js Docker image. It exposes your application on port 8080, copies server.js to the image, and runs it as a server. Your Dockerfile should look like this.

FROM node:6.9.2
EXPOSE 8080
COPY server.js .
CMD node server.js

If you’re familiar with Docker, you’re likely used to pushing your image to a registry. In this case, since we’re deploying it to Minikube, you can build it using the same Docker host as the Minikube virtual machine. For this to happen, you’ll need to use the Minikube Docker daemon.

$ eval $(minikube docker-env)

Now you can build your Docker image with the Minikube Docker daemon.

$ docker build -t hello-node:v1 .

Huzzah! Now you have an image Minikube can run.

Create Minikube deployment 🔗

If you remember from the first part of this series, deployments watch your application’s health and reschedule it if it dies. Deployments are the supported way of creating and scaling pods. kubectl run creates a deployment to manage a pod. We’ll create one that uses the hello-node Docker image we just built.

$ kubectl run hello-node --image=hello-node:v1 --port=8080

Next, check that the deployment was created successfully.

$ kubectl get deployments
NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
hello-node   1         1         1            1           30s

Creating the deployment also creates the pod where the application is running. You can view the pod with this command.

$ kubectl get pods
NAME                          READY     STATUS    RESTARTS   AGE
hello-node-1644695913-k2314   1/1       Running   0          3

Finally, let’s look at what the configuration looks like. If you’re familiar with Ansible, the configuration files for Kubernetes also use easy-to-read YAML. You can see the full configuration with this command.

$ kubectl config view

kubectl does many things. To read more about what you can do with it, you can read the documentation.

Create service 🔗

Right now, the pod is only accessible inside of the Kubernetes pod with its internal IP address. To see it in a web browser, you’ll need to expose it as a service. To expose it as a service, run this command.

$ kubectl expose deployment hello-node --type=LoadBalancer

The type was specified as a LoadBalancer because Kubernetes will expose the IP outside of the cluster. If you were running a load balancer in a cloud environment, this how you’d provision an external IP address. However, in this case, it exposes your application as a service in Minikube. And now, finally, you get to see your application. Running this command will open a new browser window with your application.

$ minikube service hello-node

Congratulations, you deployed your first containerized application via Kubernetes! But now, what if you need to our small Hello World application?

How do we push changes? 🔗

The time has come when you’re ready to make an update and push it. Edit your server.js file and change “Hello world!” to “Hello again, world!”

response.end('Hello again, world!');

And we’ll build another Docker image. Note the version bump.

$ docker build -t hello-node:v2 .

Next, you need to give Kubernetes the new image to deploy.

$ kubectl set image deployment/hello-node hello-node=hello-node:v2

And now, your update is pushed! Like before, run this command to have it open in a new browser window.

$ minikube service hello-node

If your application doesn’t come up any different, double-check that you updated the right image. You can troubleshoot by getting a shell into your pod by running the following command. You can get the pod name from the command run earlier (kubectl get pods). Once you’re in the shell, check if the server.js file shows your changes.

$ kubectl exec -it <pod-name> bash

Cleaning up 🔗

Now that we’re done, we can clean up the environment. To clear up the resources in your cluster, run these two commands.

$ kubectl delete service hello-node
$ kubectl delete deployment hello-node

If you’re done playing with Minikube, you can also stop it.

$ minikube stop

If you’re done using Minikube for a while, you can unset Minikube Docker daemon that we set earlier in this guide.

$ eval $(minikube docker-env -u)

Learn more about Kubernetes 🔗

You can find the original tutorial in the Kubernetes documentation. If you want to read more, there’s plenty of great information online. The documentation provided by Kubernetes is thorough and comprehensive.

Questions, Minikube stories, or tips for beginners? Add your comments below.

This article is part of a short series that introduces Kubernetes. This beginner-oriented series covers some higher level concepts and gives examples of using Kubernetes on Fedora.

The information technology world changes daily, and the demands of building scalable infrastructure become more important. Containers aren’t anything new these days, and have various uses and implementations. But what about building scalable, containerized applications? By itself, Docker and other tools don’t quite cut it, as far as building the infrastructure to support containers. How do you deploy, scale, and manage containerized applications in your infrastructure? This is where tools such as Kubernetes comes in. Kubernetes is an open source system that automates deployment, scaling, and management of containerized applications. Kubernetes was originally developed by Google before being donated to the Cloud Native Computing Foundation, a project of the Linux Foundation. This article gives a quick precursor to what Kubernetes is and what some of the buzzwords really mean.

What is Kubernetes? 🔗

Kubernetes simplifies and automates the process of deploying containerized applications at scale. Just like Ansible orchestrates software, Kubernetes orchestrates deploying infrastructure that supports the software. There are various “layers of the cake” that make Kubernetes a strong solution for building resilient infrastructure. It also assists with making systems that can grow at scale. If your application has increasing demands such as higher traffic, Kubernetes helps grow your environment to support increasing demands. This is one reason why Kubernetes is helpful for building long-term solutions for complex problems (even if it’s not complex… yet).

At a high level overview, imagine three different layers.

• Users: People who deploy or create containerized applications to run in your infrastructure
• Master(s): Manages and schedules your software across various other machines, for example in a clustered computing environment
• Nodes: Various machines to support the application, called kubelets

These three layers are orchestrated and automated by Kubernetes. One of the key pieces of the master (not included in the visual) is etcd. etcd is a lightweight and distributed key/value store that holds configuration data. Each node, or kubelet, can access this data in etcd through a HTTP/JSON API interface. The components of communication between master and node such as etcd are explained in the official documentation.

Another important detail not shown in the diagram is that you might have many masters. In a high-availability (HA) set-up, you can keep your infrastructure resilient by having multiple masters in case one happens to go down.

Terminology 🔗

It’s important to understand the concepts of Kubernetes before you start to play around with it. There are many core concepts in Kubernetes, such as services, volumes, secrets, daemon sets, and jobs. However, this article explains four that are helpful for the next exercise of building a mini Kubernetes cluster. The three concepts are pods, labels, replica sets, and deployments.

Pods 🔗

If you imagine Kubernetes as a Lego® castle, pods are the smallest block you can pick out. By themselves, they are the smallest unit you can deploy. The containers of an application fit into a pod. The pod can be one container, but it can also be as many as needed. Containers in a pod are unique since they share the Linux namespace and aren’t isolated from each other. In a world before containers, this would be similar to running an application on the same host machine.

When the pods share the same namespace, all the containers in a pod:

• Share an IP address
• Share port space
• Find each other over localhost
• Communicate over IPC namespace
• Have access to shared volumes

But what’s the point of having pods? The main purpose of pods is to have groups of “helping” containers on the same namespace (co-located) and integrated together (co-managed) along with the main application container. Some examples might be logging or monitoring tools that check the health of your application, or backup tools that act when certain data changes.

In the big picture, containers in a single pod are always scheduled together too. However, Kubernetes doesn’t automatically reschedule them to a new node if the node dies (more on this later).

Labels 🔗

Labels are a simple but important concept in Kubernetes. Labels are key/value pairs attached to objects in Kubernetes, like pods. They let you specify unique attributes of objects that actually mean something to humans. You can attach them when you create an object, and modify or add them later. Labels help you organize and select different sets of objects to interact with when performing actions inside of Kubernetes. For example, you can identify:

• Software releases: Alpha, beta, stable
• Environments: Development, production
• Tiers: Front-end, back-end

Labels are as flexible as you need them to be, and this list isn’t comprehensive. Be creative when thinking of how to apply them.

Replica sets 🔗

Replica sets are where some of the magic begins to happen with automatic scheduling or rescheduling. Replica sets ensure that a number of pod instances (called replicas) are running at any moment. If your web application needs to constantly have four pods in the front-end and two in the back-end, the replica sets are your insurance that number is always maintained. This also makes Kubernetes great for scaling. If you need to scale up or down, change the number of replicas.

When reading about replica sets, you might also see replication controllers. They are somewhat interchangeable, but replication controllers are older, semi-deprecated, and less powerful than replica sets. The main difference is that sets work with more advanced set-based selectors – which goes back to labels. Ideally, you won’t have to worry about this much today.

Even though replica sets are where the scheduling magic happens to help make your infrastructure resilient, you won’t actually interact with them much. Replica sets are managed by deployments, so it’s unusual to directly create or manipulate replica sets. And guess what’s next?

Deployments 🔗

Deployments are another important concept inside of Kubernetes. Deployments are a declarative way to deploy and manage software. If you’re familiar with Ansible, you can compare deployments to the playbooks of Ansible. If you’re building your infrastructure out, you want to make sure it is easily reproducible without much manual work. Deployments are the way to do this.

Deployments offer functionality such as revision history, so it’s always easy to rollback changes if something doesn’t work out. They also manage any updates you push out to your application, and if something isn’t working, it will stop rolling out your update and revert back to the last working state. Deployments follow the mathematical property of idempotence, which means you define your specs once and use them many times to get the same result.

Deployments also get into imperative and declarative ways to build infrastructure, but this explanation is a quick, fly-by overview. You can read more detailed information in the official documentation.

Installing on Fedora 🔗

If you want to start playing with Kubernetes, install it and some useful tools from the Fedora repositories.

sudo dnf install kubernetes

This command provides the bare minimum needed to get started. You can also install other cool tools like cockpit-kubernetes (integration with Cockpit) and kubernetes-ansible (provisioning Kubernetes with Ansible playbooks and roles).

Learn more about Kubernetes 🔗

If you want to read more about Kubernetes or want to explore the concepts more, there’s plenty of great information online. The documentation provided by Kubernetes is fantastic, but there are also other helpful guides from DigitalOcean and Giant Swarm. The next article in the series will explore building a mini Kubernetes cluster on your own computer to see how it really works.

Questions, Kubernetes stories, or tips for beginners? Add your comments below.