This week is the last week of the fall 2017 semester at RIT. This semester, I spent time with the MetaBrainz community working on ListenBrainz for an independent study. This post explains what I was working on in the last month and reflects back on my original objectives for the independent study.

Running my own ListenBrainz 🔗

The RIT Linux Users Group hosts various virtual machines for our projects. I requested one to set up and host a “production” ListenBrainz site. The purpose of doing this was to…

1. Test my changes in a “production” environment
2. Offer a service for the RIT Linux Users Group to poke around with

I spent most of this time working with our system administrator to set up the machine and adjust hardware specs for ListenBrainz. Once we fixed storage space and memory issues, it was easy to set it up and get ListenBrainz running. My experience writing the development guide made it easy to get set up and get working. On the first run, it worked!

Now, listen.ritlug.com is live.

Figuring out HTTPS 🔗

My next challenge for the site is to set up HTTPS. I tried using a reverse proxy in nginx to set up HTTPS, but I received 502 Bad Gateway errors. I realized I spent too much time figuring this out on my own and decided to ask for help in the MetaBrainz community forums.

Proposing new statistics 🔗

Halfway through the independent study, I realized I would fall short of my original objective of implementing basic statistics in ListenBrainz. To compromise, I wrote a proposal for new statistics to start in the project. My proposal looked at other proprietary platforms that compete with ListenBrainz to see some of their statistics. I also came up with some of my own.

I proposed this to the MetaBrainz community on the community forums. I’m awaiting feedback on my ideas. Once I get feedback, I plan to file new tickets for each statistic to track their implementation over time.

I don’t expect statistics being at the forefront of ListenBrainz for some time. A lot of work is going towards other areas of the project. But later in 2018, I expect more focus on the user-facing side of the project.

My statistic and Google BigQuery 🔗

My biggest blocker over the last month was Google BigQuery. I wrote a statistic to calculate play counts over a time period, but was asked to test my statistic. To test my statistic, I needed real data to work with.

Originally, I tried using the Simple Last.fm Scrobbler to submit listens to the local IP address for my development environment, but I wasn’t able to get the app to reach my ListenBrainz server. To get the data, I had to set up Google BigQuery credentials so I could make queries against data on the production site, listenbrainz.org.

I tried working through the Google BigQuery documentation. There’s a lot of documentation for using BigQuery as a developer, but it was confusing where to find the information I needed to set it up in my development environment. I tried creating a new project in the Google Cloud Platform, but I was confused because it prompted me to upload my own data instead of accessing data already in BigQuery.

Too late, I realized I spent too much time on my own and not asking for help. I submitted a pull request with the statistic I made and asked for help in the MetaBrainz community. I also offered to write documentation for setting this up once I learn how to do it.

Reflecting back 🔗

I looked back on my original objectives for the independent study, and I was satisfied and dissatisfied.

Not enough programming 🔗

I wanted this independent study to enhance my programming knowledge. I especially wanted to focus on Python because I wanted to become more familiar with the language. However, I actually didn’t do much programming during the independent study, to my own fault.

My biggest challenge was I bit off more than I could chew. I wanted to write code, and made a big goal before I knew the code base of the project. Even now, I still am not completely comfortable with the code yet. It’s a big project with a lot of things going on. I was able to understand the things I did work on, but there’s still a lot.

I realized that next time, I need to spend more time evaluating the code base of a project before writing out my milestones. I wish I set more realistic, smaller milestones for myself. My milestone of implementing basic reports was lofty given my existing programming knowledge.

Successes 🔗

One of my other objectives was to write documentation for the project. I felt I succeeded in this milestone, and actually found it enjoyable and interesting to do! I helped separate out documentation from the README into the dedicated ReadTheDocs site. I wrote the development environment guide and helped fix some build issues with the docs site. I also plan to write more for some of the other pain points I found, like Google BigQuery.

My last milestone was to create a use case for a data visualization course at RIT. While I didn’t implement my basic reports, I did create the proposal and make an effort to write new statistics. There’s a lot of potential now to work with the data in Google BigQuery and do front-end work with tools like D3.js and Plotly.js. I believe there’s significant potential to use ListenBrainz as a hands-on project for students to explore data visualization with real data. I hope to support my independent study professor, Prof. Roberts, with questions and logistics of using it as a tool for learning in the future.

Unexpected success 🔗

I also think I had an unplanned success too. I immersed myself in the community for ListenBrainz too. Over the last few months, I realized that many of my strengths are in community management and tooling. During my time in the community, I did the following:

• Fixed SELinux labels in Docker
• Contributed a pull request template
• Drafted contributing guidelines
• Fixed a PostgreSQL bug
• And more…

To the future! 🔗

This ends my independent study with ListenBrainz, but it doesn’t end my time contributing! I chose ListenBrainz because it’s a project I’m passionate about. An independent study allowed me to justify more time on it than a side project in my free time. I’m happy to have that opportunity, but I don’t want to end here!

I want to follow through on the statistics because I’m passionate about understanding music listening trends. I think there’s a lot of power for psychological research through music data. To this point, I filed a ticket to request tagging listens with “emotion” words that are synced back to MusicBrainz entities.

I won’t have as much time to work on the project without the course credit, but I hope to stay involved for the future. I love the project and I love the community. I’m thankful for the opportunity to work on this project as an independent study, and learn some things along the way.

My progress with ListenBrainz slowed, but I am resuming the pace of contributing and advancing on my independent study timeline. This past week, I finished out assigned tasks to discuss contributor-related documentation, like a Code of Conduct, contributor guidelines, and a pull request template. I began research on user statistics and found some already created. I wrote one of my own, but need to learn more about Google BigQuery to advance further.

Paving the contributor pathway 🔗

Earlier, I identified weaknesses for the ListenBrainz contributor pathway and found ways we could improve the pathway. This started with the development environment documentation. Now, I helped draft first revisions of our contributor guidelines, Code of Conduct reference, and pull request templates. Together, these three documents have two goals.

1. Make it easier to contribute to ListenBrainz
2. Have a better experience and have fun contributing!

Adding these documents addresses these goals. Additionally, the GitHub community profile also highlights these deliverables as ways to meet these goals. After getting feedback and seeing what others think, we make more revisions later (with some trial runs).

Back to SELinux context flags 🔗

Recently, I set my desktop back up and installed Docker for the first time on this machine; however, the development environment still failed to start. When I ran the script, it would eventually error out because of a permission denial. The web server image for ListenBrainz was failing.

After debugging, I noticed that I missed the SELinux volume tags for the ListenBrainz web server images in my original pull request, #257. When I created the pull request, I might have had cached data that let my laptop run the development environment without a problem. In either case, it was an easy fix and I knew what the issue was when it happened. Therefore, I submitted a new fix in #290.

Writing new user statistics 🔗

The most interesting part of my independent study is working with the music data to build and generate interesting statistics. I finally began exploring the existing statistics in ListenBrainz. The statistic queries use BigQuery standard SQL. BigQuery helps rapidly scan and scale data queries to help with performance (I have a lot to learn about BigQuery).

Two types of statistics 🔗

Additionally, ListenBrainz generates two types of statistics:

1. Site-wide statistics
2. User statistics

Site-wide statistics are metrics non-specific to a single user. There is only one site-wide query now. It counts how many artists were ever submitted to this ListenBrainz instance and returns an integer. There’s room for expansion in site-wide statistics.

On the other hand, user statistics are metrics specific to a single user. There’s a fair number already, like the top artists and songs in a time period and the number of artists you’ve listened to. These are a little more complete and offer more expansion for doing cool front-end work with something like D3.js.

Writing user statistics 🔗

Of course, I had to try writing my own. One helpful query I thought of was getting a count of the songs you listened to over a time period (e.g. “you listened to 500 songs this week!”). I haven’t tested it yet, but I have this in a local branch and hope to test it with real data soon.

def get_play_count(musicbrainz_id, time_interval=None): 
 
 filter_clause = "" 
 if time_interval: 
     filter_clause = "AND listened_at >=
     TIMESTAMP_SUB(CURRENT_TIME(), 
     INTERVAL {})".format(time_interval) 
 
 query = """SELECT COUNT(release_msid) as listen_count 
            FROM {dataset_id}.{table_id} 
            WHERE user_name = @musicbrainz_id 
            {time_filter_clause} 
            LIMIT {limit} 
         """.format( 
                 dataset_id=config.BIGQUERY_DATASET_ID, 
                 table_id=config.BIGQUERY_TABLE_ID, 
                 time_filter_clause=filter_clause, 
                 limit=config.STATS_ENTITY_LIMIT, 
            ) 
 
 parameters = [ 
     { 
         'type': 'STRING', 
         'name': 'musicbrainz_id', 
         'value': musicbrainz_id 
     } 
 ] 
 
 return stats.run_query(query, parameters)

Researching Google BigQuery 🔗

My next steps for the independent study are researching Google BigQuery. After going through the existing statistics and understanding how ListenBrainz generates them, an understanding of Google BigQuery is essential to writing effective queries. When I become more comfortable with the tooling and how it works, I want to map out a plan of statistics to generate and measure.

Until then, the hacking continues! As always, keep the FOSS flag high…

Welcome back to the Kubernetes and Fedora series. Each week, we build on the previous articles in the series to help introduce you to using Kubernetes. This article picks up from where we left off last when you installed Tectonic to Amazon Web Services (AWS). By the end of this article, you will…

• Start up Redis master and slave pods
• Start a front-end pod that interacts with the Redis pods
• Deploy a simple web app for all of your friends to leave you messages

Compared to previous articles, this article will be a little more hands-on. Also like before, this is based off an excellent tutorial in the upstream Kubernetes documentation. Let’s get started!

Pre-requisites 🔗

This tutorial assumes you followed the Minikube how-to earlier in this series and that you already have a Tectonic installation running (doesn’t have to be on AWS). In case you’re jumping in now, make sure you have the Kubernetes client tools installed on your Fedora system, like kubectl. If not, you can install them now.

$ sudo dnf install kubernetes-client

Configure kubectl for Tectonic 🔗

To use kubectl with your Tectonic installation, you need to have a valid configuration in ~/.kube/config for your cluster. This is how kubectl knows where and how to talk to Tectonic. To get these values, first log into the Tectonic Console you installed.

1. Click username (usually admin) > My Account on the bottom left.
2. Click Download Configuration.
3. When the Set Up kubectl window opens, click Verify Identity.
4. Enter your username and password, and click Login.
5. From the Login Successful screen, copy the provided code.
6. Switch back to Tectonic and enter the code in the field.

Now you will be able to download kubectl-config from Tectonic. There’s two ways to proceed from here.

Add a new configuration 🔗

If this is your first time using kubectl, your configuration is likely empty. If it’s empty or you don’t care about overwriting an old configuration, you can run the following commands to add the configuration.

$ mkdir ~/.kube/
$ mv ~/Downloads/minikube-config ~/.kube/config
$ chmod 600 ~/.kube/config

Append to an existing configuration 🔗

If you already have a configuration, like from Minikube, you might not want to wipe it all out. In this case, you can merge the files manually together. You’ll need to copy the clusters, users, and contexts from the Tectonic configuration into your existing one. The benefit of doing this is that you’ll be able to change contexts to switch from one cluster to another.

Test your configuration 🔗

Once you finished your configuration, test to see if it works.

$ kubectl config use-context tectonic       # if you have multiple contexts in config
$ kubectl get nodes
NAME                                        STATUS    AGE
ip-10-0-0-59.us-east-2.compute.internal     Ready     1d
ip-10-0-23-239.us-east-2.compute.internal   Ready     1d
ip-10-0-44-211.us-east-2.compute.internal   Ready     1d
ip-10-0-61-218.us-east-2.compute.internal   Ready     1d
ip-10-0-67-239.us-east-2.compute.internal   Ready     1d
ip-10-0-95-51.us-east-2.compute.internal    Ready     1d

Huzzah! Now we’re ready to get to work.

Getting the deployment and service files 🔗

All of the example files come from the official Kubernetes GitHub repo. You can find them in the Guestbook example. To get started, create a new directory and download all of the files.

$ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/examples/guestbook/redis-{master,slave}-{deployment,service}.yaml \
       https://raw.githubusercontent.com/kubernetes/kubernetes/master/examples/guestbook/frontend-{deployment,service}.yaml

We’ll explain what all of these do in next steps. All of these next steps will start with the command to run, followed by a short explanation of what’s actually happening.

Start the Redis master 🔗

$ kubectl create -f redis-master-service.yaml
service "redis-master" created
$ kubectl create -f redis-master-deployment.yaml
deployment "redis-master" created

Define the deployment 🔗

The redis-master-deployment.yaml file downloaded earlier defines the deployment and its characteristics. In this case, we have one pod that runs the Redis master in a container. Since we’re using a deployment, that means if our pod goes down, Kubernetes will spin up a new pod to replace it. Worth noting in this example, if the pod did go down, there would be a potential for data loss until the new one replaces the old one (since the Redis master is not highly available, i.e. there are multiple).

Define the service 🔗

Our service in this example is a named load balancer that proxies traffic across one or many containers. Even though we only have one Redis master pod, we still want to use a service. This is a deterministic way of making the route to the master with a dynamic (or elastic) IP address.

Labeling the pods is important in this case, as Kubernetes will use the pods’ labels to determine which pods receive the traffic sent to the service, and load balance it accordingly.

Create the service 🔗

The next important step is to create the service. Note that we’re doing this before we create the deployment. It’s best practice to create the service first. This allows the scheduler to later spread the service across the deployments you create to support your application.

After creating the service, you can check its status by running this command. You should see similar output.

$ kubectl get services
NAME              CLUSTER-IP       EXTERNAL-IP       PORT(S)       AGE
redis-master      10.0.76.248      <none>            6379/TCP      1s

Now your Redis master serivce is up and running! The next step will be to create the Redis master deployment.

If you look at the service configuration file, you’ll notice port and targetPort are two defined variables. Once everything is up and running, these will be important for determining how the traffic from the slaves to the masters is routed.

1. Redis slave connects to port on Redis master service
2. Traffic forwarded from service’s port to targetPort on pod the service listens to

Create the deployment 🔗

Next, we created the Redis master pod in the cluster. To see our deployment and pods, we can run the following commands to see what was created.

$ kubectl get deployments
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
redis-master   1         1         1            1           27s

$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
redis-master-2353460263-1ecey   1/1       Running   0          1m
...

You should see all of the pods in your cluster so far. For now, that’s just the Redis master. Let’s give it some friends!

Start the Redis slaves 🔗

$ kubectl create -f redis-slave-service.yaml
service "redis-slave" created
$ kubectl create -f redis-slave-deployment.yaml
deployment "redis-slave" created

Defining the deployment 🔗

In the configuration file, we defined two replicas, unlike the master. By doing this, it tells Kubernetes that the minimum number of pods that should always be running is two. If one of your pods goes down, Kubernetes automatically creates a new one to support the application. If you want, you can try killing the Docker process for one of your pods to see it happen in real time.

Start the guestbook front-end 🔗

$ kubectl create -f frontend-service.yaml
service "frontend" created
$ kubectl create -f frontend-deployment.yaml
deployment "frontend" created

The front-end is a PHP application with an AJAX interface and Angular-based UI. When using the form on the front-end application, it talks to the Redis master or slave, depending on if it’s reading or writing to Redis. Again, we’re deploying the front-end with multiple replicas. In this case, there will be three pods to support the front-end.

Say hello! 🔗

Once you’ve finished deploying everything, your web app should now be accessible! To get the full domain from AWS, run this command to figure out where to look.

$ kubectl get deploy/frontend svc/frontend -o wide
NAME           CLUSTER-IP   EXTERNAL-IP                                                             PORT(S)        AGE       SELECTOR
svc/frontend   10.3.0.175   aaebd8247ef2311e6a045021d1620193-54019671.us-east-2.elb.amazonaws.com   80:31020/TCP   1m        k8s-app=guestbook,tier=frontend

NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/frontend   3         3         3            3           1m

Congratulations, we’re all finished!

Cleaning up 🔗

Once you’re finished or when you want to stop running the guestbook, it’s easy to get rid of the deployments and services we created. Using labels, all the deployments and services can be deleted with one command.

$ kubectl delete deployments,services -l "app in (redis, guestbook)"

And now your guestbook application is offline. (It was nice while it lasted!)

Learn more about Kubernetes and Tectonic 🔗

If you want to explore more about Kubernetes, you can read some of the earlier articles in this series. You can also read the original tutorial published by Kubernetes on GitHub. Additionally, the upstream documentation for Kubernetes and Tectonic is thorough and can help answer more advanced questions.

Questions, Tectonic stories, or tips for beginners? Add your comments below.

Welcome back to the Kubernetes and Fedora series. Each week, we build on the previous articles in the series to help introduce you to using Kubernetes. This article takes off from running Kubernetes on your own hardware and moves us one step closer to the cloud. By the end of this article, you will…

• Understand what CoreOS Tectonic is
• Set up Amazon Web Services (AWS) for Tectonic
• Deploy Tectonic to AWS

This article is also based off of the excellent tutorial provided in the CoreOS documentation. Let’s get started!

What is Tectonic? 🔗

In the first article, some of the key concepts of Kubernetes and why it’s useful were explained. Kubernetes automates the deployment and setting up of your infrastructure across the three layers (users, masters, nodes). If you’re working on your own at a small scale, Kubernetes itself can be plenty to meet your needs. However, there is still a decent amount of human involvement in managing the different pieces of Kubernetes. If you’re working with multiple people in a team and across different environments, vanilla Kubernetes can be a lot to manage. For an enterprise environment, there’s still some unmet needs. This is where Tectonic steps in.

Tectonic is a commercial product offered by CoreOS, the providers of Container Linux and the original developers of etcd, now one of the core components of Kubernetes. Tectonic takes all of the open source components and pre-packages them. The self-proclaimed goal of doing this is to let anyone build a Google-style infrastructure into a cloud or on-premise environment. The outcome for the user is that it’s easy to install a Kubernetes infrastructure across many different environments. In addition to simplifying the installation of the various components of a Kubernetes stack, Tectonic also provides a management console, a container registry for building and sharing containers, additional tools for deployment, and a few other nice features.

If we think about Kubernetes as a cake like we did before with three layers, Tectonic is like the box you set it in. Now, you can take your cake anywhere, move it around, and stack it with other cakes-in-a-box. All of your cakes are in their own boxes and you don’t have to worry about them accidentally being damaged. If you’re still a little confused, this diagram might help make more sense of it.

Fortunately, Tectonic has a free license that lets you use it for ten nodes. In this example, we’ll register, get a free license, and deploy it into AWS.

(Note: If you want to revert anything we do in this example, there’s an easy way to dismantle it across AWS and bring your bill to $0.00.)

Pre-requisites 🔗

In order to successfully run this guide, there’s a few things you’ll need first.

• Amazon Web Services (AWS) account (free)
  • Register here
• CoreOS Tectonic account and license (free)
  • Register here
• A root-level or sub-domain (e.g. example.com or k8s.example.com)
  • If you look around, you can probably find some for less than USD$1 a year if you need one
• Curiosity!

Setting up DNS with Route 53 🔗

The first things we’ll do is set up our domain with Route 53 in AWS. Route 53 can do a lot of things, like DNS management, traffic management, availability monitoring, domain registration, and more. However, we’re only going to be using it for DNS management. Tectonic will use this to automatically provision DNS records for internal and external use.

Add your domain 🔗

To add your domain to Route 53, follow these steps from AWS.

1. From Services, select Networking & Content Delivery > Route 53.
2. Select Hosted zones from the left pane and click Create Hosted Zone.
3. Enter your domain or sub-domain, add a comment if you want, and choose a Public Zone for the type.

Once you’ve done this, you can go ahead and click “Create”.

Change the nameservers 🔗

After adding the hosted zone to Route 53, you’ll need to change the nameservers for your domain via the domain registrar (whoever you bought the domain from). Usually it should be easy to find this, but it varies among registrars. If you’re having a hard figuring out how to do this, try searching for a how-to or contacting your registrar’s support.

After you added the hosted zone, you should see the nameservers in Route 53. There will be four nameservers provided there. You can copy and paste them from Route 53 to your domain registrar. Also note that if you’re using a subdomain, the instructions might be a little different. You can read how to do this in the Route 53 documentation.

The nameservers could take minutes or hours to update, depending on how lucky you are. If you’re impatient and want to check, open up a terminal and run this command. If you see the AWS nameservers in the output, then your domain has propagated and is now usable by Route 53.

dig -t ns <example.com>

Configuring EC2 with SSH key pair 🔗

This guide assumes you already have an SSH key pair created on your system. If you don’t have one generated, you can read how to generate one here.

The next step for us is to add an SSH key pair to EC2, one of the compute engine products offered by AWS. We’ll import an existing key on your system into EC2.

1. From AWS, go to Services > Compute > EC2.
2. Confirm that you are in the correct EC2 region by checking the location next to your name in the menu bar.
3. Under Network & Security, click Key Pairs.
4. Click Import Key Pair.
5. Either upload your public key file (~/.ssh/id_rsa.pub) or paste it into the text field. Don’t forget to give it a name.

And that’s all you need to do!

Assigning AWS user privileges 🔗

Tectonic does the magic of setting up AWS for you, so you don’t have to manually add and create the services from the web interface. In order to do this, you need to add a user account that Tectonic can use to do all of the provisioning it needs. To do this, you’ll need to create a new Access ID and Secret key pair from AWS.

1. Select Services > Security, Identity & Compliance > IAM.
2. From the left hand pane, click Users, then click Add user.
3. Set the user details:
   1. User name can be anything you like (I used tectonic-mydomain.com)
   2. Access type only needs to be Programmatic access
4. For permissions, click Add user to group and create a new group for your user.
5. When creating a new group, attach only the policies needed by Tectonic to operate correctly:
   1. AmazonEC2FullAccess
   2. IAMFullAccess
   3. AmazonS3FullAccess
   4. AmazonVPCFullAccess
   5. AmazonRoute53FullAccess
6. Finish creating the user. You’ll then see the Access key ID and Secret access key. Hold onto these, you’ll need them later. You won’t get to see the secret key again!

Now we’re ready to install Tectonic! Let’s grab your credentials next.

Download Tectonic credentials 🔗

Jump back over to the CoreOS accounts page. When you’re logged in, you’ll see the Account Assets area. Download the CoreOS license file and pull secret. Later on in the installer, you’ll need to insert these to finish the installation.

Running the installer 🔗

Now things get interesting! We finally get to install and deploy Tectonic into AWS. The installer takes the form of a graphical installer in your web browser. To use the installer, you need to download the binary and run it. If you’re curious, you can find the installer source code on GitHub.

Download and run installer 🔗

First, open up a new terminal window and navigate to a directory you want to download the installer to. Even though you likely won’t need to run the installer again, you will want to hang on to this if you ever want to easily dismantle everything in AWS later.

curl -O https://releases.tectonic.com/tectonic-1.6.4-tectonic.1.tar.gz

Next, extract the tarball and navigate into the directory.

tar -xzvf tectonic-1.6.4-tectonic.1.tar.gz
cd tectonic/tectonic-installer

Now execute the installer binary. After running this, a new browser window will open that features the graphical installer.

./linux/installer

Running the installer 🔗

The installer is thorough and assumes safe defaults for most of the steps. Be sure to have your AWS Access and Secret ID keys on hand. You should be able to run through the installer without issue. If you’re confused about what any of the values mean or want to make custom changes, you can read more in the upstream documentation.

Once you’re finished, congrats! You’ve successfully installed Tectonic!

Check out your Tectonic install 🔗

Once you finish the installation successfully, your Tectonic installation will be accessible within AWS. You can navigate to the domain you specified during the install to find it. Unless you added a CA authority and certificates, your browser will probably complain about invalid SSL certificates, but you can ignore the warning safely. It might also take a few minutes before the URL is accessible, so if you were looking for a coffee or tea break, now would be a good time!

Once you’re logged in, you should see something like this.

Blow it all away! 🔗

If you’re like me, you might be frustrated by guides that tell you how to install things but not how to take it all apart. Fortunately, this guide not only tells you how to do that, but the Tectonic installer also makes it super easy to do. If you’re sure that you’re done with Tectonic and don’t want any leftovers to remain in AWS, this is the best way to do it, instead of deleting everything manually from the AWS Console.

Every installation has a time-stamped folder in the tectonic directory we used earlier. First, you need to navigate into the specific folder for the cluster you installed. It’s important to be inside of this directory first.

cd tectonic/tectonic-installer/linux/clusters/<CLUSTERNAME>

<CLUSTERNAME> will be the time-stamped directory. Once you’re in the folder, run this command to trigger the uninstaller. After running this, you’ll see the installer slowly dismantle everything and delete any leftovers in AWS.

../../terraform destroy

Once it finishes, you should see an output message confirming how many AWS resources were destroyed. And now you’re back to where you started.

Learn more about Tectonic 🔗

If you thought this was exciting and want to learn more, there is no shortage of resources for you to read. You can learn more about Tectonic from the CoreOS website or the original release announcement. You can also dig into the installer’s source code on GitHub. If you’re still trying to wrap your head around Tectonic, there’s a good write-up on virtualizationreview.com.

Next week, we’ll install a simple guestbook application to our Tectonic installation to see how it all works and what you can do with it. Stay tuned!

Questions, Tectonic stories, or tips for beginners? Add your comments below.

This is a short series to introduce Kubernetes, what it does, and how to experiment with it on Fedora. This is a beginner-oriented series to help introduce some higher level concepts and give examples of using it on Fedora. In the first post, we covered key concepts in Kubernetes. This second post shows you how to build a single-node Kubernetes deployment on your own computer.

Once you have a better understanding of what the key concepts and terminology in Kubernetes are, getting started is easier. Like many programming tutorials, this tutorial shows you how to build a “Hello World” application and deploy it locally on your computer using Kubernetes. This is a simple tutorial because there aren’t multiple nodes to work with. Instead, the only device we’re using is a single node (a.k.a. your computer). By the end, you’ll see how to deploy a Node.js application into a Kubernetes pod and manage it with a deployment on Fedora.

This tutorial isn’t made from scratch. You can find the original tutorial in the official Kubernetes documentation. This article adds some changes that will let you do the same thing on your own Fedora computer.

Introducing Minikube 🔗

Minikube is an official tool developed by the Kubernetes team to help make testing it out easier. It lets you run a single-node Kubernetes cluster through a virtual machine on your own hardware. Beyond using it to play around with or experiment for the first time, it’s also useful as a testing tool if you’re working with Kubernetes daily. It does support many of the features you’d want in a production Kubernetes environment, like DNS, NodePorts, and container run-times.

Installation 🔗

This tutorial requires virtual machine and container software. There are many options you can use. Minikube supports virtualbox, vmwarefusion, kvm, and xhyve drivers for virtualization. However, this guide will use KVM since it’s already packaged and available in Fedora. We’ll also use Node.js for building the application and Docker for putting it in a container.

Pre-requirements 🔗

You can install the prerequisites with this command.

$ sudo dnf install kubernetes libvirt-daemon-kvm kvm nodejs docker

After installing these packages, you’ll need to add your user to the right group to let you use KVM. The following commands will add your user to the group and then update your current session for the group change to take effect.

$ sudo usermod -a -G libvirt $(whoami)
$ newgrp libvirt

Docker KVM drivers 🔗

If using KVM, you will also need to install the KVM drivers to work with Docker. You need to add Docker Machine and the Docker Machine KVM Driver to your local path. You can check their pages on GitHub for the latest versions, or you can run the following commands for specific versions. These were tested on a Fedora 25 installation.

Docker Machine 🔗

$ curl -L https://github.com/docker/machine/releases/download/v0.12.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine
$ chmod +x /tmp/docker-machine
$ sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

Docker Machine KVM Driver 🔗

This installs the CentOS 7 driver, but it also works with Fedora.

$ curl -L https://github.com/dhiltgen/docker-machine-kvm/releases/download/v0.10.0/docker-machine-driver-kvm-centos7 >/tmp/docker-machine-driver-kvm
$ chmod +x /tmp/docker-machine-driver-kvm
$ sudo cp /tmp/docker-machine-driver-kvm /usr/local/bin/docker-machine-driver-kvm

Installing Minikube 🔗

The final step for installation is getting Minikube itself. Currently, there is no package in Fedora available, and official documentation recommends grabbing the binary and moving it your local path. To download the binary, make it executable, and move it to your path, run the following.

$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
$ chmod +x minikube
$ sudo mv minikube /usr/local/bin/

Now you’re ready to build your cluster.

Create the Minikube cluster 🔗

Now that you have everything installed and in the right place, you can create your Minikube cluster and get started. To start Minikube, run this command.

$ minikube start --vm-driver=kvm

Next, you’ll need to set the context. Context is how kubectl (the command-line interface for Kubernetes) knows what it’s dealing with. To set the context for Minikube, run this command.

$ kubectl config use-context minikube

As a check, make sure that kubectl can communicate with your cluster by running this command.

$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Build your application 🔗

Now that Kubernetes is ready, we need to have an application to deploy in it. This article uses the same Node.js application as the official tutorial in the Kubernetes documentation. Create a folder called hellonode and create a new file called server.js with your favorite text editor.

var http = require('http');

var handleRequest = function(request, response) {
 console.log('Received request for URL: ' + request.url);
 response.writeHead(200);
 response.end('Hello world!');
};
var www = http.createServer(handleRequest);
www.listen(8080);

Now try running your application and running it.

$ node server.js

While it’s running, you should be able to access it on localhost:8080. Once you verify it’s working, hit Ctrl+C to kill the process.

Create Docker container 🔗

Now you have an application to deploy! The next step is to get it packaged into a Docker container (that you’ll pass to Kubernetes later). You’ll need to create a Dockerfile in the same folder as your server.js file. This guide uses an existing Node.js Docker image. It exposes your application on port 8080, copies server.js to the image, and runs it as a server. Your Dockerfile should look like this.

FROM node:6.9.2
EXPOSE 8080
COPY server.js .
CMD node server.js

If you’re familiar with Docker, you’re likely used to pushing your image to a registry. In this case, since we’re deploying it to Minikube, you can build it using the same Docker host as the Minikube virtual machine. For this to happen, you’ll need to use the Minikube Docker daemon.

$ eval $(minikube docker-env)

Now you can build your Docker image with the Minikube Docker daemon.

$ docker build -t hello-node:v1 .

Huzzah! Now you have an image Minikube can run.

Create Minikube deployment 🔗

If you remember from the first part of this series, deployments watch your application’s health and reschedule it if it dies. Deployments are the supported way of creating and scaling pods. kubectl run creates a deployment to manage a pod. We’ll create one that uses the hello-node Docker image we just built.

$ kubectl run hello-node --image=hello-node:v1 --port=8080

Next, check that the deployment was created successfully.

$ kubectl get deployments
NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
hello-node   1         1         1            1           30s

Creating the deployment also creates the pod where the application is running. You can view the pod with this command.

$ kubectl get pods
NAME                          READY     STATUS    RESTARTS   AGE
hello-node-1644695913-k2314   1/1       Running   0          3

Finally, let’s look at what the configuration looks like. If you’re familiar with Ansible, the configuration files for Kubernetes also use easy-to-read YAML. You can see the full configuration with this command.

$ kubectl config view

kubectl does many things. To read more about what you can do with it, you can read the documentation.

Create service 🔗

Right now, the pod is only accessible inside of the Kubernetes pod with its internal IP address. To see it in a web browser, you’ll need to expose it as a service. To expose it as a service, run this command.

$ kubectl expose deployment hello-node --type=LoadBalancer

The type was specified as a LoadBalancer because Kubernetes will expose the IP outside of the cluster. If you were running a load balancer in a cloud environment, this how you’d provision an external IP address. However, in this case, it exposes your application as a service in Minikube. And now, finally, you get to see your application. Running this command will open a new browser window with your application.

$ minikube service hello-node

Congratulations, you deployed your first containerized application via Kubernetes! But now, what if you need to our small Hello World application?

How do we push changes? 🔗

The time has come when you’re ready to make an update and push it. Edit your server.js file and change “Hello world!” to “Hello again, world!”

response.end('Hello again, world!');

And we’ll build another Docker image. Note the version bump.

$ docker build -t hello-node:v2 .

Next, you need to give Kubernetes the new image to deploy.

$ kubectl set image deployment/hello-node hello-node=hello-node:v2

And now, your update is pushed! Like before, run this command to have it open in a new browser window.

$ minikube service hello-node

If your application doesn’t come up any different, double-check that you updated the right image. You can troubleshoot by getting a shell into your pod by running the following command. You can get the pod name from the command run earlier (kubectl get pods). Once you’re in the shell, check if the server.js file shows your changes.

$ kubectl exec -it <pod-name> bash

Cleaning up 🔗

Now that we’re done, we can clean up the environment. To clear up the resources in your cluster, run these two commands.

$ kubectl delete service hello-node
$ kubectl delete deployment hello-node

If you’re done playing with Minikube, you can also stop it.

$ minikube stop

If you’re done using Minikube for a while, you can unset Minikube Docker daemon that we set earlier in this guide.

$ eval $(minikube docker-env -u)

Learn more about Kubernetes 🔗

You can find the original tutorial in the Kubernetes documentation. If you want to read more, there’s plenty of great information online. The documentation provided by Kubernetes is thorough and comprehensive.

Questions, Minikube stories, or tips for beginners? Add your comments below.

This article is part of a short series that introduces Kubernetes. This beginner-oriented series covers some higher level concepts and gives examples of using Kubernetes on Fedora.

The information technology world changes daily, and the demands of building scalable infrastructure become more important. Containers aren’t anything new these days, and have various uses and implementations. But what about building scalable, containerized applications? By itself, Docker and other tools don’t quite cut it, as far as building the infrastructure to support containers. How do you deploy, scale, and manage containerized applications in your infrastructure? This is where tools such as Kubernetes comes in. Kubernetes is an open source system that automates deployment, scaling, and management of containerized applications. Kubernetes was originally developed by Google before being donated to the Cloud Native Computing Foundation, a project of the Linux Foundation. This article gives a quick precursor to what Kubernetes is and what some of the buzzwords really mean.

What is Kubernetes? 🔗

Kubernetes simplifies and automates the process of deploying containerized applications at scale. Just like Ansible orchestrates software, Kubernetes orchestrates deploying infrastructure that supports the software. There are various “layers of the cake” that make Kubernetes a strong solution for building resilient infrastructure. It also assists with making systems that can grow at scale. If your application has increasing demands such as higher traffic, Kubernetes helps grow your environment to support increasing demands. This is one reason why Kubernetes is helpful for building long-term solutions for complex problems (even if it’s not complex… yet).

At a high level overview, imagine three different layers.

• Users: People who deploy or create containerized applications to run in your infrastructure
• Master(s): Manages and schedules your software across various other machines, for example in a clustered computing environment
• Nodes: Various machines to support the application, called kubelets

These three layers are orchestrated and automated by Kubernetes. One of the key pieces of the master (not included in the visual) is etcd. etcd is a lightweight and distributed key/value store that holds configuration data. Each node, or kubelet, can access this data in etcd through a HTTP/JSON API interface. The components of communication between master and node such as etcd are explained in the official documentation.

Another important detail not shown in the diagram is that you might have many masters. In a high-availability (HA) set-up, you can keep your infrastructure resilient by having multiple masters in case one happens to go down.

Terminology 🔗

It’s important to understand the concepts of Kubernetes before you start to play around with it. There are many core concepts in Kubernetes, such as services, volumes, secrets, daemon sets, and jobs. However, this article explains four that are helpful for the next exercise of building a mini Kubernetes cluster. The three concepts are pods, labels, replica sets, and deployments.

Pods 🔗

If you imagine Kubernetes as a Lego® castle, pods are the smallest block you can pick out. By themselves, they are the smallest unit you can deploy. The containers of an application fit into a pod. The pod can be one container, but it can also be as many as needed. Containers in a pod are unique since they share the Linux namespace and aren’t isolated from each other. In a world before containers, this would be similar to running an application on the same host machine.

When the pods share the same namespace, all the containers in a pod:

• Share an IP address
• Share port space
• Find each other over localhost
• Communicate over IPC namespace
• Have access to shared volumes

But what’s the point of having pods? The main purpose of pods is to have groups of “helping” containers on the same namespace (co-located) and integrated together (co-managed) along with the main application container. Some examples might be logging or monitoring tools that check the health of your application, or backup tools that act when certain data changes.

In the big picture, containers in a single pod are always scheduled together too. However, Kubernetes doesn’t automatically reschedule them to a new node if the node dies (more on this later).

Labels 🔗

Labels are a simple but important concept in Kubernetes. Labels are key/value pairs attached to objects in Kubernetes, like pods. They let you specify unique attributes of objects that actually mean something to humans. You can attach them when you create an object, and modify or add them later. Labels help you organize and select different sets of objects to interact with when performing actions inside of Kubernetes. For example, you can identify:

• Software releases: Alpha, beta, stable
• Environments: Development, production
• Tiers: Front-end, back-end

Labels are as flexible as you need them to be, and this list isn’t comprehensive. Be creative when thinking of how to apply them.

Replica sets 🔗

Replica sets are where some of the magic begins to happen with automatic scheduling or rescheduling. Replica sets ensure that a number of pod instances (called replicas) are running at any moment. If your web application needs to constantly have four pods in the front-end and two in the back-end, the replica sets are your insurance that number is always maintained. This also makes Kubernetes great for scaling. If you need to scale up or down, change the number of replicas.

When reading about replica sets, you might also see replication controllers. They are somewhat interchangeable, but replication controllers are older, semi-deprecated, and less powerful than replica sets. The main difference is that sets work with more advanced set-based selectors – which goes back to labels. Ideally, you won’t have to worry about this much today.

Even though replica sets are where the scheduling magic happens to help make your infrastructure resilient, you won’t actually interact with them much. Replica sets are managed by deployments, so it’s unusual to directly create or manipulate replica sets. And guess what’s next?

Deployments 🔗

Deployments are another important concept inside of Kubernetes. Deployments are a declarative way to deploy and manage software. If you’re familiar with Ansible, you can compare deployments to the playbooks of Ansible. If you’re building your infrastructure out, you want to make sure it is easily reproducible without much manual work. Deployments are the way to do this.

Deployments offer functionality such as revision history, so it’s always easy to rollback changes if something doesn’t work out. They also manage any updates you push out to your application, and if something isn’t working, it will stop rolling out your update and revert back to the last working state. Deployments follow the mathematical property of idempotence, which means you define your specs once and use them many times to get the same result.

Deployments also get into imperative and declarative ways to build infrastructure, but this explanation is a quick, fly-by overview. You can read more detailed information in the official documentation.

Installing on Fedora 🔗

If you want to start playing with Kubernetes, install it and some useful tools from the Fedora repositories.

sudo dnf install kubernetes

This command provides the bare minimum needed to get started. You can also install other cool tools like cockpit-kubernetes (integration with Cockpit) and kubernetes-ansible (provisioning Kubernetes with Ansible playbooks and roles).

Learn more about Kubernetes 🔗

If you want to read more about Kubernetes or want to explore the concepts more, there’s plenty of great information online. The documentation provided by Kubernetes is fantastic, but there are also other helpful guides from DigitalOcean and Giant Swarm. The next article in the series will explore building a mini Kubernetes cluster on your own computer to see how it really works.

Questions, Kubernetes stories, or tips for beginners? Add your comments below.