However, the times are a-changing! Containers are entering as real players in the HPC space. Previously, containers were brushed off as incompatible with most HPC workflows. Now, several open source projects are emerging with unique approaches to enabling containers for HPC workloads. This blog post evaluates four container run-times in an HPC context, as they stand in July 2019:

• Charliecloud
• Shifter
• Singularity
• Podman

Research requirements 🔗

My research focused around a specific set of requirements. To receive a favorable review, a container run-time needed to meet three basic requirements:

• Support CentOS/RHEL 7.5+
• Compatibility with Univa GridEngine
• Support for very large numbers of users

Obviously there are security concerns with the third requirement. This is one reason containers have not made a strong showing in the HPC world yet. With the Docker security model, root access is a requirement to build and run containers. In a production HPC environment where users do not trust other users, this is a hard blocker.

Other HPC environments may differ. If you are an HPC administrator and also considering containers in your environment, consider my requirements. My research was exclusively framed through these three requirements.

Charliecloud 🔗

Charliecloud is an open source project based on a user-defined software stack (UDSS). Like most container implementations, it uses Linux user namespaces to run unprivileged containers. It is designed to be as minimal and lightweight as possible, to the point of not adding features that could conflict with any specific use cases. This can be a positive or a negative, depending on how complex your environment is.

However, I abandoned my research on Charliecloud early on after reading this PLOS research paper:

The software makes use of kernel namespaces that are not deemed stable by multiple prominent distributions of Linux (e.g. no versions of Red Hat Enterprise Linux or compatibles support it), and may not be included in these distributions for the foreseeable future.

The software is emphasized for its simplicity and being less than 500 lines of code, and this is an indication of having a lack of user-driven features. The containers are not truly portable because they must be extracted from Docker and configured by an external C executable before running, and even after this step, all file ownership and permissions are dependent on the user running the workflow.

Singularity: Scientific containers for mobility of compute, May 2017 (Gregory M. Kurtzer, Vanessa Sochat, Michael W. Bauer)

However, it is worth noting this paper was written in support of Singularity. It was also written by the Singularity project lead and others from the Singularity open source community. If you are conducting your own independent research, consider looking closer at Charliecloud, since at the time of writing it is still actively developed. The research paper was written in May 2017.

Edit: This situation already changed and Charliecloud is probably worth a deeper look:

This is a fantastic write up, one thing to mention is that abandoning the CharlieCloud research based solely on lack of support of the user kernel namespace is no longer a blocker. For example, PodMan now uses the same technology and it was released in RHEL8.

— Apptainer (formerly Singularity) (@SingularityApp) August 20, 2019

Shifter 🔗

Shifter is another container run-time implementation focused on HPC users. At time of writing, it is almost exclusively backed by the National Energy Research Scientific Computing Center and Cray. Most documented use cases use Slurm for cluster management / job scheduling. Instead of a Docker/OCI format, it uses its own Shifter-specific format, but this is reverse-compatible with Docker container images. It requires hosting a registry service and a Shifter Image Gateway.

The Shifter Image Gateway is a REST interface implemented with Python Flask. It pulls images from the registry service and converts them to the Shifter image format. MPI integration is supported but its implementation is MPICH-centric.

The downside to Shifter is lack of community. There are not many other organizations other than NERSC and Cray that appear to support Shifter. Documentation exists, but at writing time (July 2019), the last significant contribution was April 2018. Some bugs and feature requests are triaged, but there is not much of a maintainer presence in these issues. Most follow-up discussion to new issues are from a handful of outside contributors without commit access.

Additionally, there are several signs of stagnant development, such as NERSC/shifter#172 to add better MPI integration. However, the PR stalled out since it was first opened in April 2017. Furthermore, there is a high bus factor: most contributions and pull requests come from the same two developers, indicating low engagement from the wider HPC community. Code is regularly tested, but integration tests only exist for Slurm. For more details, check out the GitHub project pulse.

A detail worth noting is Shifter was one of the first real container run-times for HPC. A former Shifter collaborator branched off from Shifter to start Singularity (and eventually, a for-profit company to support it, Sylabs). It invites room for personal biases when evaluating Shifter and Singularity, specifically if you are not a newcomer in the HPC community.

Singularity 🔗

Singularity is the third and last HPC-specific player in the container run-time world. The vendor is Sylabs Inc. There are a few different factors that make Singularity interesting, and in my opinion, the most promising HPC container implementation.

General overview 🔗

Singularity v3.x.x is written almost entirely in Golang. It supports two image formats: Docker/OCI and Singularity’s native Single Image Format (SIF). As of September 2018, there are an estimated 25,000+ systems running Singularity, including users like TACC, San Diego Supercomputer Center, and Oak Ridge National Laboratory. Additionally, Univa announced a partnership with Sylabs in July 2018 to bring Singularity workflows to Univa GridEngine.

Sylabs offers Singularity (free and open source) and SingularityPRO (paid and proprietary). The commercial version comes with a support contract and long-term support for some releases (among other things).

Admin/root access is not required to run Singularity containers and it requires no additional configuration to do this out of the box. Containers are run under the Linux user ID that launches them (see Security and privilege escalation).

At a quick glance, Sylabs developers appear to be actively engaged in the Kubernetes development community, particularly around Red Hat technology. They also seem to keep their promises: in early 2018, blog posts show ambitious feature promises for the then-upcoming v3.0.0 release at the end of the year. Near the end of 2018, the release was delivered on-time with most/all of the promised functionality.

Image formats 🔗

The Singularity Image Format (SIF) is a single-image format (i.e. no layers involved). This was a design decision specifically for HPC workloads. SIFs are treated like a binary executable by a Linux user. Additionally, it is possible to create SIFs using the Definition File spec.

However, Singularity is also compatible with Docker/OCI images and OCI is given active development focus by upstream Singularity. Docker/OCI images are converted on-the-fly to a SIF. Docker/OCI images can be used locally or pulled from a remote registry like Docker Hub or Quay. To the user, if using a Docker/OCI image, the conversion is seamless and does not require additional configuration to use.

See this Sylabs blog post for a deeper dive on how SIFs were designed.

Flexible configuration 🔗

Singularity (uniquely?) offers advanced configuration options for HPC administrators. Some highlights are detailed here:

• Controlling bind mounts:
  • mount dev = minimal: Only binds null, zero, random, urandom, and shm into container
  • mount home = {yes,no}, mount tmp = {yes,no}: Choose to enable or disable these bind mounts globally
  • bind path = "": Bind specific paths into containers by default
  • user bind control = {yes,no}: Allow users to include their own bind mount paths or limit it to an admin-approved set of paths (above)
• Controlling containers:
  • limit container paths =: Possible to limit SIFs provided at a specific path and nowhere else

HPC community engagement 🔗

These notes only apply to Singularity free, not the proprietary SingularityPRO product.

The signals from their open source community engagement are positive and strong. They appear authentic and genuine to an open source commitment (i.e. not open-core business model). This is demonstrated in a few ways:

First, they have thorough user documentation, intended for end-users in HPC environments using Singularity. They have a less thorough but still useful admin documentation.

Second, all issues are triaged quickly and get feedback from core developers or outside contributors at a consistent pace. Pull requests don’t stagnate either: the oldest PR is less than six months old.

Third, code is regularly tested (1, 2). The code generally follows best practices (i.e. it is not atrocious to work with).

Fourth, there are also a handful of active contributors (both developers and in the community support channels) who come from outside of Sylabs, which indicates more engagement by a wider audience of people.

For more statistics, check out the GitHub project pulse.

Podman 🔗

tl;dr: Podman is an underdog that shows promise, but likely needs another one or two years of time for most HPC use cases.

Podman is a container run-time developed by Red Hat. Its primary goal is to be a drop-in replacement for Docker. While it is not explicitly designed with HPC use cases in mind, it intends to be a lightweight “wrapper” to run containers without the overhead of the full Docker daemon. Furthermore, the Podman development team is recently looking into better support for HPC use cases.

Podman is currently lacking for a HPC use case for some of these reasons:

1. Missing support for parallel filesystems (e.g. IBM Spectrum Scale)
2. Rootless Podman was designed to use kernel user namespaces which is not compatible with most parallel filesystems (might change in a year or two)
3. Not yet possible to set system site policy defaults
4. Pulling Docker/OCI images requires multiple subuids/subgids (might change in a year or two)

Where Podman does shine is providing a way to run and build containers without root access or setuid.

The same challenges and problems required for Podman to run OCI containers in an HPC environment are the same problems faced by Singularity to build SIF images without root in the HPC environment: mapping UIDs to subuids/subgids on the compute nodes. More interestingly, Buildah offers a promising way to enable users to build container images as Docker/OCI images all without root. It is plausible to use Buildah as the container image delivery mechanism and swap out the container run-time implementation (Podman vs. Singularity) depending on specific needs and requirements.

What do you think? 🔗

I hope other folks out there in the HPC world find this preliminary research useful. Do you agree or disagree with any parts of this write-up? Is something out-of-date? Drop a comment down below.

Open source projects are built online and a lot of their community members are placed all over the world. Even though projects have people from around the world, this doesn’t stop ambitious community members to organize open source conferences or events in their own cities. Whether they’re focused generally to open source or for a specific project, you can find a variety of conferences, hackathons, workshops, or meet-ups all over the world. Fedora benefits from having Ambassadors to attend these events to introduce Fedora and spread the word about the community. It’s not uncommon to see Fedora participating in these events, and Linux Weekend 2017 in Tirana, Albania was not an exception.

From March 25-26, 2017 in Tirana, Albania, nearly 130 people attended the first-ever Linux Weekend 2017. Linux Weekend was organized by Open Labs Hackerspace at the Universiteti Politeknik i Tiranës as an introduction to Linux for beginners. Throughout Tirana, universities have a strong focus on Windows or macOS operating systems and little focus is given to Linux. Open Labs community members wanted to organize an event that would promote Linux as an open source alternative and demonstrate some of its benefits over proprietary environments. The event collected representatives from various communities, including Fedora, Ubuntu, OpenSUSE, NextCloud, MusicBrainz, and more.

Organizing Linux Weekend 🔗

The Open Labs community is not unfamiliar to organizing open source events in Tirana. Their portfolio includes Fedora release parties, OpenStreetMap map-a-thons, Wikipedia edit-a-thons, and Fedora community meet-ups. However, these events have been targeted towards people who already had prior interest or knowledge about open source communities. The organization and planning for Linux Weekend began in the middle of January as an idea to introduce Linux to complete beginners.

Planning and organizing Linux Weekend was a community effort. Jona Azizaj is a board member of Open Labs and a Fedora contributor, and was involved as an organizer for the event. “This was the first edition of Linux Weekend, so the main focus is to raise awareness about Linux and open source. We wanted to educate beginners and spread the word about the choices that are available,” Azizaj explained. The two tracks for the event were full of talks and workshops to introduce attendees not only to Linux, but also various software and applications of what someone can do with Linux.

Many of the attendees were students who had either heard of Linux in their classes or from their peers. Other attendees included industry professionals or other open source community members. Several representatives of Fedora from different parts of the community were in attendance as well. Ambassadors, designers, and translators were available to answer questions and teach newcomers about Fedora.

Albanian students learn Linux 🔗

Since the focus of the event was to teach newcomers about Linux and how it can be used, Linux Weekend was organized to be an introduction to various parts of the Linux ecosystem. Representatives from Fedora, Ubuntu, Arch Linux, OpenSUSE, Linux Mint, and elementaryOS gave introductions to the operating systems and their communities. In addition to distributions, there were also sessions on open source licenses, NextCloud, text editors, MusicBrainz, and more. The benefit of these sessions were demonstrating the different ways Linux can be used to accomplish various tasks.

In addition to the various sessions during the weekend, there were three community booths for Fedora, OpenSUSE, and Mozilla. Attendees had a chance to get swag from each project and also talk with representatives about their own experiences or how to get started using their project.

Sidorela Uku was both an attendee and a speaker at Linux Weekend. Her talk, “Programming in Linux, editors, and tools”, introduced various text editors and other tools to help customize any Linux distribution to someone’s needs or personal preferences. In addition to sharing her own knowledge, she was also excited to discover new things. “I wanted to attend the talks and workshops to learn as much as possible. I also wanted to figure out the next steps to find a project and get involved as a contributor,” Uku explained. “I also wanted to share the things I know with others to help them get started with Linux.” This was Uku’s first time speaking at an event and she looks forward to more open source events in Tirana in the future.

Fedora contributors introduce community 🔗

Various members of the Fedora community were also in attendance. Some of the Fedora presentations over the weekend introduced the project to newcomers, detailed the translation efforts to bring Fedora to Albanian, and also guided attendees on how to make their first steps as contributors. The Fedora presence aimed to help give newcomers a taste of the operating system but also to show the impact someone can have if they decide to contribute.

Mariana Balla was one of the first speakers on Saturday morning with her talk titled, “Introduction to Fedora Project and how to be a part of the community”. She started with localizing Fedora into Albanian in early 2016 and more recently started to become an advocate as well. “Fedora is one of the most used distributions, and it was great to have Fedora here to spread the word and show what our community is all about,” Balla said. “One thing I hoped to show in my talk was that technical skills aren’t required to contribute to Fedora. There’s so many things that aren’t code that people can help with!” One of the highlights of Balla’s presentation was breaking down the different sub-projects in the community and how they contribute to making Fedora what it is. One site that was mentioned was whatcanidoforfedora.org, a site anyone can click their way through to find an area that interests them.

One key contribution area that was important for the local community was localization. Many attendees and speakers alike thought it was important to have software translated into their native language. Anxhela Hyseni is a Fedora Ambassador and led the workshop on “Translation of Fedora”. “It’s important for Albanians to have software in Albanian because we are Albanians!” Hyseni laughs. “People are better able to understand Fedora and it makes it more accessible for us to have it in our local language.” She hopes that attendees left Linux Weekend with plenty of new contacts in the open source community and a better idea of what Linux is all about. She and others also helped Linux newcomers install Fedora 25 as a dual-boot or for virtual machines in the installfest on Saturday morning.

Wrapping up 🔗

After two days of talks, workshops, and hallway discussions, the final talk finished around 4:00pm on Sunday. Azizaj closed out with some final words of encouragement and thanks for attendee participation.

The presence of Fedora, open source software, and its philosophy was present during the entire weekend. Between stickers, install media, and brochures from community tables and the various presentations during the weekend, participants and organizers felt the event was worthwhile. “We had limited time to plan, but it was important to bring Linux to people to introduce the philosophy, show them how to use it, and also how they can give back,” Anxhelo Lushka, an event organizer and Fedora contributor, explained. “We hope attendees had new experiences and learned something new and useful for the real world, for jobs or studying. We also hope we convinced them to contribute and give back, even if in a small way.”

Now that Linux Weekend 2017 is finished, the organizing team is switching their focus to Open Source Conference Albania (OSCAL), the largest open source conference in the region.

Find Fedora near you 🔗

Open source events are happening all around the world, and Fedora might be closer to you than you think! Check for local user or meet-up groups near you to get involved in some of these events. Meetup.com is a great way to find local events happening in your community. You can also keep an eye on what Fedora is up to by following us on Facebook, Twitter, Google+, or Instagram.

We hope to see you at an event in the coming future!

The popular Voice Over IP (VoIP) program, Mumble, is being repackaged again for Fedora 22 and 23. Fedora contributor fedpop unretired the package from the Fedora Package Database and is working on getting it added to the stable repositories.

Mumble is available for testing for Fedora 22 and 23 users. Once enough positive feedback is received, it will be added back to the stable repositories for all users. Testers are welcome, especially for Fedora 22!

How to test Mumble 🔗

To test, open a command line and run the following command.

$ sudo dnf install mumble --enablerepo=updates-testing

Confirm the installation and the application will appear on your system. Give it a run and make sure everything works as expected! If it all checks out, leave feedback for the build in Bodhi so the package. This helps move it closer to being packaged for the stable repositories (links are below).

For help enabling the testing repository, see the QA Testing wiki article.

Why I love Mumble 🔗

Mumble is a package I originally installed when I first began using Fedora in Fedora 20. It is one of the few major VoIP clients that can be classified as “FOSS” (Free and Open Source) and is available on multiple platforms. It’s fairly simple to get a server (Murmur) up and running, and it’s a great way for communicating over voice with friends, family, or communities. I used to use it often when I was active in various Minecraft server communities, and for a short time, I used it for my own Minecraft community as well.

When it originally disappeared in Fedora 21, I was disappointed since Mumble is an important piece of software that I think is important in any modern Linux distribution. Finally, Fedora users will once again be able to seamlessly install it without having to manually compile the software.

Big hat tip to fedpop for repackaging this awesome piece of software! This also seems to be his first experience as a Fedora packager, so a big welcome goes out to him as well. I can’t wait to start recommending Mumble again for Fedora users.

Find it in Bodhi 🔗

Fedora 22

Fedora 23

Netflix, Linux, and 60 seconds 🔗

While they have their differences on the desktop, there is one place where Netflix and Linux get along beautifully: the cloud. Netflix system administrator Brendan Gregg recently published an article on the Netflix blog about what their administrators do in the first 60 seconds when analyzing performance. You should give it a read!

The Negatives 🔗

The desktop is broken 🔗

Without a moment’s delay, the bold statement was offered by Levente: as of right now, the Linux desktop is broken.

56.26% of the desktop market share is Windows 7. 18.26% is Windows XP, 13.52% is Windows 8, 5.26% is OS X, 1.34% is Linux. Of the Linux distros out there, Mint, Ubuntu, Debian, OpenSUSE, and Fedora are the top five. Yet this doesn’t seem that impressive considering how small of a position Linux as a whole takes in the desktop realm.

In terms of mobile-oriented operating systems, it’s almost spectacularly reverse: 84.04% Android, 11.07% iOS, 2.09% Windows Phone, 2.80% is others. As many already know, Android is based off of the Linux kernel. So it’s clear that we’re doing something right with the mobile market. What can we take away from this and apply to the universe of the desktop?

Answer: We must be doing something wrong on the desktop!

The masses 🔗

Choice 🔗

One reason why Linux doesn’t appeal to the masses is the plethora of desktop environments; or in other words, choice! There are several different environments which provide different user interfaces… and they’re all different. An end user can get lost in this as they’re not going to care about how it runs under the hood – they just want it to be easy to use and have it be the same every time they use it. Android has about the same UI and  looks every year and every update. On the other hand, the controversial transition from GNOME 2 to GNOME 3 completed changed the UI of the Linux desktop, subsequently not offering much consistency for users who avoid having to “learn something new”.

In the big picture, they’re all different. For developers and daily Linux users, this is good and awesome because they like choice and decision! But for the regular end user, this isn’t good because they want consistency in how they use their computer. For the masses, the difference is bad.

Perhaps the answer is to never upgrade the computer, but that’s bad from a security standpoint. Not to mention, it’s certainly not very “cool” either. More often than that, having the latest and greatest in software is in the interest of users.

Configuration 🔗

The next big complaint is configuration. There’s countless configuration applications, all with different settings, and then sometimes they all don’t share the same settings. Again, we see the sad but true pattern of no consistency. Compare between KDE and MATE: you might configure something like printers in one desktop environment, but then you move to a different environment and they’re all lost to the oblivion. You have to go through a painful workaround to reconfigure the same thing twice. This is probably why people love OS X computers because they promise to “just work”, which they usually deliver on, at the expense of losing the freedom that a Linux distribution would offer.

Distributions 🔗

The next complaint, and possibly most controversial, is distributions themselves… we all love choice and variety, but for end users, with so many different distributions, it’s another mess. Take installing Firefox. If you’re running Fedora and want to install Firefox, but you found a .deb file, you’re not going to have success because it’s a Debian package, not an RPM package. If you’re a computer newbie, you’re likely not going to know (or care, for that matter) about the distribution that you’re running. It’ll just end in frustration and confusion for a non-technical end user. They don’t want to figure out why it doesn’t work or how to fix it – they just want it to work.

Examples of the above being yum install on Ubuntu, development packages in Fedora repositories, and… how do I set up my printers? These are reasons why people will switch from Linux back to Windows or OS X.

“The efforts to standardize on a kernel and a set of core libraries were undermined by the Distro of the Day that held the position of power.” - Miguel de Icaza

Graphics 🔗

Another frustrating aspect of Linux is that the kernel can fully support a 4096-processor supercomputer, but a laptop can have difficulties waking up from sleep.

This brings us into the next reason why the year isn’t coming soon: VGA and graphics. We miss the “it just works” moments, and sometimes the issues with graphics drivers can be more frustrating and challenging than anything near enjoyable. In the kernel, it was suggested and subsequently dismissed to support graphics via VGA, but for reasons buried in the mailing lists, it’s not in the interest of the kernel to support graphics. It’s more of the desktop developers that do this.

But part of the issue is also with hardware manufacturers. If hardware manufacturers made their software utilities for Linux, it would allow for more powerful troubleshooting and debugging.

Reverse Compatibility 🔗

Then we can look at reverse compatibility on the Linux desktop. Before looking at Linux, it’s important to consider the number one adversary, Windows. Take  Win32k… Windows 98 apps can still run on Windows 8. That’s 19 years of reverse compatibility. As much as we all love to hate on Microsoft, that’s pretty incredible. Take GTK and Qt – neither are backwards compatible with older versions.

This goes hand in hand with the next point, rate of change. Windows updates its major version every two to three years, sometimes longer. There’s usually no big changes in the interim period between major releases. You could be on XP and upgrade service packs and you won’t notice anything significant. On the other hand, Linux distributions release far more frequently and sometimes introduce major changes, such as GNOME 2 to GNOME 3.

So these are the negatives of the Linux desktop. But! There are positives…

The Positives 🔗

Gaming 🔗

Probably didn’t expect to see this as a positive, did you? The gaming side of Linux is making leaps and bounds, and it’s worth noting. Perhaps the most notable example is SteamOS, which is completely based off of Linux. CryTek Engine, Unreal Engine 4.1, all of these major game design engines are turning to Linux and are encouraging more support for Linux platforms. This is a major pull factor for users.

For gaming hardware, there are also positives. Nvidia is now compatible with bumblebee (which developers more familiar with this are praising), and there are many improvements to GPU drivers, such as performance, stability, and so much more. Some of the things that make us love Linux as a desktop.

When is our year? 🔗

So… when is our year?! It’s going to come… one day. Just not soon. There’s too many issues in terms of appealing to the masses and making the Linux experience easier for end users.

Some people believe that the time has come and gone for the desktop, but the mobile market shines a light of a positive future for us. Is Android a better idea of the future than a Linux desktop environment? Only time will tell.

Sources 🔗

This talk was given by Levente Kurusa. Levente is a Linux enthusiast focusing mostly on kernel development. He is also a frequent speaker at various Linux events, where he talks about the various way an individual can join the kernel development community. He participated in an annual open source competition called Google Code-In, where he was a finalist for KDE and became a KDE developer in the process. He currently works for Red Hat on the Virtualization Team.

The majority of the content of this article was taken by my own notes I took during his talk – if you want a less pretty version, you can still read my Pastebin summary.