This post is a reflection on the past year in my work with the UNICEF Venture Fund. We integrated new open source tools to capture metrics and data about open source repositories connected to UNICEF portfolio companies and created a shortlist of key metrics that map to business sustainability metrics. Now, we are better positioned to look back on past, current, and upcoming portfolio companies and mentor support programs.

As we move into 2022, this post covers my current thinking on these points:

1. Defining investments.
2. How do these investments impact sustainability?
3. CHAOSS metrics as an open source tool for an investment lens on sustainability.
4. What next?

Defining investments. 🔗

When we talk about investing in open source, what do we mean? What are the known inputs? What are the expected outputs? “Investments” and “investing” are broad terms. Investments typically mean sizeable financial injections of support and growth, but can also include non-financial investments too. Investments can also take the form of both time and energy (i.e. electricity and digital infrastructure).

The UNICEF Venture Fund provides equity-free funding for start-up companies building open source solutions of interest to UNICEF. All the start-up companies are registered companies in UNICEF program countries. As part of the Venture Fund’s location in the Office of Innovation, it is also a vehicle for UNICEF to explore frontier technology areas through the investments. When a start-up company is receiving investment from UNICEF, the company receives both funding and tailored mentorship about business and open technology.

A question I want to know is, what is the impact of the received funding plus guided mentorship? How does this approach enable the companies to be successful after graduating? What discoveries or knowledge could be shared with others to assist the development of their own open programs?

To summarize, an investment can be financial or non-financial. Financial investments include direct funding, grants, venture capital, fellowships, or any other exchange of capital. Non-financial investments include time spent in coaching sessions, personalized content for companies, and shared digital infrastructure. Neither list is exhaustive.

How do these investments impact sustainability? 🔗

Data makes introspection easier. Bitergia’s Cauldron.io was a champion tool for kickstarting an open source metrics strategy for the UNICEF Venture Fund. Its introduction as a tool opened up a wider span of data to look at. There are new opportunities to ask questions and explore growth, scale, and sustainability.

In order to come to a conclusion on sustainability impact, we need streamlined data to test a thesis. The Venture Fund team improved internal processes to how metrics are collected from portfolio companies. The team is unifying behind fewer tools and methods to ensure we see the same data and have the same view of the data points we measure. This also provides a fresh opportunity to review how we measure open source impact across portfolio companies. Many have dashboards on Cauldron.io, but data needs a storyteller for it to make meaning. So, the next step is to ask questions with this new data and frame a thesis to measure and test the sustainability of Venture Fund investments into open source.

Many have traveled before me on the same trail of thought. I started first with the Community Health Analytics Open Source Software (CHAOSS) project and its metrics releases. This served as the initial point of brainstorming to frame questions and different scenarios of risk, evolution, DEI, and value.

CHAOSS metrics as an open source tool for an investment lens on sustainability. 🔗

I reviewed the latest release of CHAOSS metrics and narrowed down four metrics I want to measure in the next year. I also shared thoughts on why collect this data and how to do it. This blog post is no more than me wondering out loud, to help me frame an analytical approach for this metrics strategy.

The four metrics are detailed below:

1. Contribution Attribution
2. Contributors
3. Collaboration Platform Activity
4. Labor Investment

Contribution Attribution 🔗

Question: Who has contributed to an open source project and what attribution information about people and organizations is assigned for contributions?

chaoss.community/metric-contribution-attribution/

This metric is insightful because it is targeted deeply into team and project culture. This metric is a good representation of how much the project leans into an open source model of building their project. This work ethos and intention to forge on an open source path is difficult to understand at times. If a team takes care to attribute their software dependencies and other contributors to their code (if any), this is a good sign that the team accepts collaboration as a value and encourages working with others.

I would measure this across two types of contributions: attributions for software dependencies including those with permissive licenses, and for any other direct contributors to the code and how they are recognized for their participation. This could be filtered in a red-yellow-green light approach:

1. Red: No attributions are made, or all attributions are inadequate.
2. Yellow: One of two attributions are made, or one attribution type is inadequately attributed.
3. Green: All dependencies and used works are correctly attributed.

Contributors 🔗

Question: Who are the contributors to a project?

chaoss.community/metric-contributors/

This metric explores a more human dimension of the people and participants to an open source project. The metric defines contributors and contributions broadly, as “anyone who contributes to the project in any way.” Understanding the people participating in a community, their motivations, goals, and why they choose to be in that community is important to understand sustainability. Otherwise, you may lose out on good opportunities to attract contributions from people who are already engaged, and new engagements may be difficult because of a mismatch of expectations.

This metric is more a means than it is an end; that is, it provides opportunities to ask more questions than provide detailed answers. Nevertheless, it does provide some guidance towards understanding contributors in a project, and it can lead to some concrete actions based on gathered insights. For example, this metric will enable deeper looks in areas of diversity, equity, and inclusion.

Since I work with start-up companies with small, lean development teams, I look to understand the motivations of the developers working on their projects and where the motivations may align with another open source solution. This enables the two communities to leverage their combined brainstorming for meeting complimentary goals around development and innovation.

To collect this data, I would have the team define what areas of contribution they seek for their open source solutions and then map those desired contributions to a specific project area or different team members. This enables a form of consistent accountability for checking expectations with reality and understanding team capacity. Each area could be a key-value pair, where the value is the project area, team lead, or delegated team member for the type of contribution solicited.

Collaboration Platform Activity 🔗

Question: What is the count of activities across digital collaboration platforms (e.g., GitHub, GitLab, Slack, email) used by a project?

chaoss.community/metric-collaboration-platform-activity/

Collaboration platform activity is one effective proxy metric for community engagement if measured accurately. The metric does not define collaboration as much as it provides a data structure to measure it. It abstracts collaboration into key data points like timestamp, sender, whether the platform has threaded or non-threaded discussions, data collection date, and platform message identifier. To a degree, collaboration can be abstracted out in this way: a person takes any given action at a given time in a given way, and this action is measured as project-related activity on the collaboration platform.

There are a few possible approaches to collecting this data from UNICEF Venture Fund companies. Each approach does not cancel out another, but each approach could be combined with the others:

1. Measure common git activity like commits, issues, pull/merge requests. We already measure this data, but use it only in connection to validating Venture Fund workplans for each team with UNICEF portfolio manager(s).
2. Count communications like comments, reviews, public messages, and other outreach. Communications strategies and tools are typically inferred from common git activity. Measuring for engagement and stratifying those metrics into a smaller group could allow for deeper insights to the evolution of early-stage open source communities.
3. Make community hubs first-class citizens in the data curation process to infer about informal engagement. Both open source projects and UNICEF Venture Fund portfolio companies use a variety of tools to communicate, especially in view of COVID-19 and its seismic impact on how we work. Platforms like Discord, Telegram, Mattermost, Slack, Rocket.chat, Matrix, and others are focal points where projects collaborate, ask questions, and support others. Bringing this data stream into the mix offers deeper insights into how teams engage and build community around their work, and also guidance on when to push for contribution opportunities at the right time.

The satisfaction of these three options in their totality is not enough. To leverage the fullest impact, these metrics must tie into each other, and need to be connected back to a narrative. Why is this data being collected and what actions are influenced by the knowledge of this data? The data collection enables the evaluation of sustainability and understanding the birth, growth, and evolution of an open source technology product. Influenced actions can include moving more human resources (i.e. contractors or staff) to support a project, adopting a new open source best practice, and/or engaging new customers, talent, or other leads based on participation in the community.

Measuring collaboration platform activity is not black and white. Many new questions would likely come forward as part of measuring this activity. Yet this is the point—it lays the foundation for the next layer to the data collection, analysis, and reporting process around sustainability.

Labor Investment 🔗

Question: What was the cost of an organization for its employees to create the counted contributions (e.g., commits, issues, and pull requests)?

chaoss.community/metric-labor-investment/

This metric is perhaps the most ambitious of the group. How do you measure labor investment into an open source project? Or literally, the number of person-hours that go into software design, development, co-creation, and community management? It feels like a gargantuan effort, but there may be better ways to measure this in connection to other data the UNICEF Venture Fund is already connected about the businesses.

Measuring labor investment impacts two narratives: the rate of development on the open source work, and the impact of UNICEF investment into a company backing an open source work.

Firstly, understanding the rate of development on an open source work is easier to infer by understanding who is allocated on a project and how much of their time they dedicate to it. If a team of three contributors spares a few hours a week, it will mean something different compared to a team of five engineers spread across different disciplines working full-time. Mapping the labor investment for open source projects supported by UNICEF would enable better planning by understanding the typical labor investment in open source workplan tasks as piloted by other Venture Fund portfolio companies.

Secondly, this gives us a new way of talking about the impact of UNICEF Venture Fund investments as an investment not only in software products but also in labor. It gives us insight into the investment of labor in software engineering talent among portfolio companies. How does this measurement change over time of the investment? Do projects receive more or less investment of labor during the 12 month period we work with them? This could also be used as a proxy metric for the impact of our unique mentorship and coaching opportunities.

What next? 🔗

Knowing is half the journey. Even if the knowledge is not yet firmly rooted. The analysis and introspection are from me as an individual working among the UNICEF Venture Fund and do not represent the views and beliefs of UNICEF or the UN in any capacity. My intent is that by sharing this analysis in the open, it allows for a space where conversation can spark where it could not before. It also invites others to share ideas, feedback, and constructive criticism of an emerging metrics strategy for investments made into the open source ecosystem.

Next, more layers can be added and internal and external validation can help to keep this moving forward. An implementation plan would be the next step to follow this post. The implementation plan considers the process of how start-up companies move through the Venture Fund from start to finish. Who interacts with the companies and when? At what point is a company ready to begin building in a new metric or count in their monthly metrics? Do they understand the implications and assessments of these metrics? At what points in the process is data already being collected? Could these new data requests be added to existing requests? And so on.

I hope to formalize some of this new reporting and metrics strategy in upcoming cohorts in 2022, as part of a renewed effort into communicating how our open source investments tie into sustainable impact towards the U.N. Sustainable Development Goals.

This post will serve as a milestone marker on the metrics strategy discussion in the coming one to two months. See you in 2022.

Featured photo by Edward Howell on Unsplash. Modified by Justin Wheeler. CC BY-SA 4.0.

Learn about open source project health metrics and tools used by open source projects, communities, and engineering teams to track and analyze their community work. This conference will provide a venue for discussing open source project health, CHAOSS updates, use cases, and hands-on workshops for developers, community managers, project managers, and anyone interested in measuring open source project health. We will also share insights from the CHAOSS working groups on Diversity and Inclusion, Evolution, Risk, Value, and Common Metrics.

chaoss.community/chaosscon-2020-eu/

This is my second time attending CHAOSScon. I attended on behalf of RIT LibreCorps to represent our engagement with the UNICEF Office of Innovation and the Innovation Fund. For CHAOSScon EU 2020, I arrived hoping to learn more about effective metric collection strategies for open source communities and also get a deeper understanding of the technology behind GrimoireLab.

Sessions: Play-by-play 🔗

Event reports take many forms. My form is an expanded version of my session notes along with key takeaways. Said another way, my event report is biased towards what is interesting to me. You can also skim the headings to find what interests you.

Also, I live-tweeted several sessions of CHAOSScon EU 2020, so some sections include tweet excerpts with pictures.

Ethics: What You Know & What You Don’t Know 🔗

Metrics give us a lot of insight into what’s going on with our projects – but only if we have enough context. The catch is that personal biases and pre-conceptions are extremely hard to filter for so no one person can see everything. Those gaps means that even the most well-meaning of us could end up doing more harm than good. In this talk, I’ll share a few examples of the ways human brains make assumptions, some of our typical blind spots and how incomplete data can cause harm when deployed without any checks. For folks who collect, analyze and ultimately make mission critical decisions with metrics, the challenge is to constantly maintain a bit of healthy skepticism of what the numbers seem to be telling us.

Deb Nicholson

Deb’s session was a great opener to a day focused on metrics and automated reporting. Her talk underscored unconscious bias and the ways poorly-designed metrics fail us. We should take an active approach to questioning our perspectives and take suggestions with grace.

During Q&A, my colleague Mike asked for advice on pushing back on management when they ask for bad metrics. Deb suggested a clever strategy to respond in these situations:

1. Provide the metric asked for
2. Provide a metric that contradicts what was asked for
3. Provide the metric they should have asked for

Heat, Light, and Love: Telling Twitter’s Open Source Program Office’s Story 🔗

Making it simple to trust, ship, and grow open source at Twitter demanded visibility into contribution trends that were not readily available. Twitter’s open source program office and the Augur project collaborated with CHAOSS using student interns from Google’s Summer of Code, Twitter’s intern program and the University of Missouri. First, this talk describes Twitter’s engaging interface design and its fit to the needs of Open Source stakeholders within Twitter, but also outside the open source program office. Second, we describe how an open API and data collection tool, Augur, evolved in close collaboration with Twitter.

Sean Goggins & Remy DeCausemaker

This one inspired me. It represented a dream of mine for many open source communities. Sean and Remy presented on three years of building an automated metrics visualization site with GrimoireLab and Augur. The visualization was powerful to communicate the value of open source engagement, participation, and investment to internal and external stakeholders.

The work was based on a philosophy familiar to me from working with Remy years ago in the Fedora Project community:

If we are speaking in metaphors, one way to extend Remy’s “light” metaphor is tying in a theatrical concept: stage lights and backstage lights.

Stage lights are the metrics and insights to present out to the world. They are front and center! These insights validate your mission to an external audience.

Backstage lights are for an internal audience. These metrics and insights illuminate the inside of an organization and build understanding of who and what is there. These are useful to validate work and efforts to funders, or perhaps your CFO.

This talk served as inspiration for something I hope to explore later in 2020.

Measuring Culture 🔗

They say you can’t be certain of anything unless it can be measured – so what about culture? We know it exists and can be passed on to others, so there must be a way to measure it. Using Apache Kibble we attempt to look for cultural indicators at the Apache Software Foundation, set a cultural benchmark then match real Apache projects against it. In theory we would expect a perfect match but as the results show, culture isn’t that simple.

Sharan Foga

Sharan introduced five factors to measure culture: bus factor, contributor experience, contributor retention, sentiment analysis, and key phrase extraction.

Sharan defined culture as what we believe and what we do. So, she introduced five indicators for open source communities to measure culture. These came from learnings at the Apache Software Foundation’s Kibble project:

• Pony (“bus”) factor:
  • “Committership”
  • Authorship
  • Meta factor (# of organizations involved)
• Contributor experience / contributor retention:
  • People who (re)joined
  • People who quit (why?)
  • People retained
  • Active contributors
  • Cross the above over time (e.g. # of months/years involved)
• Mood, or sentiment analysis: Look for these moods…
  • Trust
  • Surprise
  • Neutrality
  • Sadness
  • Positivity
  • Anger
  • Negativity
  • Joy
  • Anticipation
  • Disgust
  • Fear
• Key phrase analysis: Understand community jargon (e.g. if “+1” is used to indicate agreement)

I had two key takeaways. First, nothing matches completely, so don’t over-rely on one factor or metric. Second, identify your baseline indicators and iterate on those. For the above indicators, maybe these are specific to Apache projects. Maybe they apply to your projects too… or maybe they don’t!

Meet Cauldron! Your SaaS Software Development Analytics solution 🔗

Cauldron is a brand-new free open source SaaS able to analyze software development community and processes. It relies on CHAOSS GrimoireLab to collect and process data from different tools used in software development such as git, GitLab, GitHub and/or social platforms like Meetup, and Open Distro for ElasticSearch to store and to display the project data. The talk shows how Cauldron can be easily put into action to analyze the software projects that matter to the attendees, and to get some CHAOSS metrics about them.

Jose Manrique Lopez de la Fuente

I had no written notes other than “wow!”

Finally, Bitergia built a hosted GrimoireLab service, called Cauldron. Cauldron takes the pain out of hosting your own GrimoireLab infrastructure. It is in early alpha and does not support all the data sources available in GrimoireLab yet, but it is promising.

In the FOSS@MAGIC community, we want GrimoireLab to understand our open source community at the Rochester Institute of Technology, home of the first academic minor in free/open source. But as I go deeper, I realize GrimoireLab is very complex. Since my contract with RIT ends in a month, I want to leave something sustainable for my successors. Cauldron seems like a valid option.

I have some homework to do!

Language barriers for open source contributors 🔗

While open source development is common across the world, projects from countries outside North America and Europe, such as China, have difficulties attracting international contributors. One major reason for this is language differences. The presentation provides an overview of the diversity of non-native English speakers in different open source ecosystems. Furthermore, we will provide results about whether or not those contributors face a language barrier when contributing to open source projects. The presentation concludes with a discussion on how we can create metrics to assess the spoken language diversity in open source communities and how to include those people in the community more easily.

Isabella Vieira Ferreira

Fascinating work from @isaferreira_57 on language barrier in #opensource. #choasscon

Contributing + participating in a project operating in a language (technical too!) that's not one's first always require more effort– a useful reminder to be kind and empathetic 💖💖 https://t.co/3cJ6qWWSsH

— Emmy Tsang (@emmy_ft) January 31, 2020

Isabella’s talk was insightful and challenged my unconscious bias. An interesting insight from two case studies was translators did not transition over time to software development contributions.

I believe a bias exists in open source projects with active localization communities. I have seen project leaders invest in localization efforts as one way to bring new people into a community. While it does bring new people in, there is a subtle implication that “bringing new people in” means bringing software contributions. That’s not always the case. Instead, the value of translation work needs to be quantified beyond just software contributions.

This is especially true for supporting grassroots advocacy communities. Localization may not build your developer community, but it can support stronger local communities in regions where English is not the first language.

Metrics are more than the sum of their parts 🔗

We are in a state of Community building that maps success to metrics like lines of code and GitHub stars. I would like us to explore where these metrics fall short on capturing the most significant value of a community, and how we can use other metrics to show the much larger impact Community organizations like Developer Relations (DevRel) can have on a business. This improved process begins with mapping value to business need, pairing it with a community need, and back to the business opportunities. There will be practical advice on how to pitch and position Community, whether you’re a Community Manager, Developer Advocate, or other Developer Relations professional.

Matt Broberg

Matt’s CHAOSScon EU 2020 session pitched strategy on communicating the value of developer relations to our organizations. He covered a lot of material and left me many thoughts to reflect on. I look forward to seeing the video recording of this talk to reflect further on integrating his ideas into my day-to-day. The themes of Matt’s talk struck a chord with me for challenging advocacy work ahead for me in 2020.

Emerging Tech Open-Source Scoring System (ETOSS) 🔗

This is a framework used by the UNICEF Innovation Fund, to evaluate and track the strategic position and growth potential of the early stage startups in its portfolio. The focus is on startups building open source frontier tech: Blockchain, Data Science, Artificial Intelligence and Extended Reality. It classifies the startups into four quadrants, based on two dimensions: the software developed (technology), and the financial sustainability of the project (business sustainability).

Cecilia Chapiro & Justin Wheeler

.@jflory7: The UNICEF tech KPIs are quite similar to @CHAOSSproj metrics - there are also business sustainability metric to measure, placing organisations / tech projects on a sustainable and tech scale... #CHAOSScon pic.twitter.com/e3M440FMS4

— yo yehudi stays away from humans! 🏳️‍🌈 (@yoyehudi) January 31, 2020

Interested in getting involved? Contact details for @jflory7 and @ceciliachapiro here! #CHAOSScon pic.twitter.com/jMsfiCK9Rg

— yo yehudi stays away from humans! 🏳️‍🌈 (@yoyehudi) January 31, 2020

@jflory7 thanks for the great talk, great to hear @UNICEFinnovate is actively funding OSS for the greater-good! #CHAOSScon pic.twitter.com/g81koSI4Ov

— Farbod Saraf (@farbodsaraf) January 31, 2020

#opensource rubric mentioned by @jflory7 at #CHAOSScon https://t.co/U2pIZsgrMz

It helped me evaluate the last proposal we received at @eLifeInnovation 🙌🙌

— Emmy Tsang (@emmy_ft) January 31, 2020

Hey, it’s me!

My colleague Cecilia Chapiro proposed a session for CHAOSScon EU 2020 on how the UNICEF Innovation Fund evaluates the health and success of funded open source projects. However, due to travel complications, she was unable to attend. I presented her slides and work on her behalf at CHAOSScon EU 2020.

Many people were excited, if not also surprised, to hear that UNICEF funds humanitarian open source. In Cecilia’s evaluation framework, there are two key areas: business sustainability indicators and technology indicators. There were good questions about how UNICEF measures on business sustainability. I hope to further explore GrimoireLab/Cauldron to better automate how we measure sustainability with industry-standard tools from the open source ecosystem.

I also gave a shout-out to RIT LibreCorps and the resources we maintain to help UNICEF and the Innovation Fund cohorts build and support sustainable open source projects.

Open and Reproducible Research with Community-driven Research Software Metrics 🔗

What do artificial neural networks, images of the black hole, and SETI@home all have in common? They all trace their roots to research software. In this session, we hope to explore metrics and best practices for evaluating open research software. Research software engineers are evaluated largely based on the numbers of publications and citations. However, metrics based on the number of publications encourage re-creating ‘new’ things or re-implementations instead of reuse, and there are huge variations in existing practices to cite software, meaning software isn’t always cited or citations aren’t always captured. Better metrics that reflect the health of research software is crucial to avoid gamification, incentivizing research software engineers to follow best software development practices by recognizing and rewarding their effort appropriately, and ultimately advancing open science and improving the reproducibility of research.

Yo Yehudi

Yo teamed up with Emmy Tsang and Mateusz Kuzak to deliver this session. I appreciated the audience participation in the second half, which was a creative idea for a session later in the day. It helped revive the conference energy!

Yo and the team captured excellent notes from their time across Brussels, so look there for a deeper dive if this is interesting to you.

Diversity & Inclusion Badging Program 🔗

Within IT, open source software exhibits some of the worst D&I. The open source survey run by GitHub states that only 3% of contributors to open source software are female (GitHub, 2017). While open source has made strides in some aspects of D&I, such as event inclusivity (Irwin, 2017) and community engagement to support organizational diversity (Alexander, 2019), work remains. The proposed is aimed at contributing to this positive growth through the development of a D&I badging system. In open source, badges are available for community leaders to publicly display, providing a way for projects to signal their attention to such things as community health and software security. We can draw from this badging work to improve D&I in open source. In this talk, I will discuss complexities associated with D&I metrics and propose a D&I badging system that provides open source communities the ability to make diversity and inclusion within open source projects more transparent.

Matt Germonprez

First, I must pay my full respect and gratitude to Matt for digging into this topic. It is hard. It is tricky. Nobody I know of is working on this. So, my feedback comes from a place of love because this is an important topic to me too, and I appreciate Matt’s work! Thank you Matt and the CHAOSS D&I Working Group.

Critique: Is a badging system how to do this? 🔗

Matt presented a strategy to incentivize organizations and projects to follow best practices for diversity and inclusion (D&I) through a badging system. Organizations embed a badge on their project website or README when following best practices. This provides an easy way for new contributors to understand if a project community is inclusive and values equality before contributing. Since best practices are defined not by a single organization, there is a way to measure if an organization is living up to their commitment to D&I.

However, a badging system seems like a difficult way to practically measure an organization’s commitment to D&I. There is a risk for manipulating trust of the system at the expense of open source novices. Novices have the least context and direct experience to see through an unauthentic commitment. Also, ongoing compliance seems difficult to enforce over time with a badging system. Maybe initially an organization or project is “checked off” for D&I best practices. But if project leadership or management changes, community culture can also shift. How do you account for change?

Instead of badges, perhaps a central, inter-organizational certification body makes more sense to achieve this goal. A certification body acts as a central authority for validating accountability. They also have an authority to assign (or revoke) public recognition of organizations and projects with authentic, ongoing commitments to diversity and inclusion.

There is potential here, but I don’t think gamifying D&I best practices with a badging system is the way. We need a long-term, human-centered process in evaluating D&I. We need to build in human-centered evaluation as a “feature” to truly hold projects accountable to their commitments.

Addendum: Events and inclusivity 🔗

There are a few things I’m genuinely proud to have contributed to open source. Most of them aren’t code. International candy swaps at conferences are one of those things.

Time for the international candy swap! There are so many things to love about #DevConf_CZ but the geographic diversity of attendees might be my favorite part. Thank you for organizing, @jonatoni & @jflory7! pic.twitter.com/rU1ETp5aTa

— Mary Thengvall (she/her); mary-grace.bsky.social (@mary_grace) January 25, 2020

DevConf CZ 2020 candy swap, mentioned in a previous blog post

Candy swaps are fun, informal events hosted during a conference. Attendees bring local candies, sweets, or snacks from their hometowns to the conference. For an hour or so, folks gather together to swap candies. Before sampling the sweets, everyone introduces themselves, where they are from, and anything special about what they brought with them. When everyone introduces themselves and their candies, the floor is open to sample from the collected assortment.

This is a fun and engaging way to get to know other attendees beyond the conference’s context. Conferences collect people around a specific context: be it technology, metrics, design, engineering, or something else. But beyond that specific context, we are also people! We have families, we have friends, and we eat food. We have things we miss about home when we travel hundreds or even thousands of miles from home. Events like candy swaps help us connect to other people not just as conference attendees, but as people. You learn something new about each other, and if you’re bold, you try something new too.

While it is not a metric in itself, it would be cool to see things like this included as practical examples of how to evaluate “family friendliness” or inclusivity at events.

True Confessions: The Seven Deadly Sins of Open Source Communities 🔗

This talk would take a humorous approach to explaining what really no one should do when managing and measuring open source community best practices, with the primary goal of instilling positive open source development practices to the audience. Such ““don’ts”” include not documenting anything, establishing no clear paths to success, and not bothering to communicate.

Brian Proffitt

Brian explained what to avoid in building healthy open source communities. While I was already familiar with most of them, the most interesting “sin” for me was the last one: “not seeing ourselves in others.”

Brian offered mitigation strategies and metrics for the first six sins, but he acknowledged “not seeing yourself in others” is hard to mitigate and measure. However, I think things like candy swaps (explained above) are one possible way to do this. They connect us to others in an authentically human way, beyond the specific context of a conference we traveled for.

This is more challenging when interaction is virtual. Virtual systems often take away communication cues that make us human! I’m still thinking on how to mitigate “not seeing ourselves in others” if interaction is mostly virtual. (For what it is worth, I don’t think running open source project meetings in V.R. is the answer either.)

Beyond CHAOSScon EU 2020 🔗

An event report wouldn’t be complete without mentioning the hallway track. It was great to see familiar faces and also meet new folks at CHAOSScon EU 2020.

I had a great lunch conversation with Erik Albers from Free Software Foundation Europe. It was cool to meet a facilitator of the Public Money? Public Code! campaign. Hopefully there are future opportunities to collaborate with RIT LibreCorps and AI freedoms.

As always, face time with Mike Nolan and an OG mentor of mine, Remy DeCausemaker, is appreciated and fulfilling. I wish I could get more of it around the year.

Thanks folx! 🔗

To wrap up this CHAOSScon EU 2020 report, a few thank-yous are in order:

• Cecilia Chapiro: Thanks for putting your work forward to the wider open source community!
• Stephen Jacobs: For always being supportive for yet another trip abroad and helping me push my career forward in a number of ways (and footing the bill!)
• Mike Nolan: My co-conspirator, partner in FOSS, and comrade in arms

I hope to continue conversations started here later into 2020 and keep talking with folks doing great things for the ecosystem. Until next time!

DevConf.CZ 2020 is the 12th annual, free, Red Hat sponsored community conference for developers, admins, DevOps engineers, testers, documentation writers and other contributors to open source technologies. The conference includes topics on Linux, Middleware, Virtualization, Storage, Cloud and mobile. At DevConf.CZ, FLOSS communities sync, share, and hack on upstream projects together in the beautiful city of Brno, Czech Republic.

devconf.info/cz/

This is my third time attending DevConf CZ. I attended on behalf of RIT LibreCorps for professional development, before a week of work-related travel. DevConf CZ is also a great opportunity to meet friends and colleagues from across time zones. This year, I arrived hoping to better understand the future of Red Hat’s technology, see how others are approaching complex problems in emerging technology and open source, and of course, to have yummy candy.

Sessions: Play-by-play 🔗

Event reports take many forms. My form is an expanded version of my session notes along with key takeaways. Said another way, my event report is biased towards what is interesting to me. You can also skim the headings to find what interests you.

Diversity and inclusion meet-up 🔗

Would you like to meet other attendees who stand under the umbrella of “Diversity and Inclusion” or would you like a introduction into what Diversity and inclusion is and why it’s a good thing? this is the session for you! All are welcome!

Imo Flood-Murphy

This was a short, informal session ran by Imo to network and get a high-level introduction to diversity and inclusion in open source. Everyone in the room introduced themselves and gave a short explanation of who they were or what projects they represent. I appreciated the opportunity to meet others and better understand how Red Hat approaches diversity and inclusion.

A suggestion for next time is to allow more unstructured time for conversations. I think fun icebreakers get folks comfortable in a short amount of time to help make connections for the rest of the weekend.

Lessons learned from testing over 200,000 lines of Infrastructure Code 🔗

If we are talking that infrastructure is code, then we should reuse practices from development for infrastructure, i.e.

1. S.O.L.I.D. for Ansible.

2.Pair devops-ing as part of XP practices.

3. Infrastructure Testing Pyramid: static/unit/integration/e2e tests.

Lev Goncharov

Lev shared best practices on building sustainable, tested infrastructure. Infrastructure-as-Code (IaC) was important to how T-Systems scaled their infrastructure over time.

My key takeaways:

1. Smaller components:
   1. More sustainable
   2. Easier to maintain
   3. Easier to test
2. Ansible Roles encourage best use practices for Ansible
3. Spreading knowledge is essential (if nobody understands it, the code is broken)
4. Code review creates accountability
5. Use static analysis tools (Shellcheck, Pylint, Ansible Lint)
6. Write unit tests (shUnit2, Rspec, Pytest, Testinfra, Ansible Molecule)

Content as code, technical writers as developers 🔗

In the open-source project Kyma, documentation is an integral part of code delivery. We, the project’s Information Developers, believe that using the same tools and methodology as your good old code developers, we can create comprehensive and accurate documentation. During our talk, we’ll share the whys and hows of our approach, showing you that the “developer” in “Information Developer” isn’t there just because it sounds cool. We’ll prove that creating documentation goes beyond linguistic shenanigans and salvaging whatever information there is from a trainwreck that is the developer’s notes. Testing solutions, finding our way around Kubernetes, tweaking the website, engaging with the community are just a few examples of what keeps us busy every day.

Barbara Czyz, Tomasz Papiernik

“Information Developers” is a cool phrase I learned. Barbara and Tomasz explained the value of technical writing and asserted documentation should live close to project code.

My key takeaways:

1. Documenting processes like release notes enables others to join with less barriers
2. Docs-as-Code (DaC): Visibility of docs across development process is important
   1. Placing docs with code encourages feedback loops and avoids silos
3. Put links to docs in visible places (e.g. API messages, console messages)
4. Management aside: Emphasize/incentivize value of technical writing in your team
5. Remember bridges across skill areas is possible (technical writers can also be community-oriented people too)

Uncharted waters: Documenting emerging technology 🔗

We can’t help but feel the lure towards the hot new thing, especially when it comes to technology. Part of that lure is the breaking of ground, venturing into the unknown, and working on solutions to new problems. But a lot of the same things that make emerging technology fun and exciting to work on are exactly why it can be difficult to document. These challenges are quite different to those associated with mature products.

This talk is for anyone working on new products and emerging technology, or just interested in learning about fast-moving documentation. It is for the developer as much as it is for the writer, since it usually falls to them to write the early docs before a writer is added to the team.

Andrew Burden

This was the talk I didn’t know I needed to go to.

Lately I work with “emerging technology,” which means different things to different people. Regardless of what emerging tech means to you, Andrew focused on how to write documentation in a fast-paced environment with “pre-release” technology, where things change fast and suddenly. Normally this is an excuse to not write docs, but Andrew showed, yes! It is possible to write good docs, even when context changes fast and often.

Key considerations of fast-paced technical writers 🔗

An even balance of these considerations helps get into a user’s mindset:

1. Scope / scale of release
2. Release schedule
3. Developer meetings / face-time
4. Exposure with $TECHNOLOGY
5. Deployment experience with $TECHNOLOGY

Surviving the information wall 🔗

The “information wall” is the endless wall of information and things to know about a project. If information is endless, how do technical writers survive?

• Take notes: Be like a scientist
• Take notes about your notes
• Be organized with your notes

Obviously Andrew was getting at the value of note-taking. Practicing note-taking skills is critical to keep up with the pace of change.

“Multi-Version Syndrome” 🔗

Sometimes you are writing features for things that will not be released in the next release. There is a risk of losing information across multiple releases (e.g. publishing the wrong thing too soon, or the right thing too late). Clarify the release schedule as you go. A good safeguard against losing information is to rigorously understand release cycle cadence and priority.

If your product isn’t mature yet, anticipate change instead of avoiding it.

Access to technology is critical 🔗

Technical writers are often User 0. To understand the technology, you need access. There are interactive and non-interactive ways of getting access. Interactive ways are preferred because they are always reproducible.

• Interactive
  • Deploy your own
  • Get someone else to deploy it for you (but lose install context)
• Non-interactive
  • Live demos
  • Demo videos
  • Asciicinema (CLI-oriented)

Other takeaways 🔗

• Screenshots have high maintainability cost; avoid if possible
  • Sometimes good stop-gaps until something more maintainable
• Where to begin? Make a table-of-contents for the Minimum Viable Product
  • Never underestimate outlines (ahem, like how I wrote this blog post…)
• Avoid documentation scramble near release day:
  • Make lists / check-lists
  • Take more notes
  • Pre-release checklist
  • Think now, and for the future
• Audit your docs: On-boarding new people is a powerful opportunity to test out your docs

Thanks Andrew for a deep dive on this narrow but important topic.

Community management: not less than a curry 🔗

Every volunteer joins an Open Source community for a reason. The reasons could range from technical gains to finding his/her/their passion. This community of diverse volunteers require a leader who can not just mentor them with their interests but also a manager managing the community activities in terms of community engagement and planning. A community manager is not less than a candle of light and in this presentation, I would be highlighting my learnings and experiences about starting a community from scratch around a project and maintaining a healthy community management practices.

Prathamesh Chavan

Prathamesh designed an activity to help the audience understand community management. My key takeaway was community management is about connecting and understanding others as their authentic self.

In the activity, Prathamesh passed papers and pens to the audience. His session had three steps. Between each step, all attendees traded papers with another attendee:

1. Define a project idea (why, how, what)
2. Identify challenges to idea (i.e. questions)
3. Answer challenges

It reminded me of a similar workshop I attended before. This inspired me to work on my own workshop idea for a future conference.

Cognitive biases, blind spots, and inclusion 🔗

Open source thrives on diversity. The last couple of years has seen huge strides in that aspect with codes of conduct and initiatives like the Contributor Covenant. While these advancements are crucial, they are not enough. In order to truly be inclusive, it’s not enough for the community members to be welcoming and unbiased, the communities’ processes and procedures really support inclusiveness by not only making marginalized members welcome, but allowing them to fully participate.

Allon Mureinik

Allon recognizes the importance of diversity, but asking for improved diversity is one side of the coin. A friend recently shared a powerful quote with me: “If diversity is being invited to the party, inclusion is being invited to dance.” Allon’s message was to dig deeper on including marginalized people in our project communities.

He identified ways we accidentally make our communities less inclusive because of our cognitive/unconscious biases. Everyone has blind spots! Allon suggested ways to be more conscious about inclusion in open source:

• Knowledge barriers
  • Procedural knowledge, not just technical
    • How do you submit code? File a bug? Make meaningful contributions? These need to be documented
  • Documentation fosters inclusivity
• Language barriers
  • Working proficiency in English not universal
  • Conversations have extra barriers (e.g. communicating complex ideas, understanding advanced words)
• Time barriers
  • Work schedules no longer 9 to 5
  • Remember other folks in different time zones
  • On giving feedback: Fast is not a metric! Be smart
    • Merging PRs while others are away, or shortly after opening it
    • Allow time for input on non-trivial changes
• Transparency barriers
  • If not in the open, it didn’t open
  • Negative example: Contributor makes a PR, reviewer has face-to-face conversation with contributor, reviewer merges PR without public context
  • Default to open: in many ways
    • If you can’t be open, at least be transparent

Diversity in open source: show me the data! 🔗

How diverse is your work environment? Diverse communities are more effective, they allow us to share the strengths of the individuals who make up the community. Have you ever looked around and noticed that most of our Open Source communities are predominantly male? Why do you think that is? We’ll use gender diversity as a case study and share some intriguing data points. Let us convince you why it’s so important.

Regardless of your gender, we would love for you to join us! We will also give you some tips on how you can make a difference.

Serena Chechile Nichols, Denise Dumas

Serena and Denise divided the talk into two sections: metrics and action. The way they presented, they brought the audience on the same page by visiting a variety of metrics and then transitioned to an empowering discussion about changing trends we see.

Next time, I hope to see expanded messaging by defining diversity beyond only women. Diversity was frequently tied to gender participation metrics in open source. While women are underrepresented, there are additional aspects of identity that can compound discrimination. Race, socioeconomic status, nationality, sexual orientation, and more also play a part in understanding challenges collectively faced in inclusion work.

The data 🔗

• Gender differences by # of contributors:
  • GSoC 2018: 11.6% female-identifying contributors
  • OpenStack: 10.4% female-identifying contributors
  • Linux kernel: 9.9% female-identifying contributors
• U.S. Dept. of Labor: 22.2% of technical roles filled by women
  • 2014-2019: More women entering tech jobs at companies like Apple, Microsoft, Google, etc.
• Years of experience by gender (<9 years):
  • 66.2% female
  • 52.9% non-binary/queer
  • 50.1% male
• GitHub user and developer survey:
  • 95% male
  • 3% female
  • 1% non-binary

Let’s make things better 🔗

Serena and Denise asserted diversity creates change. All changes come with challenges. Diversity can increase the friction in the process, but that is okay. They emphasized the need for multiple perspectives see past our initial biases (conveniently covered by Allon in the previous talk).

This transitioned to questions, comments, and thoughts from the audience. One interesting point was using the phrase, “we don’t do that here” to create and set norms. I gave a suggestion to look at projects you already participate in and see if there is a diversity and inclusion effort there already. If there is, see if there are ways to help and get involved. If not, consider starting one (or network with the Open Source Diversity community).

To wrap up, one recurring theme of Serena and Denise’s talk is to make time to step back and evaluate the bigger picture. Questioning our biases is an important skill to practice. We need the space and time to recompute!

Candy Swap 🔗

Do you have a unique sweet dessert or candy from your country or hometown? Do you love to try new and exciting foods from around the world? Spend an hour with fellows as we share stories and candies from the world with each other. Participants are invited to bring a unique confectionary or candy from their country or city to share with multiple other people. Before going around to try yummy things, all participants explain what item they bring and any story about its origins or where it is normally used. After sharing, everyone who brought something rotates around to try candies brought by others. After all participants have had a chance to sample, the rest of the community is invited to come and try anything remaining.

Jona Azizaj, Justin Wheeler

I am biased when I say this is one of my favorite parts of conferences I go to. Jona originally proposed the candy swap for DevConf CZ. After unexpectedly adding DevConf CZ to my travel list for 2020, we teamed up to share the sweet tradition from Fedora Flock to DevConf CZ! This is one of my favorite conference traditions because I get to know other attendees in a context outside of technology. And food is always an easy way to win me over.

Instead of listening to me, see what other people have to say about it:

Time for the international candy swap! There are so many things to love about #DevConf_CZ but the geographic diversity of attendees might be my favorite part. Thank you for organizing, @jonatoni & @jflory7! pic.twitter.com/rU1ETp5aTa

— Mary Thengvall (she/her); mary-grace.bsky.social (@mary_grace) January 25, 2020

The "sweetest" activity of #devconf_cz: Today at 3 PM! 🍬🍫
Join Candy Swap, share candies, sweets and stories with others from around the world! https://t.co/OlfdmgGa3a pic.twitter.com/Jnlqi3lsaq

— DevConf.CZ (@devconf_cz) January 25, 2020

Candy Swap time at #DevConf_CZ 😍 pic.twitter.com/zFCNnXZoJf

— Jona Azizaj👩🏻‍💻 🥑 @jonatoni@mastodon.social (@jonatoni) January 25, 2020

Well I experienced this for the #Flock 2019. It's a great opportunity to know the tastebuds of diverse people and enjoy! :D

— Aal (Alisha)🌻 (@withloveaal) January 26, 2020

From Outreachy to cancer research 🔗

Outreachy program is helping women and other underrepresented people to make first steps in tech career. Picking a project, making first open source contributions, working on assigned project and learning from advanced people. But what happens when this three months are over? Can Outreachy be a lifechanging experience?

I will share my story of conversion from a chemist and full time parent into a Fedora Outreachy intern and how I found my place as a junior software developer in cancer genomics research at IRB Barcelona.

Lenka Segura

This was a favorite of the weekend. “Fedora Outreachy intern Lenka Segura on how Outreachy opened the door for her career to cancer research at IRB Barcelona!”

I put effort into live-tweeting a Twitter thread. Get the full scoop there!

Connect and grow your community through meetups 🔗

Open source communities collaborate in a multitude of ways - chatting on irc, submitting issues and contributing code on GitHub, discussing and sharing ideas on reddit and other social channels. Face to face gatherings add another dimension to that, where community members can learn and share their experiences. Local meetups provide a venue for people with similar interests to socialize and connect. However, organizing meetups is not trivial. How do we encourage and motivate the community to arrange meetups, and to keep the momentum? In my one year with the Ansible community, we have doubled the number of active meetups in Europe. These meetups are community driven, rather than Red Hat. Find out how we use metrics to analyze the situation and needs, and the steps we are taking to reach our goals of connecting with even more community members. Learn from our mistakes and challenges (100 RSVPs and only 20 turned up?), plus some tips to make your meetups more inclusive.

Carol Chen

Carol explained the role of local meet-ups around the world in building communities around software projects. She emphasized that single metrics are not always useful, so it is more helpful to evaluate on multiple areas. The most useful takeaway for me was the 5 W’s: why, who, what, when, where.

• Why? Common curiosity (noticing something new in your community)
• Who? Connections and networking
• What? Concise, compelling content
  • Consider venue travel (how to make it worth their while?)
  • Provide alternatives to git-based submissions
  • All talks don’t have to be technical! Diversify to appeal to wider audiences
    • Announcements for future events, work-life talks
    • We are more than just the technology we work with
• When? Consistency
  • Helps with venue scheduling
  • Helps retain attendee focus and build habits

Carol also gave suggestions for common points to think about for improved inclusion. All of these need active, not passive inclusion.

• Special needs / disabilities
• Food allergies
• Beverage preference (often alcohol/non-alcoholic)
• Language
• Traffic-light communication stickers
• Photography lanyards
• Gender pronouns

Beyond DevConf CZ 🔗

While the sessions are excellent and fulfilling (and sometimes frustrating when you miss a good talk with a full room), DevConf is also more than the sessions. It’s also the people and conversations that happen in the “hallway track.” It was nice to see many old friends and make new ones.

I spent a few extra days before and after DevConf CZ in Brno. In some of that time, my colleague Mike Nolan and I rehearsed in-person for our FOSDEM talk the following weekend (to come in a future blog post). I also enjoyed coffee and waffles with Marie, Sumantro, and Misc!

A few memories of a great week in Brno

Until next time! 🔗

I learn a lot and have a lot of fun at DevConf CZ. I’m happy to return for a third year. Hats-off to the organizers and volunteers who pulled it all off. Each year, DevConf gradually makes improvements. It’s nice to see inclusion prioritized across the board.

Thanks also goes out to RIT LibreCorps for sponsoring my trip. Extra thanks to Jona Azizaj!

My progress with ListenBrainz slowed, but I am resuming the pace of contributing and advancing on my independent study timeline. This past week, I finished out assigned tasks to discuss contributor-related documentation, like a Code of Conduct, contributor guidelines, and a pull request template. I began research on user statistics and found some already created. I wrote one of my own, but need to learn more about Google BigQuery to advance further.

Paving the contributor pathway 🔗

Earlier, I identified weaknesses for the ListenBrainz contributor pathway and found ways we could improve the pathway. This started with the development environment documentation. Now, I helped draft first revisions of our contributor guidelines, Code of Conduct reference, and pull request templates. Together, these three documents have two goals.

1. Make it easier to contribute to ListenBrainz
2. Have a better experience and have fun contributing!

Adding these documents addresses these goals. Additionally, the GitHub community profile also highlights these deliverables as ways to meet these goals. After getting feedback and seeing what others think, we make more revisions later (with some trial runs).

Back to SELinux context flags 🔗

Recently, I set my desktop back up and installed Docker for the first time on this machine; however, the development environment still failed to start. When I ran the script, it would eventually error out because of a permission denial. The web server image for ListenBrainz was failing.

After debugging, I noticed that I missed the SELinux volume tags for the ListenBrainz web server images in my original pull request, #257. When I created the pull request, I might have had cached data that let my laptop run the development environment without a problem. In either case, it was an easy fix and I knew what the issue was when it happened. Therefore, I submitted a new fix in #290.

Writing new user statistics 🔗

The most interesting part of my independent study is working with the music data to build and generate interesting statistics. I finally began exploring the existing statistics in ListenBrainz. The statistic queries use BigQuery standard SQL. BigQuery helps rapidly scan and scale data queries to help with performance (I have a lot to learn about BigQuery).

Two types of statistics 🔗

Additionally, ListenBrainz generates two types of statistics:

1. Site-wide statistics
2. User statistics

Site-wide statistics are metrics non-specific to a single user. There is only one site-wide query now. It counts how many artists were ever submitted to this ListenBrainz instance and returns an integer. There’s room for expansion in site-wide statistics.

On the other hand, user statistics are metrics specific to a single user. There’s a fair number already, like the top artists and songs in a time period and the number of artists you’ve listened to. These are a little more complete and offer more expansion for doing cool front-end work with something like D3.js.

Writing user statistics 🔗

Of course, I had to try writing my own. One helpful query I thought of was getting a count of the songs you listened to over a time period (e.g. “you listened to 500 songs this week!”). I haven’t tested it yet, but I have this in a local branch and hope to test it with real data soon.

def get_play_count(musicbrainz_id, time_interval=None): 
 
 filter_clause = "" 
 if time_interval: 
     filter_clause = "AND listened_at >=
     TIMESTAMP_SUB(CURRENT_TIME(), 
     INTERVAL {})".format(time_interval) 
 
 query = """SELECT COUNT(release_msid) as listen_count 
            FROM {dataset_id}.{table_id} 
            WHERE user_name = @musicbrainz_id 
            {time_filter_clause} 
            LIMIT {limit} 
         """.format( 
                 dataset_id=config.BIGQUERY_DATASET_ID, 
                 table_id=config.BIGQUERY_TABLE_ID, 
                 time_filter_clause=filter_clause, 
                 limit=config.STATS_ENTITY_LIMIT, 
            ) 
 
 parameters = [ 
     { 
         'type': 'STRING', 
         'name': 'musicbrainz_id', 
         'value': musicbrainz_id 
     } 
 ] 
 
 return stats.run_query(query, parameters)

Researching Google BigQuery 🔗

My next steps for the independent study are researching Google BigQuery. After going through the existing statistics and understanding how ListenBrainz generates them, an understanding of Google BigQuery is essential to writing effective queries. When I become more comfortable with the tooling and how it works, I want to map out a plan of statistics to generate and measure.

Until then, the hacking continues! As always, keep the FOSS flag high…

The needs and demands of infrastructure environments changes every year. With time, systems become more complex and involved. But when infrastructure grows and becomes more complex, it’s meaningless if we don’t understand it and what’s happening in our environment. This is why monitoring tools and software are often used in these environments, so operators and administrators see problems and fix them in real-time. But what if we want to predict problems before they happen? Collecting metrics and data about our environment give us a window into how our infrastructure is performing and lets us make predictions based on data. When we know and understand what’s happening, we can prevent problems before they happen.

But how do we collect and store this data? For example, if we want to collect data on the CPU usage of 100 machines every ten seconds, we’re generating a lot of data. On top of that, what if each machine is running fifteen containers? What if you want to generate data about each of those individual containers too? What about by the process? This is where time-series data becomes helpful. Time-series databases store time-series data. But what does that mean? We’ll explain all of this and more and introduce you to InfluxDB, an open source time-series database. By the end of this article, you will understand…

• What time-series data / databases are
• Quick introduction to InfluxDB and the TICK stack
• How to install InfluxDB and other tools

Introducing time-series concepts 🔗

If you’re familiar with relational database management software (RDBMS), like MySQL, tables, columns, and primary keys are familiar terms. Everything is like a spreadsheet, with columns and rows. Some data might be unique, other parts might be the same as other rows. RBDMS’s like MySQL are widely used and are great for reliable transactions that follow ACID (Atomicity, Consistency, Isolation, Durability) compliance.

With relational database software, you’re usually working with data that is something you could model in a table. You might update certain data by overwriting and replacing it. But what if you’re collecting on data on something that generates a lot of data and you want to watch change over time? Take a self-driving car. The car is constantly collecting information about its environment. It takes this data and it analyzes changes over time to behave correctly. The amount of data might be tens of gigabytes an hour. While you could use a relational database to collect this data, they’re not built for this. When it comes to scaling and usability of the data you’re collecting, an RBDMS isn’t the best tool for the job.

Why time-series is a good fit 🔗

And this is where time-series data makes sense. Let’s say you’re collecting data about a city traffic, temperature from farming equipment, or the production rate of an assembly line. Instead of going into a table with rows and columns, imagine pushing multiple rows of data that are uniquely sorted by a timestamp. This visual might help make more sense of this.

Having the data in this format makes it easier to track and watch change over time. When data accumulates, you can see how something behaved in the past, how it’s behaving now, and how it might behave in the future. Your options to make smarter data decisions expands!

Curious how the data is stored and formatted? It depends on the time-series database (TSDB) you use. InfluxDB stores the data in the Line Protocol format. Queries return the data in JSON.

If you’re still confused or trying to understand time-series data or why you would want to use it over another solution, you can read an excellent, in-depth explanation from Timescale’s blog or InfluxData’s blog.

InfluxDB: A time-series database 🔗

InfluxDB is an open source time-series database software developed by InfluxData. It’s written in Go (a compiled language), which means you can start using it without installing any dependencies. It supports multiple data ingestion protocols, such as Telegraf (also from InfluxData), Graphite, collectd, and OpenTSDB. This leaves you with flexible options for how you want to collect data and where you’re pulling it from. It’s also one of the fastest-growing time-series database software available. You can find the source code for InfluxDB on GitHub.

This article will focus on three tools in InfluxData’s TICK stack for how you can build a time-series database and begin collecting and processing data.

TICK stack 🔗

InfluxData creates a platform based on four open source projects that work and play well with each other for time-series data. When used together, you can collect, store, process, and view the data easily. The four pieces of the platform are known as the TICK stack. This stands for…

• _T_elegraf: Plugin-driven server agent for collecting / reporting metrics
• _I_nfluxDB: Scalable data store for metrics, events, and real-time analytics
• _C_hronograf: Monitoring / visualization UI for TICK stack (not covered in this article)
• _K_apacitor: Framework for processing, monitoring, and alerting on time-series data

These tools work and integrate well with the other pieces by design. However, it’s also easy to substitute one piece out for another tool of your choice. For this article, we’ll explore three parts of the TICK stack: InfluxDB, Telegraf, and Kapacitor.

InfluxDB 🔗

As mentioned before, InfluxDB is the time-series database (TSDB) of the TICK stack. Data collected from your environment is stored into InfluxDB. There are a few things that stand out about InfluxDB from other time-series databases.

Emphasis on performance 🔗

InfluxDB is designed with performance as one of the top priorities. This allows you to use data quickly and easily, even under heavy loads. To do this, InfluxDB focuses on quickly ingesting the data and using compression to keep it manageable. To query and write data, it uses an HTTP(S) API.

The performance notes are noteworthy standing up the amount of data InfluxDB is capable of handling. It can handle up to a million points of data per second, at a precise level even to the nanosecond.

SQL-like queries 🔗

If you’re familiar with SQL-like syntax, querying data from InfluxDB will feel familiar. It uses its own SQL-like syntax, InfluxQL, for queries. As an example, imagine you’re collecting data on used disk space on a machine. If you wanted to see that data, you could write a query that might look like this.

SELECT mean(diskspace_used) as mean_disk_used
FROM disk_stats
WHERE time() >= 3m
GROUP BY time(10d)

If you’re familiar with SQL syntax, this won’t feel too different. The above statement will pull the mean values of used disk space from a three-month period and group them by every ten days.

Downsampling / data retention 🔗

When working with large amounts of data, storing it becomes a concern. Over time, it can accumulate to huge sizes. With InfluxDB, you can downsample into less precise, but smaller metrics that you can store for longer periods of time. Data retention policies for your data enable you to do this.

For example, pretend you have sensors collecting data on the amount of RAM in a number of machines. You might collect metrics on the amount of memory in use by multiple users, the system, cached memory, and more. While it might make sense to hang on to that data for thirty days to watch what’s happening, after thirty days, you might not need it that precise. Instead, you might only want the ratio of total memory to memory in use. Using data retention policies, you can tell InfluxDB to hang on to the precise data for all the different usages for thirty days. After thirty days, you can average data to be less precise, and you can hold on to that data for six months, forever, or however long you like. This compromise meets in the middle between keeping historical data and reducing disk usage.

Telegraf 🔗

If InfluxDB is where all of your data is going, you need a way to collect and gather the data first. Telegraf is a metric collection daemon that gathers various metrics from system components, IoT sensors, and more. It’s open source and written completely in Go. Like InfluxDB, Telegraf is also written by the InfluxData team and is built to work with InfluxDB. It also includes support for different databases, such as MySQL / MariaDB, MongoDB, Redis, and more. You can read more about it on InfluxData’s website.

Telegraf is modular and heavily based on plugins. This means that Telegraf is either lean and minimal or as full and complex as you need it. Out of the box, it supports over a hundred plugins for various input sources. This includes Apache, Ceph, Docker, IPTables, Kubernetes, NGINX, and Varnish, just to name a few. You can see all the plugins, including processing and output plugins in their README.

Even if you’re not using InfluxDB as a data store, you may find Telegraf useful as a way to collect this data and information about your systems or sensors.

Kapacitor 🔗

Now we have a way to collect and store our data. But what about doing things with it? Kapacitor is the piece of the stack that lets you process and work with the data in a few different ways. It supports both stream and batch data. Stream data means you can actively work and shape the data in real-time, even before it makes it to your data store. Batch data means you retroactively perform actions on samples, or batches, of the data.

One of the biggest pluses for Kapacitor is that it enables you to have real-time alerts for events happening in your environment. CPU usage overloading or temperatures too high? You can set up several different alert systems, including but not limited to email, triggering a command, Slack, HipChat, OpsGenie, and many more. You can see the full list in the documentation.

Like the previous tools, Kapacitor is also open source and you can read more about the project in their README.

Installing the TICK stack 🔗

Packages are available for nearly every distribution. You can install these packages from the command line. Use the instructions for your distribution.

Fedora 🔗

sudo dnf install https://dl.influxdata.com/influxdb/releases/influxdb-1.3.1.x86_64.rpm \
https://dl.influxdata.com/telegraf/releases/telegraf-1.3.4-1.x86_64.rpm \
https://dl.influxdata.com/kapacitor/releases/kapacitor-1.3.1.x86_64.rpm

CentOS 7 / RHEL 7 🔗

sudo yum install https://dl.influxdata.com/influxdb/releases/influxdb-1.3.1.x86_64.rpm \
https://dl.influxdata.com/telegraf/releases/telegraf-1.3.4-1.x86_64.rpm \
https://dl.influxdata.com/kapacitor/releases/kapacitor-1.3.1.x86_64.rpm

Ubuntu / Debian 🔗

wget https://dl.influxdata.com/influxdb/releases/influxdb_1.3.1_amd64.deb \
https://dl.influxdata.com/telegraf/releases/telegraf_1.3.4-1_amd64.deb \
https://dl.influxdata.com/kapacitor/releases/kapacitor_1.3.1_amd64.deb
sudo dpkg -i influxdb_1.3.1_amd64.deb telegraf_1.3.4-1_amd64.deb kapacitor_1.3.1_amd64.deb

Other distributions 🔗

For help with other distributions, see the Downloads page.

See the data, be the data 🔗

Now that you have the tools installed, you can experiment with some of these tools. There’s plenty of upstream documentation on all three projects. You can the docs here:

• InfluxDB documentation
• Telegraf documentation
• Kapacitor documentation

Additionally, for more help, you can visit the InfluxData community forums. Happy hacking!

Yesterday, Fedora contributor and CommOps team member Bee Padalkar published an article on her blog about measuring the impact of Fedora’s participation at the FOSDEM conference in Europe.

Looking at FOSDEM 🔗

In her analysis, Bee looked at people who scanned the FOSDEM badges for 2014, 2015, 2016. Leveraging tools like fedmsg, she was able to draw conclusive evidence of how people who scanned the badge began contributing for the first time or started contributing more than before the conference. The statistics are fascinating and the analysis is comprehensive in how it measures contributions. It’s worth the full time to read how we’re making an impact at conferences!

Looking ahead 🔗

The other awesome factor of this is that these kinds of reports are extendable to other events in the world of Fedora. Other Ambassadors can use tools like Fedora Badges and track metrics of how they impact and affect the people they engage with at conferences and hackathons. I’m hoping for us to be able to use these kinds of analytics for the past event at BrickHack 2016 that I helped organize as an Ambassador. Stay tuned for an event report and plenty more on the Community Blog with details about BrickHack.

Read all about it! 🔗

Read the full analysis on her blog!