Survey says: What? 🔗

At some point, after we opened the survey, a question emerged about how to handle an unruly response. In the ongoing responses, our data manager noticed one response that was objectively harmful. The person was strongly against the D.E.I. initiative that organized the survey. The response was written in a hostile tone, made insulting and derogatory comments about groups of people, and was entirely opposed to the project spending any time and resources on diversity, equity, and inclusion. The question asked to our group was whether we would include this response in the published data, or whether we would omit it.

There were two perspectives. Some elected to remove this response from the final report and any published data. Others felt it was important to wait and see if this response would become a pattern as we ran the survey. I found myself in the second group that felt it was important to wait and see first. I want to unpack this rationale, both for future me and perhaps someone else reading.

On discarding the survey response 🔗

There were good points about removing the harmful response.

Firstly, the response used harmful language and was likely triggering. This particular response included angry rhetoric that was reflective, to a degree, of the social and political “climate” of our world today. Including the response in our final reporting could also be giving it a platform, which would arguably be a harmful act. It would validate that input as acceptable input. Our group was not in disagreement that the response was harmful and not behavior the community should tolerate.

Second, the response did not provide actionable insight or useful asks to the project and community. It was written in an aggressive, angry tone towards the reader and did not offer workable suggestions other than ending and divesting from all D.E.I. work immediately. Given this was not an acceptable option, there wasn’t much there for us to learn or understand about CHAOSS from this individual response. So, why include or save this response?

There is an option to ignore feedback by intentionally discarding it, but what if the individual feedback represents a larger trend?

What is community culture? 🔗

It is important to be aware of threats to community culture. What is community culture? My improvised definition is any organizational culture oriented towards the care, well-being, and thriving of others (including the self) within a single, shared community environment. Regardless of other values and goals in a project, the shared culture of the project can either lean towards a collective, communal-oriented approach or an independent, individual-oriented approach. The communal approach that prioritizes the well-being of all instead of a privileged view could also be considered as community culture. Many traditional “Open” projects skew toward a strong community culture.

On monitoring survey responses for a pattern 🔗

Coming back to the survey response, what if omitting the data leaves holes in the story of your community? If there is not just one, but several of these kinds of responses, what comments does that make about the community culture? Is there already a strong community culture, or is there resistance and challenges to building a more cooperative, caring environment? There is real work to do at both ends of the spectrum, but what that work might look like depends on which side you are on.

I posit that omitting the “unhappy” or harmful responses can create a dangerous blind spot to toxicity within a community culture. When it comes to direct, interpersonal interactions with others (e.g. meetings, emails, chats, etc.), stewards of the community culture need to take direct action against visible challenges and threats to the community culture. If someone starts swearing out at someone in a meeting, that is a hard-to-miss action. It is visible, and anyone could observe it or even record it.

In anonymous surveys, you might find a more subtle layer of the community culture than what is shown by the actions of a small few. There can be greater trust that someone’s comments will not be tied back to their identity, so some responders may feel emboldened with their words and true opinions.

Don’t discard a blind spot. 🔗

The point of this is that especially in larger communities, it is worth noting negative and harmful responses and not totally ignoring them. Communities that organize in more decentralized ways will always have supporters, users, and contributors from both the core and the periphery. The core project membership may not interact or engage often with the periphery often, so there can be a blind spot to parts of the project that identify with the community but are a few degrees removed from the inner ring of the project community.

Noting whether something is indicative of a larger pattern is important. If your community has a ton of jerks, you need to know that your community is full of jerks so that you don’t waste time persuading people otherwise, when the lived experience is very different.

In the original conversation with the CHAOSS Project team, this data scrubbing question emerged in the process of running the survey instead of after the data collection concluded. The survey later closed and our data manager confirmed that the flagged response from earlier was the only one of its kind. As a group, we then felt more confident in discarding that one outlier as an anomaly since the survey was open to the general public.

Feature photo by JESHOOTS.COM on Unsplash. Modified by Justin Wheeler.

My brief readout is a guided reflection using questions shared by Matt Germonprez. This reflects my review of our work as a team to date and also shares some of my hopeful outlooks for what our amazing team can accomplish together. This readout will cover (1) our accomplishments as a team, (2) what was expected and surprising, and (3) what we could change in the next year.

CHAOSS accomplishments & learnings 🔗

Three achievements and aspirations stand out over the past year:

1. Established process management and a team workflow.
2. Created a small but active Community of Practice (CoP).
3. Sharing our results with CHAOSS and the Open ecosystem.

Processes & workflow 🔗

For direct participants of the team, the Ford Foundation funding did not come with strict requirements or success metrics. As we assembled our team, we were given the discretion of how to conduct a D.E.I. review for the project and determine the best course of doing that. This allowed for creative freedom to figure out what would work best for CHAOSS. Additionally, I could not identify a straightforward way to discover other Open communities and projects doing our kind of work. Since there were also not many other known successful models to follow, we combined our shared experiences across multiple Open communities to build our team, identify main areas of focus, and engage the community around our efforts.

This is an achievement because we collectively created an active group that makes incremental, positive changes to CHAOSS. This is a model we could share with other projects so that others can learn from our experiences.

Community of Practice 🔗

Our team is a small but engaged group of D.E.I. practitioners. We share a connection through our ongoing review of the CHAOSS Project, but we also give and take from our own personal experiences outside of CHAOSS. Our group regularly meets and discusses complex, difficult issues that are both (a) not easy to discuss openly and (b) applicable to many communities beyond only CHAOSS. Our team meetings are a safe space that promotes honest and constructive discussion centered on diversity, equity, and inclusion. In addition to our recommendations and direct efforts with CHAOSS, I often reflect on our conversations as a team when working with other Open communities. An example of this is how we built a list of questions to get a “pulse” from the community on their feelings about CHAOSS.

Sharing results with CHAOSS and beyond 🔗

This is aspirational and not yet fully realized. Our team has collected a solid portfolio of stories and experiences that other communities would stand to benefit learning from. I consider this a current achievement because while our work does specifically look at CHAOSS, we also often reflect from a general perspective and how a topic of interest might look in other communities. When the time comes to package our findings, I believe we are setting ourselves up for easier messaging and outreach opportunities in the future.

According to expectations 🔗

While I have worked in Open Source D.E.I. communities since 2015, I have never conducted an applied research review for community D.E.I. before. I did not come into this with strong immediate expectations because it would inevitably reflect the backgrounds and strengths of the team we would assemble. However, I did have specific hopes or things I hoped would be realized by this work.

As expected 🔗

• Data-driven approach: We began this work without a strong representation of the state of CHAOSS. What do contributors think about the project? While data is not a universal panacea, we gravitated to a community survey early on because we needed to understand the community experience better first before making serious suggestions.
• Time zones are hard: Our team was spread out across North America, Africa, LATAM, and Europe. Additionally, the work with CHAOSS was also a part-time venture for most of us, in addition to primary employment. Calendars and schedules are hard to get right. Since our team’s organization was ad-hoc, momentum would occasionally slow for some periods.
• We have an amazing team! I expected great things once we identified our roster. We have also had more amazing people join us over time and add new passion and insight to our focus as a group.

Surprises 🔗

• Documenting our impact is not always intuitive: While we have done internal storytelling work within the CHAOSS Project, we do not have a good record of our achievements to date. Our linear progression does not lend itself easily to self-reflection and recalibration. Although much of our focus is on the CHAOSS community survey and CHAOSS Africa, we also facilitated several other notable achievements in the project in the last year. See the following examples:
  • Supporting the establishment of a Code of Conduct Committee.
  • Community office hours for newcomers.
  • Improved, peer-to-peer onboarding experience in CHAOSS.
  • Increased efforts in CHAOSS mentored projects (e.g. Outreachy and GSoC).
  • Recommending changes to the project and community, like broader localization to Chinese & Spanish and establishing a D.E.I. council.
• Losing and regaining steam on the survey: Although the community pulse survey was one of the earliest tasks identified in our work, launching a first survey proved to take a lot of resources from the team. We briefly stalled out on the survey effort while focused on other areas (like listed above). While our team was able to achieve many smaller victories for CHAOSS with low-hanging fruits, it took a sustained focus and slowdown on new topics to achieve larger contributions like the community pulse survey.

Changes for the CHAOSS team next year 🔗

Looking ahead to 2023, I hope to strengthen our efforts as a team in these areas:

1. Packaging our work
2. Dissemination of our work

Packaging 🔗

Our work stream was linearly ordered and we took a forward-looking approach. Now is a good time to look back and reflect on our results to date. What are our key findings and observations? What suggestions will we make to CHAOSS? How could other communities learn from our experience running this review? One task for us as a team is to identify key messages and themes so that dissemination into broader domains is possible.

Dissemination 🔗

Once we package our work, notes, and reflections, we should take an active approach to disseminating and sharing our work. This includes both the CHAOSS Project and a more general audience. For the CHAOSS Project, this could be a written report, presentations to the CHAOSS board, speaking at CHAOSScon, and outreach to the multiple Working Groups. For a general audience, this could include speaking at industry conferences, sharing our work with other Communities of Practice, social media, or other ways of promoting our deliverables.

This post is a reflection on the past year in my work with the UNICEF Venture Fund. We integrated new open source tools to capture metrics and data about open source repositories connected to UNICEF portfolio companies and created a shortlist of key metrics that map to business sustainability metrics. Now, we are better positioned to look back on past, current, and upcoming portfolio companies and mentor support programs.

As we move into 2022, this post covers my current thinking on these points:

1. Defining investments.
2. How do these investments impact sustainability?
3. CHAOSS metrics as an open source tool for an investment lens on sustainability.
4. What next?

Defining investments. 🔗

When we talk about investing in open source, what do we mean? What are the known inputs? What are the expected outputs? “Investments” and “investing” are broad terms. Investments typically mean sizeable financial injections of support and growth, but can also include non-financial investments too. Investments can also take the form of both time and energy (i.e. electricity and digital infrastructure).

The UNICEF Venture Fund provides equity-free funding for start-up companies building open source solutions of interest to UNICEF. All the start-up companies are registered companies in UNICEF program countries. As part of the Venture Fund’s location in the Office of Innovation, it is also a vehicle for UNICEF to explore frontier technology areas through the investments. When a start-up company is receiving investment from UNICEF, the company receives both funding and tailored mentorship about business and open technology.

A question I want to know is, what is the impact of the received funding plus guided mentorship? How does this approach enable the companies to be successful after graduating? What discoveries or knowledge could be shared with others to assist the development of their own open programs?

To summarize, an investment can be financial or non-financial. Financial investments include direct funding, grants, venture capital, fellowships, or any other exchange of capital. Non-financial investments include time spent in coaching sessions, personalized content for companies, and shared digital infrastructure. Neither list is exhaustive.

How do these investments impact sustainability? 🔗

Data makes introspection easier. Bitergia’s Cauldron.io was a champion tool for kickstarting an open source metrics strategy for the UNICEF Venture Fund. Its introduction as a tool opened up a wider span of data to look at. There are new opportunities to ask questions and explore growth, scale, and sustainability.

In order to come to a conclusion on sustainability impact, we need streamlined data to test a thesis. The Venture Fund team improved internal processes to how metrics are collected from portfolio companies. The team is unifying behind fewer tools and methods to ensure we see the same data and have the same view of the data points we measure. This also provides a fresh opportunity to review how we measure open source impact across portfolio companies. Many have dashboards on Cauldron.io, but data needs a storyteller for it to make meaning. So, the next step is to ask questions with this new data and frame a thesis to measure and test the sustainability of Venture Fund investments into open source.

Many have traveled before me on the same trail of thought. I started first with the Community Health Analytics Open Source Software (CHAOSS) project and its metrics releases. This served as the initial point of brainstorming to frame questions and different scenarios of risk, evolution, DEI, and value.

CHAOSS metrics as an open source tool for an investment lens on sustainability. 🔗

I reviewed the latest release of CHAOSS metrics and narrowed down four metrics I want to measure in the next year. I also shared thoughts on why collect this data and how to do it. This blog post is no more than me wondering out loud, to help me frame an analytical approach for this metrics strategy.

The four metrics are detailed below:

1. Contribution Attribution
2. Contributors
3. Collaboration Platform Activity
4. Labor Investment

Contribution Attribution 🔗

Question: Who has contributed to an open source project and what attribution information about people and organizations is assigned for contributions?

chaoss.community/metric-contribution-attribution/

This metric is insightful because it is targeted deeply into team and project culture. This metric is a good representation of how much the project leans into an open source model of building their project. This work ethos and intention to forge on an open source path is difficult to understand at times. If a team takes care to attribute their software dependencies and other contributors to their code (if any), this is a good sign that the team accepts collaboration as a value and encourages working with others.

I would measure this across two types of contributions: attributions for software dependencies including those with permissive licenses, and for any other direct contributors to the code and how they are recognized for their participation. This could be filtered in a red-yellow-green light approach:

1. Red: No attributions are made, or all attributions are inadequate.
2. Yellow: One of two attributions are made, or one attribution type is inadequately attributed.
3. Green: All dependencies and used works are correctly attributed.

Contributors 🔗

Question: Who are the contributors to a project?

chaoss.community/metric-contributors/

This metric explores a more human dimension of the people and participants to an open source project. The metric defines contributors and contributions broadly, as “anyone who contributes to the project in any way.” Understanding the people participating in a community, their motivations, goals, and why they choose to be in that community is important to understand sustainability. Otherwise, you may lose out on good opportunities to attract contributions from people who are already engaged, and new engagements may be difficult because of a mismatch of expectations.

This metric is more a means than it is an end; that is, it provides opportunities to ask more questions than provide detailed answers. Nevertheless, it does provide some guidance towards understanding contributors in a project, and it can lead to some concrete actions based on gathered insights. For example, this metric will enable deeper looks in areas of diversity, equity, and inclusion.

Since I work with start-up companies with small, lean development teams, I look to understand the motivations of the developers working on their projects and where the motivations may align with another open source solution. This enables the two communities to leverage their combined brainstorming for meeting complimentary goals around development and innovation.

To collect this data, I would have the team define what areas of contribution they seek for their open source solutions and then map those desired contributions to a specific project area or different team members. This enables a form of consistent accountability for checking expectations with reality and understanding team capacity. Each area could be a key-value pair, where the value is the project area, team lead, or delegated team member for the type of contribution solicited.

Collaboration Platform Activity 🔗

Question: What is the count of activities across digital collaboration platforms (e.g., GitHub, GitLab, Slack, email) used by a project?

chaoss.community/metric-collaboration-platform-activity/

Collaboration platform activity is one effective proxy metric for community engagement if measured accurately. The metric does not define collaboration as much as it provides a data structure to measure it. It abstracts collaboration into key data points like timestamp, sender, whether the platform has threaded or non-threaded discussions, data collection date, and platform message identifier. To a degree, collaboration can be abstracted out in this way: a person takes any given action at a given time in a given way, and this action is measured as project-related activity on the collaboration platform.

There are a few possible approaches to collecting this data from UNICEF Venture Fund companies. Each approach does not cancel out another, but each approach could be combined with the others:

1. Measure common git activity like commits, issues, pull/merge requests. We already measure this data, but use it only in connection to validating Venture Fund workplans for each team with UNICEF portfolio manager(s).
2. Count communications like comments, reviews, public messages, and other outreach. Communications strategies and tools are typically inferred from common git activity. Measuring for engagement and stratifying those metrics into a smaller group could allow for deeper insights to the evolution of early-stage open source communities.
3. Make community hubs first-class citizens in the data curation process to infer about informal engagement. Both open source projects and UNICEF Venture Fund portfolio companies use a variety of tools to communicate, especially in view of COVID-19 and its seismic impact on how we work. Platforms like Discord, Telegram, Mattermost, Slack, Rocket.chat, Matrix, and others are focal points where projects collaborate, ask questions, and support others. Bringing this data stream into the mix offers deeper insights into how teams engage and build community around their work, and also guidance on when to push for contribution opportunities at the right time.

The satisfaction of these three options in their totality is not enough. To leverage the fullest impact, these metrics must tie into each other, and need to be connected back to a narrative. Why is this data being collected and what actions are influenced by the knowledge of this data? The data collection enables the evaluation of sustainability and understanding the birth, growth, and evolution of an open source technology product. Influenced actions can include moving more human resources (i.e. contractors or staff) to support a project, adopting a new open source best practice, and/or engaging new customers, talent, or other leads based on participation in the community.

Measuring collaboration platform activity is not black and white. Many new questions would likely come forward as part of measuring this activity. Yet this is the point—it lays the foundation for the next layer to the data collection, analysis, and reporting process around sustainability.

Labor Investment 🔗

Question: What was the cost of an organization for its employees to create the counted contributions (e.g., commits, issues, and pull requests)?

chaoss.community/metric-labor-investment/

This metric is perhaps the most ambitious of the group. How do you measure labor investment into an open source project? Or literally, the number of person-hours that go into software design, development, co-creation, and community management? It feels like a gargantuan effort, but there may be better ways to measure this in connection to other data the UNICEF Venture Fund is already connected about the businesses.

Measuring labor investment impacts two narratives: the rate of development on the open source work, and the impact of UNICEF investment into a company backing an open source work.

Firstly, understanding the rate of development on an open source work is easier to infer by understanding who is allocated on a project and how much of their time they dedicate to it. If a team of three contributors spares a few hours a week, it will mean something different compared to a team of five engineers spread across different disciplines working full-time. Mapping the labor investment for open source projects supported by UNICEF would enable better planning by understanding the typical labor investment in open source workplan tasks as piloted by other Venture Fund portfolio companies.

Secondly, this gives us a new way of talking about the impact of UNICEF Venture Fund investments as an investment not only in software products but also in labor. It gives us insight into the investment of labor in software engineering talent among portfolio companies. How does this measurement change over time of the investment? Do projects receive more or less investment of labor during the 12 month period we work with them? This could also be used as a proxy metric for the impact of our unique mentorship and coaching opportunities.

What next? 🔗

Knowing is half the journey. Even if the knowledge is not yet firmly rooted. The analysis and introspection are from me as an individual working among the UNICEF Venture Fund and do not represent the views and beliefs of UNICEF or the UN in any capacity. My intent is that by sharing this analysis in the open, it allows for a space where conversation can spark where it could not before. It also invites others to share ideas, feedback, and constructive criticism of an emerging metrics strategy for investments made into the open source ecosystem.

Next, more layers can be added and internal and external validation can help to keep this moving forward. An implementation plan would be the next step to follow this post. The implementation plan considers the process of how start-up companies move through the Venture Fund from start to finish. Who interacts with the companies and when? At what point is a company ready to begin building in a new metric or count in their monthly metrics? Do they understand the implications and assessments of these metrics? At what points in the process is data already being collected? Could these new data requests be added to existing requests? And so on.

I hope to formalize some of this new reporting and metrics strategy in upcoming cohorts in 2022, as part of a renewed effort into communicating how our open source investments tie into sustainable impact towards the U.N. Sustainable Development Goals.

This post will serve as a milestone marker on the metrics strategy discussion in the coming one to two months. See you in 2022.

Featured photo by Edward Howell on Unsplash. Modified by Justin Wheeler. CC BY-SA 4.0.

Making graphs is easy. Making intuitive, easy-to-understand graphs? It’s harder than most people think. At the Rochester Institute of Technology, the ISTE-260 (Designing the User Experience) course teaches the language of design to IT students. For an introductory exercise in the class, students are tasked to visualize any set of data they desire. Students David Kim, Jathan Anandham, Justin Wheeler, and Scott Tinker used the MusicBrainz database to look at how five different Queen songs went mainstream in different ways.

Five factors of Queen 🔗

Our mini data science experiment decided to look at five unique data points available to us via MusicBrainz Works:

• Number of recorded covers
• Number of artists who covered a song
• Release year
• Year of last recorded cover
• Time elapsed between release year and year of last recorded cover

Originally, we looked at songs from different artists, but decided to look at five recordings from the same artist. With Queen being a notoriously famous band, there were several data points to work with in terms of how often a song was covered.

Making sense of the data 🔗

A few explanations are necessary for some of the data, especially the difference in number of covers and number of artists. Don’t Stop Me Now, Fat Bottomed Girls, and We Will Rock You had the same number of recorded covers as number of artists who have covered the song. Why were Another One Bites the Dust and Bohemian Rhapsody different?

As it turns out, Another One Bites the Dust had more covers than the number of artists who have covered the song. This happens because some artists have covered the song twice (e.g. once on a studio release and another on a live recording release). On the other hand, Bohemian Rhapsody had more artists covering it than number of covers because some recordings featured multiple artists on the same cover (e.g. the 1992 live performance with Elton John and Axl Rose).

The data opens many interesting questions. Why have some songs persisted longer than others (in terms of recent covers)? Have these songs impacted culture and society in different ways? How have they permeated culture? Is there geographical bias in the data?

This exercise was an exploratory assignment, but we had fun visualizing it and ended up learning an interesting pattern in music data.

Check out the presentation and paper 🔗

If you’re interested for the full details, the slides and a short paper about the presentation are available online. They provide deeper context for the research and the visualization details based on different design concepts.

You can see what else David Kim, Jathan Anandham, Justin Wheeler, and Scott Tinker are up to on LinkedIn. Thanks for tuning in to this adventure into music data analysis, powered by MusicBrainz!

Photo by Matthias Wagner on Unsplash.