Navigating Community Metrics: The Importance of Contextual Understanding

This chapter explores the complexities of analyzing community metrics within large and diverse project portfolios. It underlines the significance of contextual knowledge and human insight in interpreting data visualizations and advocates for advanced data collection tools to improve metrics analysis.

Transcript

chevron_right

Play full episode

chevron_right

Transcript

Episode notes

Thank you to the folks at Sustain for providing the hosting account for CHAOSSCast!

CHAOSScast – Episode 91

In this episode of CHAOSScast, host Matt Germonprez is joined by Red Hat's Senior Data Scientist Cali Dolfi and Community Architect Josh Berkus to discuss their experiences in measuring and maintaining open source community health. They delve into their day-to-day roles, challenges, and key projects like Project Aspen, the importance of contextual metrics, and the impact of generative AI on their work. Also, they emphasize the importance of goal-oriented metrics and establishing repeatable processes in OSPOs. Press download to hear much more!

[00:00:40] Cali and Josh share their backgrounds.

[00:02:02] Cali talks about her work as a data scientist at Red Hat, focusing on community open source metrics and mentions her recent projects, including Project Aspen, and her role in developing platforms for data visualization and metrics.

[00:04:34] Josh discusses his day-to-day responsibilities which include stewarding Red Hat’s involvement in cloud native projects and committee work with the Cloud Native Computing Foundation.

[00:06:17] The discussion shifts towards the health of collections of projects or ecosystems and Cali and Josh share their thoughts on how they approach ecosystem health, particularly with the cloud native space. Josh focuses on Kubernetes and its connection to various projects.

[00:09:17] Matt questions if Red Hat often plays a stabilizing role within these ecosystems, especially in times of crisis or instability.

[00:10:29] Cali discusses current hot topics in open source community health at Red Hat, focusing on SBOM (Software Bill of Materials) analysis and its implications for security and maintenance within the tech industry. They discuss the importance of understanding vulnerabilities within open source projects and the role of maintainers in mitigating these vulnerabilities.

[00:14:51] Matt asks about identifying vulnerabilities in upstream projects and notes the challenges of visibility due to numerous projects. Cali explains their approach of analyzing the entire codebase, using visualizations on the ‘8not dashboard’ to monitor active maintainers in different project areas.

[00:16:43] Josh discusses mainstream tooling focused on known vulnerabilities and emphasizes the need to predict future vulnerabilities.

[00:19:16] Matt inquires about handling the variability and contextual specificity of metrics across numerous projects. Cali discusses the importance of contextual understanding in interpreting data and metrics, emphasizing the need for community involvement to enrich the interpretation. Josh argues that improving data collection methods to incorporate contextual knowledge is crucial, aiming to shift some analytical responsibilities from humans to algorithms.

[00:24:19] A discussion starts on the role of generative AI in current tech, prompting Cali to reflect on the impact of AI hype cycles on resource allocation within the industry. Josh acknowledges that while some open source machine learning tools have benefited from increased resources due to the AI wave, the introduction of generative AI in community projects has often been problematic.

[00:30:03] The conversation shifts back to the challenge of AI-generated contributions to open source projects. Josh and Matt discuss the potential need for Red Hat’s Open Source Program Office (OSPO) to adapt its analytics and policies to manage the influx of such contributions.

[00:31:35] We close with Cali offering advice to new OSPOs on setting up robust data analysis infrastructures from the start, and Josh reinforces the need for goal-oriented metrics and processes advising OSPOs to design operations that are sustainable and scalable.

Value Adds (Picks) of the week: