Proposed template to gather info from projects and maintainers

[edsadr]

Questionnaire and Metrics for OpenJS Foundation Projects

Questionnaire for Project Members

Project Details

• Capture essential information about each project, including its name and overview. Document key milestones or achievements from the past year to understand development trajectory and successes.

Challenges and Solutions

• Detail significant challenges faced and the strategies or solutions implemented to overcome these issues. This helps identify where the foundation can assist.

Future Outlook

• Outline short-term and long-term goals. Helps the foundation understand project direction and support their path.

Community Engagement

• Discuss methods and channels used for engaging with the community, including forums, social media, and other communication channels.

Contributor Analysis

• Provide insights into the number and activity of contributors, tracking trends over the year.

User Feedback

• Gather common praises or complaints from users to guide future development.

Foundation Support

• Query about the support received from the OpenJS Foundation and additional support needed.

Ecosystem Improvement

• Invite insights and suggestions on improving the overall JavaScript ecosystem.

Additional Insights

• Allow projects to share other relevant information or experiences not covered in other sections.

Metrics for Measuring Project Status

Contribution Metrics

• Monitor pull requests, issues, and commits to quantify project activity and community involvement.

Community Metrics

• Track growth in contributors and followers, and subscriber metrics to assess popularity and reach.

Project Activity

• Count frequency of updates and releases to gauge project dynamism and responsiveness.

Engagement Levels

• Analyze activity in discussions and support channels to measure community health and communication effectiveness.

Adoption Metrics

• Measure downloads, installations, and external mentions to quantify project adoption and relevance.

Diversity in Contributions

• Evaluate contributions across various areas like documentation, code, and community management.

Survey on Maintainer Engagement

Engagement

• Measure maintainers' satisfaction with the engagement level within the community.

Support

• Assess the effectiveness of support provided by the foundation and identify areas needing additional resources.

Collaboration

• Understand how well team members and external stakeholders collaborate.

Community Spirit

• Describe the spirit or atmosphere within the project community, impacting volunteer retention and satisfaction.

[edsadr]

cc @openjs-foundation/ecosystem-report

This is first proposal for the questionnaire that we would send to every project, the idea is to to discuss in this issue and finalize (if possible) in the meeting next week.

[PaulaPaul]

Hi @edsadr - I'd like to help with this and am wondering if we've picked a mechanism for providing the survey to members (Google Form, GitHub, other?)
GitHub would be interesting as it is open and can be directly correlated with the project, but can be more challenging to correlate the results. Google Form is easy to work with, but it requires that we have up-to-date email addresses for the project maintainers, and it is harder to track whether people received the request...
If the survey mechanism has been decided, let me know and I'm happy to create a draft for collaboration/review-
Thanks!

[ljharb]

A Google form can be distributed via GitHub.

[PaulaPaul]

From today's call, there were suggestions to organize these metrics into those can be specifically measured / quantified, and those that are more subjective. Also, end user vs. maintainer/team metrics. In addition, provide some thoughts on which metrics would be actionable or lead to action vs. those that are just for information and open sharing.

Providing data vs. providing opinions was discussed, and avoiding becoming prescriptive or interpreting the data before we have consistent results.

The topic of how the metrics could be captured was discussed (e.g. use LFX Insights or create a bespoke metrics capture capability for the OpenJS Foundation using https://github.com/chaoss/grimoirelab which underlies LFX).

Perhaps we can help people understand what metrics really mean. What would 'green yellow red' really mean? Ensure people understand the value in numbers alone vs. trends over time. Creating a baseline would be an important step there.

Should we start small?

Attendees were asked to review and rank the metrics in this issue. These are for the project, not for the users. LFX also has project insights, and @rginn can connect us to the fact sheet. Note that not all projects opt in to metrics collection and we will need to work with projects directly in some (or many) cases.

@edsadr will mark the items here that can be collected directly form GitHub or other source by automation

[mhdawson]

@edsadr your set of questions were a great starting point for the discussion today. Using those as input these are a few of my thoughts:

Goals

Some of the goals that I think we discused as well as some thoughts I had while writing this down:

• Help end users understand how Foundation projects fit into their plans
• Share the state of each of the projects within the Foundation (active, emeritus)
  • For active projects share more detailed info
    • Include metrics that are important to users of a project, for example release cadence, last release etc.
    • Include current end user satisfaction, as well as pain points
    • Include sentiment/health of collaborators including available resources versus resources available, and how the Foundation is helping projects
• Provide a channel for end users to share their requests and pain points on Foundation projects
• Share information to help build end user involvement and support of Foundation projects
  Capture data to help the Foundation make its projects successful

[mhdawson]

And having read through the questions above, a the report might look like in terms of what I think would be interesting. It covers some of the same things you had above but as you asked for in the meeting I trimmed it down to the ones I thought were most interesting and tweaked into what I see supporting the goals we discussed.

Summary

Summarize interesting points from the data captured and shared across the projects

Foundation (from user survey)

• Percentage of respondents who company are a member of the Foundation

• X% of respondents are planning on Joining the foundation this year

• Y% of respondents indicates that a project being part of the Foundation increases their likelihood of using a project

• Most end users expect the following from the Foundation: a, b,c

• End users also support projects by:

Projects

Project 1

Key project details

• Created
• Collaborators/contributors
• Collaborator diversity
• Last release
• Number of releases this year

Adoption

• Downloads or equivalent

Future Outlook

• Planned Release cycle
• Major plans for next year

Collaborator sentiment and resource needs

• Does the project have the resources and people it needs to be successful
• How good a place is the project to work in (community Spirit)
• Resources currently provided by the Foundation
• Satisfaction with support received by the Foundation
• Major challenges

User Feedback (From User survey)

• User happiness with current state (1-10)
• Major Feature requests
• Major pain points
• Plan to use more, or plan to use less

Help Needed

• Help needed by project
• Ways to get involved
• Ways to support the project financially

[edsadr]

Based on our last meeting, I sent a PR to proposing the goals we discussed, please check it here: #5

[edsadr]

I also created this document specifying what metrics can be collected automatically and what will require manual effort, I would suggest we discuss this in a Google Drive Document here: https://docs.google.com/document/d/1tz1YHM_6NZs0bDfjvtT_ff1XkPBWHb_4otzhht5vi_o/edit#heading=h.u27qs6aj6wpc