[crawler] Capture project details

[themightychris]

The prototype crawler from #19 only captures a few initial basic project details, Code for Kenya/HURUmap provides a good example of a record with all fields filled in well and some natural changes to them in the history already

What are some more details a v2 should capture? Any thoughts on how we should organize it? (TOML has great support for grouping things any number of levels deep)

I don't think we want to capture any details in the index that routinely change day-by-day in the life of a project (e.g. number of open issues, number of contributors), BUT maybe we do capture things like that as binary or tiered buckets (e.g. has-issue=true or contributors=5-10)

I think we should pull in the GitHub description and/or opening paragraph of the README directly, and then for other big wordy things record their presence, link to them, and maybe measure their health or summarize them if there's a valuable way to do so. (e.g. we can record which license is used and link to the license, we can record which of GitHub's standard community health files are present and link to them)

We should also record the presence of any civic.json or publiccode.yaml file, and pull in some or all of their contents into a normalized form.

[themightychris]

I just found that publiccode.yaml got formalized and submitted to the OSI to be incubated as a stewarded project, it's been through a ton more iteration than civic.json and has first class support for describing that a project isBasedOn another project -- something our network sorely needs that is noticeably missing from civic.json

So it seems to me we should hew as closely as we can to publiccode.yml's schema and essentially just do our best to progressively fill out a massive database of project records that are autofilled as best we can

[nikolajbaer]

Sorry, itchy trigger finger.

i have a couple notes on this from the statusboard perspective. These are debatable, as they are more helpful for reading through it without a secondary "processing" task.

1. It would be nice to have a "manifest" at the organization folder so we can see what organizations are there. This might also help with routing renamed groups.

2. An explicit "name" in the .toml files for both orgs and projects would be nice, rather than the implicit folder / filename convention.

3. Likewise a manifest in the brigade's projects folder, maybe even indicating which ones are active vs. archived, would be helpful in processing.

As to civicjson and publiccode.yaml, i am looking forward to seeing that data in the .toml files!

[gregboyer]

Two questions I'm wondering about, and maybe they don't belong at this step but rather when presenting info:

1. How do we know which brigades to include? (Maybe scrape everything but only display opt-in brigades?)
2. Ditto, but more important for projects

My reasoning is that we need to have a threshold that allows us to present accurate and helpful information, but not add noise such as old or poorly documented projects

[gregboyer]

I do also want to add that I love this middle layer for data standardization and how it'll allow us to make more flexible decisions in the future regarding our sources of information without impacting anything downstream.

[themightychris]

@gregboyer for which brigades to include, we're pulling from the already-curated dataset maintained for cfapi: https://github.com/codeforamerica/brigade-information

My thinking is that we scrape everything into the index repo so people can build different sorts on tools/analysis on top of it. Whatever quality/completion metrics we can layer into the data for easy filtering by tools like what aspiring contributors might search through

[tdooner]

I'd like to add to the conversation a desire to have some information about project activity, for purposes of more easily determining what a Brigade is actively working on.

For example, imagine being able to sort OpenOakland's projects by last commit timestamp (i.e. Github default sort).

When combined with other relevance signals, this gets pretty close to making a fully useful search engine for brigade projects.

I don't think we want to capture any details in the index that routinely change day-by-day in the life of a project (e.g. number of open issues, number of contributors), BUT maybe we do capture things like that as binary or tiered buckets (e.g. has-issue=true or contributors=5-10)

I assume that this preference is a consequence of the fact that we're storing the entire revision history in Git and don't want it to be too big with daily changes to this stuff. I think we should give it a try, though, given how valuable this kind of thing could be. Perhaps if we change the Git commit pattern to be one commit per day instead of one commit per day per organization then the history will be a bit more scannable.

[themightychris]

Git could handle the volume, but we want to distribute the dataset in a fan-out to many applications and keeping track of granularity down to "time of last commit" is just needlessly noisey for all involved.

For I think almost all intents and purposes, it would be just as valuable if we tracked something like last_commit_within = week | month | quarter | year | 2years | 5years | 10years

That would provide a much cleaner signal over time about how active a project is. Think evolving classifications rather than time-series datapoints. We'll have the URL to the git repo in there too if anyone wants to go analyze the timestamps of every commit for a project

[nikolajbaer]

Both are great thoughts! I agree on the signal/noise concerns, but also the value of a metric of activity.

What if we checked the most recent commit on a trigger, and then only update the index data if it slips from one "bucket" to another, e.g. when it transitions between the "last_commit_within" buckets (although maybe start at "quarter" as that might reduce the "noise" a bit). The bottom rung could be "active" for anything that has commit activity within the last 3 months. The challenge will be getting the threshold right so we don't have a lot of projects flip/flopping between "quarter" and "active".

Edit: realizing this was approach was already somewhat mentioned in the original and follow-up comments, so sorry for duplication but 👍 for the tiered idea.