Adapt data entry to data analysis needs

[ebengoechea]

Once the main additional metadata is added to DYE, I'm thinking of enabling some setting option to restrict the available input data to sets or fixed categories so that metadata can be used for data analysis. This will likely take a while, though. See Diaspora's Bayesian Bears' project.

[JonnyTran]

@ebengoechea I too echo this. I'm currently working on a dataset from the Visualizer.coffee, and the data is awesome, but there are some data entries issues causing the dataset to be less than perfect.

I would suggest that the grinder_setting input field should not be carried over from the last shot. Since the users changes grind setting often and are very likely to forget updating this field in every shot, this issue will cause many incorrect data entries that are very difficult to remove and would mess up the data analysis.
I think the grinder_setting field should be a null value by default unless entered by the user. If the android app would let the user enter numpad, that would be stellar.

[ebengoechea]

@JonnyTran To me grinder setting absolutely needs to be propagated from one shot to the next. Nobody has ever requested this not to be so (except when no field is propagated), so I think your workflow in this respect is not common at all.

[JonnyTran]

That's fair enough, I guess in the majority of the cases the grinder_setting propagated values would be correct.
That brings me to your original topic, of

restrict the available input data to sets or fixed categories so that metadata can be used for data analysis

As you know, users have a free form to enter grinder_model, bean_brand, bean_type and roast_level. As these are categorical variables, the user-entered values can have many variations, typos, capitalizations, and abbreviations for the same category. For data analysis, if these values are not consistent and normalized, we'd end up with smaller sample sizes and oddities in the dataset that make it less useful to extract insight from.

For an example on `grinder_model`, here are the top 20 most frequent values:

Niche Zero              6025
Niche                   4042
Lagom P100               910
Bentwood Vertical 63     810
                         760
EK43 with SSP HU         716
Monolith Flat            678
Lagom P64                676
DF64                     576
Mahlkönig EK43           437
Bentwood 63              396
Titus M3                 394
P64                      384
Sette 270wi              381
niche zero               352
bentwood                 346
Mahlkonig EK43           343
Kafatek flat             337
M47 Simplicity           326
Sette 270Wi              317

You can see that {Niche, niche zero, nz, zero} should be normalized to "Niche Zero", and {bentwood, Bentwood 63} to "Bentwood Vertical 63". There are also numerous empty space "" values.

Another point is that some users would enter the burr option along with the grinder_model e.g. "EK43 with SSP HU". IMO this grinder_model field should be normalized to grinder model only, and the burr set should be another separate field. This would make it easier to group shots from the same grinders which has the same grinder characteristics and grind_setting scale, rather than creating another grinder category for a different burr set.

To deal with these issues, I would suggest:

1. Collect a comprehensive list of known grinder model names, so as the user type on DYE, use string matching to suggest the closest grinder models to choose from - saving typing time and helps user find the correct model name, but also allow the user to set their own if they have a novel grinder. This potentially let us join metadata about the grinder characteristics such as RPM, default burr set, avg. retention, etc. Perhaps I can reach out to @miharekar on the Visualizer.coffee end to create an updated list.
2. Create another burr set field that the user can enter, or automatically inferred from the grinder_model's default burr set.
3. For other free-form fields such as bean_type, the comprehensive list of bean names for each roasters can also be collected. Perhaps drinktrade.com would have a very comprehensive list.
4. For other fields such as roast_level that is on a standard scale, users can only enter fixed categories, e.g. "light", "medium-light", "medium", "medium-dark", "dark" or Agtron scale.

I'm not familiar with the DE1app code base, and not sure if TK allows for this interactive string matching lookup in the free-form text fields.

[miharekar]

The biggest problem is how few people even enter this. Cause 716 EK43 with SSP HU? That's all just my shots, and it's the 5th most common entry. 😂

So you'd have to make the workflow of entering this much more streamlined and desired for people actually to do it. Because now most of them do not.

[ebengoechea]

All of this is planned as a feature optional setting. But it's not among my priorities, so it will likely take a while.