[CEP] Supporting domains without privilege while GAing feature-flags to paid privileges

[sravfeyn]

Abstract

This proposes a solution on how to migrate feature-flags/toggles (when GA-ing) into paid privileges for domains that may not have a paid plan or are below the plan that the toggle is being upgraded into.

The solution proposes to migrate the StaticToggle document into a new FrozenToggle document (thereby disabling its Edits in the Edit Toggle UI for further edits) and patch domain_has_privilege with a decorator that checks the FrozenToggle for enablement for a given domain.

Motivation

Currently, we are in the process of GA-ing all the feature flags and deleting them after GA. It is often the case that some of the flags need to be converted into a new privilege under one of the paid subscription plans. The domains that have these flags enabled may not be subscribed to these plans and we might want to still enable the functionality provided via the new privilege to these domains.

There is an existing strategy that has been used to do one of migrations such as ERM migration. This involves creating custom software plans and roles to grandfather the domains under the new privilege. While this works well for one-off cases, it has few below drawbacks.

1. Extra custom subscription (role, plan) model objects will need to be created for each of the domains and privilege to be migrated (potentially around the order of Number of Domains X Number of toggles). This could get more complicated as the toggles will be migrated one after another, hence creating custom plans/roles on top of each other to capture various combinations of the feature flags on various domains.
2. The process involves multi stage migration.

Given that we want to migrate a large number of toggles, this becomes specially challenging, so a simpler method is desired.

Specification

1. A new FrozenToggle class will be added with pretty much the same spec as Toggle class except that the 'enabled_users' attribute on FrozenToggle class (which holds enabled domains also) will be frozen for further modifications.
2. The FrozenToggle document will have an additional attribute to hold the new privilege which the flag will be migrated to.
3. domain_has_privilege function will be updated to check FrozenToggle document to lookup domains that have the toggle enabled historically.

The migration process involves

1. removing the StaticToggle for the feature flag,
2. creating a new FrozenToggle object with the new privilege and
3. writing a Django migration that converts the Toggle document into FrozenToggle document

The migration can be run as part of the migration inside the deploy. After the deploy the toggle will not be available for editing in the UI. So this solution can be deployed in one go without a manual maintenance.

Impact on users

For the purpose of this CEP, this doesn't impact the users directly.

Impact on hosting

There is no manual migrations to be run, so this will be a simpler process.

Release Timeline

This will be implemented as soon as the it is deemed good enough and other the feedback is integrated.

Open questions and issues

1. One of the goals of GAing features is to be able to delete all the toggle related code (?). While this solution still requires various Toggle related code in place for now, when we do want to remove the code it will still be possible by converting all the FrozenToggle documents to a simple static JSON documents to lookup the privilege for old domains with historical privilege for a toggle. (Think of FrozenToggle as a place to keep track of a new privilege and enabled-domains, it can exist without any of the toggle related code when/if we want to)
2. This doesn't/can't address the user only feature-flags. As there are only 7 such feature-flags it will need to be dealt in a different way at a different time.

[sravfeyn]

Please review this CEP, any feedback will be much appreciated. Thanks! @dimagi/team-commcare-hq

[sravfeyn]

Pinging those who might care about this more @snopoke @dannyroberts @biyeun @gherceg @kaapstorm

[millerdev]

It sounds less than ideal for long-term maintainability to use a new toggle type for grandfathered domains. It makes the process for determining if a feature is available more complicated since there would be two code paths to enabling the feature: 1) through the software plan system 2) through the toggle system.

Rather than creating a new toggle type, would it be possible to create a management command to automatically find the domains for which a given toggle is enabled and create custom software plans and roles to grandfather the domains under the new privilege?

The management command would take an argument (the toggle slug). It may need to generate code to be committed to commcare-hq, and it also may need to have a mode to assign the custom software plans in the database once the generated code has been deployed.

[sravfeyn]

It sounds less than ideal for long-term maintainability to use a new toggle type for grandfathered domains.

This is really not a toggle, in that it can be not be enabled/disabled. Abstractly its just a record of domains with a specific privilege.

It makes the process for determining if a feature is available more complicated since there would be two code paths to enabling the feature: 1) through the software plan system 2) through the toggle system.

From a code path point of view, I was planning to patch domain_has_privilege itself and make the toggle part invisible from an abstraction point of view.

would it be possible to create a management command to automatically find the domains for which a given toggle is enabled and create custom software plans and roles to grandfather the domains under the new privilege?

We do have that currently, and its downsides are listed in the Motivation section. The main issue is that there will be too many custom roles and privileges on top of each other as we GA toggles one after another. Apart from that it will involve multistage migration for each GA (migration of toggles to plan and then killing the toggle).

It sounds less than ideal for long-term maintainability to use a new toggle type for grandfathered domains.

I agree, it does. It adds an extra model to replace the existing Toggle but the model is essentially a static record (it doesn't require any UI or CRUD components), so there wouldn't be much code around it. The main advantage of this is that it makes GAing feature flags easier while still keeping the option of migrating even this static data model to custom software plans open.

If we do want to remove this probono privilege for these domains say after an year for not upgrading, this will involve just deleting this data model without again having to edit the software plans for these domains.

[orangejenny]

when we do want to remove the code it will still be possible by converting all the FrozenToggle documents to a simple static JSON documents to lookup the privilege for old domains with historical privilege for a toggle

What would this entail? I do think that a major benefit of removing toggles - probably the biggest code benefit - is removing the complexity of toggle checks. I recognize (and applaud!) plans to remove a lot of flags, but losing the code simplification that usually comes with GAing a feature flag is a big loss. Misunderstanding, see further conversation.

[mkangia]

Currently, we are in the process of GA-ing all the feature flags and deleting them after GA

@sravfeyn where can one read more about this?

[snopoke]

writing a Django migration that converts the Toggle document into FrozenToggle document

I don't see the need for any kind of data migration. Changing the class type from StaticToggle to FrozenToggle should be enough. The actual data storage layer shouldn't need to change at all.

[biyeun]

Agree with Simon's note that a migration from StaticToggle to FrozenToggle isn't needed, as this is data that lives in couch db and the structure itself is not changing.

I also agree with Jenny's comment that the benefit to GAing feature flags is a reduction in code complexity and I would prefer a process that creates custom SoftwarePlans for the grandfathered domains. @gherceg created a very good process with this kind of migration when working on the ERM code. It might be best to chat with him and, ideally, formalize this process in the documentation.

[sravfeyn]

but losing the code simplification that usually comes with GAing a feature flag is a big loss

The toggle related code can all be deleted in this approach. I think I didn't do a good job of explaining the exact strategy particularly the part where it doesn't need any of the Toggle related code which can be cleaned up.

So, to make it easy for others to understand the implications of both I have coded out both the approaches.

The existing approach using custom subscription plans

Here is the code. What this does is below

1. Adds a management command that updates/creates various subscription models (role, version and plan) to grant the privilege for all domains that have a particular toggle enabled.
2. The exact details of what gets updated and created can be seen here in ERM migration.

This will end up creating multiple custom plans/versions for each combination of toggles. For example,

• For domain d1 with flag/privilege a, b a custom plan such as pro_v0_ab may need to be created.
• For domain d2 with flag/privilege b, c a custom plan such as pro_v0_bc may need to be created
• For domain d3 with flag/privilege a, c a custom plan such as pro_v0_ac may need to be created.

So a plan needs to exist in every combination that the toggles are enabled for the domains. Further even if domains have same combination of toggles but are on a different plan that will end up creating other plans such as advanced_v0_ab etc. This will end up creating a lot of plans.

The proposed approach of using a new model

Here I have implemented the above described approach. Note that this does not need any Toggle related code at all. All of the toggle code can be cleaned up. This adds a single extra model and modifies domain_has_privilege with couple of lines.

I do agree that looking up the privilege is now ore complicated by having to look at this new model, but this seems much better compared to creating many combinations of custom plans.

Is there a better way?
################

It would be great if those with more accounting app knowledge know of better a way to implement this in a more neat way without the drawbacks of above two approaches.

Meanwhile, implementing the proposed approach would unblock most of the other work related to GAing feature flags. If we find a better method, it should be easy to transition from the proposed approach.

[orangejenny]

@sravfeyn Thank you for the code example - I did misunderstand, disregard my earlier comment.

[sravfeyn]

writing a Django migration that converts the Toggle document into FrozenToggle document

I don't see the need for any kind of data migration. Changing the class type from StaticToggle to FrozenToggle should be enough. The actual data storage layer shouldn't need to change at all.

We will need to store the new privilege on the doc for which a migration becomes necessary. Also, I think the doc_type will still be StaticToggle if not migrated and it will also not allow to cleanup the toggle related code. The other issue is we will need to create a couch view to lookup by privilege. Given all these drawbacks with the FrozenToggle, I have isolated all of this into a separate model that doesn't need any of the toggle code. Here is the approach.

[sravfeyn]

I hope the comments are addressed and it's good to go ahead with this approach