1092 - Migration to update column comments

[samathad2023]

🎫 Ticket

Link to the relevant ticket:
https://gitlab.login.gov/lg-teams/Team-Data/data-warehouse-ag/-/issues/1092

🛠 Summary of changes

Added migration to encrypted_email column in the email_address table. Specifically, it changed the sensitive flag in the comments from true to false.

📜 Testing Plan

Provide a checklist of steps to confirm the changes.

• Deploy and apply
• Verify updates in idp
• Check the dms task transformation in analytics

[vrajmohan]

Isn't this a problem as the encryption is two-way?

[samathad2023]

we need to decrypt the data for downstream processing for dataware house needs. Are there specific concerns ?

[vrajmohan]

Here are my assumptions (happy to learn if they are incorrect):

1. All sensitive information (PII and others) is encrypted in the Rails database. Despite the encryption, we do not propagate sensitive information to the data warehouse, as that is potentially open to a much larger audience, by marking sensitive information with comment: "sensitive=true".
2. The user's email address is one such data item. Furthermore, it is encrypted with a two-way algorithm so that we can retrieve the plaintext email address so that we can, say, send email to the user.
3. If we propagate the email address as well to the data warehouse, that would make it vulnerable to an attacker as they would possess a large dataset that they can analyze.
4. Why are we doing this for this field? Doesn't it negate the work that we did annotating the schema?

[samathad2023]

Thanks for raising these points—they are all valid and important.
Our goal is to only have decrypted data in the data warehouse for specific, permission-based roles with strict security constraints (like admin-level access). The comment: "sensitive=true" annotation is meant to flag this information specifically for the data warehouse process and isn't used anywhere else.
We believe that turning off the flag is the simplest way to proceed without a major refactor. Adding others to provide more context @MrNagoo @astrogeco @ambuj-neupane-tts

[MrNagoo]

@vrajmohan Security has approved not only the decryption but the storage of email as plaintext within the Data Warehouse. While we are now bringing and decrypting the email, we are still going to add column level security to the column only to be accessible by roles privileged to that information (fraud investigation/FCMS)

[ambuj-neupane-tts]

to be clear @MrNagoo @vrajmohan, Security has approved the plan to do it (ingest encrypted email address and decrypt to the warehouse). But we are not yet authorized to make the change. CC @astrogeco

[samathad2023]

Following our team discussion, we're adding the FCMS migration changes as a draft. They'll be merged when we're ready for the deployment.