Skip to content

feat(users): store and retrieve lineage_context from DB instead of Redis #7940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

tsdk02
Copy link
Contributor

@tsdk02 tsdk02 commented Apr 29, 2025

Type of Change

  • Bugfix
  • New feature
  • Enhancement
  • Refactoring
  • Dependency updates
  • Documentation
  • CI/CD

Description

Previously, the user's lineage_context (userid, org_id, merchant_id, profile_id, role_id, tenant_id) was cached in Redis.
This PR moves the storage and retrieval of lineage_context to the PostgreSQL users table instead of Redis.

This ensures permanent persistence of lineage_context in DB, as opposed to the 7 days TTL when stored in Redis.

Key Changes

  • Schema Change:

    • Added lineage_context as a JSONB column in the users table.
  • During Login:

    • Attempt to fetch lineage_context from DB (users.lineage_context).
    • If present:
      • Validate the context by checking if a matching user role exists (checks both v1 and v2 user roles).
      • If valid → use it for JWT generation.
      • If not → fallback to default DB-based role resolution.
    • If not present → fallback directly to default role resolution.
    • DB update for the new lineage context is now done via tokio::spawn to avoid blocking response.
  • During Org/Merchant/Profile Switch:

    • After switching, the new lineage_context is serialized and persisted in the users table.
    • This enables restoring the last used context during the next login.
    • The update is done in the background using tokio::spawn with success and failure logs.
  • Error Handling:

    • All DB failures or deserialization errors are logged.
    • Failures do not interrupt the login/switch flow — safe fallback ensures continuity.

Additional Changes

  • This PR modifies the API contract
  • This PR modifies the database schema
  • This PR modifies application configuration/environment variables

Motivation and Context

By moving the lineage_context to the users table in DB:

  • We ensure the context is consistently available regardless of deployment environment.
  • The last used account context becomes part of the user's persistent record.
  • It simplifies logic and avoids prefix mismatches across tenants.

How did you test it?

Screenshot 2025-04-29 at 11 45 43 PM

Test Case 1: Lineage Context Caching on Login

  1. Log in as a user with an assigned role.
  2. Check that lineage_context is stored in the users table (PostgreSQL).
  3. Log out and log back in → verify that the JWT reflects the same org/merchant/profile context without role fallback.

Test Case 2: Lineage Context Update on Org/Profile Switch

  1. Log in as a user and switch to another org/merchant/profile.
  2. Confirm that the updated context is persisted in the lineage_context field in the DB.
  3. Log out and log in again → JWT should reflect the newly switched context.

Test Case 3: Fallback to Role Resolution upon clearing lineage_context in db (Tested on local)

  1. Manually removed lineage_context from the DB for a user.
  2. Logged in again → verified that the system correctly falls back to the default role resolution logic.
  3. Confirm that the context is repopulated in the DB after login.

Test Case 4: Fallback to Role Resolution upon deleting user_role (Tested on local)

  1. Manually deleted user_role from the DB for a user (user_role corresponds to user invited to another account)
  2. Tried to log in again → verified that the system correctly falls back to the default role resolution logic.
  3. Confirm that the context is repopulated in the DB after login.

Checklist

  • I formatted the code cargo +nightly fmt --all
  • I addressed lints thrown by cargo clippy
  • I reviewed the submitted code
  • I added unit tests for my changes where possible

@tsdk02 tsdk02 added C-feature Category: Feature request or enhancement A-users Area: Users labels Apr 29, 2025
@tsdk02 tsdk02 self-assigned this Apr 29, 2025
@tsdk02 tsdk02 requested review from a team as code owners April 29, 2025 19:00
Copy link

semanticdiff-com bot commented Apr 29, 2025

Review changes with  SemanticDiff

Changed Files
File Status
  crates/common_utils/src/types.rs  34% smaller
  crates/router/src/core/user.rs  17% smaller
  crates/router/src/utils/user.rs  10% smaller
  crates/router/src/types/domain/user/decision_manager.rs  9% smaller
  crates/router/src/types/domain/user.rs  1% smaller
  crates/api_models/src/user/theme.rs  0% smaller
  crates/common_utils/src/types/user.rs  0% smaller
  crates/common_utils/src/types/user/core.rs  0% smaller
  crates/common_utils/src/types/user/theme.rs  0% smaller
  crates/diesel_models/src/query/user/theme.rs  0% smaller
  crates/diesel_models/src/schema.rs  0% smaller
  crates/diesel_models/src/schema_v2.rs  0% smaller
  crates/diesel_models/src/user.rs  0% smaller
  crates/diesel_models/src/user/theme.rs  0% smaller
  crates/router/src/configs/settings.rs  0% smaller
  crates/router/src/consts/user.rs  0% smaller
  crates/router/src/core/recon.rs  0% smaller
  crates/router/src/core/user/theme.rs  0% smaller
  crates/router/src/db/kafka_store.rs  0% smaller
  crates/router/src/db/user.rs  0% smaller
  crates/router/src/db/user/theme.rs  0% smaller
  crates/router/src/routes/user/theme.rs  0% smaller
  crates/router/src/services/email/types.rs  0% smaller
  crates/router/src/utils/user/theme.rs  0% smaller
  crates/router/src/workflows/api_key_expiry.rs  0% smaller
  migrations/2025-04-29-144409_add_lineage_context_to_users/down.sql Unsupported file format
  migrations/2025-04-29-144409_add_lineage_context_to_users/up.sql Unsupported file format

@tsdk02 tsdk02 linked an issue Apr 29, 2025 that may be closed by this pull request
2 tasks
@hyperswitch-bot hyperswitch-bot bot added the M-database-changes Metadata: This PR involves database schema changes label Apr 29, 2025
Copy link
Contributor

@ThisIsMani ThisIsMani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR should also read from redis for backwards compatibility.

Comment on lines 3185 to 3204
let _ = state
.global_store
.update_user_by_user_id(
&user_from_token.user_id,
storage_user::UserUpdate::LineageContextUpdate {
lineage_context: Some(
serde_json::to_value(&lineage_context)
.change_context(UserErrors::InternalServerError)
.attach_printable("Failed to serialize LineageContext to JSON")?,
),
},
)
.await
.map_err(|e| {
logger::error!(
"Failed to update lineage context for user {}: {:?}",
user_from_token.user_id,
e
)
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can implement this on LineageContext or the UserFromStorage type if it is still big after the other changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But generally, db calls are not supposed to be isolated right, so added the code in the main function itself. Do you recommend having this in the impl on lineageContext?

Comment on lines 244 to 247
storage::User {
lineage_context: lineage_context.clone(),
..user.to_owned()
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not updating last_modified_at here? Are we not supposed to?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not updating last_modified_at in any kind of update for MockDb

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated last_modified_at for all kinds of updates in MockDb now.

@tsdk02 tsdk02 requested a review from a team as a code owner April 30, 2025 18:43
@tsdk02 tsdk02 requested a review from Copilot May 2, 2025 14:25
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the handling of the user's lineage_context by persisting it in PostgreSQL instead of Redis and updates several type references from the theme namespace to the user namespace. Key changes include adding the lineage_context column in the users table, updating DB interactions to handle the new column (including async updates via tokio::spawn), and refactoring type imports across multiple files.

Reviewed Changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
crates/router/src/services/email/types.rs Updated import for EmailThemeConfig to use the user namespace.
crates/router/src/routes/user/theme.rs Refactored ThemeLineage import to use the user namespace.
crates/router/src/db/user/theme.rs Updated ThemeLineage import to use the user namespace.
crates/router/src/db/user.rs Added lineage_context assignments and last_modified_at updates in user update functions.
crates/router/src/db/kafka_store.rs Updated ThemeLineage import to use the user namespace.
crates/router/src/core/user/theme.rs Updated ThemeLineage import to use the user namespace.
crates/router/src/core/user.rs Replaced lineage context cache update with an async DB update using tokio::spawn.
crates/router/src/core/recon.rs Updated ThemeLineage import to use the user namespace.
crates/router/src/consts/user.rs Removed unused lineage_context constants related to Redis.
crates/router/src/configs/settings.rs Updated EmailThemeConfig import to use the user namespace.
crates/diesel_models/src/user/theme.rs Updated EmailThemeConfig and ThemeLineage imports to use the user namespace.
crates/diesel_models/src/user.rs Added lineage_context field and updated UserUpdate variants accordingly.
crates/diesel_models/src/schema_v2.rs & schema.rs Added lineage_context column to the users table schema.
crates/diesel_models/src/query/user/theme.rs Updated ThemeLineage import to use the user namespace.
crates/common_utils/src/types/user/core.rs Introduced the new LineageContext type definition.
crates/common_utils/src/types/user.rs & types.rs Organized and re-exported user types instead of theme types.
crates/api_models/src/user/theme.rs Updated ThemeLineage and EmailThemeConfig imports to use the user namespace.

lineage_context
.try_set_lineage_context_in_cache(&state, user_from_token.user_id.as_str())
.await;
tokio::spawn({
Copy link
Preview

Copilot AI May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding a retry mechanism for the asynchronous update of lineage_context in the background task to improve resilience against transient DB failures.

Copilot uses AI. Check for mistakes.

lineage_context
.try_set_lineage_context_in_cache(&state, user_from_token.user_id.as_str())
.await;
tokio::spawn({
Copy link
Preview

Copilot AI May 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evaluate whether a consistent retry or fallback strategy is needed across all lineage_context update tasks (e.g. in switch_merchant and switch_profile) to ensure reliable DB updates.

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-users Area: Users C-feature Category: Feature request or enhancement M-database-changes Metadata: This PR involves database schema changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Persist Last Used Lineage Context in Database Instead of Redis
2 participants