Cursor rules refinery gateway (#377)

* First draft cursor rules for refinery gateway

* Submodules update

* Database migrations rules

* Database migrations rules

* PR comments

* PR comments

* PR comments

Author: lumburovskalina

.cursor/rules/api-models.mdc

@@ -0,0 +1,75 @@
+---
+description: Rules for Pydantic models and request/response validation
+globs: ["fast_api/models.py"]
+alwaysApply: true
+---
+
+# API Models Guidelines
+
+Pydantic models validate request bodies and ensure type safety.
+
+## Model Definition
+
+**Basic structure:**
+```python
+from pydantic import BaseModel, StrictBool, StrictFloat, StrictInt, StrictStr
+from typing import Optional, List, Dict, Any
+
+class CreateProjectBody(BaseModel):
+    name: StrictStr
+    description: Optional[StrictStr] = None
+    organization_id: StrictStr
+
+class CreateLabelsBody(BaseModel):
+    labelingTaskId: StrictStr
+    labels: List[StrictStr]
+
+class SearchRecordsExtendedBody(BaseModel):
+    filterData: Any = None
+    offset: StrictInt
+    limit: StrictInt
+```
+
+## Naming Conventions
+
+- Request bodies: `CreateProjectBody`, `UpdateProjectBody`, `DeleteRecordBody`
+- Use descriptive names ending in `Body`
+
+## Usage in Routes
+
+```python
+from fast_api.models import CreateProjectBody
+
+@router.post("/create-project")
+def create_project(body: CreateProjectBody):
+    project = manager.create_project(
+        name=body.name,
+        description=body.description,
+        organization_id=body.organization_id,
+    )
+    return pack_json_result(sql_alchemy_to_dict(project))
+```
+
+## Field Validation
+
+```python
+from pydantic import field_validator, Field
+
+class CreateProjectBody(BaseModel):
+    name: StrictStr = Field(min_length=1, max_length=100)
+    
+    @field_validator('name')
+    @classmethod
+    def validate_name(cls, v):
+        if not v or not v.strip():
+            raise ValueError('Project name cannot be empty')
+        return v.strip()
+```
+
+## Best Practices
+
+1. Use strict types (`StrictStr`, `StrictInt`, etc.) for validation
+2. Provide defaults for optional fields
+3. Use descriptive model names ending in `Body`
+4. Validate inputs using validators or Field constraints
+5. Use proper type hints for all fields

.cursor/rules/controllers.mdc

@@ -0,0 +1,120 @@
+---
+description: Rules for controller modules and business logic
+globs: ["controller/**/*.py"]
+alwaysApply: true
+---
+
+# Controllers Guidelines
+
+Controllers contain business logic and orchestrate operations between routes, submodules, and external services.
+
+## Import Patterns
+
+```python
+# Submodules
+from submodules.model.business_objects import project, record
+from submodules.model import Project, enums
+from submodules.model.exceptions import EntityNotFoundException
+from submodules.s3 import controller as s3
+
+# Other controllers
+from controller.auth import manager as auth_manager
+
+# Utilities
+from util import notification
+from util.miscellaneous_functions import chunk_list
+```
+
+## Function Patterns
+
+**Get operations:**
+```python
+def get_project(project_id: str) -> Project:
+    proj = project.get(project_id)
+    if not proj:
+        raise EntityNotFoundException(f"Project {project_id} not found")
+    return proj
+```
+
+**Create operations:**
+```python
+def create_project(name: str, organization_id: str, user_id: str, with_commit: bool = True) -> Project:
+    if not name or not name.strip():
+        raise ValueError("Project name cannot be empty")
+    
+    org = organization.get(organization_id)
+    if not org:
+        raise EntityNotFoundException(f"Organization {organization_id} not found")
+    
+    return project.create(name=name, organization_id=organization_id, created_by=user_id, with_commit=with_commit)
+```
+
+**Update operations:**
+```python
+def update_project(project_id: str, changes: Dict[str, Any], with_commit: bool = True) -> Project:
+    proj = project.get(project_id)
+    if not proj:
+        raise EntityNotFoundException(f"Project {project_id} not found")
+    
+    for key, value in changes.items():
+        setattr(proj, key, value)
+    
+    if with_commit:
+        general.commit()
+    return proj
+```
+
+## Business Logic Patterns
+
+**Validation:**
+```python
+def create_attribute(project_id: str, name: str, data_type: str) -> Attribute:
+    proj = project.get(project_id)
+    if not proj:
+        raise EntityNotFoundException(f"Project {project_id} not found")
+    
+    if not name or not name.strip():
+        raise ValueError("Attribute name cannot be empty")
+    
+    existing = attribute.get_by_name(project_id, name)
+    if existing:
+        raise ValueError(f"Attribute '{name}' already exists")
+    
+    return attribute.create(project_id=project_id, name=name, data_type=data_type, with_commit=True)
+```
+
+**Transaction management:**
+```python
+def bulk_create_records(project_id: str, records_data: List[Dict[str, Any]], with_commit: bool = True):
+    created_records = []
+    for data in records_data:
+        rec = record.create(project_id=project_id, data=data, with_commit=False)
+        created_records.append(rec)
+    if with_commit:
+        general.commit()
+    return created_records
+```
+
+**Async operations:**
+```python
+from submodules.model import daemon
+
+def run_weak_supervision(project_id: str, user_id: str) -> None:
+    daemon.run_with_db_token(weak_supervision_service.initiate_weak_supervision, project_id, user_id)
+    notification.create_notification(user_id=user_id, project_id=project_id, type=NotificationType.WEAK_SUPERVISION_STARTED)
+```
+
+## Constants
+
+```python
+ALL_PROJECTS_WHITELIST = {"id", "name", "description", "tokenizer", "status", "created_at", "created_by"}
+```
+
+## Best Practices
+
+1. Single responsibility per function
+2. Always validate inputs
+3. Use type hints for all parameters
+4. Use `with_commit` parameter appropriately
+5. Use submodule business objects, never SQLAlchemy directly
+6. Keep business logic in controllers, not routes

.cursor/rules/exceptions.mdc

@@ -0,0 +1,66 @@
+---
+description: Rules for exception handling and custom exceptions
+globs: ["exceptions/**/*.py", "**/*.py"]
+alwaysApply: true
+---
+
+# Exceptions Guidelines
+
+## Exception Locations
+
+**Gateway exceptions** (`exceptions/exceptions.py`):
+- `MissingArgumentsException`, `ProjectAccessError`, `ServiceRequestsError`
+- `DatabaseSessionError`, `AuthManagerError`, `EmbeddingConnectorError`
+
+**Submodule exceptions**:
+```python
+from submodules.model.exceptions import EntityNotFoundException, EntityAlreadyExistsException
+```
+
+## Usage Patterns
+
+**Raising exceptions:**
+```python
+# Validation
+if not name:
+    raise MissingArgumentsException("Project name is required")
+
+# Not found
+proj = project.get(project_id)
+if not proj:
+    raise EntityNotFoundException(f"Project {project_id} not found")
+
+# Business logic
+if not has_access(user_id, project_id):
+    raise ProjectAccessError(f"User {user_id} does not have access")
+```
+
+**Handling in routes:**
+```python
+try:
+    result = manager.operation(project_id)
+    return pack_json_result(result)
+except EntityNotFoundException as e:
+    return pack_json_result({"error": str(e)}, status_code=404)
+except ProjectAccessError as e:
+    return pack_json_result({"error": str(e)}, status_code=403)
+except Exception as e:
+    logger.error(f"Error: {e}", exc_info=True)
+    return GENERIC_FAILURE_RESPONSE
+```
+
+## HTTP Status Code Mapping
+
+- `400`: `ValueError`, `MissingArgumentsException`
+- `403`: `ProjectAccessError`
+- `404`: `EntityNotFoundException`
+- `409`: `EntityAlreadyExistsException`
+- `500`: `ServiceRequestsError`, `DatabaseSessionError`
+
+## Best Practices
+
+1. Use specific exception types, not generic `Exception`
+2. Provide clear error messages with context
+3. Log exceptions before raising or handling
+4. Map exceptions to appropriate HTTP status codes
+5. Don't swallow exceptions silently

.cursor/rules/fastapi-routes.mdc

@@ -0,0 +1,120 @@
+---
+description: Rules for FastAPI route definitions and HTTP handling
+globs: ["fast_api/routes/**/*.py"]
+alwaysApply: true
+---
+
+# FastAPI Routes Guidelines
+
+Routes handle HTTP request/response logic. Keep routes thin - validate input, call controllers, format responses.
+
+## Route Structure
+
+```python
+from fastapi import APIRouter, Depends, Request
+from controller.auth import manager as auth_manager
+from fast_api.routes.client_response import pack_json_result, get_silent_success
+
+router = APIRouter()
+
+@router.get(
+    "/{project_id}/data",
+    dependencies=[Depends(auth_manager.check_project_access_dep)],
+)
+def get_data(project_id: str):
+    data = manager.get_data(project_id)
+    return pack_json_result(data)
+```
+
+## Authentication
+
+```python
+# Project access
+dependencies=[Depends(auth_manager.check_project_access_dep)]
+
+# Admin operations
+dependencies=[
+    Depends(auth_manager.check_project_access_dep),
+    Depends(auth_manager.check_group_auth),
+]
+
+# User context
+org_id = auth_manager.get_organization_id_by_info(request.state.info)
+```
+
+## Response Patterns
+
+```python
+# With data
+return pack_json_result(data)
+
+# No data
+return get_silent_success()
+
+# Custom status
+return pack_json_result(None, status_code=204)
+
+# Error handling
+from fast_api.routes.client_response import GENERIC_FAILURE_RESPONSE
+try:
+    result = manager.operation()
+    return pack_json_result(result)
+except EntityNotFoundException:
+    return pack_json_result({"error": str(e)}, status_code=404)
+except Exception as e:
+    logger.error(f"Error: {e}")
+    return GENERIC_FAILURE_RESPONSE
+```
+
+## Data Transformation
+
+Always use whitelists when converting SQLAlchemy objects:
+
+```python
+from submodules.model.util import sql_alchemy_to_dict
+
+ATTRIBUTES_WHITELIST = {"id", "name", "data_type", "is_primary_key"}
+
+@router.get("/{project_id}/all-attributes")
+def get_attributes(project_id: str):
+    data = manager.get_all_attributes(project_id)
+    data_dict = sql_alchemy_to_dict(data, column_whitelist=ATTRIBUTES_WHITELIST)
+    return pack_json_result(data_dict)
+```
+
+## Request Parameters
+
+```python
+# Path parameters
+@router.get("/{project_id}/record/{record_id}")
+def get_record(project_id: str, record_id: str):
+    pass
+
+# Query parameters
+from fastapi import Query
+from typing import List, Union
+
+@router.get("/{project_id}/all-attributes")
+def get_attributes(
+    project_id: str,
+    state_filter: Union[List[str], None] = Query(default=None),
+):
+    pass
+
+# Request body
+from fast_api.models import CreateProjectBody
+
+@router.post("/create-project")
+def create_project(body: CreateProjectBody):
+    pass
+```
+
+## Best Practices
+
+1. Keep routes thin - delegate to controllers
+2. Use dependencies for authentication
+3. Always use `pack_json_result()` or `get_silent_success()`
+4. Use whitelists with `sql_alchemy_to_dict()`
+5. Use type hints for all parameters
+6. Handle exceptions appropriately
+7. Use kebab-case for route paths: `/all-projects`, `/project-by-project-id`