diff --git a/docs/adr/0050-classifier-deployment-strategy.md b/docs/adr/0050-classifier-deployment-strategy.md new file mode 100644 index 00000000..03a4c074 --- /dev/null +++ b/docs/adr/0050-classifier-deployment-strategy.md @@ -0,0 +1,140 @@ +--- +name: Classifier Deployment Strategy +description: Exploration of multi-tenant ML model serving strategies — managed training with flexible deployment (self-hosted or managed). +tags: [classifier, ml, multi-tenant, architecture, deployment] +author: emil +date: 2026-04-24 +status: Proposed +--- + +# 0050. Classifier Deployment Strategy + +Date: 2026-04-24 + +## Status + +Proposed — open for discussion, not yet decided. + +## Context + +KSeF Hub uses three sidecar services: + +| Service | Coupling | Tenant-specific? | +|---------|----------|-----------------| +| **pdf-renderer** | Tightly coupled to the app (FA(3) XML → PDF) | No — same logic for all tenants | +| **invoice-extractor** | Tightly coupled to the app (PDF → structured JSON) | No — generic extraction | +| **invoice-classifier** | Loosely coupled, depends on tenant data | **Yes** — categories, tags, and classification patterns are per-company | + +The classifier is fundamentally different: its accuracy depends on the tenant's own data. A model trained on one company's invoices (e.g., a media agency) will mis-classify for another (e.g., a construction firm). This means multi-tenant SaaS needs per-tenant models. + +### Current state + +- Single classifier service behind a per-company config (ADR 0049) +- Each company can override the classifier URL, token, and confidence thresholds +- One shared model serves all companies — predictions degrade as company profiles diverge +- Training data export exists (CSV with extended columns from the Services tab) + +### Problem + +As we scale to more tenants, a shared model won't work. We need a strategy for: +1. **Training** — how per-tenant models are created and updated +2. **Serving** — how predictions are served at inference time +3. **Cost** — who pays for compute (us or the tenant) +4. **Ops** — how much operational burden falls on us vs. the tenant + +## Options Considered + +### Option A — Managed multi-tenant model serving + +We own both training and serving. A model-management sidecar loads per-tenant models on demand. + +**How it works:** +- Tenant clicks "retrain" in the UI (or we auto-retrain on a schedule) +- We train the model from their approved invoices using our training pipeline +- A model-serving sidecar keeps recently-used models in memory, evicts cold ones +- Prediction requests are routed to the correct tenant model + +**Pros:** +- Best UX — tenant never touches infrastructure +- We control model quality, versioning, rollback +- Can optimize hardware (shared GPU/CPU, model caching) + +**Cons:** +- Operational complexity — cache eviction, cold starts, memory pressure, OOM risk +- We bear infra cost per model (though classification models are small, ~MBs) +- Single point of failure — our serving layer goes down, all tenants lose classification + +### Option B — Dedicated self-hosted service per tenant + +We open-source the classifier. Each tenant trains and deploys their own instance. + +**How it works:** +- We provide training tools and a Docker image +- Tenant deploys to their own cloud (GCP Cloud Run, AWS ECS, etc.) +- Tenant configures their classifier URL in KSeF Hub (already supported via ADR 0049) + +**Pros:** +- Zero infra cost for us +- Tenant data never leaves their environment (compliance/GDPR win) +- Tenant has full control over model, hardware, scaling + +**Cons:** +- Terrible UX — requires cloud knowledge, deployment pipeline, monitoring +- Support burden when tenant's service breaks +- Fragmented versions — tenants may run stale images + +### Option C — Managed training + flexible deployment (preferred direction) + +We own the training pipeline. The tenant chooses where to serve: our managed infrastructure or their own cloud. + +**How it works:** +- **Training**: We provide a managed training pipeline. Tenant's approved invoices feed into it. We produce a model artifact (e.g., ONNX, TensorFlow SavedModel, scikit-learn pickle). +- **Deployment option 1 — Managed**: We host the model in our serving layer (same as Option A). Included in the subscription or as a paid tier. +- **Deployment option 2 — Self-hosted on cloud ML platforms**: Tenant deploys the model artifact to a managed ML service: + - **Google Cloud**: Vertex AI Endpoints (upload model → get prediction API), Cloud Run (containerized) + - **AWS**: SageMaker Endpoints (upload model → get prediction API), Lambda (lightweight) + - **Azure**: Azure ML Endpoints +- We provide clear documentation/tooling for each cloud platform deployment. +- Tenant points their classifier URL to their deployed endpoint (ADR 0049 already supports this). + +**Pros:** +- We own the hardest part (training pipeline, data prep, feature engineering) +- Tenant gets a simple choice: "we host it" or "you host it on [GCP/AWS/Azure]" +- Cloud ML platforms handle scaling, monitoring, versioning natively +- Self-hosted option satisfies compliance-sensitive tenants +- Model artifact is portable — not locked to our infrastructure + +**Cons:** +- Need to support multiple model export formats (or standardize on one like ONNX) +- Documentation/guides needed for each cloud platform +- Two code paths for model delivery (managed vs. self-hosted) + +## Open Questions + +1. **Model format**: What format do we standardize on? ONNX is portable but may lose framework-specific optimizations. scikit-learn pickle is simple but Python-only. +2. **Retraining frequency**: On-demand (tenant clicks button), scheduled (weekly), or continuous (on every N new approved invoices)? +3. **Minimum training data**: How many approved invoices does a company need before a per-tenant model outperforms the shared default model? +4. **Cold start**: For new tenants with no data, do we use a shared "bootstrap" model trained on anonymized cross-tenant data? +5. **Pricing**: Is managed serving included in the base plan, or a paid add-on? +6. **Model versioning**: How do we handle A/B testing between model versions? Rollback on accuracy regression? + +## Decision + +Not yet decided. This ADR captures the current thinking to be revisited when we approach multi-tenant scaling. + +**Leaning toward Option C** — managed training with flexible deployment. It gives us the best of both worlds: great UX for tenants who want "it just works", and full control for tenants with compliance requirements or existing cloud infrastructure. + +## Next Steps + +- [ ] Research Vertex AI / SageMaker model deployment APIs — what's the minimal integration? +- [ ] Prototype: export a trained model artifact, deploy to Vertex AI, verify prediction API compatibility +- [ ] Define minimum viable training pipeline (how many invoices, what features, what accuracy threshold) +- [ ] Estimate serving costs per tenant for the managed option +- [ ] Decide on model format standard + +## Consequences + +Deferring this decision is acceptable for now — the current shared model + per-company config (ADR 0049) works for early tenants. But this becomes blocking when: +- A second tenant with a very different business profile onboards +- Classification accuracy drops below the confidence thresholds for any company +- A tenant asks "can I train my own model?" diff --git a/docs/architecture.md b/docs/architecture.md index c1fcd3ab..992c068a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -13,6 +13,7 @@ Quick reference for developers. Before touching a feature area: | Feature | Key files | |---------|-----------| | Invoice export (ZIP + CSV) | `lib/ksef_hub/exports.ex`, `exports/csv_builder.ex`, `exports/export_batch.ex`, `live/export_live/index.ex` | +| Training CSV export | `lib/ksef_hub/exports.ex` (`list_training_invoices/3`), `exports/csv_builder.ex` (`extended: true`), `controllers/training_csv_controller.ex` | | KSeF sync | `lib/ksef_hub/sync/sync_worker.ex`, `sync/invoice_fetcher.ex`, `ksef_client/` | | FA(3) XML parsing | `invoices/parser.ex`, `docs/fa3-xml.md` (field mapping reference) | | Invoice CRUD & business logic | `lib/ksef_hub/invoices.ex` (facade) + `invoices/` sub-modules | @@ -109,3 +110,4 @@ Read only the ADR(s) relevant to your task — the summaries below tell you whic | 0047-approver-analyst-roles.md | Approver and Analyst Roles | Accepted | Rename `:reviewer` → `:approver` and `:viewer` → `:analyst`; analyst has same data scope as approver (access grants required for restricted invoices) | | 0048-public-landing-page.md | Public Landing Page as Standalone Astro Project | Accepted | Ship marketing site as separate Astro 5 + Tailwind v4 project in `landing/`, deployed to GitHub Pages; zero runtime coupling with Phoenix | | 0049-per-company-classifier-config.md | Per-Company Classifier Configuration | Accepted | Company-scoped classifier settings in `classifier_configs` table with env-var fallback; only classifier supports per-company overrides | +| 0050-classifier-deployment-strategy.md | Classifier Deployment Strategy | Proposed | Multi-tenant ML model serving — managed training with flexible deployment (self-hosted on Vertex AI/SageMaker or our managed infra) | diff --git a/lib/ksef_hub/exports.ex b/lib/ksef_hub/exports.ex index ce741d34..ab7f99d1 100644 --- a/lib/ksef_hub/exports.ex +++ b/lib/ksef_hub/exports.ex @@ -94,6 +94,34 @@ defmodule KsefHub.Exports do |> Repo.all() end + @doc "Builds an extended training CSV binary from a list of invoices." + @spec build_training_csv([Invoice.t()], keyword()) :: binary() + defdelegate build_training_csv(invoices, opts \\ []), to: CsvBuilder, as: :build + + @doc """ + Lists invoices for training CSV export with extended preloads. + + Unlike `list_exportable_invoices/1`, this takes a company_id and date range + directly — no ExportBatch required, no download tracking. Includes all invoice + types with no date range cap (CSV generation is fast, unlike PDF rendering). + """ + @spec list_training_invoices(Ecto.UUID.t(), Date.t(), Date.t()) :: [Invoice.t()] + def list_training_invoices(company_id, date_from, date_to) do + %{ + company_id: company_id, + date_from: date_from, + date_to: date_to, + invoice_type: nil, + only_new: false, + user_id: nil, + category_id: nil + } + |> exportable_invoices_query() + |> order_by([i], asc: i.issue_date, asc: i.invoice_number) + |> preload([:category, :payment_requests, :created_by, :inbound_email]) + |> Repo.all() + end + @doc "Counts invoices matching the given export filters without loading them." @spec count_exportable_invoices(Ecto.UUID.t(), map()) :: non_neg_integer() def count_exportable_invoices(company_id, filters) do diff --git a/lib/ksef_hub/exports/csv_builder.ex b/lib/ksef_hub/exports/csv_builder.ex index d93cef8e..797a1ab1 100644 --- a/lib/ksef_hub/exports/csv_builder.ex +++ b/lib/ksef_hub/exports/csv_builder.ex @@ -3,7 +3,7 @@ defmodule KsefHub.Exports.CsvBuilder do alias KsefHub.Invoices.Invoice - @headers [ + @standard_headers [ "Invoice Number", "Issue Date", "Sales Date", @@ -38,21 +38,41 @@ defmodule KsefHub.Exports.CsvBuilder do "Correction Reason" ] + @extended_headers [ + "Invoice ID", + "Company ID", + "Cost Line", + "Project Tag", + "Is Excluded", + "Access Restricted", + "Payment Status", + "Payment Date", + "Category Identifier", + "Prediction Status", + "Predicted Category", + "Predicted Tag", + "Category Confidence %", + "Tag Confidence %", + "Extraction Status" + ] + @doc "Builds a CSV binary (UTF-8 with BOM) from a list of invoices." - @spec build([Invoice.t()]) :: binary() - def build(invoices) do - rows = Enum.map(invoices, &invoice_to_row/1) + @spec build([Invoice.t()], keyword()) :: binary() + def build(invoices, opts \\ []) do + extended = Keyword.get(opts, :extended, false) + headers = if extended, do: @standard_headers ++ @extended_headers, else: @standard_headers + rows = Enum.map(invoices, &invoice_to_row(&1, extended)) csv = - [@headers | rows] + [headers | rows] |> Enum.map_join("\r\n", &encode_row/1) <<0xEF, 0xBB, 0xBF>> <> csv <> "\r\n" end - @spec invoice_to_row(Invoice.t()) :: [String.t()] - defp invoice_to_row(invoice) do - [ + @spec invoice_to_row(Invoice.t(), boolean()) :: [String.t()] + defp invoice_to_row(invoice, extended) do + standard = [ s(invoice.invoice_number), format_date(invoice.issue_date), format_date(invoice.sales_date), @@ -86,8 +106,74 @@ defmodule KsefHub.Exports.CsvBuilder do s(invoice.corrected_invoice_ksef_number), s(invoice.correction_reason) ] + + if extended do + standard ++ extended_fields(invoice) + else + standard + end + end + + @spec extended_fields(Invoice.t()) :: [String.t()] + defp extended_fields(invoice) do + {payment_status, payment_date} = extract_payment_info(invoice) + + [ + s(invoice.id), + s(invoice.company_id), + s(invoice.expense_cost_line), + s(invoice.project_tag), + format_boolean(invoice.is_excluded), + format_boolean(invoice.access_restricted), + payment_status, + payment_date, + format_category_identifier(invoice), + s(invoice.prediction_status), + s(invoice.prediction_expense_category_name), + s(invoice.prediction_expense_tag_name), + format_confidence(invoice.prediction_expense_category_confidence), + format_confidence(invoice.prediction_expense_tag_confidence), + s(invoice.extraction_status) + ] end + @spec extract_payment_info(Invoice.t()) :: {String.t(), String.t()} + defp extract_payment_info(%{payment_requests: prs}) when is_list(prs) do + case Enum.find(prs, &(&1.status == :paid)) do + %{status: status, paid_at: paid_at} -> + {s(status), format_datetime(paid_at)} + + nil -> + case List.first(prs) do + nil -> {"", ""} + %{status: status} -> {s(status), ""} + end + end + end + + defp extract_payment_info(_), do: {"", ""} + + @spec format_category_identifier(Invoice.t()) :: String.t() + defp format_category_identifier(%{category: %{identifier: id}}) when is_binary(id), do: id + defp format_category_identifier(_), do: "" + + @spec format_boolean(boolean() | nil) :: String.t() + defp format_boolean(true), do: "true" + defp format_boolean(false), do: "false" + defp format_boolean(nil), do: "" + + @spec format_confidence(float() | nil) :: String.t() + defp format_confidence(nil), do: "" + + defp format_confidence(value) when is_float(value) do + value + |> Kernel.*(100) + |> Float.round(1) + |> Float.to_string() + end + + # --- Shared formatting helpers --- + @spec s(term()) :: String.t() defp s(nil), do: "" defp s(value), do: to_string(value) diff --git a/lib/ksef_hub/invoices/invoice.ex b/lib/ksef_hub/invoices/invoice.ex index 10a2a976..54867a91 100644 --- a/lib/ksef_hub/invoices/invoice.ex +++ b/lib/ksef_hub/invoices/invoice.ex @@ -88,6 +88,7 @@ defmodule KsefHub.Invoices.Invoice do belongs_to :pdf_file, KsefHub.Files.File field :tags, {:array, :string}, default: [] has_many :comments, KsefHub.Invoices.InvoiceComment + has_many :payment_requests, KsefHub.PaymentRequests.PaymentRequest belongs_to :created_by, KsefHub.Accounts.User has_one :inbound_email, KsefHub.InboundEmail.InboundEmail has_many :access_grants, KsefHub.Invoices.InvoiceAccessGrant diff --git a/lib/ksef_hub_web/components/settings_components.ex b/lib/ksef_hub_web/components/settings_components.ex index ff867588..d75c7354 100644 --- a/lib/ksef_hub_web/components/settings_components.ex +++ b/lib/ksef_hub_web/components/settings_components.ex @@ -76,7 +76,7 @@ defmodule KsefHubWeb.SettingsComponents do {:manage_certificates, "Certificates", ~p"/c/#{id}/settings/certificates", "hero-shield-check"}, {:manage_team, "Activity Log", ~p"/c/#{id}/settings/activity-log", "hero-clock"}, - {:manage_services, "Services", ~p"/c/#{id}/settings/services", "hero-server-stack"} + {:manage_services, "Classifier", ~p"/c/#{id}/settings/services", "hero-cpu-chip"} ] |> Enum.filter(fn {perm, _label, _path, _icon} -> is_nil(perm) or Authorization.can?(role, perm) diff --git a/lib/ksef_hub_web/controllers/training_csv_controller.ex b/lib/ksef_hub_web/controllers/training_csv_controller.ex new file mode 100644 index 00000000..5fcc8896 --- /dev/null +++ b/lib/ksef_hub_web/controllers/training_csv_controller.ex @@ -0,0 +1,54 @@ +defmodule KsefHubWeb.TrainingCsvController do + @moduledoc "Controller for downloading training CSV files with extended invoice data." + + use KsefHubWeb, :controller + + import KsefHubWeb.AuthHelpers, only: [resolve_role: 2] + import KsefHubWeb.FilenameHelpers, only: [send_attachment: 4] + + alias KsefHub.Authorization + alias KsefHub.Exports + + @doc "Downloads an extended CSV of invoices for ML training." + @spec download(Plug.Conn.t(), map()) :: Plug.Conn.t() + def download(conn, %{"company_id" => company_id, "date_from" => from, "date_to" => to}) do + user_id = conn.assigns.current_user.id + role = resolve_role(user_id, company_id) + + if Authorization.can?(role, :manage_services) do + do_download(conn, company_id, from, to) + else + conn + |> put_flash(:error, "You don't have permission to export training data.") + |> redirect(to: ~p"/c/#{company_id}/settings/services") + end + end + + def download(conn, %{"company_id" => company_id}) do + conn + |> put_flash(:error, "Date range is required.") + |> redirect(to: ~p"/c/#{company_id}/settings/services") + end + + @spec do_download(Plug.Conn.t(), String.t(), String.t(), String.t()) :: Plug.Conn.t() + defp do_download(conn, company_id, date_from_str, date_to_str) do + with {:ok, date_from} <- Date.from_iso8601(date_from_str), + {:ok, date_to} <- Date.from_iso8601(date_to_str), + :ok <- validate_date_order(date_from, date_to) do + invoices = Exports.list_training_invoices(company_id, date_from, date_to) + csv_binary = Exports.build_training_csv(invoices, extended: true) + filename = "training_#{date_from}_#{date_to}.csv" + send_attachment(conn, "text/csv", filename, csv_binary) + else + _ -> + conn + |> put_flash(:error, "Invalid date range.") + |> redirect(to: ~p"/c/#{company_id}/settings/services") + end + end + + @spec validate_date_order(Date.t(), Date.t()) :: :ok | :error + defp validate_date_order(from, to) do + if Date.compare(to, from) != :lt, do: :ok, else: :error + end +end diff --git a/lib/ksef_hub_web/live/export_live/index.ex b/lib/ksef_hub_web/live/export_live/index.ex index 909e62a4..b94d1c1f 100644 --- a/lib/ksef_hub_web/live/export_live/index.ex +++ b/lib/ksef_hub_web/live/export_live/index.ex @@ -270,20 +270,21 @@ defmodule KsefHubWeb.ExportLive.Index do -