Skip to content

Commit 2aaefc7

Browse files
authored
feat: automated project archival with data retention (#678)
1 parent a0daa05 commit 2aaefc7

6 files changed

Lines changed: 898 additions & 0 deletions

File tree

backend/docs/PROJECT_ARCHIVAL.md

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Automated Project Archival & Data Retention
2+
3+
Projects that go quiet are archived automatically, retained for a configurable
4+
window, and then purged — with restoration available at any point before the
5+
purge runs.
6+
7+
Service: [`backend/src/services/project-archival/index.ts`](../src/services/project-archival/index.ts)
8+
Routes: [`backend/src/routes/project-archival.ts`](../src/routes/project-archival.ts) — mounted at `/api/v1/project-archival`
9+
Schedule: `project-archival-sweep` in [`backend/src/config/scheduled-tasks.ts`](../src/config/scheduled-tasks.ts)
10+
11+
## Lifecycle
12+
13+
```
14+
active/completed ──(inactive ≥ archiveAfterDays)──▶ archived ──(retention elapsed)──▶ purged
15+
16+
└──(restore)──▶ previous status
17+
```
18+
19+
1. **Sweep** — the daily job scans every project and resolves the retention
20+
policy that applies to it.
21+
2. **Archive** — projects whose status is listed in `eligibleStatuses` and whose
22+
last activity (`endDate`, else `updatedAt`) is older than `archiveAfterDays`
23+
are archived. An `ArchiveRecord` captures the previous status, milestone
24+
count, budget, and the `purgeEligibleAt` timestamp.
25+
3. **Warn** — an archive within 7 days of its purge date emits a `purge_due`
26+
notification to the project owner and client.
27+
4. **Purge** — once `purgeAfterDays` has elapsed the project, its milestones,
28+
and its payment releases are deleted permanently.
29+
5. **Restore** — restoring before the purge returns the project to its
30+
pre-archive status and closes the archive record.
31+
32+
## Retention policies
33+
34+
Policies resolve most-specific-first: `owner``client``global`. The seeded
35+
`default` policy is global and cannot be deleted.
36+
37+
| Field | Meaning |
38+
| --- | --- |
39+
| `scope` / `scopeId` | `global`, or `client`/`owner` bound to an id |
40+
| `archiveAfterDays` | Inactivity required before archival |
41+
| `purgeAfterDays` | Retention window after archival (must be ≥ `archiveAfterDays`) |
42+
| `eligibleStatuses` | Statuses eligible for archival (default `completed`, `abandoned`) |
43+
| `enabled` | Disabled policies are skipped during resolution |
44+
| `notify` | Whether the policy emits archival notifications |
45+
46+
Defaults come from `PROJECT_ARCHIVE_AFTER_DAYS` (90) and
47+
`PROJECT_PURGE_AFTER_DAYS` (365).
48+
49+
## Scheduling
50+
51+
The sweep runs daily at `04:00 UTC`. Override it without a code change:
52+
53+
```bash
54+
SCHEDULE_OVERRIDE_PROJECT_ARCHIVAL_SWEEP="0 */6 * * *"
55+
```
56+
57+
## API
58+
59+
| Method | Path | Purpose |
60+
| --- | --- | --- |
61+
| `GET` | `/policies` | List retention policies |
62+
| `POST` | `/policies` | Create or update a policy (pass `id` to update) |
63+
| `DELETE` | `/policies/:policyId` | Delete a non-default policy |
64+
| `GET` | `/candidates` | Preview what the next sweep would archive |
65+
| `POST` | `/run` | Run the sweep now; `{ "dryRun": true }` for a no-op preview |
66+
| `POST` | `/archive` | Archive one project immediately, bypassing the window |
67+
| `GET` | `/archives` | List archives (`clientId`, `ownerId`, `policyId`, `includeRestored`, `includePurged`) |
68+
| `POST` | `/restore/:projectId` | Restore the project's latest active archive |
69+
| `GET` | `/analytics` | Archival analytics |
70+
| `GET` | `/notifications` | Notification feed (`limit`, `projectId`) |
71+
72+
Reads require the `projects:read` permission, writes `projects:write`, and
73+
policy deletion `projects:delete`.
74+
75+
## Analytics
76+
77+
`GET /analytics` returns archived/retained/restored/purged totals, the budget
78+
value currently held in archives, pending candidate count, restoration rate,
79+
average inactivity at archival, average time spent in retention, breakdowns by
80+
reason, policy, and month, the next 20 upcoming purges, and `lastRunAt`.
81+
82+
## Notifications
83+
84+
Four notification types are emitted to the owner and client: `archived`,
85+
`purge_due`, `purged`, and `restored`. The feed is capped at the 500 most recent
86+
entries and each is mirrored to the application log.

backend/src/config/scheduled-tasks.ts

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ import { getArchivalService } from '../services/archival/index.js';
2424
import { getBridgeMonitorService } from '../services/bridge-monitor/bridge-monitor.js';
2525
import { runScheduledReconciliation } from '../services/payment-reconciliation/index.js';
2626
import { runEscalationEvaluation } from '../jobs/escalation.job.js';
27+
import { runProjectArchivalSweep } from '../services/project-archival/index.js';
2728
import { ethers } from 'ethers';
2829

2930
// ---------------------------------------------------------------------------
@@ -246,6 +247,19 @@ const RAW_TASKS: (Omit<ScheduledTaskMeta, 'schedule'> & { defaultSchedule: strin
246247
}
247248
},
248249
},
250+
{
251+
id: 'project-archival-sweep',
252+
name: 'Automated Project Archival',
253+
description:
254+
'Archives projects that have exceeded their retention policy window, warns about upcoming purges, and purges archives past retention.',
255+
defaultSchedule: '0 4 * * *',
256+
timezone: 'UTC',
257+
timeoutMs: 10 * 60 * 1000,
258+
priority: 'normal',
259+
handler: () => {
260+
runProjectArchivalSweep();
261+
},
262+
},
249263
{
250264
id: 'escalation-evaluation',
251265
name: 'Escalation SLA Evaluation',

backend/src/index.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ import { paymentLinksRouter } from './routes/payment-links.js';
8282
import { paymentStrategiesRouter } from './routes/payment-strategies.js';
8383
import { taxRouter } from './routes/tax.js';
8484
import { projectsRouter } from './routes/projects.js';
85+
import { projectArchivalRouter } from './routes/project-archival.js';
8586
import { graphQLRouter, graphQLWsRouter } from './graphql/gateway.js';
8687
import { fraudDetectionRouter } from './routes/fraud-detection.js';
8788
import { bridgeRouter } from './routes/bridge.js';
@@ -333,6 +334,7 @@ apiV1Router.use('/emails', emailRouter);
333334
apiV1Router.use('/portfolio', portfolioRouter);
334335
apiV1Router.use('/backup', backupRouter);
335336
apiV1Router.use('/archival', archivalRouter);
337+
apiV1Router.use('/project-archival', projectArchivalRouter);
336338
apiV1Router.use('/admin/contracts/upgrade', upgradeValidatorRouter);
337339
apiV1Router.use('/bridge/monitor', bridgeMonitorRouter);
338340
apiV1Router.use('/ip-allowlist', ipAllowlistRouter);
@@ -433,6 +435,9 @@ app.use('/api/v1/exports', streamingExportRouter);
433435
// Project + milestone delivery approval workflow
434436
app.use('/api/v1/projects', projectsRouter);
435437

438+
// Automated project archival + data retention
439+
app.use('/api/v1/project-archival', projectArchivalRouter);
440+
436441
// Payment categories — Issue #251
437442
app.use('/api/v1/categories', categoriesRouter);
438443

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
/**
2+
* project-archival.ts — automated project archival with data retention.
3+
*
4+
* Mounted at /api/v1/project-archival. Kept separate from the projects
5+
* router so the collection paths do not collide with its "/:id" routes.
6+
*/
7+
8+
import { Router } from 'express';
9+
import { z } from 'zod';
10+
import { validate } from '../middleware/validate.js';
11+
import { requireEnhancedPermission } from '../middleware/permissions.js';
12+
import { projectArchivalService } from '../services/project-archival/index.js';
13+
14+
export const projectArchivalRouter = Router();
15+
16+
const projectStatus = z.enum(['active', 'completed', 'archived', 'disputed', 'abandoned']);
17+
18+
const policySchema = z.object({
19+
id: z.string().min(1).optional(),
20+
name: z.string().min(1),
21+
scope: z.enum(['global', 'client', 'owner']).optional(),
22+
scopeId: z.string().min(1).nullable().optional(),
23+
archiveAfterDays: z.number().int().min(0),
24+
purgeAfterDays: z.number().int().min(0),
25+
eligibleStatuses: z.array(projectStatus).optional(),
26+
enabled: z.boolean().optional(),
27+
notify: z.boolean().optional(),
28+
});
29+
30+
const runSchema = z.object({
31+
dryRun: z.boolean().optional(),
32+
});
33+
34+
const restoreSchema = z.object({
35+
restoredBy: z.string().min(1).optional(),
36+
});
37+
38+
const archiveSchema = z.object({
39+
projectId: z.string().min(1),
40+
actor: z.string().min(1).optional(),
41+
});
42+
43+
function actorOf(req: unknown): string | undefined {
44+
return (req as { user?: { id?: string } }).user?.id;
45+
}
46+
47+
// ── Retention policies ──────────────────────────────────────────────────────
48+
49+
projectArchivalRouter.get(
50+
'/policies',
51+
requireEnhancedPermission('projects', 'read'),
52+
(_req, res, next) => {
53+
try {
54+
const policies = projectArchivalService.listPolicies();
55+
res.json({ policies, count: policies.length });
56+
} catch (err) { next(err); }
57+
},
58+
);
59+
60+
projectArchivalRouter.post(
61+
'/policies',
62+
requireEnhancedPermission('projects', 'write'),
63+
validate(policySchema),
64+
(req, res) => {
65+
try {
66+
res.status(201).json(projectArchivalService.configurePolicy(req.body));
67+
} catch (err) {
68+
res.status(400).json({ error: (err as Error).message });
69+
}
70+
},
71+
);
72+
73+
projectArchivalRouter.delete(
74+
'/policies/:policyId',
75+
requireEnhancedPermission('projects', 'delete'),
76+
(req, res, next) => {
77+
try {
78+
const deleted = projectArchivalService.deletePolicy(String(req.params.policyId));
79+
if (!deleted) {
80+
res.status(400).json({ error: 'Policy not found or cannot be deleted' });
81+
return;
82+
}
83+
res.json({ deleted: true, policyId: req.params.policyId });
84+
} catch (err) { next(err); }
85+
},
86+
);
87+
88+
// ── Archival sweep ──────────────────────────────────────────────────────────
89+
90+
projectArchivalRouter.get(
91+
'/candidates',
92+
requireEnhancedPermission('projects', 'read'),
93+
(_req, res, next) => {
94+
try {
95+
const candidates = projectArchivalService.previewArchival();
96+
res.json({ candidates, count: candidates.length });
97+
} catch (err) { next(err); }
98+
},
99+
);
100+
101+
projectArchivalRouter.post(
102+
'/run',
103+
requireEnhancedPermission('projects', 'write'),
104+
validate(runSchema),
105+
(req, res, next) => {
106+
try {
107+
res.json(projectArchivalService.runArchival({ dryRun: Boolean(req.body?.dryRun) }));
108+
} catch (err) { next(err); }
109+
},
110+
);
111+
112+
projectArchivalRouter.post(
113+
'/archive',
114+
requireEnhancedPermission('projects', 'write'),
115+
validate(archiveSchema),
116+
(req, res, next) => {
117+
try {
118+
const record = projectArchivalService.archiveNow(
119+
req.body.projectId,
120+
req.body.actor ?? actorOf(req),
121+
);
122+
if (!record) {
123+
res.status(404).json({ error: 'Project not found or already archived' });
124+
return;
125+
}
126+
res.status(201).json(record);
127+
} catch (err) { next(err); }
128+
},
129+
);
130+
131+
// ── Archives, restoration, analytics, notifications ─────────────────────────
132+
133+
projectArchivalRouter.get(
134+
'/archives',
135+
requireEnhancedPermission('projects', 'read'),
136+
(req, res, next) => {
137+
try {
138+
const archives = projectArchivalService.listArchives({
139+
clientId: typeof req.query.clientId === 'string' ? req.query.clientId : undefined,
140+
ownerId: typeof req.query.ownerId === 'string' ? req.query.ownerId : undefined,
141+
policyId: typeof req.query.policyId === 'string' ? req.query.policyId : undefined,
142+
includeRestored: req.query.includeRestored === 'true',
143+
includePurged: req.query.includePurged === 'true',
144+
});
145+
res.json({ archives, count: archives.length });
146+
} catch (err) { next(err); }
147+
},
148+
);
149+
150+
projectArchivalRouter.post(
151+
'/restore/:projectId',
152+
requireEnhancedPermission('projects', 'write'),
153+
validate(restoreSchema),
154+
(req, res, next) => {
155+
try {
156+
const restored = projectArchivalService.restoreProject(
157+
String(req.params.projectId),
158+
req.body?.restoredBy ?? actorOf(req),
159+
);
160+
if (!restored) {
161+
res.status(404).json({ error: 'No restorable archive found for this project' });
162+
return;
163+
}
164+
res.json(restored);
165+
} catch (err) { next(err); }
166+
},
167+
);
168+
169+
projectArchivalRouter.get(
170+
'/analytics',
171+
requireEnhancedPermission('projects', 'read'),
172+
(_req, res, next) => {
173+
try {
174+
res.json(projectArchivalService.getAnalytics());
175+
} catch (err) { next(err); }
176+
},
177+
);
178+
179+
projectArchivalRouter.get(
180+
'/notifications',
181+
requireEnhancedPermission('projects', 'read'),
182+
(req, res, next) => {
183+
try {
184+
const limit = typeof req.query.limit === 'string' ? parseInt(req.query.limit, 10) : 50;
185+
const projectId = typeof req.query.projectId === 'string' ? req.query.projectId : undefined;
186+
const notifications = projectArchivalService.listNotifications(
187+
Number.isFinite(limit) ? limit : 50,
188+
projectId,
189+
);
190+
res.json({ notifications, count: notifications.length });
191+
} catch (err) { next(err); }
192+
},
193+
);

0 commit comments

Comments
 (0)