Skip to content

Commit 42dcce3

Browse files
author
lihua
committed
feat: add processing module with configurations and types for TOA processing
1 parent a28392f commit 42dcce3

32 files changed

Lines changed: 4075 additions & 133 deletions

‎README.md‎

Lines changed: 48 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -131,13 +131,15 @@ psrchive-ele/
131131
│ │ └── index.ts # Context bridge (IPC → renderer)
132132
│ ├── shared/
133133
│ │ ├── commands.ts # Shared command ids for menu / shortcuts / UI
134+
│ │ ├── processing.ts # Shared processing/session/TOA/batch types
134135
│ │ └── update.ts # Shared updater state types
135136
│ └── renderer/src/
136137
│ ├── App.tsx # Root: data loading, shared command handlers
137138
│ ├── components/
138139
│ │ ├── TitleBar.tsx # Icon command menu + updater status
139140
│ │ ├── Sidebar.tsx
140141
│ │ ├── MainPanel.tsx
142+
│ │ ├── ProcessingInspector.tsx # Session-based PSRCHIVE workflow UI
141143
│ │ ├── StatusBar.tsx
142144
│ │ ├── SettingsPanel.tsx # Categorized settings center
143145
│ │ ├── HelpPanel.tsx
@@ -159,11 +161,14 @@ psrchive-ele/
159161
│ │ ├── main.py # FastAPI app
160162
│ │ ├── routes.py # REST endpoints
161163
│ │ ├── data_provider.py # Mock / real psrchive data
164+
│ │ ├── processing.py # Session materialization + paz/pam/pat/pac orchestration
162165
│ │ └── psrcat.py # PSRCAT database parser
163166
│ └── requirements.txt
164167
├── docs/ # Developer documentation
165168
│ ├── architecture.md # System overview & communication flows
166169
│ ├── api.md # Backend REST API reference
170+
│ ├── processing-guide.md # End-user session processing workflow guide
171+
│ ├── psrchive-features.md # Implemented vs planned PSRCHIVE capability matrix
167172
│ ├── components.md # React component reference
168173
│ ├── state.md # Jotai atom reference
169174
│ ├── shortcuts.md # Keyboard shortcuts
@@ -179,12 +184,20 @@ psrchive-ele/
179184
| Method | Path | Description |
180185
|--------|---------------------------|----------------------------------|
181186
| GET | `/api/health` | Backend status + provider name |
187+
| GET | `/api/capabilities` | Runtime/provider/CLI processing capabilities |
182188
| GET | `/api/files?dir=` | List archive files in directory |
183189
| GET | `/api/archive?path=` | Archive metadata |
184190
| GET | `/api/archive/profile` | Pulse profile (Stokes I/Q/U/V) |
185191
| GET | `/api/archive/waterfall` | Frequency × Phase heatmap |
186192
| GET | `/api/archive/time-phase` | Time × Phase heatmap |
187193
| GET | `/api/archive/bandpass` | Mean intensity per channel |
194+
| POST | `/api/sessions` | Create a non-destructive processing session |
195+
| PATCH | `/api/sessions/{id}/recipe` | Update the active processing recipe |
196+
| GET | `/api/sessions/{id}/preview/*` | Session preview metadata + charts |
197+
| POST | `/api/sessions/{id}/export` | Export a processed archive copy |
198+
| POST | `/api/sessions/{id}/toa` | Run `pat` and return TOA + residual preview |
199+
| POST | `/api/sessions/{id}/calibration/preview` | Inspect the active `pac` preview command/log |
200+
| DELETE | `/api/sessions/{id}` | Destroy a processing session |
188201
| GET | `/api/psrcat/pulsars` | All PSRCAT pulsars |
189202
| GET | `/api/psrcat/pulsar/{n}` | Single pulsar by name |
190203
| GET | `/api/psrcat/stats` | PSRCAT summary statistics |
@@ -218,12 +231,38 @@ git clone git://git.code.sf.net/p/psrchive/code psrchive
218231
cd psrchive && ./bootstrap && ./configure && make && make install
219232
```
220233

234+
## PSRCHIVE Processing Workflow
235+
236+
The app now ships a session-based processing workflow:
237+
238+
- opening an archive creates a non-destructive backend processing session
239+
- the right-side `Processing` inspector drives `paz`, `pam`, `pat`, and `pac`
240+
- chart previews always come from `/api/sessions/{id}/preview/*`
241+
- `Save Archive` exports a new processed copy instead of mutating the source file
242+
243+
Implemented in v1:
244+
245+
- interactive channel zapping from the waterfall
246+
- live `pam` controls for `dedisperse`, `tscrunch`, `fscrunch`, `bscrunch`, and `phase rotate`
247+
- `pat` TOA extraction with an observed/template/difference residual preview
248+
- calibration preview from an existing search path / `database.txt` / solution file
249+
- local workspace-scoped batch recipe save/load/run support
250+
251+
Current limitations:
252+
253+
- zapping is channel-only for now
254+
- TOA uses `pat` and visual residuals, not full `tempo2` timing residuals yet
255+
- calibration does not build new databases from raw calibrator observations
256+
- batch processing is sequential foreground orchestration, not a background job queue
257+
221258
## Developer Docs
222259

223260
| Document | Description |
224261
|----------|-------------|
225262
| [docs/architecture.md](docs/architecture.md) | System overview, data flow, state management |
226263
| [docs/api.md](docs/api.md) | Full REST API reference with request/response schemas |
264+
| [docs/processing-guide.md](docs/processing-guide.md) | How to use the Processing Inspector, export flow, TOA, calibration, and batch recipes |
265+
| [docs/psrchive-features.md](docs/psrchive-features.md) | Implemented vs planned PSRCHIVE capabilities and runtime requirements |
227266
| [docs/data-flow.md](docs/data-flow.md) | Local vs Docker runtime, archive-to-chart pipeline, and exact `psrchive` call path |
228267
| [docs/components.md](docs/components.md) | React component props, behavior, and sub-components |
229268
| [docs/state.md](docs/state.md) | All Jotai atoms — types, defaults, persistence |
@@ -242,11 +281,15 @@ cd psrchive && ./bootstrap && ./configure && make && make install
242281
- [x] Multi-window support
243282
- [x] Shared command menu across title bar, shortcuts, and macOS menu bar
244283
- [x] Categorized settings center with updater and backend controls
245-
- [ ] Interactive RFI zapping (click/box-select on waterfall)
246-
- [ ] TOA extraction with visual residuals
247-
- [ ] Polarization calibration wizard
248-
- [ ] Real-time parameter adjustment (sliders for pam operations)
249-
- [ ] Batch processing pipeline configuration
284+
- [x] Interactive RFI zapping (channel click / box-select on waterfall, v1)
285+
- [x] TOA extraction with visual residuals (v1 `pat` workflow)
286+
- [x] Polarization calibration wizard (v1 existing database / solution inputs)
287+
- [x] Real-time parameter adjustment (v1 `pam` controls)
288+
- [x] Batch processing pipeline configuration (v1 saved recipes + sequential execution)
289+
- [ ] Full `tempo2` timing residual workflow
290+
- [ ] Automatic / subint / phase-bin RFI tooling
291+
- [ ] Calibration database builder from raw calibrator observations
292+
- [ ] Background batch queue with retry/history
250293

251294
## References
252295

‎README.zh.md‎

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -131,13 +131,15 @@ psrchive-ele/
131131
│ │ └── index.ts # Context bridge（IPC → renderer）
132132
│ ├── shared/
133133
│ │ ├── commands.ts # 菜单 / 快捷键 / UI 共享 command id
134+
│ │ ├── processing.ts # processing/session/TOA/batch 共享类型
134135
│ │ └── update.ts # 共享 updater 状态类型
135136
│ └── renderer/src/
136137
│ ├── App.tsx # 根组件：数据加载与 command handlers
137138
│ ├── components/
138139
│ │ ├── TitleBar.tsx # 图标命令菜单 + 更新状态
139140
│ │ ├── Sidebar.tsx
140141
│ │ ├── MainPanel.tsx
142+
│ │ ├── ProcessingInspector.tsx # 基于 session 的 PSRCHIVE 处理工作流 UI
141143
│ │ ├── StatusBar.tsx
142144
│ │ ├── SettingsPanel.tsx # 分类设置中心
143145
│ │ ├── HelpPanel.tsx
@@ -159,13 +161,18 @@ psrchive-ele/
159161
│ │ ├── main.py # FastAPI app
160162
│ │ ├── routes.py # REST endpoints
161163
│ │ ├── data_provider.py # Mock / 真正的 psrchive 数据提供者
164+
│ │ ├── processing.py # session materialization + paz/pam/pat/pac 编排
162165
│ │ └── psrcat.py # PSRCAT 数据库解析器
163166
│ └── requirements.txt
164167
├── docs/ # 开发者文档
165168
│ ├── architecture.md
166169
│ ├── architecture.zh.md
167170
│ ├── api.md
168171
│ ├── api.zh.md
172+
│ ├── processing-guide.md
173+
│ ├── processing-guide.zh.md
174+
│ ├── psrchive-features.md
175+
│ ├── psrchive-features.zh.md
169176
│ ├── components.md
170177
│ ├── components.zh.md
171178
│ ├── data-flow.md
@@ -190,12 +197,20 @@ psrchive-ele/
190197
| Method | Path | 说明 |
191198
|--------|------|------|
192199
| GET | `/api/health` | 后端状态 + provider 名称 |
200+
| GET | `/api/capabilities` | runtime/provider/CLI processing 能力探测 |
193201
| GET | `/api/files?dir=` | 列出目录中的归档文件 |
194202
| GET | `/api/archive?path=` | 归档元数据 |
195203
| GET | `/api/archive/profile` | 脉冲轮廓（Stokes I/Q/U/V） |
196204
| GET | `/api/archive/waterfall` | Frequency × Phase 热力图 |
197205
| GET | `/api/archive/time-phase` | Time × Phase 热力图 |
198206
| GET | `/api/archive/bandpass` | 每个频率通道的平均强度 |
207+
| POST | `/api/sessions` | 创建非破坏 processing session |
208+
| PATCH | `/api/sessions/{id}/recipe` | 更新当前 processing recipe |
209+
| GET | `/api/sessions/{id}/preview/*` | 读取 session preview 元数据与图表 |
210+
| POST | `/api/sessions/{id}/export` | 导出处理后的 archive 副本 |
211+
| POST | `/api/sessions/{id}/toa` | 运行 `pat`，返回 TOA 和 residual preview |
212+
| POST | `/api/sessions/{id}/calibration/preview` | 查看当前 `pac` preview 的命令与日志 |
213+
| DELETE | `/api/sessions/{id}` | 销毁 processing session |
199214
| GET | `/api/psrcat/pulsars` | 全部 PSRCAT pulsars |
200215
| GET | `/api/psrcat/pulsar/{n}` | 按名称查询单个 pulsar |
201216
| GET | `/api/psrcat/stats` | PSRCAT 汇总统计 |
@@ -232,12 +247,38 @@ git clone git://git.code.sf.net/p/psrchive/code psrchive
232247
cd psrchive && ./bootstrap && ./configure && make && make install
233248
```
234249

250+
## PSRCHIVE Processing 工作流
251+
252+
应用现在已经内置基于 session 的 processing workflow：
253+
254+
- 打开 archive 时会创建一个非破坏 backend processing session
255+
- 右侧 `Processing` inspector 统一驱动 `paz`、`pam`、`pat`、`pac`
256+
- 图表预览统一来自 `/api/sessions/{id}/preview/*`
257+
- `Save Archive` 永远导出新文件，不会修改原始 archive
258+
259+
当前 v1 已实现：
260+
261+
- 在 waterfall 上做频道级点击 / 框选 zapping
262+
- `dedisperse`、`tscrunch`、`fscrunch`、`bscrunch`、`phase rotate` 的实时 `pam` 控件
263+
- 带 observed/template/difference residual preview 的 `pat` TOA 提取
264+
- 基于已有 search path / `database.txt` / solution file 的 calibration preview
265+
- 按 workspace 保存、加载、执行 batch recipe
266+
267+
当前已知限制：
268+
269+
- zapping 目前只做到频道级
270+
- TOA 目前是 `pat` + visual residual，还不是完整 `tempo2` timing residual
271+
- calibration 还不支持从原始 calibrator observation 建数据库
272+
- batch 目前是前台顺序执行，不是后台任务队列
273+
235274
## 开发文档
236275

237276
| 文档 | 说明 |
238277
|------|------|
239278
| [docs/architecture.zh.md](docs/architecture.zh.md) | 系统总览、通信流、状态管理入口 |
240279
| [docs/api.zh.md](docs/api.zh.md) | 完整 REST API 参考与请求 / 响应结构 |
280+
| [docs/processing-guide.zh.md](docs/processing-guide.zh.md) | Processing Inspector、导出、TOA、calibration、batch recipe 的使用手册 |
281+
| [docs/psrchive-features.zh.md](docs/psrchive-features.zh.md) | 已实现 / 未实现 PSRCHIVE 能力矩阵与 runtime 前提 |
241282
| [docs/data-flow.zh.md](docs/data-flow.zh.md) | local / Docker runtime 对比、文件到图表的数据链路、`psrchive` 实际调用路径 |
242283
| [docs/components.zh.md](docs/components.zh.md) | React 组件职责、props、行为与子结构 |
243284
| [docs/state.zh.md](docs/state.zh.md) | 所有 Jotai atoms、类型、默认值与持久化规则 |
@@ -256,11 +297,15 @@ cd psrchive && ./bootstrap && ./configure && make && make install
256297
- [x] 多窗口支持
257298
- [x] 标题栏、自定义菜单、快捷键、macOS 顶部菜单的统一 command system
258299
- [x] 带 updater 与 backend 控件的分类设置中心
259-
- [ ] 交互式 RFI zapping（在 waterfall 上点击 / 框选）
260-
- [ ] 带可视化 residuals 的 TOA 提取
261-
- [ ] 偏振标定向导
262-
- [ ] 实时参数调节（pam 操作滑杆）
263-
- [ ] 批处理流水线配置
300+
- [x] 交互式 RFI zapping（waterfall 上的频道点击 / 框选，v1）
301+
- [x] 带可视化 residuals 的 TOA 提取（v1 `pat` 工作流）
302+
- [x] 偏振标定向导（v1，仅已有 database / solution 输入）
303+
- [x] 实时参数调节（v1 `pam` 控件）
304+
- [x] 批处理流水线配置（v1，recipe 保存 + 顺序执行）
305+
- [ ] 完整 `tempo2` timing residual 工作流
306+
- [ ] 自动 / subint / phase-bin RFI 工具
307+
- [ ] 从原始 calibrator observation 构建 calibration database
308+
- [ ] 带重试 / 历史的后台 batch queue
264309

265310
## 参考资料
266311

‎backend/app/data_provider.py‎

Lines changed: 54 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -25,17 +25,30 @@ def get_metadata(self, path: str) -> dict[str, Any]: ...
2525

2626
@abstractmethod
2727
def get_profile(
28-
self, path: str, *, subint: int | None = None, chan: int | None = None
28+
self,
29+
path: str,
30+
*,
31+
subint: int | None = None,
32+
chan: int | None = None,
33+
dedisperse: bool = True,
2934
) -> dict[str, Any]: ...
3035

3136
@abstractmethod
3237
def get_waterfall(
33-
self, path: str, *, subint: int | None = None
38+
self,
39+
path: str,
40+
*,
41+
subint: int | None = None,
42+
dedisperse: bool = True,
3443
) -> dict[str, Any]: ...
3544

3645
@abstractmethod
3746
def get_time_phase(
38-
self, path: str, *, chan: int | None = None
47+
self,
48+
path: str,
49+
*,
50+
chan: int | None = None,
51+
dedisperse: bool = True,
3952
) -> dict[str, Any]: ...
4053

4154
@abstractmethod
@@ -106,7 +119,12 @@ def get_metadata(self, path: str) -> dict[str, Any]:
106119
}
107120

108121
def get_profile(
109-
self, path: str, *, subint: int | None = None, chan: int | None = None
122+
self,
123+
path: str,
124+
*,
125+
subint: int | None = None,
126+
chan: int | None = None,
127+
dedisperse: bool = True,
110128
) -> dict[str, Any]:
111129
p = self._parse_params(path)
112130
rng = p["rng"]
@@ -132,7 +150,11 @@ def get_profile(
132150
}
133151

134152
def get_waterfall(
135-
self, path: str, *, subint: int | None = None
153+
self,
154+
path: str,
155+
*,
156+
subint: int | None = None,
157+
dedisperse: bool = True,
136158
) -> dict[str, Any]:
137159
p = self._parse_params(path)
138160
rng = p["rng"]
@@ -167,7 +189,11 @@ def get_waterfall(
167189
}
168190

169191
def get_time_phase(
170-
self, path: str, *, chan: int | None = None
192+
self,
193+
path: str,
194+
*,
195+
chan: int | None = None,
196+
dedisperse: bool = True,
171197
) -> dict[str, Any]:
172198
p = self._parse_params(path)
173199
rng = p["rng"]
@@ -250,11 +276,17 @@ def get_metadata(self, path: str) -> dict[str, Any]:
250276
}
251277

252278
def get_profile(
253-
self, path: str, *, subint: int | None = None, chan: int | None = None
279+
self,
280+
path: str,
281+
*,
282+
subint: int | None = None,
283+
chan: int | None = None,
284+
dedisperse: bool = True,
254285
) -> dict[str, Any]:
255286
ar = self._psrchive.Archive_load(path)
256287
ar.remove_baseline()
257-
ar.dedisperse()
288+
if dedisperse:
289+
ar.dedisperse()
258290

259291
if subint is None:
260292
ar.tscrunch()
@@ -279,11 +311,16 @@ def get_profile(
279311
return result
280312

281313
def get_waterfall(
282-
self, path: str, *, subint: int | None = None
314+
self,
315+
path: str,
316+
*,
317+
subint: int | None = None,
318+
dedisperse: bool = True,
283319
) -> dict[str, Any]:
284320
ar = self._psrchive.Archive_load(path)
285321
ar.remove_baseline()
286-
ar.dedisperse()
322+
if dedisperse:
323+
ar.dedisperse()
287324
if subint is None:
288325
ar.tscrunch()
289326

@@ -302,11 +339,16 @@ def get_waterfall(
302339
}
303340

304341
def get_time_phase(
305-
self, path: str, *, chan: int | None = None
342+
self,
343+
path: str,
344+
*,
345+
chan: int | None = None,
346+
dedisperse: bool = True,
306347
) -> dict[str, Any]:
307348
ar = self._psrchive.Archive_load(path)
308349
ar.remove_baseline()
309-
ar.dedisperse()
350+
if dedisperse:
351+
ar.dedisperse()
310352
if chan is None:
311353
ar.fscrunch()
312354

0 commit comments

Comments
 (0)