Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ All notable changes to this project are documented here. The format follows [Kee

### Fixed

- IBKR Flex Query XML statements can be uploaded on Import → File upload, and their times are read in the statement timezone, as broker sync does. Previously an XML upload was not recognized and could not be imported, so the Default import timezone never applied ([#16](https://github.com/LuxAlgo/trade-journal/issues/16)).
- Password protection now verifies the session signature on every API route. Previously, when `JOURNAL_PASSWORD` was set, any request carrying a cookie of the right name was accepted, so a forged cookie could read the journal.

## [0.1.0] - 2026-09-03
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ One primitive drives everything: a raw **execution** (a fill). Executions come i
```mermaid
flowchart LR
BS["Broker sync (broker-sdk)"] --> EX["Executions (deduped fills)"]
IM["14 documented statement formats + CSV mapper"] --> EX
IM["15 documented statement formats + CSV mapper"] --> EX
MN["Manual entry"] --> EX
EX --> RT["Round-trip engine (FIFO / LIFO / WAVG)"]
RT --> D["Dashboard + calendar"]
Expand All @@ -129,7 +129,7 @@ flowchart LR
| | |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Broker sync** | Read-only sync via [`@luxalgo/broker-sdk`](https://github.com/LuxAlgo/broker-sdk): Alpaca, Binance, Bybit, Coinbase, Kraken, OKX, Tradier, IBKR Flex, Hyperliquid, Questrade, Topstep, Trading212, Webull, Crypto.com, E*TRADE, Public, Charles Schwab, TradeStation, tastytrade, Robinhood Crypto, Gemini, and KuCoin. Connect forms render straight from SDK metadata. |
| **Statement import** | **14 documented formats**, auto-detected: **TradeZella** and **Tradervue** (CSV migration), TradingView (paper and strategy exports), MetaTrader 4 and 5, ThinkorSwim, IBKR (activity + Flex), NinjaTrader, Tradovate, TopstepX, Webull, DAS Trader, plus a column mapper for any other CSV. Matching execution records dedupe within an account; review warnings and skipped rows before committing an import. Have an export we don't recognize? Open an issue with an anonymized sample; real files are the most useful contribution this repo can get. |
| **Statement import** | **15 documented formats**, auto-detected: **TradeZella** and **Tradervue** (CSV migration), TradingView (paper and strategy exports), MetaTrader 4 and 5, ThinkorSwim, IBKR (activity + Flex), NinjaTrader, Tradovate, TopstepX, Webull, DAS Trader, plus a column mapper for any other CSV. Matching execution records dedupe within an account; review warnings and skipped rows before committing an import. Have an export we don't recognize? Open an issue with an anonymized sample; real files are the most useful contribution this repo can get. |
| **Round-trip engine** | Flat-to-flat position cycles from raw fills. FIFO / LIFO / weighted-average per account. Partial fills, scale-ins, flips, futures multipliers. Annotations survive rebuilds. |
| **Analytics** | Net/gross P&L, win and day-win rates, profit factor, expectancy, R multiples, streaks, drawdown and recovery, profit concentration, duration/time-of-day/weekday performance, per-symbol/tag/mistake/playbook breakdowns. Advanced filters on every dimension, comparison groups, and a two-way cross-analysis matrix. Reports also include rolling performance trends and a trade explorer scatter plot, with saved MAE/MFE estimates where available. |
| **Dashboard** | P&L calendar with weekly totals, cumulative and daily P&L, gauges, the open **Edge Score v2** radar, open positions, time-of-day performance. Drag cards to rearrange, hide what you don't use, save named layouts. Calendar insights highlight daily patterns. Light/dark themes, a collapsible sidebar, and mobile navigation support smaller screens. |
Expand All @@ -142,8 +142,8 @@ flowchart LR
| **Prop firms** | Evaluation and reset expenses, refunds, payout requests, partial receipts, reversals, cash ROI, account phases, renewal reminders, attachments, and generic cash CSV import/export. Separate from trade P&L. |
| **Privacy & export** | Privacy mode masks every monetary value (charts keep their shape) and persists across tabs. Trades export to CSV, reviews to PDF or PNG, and journal records to JSON; credentials, candle datasets, and attachment binaries are excluded from that export. |

IBKR broker sync uses **Settings → General → Default import timezone** for Flex
timestamps without an offset. Existing synced accounts with unknown or different
IBKR broker sync and uploaded Flex XML statements use **Settings → General →
Default import timezone** for Flex timestamps without an offset. Existing synced accounts with unknown or different
timezone provenance require recovery into a separate account; see
[IBKR timezone recovery](docs/importers.md#ibkr-broker-sync-and-timezone-recovery).

Expand Down
24 changes: 20 additions & 4 deletions apps/web/src/app/api/import/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ import { getImportTimeZone } from "@/server/settings";
import { isTimeZone } from "@/lib/timezone";
import type { ImportReviewOptions } from "@/lib/import-review";
import { previewNinjaTraderImport, commitNinjaTraderImport } from "@/server/ninjatrader-import";
import {
IBKR_FLEX_XML,
flexUploadSyncProblem,
looksLikeFlexXml,
parseFlexXmlImport,
} from "@/server/ibkr-flex-xml-import";
import type { AiImportOptions } from "@/lib/ai-import";
import {
createAiImportPreview,
Expand Down Expand Up @@ -75,7 +81,9 @@ export const POST = handler(async (request: Request) => {
: readAiImportPreview(statement, body.aiPreviewToken!)
: body.mapping
? parseWithMapping(body.content, body.mapping, { timeZone })
: parseAuto(body.content, { timeZone, fileName: body.fileName, symbol: body.symbol });
: looksLikeFlexXml(body.content)
? parseFlexXmlImport(body.content, timeZone)
: parseAuto(body.content, { timeZone, fileName: body.fileName, symbol: body.symbol });

if (!parsed) {
return ok({
Expand All @@ -86,6 +94,12 @@ export const POST = handler(async (request: Request) => {
});
}

const syncProblem =
parsed.format === IBKR_FLEX_XML.id && body.accountId
? flexUploadSyncProblem(body.accountId, timeZone)
: null;
const errors = [...(parsed.errors ?? []), ...(syncProblem ? [syncProblem] : [])];

if (body.mode === "preview") {
const symbols = [...new Set(parsed.executions.map((e) => e.symbol))];
return ok({
Expand Down Expand Up @@ -113,7 +127,7 @@ export const POST = handler(async (request: Request) => {
),
},
warnings: parsed.warnings,
errors: parsed.errors,
errors: syncProblem ? errors : parsed.errors,
needsSymbol: parsed.needsSymbol,
reconciliation:
parsed.format === "ninjatrader" && body.accountId
Expand All @@ -123,7 +137,7 @@ export const POST = handler(async (request: Request) => {
}

if (!body.accountId) return bad("accountId is required to commit");
if (parsed.errors?.length) return bad(parsed.errors.join(" "));
if (errors.length) return bad(errors.join(" "));
if (parsed.executions.length === 0) return bad("No executions to import");
const result =
parsed.format === "ninjatrader"
Expand All @@ -148,4 +162,6 @@ export const POST = handler(async (request: Request) => {
});

/** The import page lists what auto-detection understands. */
export const GET = handler(() => ok({ formats: FORMATS.map(({ id, label }) => ({ id, label })) }));
export const GET = handler(() =>
ok({ formats: [...FORMATS, IBKR_FLEX_XML].map(({ id, label }) => ({ id, label })) }),
);
10 changes: 7 additions & 3 deletions apps/web/src/app/import/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -403,8 +403,12 @@ function FileImport() {
) : (
<>
Auto-detected:{" "}
{formatData?.formats.map((format) => format.label.split(" (")[0]).join(", ")} —
anything else via column mapping.
{[
...new Set(
formatData?.formats.map((format) => format.label.split(" (")[0]) ?? [],
),
].join(", ")}{" "}
— anything else via column mapping.
</>
)}
</span>
Expand Down Expand Up @@ -766,7 +770,7 @@ function BrokerConnect() {
{settingsError
? "Could not load the import timezone. Refresh and try again."
: settingsData
? `Statement timezone: ${settingsData.importTimeZone}. Used for Flex timestamps without an offset. Change it in Settings → Journal before connecting.`
? `Statement timezone: ${settingsData.importTimeZone}. Used for Flex timestamps without an offset, and required for Flex XML files uploaded into this account. Change it in Settings → Journal before connecting.`
: "Loading statement timezone…"}
</p>
)}
Expand Down
28 changes: 28 additions & 0 deletions apps/web/src/server/broker-trades.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import type { Trade } from "@luxalgo/broker-sdk";
import type { ImportedExecution } from "@luxalgo/journal-importers";

/**
* SDK fills as journal executions. Broker sync and statement uploads share this
* mapping so the same fill has the same content hash from either path.
*/
export const brokerTradesToExecutions = (
trades: Trade[],
): { timed: ImportedExecution[]; untimed: number } => {
const rows: ImportedExecution[] = trades.map((trade) => ({
symbol: trade.symbol,
side: trade.side,
quantity: trade.quantity,
price: trade.price,
fee: trade.fee ?? 0,
// The SDK omits unparseable timestamps; a fill with no time can't be
// journaled meaningfully, so it is dropped rather than guessed at.
executedAt: trade.executedAt ?? "",
}));
const timed = rows.filter((row) => row.executedAt !== "");
return { timed, untimed: rows.length - timed.length };
};

export const untimedFillsReason = (untimed: number, statementTimeZone: boolean): string =>
statementTimeZone
? `${untimed} fill(s) had no usable timestamp. Check for missing or invalid dates, unsupported timezone suffixes, or ambiguous or nonexistent daylight-saving times.`
: `${untimed} fill(s) had no usable timestamp.`;
122 changes: 122 additions & 0 deletions apps/web/src/server/ibkr-flex-xml-import.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
import { BrokerError } from "@luxalgo/broker-sdk";
import { parseFlexStatement } from "@luxalgo/broker-sdk/adapters";
import type { ParsedImport } from "@luxalgo/journal-importers";
import { eq } from "drizzle-orm";
import { accounts, db } from "@/db";
import { RequestError, requireValue } from "./api";
import { brokerTradesToExecutions, untimedFillsReason } from "./broker-trades";
import {
assertIbkrSyncTimeZone,
canonicalImportTimeZone,
isIbkrSyncAccount,
} from "./ibkr-sync-timezone";
import { getImportTimeZone } from "./settings";

export const IBKR_FLEX_XML = { id: "ibkr-flex-xml", label: "Interactive Brokers (Flex Query XML)" };

/** A Flex Query response or statement, optionally after an XML declaration and comments. */
export const looksLikeFlexXml = (content: string): boolean =>
/^\s*(?:<\?xml[^>]*\?>\s*)?(?:<!--[\s\S]*?-->\s*)*<(?:FlexQueryResponse|FlexStatements?)[\s>]/.test(
content.replace(/^/, ""),
);

/** Attribute maps for one element name, read the same way as the SDK's Flex parser. */
const elements = (xml: string, tag: string): Record<string, string>[] =>
[...xml.matchAll(new RegExp(`<${tag}\\s([^>]*?)/?>`, "g"))].map((match) =>
Object.fromEntries(
[...(match[1] ?? "").matchAll(/([\w-]+)="([^"]*)"/g)].map((pair) => [pair[1]!, pair[2]!]),
),
);

const isSummaryRow = (trade: Record<string, string>): boolean =>
Boolean(trade.levelOfDetail) && trade.levelOfDetail!.toUpperCase() !== "EXECUTION";

/**
* Parse an uploaded Flex Query XML statement with the broker SDK's Flex parser,
* the same parser broker sync uses. Naive `dateTime` values are read in the
* statement timezone, so the "Default import timezone" setting applies to
* uploads exactly as it applies to sync.
*/
export const parseFlexXmlImport = (content: string, timeZone: string): ParsedImport => {
requireValue(
!/^[+-]/.test(timeZone.trim()),
"Choose a named IANA statement timezone for IBKR Flex statements, such as America/New_York, not a UTC offset.",
);
const zone = canonicalImportTimeZone(timeZone);
const errors: string[] = [];
const warnings: string[] = [];

const rows = elements(content, "Trade");
const accountIds = new Set(
[...elements(content, "FlexStatement"), ...rows]
.map((attrs) => attrs.accountId?.trim())
.filter((id): id is string => Boolean(id)),
);
if (accountIds.size > 1)
errors.push(
`This Flex statement contains ${accountIds.size} IBKR accounts (${[...accountIds].join(", ")}). Export one account per Flex Query, or import each account's statement into its own journal account.`,
);

// Order, closed-lot and summary rows repeat the executions they summarize.
const summaries = rows.filter(isSummaryRow);
const executionXml = summaries.length
? content.replace(/<Trade\s[^>]*?\/?>/g, (tag) =>
isSummaryRow(elements(tag, "Trade")[0] ?? {}) ? "" : tag,
)
: content;
if (summaries.length)
warnings.push(
`${summaries.length} Trade summary row(s) (${[...new Set(summaries.map((row) => row.levelOfDetail))].join(", ")}) were skipped; only execution rows are imported.`,
);

let parsed: ReturnType<typeof parseFlexStatement>;
try {
parsed = parseFlexStatement(executionXml, { statementTimeZone: zone });
} catch (error) {
if (error instanceof BrokerError) requireValue(false, error.message);
throw error;
}

const { timed, untimed } = brokerTradesToExecutions(parsed.trades);
const dropped = rows.length - summaries.length - parsed.trades.length;
if (dropped > 0)
warnings.push(
`${dropped} Trade row(s) were skipped: a missing symbol, side, quantity or price, or a zero price such as an option expiry or assignment.`,
);
if (untimed > 0)
warnings.push(
`${untimedFillsReason(untimed, true)} Flex dates must use the default yyyyMMdd date and HHmmss time formats.`,
);
if (timed.length === 0 && errors.length === 0)
errors.push(
"No executions could be read from this Flex statement. Include the Trades section at Execution level in the Flex Query.",
);

return {
format: IBKR_FLEX_XML.id,
executions: timed,
skippedRows: dropped + untimed,
warnings,
...(errors.length ? { errors } : {}),
};
};

/**
* An upload into an IBKR sync account must use the statement timezone sync
* uses, or the same fills would hash differently and be stored twice.
*/
export const flexUploadSyncProblem = (accountId: string, zone: string): string | null => {
const account = db.select().from(accounts).where(eq(accounts.id, accountId)).get();
if (!account || (!isIbkrSyncAccount(account) && account.ibkrSyncTimeZone === null)) return null;
const syncZone = canonicalImportTimeZone(account.ibkrSyncTimeZone ?? getImportTimeZone());
const uploadZone = canonicalImportTimeZone(zone);
if (uploadZone !== syncZone)
return `This account syncs IBKR statements in ${syncZone}, but this upload uses ${uploadZone}. Upload with the ${syncZone} statement timezone so uploaded and synced fills match, or import into a separate account.`;
try {
assertIbkrSyncTimeZone(account, uploadZone);
} catch (error) {
if (error instanceof RequestError) return error.message;
throw error;
}
return null;
};
23 changes: 4 additions & 19 deletions apps/web/src/server/sync.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import { connect, listBrokers, type BrokerId } from "@luxalgo/broker-sdk";
import { eq } from "drizzle-orm";
import type { ImportedExecution } from "@luxalgo/journal-importers";
import { accounts, db } from "@/db";
import { decryptJson, encryptJson } from "./crypto";
import { nowIso } from "./ids";
import { brokerTradesToExecutions, untimedFillsReason } from "./broker-trades";
import { insertExecutions, type InsertResult } from "./executions";
import { getImportTimeZone } from "./settings";
import { requireValue } from "./api";
Expand Down Expand Up @@ -52,20 +52,9 @@ export const syncAccount = async (accountId: string): Promise<SyncOutcome> => {
const snapshot = await connection.fetchSnapshot();
const syncedAt = nowIso();

const rows: ImportedExecution[] = snapshot.accounts.flatMap((brokerAccount) =>
brokerAccount.trades.map((trade) => ({
symbol: trade.symbol,
side: trade.side,
quantity: trade.quantity,
price: trade.price,
fee: trade.fee ?? 0,
// The SDK omits unparseable timestamps; a fill with no time can't be
// journaled meaningfully, so it is dropped rather than guessed at.
executedAt: trade.executedAt ?? "",
})),
const { timed, untimed } = brokerTradesToExecutions(
snapshot.accounts.flatMap((brokerAccount) => brokerAccount.trades),
);
const timed = rows.filter((row) => row.executedAt !== "");
const untimed = rows.length - timed.length;

const equity = snapshot.accounts.reduce((total, a) => total + a.equity, 0);
const positions = snapshot.accounts.flatMap((a) => a.positions);
Expand Down Expand Up @@ -100,11 +89,7 @@ export const syncAccount = async (accountId: string): Promise<SyncOutcome> => {
if (untimed > 0) {
result.skipped += untimed;
if (result.skippedReasons.length < 5)
result.skippedReasons.push(
statementTimeZone !== undefined
? `${untimed} fill(s) had no usable timestamp. Check for missing or invalid dates, unsupported timezone suffixes, or ambiguous or nonexistent daylight-saving times.`
: `${untimed} fill(s) had no usable timestamp.`,
);
result.skippedReasons.push(untimedFillsReason(untimed, statementTimeZone !== undefined));
}

return { accountId, ...result, equity, positions: positions.length, syncedAt };
Expand Down
Loading
Loading