Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ All notable changes to this project are documented here. The format follows [Kee

### Fixed

- Webull sync imports trade history ([#28](https://github.com/LuxAlgo/trade-journal/issues/28), proposed by @anthony-som in [#29](https://github.com/LuxAlgo/trade-journal/pull/29)). A fill that closes a position opened before Webull's roughly one year of history is skipped, or kept only for the part the history covers, and reported, instead of showing a position that never existed. An option that expired while held closes at 0 at the 4pm New York expiry, and Webull's option contract sizes are saved as multipliers unless you set one. Later syncs only ask for orders from 30 days before the previous sync onward.
- Broker sync and IBKR Flex XML uploads keep every fill when an order fills in identical pieces (same symbol, side, size, price and time). Previously the later pieces were dropped as duplicates, so volume, fees and P&L were under-counted and a closed position could stay open, as with a closed Hyperliquid short ([#50](https://github.com/LuxAlgo/trade-journal/issues/50), [#52](https://github.com/LuxAlgo/trade-journal/issues/52)). Fills already stored keep their identity; the next sync adds the missing pieces that are still in the broker's history. IBKR sync now skips order, closed-lot and summary rows in the Flex statement, as file uploads already did, so they are no longer stored as extra fills.
- IBKR Flex Query XML statements can be uploaded on Import → File upload, and their times are read in the statement timezone, as broker sync does. Previously an XML upload was not recognized and could not be imported, so the Default import timezone never applied ([#16](https://github.com/LuxAlgo/trade-journal/issues/16)).
- Password protection now verifies the session signature on every API route. Previously, when `JOURNAL_PASSWORD` was set, any request carrying a cookie of the right name was accepted, so a forged cookie could read the journal.
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,11 @@ Default import timezone** for Flex timestamps without an offset. Existing synced
timezone provenance require recovery into a separate account; see
[IBKR timezone recovery](docs/importers.md#ibkr-broker-sync-and-timezone-recovery).

Webull sync reads about a year of order history. A synced fill that closes a position opened before
that is skipped, or kept only for the part the history covers, and the sync reports it, rather than
showing a position that never existed. Options that expired while held close at 0 at the 4pm New
York expiry.

## Screenshots

Captured from the local app with generated demo trades and fictional prop firm records. Expand a view to inspect it at full width.
Expand Down
2 changes: 1 addition & 1 deletion apps/web/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"@ai-sdk/openai-compatible": "3.0.46",
"@dnd-kit/core": "6.3.1",
"@dnd-kit/sortable": "10.0.0",
"@luxalgo/broker-sdk": "0.5.1",
"@luxalgo/broker-sdk": "0.6.0",
"@luxalgo/journal-core": "workspace:*",
"@luxalgo/journal-importers": "workspace:*",
"@luxalgo/vela": "^0.6.8",
Expand Down
153 changes: 153 additions & 0 deletions apps/web/src/server/broker-history.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
/**
* A broker's trade history starts somewhere (Webull keeps about a year), so a
* sync can see a fill that closes a position opened before that start. Booked
* as is, it shows a position that never existed: selling shares bought before
* the start becomes a short. These pure helpers decide how much of each new
* fill the journal keeps, and close option positions that expired while held.
*/

export interface SyncedFill {
symbol: string;
side: "buy" | "sell";
quantity: number;
executedAt: string;
assetClass?: string | null;
/** Broker-stated: whether the fill opened or closed a position. */
positionEffect?: "open" | "close";
/** A separately held position on the same symbol, as round trips group it. */
group?: string;
}

const EPS = 1e-9;
const direction = (fill: SyncedFill): number => (fill.side === "buy" ? 1 : -1);

/**
* How much to keep of each `fresh` fill that reaches back before the start of
* the broker's history, by index into `fresh`; 0 drops the fill. Fills kept
* whole are left out. `stored` are fills already journaled; the caller leaves
* out fresh fills that repeat a stored one, since both feed one position.
*
* - A fill the broker labels as a close, with nothing open on that side, is
* dropped. One that closes more than the history opened (sell 3 against 1)
* keeps what the history covers; the rest closed shares bought before the
* start, at a cost the journal never saw.
* - Unlabeled fills of the `checked` asset class, when the broker's position
* list disagrees with where the replay ends: the earliest fill that opened
* the wrong position is cut back, again and again. One that opened from
* flat is dropped; one that crossed flat keeps only its closing part.
*
* Each position (symbol and group) replays on its own, as round trips do.
*
* ponytail: a real flip the broker's list agrees with in direction (sell 3
* against 2 bought before the start, leaving a short of 1) keeps all 3.
*/
export const keepWithinHistory = (
stored: SyncedFill[],
fresh: SyncedFill[],
brokerQuantity: (symbol: string) => number | undefined,
checked = "equity",
): Map<number, number> => {
const keep = new Map<number, number>();
const positions = new Map<string, { fill: SyncedFill; index: number }[]>();
const add = (fill: SyncedFill, index: number) => {
const key = JSON.stringify([fill.symbol, fill.group ?? null]);
const list = positions.get(key);
if (list) list.push({ fill, index });
else positions.set(key, [{ fill, index }]);
};
stored.forEach((fill) => add(fill, -1));
fresh.forEach(add);

for (const fills of positions.values()) {
fills.sort((a, b) => a.fill.executedAt.localeCompare(b.fill.executedAt));
const replay = () => {
let held = 0;
// Fresh fills that opened a position, and how much of each closed the one before.
const openers: { index: number; closing: number }[] = [];
for (const { fill, index } of fills) {
const quantity = keep.get(index) ?? fill.quantity;
if (quantity < EPS) continue;
const change = direction(fill) * quantity;
if (index >= 0 && fill.positionEffect === "close") {
if (held * change >= -EPS) {
keep.set(index, 0);
continue;
}
if (quantity > Math.abs(held) + EPS) {
keep.set(index, Math.abs(held));
held = 0;
continue;
}
}
const after = held + change;
const flips = Math.abs(after) > EPS && Math.sign(after) !== Math.sign(held);
if (index >= 0 && fill.positionEffect === undefined && (Math.abs(held) < EPS || flips)) {
openers.push({ index, closing: Math.abs(held) });
}
held = after;
}
return { held, openers };
};

let { held, openers } = replay();
const { symbol, group } = fills[0]!.fill;
// The broker's list nets each symbol, so it only speaks for an ungrouped position.
const expected = group === undefined ? brokerQuantity(symbol) : undefined;
if (expected === undefined || !fills.some(({ fill }) => fill.assetClass === checked)) continue;
while (Math.abs(held - expected) > EPS && Math.sign(held) !== Math.sign(expected)) {
const opener = openers.find(({ index }) => direction(fresh[index]!) === Math.sign(held));
if (!opener) break;
keep.set(opener.index, opener.closing);
({ held, openers } = replay());
}
}
for (const [index, quantity] of keep) {
if (quantity > fresh[index]!.quantity - EPS) keep.delete(index);
}
return keep;
};

const OCC_SYMBOL = /^\S+ (\d{2})(\d{2})(\d{2})[CP]\d/;
const newYorkHour = new Intl.DateTimeFormat("en-US", {
timeZone: "America/New_York",
hour: "numeric",
hourCycle: "h23",
});

/** 4pm New York, the US options close, on a `YYMMDD` expiry: 20:00Z in summer, 21:00Z in winter. */
const expiryClose = (yy: string, mm: string, dd: string): string => {
const at = (utcHour: number) =>
new Date(Date.UTC(2000 + Number(yy), Number(mm) - 1, Number(dd), utcHour));
return (newYorkHour.format(at(20)) === "16" ? at(20) : at(21)).toISOString();
};

/**
* Closing fills for option contracts still held after they expired. Webull
* records no fill when an option expires worthless, so the contract would
* otherwise stay open forever and its loss never count. Each closes at 0 at
* the 4pm New York expiry; that fixed time gives it the same hash every sync.
*
* ponytail: a contract exercised or assigned in the money also closes at 0.
* Telling the two apart needs the underlying's price at expiry.
*/
export const expiredOptionCloses = (optionFills: SyncedFill[], now: string): SyncedFill[] => {
const held = new Map<string, number>();
for (const fill of optionFills) {
held.set(fill.symbol, (held.get(fill.symbol) ?? 0) + direction(fill) * fill.quantity);
}
const closes: SyncedFill[] = [];
for (const [symbol, quantity] of held) {
const expiry = OCC_SYMBOL.exec(symbol);
if (!expiry || Math.abs(quantity) < EPS) continue;
const executedAt = expiryClose(expiry[1]!, expiry[2]!, expiry[3]!);
if (executedAt >= now) continue;
closes.push({
symbol,
side: quantity > 0 ? "sell" : "buy",
quantity: Math.abs(quantity),
executedAt,
assetClass: "option",
});
}
return closes;
};
40 changes: 27 additions & 13 deletions apps/web/src/server/broker-trades.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,33 @@ const numberIdenticalFills = (rows: ImportedExecution[]): ImportedExecution[] =>
*/
export const brokerTradesToExecutions = (
trades: Trade[],
): { timed: ImportedExecution[]; untimed: number } => {
const rows: ImportedExecution[] = trades.map((trade) => ({
symbol: trade.symbol,
side: trade.side,
quantity: trade.quantity,
price: trade.price,
fee: trade.fee ?? 0,
// The SDK omits unparseable timestamps; a fill with no time can't be
// journaled meaningfully, so it is dropped rather than guessed at.
executedAt: trade.executedAt ?? "",
}));
const timed = rows.filter((row) => row.executedAt !== "");
return { timed: numberIdenticalFills(timed), untimed: rows.length - timed.length };
): {
timed: ImportedExecution[];
untimed: number;
/** Whether each timed row opened or closed a position, where the broker says so. */
positionEffects: Map<ImportedExecution, "open" | "close">;
} => {
// The SDK omits unparseable timestamps; a fill with no time can't be
// journaled meaningfully, so it is dropped rather than guessed at.
const timedTrades = trades.filter((trade) => trade.executedAt);
const timed = numberIdenticalFills(
timedTrades.map((trade) => ({
symbol: trade.symbol,
side: trade.side,
quantity: trade.quantity,
price: trade.price,
fee: trade.fee ?? 0,
executedAt: trade.executedAt!,
// The SDK's "cash" has no journal equivalent.
...(trade.assetClass && trade.assetClass !== "cash" ? { assetClass: trade.assetClass } : {}),
})),
);
const positionEffects = new Map<ImportedExecution, "open" | "close">();
timed.forEach((row, index) => {
const effect = timedTrades[index]!.positionEffect;
if (effect) positionEffects.set(row, effect);
});
return { timed, untimed: trades.length - timedTrades.length, positionEffects };
};

export const untimedFillsReason = (untimed: number, statementTimeZone: boolean): string =>
Expand Down
5 changes: 5 additions & 0 deletions apps/web/src/server/executions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ export const executionProblem = (row: unknown, source: ExecutionSource): string
(source === "manual" || !Number.isSafeInteger(r.occurrence) || r.occurrence < 1)
)
return `${label}: invalid copy number for an identical fill.`;
if (
r.hashQuantity !== undefined &&
(source !== "sync" || !isFiniteNumber(r.hashQuantity) || r.hashQuantity <= r.quantity)
)
return `${label}: invalid reported quantity for a fill kept in part.`;
const meta = r.importMetadata;
const metaOk =
!meta ||
Expand Down
6 changes: 4 additions & 2 deletions apps/web/src/server/ids.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@ export const nowIso = (): string => new Date().toISOString();
/**
* Content hash for execution dedup: the same fill inserts once per account.
* Numbered copies of an identical fill hash apart; the first copy keeps the
* plain hash, so fills stored before numbering existed still deduplicate.
* plain hash, so fills stored before numbering existed still deduplicate. A
* fill kept only in part hashes with the quantity the broker reported.
*/
export const executionHash = (execution: {
symbol: string;
side: string;
quantity: number;
hashQuantity?: number;
price: number;
executedAt: string;
importMetadata?: { id: string; group?: string };
Expand All @@ -23,7 +25,7 @@ export const executionHash = (execution: {
[
execution.symbol,
execution.side,
execution.quantity.toPrecision(12),
(execution.hashQuantity ?? execution.quantity).toPrecision(12),
execution.price.toPrecision(12),
execution.executedAt,
...(execution.importMetadata
Expand Down
Loading
Loading