Improvement: IBFlex stricter security matching

[georgemac-labs]

Note: depends on #5292

When importing from IBKR, I have had some issues with transactions being matched to the wrong securities.

The importer currently starts with CONID, which is good, as it’s closest to universal and unambiguous. However, if that fails, it falls back to ISIN, and then symbol, which are ambiguous identifiers. The following scenarios are considered a match:

• Same ISIN, even if there’s a currency mismatch (this looks like a different trading line of the security e.g. EQQQ.F vs EQQQ.L)
• Same symbol, even if there’s an ISIN mismatch (this looks like a completely different instrument!)
• Same symbol, even if there’s a currency mismatch (could be either of the above)

This PR tightens the rules as follows:

• If an ISIN is stored for a security, it must match the import transaction. If it doesn’t, it’s not a match, and a new security should be created.
• If the transaction is a trade, currency must match the stored security. If it doesn’t, it’s not a match, and a new security should be created. Caveat: this rule does not apply for dividends, as those are not always paid in the listing currency.
• The currency checks build on previous PRs¹ and recognise that the currency does match even if there is a mix of major and minor units (important because e.g. IB Flex reports can use GBP values for GBX listed securities)

Note: I amended the existing unit test, because it contained a trade and a dividend payment where the symbols matched but the CONIDs did not. I traced the origin of the test data and it seems the dividend payment is copied from a different security: #812 (comment), so it doesn't tell us anything about real IBKR data. I only have my personal data to go on, but it's almost 9000 records, and I have zero cases where a CashTransaction has a different CONID from the associated Trade. Thus: I'm working with the assumption that they should always match.

These rules make sense to me, but I may have overlooked some facts and – as always – I’m happy to hear feedback.

Closes: #4700

Footnotes

1. Improvement: IBFlex handle currency unit mismatches #5291
   Improvement: IBFlex support cross-currency dividend import #5292 ↩

[buchen]

let's wait for the other pull request to conclude - this one is independent but sits on top of the other pull request.

The checks wether CONID or ISIN conflict if the ticker is present make sense to me.

[buchen]

What is the rationale between separating the instrument matching whether it is a trade (purchase, sale) or a dividend transactions?

Wouldn't we get mismatches? Say a purchase creates a new security (strict match) but the corresponding dividend than is matched to the previous instrument (because it is not strictly matching).

[georgemac-labs]

Trades are treated differently because the currency should always match. If it doesn't, you can be sure that should be a different security in PP.

With dividends, you don't know, because they are sometimes not paid in the listing currency. Currency mismatch is not evidence for creating a security.

And yes, you can have a mismatch. Real-world example: you have EQQQ.F (EUR) and EQQQ.L (GBP) and both pay a dividend in USD. You don't have the CONIDs stored in the WKD field. Both have the same ISIN. Currency doesn't match, but that proves nothing. Where do you assign the dividend? Currently impossible to say.

You could try to solve this using the CashTransaction's listingExchange attribute, but I think that would be more complexity, a pain to maintain (mapping table with exchange codes) and still imperfect – and all that for an edge-case.

I ignored this case, because it's not easy to get into this situation. You need to:

• Own the same instrument in multiple currencies AND
• Receive a third-currency dividend AND
• Not have the CONIDs saved

... although the joke is – that's exactly what happened to me.

Really, if you're doing non-trivial stuff with IBKR, you want CONIDs saved for all securities. And I think the IBFlex importer is getting more powerful and accurate, so in the future that should be the norm. I only ended up with securities without CONIDs due to importer issues.

[buchen]

Fair. I am wondering if the user will understand the subtle difference between matching trades and dividends. I would think it is easier if it is one algorithm - and if it mixes it up, fix the meta data once and for all.

And if a new instrument is created, the user could still choose to apply them to an existing instrument:

[georgemac-labs]

Coming back to this now the parent commit has been merged...

I don't really agree on this point because I don't think the user is thinking in depth about the logic. And I think this approach will produce the right result more often than the alternative. However, it's not a hill I would die on, and it makes the code is more complex. Don't mind dropping it.

[georgemac-labs]

Here's an alternative proposal, please let me know if you agree and I will implement.

• Match on:
  • conid (no currency check — dividends can be in a non-base currency)
    else
  • ISIN (disambiguate by currency if multiple match)
    else
  • symbol (disambiguate by currency if multiple match)
• If the chosen identifier matches more than one security after currency disambiguation, use the first match and emit a warning that there is ambiguity
• Same rules for all transaction types — no dividend/trade distinction

[buchen]

I get that point (and I understand this has nothing to do with the strict mode). I am just wondering if we take the use of the FixedEchangeRateProvider a little to far. The user could still have a DEM instrument, but I am not sure if he wants to use that instead of the creating a new EUR instrument. Maybe GBX and GBP is a special case because apparently it used interchangeably.

[georgemac-labs]

I will need to look at this, but I'd say the data modelling is fundamentally a bit off. GBP and GBX are not two currencies – they are two units of the same, like Euros and Euro cents. Their relationship is quite different from DEM and EUR. But I didn't even consider opening that can of worms!

[georgemac-labs]

... I would say this has been clarified in the meantime? Solution for GBP/GBX et al in IBFlex is in 8319a86

[Nirus2000]

Closing this PR due to inactivity.
The improvement — stricter security matching rules for IBFlex imports — is well-motivated and addresses real-world mismatches (ISIN with currency mismatch, symbol with ISIN conflict). @buchen acknowledged the CONID/ISIN conflict checks as sensible.
However, the PR stalled for two reasons:

Open design question: @buchen suggested preferring a single unified matching algorithm over separate logic for trades vs. dividends, to avoid confusing users. This was never addressed.
Dependency on #5292: The PR sits on top of that one and was explicitly put on hold until it concluded. It is unclear whether #5292 has since been merged and whether this PR was ever rebased.

If someone wants to revive this, please open a new pull request that:

Rebases cleanly on the current master
Addresses the unified-algorithm concern raised by @buchen
Revisits the dividend vs. trade matching distinction with a clear rationale or simplification

Thank you to @georgemac-labs for the thorough analysis of the matching edge cases and for taking the time to contribute. The problem is real, the approach is sound — it just needs someone to pick it up and carry it over the finish line.