Add cdb schema V2

[salix5]

Problem

There are many problems in the current schema.

rule_code

See Fluorohydride/ygopro-core#823

scale

Now the scale is packed into level column.
Packing multiple unrelated properties into the same column is considered a poor practice in database design.

• Violates database normalization
  Normalization is a design principle that separates data into logical tables and columns to reduce redundancy and improve clarity.
  Packing multiple attributes into one column breaks this rule, and it makes the schema less readable, harder to understand, and more error-prone.

• Harder to query
  Queries are harder to write and maintain.
  You must remember which bit means what.
  Even simple queries become more complex, requiring constant use of bitwise operations.

• Poor indexing support
  SQLite indexes individual columns, but not individual bits inside an integer.
  If you pack 4 values into one INTEGER column, it cannot index each flag efficiently.
  Queries on individual bits won’t benefit from indexes, resulting in full table scans.
  It can hurt performance, especially when the table grows.

• Difficult to maintain
  With individual columns, adding or changing values is straightforward.
  But when values are packed into a single field, even basic read or write operations require bitwise manipulation.
  For example:
  What is the level of a card with a level column of 50528266?
  If the level needs to be changed to 11, what is the new value of level?

• Minimal space savings in SQLite
  SQLite is already very efficient at storing integers.
  It uses variable-length integers internally, which take only as many bytes as needed.
  The space saving from packing bits is negligible.

setcode

There are some cards with more than 4 setcode entries:
8512558 Utopic Onomatopoeia
55088578 Onomatokage

They are handled by table lookup now, but it is not a good approach.

Solution

Require:
Fluorohydride/ygopro-core#802

ocgcore: 802

Add the following columns to the database:
datas.rule_code,
datas.scale,
datas.setcode2,
datas.setcode3,
datas.setcode4,

Action Required for Third-Party Tool Maintainers

I suggest that third-party applications that rely on implicit SQL syntax update their queries.

Not recommended (will crash):

SELECT * FROM datas WHERE id = 10000;

INSERT INTO datas VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);

Correct (future-proof):

SELECT datas.id, datas.ot, datas.alias, datas.setcode, datas.type, datas.atk, datas.def, datas.level, datas.race, datas.attribute, datas.category
FROM datas
WHERE id = 10000;

INSERT INTO datas (id, ot, alias, setcode, type, atk, def, level, race, attribute, category) 
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);

@mercury233 @purerosefallen @Wind2009-Louse @fallenstardust

[Copilot]

Pull request overview

This PR introduces a “schema V2” evolution for the card database and updates the client DB loader to consume the new columns while attempting to remain compatible with older databases.

Changes:

• Extends the datas table with scale and additional setcode storage (setcode2..4) and migrates existing packed scale / special setcode data.
• Updates DataManager::ReadDB to read either the legacy SELECT shape or the new V2 SELECT shape.
• Removes the legacy “extra_setcode” patch-up path for V2 databases (kept only for legacy DBs).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
resource/migrate.sql	Adds schema migration steps for scale and setcode2..4 and commits within a transaction.
gframe/data_manager.cpp	Adds V2 SELECT + parsing and runtime selection between legacy/V2 DB layouts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[salix5]

Migration Plan

Database with schema V1

The current databases still work after this PR.
I suggest that run the following command to add the new columns.

Migrate to schema v2

The sqlite commands for adding new columns are in migrate.sql.
All of these columns have a default value 0.
To apply the schema change to existing .cdb files, run the command file in SQLite CLI:

sqlite3 cards.cdb < migrate.sql

[Wind2009-Louse]

I think we should make a todo list for current ecosystem, like:

• YGOPro
• YGOPro2
• MDPro3
• DataEditorX/DataEditorY
• Windbot
• and more...

[mercury233]

what about link arrows in defense column, and more categories

[purerosefallen]

I think we should make a todo list for current ecosystem, like:

• YGOPro
• YGOPro2
• MDPro3
• DataEditorX/DataEditorY
• Windbot
• and more...

I would say there are too much. Not this little, including some MyCard internal systems

[purerosefallen]

and I suggest to have a 3rd table to store the schema version.
no table means v1

[purerosefallen]

I think we should make a todo list for current ecosystem, like:

• YGOPro
• YGOPro2
• MDPro3
• DataEditorX/DataEditorY
• Windbot
• and more...

https://github.com/purerosefallen/ygopro-cdb-encode

in fact this one, and other js/ts projects reading cdb should migrate to this project

[mercury233]

I think we should make a todo list for current ecosystem, like:

• YGOPro
• YGOPro2
• MDPro3
• DataEditorX/DataEditorY
• Windbot
• and more...

https://github.com/purerosefallen/ygopro-cdb-encode

in fact this one, and other js/ts projects reading cdb should migrate to this project

Please stop shifting your responsibility to others.

[salix5]

what about link arrows in defense column, and more categories

Link markers
It is mutual exclusive with def.
Keeping it in the same column is somewhat acceptable.

Scale should be put in another column because it is not mutual exclusive with level.

Category
It can be treated as uint64 if necessary.

[purerosefallen]

and I suggest to have a 3rd table to store the schema version. no table means v1

I think a version control marker in cdb format is necessary, just as how docker-compose.yml's version works.

In case we have v3, we may route to correct SQL to parse CardData

In addition, I suggest we make a driver layer, something like class CdbReaderLegacy class CdbReaderV2

[salix5]

https://sqlite.org/pragma.html#pragma_user_version
PRAGMA schema.user_version;
PRAGMA schema.user_version = integer ;

The user_version pragma will get or set the value of the user-version integer at offset 60 in the database header. The user-version is an integer that is available to applications to use however they want. SQLite makes no use of the user-version itself.

How about storing pragma version in user_version?

[purerosefallen]

nope we need a specific table.
what if other application offers the same user_version value?
And I doubt if every language / lib (Python, sql.js) supports this field.

[purerosefallen]

and again, unless necessary, we should keep using general SQL thing rather than SQLite-exclusive stmt

[purerosefallen]

and some ORMs like TypeORM, SQLModel does not support schema version in header. it's would make things built with those libs hard to maintain.

I don't agree on using SQLite user_version

[salix5]

@mercury233 @purerosefallen @Wind2009-Louse
The schema version should be put in:

• datas
• a new table (table name?)

[purerosefallen]

@mercury233 @purerosefallen @Wind2009-Louse The schema version should be put in:

• datas
• a new table (table name?)

a new table
maybe ygopro_schema

[purerosefallen]

Converting this to draft because it's considered half-done for lack of version table while a version table plan was introduced

[mercury233]

I prefer feature detection over versioning in database.

[purerosefallen]

I prefer feature detection over versioning in database.

Could you explain more? I think versioning could be a way.

[purerosefallen]

I think this PR can be merged.
However, it could be a long time until official cdb sources like https://github.com/mycard/ygopro-database could switch to v2.

During that time, we may plan to migrate those internal systems, but it could take months to do so.

[mercury233]

Suggested change

	/sqlite3/*.c
	/sqlite3/*.h
	/sqlite3
	!/sqlite3/migrate.sql

btw I don't think it is a good idea to put that sql file here

[salix5]

Where should I put this file ?

[mercury233]

better in wiki

[DailyShana]

I suggest this is an opportunity to resolve the hardcoding of cards with double names like CARD_MARINE_DOLPHIN.

[purerosefallen]

I suggest this is an opportunity to resolve the hardcoding of cards with double names like CARD_MARINE_DOLPHIN.

any specific plans?
something like another_code?

[DailyShana]

some possible choice:

1. set a bit in rule_code
   card A (id=1) with rule name B (id=2) : rule_code=2
   card A (id=1) with another name B (id=2) : rule_code=2+(1<<32)

2. add a new another_code

3. SQLite JSON (a little over-design)

CREATE TABLE datas (
    id INTEGER PRIMARY KEY,
    rule_code TEXT
);

rule_code="{"override":2, "apend":[3,4]}"

maybe this is suitable for setcodes
setcode="[0x1,0x2,0x3,0x4,0x5]"

[purerosefallen]

some possible choice:

1. set a bit in rule_code
   card A (id=1) with rule name B (id=2) : rule_code=2
   card A (id=1) with another name B (id=2) : rule_code=2+(1<<32)
2. add a new another_code
3. SQLite JSON (a little over-design)

CREATE TABLE datas (
    id INTEGER PRIMARY KEY,
    rule_code TEXT
);

rule_code="{"override":2, "apend":[3,4]}"

maybe this is suitable for setcodes setcode="[0x1,0x2,0x3,0x4,0x5]"

I would go with 2
@salix5 planning to do?

[salix5]

ygopro is known for abusing bitwise logic.
I think adding a column is better.