Checksums: Addressing the details

[jamlung-ri]

Examples:

• How we are using checksums to differentiate between two concepts e.g. the comparison tool?
• How do Checksums affect or interact with concepts with mappings?

[jamlung-ri]

@snyaggarwal is putting together something for this - let's aim to discuss on Friday's Arch call

[bmamlin]

@snyaggarwal, I think I need your help in successfully generating checksums outside of OCL. While I was able to successfully generate checksums for a couple simple concepts & mappings, when I tried to work through all of CIEL, I quickly ran into concepts that weren't matching.

For example, I'm unable to successfully generate the 06a4e830339e7aa104bea061b604522d (standard) checksum for the following Anemia concept:

Anemia Concept

{
    "type": "Concept",
    "uuid": "8838472",
    "id": "1",
    "external_id": "1AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
    "concept_class": "Diagnosis",
    "datatype": "N/A",
    "display_name": "Anemia due to blood loss",
    "display_locale": "en",
    "names":
    [
        {
            "uuid": "210431598",
            "name": "Anemia due to blood loss",
            "external_id": "1BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "type": "ConceptName",
            "locale": "en",
            "locale_preferred": true,
            "name_type": "FULLY_SPECIFIED",
            "checksum": "b948c7063be4179b3a2acd2ac5ba97e8"
        },
        {
            "uuid": "210431599",
            "name": "Anemia por pérdida de sangre",
            "external_id": "80825BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "type": "ConceptName",
            "locale": "es",
            "locale_preferred": true,
            "name_type": "FULLY_SPECIFIED",
            "checksum": "48d198a148c14cffe0a0bbb9cbcda88a"
        },
        {
            "uuid": "210431600",
            "name": "Anemia, blood loss",
            "external_id": "87939BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "type": "ConceptName",
            "locale": "en",
            "locale_preferred": false,
            "name_type": null,
            "checksum": "3af4d5d71c7bfa570a35d32cb1426503"
        },
        {
            "uuid": "210431601",
            "name": "Anémie secondaire à une hémorragie",
            "external_id": "106367BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "type": "ConceptName",
            "locale": "fr",
            "locale_preferred": true,
            "name_type": "FULLY_SPECIFIED",
            "checksum": "6fefdbf2ba50a65e0d1abb343ec88869"
        },
        {
            "uuid": "210431602",
            "name": "خون کی کمی، خون بہنے کی وجہ سے",
            "external_id": "139759BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB",
            "type": "ConceptName",
            "locale": "ur",
            "locale_preferred": true,
            "name_type": "FULLY_SPECIFIED",
            "checksum": "b043b5fa4d049d82846bf6e1c2e0a902"
        }
    ],
    "descriptions":
    [
        {
            "uuid": "1522321",
            "description": "Anemia due to bleeding or a hemorrhagic process.",
            "external_id": "1FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
            "type": "ConceptDescription",
            "locale": "en",
            "locale_preferred": false,
            "description_type": null,
            "checksum": "07fb9df8e68e00e49c742b40774dfb0a"
        },
        {
            "uuid": "1522322",
            "description": "خون بہنے کی وجہ سے خون کی کمی",
            "external_id": "18132FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
            "type": "ConceptDescription",
            "locale": "ur",
            "locale_preferred": false,
            "description_type": null,
            "checksum": "d186b390bb5721a253a17ff9df704f68"
        }
    ],
    "extras":
    {},
    "retired": false,
    "source": "CIEL",
    "source_url": "/orgs/CIEL/sources/CIEL/",
    "owner": "CIEL",
    "owner_name": "CIEL",
    "owner_url": "/orgs/CIEL/",
    "version": "8838472",
    "created_on": "2024-01-12T15:51:25.864168Z",
    "updated_on": "2024-01-12T15:51:25.898149Z",
    "version_created_on": "2024-01-12T15:51:25.864168Z",
    "version_created_by": "ocladmin",
    "update_comment": null,
    "is_latest_version": true,
    "locale": null,
    "url": "/orgs/CIEL/sources/CIEL/concepts/1/",
    "owner_type": "Organization",
    "version_url": "/orgs/CIEL/sources/CIEL/concepts/1/8838472/",
    "previous_version_url": "/orgs/CIEL/sources/CIEL/concepts/1/8781768/",
    "parent_concept_urls":
    [],
    "child_concept_urls":
    [],
    "version_updated_on": "2024-01-12T15:51:25.898149Z",
    "version_updated_by": "ocladmin",
    "checksums":
    {
        "smart": "b05def9dd9cb057b83b7be3bbb80ddfd",
        "standard": "06a4e830339e7aa104bea061b604522d"
    }
}

I think the issue isn't checksum algorithm per se; rather, which fields should be included. From reviewing OCL code, I've come up with lists like this:

CHECKSUM_CONCEPT_INCLUSIONS = [
    "concept_class",
    "datatype",
    "description",
    "descriptions",
    "description_type",
    "extras",
    "names",
    "name",
    "name_type",
    "locale",
    "locale_preferred",
    "retired",
]

CHECKSUM_MAPPING_INCLUSIONS = [
    "map_type",
    "from_concept_code",
    "to_concept_code",
    "from_concept_name",
    "to_concept_name",
    "extras",
]

Would it be possible for you to provide a simple python script for calculating checksums (just standard checksums would be fine for now) outside of OCL?