Resolving Notice IDs appearing in real notices can be complicated

[csnyulas]

This issue is created to highlight various ways notice ID declarations or references can appear in real notices, which make the creation of unique and reliable notice URI generation, especially during the XML-to-RDF conversion process, very difficult.

Examples of GOOD data:

        <cac:NoticeDocumentReference>
            <cbc:ID schemeName="ojs-notice-id">436567-2023</cbc:ID>
        </cac:NoticeDocumentReference>

OR

       <cac:NoticeDocumentReference>
            <cbc:ID schemeName="notice-id-ref">ae6eb65a-99a4-4e63-81db-9333e19b72cf-01</cbc:ID>
        </cac:NoticeDocumentReference>

Examples of INCOMPLETE data:

      <cac:NoticeDocumentReference>
         <cbc:ID>615112-2023</cbc:ID>
      </cac:NoticeDocumentReference>		

      <cac:NoticeDocumentReference>
        <cbc:ID>0d0045ae-878e-42e7-a2ba-e2b14b6453da-01</cbc:ID>
      </cac:NoticeDocumentReference>

      <cac:NoticeDocumentReference>            
	     <cbc:ID schemeName="">00746048-2023</cbc:ID>	  
      </cac:NoticeDocumentReference>		

Examples of BAD data:

      <cac:NoticeDocumentReference>
         <cbc:ID schemeName="ojs-notice-id">af86b9de-d912-4262-93f7-2f68d8a8234f-01</cbc:ID>
      </cac:NoticeDocumentReference>

OR

        <cac:NoticeDocumentReference>
		    <cbc:ID schemeName="notice-id-ref">640293-2023</cbc:ID>
        </cac:NoticeDocumentReference>

OR

        <cac:NoticeDocumentReference>
            <cbc:ID schemeName="Lot">2c10eb51-0c92-4151-8e19-e8fa14d88759-01</cbc:ID>
        </cac:NoticeDocumentReference>

Also, it is worth mentioning that the analysis of 723K+ notices revealed that more than 25K notices did not have a notice ID specified, even though some of these had the scheme specified, but with no id value.

[cristianvasquez]

Hello @csnyulas thank you for the analysis!

Indeed, the identifiers come in different schemes/formats, therefore, a normalisation is required.

From the ted docs one can learn that:

Source XML make use of two distinct identifier formats to refer to other linked notices:

1. A ojs-notice-id format, such as 739743-2023, for referencing notices.
2. A notice-identifier format, such as 98083d34-7771-4485-8ab4-9adc73cdb892

But as you pointed out, the data many times declares an incorrect schemeName, which is a data problem.

Can you provide more context on where something like this appears?
Are the 25K notices without notice ID, Change notices?

I believe the best candidate for such normalization is to use the TED API, where you can check the ojs-notice-id and notice-id-ref of all previous notices in the same procedure