Data access request

[aalexandersson]

What is the name of your project?

Comparing the accuracy and speed of Match*Pro, fastLink, and splink at the Florida Cancer Data System

What is the purpose of your project?

Use the pseudopeople Python package to compare the accuracy and speed of Match*Pro, fastLink, and splink for linkage data requests at the Florida Cancer Data System (FCDS). Medium-to-large linkages at the FCDS means using input datasets with approximately 250,000 * 4,000,000 records, which in terms of size corresponds to the simulated "Rhode Island" pseudopeople population of 1,000,000 people. The FCDS reports to the Florida Department of Health (FDOH).

The FCDS uses Match*Pro for the NAACCR Virtual Pool Registry Cancer Linkage System (VPR-CLS) "Phase 1" linkages which do not use a clerical review. In contrast, "VPR-CLS Phase 2" linkages use a clerical review for more accuracy. The FCDS uses fastLink for regular linkage data requests, including but not limited to "VPR-CLS Phase 2" linkages, because fastLink was more accurate in prior FCDS testing. The FCDS recently used preliminary "Rhode Island"-sized data from Abraham Flaxman which was very helpful to determine that splink is a feasible alternative to fastLink.

Who is involved in the project? Which of these people will have direct access to the pseudopeople input data?

Anders Alexandersson - Senior Research Associate (FCDS) - Direct access (main person to work with the data)
Brad Wohler - Manager of Statistics (FCDS) - Direct access
David Lee - Project Director and Principal Investigator (FCDS) - Direct access
Gary Levin - Deputy Project Director (FCDS) - Direct access
Mark Rudolph - Manager of Computers/Systems Programmer (FCDS) - Direct access (for security reasons)
Heather Lake-Burger - Registries and Surveillance Administrator (FDOH) - NO direct access (will receive the report with findings)

Contact info at https://fcds.med.miami.edu/inc/staff.shtml.

What funding is the project under? What expectations with respect to open access and access to data come with that funding?

The FCDS is funded by FDOH and the Centers for Disease Control and Prevention’s National Program of Cancer Registries (CDC-NPCR). The project is funded by FDOH (Contract CODJU) and CDC through the NPCR (DP003872-04).

The funding does not come with stated expectations with respect to open access and access to data. However, the end goal of the FCDS is to have open access (public) data for fully transparent comparisons of the the accuracy and speed of probabilistic record linkage software using simulated (artificial) data. Therefore, if the project is successful, it is realistic to expect that the FCDS and FDOH would like to 1) share the findings in a report with the Match*Pro, fastLink, and splink developers, and 2) that the report will have some individual level data, for example a listing of the first or last 5 records.

We commit to:

• be responsive to further questions from interested parties
• deprecate and replace our version of the pseudopeople input data when a new version is released

What data would you like to request?

• Full US
• Rhode Island
• Other (may not be available immediately)

Other data - more explanation

The FCDS needs more noise errors to the already provided "noisy" pre-pseudopeople "Rhode Island" level data from Abraham Flaxman in at least two ways:

1. The major limitation with the provided data for the FCDS is that it has no errors in simulated Social Security Number (SSN), only 15% missingness. The FCDS needs to compare partial matches in SSN using the Damerau-Levenshtein string distance in the three software because the FCDS often has incomplete access to SSN data but SSN has, say, 3-5% noise errors such as typos (including transpositions) and fake/wrong use such as the SSN of a family member.

2. The FCDS also needs some noise in date of birth which currently is without noise in the provided "noisy" dataset, which is not realistic.

Update: We are using Python 3.11. Currently, pseudopeople is not compatible with Python 3.11.

[Ironholds]

This looks good to me - @aflaxman ?

[aflaxman]

Agree, thanks for this request. I'll follow up via email with details on how to access the Rhode Island data as well as how to configure noise in the SSN and DOB as you have described. :)

[aflaxman]

I forgot to close this issue when I send @aalexandersson his data access link... closing now.

[aalexandersson]

The report has been completed and is now under review by FDOH. I will present the findings at the 2024 IPDLN conference in September. I expect the report to be public by then. I will attend your @aflaxman workshop at IPDLN and will be happy to share an unofficial copy of the report with you before then, if you would like.

I have an ask: May I share the derived dataframes df1 and df2 privately with the lead developer Robin Linacre of Splink? I upgraded Splink from version 3.9.15 to 4.0.0 but ran in to an issue with Splink that is difficult to reproduce without the dataframes df1 and df2. See Splink discussion issue 2316. I am open to other suggestions, for example, to share the data with you and see if you can reproduce the Splink issue. Please let me know what you prefer.

[aflaxman]

@Ironholds : this is a pretty interesting edge case that we have not run into yet --- it is exactly the sort of use of pseudopeople that I was hoping this simulated data would enable. Maybe a way to think about it is a modification to @aalexandersson 's original proposal, to add Robin Linacre to the list of people with access to pseudopeople data for this project. Do you have any concerns or clarifying questions about sharing df1 and df2?

[aalexandersson]

Could the Splink discussion issue 2316 be related to the pseudopeople requirement of numpy < 2.0? I will try to test this. Are you able to use pseudopeople==1.1.1 and splink==4.0.0 in requirements.txt?

[aflaxman]

I did a quick test of pseudopeople / numpy versions on google's colab, and it seems quite fussy about versions right now! Since @Ironholds has not objected to your request to share the df1 and df2 data, you may proceed with sharing this pseudopeople output to him.

[Ironholds]

I have not because I don't!

[aalexandersson]

Thanks for re-opening the issue @aflaxman . A first report is available as a 2024 monograph report from the Florida cancer registry at fcds.med.miami.edu/inc/publications.shtml; see the tab "Monograph Reports". The title of the report is "Comparing the Linkage Performance of fastLink, Splink, and Match*Pro at the Florida Cancer Registry Using Simulated pseudopeople Data".

We would like to make these changes to the data access request:

1. We request Florida-specific data that are compatible with Python 3.12. We do not need updated data until pseudopeople is compatible with Python 3.12. I learned at IPDLN 2024 that Florida-specific data should be possible to request. If Florida-specific data are not directly available, then it should be possible to extract the Florida data from the full US dataset. We are only interested in the Florida data since we are the Florida cancer registry. The purpose is essentially the same as originally stated. Using Florida-specific data will be more realistic for Florida both in terms of scale and ethnicity. This will enable us to update the analyses. Both Match*Pro and fastLink are expected to have major new releases no later than early 2026. We have already upgraded from Splink 3.x to Splink 4.x. For 2025 and 2026 we expect Splink to provide frequent 4.x releases.
2. Make the derived analysis data publicly available, eventually. This part will not happen in 2025, but hopefully in 2026 or 2027. If the updated analyses are significant, then we will also consider to submit the findings to a suitable journal such as the Journal of Registry Management.
3. Maybe later add a few more people to the project. For example, FCDS or FDOH may want to add a data analyst to the project or you or Match*Pro/fastLink/Splink might be interested in co-authoring, if it comes to that later. We will request specifics as soon as possible regarding possible new project members.

[aflaxman]

Thanks again for your continued interest in pseudopeople! I think this should be easy to approve, but due to competing priorities, it might take a lot longer to make a Florida-specific dataset for you to download than it will to give you the full USA data that you can use with the state='FL' option as a filter (see e.g. here for the call for decennial-style data that includes the state parameter).

When you make your derived analysis data publicly available, would this include pseudopeople output for all of the simulated individuals in Florida? Or is the derived data more about the results of your investigation?

[aalexandersson]

We are not in a hurry -- We will not need these pseudopeople data until second half of 2025. This project will be for 1 July 2025 - 30 June 2026 (FY26), and the results will be shared internally throughout the year. Then, depending on the results, we will determine if and how to proceed with making the data public, if you allow it. I can remind you again about this data access request at the end of the summer 2025, if needed.

At the beginning, the goal is to merely create and use the datasets internally. I plan to create pseudopeople data for about 4 million individuals in one dataframe to simulate Florida cancer registry data (say df1), and for 10,000-500,000 individuals in a few smaller dataframes to simulate typical researchers' "finder files" (say df2a, df2b, and df2c) to be linked against the larger dataframe (that is, df1). We don't need simulated 20+ million people, we only need what is realistic for linkage data requests at the Florida cancer registry.

For later once we have some interesting results worth sharing, likely sometime after July 2026 (FY27), I hope to be able to provide publicly available pseudopeople data for these datasets and possibly related to publication submission. The 2024 FCDS Monograph Supplement created dataframes df1 and df2 which were used for testing the performance of Match*Pro, fastLink, and Splink. Now, the plan is to create and eventually publicly provide similar datasets but for a more realistic Florida cancer registry dataset (df1) and for more realistic finder files (df2a, df2b, and df2c).

I did not plan to provide public data only about the results of our investigation but it too might be worthwhile eventually. I expect that the earlist we will have preliminary results will be for June 2026 in a conference presentation, if accepted, and we will likely not need to share the data already then.

[aflaxman]

Technically, I think it will be easiest for me to make df1 and df2a, df2b, df2c and share those to you. You can come up with configurations that best mimic the noise levels you are most interested it and I can run them.

This is more simulated data than we have shared openly in the past, but perhaps not too much more. @Ironholds , I would be very interested in your thoughts on making 5 million simulated records openly available!