Map charge descriptions to crosswalk categories

[bepetersn]

For statutes with multiple IUCR possibilties, make csv mapping between charge descriptions and crosswalk categories. Also to IUCR codes, if an obvious mapping can be made, and eventually to our own categories of interest.

[bepetersn]

@ghing,

It turns out that charges descriptions have more reliable mappings to IUCR categories than ILCS statutes, but they're still not perfect. 15.6% of unique charge descriptions map to multiple IUCR categories.

Under Miscellaneous convictions data I've uploaded two JSON files:

• one containing the mapping of all charge descriptions to the IUCR categories they appear with in our data
• one containing the same mapping, filtered to ones where there are multiple IUCR categories for a given charge description.

More to come...

[ghing]

@bepetersn. I'll take a look at this today. We might just have to make our own call for mapping the ambiguous ones and document it clearly.

[ghing]

@bepetersn I'm a little confused by these. For example, this mapping from chrgdesc_to_category__multiples.json:

    "ATT.FORGERY": [
        "Battery",
        "Burglary"
    ],

I don't see why this description would match to either of these categories. Are the JSON files based on ILCS -> IUCR, or on charge description or a combination? Do you have code that implements your methodology for generating these mappings.

[bepetersn]

See my code at: #10

Several notes:

• I looked at dispositions here, instead of convictions, if that matters. My database is out of date in terms of some the "final" fields you added, and as a result I couldn't figure out how to access these fields from the Conviction model.
• With regard to your question about that example of "ATT.FORGERY" in the charge description field and "Battery" and "Burglary" in the IUCR categories, it is weird. Running the SQL on my database, however, I can confirm that this is what I see.
• Finally, if my belief is correct, this mapping from charge descriptions to IUCR categories doesn't represent any dispositions that couldn't be given an IUCR code, where I think we got the category from. Thus it doesn't yet fulfill our goal of getting more categories from the data. Realizing this now, I need to iterate on this to build a more full list by starting with statutes again.

[ghing]

@bepetersn I'll take a look at #10.

Thanks for clarifying the "ATT. FORGERY" issue. It sounds like there might be some disconnect for some records between the statute and the chargedesc. I'll take a quick peek and let you know what I find.

It shouldn't matter that you looked at dispositions since that's the source for the convictions anyway. It just means that you're looking through more records.

The mappings from charge description to IUCR category won't capture records that didn't have an IUCR code calculated from statute.

I think the first step moving forward would be to start making our own mapping from (final)_chargedesc to IUCR categories based on looking at the values of the charge description. For instance, "ATT. FORGERY" seems like it clearly maps to the "ATTEMPTED FORGERY" IUCR category. I think we could figure out mappings for most descriptions. Does this make sense to you?

[ghing]

@bepetersn, FYI I've uploaded a recent snapshot of the database to drive.

[ghing]

@bepetersn, I took a look at just the "ATT.FORGERY" case. It seems like there might have been some difficulty parsing the statute field to get the IUCR code/category which your management command was using to grab the categories. This makes sense because, at least for these dispositions, it looks like they tried to cram two different statutes into one field. 😿

dispositions = Disposition.objects.filter(final_chrgdesc="ATT.FORGERY")
for d in dispositions:
    print(d.case_number, d.final_statute)

The output is:

2007CR1475501 720-5/8-4(720-5/17-3(A)(1)
2007CR1475501 720-5/8-4(720-5/17-3(A)(1)
2007CR1475501 720-5/8-4(720-5/17-3(A)(1)
2008CR0853801 720-5/8-4(720-5/17-3(A)(2)
2008CR0853801 720-5/8-4(720-5/17-3(A)(2)
2008CR1582701 720-5(8-4)/17-3(A)(2)

720-5/17-3 maps to forgery, so I'm not sure where the Battery and Burglary mappings are coming from.

This makes me think that some of the more weird mappings might be due to parse issues. I wonder if we're better off making our own map of chargedesc to category.

Have you done any more digging into this?