Move HIE counts to res model view

[wrridgeway]

In anticipation of removing HIE handling in the res modeling pipeline, we're moving HIE status tracking into dbt. Totally open to changing up the naming I've gone with here.

Row counts for default.vw_card_res_char

select 'new' as source,
	count(*) as count
from z_ci_add_hies_to_shared_model_view_default.vw_card_res_char
union all
select 'old' as source,
	count(*) as count
from default.vw_card_res_char

source	count
old	30656507
new	30656507

Row counts for model.vw_card_res_input

select 'new' as source,
	count(*) as count
from z_ci_add_hies_to_shared_model_view_model.vw_card_res_input
union all
select 'old' as source,
	count(*) as count
from model.vw_card_res_input

source	count
old	30607376
new	30607376

[wrridgeway]

Sorry, was getting different row counts for model.vw_card_res_input and z_ci_add_hies_to_shared_model_view_model.vw_card_res_input, but that was just an issue with z_ci_add_hies_to_shared_model_view_model.vw_pin_shared_input needing to be rebuilt.

[wrridgeway]

We can use mailed_hie from default.vw_pin_value to confirm these start/end dates are inclusive:

select vpv.pin,
	vpv.year,
	vpv.mailed_hie,
	oby.user10,
	oby.user14
from default.vw_pin_value vpv
	left join iasworld.oby on vpv.pin = oby.parid
	and vpv.year = oby.taxyr
	and oby.cur = 'Y'
	and oby.deactivat is null
	and oby.class = '288'
where vpv.pin = '01012130010000'
and vpv.year >= '2019'
order by vpv.year desc

pin	year	mailed_hie	user10	user14
01012130010000	2026
01012130010000	2025	0
01012130010000	2024	2352	2021	2024
01012130010000	2023	2352	2021	2024
01012130010000	2022	2352	2021	2024
01012130010000	2021	2352	2021	2024
01012130010000	2020	0
01012130010000	2019

[wrridgeway]

Mirella confirmed this is the case for addn, Ray confirmed it's the case for oby.

[wrridgeway]

I realize "after the current year" is a little wonky, but it seems like the most straightforward way to build this given the inclusive nature of the start/end years.

[jeancochrane]

This seems great to me! I'm still curious if and how ADDN and OBY record the individual HIE changes, and whether they tie back to specific cards, but I'm now convinced that we don't need to care about that for modeling purposes. We can dig into it further if we decide we ever need that info for analysis.

One thing that's confusing me is that it seems relatively common for a PIN to have fewer cards than it has active or expiring HIEs. Is it possible for a card to have multiple HIEs in a given year?

select meta_pin_num_cards, hie_num_active, hie_num_expiring, count(*)
from z_ci_add_hies_to_shared_model_view_model.vw_card_res_input
where meta_pin_num_cards < hie_num_active
    or meta_pin_num_cards < hie_num_expiring
group by 1, 2, 3

[jeancochrane]

[Thought, non-blocking] You may have noticed this already, but heads up that there are some obviously incorrect dates in the hie_start and hie_end fields between the two tables, which will cause HIE counts greater than 0 for a handful of PINs:

hie_start query

select
    hie_start,
    count(*) as cnt
from (
    SELECT
        taxyr AS year,
        CAST(userval1 AS INT) AS hie_start
    FROM iasworld.addn
    WHERE lline > 0
        AND cur = 'Y'
        AND deactivat IS NULL
        AND userval1 IS NOT NULL
        AND userval2 IS NOT NULL
    union all
    SELECT
        taxyr AS year,
        CAST(user10 AS INT) AS hie_start
    FROM iasworld.oby
    WHERE cur = 'Y'
        AND deactivat IS NULL
        AND user10 IS NOT NULL
        AND user14 IS NOT NULL
        AND class = '288'
) as all_hie
where hie_start not between 2000 and 2026
group by 1
order by 1

hie_start	cnt
204	3
2030	3
2032	20
2033	4
2053	4
2055	3

hie_end query

select
    hie_end,
    count(*) as cnt
from (
    SELECT
        taxyr AS year,
        CAST(userval2 AS INT) AS hie_end
    FROM iasworld.addn
    WHERE lline > 0
        AND cur = 'Y'
        AND deactivat IS NULL
        AND userval1 IS NOT NULL
        AND userval2 IS NOT NULL
    union all
    SELECT
        taxyr AS year,
        CAST(user14 AS INT) AS hie_end
    FROM iasworld.oby
    WHERE cur = 'Y'
        AND deactivat IS NULL
        AND user10 IS NOT NULL
        AND user14 IS NOT NULL
        AND class = '288'
) as all_hie
where hie_end not between 2022 and 2032
group by 1
order by 1

hie_end	cnt
2205	4

There are few enough of these that it feels like it might be possible to get Valuations to fix them. But I wonder if it's worth it, since there are so few? If we think it's worth it, we could pull this out into a separate view and add some data integrity tests on top of it.

[jeancochrane]

[Question, non-blocking] Somehow we're ending up with nulls here. Is that a problem?

select count(*)
from z_ci_add_hies_to_shared_model_view_model.vw_card_res_input
where hie_num_active is null or hie_num_expiring is null

> 185

[wrridgeway]

I noticed this too. There are no nulls in default.vw_card_res_char. It's a consequence of class mismatches between dweldat and pardat. For example pin 19321300500000 is not in dweldat after 2020 because it switched from class 202 to class 100. But it was still class 202 in pardat in 2021. So because vw_pin_shared_input uses class from pardat for its universe but the downstream res model view draws characteristics from dweldat some pins for some years end up with nulls for for more than just the new columns.

[wrridgeway]

Yes, I believe it's possible for single cards to have multiple HIEs in a given year. It's a little weird: Mirella made it clear we don't filter on class in addn so ignore the 288/not 288 distinction here:

Though this begs a different question I'm afraid I don't know the answer to: is every row in addn an HIE? Or can multiple rows be part of one HIE?

Edit: after checking with Mirella, we're defining individual HIEs as different start/end dates within cards.