Ibis map dtype

[mark-druffel]

Is your feature request related to a problem? Please describe.
I totally screw this up initially so correcting it, sorry for all the confusion @deepyaman!

Ibis dtype map is not implemented, not sure exactly what the proper syntax would be but here's what I was thinking:

import ibis
import pandera.ibis as pa
import pandas as pd

data = pd.DataFrame({
    "id": ["01", "02", "03"],
    "dict_col": [{
        1: "value_1A",
        2: "value_1B",
        3: "value_1C"
    },
    {
        1: "value_2A",
        2: "value_2B",
    },
    None]
})
df = ibis.memtable(data)

class ValidateSchema(pa.DataFrameModel):
    id: str = pa.Field(nullable=False)
    dict_col: ibis.dtype("map<string, string>") = pa.Field(nullable=True)
ValidateSchema.validate(data)

> TypeError: Data type 'object' not understood by Engine.

Describe the solution you'd like
Implement Ibis Map in ibis_engine.py.

Describe alternatives you've considered
I don't think there's a workaround other than using the pyspark dataframe underneath ibis for validations which is a separate can of worms for us.

Additional context
Add any other context or screenshots about the feature request here.

[deepyaman]

@mark-druffel Do you mind fixing the example?

1. The dict key is getting reused/clobbered.
2. Are you looking to have an array of strings or structs?

[mark-druffel]

Hey @deepyaman, thanks for the response! Adding @jakepenzak, he helped me work through your second question (hopefully!).

1. Sorry, I screwed that up. Fixed the example above.
2. I think I misunderstood ibis array vs. map, I fixed my initial ask but here's some additional context / thought

We get json columns from a few systems. I think there are basically two scenarios:
a. We know the incoming json schema and want to enforce
b. We don't know the incoming schema or don't want to enforce for whatever reason

Scenario a, I know I am getting a dictionary of key value pairs and I know it's int, string (for example):

class ValidateSchema(pa.DataFrameModel):
    id: str = pa.Field(nullable=False)
    dict_col: ibis.type("map<int64, string>") ) = pa.Field(nullable=True)
ValidateSchema.validate(data)

Scenario b, I know I am getting a dictionary of key value pairs, but I don't know or want to enforce the underlying types (not sure if leaving map empty is valid for ibis but I'm assuming so?):

class ValidateSchema(pa.DataFrameModel):
    id: str = pa.Field(nullable=False)
    dict_col: ibis.dtype("map") = pa.Field(nullable=True)
ValidateSchema.validate(data)

Sorry for all the confusion, I'm still trying to learn the ibis type system and I'm honestly still pretty naive with Python's so I get myself a bit confused 😵

Just let me know if this makes sense now and thanks a million for all your help!