Visualising pandas dataframes with ipythonblocks

[psychemedia]

Hi

Ever since I first saw ipythonblocks a few weeks ago, I started to wonder about the extent to which it might be useful as a way visualising the structure of pandas dataframes, and the results of applying various transformations to them.

I've started sketching some proof of concept ideas out here: http://nbviewer.ipython.org/gist/psychemedia/9795643

If anyone else is interested in chatting through ideas about how to push pandas DataFrame visualisations using ipythonblocks, or is already further on than my experiments are (or has taken a different approach to the same or related issues), please let me know...

[jiffyclub]

Very neat idea, I like it! I've posted a link to this issue on some social media in case others are thinking about this application.

[sburns]

+1. This would be very useful to demonstrate how df.map, df.apply, etc work. I guess it could extend to joins too, right? That would be a very neat way to teach relational operations.

edit: I feel dumb for not looking at the notebook sooner before raising points you've clearly thought about @psychemedia --great work!

[psychemedia]

@sburns How do you see the apply and map stuff working?

[sburns]

@psychemedia df.apply returns the same or lower dimensional structure, depending on the function. If we think about generalizing the colourNA to something like applyColour...

def pBlock_applyColour(df, apply_function, colour_func, *args, **kwargs):
    applied = df.apply(apply_function)
    (y, x) = applied.shape
    b = BlockGrid(x, y)
    for i in range(x);
        for j in range(y):
            b[j, i] = colour_func(applied.iloc[j, i])
    return b

func = pd.isnull # which we know returns only True or False
colour_func = lambda val: (123, 234, 123) if val else (0, 0, 0)
pBlock_applyColour(df, func, colour_func=colour_func)

Now, func & colour_func are pretty dependent on one another but I think that's ok. Since we pass in an arbitrary function, we're free to color based on ranges, types, whatever you can think of.

On the other hand, this whole approach might be too generalized (& hence complex 👎) because after all ipythonblocks is aimed at learners.

[psychemedia]

@sburns Ah, okay. I think what I was thinking was a set of canned functions that extend pandas in order to support visualisation of dataframe operations. So I'm not thinking about learners learning how to write and manipulate ipythonblocks functions. Instead, I'm wondering about some additional methods that can be called to help learners see what's going on in their pandas code; so I guess it's debugging support for pandas too...

[sburns]

@psychemedia Your approach would ultimately be a lot more sustainable & useful to people of all levels of pandas-ability. I know the pandas devs are actively re-thinking how to display dataframes in many varied contexts.