modin issue

[Shine226]

1. Run Wiggum in a regular terminal rather than vs studio since error occurs in vs studio.
2. Still need to import pandas for pandas.core
3. New error

  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum_app/controller.py", line 273, in main
    labeled_df_setup.get_subgroup_trends_1lev(trend_list)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/detectors.py", line 239, in get_subgroup_trends_1lev
    groupby_vars = self.get_vars_per_role('splitby')
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/labeled_dataframe.py", line 490, in get_vars_per_role
    return list(all_vars[is_target_role & drop_ignore])
  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4307, in __getitem__
    result = getitem(key)
IndexError: too many indices for array

[cegme]

@Shine226 Can you try and repoduce the error without using the all_vars index? Just use all_vars.

[Shine226]

First error:
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum_app/controller.py", line 273, in main
    labeled_df_setup.get_subgroup_trends_1lev(trend_list)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/detectors.py", line 259, in get_subgroup_trends_1lev
    cur_trend.get_trend_vars(self)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/trend_components/base_getvars.py", line 181, in get_trend_vars
    ['ordinal','continuous'],['ordinal','continuous'])
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/trend_components/base_getvars.py", line 102, in set_weights_regression
    indep_vars = labeled_df.get_vars_per_roletype('independent', i_type)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/labeled_dataframe.py", line 528, in get_vars_per_roletype
    drop_ignore)]
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/labeled_dataframe.py", line 527, in <listcomp>
    target_rows = [r & t & d for r,t,d in zip(is_target_role,is_target_type,
TypeError: unsupported operand type(s) for &: 'str' and 'str'

[brownsarahm]

that error says that the wrong type got passed into the variables in that zip. those should all be boolean so that the & works.

[Shine226]

Before changing to modin:

variable
sepal length     True
sepal width      True
petal length    False
petal width     False
class           False
dtype: bool
variable
sepal length     True
sepal width      True
petal length     True
petal width      True
class           False
dtype: bool
variable
sepal length    True
sepal width     True
petal length    True
petal width     True
class           True
dtype: bool
<class 'pandas.core.series.Series'>
<class 'pandas.core.series.Series'>
<class 'pandas.core.series.Series'>

After applying Modin:

              __reduced__
variable                 
sepal length         True
sepal width         False
petal length         True
petal width          True
class               False
              __reduced__
variable                 
sepal length         True
sepal width          True
petal length         True
petal width          True
class               False
              __reduced__
variable                 
sepal length         True
sepal width          True
petal length         True
petal width          True
class                True
<class 'modin.pandas.dataframe.DataFrame'>
<class 'modin.pandas.dataframe.DataFrame'>
<class 'modin.pandas.dataframe.DataFrame'>

[Shine226]

We can use .squeeze() to convert back to series.

[Shine226]

New error:

UserWarning: `Series.align` defaulting to pandas implementation.
UserWarning: Distributing <class 'pandas.core.series.Series'> object. This may take some time.
UserWarning: Distributing <class 'list'> object. This may take some time.
127.0.0.1 - - [03/Jun/2021 15:47:44] "POST / HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum_app/controller.py", line 274, in main
    labeled_df_setup.get_subgroup_trends_1lev(trend_list)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/detectors.py", line 274, in get_subgroup_trends_1lev
    agg_trends = cur_trend.get_trends(self.df,'agg_trend')
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/trend_components/statistical.py", line 171, in get_trends
    groupby_name = groupby_name_by_type[type(data_df)](data_df)
KeyError: <class 'modin.pandas.dataframe.DataFrame'>

[Shine226]

The groupby_name_by_type is using pandas.core.frame.Dataframe:

groupby_name_by_type = {pandas.core.groupby.DataFrameGroupBy:lambda df: df.keys,
                                pandas.core.frame.DataFrame:lambda df: None}

[Shine226]

Change code to:

import modin.pandas as pd
groupby_name_by_type = {pd.groupby.DataFrameGroupBy:lambda df: df.keys,
                                pd.dataframe.DataFrame:lambda df: None}

new error:

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum_app/controller.py", line 274, in main
    labeled_df_setup.get_subgroup_trends_1lev(trend_list)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/detectors.py", line 284, in get_subgroup_trends_1lev
    curgroup_trend_df = cur_trend.get_trends(cur_grouping,'subgroup_trend')
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/trend_components/statistical.py", line 173, in get_trends
    groupby_name = groupby_name_by_type[type(data_df)](data_df)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/trend_components/statistical.py", line 6, in <lambda>
    groupby_name_by_type = {pd.groupby.DataFrameGroupBy:lambda df: df.keys,
  File "/opt/anaconda3/lib/python3.7/site-packages/modin/pandas/groupby.py", line 125, in __getattr__
    raise e
  File "/opt/anaconda3/lib/python3.7/site-packages/modin/pandas/groupby.py", line 121, in __getattr__
    return object.__getattribute__(self, key)
AttributeError: 'DataFrameGroupBy' object has no attribute 'keys'

[Shine226]

Changing keys to _idx_name will solve no attribute 'keys', but not good for extracting a compound key such as ['sex', 'race']:

groupby_name_by_type = {pd.groupby.DataFrameGroupBy:lambda df: df._idx_name,
                                pd.dataframe.DataFrame:lambda df: None}

[Shine226]

New error for corr function:

127.0.0.1 - - [09/Jun/2021 11:29:20] "POST / HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/anaconda3/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum_app/controller.py", line 273, in main
    labeled_df_setup.get_subgroup_trends_1lev(trend_list)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/detectors.py", line 284, in get_subgroup_trends_1lev
    curgroup_trend_df = cur_trend.get_trends(cur_grouping,'subgroup_trend')
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/trend_components/statistical.py", line 179, in get_trends
    corr_data = self.compute_correlation_table(data_df,trend_col_name)
  File "/Users/chenguangxu/Documents/GitHub/detect_simpsons_paradox_dev/wiggum/trend_components/statistical.py", line 88, in compute_correlation_table
    corr_mat = data_df[corr_var_list].corr(method=self.corrtype)
  File "/opt/anaconda3/lib/python3.7/site-packages/modin/pandas/groupby.py", line 622, in corr
    return self._default_to_pandas(lambda df: df.corr(**kwargs))
  File "/opt/anaconda3/lib/python3.7/site-packages/modin/pandas/groupby.py", line 980, in _default_to_pandas
    return self._df._default_to_pandas(groupby_on_multiple_columns, *args, **kwargs)
  File "/opt/anaconda3/lib/python3.7/site-packages/modin/pandas/base.py", line 400, in _default_to_pandas
    result = op(pandas_obj, *args, **kwargs)
  File "/opt/anaconda3/lib/python3.7/site-packages/modin/pandas/groupby.py", line 974, in groupby_on_multiple_columns
    by=by, axis=self._axis, squeeze=self._squeeze, **self._kwargs
  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py", line 6727, in groupby
    dropna=dropna,
  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 568, in __init__
    dropna=self.dropna,
  File "/opt/anaconda3/lib/python3.7/site-packages/pandas/core/groupby/grouper.py", line 811, in get_grouper
    raise KeyError(gpr)
KeyError: 'class'

[Shine226]

Set index for DataFrame before groupby:

  # Modin issue: set index before groupby for corr() in get_trends
  self.df.index = self.df[groupbyAttr]
  
  #condition the data
  cur_grouping = self.df.groupby(groupbyAttr)