Feature Group by Time

[jeff-hernandez]

Currently, there isn't a direct way to aggregate features by time intervals. For example, given this dataset of transactions, an expected output can be the total amount on each day.

 ID  Amount                       Time
  0     1.5 2020-06-02 19:06:40.012210
  1     5.0 2020-06-02 10:06:40.012241
  2     2.3 2020-06-02 09:06:40.012252
  3     6.0 2020-06-01 18:06:40.012261
  4     9.0 2020-06-01 13:06:40.012271
  5     3.0 2020-06-01 11:06:40.012280

grouper = pd.Grouper(key='Time', freq='1d')
df.groupby(grouper)['Amount'].sum()

Time
2020-06-01    18.0
2020-06-02     8.8
Freq: D, Name: Amount, dtype: float64

[jeff-hernandez]

Time-Based Entity

Perhaps this can be done by adding a method that creates a time-based entity. For example, this entity set has a single entity for transactions.

Entityset: Demo
  Entities:
    Transactions [Rows: 6, Columns: 3]
  Relationships:
    No relationships

es['Transactions'].df

 ID  Amount                       Time
  0     1.5 2020-06-02 19:06:40.012210
  1     5.0 2020-06-02 10:06:40.012241
  2     2.3 2020-06-02 09:06:40.012252
  3     6.0 2020-06-01 18:06:40.012261
  4     9.0 2020-06-01 13:06:40.012271
  5     3.0 2020-06-01 11:06:40.012280

We can then make a new entity for daily transactions based on the transaction time.

es.entity_from_time(
    base_entity_id='Transactions',
    new_entity_id='Daily Transactions',
    index='Time',
    freq='1d',
)

Entityset: Demo
  Entities:
    Transactions [Rows: 6, Columns: 4]
    Daily Transactions [Rows: 2, Columns: 1]
  Relationships:
    Transactions.Time_1d -> Daily Transactions.Time_1d

Now, we can apply DFS on daily transactions.

fm, fd = ft.dfs(
    target_entity='Daily Transactions',
    entityset=es,
    agg_primitives=['sum'],
    ...
)

            SUM(Transactions.Amount)
Time_1d                             
2020-06-01                      18.0
2020-06-02                       8.8

[kmax12]

if we're dealing with just a single table that we are trying to aggregate a different frequencies, I wonder if we should think outside of entity sets or dfs.

we could potentially allow a user to just define a single entity (issue about this #1000), then we could have a new function that takes in that entity, a frequency, and list of aggregation primitives

something like

import featuretools as ft

my_entity = ft.Entity(
    df=df
    index=index,
    time_index=time_index,
    variable_types={...}
)

df, features = ft.aggregate_by_time(
    entity=my_entity,
    frequency='1d'
    agg_primitives=[...]
)

Open questions

• how do you recalculate on new data?

[jeff-hernandez]

It does make sense to consider this at the entity level. On a different note, feature calculations are the same independent of frequency. For example, the total amount is calculated the same way whether it is an hour or a day of data. So, I wonder if we can set the frequency as a property of the entity instead of the feature. Also, if the features are a callable object and a function of the entity that returns the feature matrix, I wonder if this can make recalculating on new data more straightforward.

import featuretools as ft

entity = ft.Entity(df)
features = entity.aggregate(primitives=[...])
daily = features(entity.by(frequency='1d'))
hourly = features(entity.by(frequency='1h'))

# recalculating on new data
entity = ft.Entity(new_df)
daily = features(entity.by(frequency='1d'))
hourly = features(entity.by(frequency='1h'))

[dclong]

IMHO, it makes more sense to work on a single DataFrame (entity).

1. In real situations, the joining condition might be very complicated and come with additional filtering conditions.
2. The DFS API makes users think of multiple things at the same. First, how to join tables. Second, how to generate features. By support better APIs on a single DataFrame (entity), users can get more focused on each step.