chef changes time concepts

[semio]

chef changes the day concept while it shouldn't, probably some parsing thing:

https://github.com/open-numbers/ddf--open_numbers--covid_government_response/blob/master/ddf--datapoints--stringency_index--by--geo--time.csv

https://github.com/open-numbers/ddf--oxford--covid_government_response/blob/master/ddf--datapoints--stringency_index--by--country--day.csv

[semio]

here is the thing : time concept was converted to int16

For 4 digits year number it works well but numbers like 20200520 is bigger than the upper bound of int16

[semio]

I didn't use string as dtype here because it's not space efficient and in our datasets the time values are integers. Using categorical is better choice which is more space efficient, but it's not easy to get all categories (i.e. all possible values of the time concept).

But the decision was made before I used dask to load datapoints. Dask solves the data-bigger-than-memory issue, so possibly for now it's ok to just use string for time. Need to test it

[jheeffer]

Time concept is not neccesarily numeric. See the documentation. Week, Quarter and Month contain w, q and - respectively.
Also undocumented in DDF but following ISO8601, when adding time, there would be a t seperator between date and time and . separating seconds from milliseconds

If you can internally parse it to datetime and format it back when outputting that'd be fine too. Might be more memory efficient. But this may be hard, especially in case of time concept, whose format is ambiguous: should midnight 1st of january 2020 be formatted as 20200101 or 2020 or 2020-01? Ambiguity might be solved by using the format the data was in when you parsed it. Might be difficult to trace that?

[semio]

Ok, as we have defined the format for year, month etc, I think it's possible to parse these columns to datetime object internally. For time, we can use string for now, or maybe add some way to specify the format in recipe or datapackage.json

[semio]

After implementing reading time columns as datetime objects, we also need to change the filtering function in ingredient filter and filter procedure, because datetime object is not comparable to int (e.g filter time > 1900 not working). But I found that the performance with eval() and query() (which are used in filter) is very poor

# df is population from WPP
>>> df.shape
(6130902, 5)

>>> df.columns
Index(['age1yearinterval', 'country', 'gender', 'time', 'population'], dtype='object')

>>> %time df.eval("time > '2020'")
CPU times: user 50.4 s, sys: 614 ms, total: 51 s
Wall time: 51 s

>>>  %time df['time'] > '2020'
CPU times: user 7.45 ms, sys: 0 ns, total: 7.45 ms
Wall time: 7.05 ms

# below is performance when using string
>>> %time df['time'] > "2020"
CPU times: user 253 ms, sys: 0 ns, total: 253 ms
Wall time: 252 ms

>>> %time df.eval('time > "2020"')
CPU times: user 255 ms, sys: 23.8 ms, total: 279 ms
Wall time: 276 ms

... which I think is not acceptable. And there will be many changes to make if we don't use df.query(). So I think we for now settle with strings for time concepts because filter should just work.

>>> '2010q4' < '2020q2'
True

>>> '2010w51' < '2020w01'
True

Comments or OK?

[semio]

chef will read time columns as strings now. I also sent a bug report to pandas