datetime_key only allows datetime incremental watermarks

[xarasco]

Problem
In tables that use any form of epoch time, a view must be applied in order to convert epoch time into a datetime data type. Other methods of watermarking a table also use integers as well. For example, if a postgres xmin field were carried into Big Query table, it is an effective watermark for table activity.

In the case of epoch time, the only current solution I can think of is to create a view on the bigquery table.

Solution
Abstract datetime_key to allow integer watermarks as well.

[daigotanaka]

I understand this limits the application. Another tap I created (tap-rest-api) supports datetime, timestamp, and index. https://github.com/anelendata/tap-rest-api#timestamp_key-datetime_key-index_key

My initial use case was exactly what you described. I created a view in BQ to be consumed by this tap.

May be I can reuse helper.py from tap-rest-api (I probably clean it up & submodule it together with json2schema.py) Would you take a look at tap-rest-api and let me know if this is something you want?
https://github.com/anelendata/tap-rest-api/blob/eaa073da0c2c9c8cc8cafa6e7f1e5b86657d90da/tap_rest_api/helper.py#L120

[xarasco]

I think that is the general idea of what I'm thinking about. Thanks.

[daigotanaka]

This was removed from v0.3.2 milestone as #13 was critical bug to release immediately.