Bring current with changes in Glue

[dacort]

I left and returned to AWS. Since that time, Glue has:

• Added support for defining table names with crawlers
• Added support for schema evolution
• Upgraded to Python 3 (updates for python3 support in glue #16)
• Added support for incremental crawler runs

I want to research what impact these changes have and if they should be incorporated. Initial thoughts are that:

• Some of the manual table creation I'm doing could be replaced with crawlers
• Python3 upgrade is a must

In addition, I need to go through the backlog. :)

[CalvinLeather]

As a band-aid, opened this PR while waiting on python 3 upgrade. Happy to help contribute to / test python 3 upgrade, our team is using tool actively.

[dacort]

Thanks @CalvinLeather ! Much appreciated. I hope to be able to devote some time to this in the coming weeks.

[dacort]

Adding a note here about blueprints - they could be useful for building more comprehensive Glue deployments for this project, specifically workflows which could schedule the jobs.

[dacort]

Looked into Blueprints a little bit yesterday. Looks like they could successfully be used to bootstrap Classifiers, Crawlers, and table definitions. In addition a schedule/trigger can also be set up so they could be a good end-to-end direction to go.

We can still make use of this library – all of the table definitions and classifiers are still necessary, as are some of the source-specific transforms.

We could essentially parameterize the whole thing:

• Create a single AGSL Blueprint
• A Workflow per log type can be created from that Blueprint
• Based on the log type selected, the necessary crawlers and tables get created

[brandon-fryslie]

Is there any documentation on how to achieve what this library achieves with AWS Glue directly?