Hop Mail Action with OAuth Authentication

[MarioR83]

Faily new to Hop, in the process of setting up a Hop environment/server and migrating PDI jobs and transformations over.

Have a few workflows with Mail actions, currently using basic authentication for SMTP, with MS decommissioning soon i need to start using OAuth2. Not exactly sure where to start and the few references I've have found i don't necessary will work with Hop.

Hoping for some guidance or alternatives. i use the Mail action in varies spot throughout the workflow mostly as notifications.

[dmainou]

Hi,

This is very common
One of 2 possible solutions.

1. Best solution, you are given a service account to reach out to your provider. That goes around the Oauth and substitutes it with something else. e.g. a key, a mac address, etc. Talk to your supplier and security team.
   Here is a link to create an app pasword using google workspace an gmail. Most other providers work the same.
   I've followed this guide and successfully sent emails for personal and business accounts.
   https://support.google.com/accounts/answer/185833?hl=en

2 If your supplier is not playing ball, deploy a en email relay such as posfix on the same server you are running hop. Your mail step 'sends' the email to the relay on the localhost and the relay then does all that magic with the oauth server.

[MarioR83]

thank you for the info, i got some homework to do to see what works for me.

[dmainou]

I just implemented an OAuth yesterday. The token is good for 30 days and the provider does not want me to refresh the token until it's about to expire.

Here is what id did.

• Hpl checks the age of the token if expired or doesn't exist.
  If ok then logs the token and terminates.
  Else it triggers next hpl (create/ refresh)
• HPL checks token age.
  If no token uses client ID, project ID, user name and password to get initial token and refresh token. Else uses client ID and project ID and refresh token to obtain a new token.
  Once we have the token we calculate the expiry (+x seconds) subtract a margin of safety (1d) to allow an evergreen refresh.
  Outputs token to SQLite or a key store.

Token is retrieved from sqlite/ store as needed throughout the solution.

If your token has a shorter expiry you can set the relevant fields as JVM variables and not have a DB or extra service in the middle.

[MarioR83]

I just implemented an OAuth yesterday. The token is good for 30 days and the provider does not want me to refresh the token until it's about to expire.

Here is what id did.

• Hpl checks the age of the token if expired or doesn't exist.
  If ok then logs the token and terminates.
  Else it triggers next hpl (create/ refresh)
• HPL checks token age.
  If no token uses client ID, project ID, user name and password to get initial token and refresh token. Else uses client ID and project ID and refresh token to obtain a new token.
  Once we have the token we calculate the expiry (+x seconds) subtract a margin of safety (1d) to allow an evergreen refresh.
  Outputs token to SQLite or a key store.

Token is retrieved from sqlite/ store as needed throughout the solution.

If your token has a shorter expiry you can set the relevant fields as JVM variables and not have a DB or extra service in the middle.

So, you are doing this within a pipeline? would you mind sharing it, (please remove all sensitive info).

[dmainou]

I'll need to find some time to confirm there is no sensitive material and to get approval to share.
But screenshots are ok.

This is the main workflow which lives either at the beginning of a batch execution or run separately to the logic that needs the token

This pipeline get's the latest token (if any) then either gets the very first ever token or refreshes an existing one. (SQL has been set to always return a line even if the table is empty)

In red first token, green refreshes the token, blue error handling. (this client wants an email. I prefer something that results in observability raising an alarm and a ticket.)

This gets the very first token

The get variables builds the body of the call looks and like this:

ps I am still developing so not all security is in place as of now. It's best if those credentials are encrypted so either use a secrets vault or keep them encrypted in the conf file and decrypt as part of the logic which I'm yet to implement.

The refresh pipeline is quite similar but starts by sourcing the refresh token from the db.

The call body on this occasion looks like this:

After landing the body I replace in string the ##REFRESH_TOKEN## with the value I obtained before.

The above is specific to this particular provider/system but the pattern is common.

You may not want to store that token in a db (it works for this client on this workflow) you may want to create it as a variable of the jvm, encrypt it and save it locally or remotely, update a conf file, or post it to a vault depending on what requirements you have.
Sky is the limit.

[MarioR83]

@dmainou thank you i appreciate this, it's going to take me a minute to digest, like i said I'm fairly new at this. If you do get the "ok" to share the actual files that would be great, but i understand.