Add simple end-to-end example of DP training of a transformer

[ryan112358]

We need a realistic example usage of the library that is simple (ideally 1 file) and easily forkable so that people can easily modify the training loop for research purposes. It should use the Jax privacy core library components rather than the higher level keras_api, the latter is primary for a managed training loop experience, and customization is harder / requires modifications to the Jax privacy core library.

[Neerajpathak07]

@ryan112358 LMK if this example description fits right for the file name:- examples/dp_sgd_transformer_example.py

"""Example of training a Transformer on text using JAX Privacy core API.

This example demonstrates how to train a simple Transformer decoder
on character-level language modeling using differentially private stochastic
gradient descent (DP-SGD) with the JAX Privacy core library components. It shows how to:
"""

Was thinking of utilizing core components like, optax, clipped_grad & noise_addition.
Will also be referencing:- https://github.com/google-deepmind/jax_privacy/blob/main/examples/jax_api_example.py

[RamSaw]

Hi folks!

Thank you very much for active contributions!

As there are already 3 pending PRs implementing the same thing, we will review and merge the first one from Neerajpathak07. moksha-hub I will close yours, sorry. debanganghosh08 see #120, I propose to transform your PR to Flax NNX as it is the current standard.

Re such race conditions: please take a look, we updated our contribution guide, comment on issue that you want to take it, then we will assign it to you.

[debanganghosh08]

Hi folks!

Thank you very much for active contributions!

As there are already 3 pending PRs implementing the same thing, we will review and merge the first one from Neerajpathak07. moksha-hub I will close yours, sorry. debanganghosh08 see #120, I propose to transform your PR to Flax NNX as it is the current standard.

Re such race conditions: please take a look, we updated our contribution guide, comment on issue that you want to take it, then we will assign it to you.

Thanks for the update, @RamSaw ! I look forward to contributing to #120 !