Add l1 and l1+l2 regularizers for PyTorch backend

[vl-dud]

L1 regularizer is added. Not quite sure about the line:

l1_loss = torch.sum(torch.stack([torch.sum(p.abs()) for p in trainable_variables]))

Perhaps regularization should not be applied to self.external_trainable_variables? Then it must be like:

l1_loss = torch.sum(torch.stack([torch.sum(p.abs()) for p in self.net.parameters()]))

However, in current l2 regularization trainable_variables (both self.net.parameters() and self.external_trainable_variables) are used.

[lululxvi]

You are right that regularization should not be applied to self.external_trainable_variables. Let us fix this first in this PR.

[vl-dud]

Fixed. Please note that I replaced

raise ValueError("L-BFGS optimizer doesn't support weight_decay > 0")

with

print(f"Warning: weight decay is ignored for {self.opt_name}")

And I removed the exception

raise ValueError("AdamW optimizer requires non-zero weight decay")

because AdamW will still start with the default weight_decay

[lululxvi]

Let us fix l2 in this PR. Use a new PR for adding l1 and l1+l2.

[vl-dud]

Take a look now please.

[lululxvi]

In line 113, emphasize that external_trainable_variables will not apply regularization.

[vl-dud]

In line 113, emphasize that external_trainable_variables will not apply regularization.

Fixed.