JAIL_INTERFACES variable must be manually set if INTERFACE variable is changed from default

[HackInTheBox]

Please include documentation about the JAIL_INTERFACES variable
if you specify an INTERFACE, you must also specify the JAIL_INTERFACES in le-config
Here is a sample config file

JAIL_IP="192.168.4.103"
DEFAULT_GW_IP="192.168.4.1"
INTERFACE="vnet1"
JAIL_INTERFACES="vnet1:bridge0"

It is best to use a unique vnet that is currently not in use by any other jail.
If you are running a plugin currently using vnet0,
Choose a different vnet for the letsencrypt jail.

For example, change INTERFACE=vnet1 then change JAIL_INTERFACES to vnet1:bridge0

Do you need to change bridge0? Probably not. For most use-cases, you will only configure one bridge per physical network interface on your TrueNAS machine. That bridge will include each vnet, and each vnet is paired with an interface the jail uses, which is called an epair.

!!! Failure to correctly configure these interfaces will not only cause your jails to fail, but may cause excessive network traffic on your entire subnet, causing the whole network segment to fail. !!!

Unfortunately, this is not well documented in the FreeBSD docs or anywhere (I can find) on the internet at all. Remember, FreeBSD does not allow DHCP service on more than 1 interface per MACHINE. Keep this in mind when planning configuration for your jails. In my experimentation, your jails count against that quota. While DHCP "may" work for both a physical interface and a vnet simultaneously, you will likely encounter regular connection drops depending on the broadcast traffic in your network.

Also,
For POSIX consistency, please consider mounting config files in the jail's /etc folder
For POSIX consistency, please consider moving/mounting scripts in the jail's /root/bin folder (acme.sh, truenas.sh, etc)

If jail creation fails, /tmp/pkgs.json remains in freenas operating system's /tmp folder
Consider moving this to ${SCRIPTPATH}/pkgs.json

[basilhendroff]

Wow! Thank you for the insight. @danb35 thoughts?

[danb35]

If jail creation fails, /tmp/pkgs.json remains in freenas operating system's /tmp folder

...until the system restarts (or possibly even earlier), at which time it goes away--that's kind of the point of /tmp. I guess we could write something into the script to delete /tmp/pkgs.json on failure before exiting, but I don't think I see any real reason to bother.

But as to the actual subject of this issue, I'm not sure. I understand that if INTERFACE is something other than vnet0, then JAIL_INTERFACES will need to match--that's simple enough to change in the documentation, both in your scripts and mine (which is, I assume, why you tagged me). But it looks like this is suggesting using a different vnet interface for each jail. I don't know that this is a bad thing, necessarily, but I also don't know why you'd want to. I have 12 jails currently running on my 12.0-U2.1 system, and I'm pretty sure they're all using vnet0 (I know the first four are). Three of them are using DHCP, and I haven't noticed any difficulty with that either.

So, on the one hand, using vnet0 for everything seems to work fine for me, and has for quite some time. On the other hand, networking in jails is something I've always treated as a black art, in part because it doesn't seem to be very well documented.

[basilhendroff]

@HackInTheBox Thank you very much for taking the time to comment on the script. I'll begin to accommodate many of your suggestions in the next script release, however, unless you have any further comment, I will temper your suggestions with the feedback from @danb35. I asked Dan to comment as I used his scripts as a template for my own.