Large mailbox - archiving possible?

[Softbox44]

I am getgin fairly large mail folders with some cur over 50GB ,mainly due to received emails with file attachments of 10-30MB. Is there a clever way to archive large emails? Or alternativelly an automated way to extract the attachments?

[Softbox44]

any ideas??

[polarathene]

Look through existing issues/discussions and you should find some advice on using various tools to manage backup archives.

If you want to filter specifically for specific conditions, you can possibly to so with Dovecot Sieve script or similar.

This isn't a common use-case for DMS so you'll need to figure this out yourself or wait until someone from the community potentially chimes in with better advice.

[casperklein]

This can be done with the find command:

find ./cur -type f -size +50M  # find files larger than 50mb
find ./cur -type f -mtime +30  # find files older than 30 days

# Move mails older than 30 days to /my-archive
find ./cur -type f -mtime +30 -exec mv {} /my-archive/ \;

[Softbox44]

Thanks. Great search terms. Thank you. Is there a way to extract attachments. The main issue is with large attachments I receive where I'd like to keep the email but remove the attachment from the mail folder.
Any tips?

[polarathene]

You would use a cron script in the container to run that find command I think, which can be paired with -exec mv to run a command for each large file found (similar to the example @casperklein provided you above that does it for -mtime +30 instead).

If you prefer systemd timers instead, or a cron job on the host, you could have the script with find command mounted in the DMS container, and have your scheduled task run docker exec dms /my-script-name.sh.

Just have it mv the attachment to a volume mount to persist the attachment.

Make sure you have a backup copy of your data to try that on, so that you are able to restore or access it at a later date should you need it.

[polarathene]

Actually it seems Dovecot has support for storing attachments separately:

On dovecot this exists since 2010 is easily configurable for example like this:

mail_home = /mailstore/local/%d/%n
mail_location = mdbox:/mailstore/local/%d/%n:ALT=/mailstore/remote/%d/%n
mail_attachment_fs = sis posix
mail_attachment_dir = /mailstore/attachments
mail_attachment_min_size = 128k

sis apparently refers to SIS (Single Instance attachment Storage).

I don't have the time to look into this right now, but I know DMS manages mail_home + mail_location, we probably don't touch the mail_attachment* settings in that quote.

You can see there's a threshold size for when the attachment is split off to attachment storage dir. So those two settings are straightforward, while mail_attachment_fs seems to already default to the value sis posix but in Dovecot 2.4 the setting name has changed and SIS is deprecated (thus only posix FS driver` supported, dedupe functionality may require FS capability to handle it instead):

• Dovecot 2.3 (which DMS uses as of DMS v15): mail_attachment_fs
• Dovecot 2.4 (DMS does not support this upgrade yet, many breaking changes): mail_ext_attachment

Likewise the equivalent settings between these versions for mail_attachment_dir / mail_ext_attachment_path have the same docs line mentioned:

With sdbox and mdbox, mail attachments can be saved to external files, which also allows single-instance storage of them.

DMS however uses maildir:

docker-mailserver/target/dovecot/10-mail.conf

Lines 30 to 31 in c9aac24

	mail_home = /var/mail/%d/%n/home/
	mail_location = maildir:/var/mail/%d/%n

DMS presently does support changing that to a different format sdbox or mdbox (both categorized as dbox under Dovecot docs) via the DOVECOT_MAILBOX_FORMAT ENV

docker-mailserver/target/scripts/startup/setup.d/dovecot.sh

Lines 59 to 72 in c9aac24

	# set mail_location according to mailbox format
	case "${DOVECOT_MAILBOX_FORMAT}" in
	( 'sdbox' | 'mdbox' )
	_log 'trace' "Dovecot ${DOVECOT_MAILBOX_FORMAT} format configured"
	sedfile -i -E "s|^(mail_home =).*|\1 /var/mail/%d/%n|" /etc/dovecot/conf.d/10-mail.conf
	sedfile -i -E \
	"s|^(mail_location =).*|\1 ${DOVECOT_MAILBOX_FORMAT}:/var/mail/%d/%n|" \
	/etc/dovecot/conf.d/10-mail.conf
	_log 'trace' 'Enabling cron job for dbox purge'
	mv /etc/cron.d/dovecot-purge.disabled /etc/cron.d/dovecot-purge
	chmod 644 /etc/cron.d/dovecot-purge
	;;

It seems that may be required for the attachment feature as the Dovecot docs suggest that maildir is not supported? (EDIT: That actually seems to be about a separate ALT storage path support in mail_location)

Migrating your existing mail to a different storage format would be another task for you to suss out, although I think this has been covered in other discussions should you need to tackle that too.

---

Here is how to get this functionality in DMS with the following (for Dovecot 2.3), I've verified it to work:

services:
  dms:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest # :15.0
    hostname: mail.example.internal
    environment:
      # Updates `mail_location` with `sdbox` or `mbox` (default is `maildir`):
      # NOTE: Support for alternative mailbox formats is not well tested in DMS
      DOVECOT_MAILBOX_FORMAT: mdbox
    volumes:
      # Use whatever you like for paths here, so long as the target path in the
      # container matches the path for the `dovecot.cf` override setting `mail_attachment_dir`
      #
      # NOTE: Dovecot expects ownership (5000:5000 default in DMS)
      # Might differ for LDAP if not corrected there, otherwise should be configurable via `DMS_VMAIL_UID`:
      # https://docker-mailserver.github.io/docker-mailserver/latest/config/environment/#dms_vmail_uid
      - ./dovecot-attachments:/srv/dovecot/attachments
    configs:
      - source: dms-accounts
        target: /tmp/docker-mailserver/postfix-accounts.cf
      - source: dovecot-override
        target: /tmp/docker-mailserver/dovecot.cf

# Remove this if you're providing your own DMS config volume for
# `/tmp/docker-mailserver` with equivalent files.
configs:
  dms-accounts:
    content: |
      john.doe@example.internal|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

  dovecot-override:
    content: |
      # Add support for storing mail attachments at a different location:
      mail_attachment_fs = posix
      mail_attachment_dir = /srv/dovecot/attachments
      mail_attachment_min_size = 128k

# Start DMS and shell into it:
$ docker compose up -d --force-recreate
$ docker compose exec -it dms bash

# Create a sparse file of 1MB size to test as an attachment:
$ truncate -s 1MB /tmp/example.dat

# Send a mail with that file attached:
$ swaks --server localhost \
  --from alice.doe@example.internal --to john.doe@example.internal \
  --header "Subject: This is an attachment example" \
  --attach-name example.dat --attach @/tmp/example.dat

# Attachment was split off into expected location:
$ du -shx /srv/dovecot/attachments
1016K   /srv/dovecot/attachments

$ du -shx /var/mail
64K     /var/mail

[polarathene]

I don't know for sure if you can leverage this approach, but presumably you can backup the content of /srv/dovecot/attachments with either find command @casperklein showed you (size and/or time preferences to filter by) to archive those files somewhere.

Make sure you verify that you can restore access to the attachments. They'll need to be restored to the same ownership at least as in the container if that wasn't persisted. If you need to persist it, it should be possible with tar archives (by default root user will untar with archived ownership retained, but non-root users convert to the non-root user as ownership IIRC, which would then need to be corrected for Dovecot in DMS to be happy).