E-Mail notifications in FreeNAS using Gmail

Setting up email alerts is a smart way of making sure nothing unexpected happens on a FreeNAS server without knowing about it or having to check the UI every now and then. A "should be straigthforward" way ough to be setting up a gmal account as the outgoing email. Nevertheless, it is not as simple as clicking "yeah, sure, email me".

Objective

Get a working E-Mail notification from FreeNAS by using a Gmail E-Mail address for sending the messages.

Overview

FreeNAS needs an smtp server and corresponding E-Mail account credentials to send E-Mails. E-Mails cannot be sent directly from within FreeNAS without an E-Mail server. Additionally, using a Gmail address with the corresponding account password does not work either.  The steps to get such a setup running are briefly as follows,

1. Get a Gmail E-Mail address,
2. Allow "App password" access to the E-Mail account, and
   
3. Set up FreeNAS E-Mail service.
   

Settings in the Google account

This is the first step in configuring sending E-Mails through a gmail account. Basically, a new App passsword is required for FreeNAS. Connecting and sending emails via the username and password merely did not work in my case, not even when enabling "Allow less secureapps". So to properly set up access the following needs to be done,

1. Log in to the Google account.
2. Go to Security.
   
   
   
   
   
3. Enable 2 Factor Authentication (this is required for step 4!).
   
   
   
   
   
4. Once 2 Factor Authentication is enabled, an option becomes avilable to create App passwords.
   
   
   
5. Create a new App password and copy it to the clipboard.
   

Once this is done, the settings from Gmail side are complete.

Settings in FreeNAS

1. Go to System/Email and adjust the settings according to the details below.
   
   
   
   
   From E-Mail: ChooseItToYourLiking
   Outgoing Mail Server: smtp.gmail.com
   Mail Server Port: 465
   Security: SSL (Implicit TLS)
   SMTP Authentication: Ticked
   Username: example@gmail.com
   Password: YourAppPassword
   

Once this is done, a test E-Mail can be sent by clicking "SEND EMAIL". If all above steps were followed, a test notification wil arrive in the E-Mail box associated with the root user (!). Following this, an alert service can be set up as well. Without too much details,

1. Go to System/Alert Services.
2. Click "Add" and choose Type/Email.
3. Fill out your email address and click "SEND TEST ALERT".

This should be working now as well and the type of alerts cn be configured based on user needs.

FreeNAS home server backups Done Right

There are plenty of guides explaining how to back up FreeNAS - now TrueNAS - servers. Personally, I found them either vague in what they actually back up and what will be the final outcome or they get way too complicated for people of home NAS servers who want something that "just works". So here it is how I am periodically backing up my FreeNAS server to USB external drives.

Expected Result

 

Inspect the following example layout of a Source_Pool and an empty Destination_Pool. I will adhere to this naming in the rest of the post. Also, make sure you understand every command before following the steps. Working with root access is always risky and data can be lost, previous backups mistakenly overwritten.

Source_Pool        ==>This is the main pool we want to backup.

    ---Dataset1        ==> This is a dataset within the pool.

    ---Dataset2

        ---Sub-dataset

    ---Dataset3

Destination_Pool                 ==>This is the destination pool we want to backup TO.
    ---Backup_Source_Pool     ==> This is the top-level dataset that will be created of the source pool.
        ---Dataset1
        ---Dataset2
            ---Sub-dataset
        ---Dataset3

Note how the original datasets of the Source_Pool all appear under Destination_Pool/Backup_Source_Pool. This means that another pool (say Source_Pool#2, not shown in the example above) may also be backed up to the Destination_Pool, e.g. to a dataset called Destination_Pool/Backup_Source_Pool#2. This would then also contain the full dataset and child dataset layout of this Source_Pool#2. Provided that the Destination_Pool has enough capacity, several pool dataset may be backed up to for example, a single USB drive.

Considerations

 

• Data deliberately deleted over time from Source_Pool should also be deleted on the Destination_Pool inside the backups. I do not want to hoard data that I deleted for a good reason.
• The snapshot used for backing up data must not be deleted until a newer backup was made using another, newer snapshot. In other words, always keep the latest snapshot of the system. Deleting all snapshots will require a full backup of the entire pool.
• Permissions and ACLs are retained.
  

If in any doubt, check man zfs.

Overview

The general idea is to create an initial snapshot of the Source_Pool and use zfs send | zfs receive to send it over to the Destination_Pool into an existing dataset named Backup_Source_Pool. Once the initial backup is done, future backups can be done as incremental, which requires two snapshots to exist, where the backup will transfer only the changes between the two snapshots. In simple terms this means that at the very least two snapshots must exist on the Source_Pool to use incremental backups.

1. Identify source and destination datasets where backups should be made from and to.
2. Create a snapshot of the source dataset.
3. Send and receive dataset stream using zfs send | zfs receive to destination pool.
4. For future backups, send incremental backups using the same method.

Initial Backup

1. Create an initial snapshot of Source_Pool. I prefer doing this through the UI using a recursive snapshot. (Recursive means that all Datasets within Source_Pool will also be snapshotted, otherwise only data directly in the main Dataset directory will be snapshotted! Short: if you want the entire Pool, use recursive.)
   Note: I highly recommend using and sticking to a naming style for snapshots. I am using,
   

   Source_Pool@BACKUP-20200814

   and will adhere to this.
2. Create a dataset called Backup_Source_Pool under the Destination_Pool. I prefer doing this from the UI as well.
   
3. Use ssh to log in to the FreeNAS server and verify the snapshot is there,
   

   zfs list -t snapshot

   if thre are too many snapshots, search for the correct one, e.g.
   

   zfs list -t snapshot | grep @BACKUP-20200814

4. Assuming both Source_Pool and Destination_Pool are present and mounted in the system, proceed by making the initial backup. Note: this requires root access, so
   

   sudo -i

   (sudo zfs send ... Does not work!) 
   

   zfs send -Rv Source_Pool@BACKUP-20200814 | zfs receive -Fdu Destination_Pool/Backup_Source_Pool

A bit of explanation from man zfs,

R --  Generate a replication stream package, which will replicate the specified filesystem, and all descendent file systems, up to the named snapshot. When received, all properties, snapshots, descendent file systems, and clones are preserved.

v --  Print verbose information about the stream package generated.

F -- Force a rollback of the filesystem to the most recent snapshot before performing the receive operation.

d -- Use the name of the sent snapshot to determine the name of the new snapshot as described in the paragraph above. See man zfs for more accurate info.

u --  Newly created file system is not mounted. (Translation: when inspecting from the UI, the copied dataset will be visible, however, going over ssh manually to /mnt/Backup_Source_Pool will show up empty. No need for panic, the data is there, its is merely not mounted. I usually cannot mount a single directory using zfs mount Backup_Pool - due to I suspect some shares being mounted or similar - but zfs mount -a works just fine and the data "shows up".

 

Incremental Backup

 

This will only work if,

1. you already have an initial backup, and
2. you still have the snapshot of that backup (including the snapshots of all Sub-datasets if it was a recursive snapshot). 

Otherwise you will get an error. Unfortunately, this means that if the first Snapshot is no longer available, an incremental backup cannot be made and a new, initial backup has to be made, transferring all the data again. Otherwise, since only the change between the latest and previous snapshot is transferred, the backup speed is greatly increased.

Note that the following code will transfer only the differences between two Snapshots and data that was removed from Source_Pool will also be deleted on Destination_Pool during the backup. To make an incremental backup, the -i option is used followed by the old and new snapshot names after one another.

1. Create a new recursive snapshot, this will be,
   Source_Pool@BACKUP-20200815
2. Use ssh to log in to the FreeNAS server and check the Snapshots available
   

   zfs list -t snapshot | grep Source_Pool@BACKUP

   Hopefully there will be,
   Source_Pool@BACKUP-20200814
   Source_Pool@BACKUP-20200815
   

3. Assuming both Source_Pool and Destination_Pool are present and mounted in the system, proceed by making the incremental backup. Note: this requires root access, so
   

   sudo -i

   (sudo zfs send ... Does not work!)
   

   zfs send -Rv -i Source_Pool@BACKUP-20200814 Source_Pool@BACKUP-20200815 | zfs receive -Fdu Destination_Pool/Backup_Source_Pool

Note that here the -i argument is added which stands for "incremental backup", man zfs. This requires two distinct snapshots to exists on the Source_Pool and the older of the two snapshots to exist on the Destination_Pool.

i -- Generate an incremental stream from snapshot1 to snapshot2. The incremental source snapshot1 can be specified as the last component of the snapshot name (for example, the part after the "@"), and it is assumed to be from the same file system as snapshot2.

Similarly to the initial backup, the Destination_Pool is unmounted when done.

 

Snapshot keeping strategy

Since for the incremental backups at least two Snapshots are needed, I always keep a minimum of two recursive snapshots of my system. This is a balance between storage space and ability to create backups or roll back to previous snapshots.

Technically between backups one can just keep a single Snapshot on the Source_Pool. Then, when backup day arrives, make a new Snapshot, do an incremental backup between the two Snapshots - the older of which exists on the Destination_Pool - and thendelete again the older snapshot of the two from Source_Pool.

In other words, for incremental backups to work, a Snapshot must not be deleted until another backup was made since that snapshot.

Reference

 

Not really necessary perhaps, but for simplicity here are some excerpts from man zfs relating to zfs send and zfs receive.

zfs send [-DvRp] [-[iI] snapshot] snapshot

Creates a stream representation of the second snapshot, which is written to standard output. The output can be redirected to a file or to a different system (for example, using ssh(1). By default, a full stream is generated.

-D

Perform dedup processing on the stream. Deduplicated streams cannot be received on systems that do not support the stream deduplication feature.

-i snapshot

Generate an incremental stream from the first snapshot to the second snapshot. The incremental source (the first snapshot) can be specified as the last component of the snapshot name (for example, the part after the @), and it is assumed to be from the same file system as the second snapshot.

If the destination is a clone, the source may be the origin snapshot, which must be fully specified (for example, pool/fs@origin, not just @origin).

-I snapshot

Generate a stream package that sends all intermediary snapshots from the first snapshot to the second snapshot. For example, -I @a fs@d is similar to -i @a fs@b; -i @b fs@c; -i @c fs@d. The incremental source snapshot may be specified as with the -i option.

-R

Generate a replication stream package, which will replicate the specified filesystem, and all descendent file systems, up to the named snapshot. When received, all properties, snapshots, descendent file systems, and clones are preserved.

If the -i or -I flags are used in conjunction with the -R flag, an incremental replication stream is generated. The current values of properties, and current snapshot and file system names are set when the stream is received. If the -F flag is specified when this stream is received, snapshots and file systems that do not exist on the sending side are destroyed.

-p

Send properties.

-v

Print verbose information about the stream package generated.

The format of the stream is committed. You will be able to receive your streams on future versions of ZFS. 

 

zfs receive [-vnFu] filesystem|volume|snapshot

zfs receive [-vnFu] [-d | -e] filesystem

Creates a snapshot whose contents are as specified in the stream provided on standard input. If a full stream is received, then a new file system is created as well. Streams are created using the zfs send subcommand, which by default creates a full stream. zfs recv can be used as an alias for zfs receive.

If an incremental stream is received, then the destination file system must already exist, and its most recent snapshot must match the incremental stream's source. For zvols, the destination device link is destroyed and recreated, which means the zvol cannot be accessed during the receive operation.

When a snapshot replication package stream that is generated by using the zfs send -R command is received, any snapshots that do not exist on the sending location are destroyed by using the zfs destroy -d command.

The name of the snapshot (and file system, if a full stream is received) that this subcommand creates depends on the argument type and the -d or -e option.

If the argument is a snapshot name, the specified snapshot is created. If the argument is a file system or volume name, a snapshot with the same name as the sent snapshot is created within the specified filesystem or volume. If the -d or -e option is specified, the snapshot name is determined by appending the sent snapshot's name to the specified filesystem. If the -d option is specified, all but the pool name of the sent snapshot path is appended (for example, b/c@1 appended from sent snapshot a/b/c@1), and if the -e option is specified, only the tail of the sent snapshot path is appended (for example, c@1 appended from sent snapshot a/b/c@1). In the case of -d, any file systems needed to replicate the path of the sent snapshot are created within the specified file system.

-d

Use all but the first element of the sent snapshot path (all but the pool name) to determine the name of the new snapshot as described in the paragraph above.

-e

Use the last element of the sent snapshot path to determine the name of the new snapshot as described in the paragraph above.

-u

File system that is associated with the received stream is not mounted.

-v

Print verbose information about the stream and the time required to perform the receive operation.

-n

Do not actually receive the stream. This can be useful in conjunction with the -v option to verify the name the receive operation would use.

-F

Force a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication stream (for example, one generated by zfs send -R -[iI]), destroy snapshots and file systems that do not exist on the sending side.