11/20/2023
read 7 minutes

How To Configure Backup To Object Storage Using Restic

/upload/iblock/bba/flqb21rblqju6n1srhi606qyvi1u6owm/3d_abstract_techno_background_with_connecting_lines_dots_opt.jpeg

Contents of Articles:

  1. Introduction
  2. Installing Restic And Connecting To Storage
  3. Example Of Working With Storage via Rustic
    1. Backup
    2. Restoring A Backup
    3. Old Copy Deletion Policy
    4. Automation
  4. Conclusion

Introduction

Many administrators set up backups while using the Rsync Utility in 2007. It was necessary to allocate a separate server for storage. And one of the frequent problems was the costs increasing due to reserving a server specifically for backups. Devices that were backed up were located next to the backup host. This violated the 3-2-1 rule, which recommends having backups in at least three locations, two of which are on different media and one offsite. You need to have hosts in different data centers to build a truly reliable system.

Today, you can contact 3HCloud to use cloud data storage services. Below we will tell you how to use the Restic utility. That is a backup system that provides tools for storing backups in a versioned repository. Restic also allows you to define rules for including and excluding files and directories in the backup.

Installing Restic And Connecting To Storage

We will use the Ubuntu 22.04 distribution with Restic installation from the repository in this example.

  1. Connect to your server and install the necessary dependencies.

    $ sudo apt update
    $ sudo apt install -y restic

  2. Update Restic and check if the required version is installed.

    restic self-update

    $ restic version
    restic 0.15.2 compiled with go1.20.3 on linux/amd64

  3. Export the environment variables and initialize the repository using 3HCloud S3 as a backend.

    $ export AWS_ACCESS_KEY_ID=<container_username>
    $ export AWS_SECRET_ACCESS_KEY=<container_password>

    $ restic -r s3:s3.dfw.3hcloud.com/<container_name> init

Set the password for the repository after entering the command and displaying the message in the terminal.

Example Of Working With Storage via Rustic

It’s necessary to know how to work with a storage object while using the utility.

Backup

Let's create a directory with a file that we will copy. If you already have a directory with data for a future backup, you can skip this step.

$ mkdir dir
$ echo line1 > dir/file1.txt
$ cat dir/file1.txt
line1

Let's start creating a backup in dry-run mode to make sure that everything will work correctly. You will need to enter the password from the repository that you set earlier during the execution of the command.

restic -r s3:s3.dfw.3hcloud.com/<container_name> backup -vv --dry-run <directory>

The terminal will output a message similar to the following:

repository e4218f76 opened (version 1)
created new cache in /root/.cache/restic
lock repository
no parent snapshot found, will read all files
load index files
start scan on [/root/dir]
start backup on [/root/dir]
scan finished in 0.243s: 1 files, 6 B
new       /root/dir/file1.txt, saved in 0.007s (6 B added)
new       /root/dir/, saved in 0.008s (0 B added, 0 B stored, 425 B metadata)
new       /root/, saved in 0.008s (0 B added, 0 B stored, 419 B metadata)

Files:           1 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Data Blobs:      1 new
Tree Blobs:      3 new
Would add to the repository: 1.034 KiB (1.304 KiB stored)

processed 1 files, 6 B in 0:00

Now you can remove the –dry-run key and make the first backup.

restic -r s3:s3.dfw.3hcloud.com/<container_name> backup <directory> repository e4218f76 opened (version 1)
no parent snapshot found, will read all files

Files:           1 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repository: 1.034 KiB (1.304 KiB stored)

processed 1 files, 6 B in 0:00
snapshot a7d06fa4 saved

You can view the list of backups (snapshots) while using the special snapshots command. In order not to enter a password every time, you can write it to a file and use the -p key.

$ echo 'pass' > backup_password
$ restic -r s3:s3.dfw.3hcloud.com/<container_name> -p backup_password snapshots
enter password for repository: 
repository e4218f76 opened (version 1)
ID        Time                 Host        Tags        Paths
----------------------------------------------------------------
a7d06fa4  2023-06-14 16:49:09  www                     /root/dir
----------------------------------------------------------------
1 snapshots

Now let's make changes to the directory, repeat the backup process, and see how it looks.

$ echo line2 >> dir/file1.txt 
$ echo line1 >> dir/file2.txt 
$ ls -l dir/
total 8
-rw-r--r-- 1 root root 12 Jun 14 16:52 file1.txt
-rw-r--r-- 1 root root  6 Jun 14 16:52 file2.txt
$ restic -r s3:s3.dfw.3hcloud.com/<container_name> -p backup_password backup ~/dir repository e4218f76 opened (version 1)
using parent snapshot a7d06fa4

Files:           1 new,     1 changed,     0 unmodified
Dirs:            0 new,     2 changed,     0 unmodified
Added to the repository: 1.377 KiB (1.646 KiB stored)

processed 2 files, 18 B in 0:00
snapshot e0f0801e saved
$ restic -r s3:s3.dfw.3hcloud.com/restic_backup -p backup_password snapshots repository e4218f76 opened (version 1)
ID        Time                 Host        Tags        Paths
----------------------------------------------------------------
a7d06fa4  2023-06-14 16:49:09  www                     /root/dir
e0f0801e  2023-06-14 16:53:02  www                     /root/dir
----------------------------------------------------------------
2 snapshots

After backing up the modified directory, a new copy appeared in the object storage.

Restoring A Backup

Restoring backups using Restic looks like this:

restic -r s3:s3.dfw.3hcloud.com/<container_name> restore <id_snapshot> --target <directory>

Old Copy Deletion Policy

To delete old backups, you can use the forget command. You can read more about it in the manual to configure the system in compliance with your demands.

Let's consider an example where we need to store daily backups for the last week, weekly backups for the last month, monthly backups for the last year, and yearly backups for the last 75 years. In this case, arguments of the following format can be passed:

forget --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 75y

Let's consider an example of using the forget command:

$ restic -r s3:s3.dfw.3hcloud.com/restic_backup -p backup_password forget --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 75y repository e4218f76 opened (version 1)
Applying Policy: keep daily snapshots within 7d, weekly snapshots within 1m, monthly snapshots within 1y, yearly snapshots within 75y
keep 1 snapshots:
ID        Time                 Host      Tags      Reasons            Paths
-------------------------------------------------------------------------------
e0f0801e  2023-06-14 16:53:02  www                 daily within 7d    /root/dir
                                                   weekly within 1m
                                                   monthly within 1y
                                                   yearly within 75y
-------------------------------------------------------------------------------
1 snapshots

remove 1 snapshots:
ID        Time                 Host      Tags      Paths
------------------------------------------------------------
a7d06fa4  2023-06-14 16:49:09  www                 /root/dir
------------------------------------------------------------
1 snapshots

[0:00] 100.00%  1 / 1 files deleted

Automation

Combining commands for backup creation and deletion (rotation) allows for process automation. For instance, by placing the following task in cron:

export AWS_ACCESS_KEY_ID=<container_username>
export AWS_SECRET_ACCESS_KEY=<container_password>
restic -r s3:s3.dfw.3hcloud.com/<container_name> -p backup_password backup <directory> 2>&1 | systemd-cat -t restic-cron-backup

Logging via systemctl-cat enables easy access to backup logs when needed:

$ journalctl -t restic-cron-backup Nov 16 23:05:57 www restic-cron-backup[9390]: using parent snapshot 25f3eb23
Nov 16 23:05:57 www restic-cron-backup[9390]: Files:           0 new,     0 changed,     2 unmodified
Nov 16 23:05:57 www restic-cron-backup[9390]: Dirs:            0 new,     0 changed,     2 unmodified
Nov 16 23:05:57 www restic-cron-backup[9390]: Added to the repository: 0 B   (0 B   stored)
Nov 16 23:05:57 www restic-cron-backup[9390]: processed 2 files, 18 B in 0:00
Nov 16 23:05:57 www restic-cron-backup[9390]: snapshot 5bfd0c72 saved
Nov 16 23:06:10 www restic-cron-backup[9458]: Applying Policy: keep daily snapshots within 7d, weekly snapshots within 1m, monthly snapshots within 1>
Nov 16 23:06:10 www restic-cron-backup[9458]: keep 1 snapshots:
Nov 16 23:06:10 www restic-cron-backup[9458]: ID        Time                 Host        Tags        Reasons            Paths
Nov 16 23:06:10 www restic-cron-backup[9458]: -----------------------------------------------------------------------------------
Nov 16 23:06:10 www restic-cron-backup[9458]: 5bfd0c72  2023-11-16 23:05:56  www                     daily within 7d    /root/dir
Nov 16 23:06:10 www restic-cron-backup[9458]:                                                        weekly within 1m
Nov 16 23:06:10 www restic-cron-backup[9458]:                                                        monthly within 1y
Nov 16 23:06:10 www restic-cron-backup[9458]:                                                        yearly within 75y
Nov 16 23:06:10 www restic-cron-backup[9458]: -----------------------------------------------------------------------------------
Nov 16 23:06:10 www restic-cron-backup[9458]: 1 snapshots
Nov 16 23:06:10 www restic-cron-backup[9458]: remove 2 snapshots:
Nov 16 23:06:10 www restic-cron-backup[9458]: ID        Time                 Host        Tags        Paths
Nov 16 23:06:10 www restic-cron-backup[9458]: ----------------------------------------------------------------
Nov 16 23:06:10 www restic-cron-backup[9458]: e0f0801e  2023-11-16 22:53:02  www                     /root/dir
Nov 16 23:06:10 www restic-cron-backup[9458]: 25f3eb23  2023-11-16 23:05:51  www                     /root/dir
Nov 16 23:06:10 www restic-cron-backup[9458]: ----------------------------------------------------------------
Nov 16 23:06:10 www restic-cron-backup[9458]: 2 snapshots
Nov 16 23:06:10 www restic-cron-backup[9458]: [0:00] 100.00%  2 / 2 files deleted

Conclusion

We have a convenient and reliable tool for working with backups and providing the necessary storage depth with the help of Restic and 3HCloud object storage. This approach helps reduce the costs associated with setting up and maintaining your own system, providing access to archives from anywhere in the world.


News
28 March 202403/28/2024
read 1 minuteread 1 min
3HCloud Brings GPU Servers to Miami
28 December 202312/28/2023
read 2 minutesread 2 min
New features and tariffs