it is a mess @#$

About me, IT and various tips and tricks.

How to backup your Linux system with LVM snapshots

Since I received my Pinebook Pro it happened to me a few times that, after I upgraded my Manjaro ARM Linux on it, I was left with a crippled system. So I had to come up with some backup strategy before I upgrade.

Since I have the system (root filesystem) on luks partition with LVM, it dawned to me that I can use LVM snaphots for this.

So let’s see if we have enough place in volume group for such an operation:

[root@pbp ~]# vgdisplay
— Volume group —
VG Name vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 12
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size <893.75 GiB
PE Size 4.00 MiB
Total PE 228799
Alloc PE / Size 134081 / 523.75 GiB
Free PE / Size 94718 / 369.99 GiB

There is almost 370GB of free space in a volume group vg0. So let’s see how big root volume is:

[root@pbp ~]# lvdisplay
— Logical volume —
LV Path /dev/vg0/rootlv
LV Name rootlv
VG Name vg0LV UUID xyZeBw-Pltj-3wXG-DyfS-fc7o-lxrc-qQyue
LV Write Access read/write
LV Creation host, time pbp-sd, 2020-02-14 13:48:33 +0100
LV snapshot status source of
rootsnap [active]
LV Status available
# open 1
LV Size 30.00 GiB
Current LE 7680
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:2

Root volume is 30GB so there is more than enough space in volume group to create a snapshot:

[root@pbp ~]# lvcreate -s -n rootsnapshot -L 30G /dev/mapper/vg0-rootlv
Logical volume “rootsnapshot” created.

Check if everything went ok. Notice “Original” and “Snapshot” labels:

[root@pbp ~]# lvscan
ACTIVE Original ‘/dev/vg0/rootlv’ [30.00 GiB] inherit
ACTIVE ‘/dev/vg0/swaplv’ [4.50 GiB] inherit
ACTIVE ‘/dev/vg0/homelv’ [459.25 GiB] inherit
ACTIVE Snapshot ‘/dev/vg0/rootsnapshot’ [30.00 GiB] inherit

You can also see how much changes have been written to original volume since snapshot was created (0.34%):

[root@pbp ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
homelv vg0 -wi-ao—- 459.25g
rootlv vg0 owi-aos— 30.00g
rootsnapshot vg0 swi-a-s— 30.00g rootlv 0.34
swaplv vg0 -wi-ao—- 4.50g

The snapshot volume can be mounted, as any other device, to access original files and optionally make a tar backup and transfer it outside of your computer:

[root@pbp ~]# mount /dev/mapper/vg0-rootsnapshot /mnt/rootsnapshot/

[root@pbp ~]# ls -l /mnt/rootsnapshot/
total 76
lrwxrwxrwx 1 root root 7 Apr 16 2019 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Feb 14 14:07 boot
drwxr-xr-x 2 root root 4096 Feb 14 13:53 dev
drwxr-xr-x 70 root root 4096 Feb 21 16:34 etc
drwxr-xr-x 3 root root 4096 Dec 4 20:30 home
lrwxrwxrwx 1 root root 7 Apr 16 2019 lib -> usr/lib
drwx—— 2 root root 16384 Dec 9 17:45 lost+found
drwxr-xr-x 4 root root 4096 Feb 19 14:48 mnt
drwxr-xr-x 2 root root 4096 Apr 16 2019 opt
-rw-r–r– 1 root root 44 Dec 9 13:46 overlay.txt
drwxr-xr-x 2 root root 4096 Feb 14 13:53 proc
drwxr-x— 18 root root 4096 Feb 19 10:22 root
drwxr-xr-x 24 root root 4096 Feb 14 07:57 run
lrwxrwxrwx 1 root root 7 Apr 16 2019 sbin -> usr/bin
drwxr-xr-x 4 root root 4096 Oct 30 12:08 srv
drwxr-xr-x 2 root root 4096 Feb 14 13:53 sys
drwxrwxrwt 30 root root 4096 Feb 9 14:01 tmp
drwxr-xr-x 9 root root 4096 Feb 20 13:01 usr
drwxr-xr-x 12 root root 4096 Feb 20 13:06 var

Now, how to recover in case your upgrade (or whatever else) goes wrong:

[root@pbp ~]# lvconvert -v –merge vg0/rootsnapshot

Logical volume vg0/rootlv contains a filesystem in use.
Can’t merge over open origin volume.

Merging of snapshot vg0/rootsnapshot will occur on next activation of vg0/rootlv

Because you are trying to recover your root volume, and it is mounted, the operation is not going to succeed. But it will be recovered after next reboot, just after LVM vary on process. After this, snaphot volume will be deleted and you will boot into the system before the upgrade/whatever else.

Move/clone your VPS from one provider to another

Recently I decided to change my VPS provider. I’ve chosen Contabo instead of OVH. Since I had Debian 10 installed in OVH I also ordered Debian in Contabo. After they established my new machine I checked for differences in partition layout and found that Contabo VPS has two partions and OVH has just one. It looks like this:

contabo# df -h

Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 395M 16M 380M 4% /run
/dev/sda2 294G 6.6G 272G 3% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda1 922M 47M 812M 6% /boot

ovh# df -h

Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 386M 39M 347M 11% /run
/dev/sda1 40G 7.3G 31G 20% /
tmpfs 1.9G 396K 1.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup

The difference is that /boot partition on Contabo VPS.

I’m lazy 🙂 So I was looking for a way to just “clone” my old VPS from OVH to Contabo and I came up with the following solution.

First backup entire filesystem on your old machine excluding some directories especially /boot directory. We will leave that partition mounted on /boot on Contabo machine untouched. So on old VPS (if you don’t have swapfile you don’t have to exclude it):

# cd /

# tar ––exclude=’./swapfile’ ––exclude=’./proc’ ––exclude=’./sys’ ––exclude=’./dev’ ––exclude=’./run’ ––exclude=’./boot’ -zcvf os.tgz .

Transfer the file with your old OS (os.tgz in my case) to your new VPS (being logged in on your new VPS):

# cd /

# sftp user@oldmachine

# cd /  (make sure you have permissions to read the os.tgz file)

# get os.tgz

 

Make a backup of the OS on new VPS similar way:

# cd /

# tar ––exclude=’./sys’ ––exclude’./dev’ ––exclude’./proc’ ––exclude=’./run’ ––exclude=’./boot’ ––exclude=’./os.tgz -zcvf os_orig.tgz .

I also copied two additonal files to have them around:

# cp /etc/fstab /

# cp /etc/network/interfaces /

After that restart your new VPS in rescue mode and mount root partition somewhere:

# mkdir /mnt/root

# mount /dev/sda2 /mnt/root (it was /dev/sda2 in my case)

# cd /mnt/root

Now remove directories that will be overwritten by those from the backup (in case you are moving to some different provider than Contabo or the OS is not Debian 10 then this list might not be precise):

# cd /mnt/root

remove below listed directories

lrwxrwxrwx 1 root root 7 Nov 29 17:58 bin -> usr/bin
drwxr-xr-x 74 root root 4096 Nov 29 13:49 etc
drwxr-xr-x 3 root root 4096 Nov 29 11:32 home
lrwxrwxrwx 1 root root 7 Nov 29 17:58 lib -> usr/lib
lrwxrwxrwx 1 root root 9 Nov 29 17:58 lib32 -> usr/lib32
lrwxrwxrwx 1 root root 9 Nov 29 17:58 lib64 -> usr/lib64
lrwxrwxrwx 1 root root 10 Nov 29 17:58 libx32 -> usr/libx32
drwxr-xr-x 2 root root 4096 Nov 29 10:15 media
drwxr-xr-x 2 root root 4096 Nov 29 10:15 mnt
drwxr-xr-x 2 root root 4096 Nov 29 10:15 opt
drwx—— 3 root root 4096 Nov 29 11:44 root
lrwxrwxrwx 1 root root 8 Nov 29 17:58 sbin -> usr/sbin
drwxr-xr-x 2 root root 4096 Nov 29 10:15 srv
drwxrwxrwt 8 root root 4096 Nov 29 13:49 tmp
drwxr-xr-x 13 root root 4096 Nov 29 10:15 usr
drwxr-xr-x 11 root root 4096 Nov 29 10:15 var

Unpack your old OS from backup (note the p switch which is there to preserve permissions):

# cd /mnt/root

# tar xvzfp os.tgz

Now the two last steps. Restore fstab and interfaces files you have copied earlier:

# cd /mnt/root

# cp fstab etc/

# cp interfaces etc/network/

Now reboot. If everything went well than all you have to do is reconfigure your services like apache, openvpn etc.

Also check what is in your /etc/resolv.conf – that too might have to be taken care of.

Good luck 😛

Resize LVM volume so it fills entire remaining VG space

This is a quick one. I have one volume and want to resize it so it will fill entire space that is left on a VG. I know there is plenty. If you want to check the VG use command “vgdisplay”. I want to resize my root volume:

root@ubuntu:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2.4G 908K 2.4G 1% /run
/dev/mapper/ubuntu–vg-ubuntu–lv 3.9G 1.7G 2.1G 45% /
tmpfs 12G 0 12G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/sda2 239M 75M 147M 34% /boot
tmpfs 2.4G 0 2.4G 0% /run/user/1000
/dev/loop0 89M 89M 0 100% /snap/core/7270

Let’s resize it.

root@ubuntu:~# lvresize -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
Size of logical volume ubuntu-vg/ubuntu-lv changed from 4.00 GiB (1024 extents) to <99.00 GiB (25343 extents).
Logical volume ubuntu-vg/ubuntu-lv successfully resized.

Resize underlying filesystem:

root@ubuntu:~# resize2fs /dev/ubuntu-vg/ubuntu-lv
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/ubuntu-vg/ubuntu-lv is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 13
The filesystem on /dev/ubuntu-vg/ubuntu-lv is now 25951232 (4k) blocks long.

Check actual size of filesystem now:

root@ubuntu:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2.4G 908K 2.4G 1% /run
/dev/mapper/ubuntu–vg-ubuntu–lv 98G 1.7G 92G 2% /
tmpfs 12G 0 12G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/sda2 239M 75M 147M 34% /boot
tmpfs 2.4G 0 2.4G 0% /run/user/1000
/dev/loop0 89M 89M 0 100% /snap/core/7270

Done 🙂

Block entire countries from accessing site using ufw on Linux

Recently one of my sites started to get very slow and unresponsive. After analyzing Apache access log I have figured out that there is a lot of strange requests coming from China.

I have enabled GEO blocking on an Apache level but day after traffic from China increased to the levels that the site started to slow down again. Apache had to display “You are not authorized” page to each request coming from China anyway so it was exhausting its resources.

And so I had to block this traffic on an OS level by firewall. There is a very handy site coming to your rescue –> http://www.ip2location.com/free/visitor-blocker where you can download a CIDR formatted text file with IP addresses from a chosen country. Get the file and save it. Rename it (in my case filename was china.txt).

After you downloaded the file everything is relatively easy:

while read line; do sudo ufw insert 1 deny from $line to any; done < china.txt

For big country it might take even hours to complete. After it’s done check ufw:

sudo ufw status

It should look similar to the screenshot below:

In case you want to remove the rules run following command:

while read line; do sudo ufw delete deny from $line; done < china.txt

Powered by WordPress & Theme by Anders Norén