Very poor read performance compared to write performance on md(raid1) / crypt(luks) / lvm

Question

I'm experiencing very poor read performance over raid1/crypt/lvm. In the same time, write speeds are about 2x+ faster on the same setup. On another raid1 setup on the same machine I get normal read speeds (maybe because I'm not using cryptsetup).

OS related disks: sda + sdb. I have raid1 configuration with two disks, both are in place. I'm using LVM over the RAID. No encryption. Both disks are WD Green, 5400 rpm.

IO test results on this raid1:

dd if=/dev/zero of=/tmp/output.img3 bs=8k count=256k conv=fsync
 - 2147483648 bytes (2.1 GB) copied, 22.3392 s, 96.1 MB/s

sync
echo 3 > /proc/sys/vm/drop_caches
dd if=/tmp/output.img3 of=/dev/null bs=8k
 - 2147483648 bytes (2.1 GB) copied, 15.9 s, 135 MB/s

And here is the problematic setup (on the same machine). Currently I have only one sdc (WD Green, 5400rpm) configured in software raid1 + crypt (luks, serpent-xts-plain) + lvm. Tomorrow I will attach another disk (sdd) to complete this two-disk raid1 setup.

IO tests results on this raid1:

dd if=/dev/zero of=output.img3 bs=8k count=256k conv=fsync
2147483648 bytes (2.1 GB) copied, 17.7235 s, 121 MB/s

sync
echo 3 > /proc/sys/vm/drop_caches
dd if=output.img3 of=/dev/null bs=8k
2147483648 bytes (2.1 GB) copied, 36.2454 s, 59.2 MB/s

We can see that the read performance is very very bad (59MB/s compared to 135MB/s when using no encryption). Nothing is using the disks during benchmark. I can confirm this because I checked with iostat and dstat.

Details on the hardware:

• disks: all are WD green, 5400rpm, 64mb cache.
• cpu: FX-8350 at stock speed
• ram: 4x4GB at 1066Mhz.

Details on the software:

• OS: Debian Wheezy 7, amd64
• mdadm: v3.2.5 - 18th May 2012
• LVM version: 2.02.95(2) (2012-03-06)
• LVM Library version: 1.02.74 (2012-03-06)
• LVM Driver version: 4.22.0
• cryptsetup: 1.4.3

Here is how I configured the slow raid1+crypt+lvm setup:

parted /dev/sdc

1. mklabel gpt
2. type: ext4
3. start: 2048s
4. end: -1

Now the raid, crypt and the lvm configuration:

1. mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdc
2. cryptsetup --cipher serpent-xts-plain luksFormat /dev/md1
3. cryptsetup luksOpen /dev/md1 md1_crypt
4. vgcreate vg_sql /dev/mapper/md1_crypt
5. lvcreate -l 100%VG vg_sql -n lv_sql
6. mkfs.ext4 /dev/mapper/vg_sql-lv-sql
7. mount /dev/mapper/vg_sql-lv_sql /sql

So guys, can you help me identify the reason and fix it? It has to be something with the cryptsetup as there is no such read slowdown on the other setup (sda+sdb) where no encryption is present. But I have no idea what to do.

Thanks!

Answer 1

It should be obvious that encryption adds considerable overhead, but disregarding that:

You're also using non-AES encryption which means you don't get the acceleration from your processor (AES hardware acceleration). You're also using a desktop-quality processor and extremely slow consumer drives.

This setup is exactly why you're having such poor performance. Put decent drives in along with a server-quality processor (is this even a server?) and use AES for LUKS.

Answer 2

It's usual not correctly done testing. Loading Linux Software RAID with 8k read operation is the way to ask it beahave poorly (you did). Try using larger bs.

---

I'm also adding this output of cryptsetup benchmark for ppl in comments below

#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   476.6 MiB/s  1891.3 MiB/s
 serpent-cbc   128b    79.1 MiB/s   222.6 MiB/s
 twofish-cbc   128b   135.0 MiB/s   158.7 MiB/s
     aes-cbc   256b   194.7 MiB/s   892.5 MiB/s
 serpent-cbc   256b    46.2 MiB/s   211.4 MiB/s
 twofish-cbc   256b   144.6 MiB/s    93.3 MiB/s
     aes-xts   256b  1542.0 MiB/s  1727.0 MiB/s
 serpent-xts   256b   114.0 MiB/s   200.2 MiB/s
 twofish-xts   256b   111.4 MiB/s   131.7 MiB/s
     aes-xts   512b   573.6 MiB/s   829.3 MiB/s
 serpent-xts   512b   199.8 MiB/s   191.1 MiB/s
 twofish-xts   512b    81.8 MiB/s   107.3 MiB/s

-- clearly seen (not always, but often) decryption is faster.