Why is `rpm -qa base*` so terribly slow?

Question

I wonder: rpm uses a (simple) database to store things like packet names. When using rpm -qa to query that database, it is terribly slow.

For example, this command took almost a minute to complete while the disk was "shuffling":

# rpm -qa rtl8812au-kmp\*
rtl8812au-kmp-preempt-5.9.3.2+git20210427.6ef5d8f_k5.3.18_57-lp153.1.1.x86_64
rtl8812au-kmp-default-5.9.3.2+git20210427.6ef5d8f_k5.3.18_57-lp153.1.1.x86_64

Of course the second invocation was much faster, as all the data was in disk cache, so it took 2.2 seconds "only". Fast is spelled differently, however.

# grep MHz /proc/cpuinfo 
cpu MHz     : 3998.528
...
# LANG= time dd if=/var/lib/rpm/Packages of=/dev/null
748816+0 records in
748816+0 records out
383393792 bytes (383 MB, 366 MiB) copied, 0.606612 s, 632 MB/s
0.30user 0.29system 0:00.60elapsed 99%CPU (0avgtext+0avgdata 1876maxresident)k
0inputs+0outputs (0major+81minor)pagefaults 0swaps
# LANG= time rpm -qa rtl8812au-kmp\*
rtl8812au-kmp-preempt-5.9.3.2+git20210427.6ef5d8f_k5.3.18_57-lp153.1.1.x86_64
rtl8812au-kmp-default-5.9.3.2+git20210427.6ef5d8f_k5.3.18_57-lp153.1.1.x86_64
2.17user 0.06system 0:02.24elapsed 99%CPU (0avgtext+0avgdata 32472maxresident)k
0inputs+0outputs (0major+6302minor)pagefaults 0swaps

So reading the complete package (names) database completely is faster than searching for a specific name!

The /var (root) filesystem is ext3 put on an LV where the PV of the VG is a LUKS-encrypted volume make from a partition of a RAID1 (imsm):

# hdparm -t /dev/mapper/system-root
/dev/mapper/system-root:
 Timing buffered disk reads: 242 MB in  3.02 seconds =  80.19 MB/sec

# hdparm -t /dev/mapper/cr_md-uuid-fe1588c4:a2140388:5b4117ba:4e4339b9-part5
/dev/mapper/cr_md-uuid-fe1588c4:a2140388:5b4117ba:4e4339b9-part5:
 Timing buffered disk reads: 240 MB in  3.03 seconds =  79.33 MB/sec

# hdparm -t /dev/disk/by-id/md-uuid-fe1588c4:a2140388:5b4117ba:4e4339b9-part5
/dev/disk/by-id/md-uuid-fe1588c4:a2140388:5b4117ba:4e4339b9-part5:
 Timing buffered disk reads: 290 MB in  3.01 seconds =  96.32 MB/sec

# hdparm -t /dev/md126
/dev/md126:
 Timing buffered disk reads: 408 MB in  3.01 seconds = 135.63 MB/sec

# hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads: 394 MB in  3.01 seconds = 130.92 MB/sec

# hdparm -t /dev/sdb
/dev/sdb:
 Timing buffered disk reads: 444 MB in  3.03 seconds = 146.61 MB/sec

Timing example

Sorry, this is a late addition, but I had to remember to make the timing test after boot before the RPM database files are in cache.

~> time rpm -qa libavd\*
libavdevice58_13-4.4-pm153.2.8.x86_64
libavdevice57-3.4.9-pm153.1.2.x86_64

real    1m15,788s
user    0m2,727s
sys 0m0,613s
~> time rpm -qa libavd\*
libavdevice58_13-4.4-pm153.2.8.x86_64
libavdevice57-3.4.9-pm153.1.2.x86_64

real    0m2,300s
user    0m2,220s
sys 0m0,080s

So still with all files in RAM cache the command needs 2.2 seconds of user CPU.

Effect of rebuilding the database

(Sorry, the locale is de_DE)

# ll /var/lib/rpm/
insgesamt 447440
-rw-r--r-- 1 root root  20901888 28. Jan 20:35 Basenames
-rw-r--r-- 1 root root     28672 28. Jan 20:35 Conflictname
-rw-r--r-- 1 root root  12189696 28. Jan 20:35 Dirnames
-rw-r--r-- 1 root root      8192 27. Jan 21:40 Enhancename
-rw-r--r-- 1 root root      8192 22. Jan 22:51 Filetriggername
-rw-r--r-- 1 root root     77824 28. Jan 20:35 Group
-rw-r--r-- 1 root root    184320 28. Jan 20:35 Installtid
-rw-r--r-- 1 root root    319488 28. Jan 20:35 Name
-rw-r--r-- 1 root root     94208 28. Jan 20:35 Obsoletename
-rw-r--r-- 1 root root 409018368 28. Jan 20:35 Packages
-rw-r--r-- 1 root root  11694080 28. Jan 20:35 Providename
-rw-r--r-- 1 root root    114688 28. Jan 20:35 Recommendname
-rw-r--r-- 1 root root   1339392 28. Jan 20:35 Requirename
-rw-r--r-- 1 root root         0 26. Okt 2014  .rpm.lock
-rw-r--r-- 1 root root    634880 28. Jan 20:35 Sha1header
-rw-r--r-- 1 root root    360448 28. Jan 20:35 Sigmd5
-rw-r--r-- 1 root root     20480 27. Jan 21:40 Suggestname
-rw-r--r-- 1 root root    667648 28. Jan 20:34 Supplementname
-rw-r--r-- 1 root root      8192 20. Nov 2018  Transfiletriggername
-rw-r--r-- 1 root root      8192 31. Dez 02:06 Triggername
# rpm --rebuilddb 
# ll /var/lib/rpm/
insgesamt 373392
-rw-r--r-- 1 root root  16920576  3. Feb 21:26 Basenames
-rw-r--r-- 1 root root     24576  3. Feb 21:26 Conflictname
-rw-r--r-- 1 root root   7684096  3. Feb 21:26 Dirnames
-rw-r--r-- 1 root root      8192  3. Feb 21:26 Enhancename
-rw-r--r-- 1 root root      8192  3. Feb 21:26 Filetriggername
-rw-r--r-- 1 root root     77824  3. Feb 21:26 Group
-rw-r--r-- 1 root root    139264  3. Feb 21:26 Installtid
-rw-r--r-- 1 root root    311296  3. Feb 21:26 Name
-rw-r--r-- 1 root root     77824  3. Feb 21:26 Obsoletename
-rw-r--r-- 1 root root 343470080  3. Feb 21:26 Packages
-rw-r--r-- 1 root root  10416128  3. Feb 21:26 Providename
-rw-r--r-- 1 root root    114688  3. Feb 21:26 Recommendname
-rw-r--r-- 1 root root   1204224  3. Feb 21:26 Requirename
-rw-r--r-- 1 root root    503808  3. Feb 21:26 Sha1header
-rw-r--r-- 1 root root    311296  3. Feb 21:26 Sigmd5
-rw-r--r-- 1 root root     16384  3. Feb 21:26 Suggestname
-rw-r--r-- 1 root root    647168  3. Feb 21:26 Supplementname
-rw-r--r-- 1 root root      8192  3. Feb 21:26 Transfiletriggername
-rw-r--r-- 1 root root      8192  3. Feb 21:26 Triggername
# 

Having upgraded to OpenSUSE Leap 15.5 the issue is still there (kernel 5.14.21-150500.55.12-default, rpm-4.14.3-150300.55.1.x86_64):

> # directly after boot and login, so most likely no data cached
> time rpm -qa >/tmp/rpmlist-poweroff

real    0m58,971s
user    0m3,016s
sys 0m0,472s
> time rpm -qa 'base*'

real    0m2,644s
user    0m2,603s
sys 0m0,040s
> # when the data is in cache, it's mostly CPU-bound

So the issue seems to be excessive (useless?) I/O.

Answer 1

I recently discovered by using the verbose flag to rpm rpm -qa -vvv that rpm on the OS I am using (RHEL 8) does signature and digest checking when querying all the packages. Adding the --nodigest and --nosignature flags significantly improved the time for me.

rpm -qa --nodigest --nosignature base*

Example timings from my system:

$ free && sync && echo 3 > /proc/sys/vm/drop_caches && free
$ time rpm -qa > /dev/null

real: 0m2.944s
user: 0m1.689s
sys:  0m0.163s
$ time rpm -qa > /dev/null

real: 0m1.740s
user: 0m1.644s
sys:  0m0.079s
$ free && sync && echo 3 > /proc/sys/vm/drop_caches && free
$ time rpm -qa --nodigest --nosignature > /dev/null

real: 0m1.319s
user: 0m0.259s
sys:  0m0.147s
$ time rpm -qa --nodigest --nosignature > /dev/null

real: 0m0.366s
user: 0m0.235s
sys:  0m0.099s

Now I just need to discover if there is a way to safely configure the system (maybe edit or override the macros) to set this as the default behavior for -qa.