4

Trying to understand more of Linux and why our diskspace keeps running full with what. Little to none experience with Linux so bear with me

/dev/sda2 is mounted on / and this disk keeps running full enter image description here

If I check a level deeper I can see

enter image description here

Now there is this 18GB on /

How can I see what is actually taking up this space? Folders like /etc /home /var I can check the content but I fail to see what files are directly on the /

All normal folder that would run full like cache or logs or /tmp are checked and emptied and still the disk is at 96% which I think is not normal.

Any other tips are welcome as well ofcourse.

Thanks in advance.

EDIT

Command ls -lst / gives the following result enter image description here

I'm not sure what to make of this but does this mean this 'swap' thing takes up the majority of the space? Or can this be ignored.

Improve this question
3
  • Hard to know for sure; but /swap is probably a file-based swap for your computer's memory. (you can run swapon to see). If it is your computer's swap space; that is a static file which will not change in size; it's size is probably a multiple of your RAM (1x, 1.5x or 2x).
    – ericx
    Commented Jul 11 at 13:34
  • There is a note half off the top of the du run indicating it could not read some things. This means you probably where not running as root, so there could be 10G of data your user just isn't allowed to see.
    – David G.
    Commented Jul 11 at 22:35
  • 1
    It's preferable to post plain text rather than screenshots, for the benefit of people using screen readers.
    – Barmar
    Commented Jul 13 at 15:28

8 Answers 8

Reset to default
5

It would have been helpful if you had provided the actual commands which gave these results (appears to be df & du).

It looks as if there is 18G used in /.

That's bad.

You should NEVER add files there. Not only does it create issues with managing the storage, it also implies that you are running data processing tasks or logging a root. Also bad.

How can I see what is actually taking up this space?

ls -lst / would be a good starting point.

You won't see files which have been deleted (but are still open). lsof | grep 'deleted' might help if you don't see what you expect from ls

Improve this answer
10
  • Thank you, I've ran the first command you gave and added the output to the post. Can you help me interpret the results? Because on sight I never seem to add up to the correct 18GB in use..
    – Filip Huysmans
    Commented Jul 11 at 12:23
  • The swap file is around 4Gb, not 18Gb. That's sort of OK where it is. You are still missing 14Gb. The total figure at the top tells me you've not omitted any significant files (don't post pictures of text you can copy copy paste as text). So you're not seeing what you EXPECT to see.
    – symcbean
    Commented Jul 11 at 18:35
  • 2
    You must remember that the 18G in / include all the subdirectories. The 4G swap plus the rest easily adds to 18G.
    – David G.
    Commented Jul 11 at 22:21
  • ....but there's 28G used in the volume.
    – symcbean
    Commented Jul 11 at 23:01
  • 1
    Do not forget that du can't inspect dirs protected from reading. Use sudo.
    – Basilevs
    Commented Jul 12 at 15:17
12

Use something like ncdu to visualize your disk space usage. There’s other similar tools as well, but this one runs w/out UI, and runs great!

https://dev.yorhel.nl/ncdu

Improve this answer
3
  • 11
    Worth mentioning that deleted files will linger until they're closed by the process having them open.
    – vidarlo
    Commented Jul 11 at 8:28
  • Good point, indeed!
    – BaronSamedi1958
    Commented Jul 11 at 11:19
  • 6
    very good point indeed, deleted docker logfiles where the culprit on this case
    – Filip Huysmans
    Commented Jul 12 at 8:45
4

Step 1, see what du can find in the active filesystem:

du -xm / | sort -n | tail -40

  • Run du on the problem filesystem (/),
  • Use -x to limit the exam to that filesystem
  • Use -m to make sorting simpler (I find it a bit more readable than using -h on both)

If it finds something big, you can examine and remove it. If it doesn't find the corresponding 28G of data, you likely have one of 2 things happening:

Improve this answer
3

Whenever this happens to me, I do a manual recursion something like:

du -hs /* > /du.out

Then when it is apparent that the bloat is from (let's say:) /var, I repeat the process:

du -hs /var/* > /var/du.out

Rinse and repeat until you drill down to your problem location.

The above could obviously be improved as some sort of script; but generally speaking, if the drive has filled up; then you only have the small space reserved for root and many programs won't run if they need scratch space on disk.

Improve this answer
2
  • 4
    If you're doing to do that, just use -d1 (or -hd1) against the directory. No need to shell glob.
    – Bob
    Commented Jul 11 at 19:01
  • 2
    @Bob is correct, but I will go one step further. The shell glob will actually MISS files and directories, notably any that begin with a period (.). The only good thing to say about it is that it will do the top-level files individually (but for that just add the -a option). Personally, I often just do du some/directory | sort -n. Or sometimes I use the KDE tool filelight.
    – David G.
    Commented Jul 12 at 0:55
3

I think the most obvious issue for why we can only see 18GB used in du of 28GB used in df is in the du image.

If you look carefully at the top, you will see the bottom half of the start of something like:

du: cannot read directory '/some/directory': Permission denied

In short, du could not read all the disk. Please redo the du as root, possibly with any other security changes needed to let it see everything.

If this doesn't account for everything, please post what you get. But do so as text, not as an image, and include the starting command and any errors that came out.

Improve this answer
3

Sometime, it`s some files was deleted, but the process still open the file, run this command maybe find out the process

lsof | grep deleted

And then kill it

kill -9 $pid

This is detailed verification steps

## check disk size
$ du -sh /data 2>/dev/null
956M    /data

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         50G 1008M   46G   3% /data

## create a 2GB file
$ dd if=/dev/zero of=/data/myfile.iso bs=1024k count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 1.49613 s, 1.4 GB/s

## check disk size again
$ du -sh /data 2>/dev/null
2.9G    /data

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         50G  3.0G   44G   7% /data

## create and run demo script
$ cat test.py 
import time
f = open('/data/myfile.iso','r')
time.sleep(1200)

$ nohup python test.py &
[1] 1522
nohup: ignoring input and appending output to ‘nohup.out’

## Delete file, and then check disk size
$ rm -f myfile.iso

## now disk size is different, "du" print "956M" and "df" print "3.0G"
$ du -sh /data 2>/dev/null
956M    /data

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         50G  3.0G   44G   7% /data

## find the deleted file and kill process pid
$ lsof |grep delete
python     1522          root    3r      REG               8,16 2097152000         12 /data/myfile.iso (deleted)

$ kill 1522
[1]+  Terminated              nohup python test.py

## This is end of disk size, du and df is eq
$ du -sh /data 2>/dev/null
956M    /data

$ df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         50G 1008M   46G   3% /data
Improve this answer
2
  • 1
    It is rare that you should start with kill -9. Start with just kill, and let the program clean up after itself if it needs to. Also be aware that even kill -9 doesn't always work.
    – David G.
    Commented Jul 12 at 3:26
  • You might also find kill -KILL easier to read and remember.  (All the signals have symbolic names: TERM, HUP, QUIT, ABRT, INT, KILL.  I'd suggest trying them in that order, with KILL as the last resort.)
    – gidds
    Commented Jul 14 at 13:22
2

You appear to have lost 10G. You might try forcing a fsck on the disk. Many modern distributions disable it completely on journaled filesystems, but I personally have seen too much corruption even with the journal to permit it to be disabled.

You either need:

  1. a distribution specific means to force the fsck, typically as part of a reboot.
  2. to switch to single user mode, preferably before / is mounted read/write, and do the fsck there.
  3. to boot from other media and fsck /dev/sda2 yourself.

If there are many issues, I would recommend use tune2fs to change the fsck requirements for the disk.

Improve this answer
2
  • Lost to OS! If he runs du under sudo he will quickly find the lost space.
    – Basilevs
    Commented Jul 12 at 15:15
  • @Basilevs Wrong answer. This answer is about corrupt filesystem. My other answer is about needing root. Depending on a security module, you might need more than just root (but I don't use Linux security modules, so I can't say what).
    – David G.
    Commented Jul 13 at 0:42
0

/var is taking 8.7G. Check out /var/logs. You can use such tools as baobab to quickly grasp what is taking too much space.

My guess is that systemd writes logs too often. Try the following:

sudo journalctl --rotate
sudo journalctl --vacuum-time=1s

If the space frees up, set SystemMaxUse=100M in /etc/systemd/journald.conf.

Improve this answer
1
  • DON'T clean /var/log before finding out what is there. There might be something repeating a LOT that needs examination. simply paging through the log files should help to notice a pattern.
    – David G.
    Commented Jul 12 at 13:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .