Finding deleted files that are still opened by a process
Recently my server ran out of space, but my checks (via du) reported that only about 60GB was actually being used. That's because if a process has a file open that is then deleted, the space is still claimed, but the file doesn't appear.
The following code snippet will show files that have been deleted, but are still being held open (and using storage space).
lsof +L1
That's it. From there, you can determine what applications were still holding open files. In my case, rsyslog was still holding open an old version of /var/log/syslog.1 and /var/log/messages.1:
rsyslogd 30120 root 1w REG 8,3 25107334316 0 1211404 /var/log/syslog.1 (deleted) rsyslogd 30120 root 2w REG 8,3 18718398817 0 1211409 /var/log/messages.1 (deleted)
You'll notice that they show deleted, and have a combined storage footprint of approximately 43GB. Restarting rsyslog allowed the files to be freed, and storage reclaimed.