Tomcat 空间占用增长异常分析一例

第一个感觉以为是 df 和 du 这两个命令显示的有差异。没错，的确有差异。但是，现在的主要问题是 df -k 命令显示的空间在增长。分析研究了 /proc 下相关进程对应的目录下的信息，未发现异常。图劳无功。

猜测tomcat 打开了某些文件描述符，虽然文件已经修改了，但是文件描述符如果没有关闭的话，一样会占用空间甚至增大空间的占用。用 lsof(LiSt Opened File) 进行分析。

注：对一些特定的路径进行了一点处理。首先获取/opt 对应的设备名字。

# lsof /dev/sdb2 | sort +6nr | head -5 
java      21247 root    1w   REG  104,6 96538188 917506 /opt/tomcat/logs/catalina.out (deleted)
java      21247 root    2w   REG  104,6 96538188 917506 /opt/tomcat/logs/catalina.out (deleted)
java      21247 root  mem    REG  104,6  1394764 901145 /opt/tomcat/common/lib/nls_charset12.jar
java      21247 root  mem    REG  104,6  1394295 901143 /opt/tomcat/common/lib/nls_charset11.jar
java      21247 root  mem    REG  104,6  1159907 901140 /opt/tomcat/common/lib/classes12dms.jar
[root@stat root]# 

经确认，/opt/tomcat/logs/catalina.out (deleted) 项占用的空间在疯狂增长。建议跟踪该文件的变化。

# tail -f /opt/tomcat/logs/catalina.out
java.net.UnknownHostException: auto at 
java.net.InetAddress.getAllByName0(InetAddress.java:1016) at 
java.net.InetAddress.getAllByName0(InetAddress.java:981) at
java.net.InetAddress.getAllByName(InetAddress.java:975) at 
java.net.InetAddress.getByName(InetAddress.java:889)  
org.apache.catalina.cluster.mcast.McastMember.getData(McastMember.jav a:122) at 
org.apache.catalina.cluster.mcast.McastServiceImpl.send(McastServiceI mpl.java:221) at
org.apache.catalina.cluster.mcast.McastServiceImpl$SenderThread.run(M castServiceImpl.java:259)  

注意到

java.net.InetAddress.getAllByName0(InetAddress.java:1016

测试可能是 Tomcat 的 Bug,把 java.net.InetAddress.getAllByName0(InetAddress.java:1016 作为搜索短语查找一下，果不其然：

Updated:2005/12/12 . 空间不释放的问题:

When you open a file, you get a pointer. Subsequent writes to this file references this file pointer. The write call does not check to see if the file is there or not. It just writes to the specified number of characters starting at a predetermined location. Regardless of whether the file exist or not, disk blocks are used by the write operation. The df command reports the number of disk blocks used while du goes through the file structure and and reports the number of blocks used by each directory. As far as du is concerned, the file used by the process does not exist, so it does not report blocks used by this phantom file. But df keeps track of disk blocks used, and it reports the blocks used by this phantom file.

下载 LSOF 工具的使用文档