Last time we had the /opt filesystem on a couple of ODA nodes filling up. It turned out that the OS Watcher archive directory structure (/opt/oracle/oak/osw/archive) contained lots of old files that should have been cleaned by OSWatcher.
When OSWatcher is started, it will also start a script called OSWatcherFM.sh that is responsible for cleaning up old files. It turned out this OSWatcherFM.sh script was not running and so the archive directory structure was not cleaned up.
Solution:
The solution is simply to restart the OSWatcher which in turn will start the OSWatcherFM.sh script. Execute the following commands as user root:
/opt/oracle/oak/osw/stopOSW.sh /opt/oracle/oak/osw/startOSW.sh 10 504 gzip
Check if both OSWatcher.sh and OSWatcherFM.sh are running using:
ps -ef|grep OSW
oracle 10046 24783 0 10:23 pts/0 00:00:00 grep OSW root 12704 1 0 2012 ? 01:11:42 /usr/bin/ksh ./OSWatcher.sh 10 504 gzip root 12922 12704 0 2012 ? 00:16:48 /usr/bin/ksh ./OSWatcherFM.sh 504
Note: it can take a couple of seconds before the OSWatcherFM.sh script is started, so if it doesn’t show try again a couple of seconds later.
The OSWachterFM.sh script will cleanup the old files after a couple of minutes, so after some minutes the /opt/oracle/oak/osw/archive directory structure will be cleaned up.