System Crash during patches causes issue

After a patch set which had a system crash our system starting having these problems.

Reports failed and the wls diagnostic log showed errors like such

java.io.IOException: No locks available

 

Our ofm11g test instance uses nfs to reduce cost, this is not a supported configuration for Oracle and on occasion will cause us issue. The nfs mount is used to share the reports and cache directory instead of ocfs.

 

When researching I found the issue was related to a crashed nfs service, some work around talked about using nolock option but that does not seem necessary although it should pose no risk as long as just your reports and cache are in these directories and not your instance files.

 

[/var/log]# /etc/init.d/nfslock status
rpc.statd dead but pid file exists

[/var/log]# rpcinfo –p

[/var/log]# /etc/init.d/nfslock restart
Stopping NFS locking: [  OK  ]
Stopping NFS statd: [  OK  ]
Starting NFS statd: [  OK  ]
[/var/log]# /etc/init.d/nfslock status
rpc.statd (pid  13972) is running...

I also found the following services in a failed status and restarted them

[/var/log]# service sendmail status
sendmail dead but pid file exists
[/var/log]# service sendmail start
Starting sendmail: [  OK  ]
[/var/log]# vi message.txt
[/var/log]# mail -s "test" <myemail> < message.txt

[/var/log]# service crond status
crond dead but pid file exists
[/var/log]# service crond start
Starting crond: [  OK  ]

After restarting the services the oracle instance needed to be bounced to clear up all issues.

If you can bounce the whole node to make sure all services are cleared up you should be in better shape.