Wednesday, December 29, 2010

Oracle CSService Hangs on Single Node with ASM, on Remote Machine

It was a while since I posted anything, don't know why...!

Any way, I faced a problem during the last two days, the network admin decided to move to new domain and one of the technicians decided to move the database at one of the remote location we have to the new domain, every thing was working fine until one of the databases on that machine crashed.

So, when I tried to login as /sysdba it kept saying insufficient privileges, but when logging using sys account with password it worked fine. I tried to startup the database but it couldn't connect with ASM due to insufficient privilege again. I was logging as domain admin to the machine, and tried the same thing using local admin with the same problem, even though, I tried to change the authentication method in the SQLNET.ORA file from NTS to NONE, but I kept receiving the same error.

I even added the the local administrator to dba_ group under windows group, but with no success. When I decided to restart the machine, the remote desktop didn't connect and the machine seemed to be hanged.

I tried to connect to the management console, it connects which mean the server is working, so , I doubted that there is an oracle service causing these problems. At first, I needed to connect to the machine using Remote Desktop, so I connected through, register editor to the remote machine, then I disabled all oracle services (in REGEDIT), and issued a restart command on the machine using shutdown line command (shutdown -r -m \\

I was connected using RDP, then I tried to start oracle services one at time, but the OracleCSService couldn't be started. So, after checking the log file of the cssd service, I noticed that crsctl.exe check boot keeps return an error and it stuck in infinite loop, I  build the OCR file used by this service using 

ORACLE_HOME of 10g\bin\clscfg.exe -local 
-o ORACLE_HOME of 10g

then I changed the logon owner of all oracle service to local system on services properties, and it worked the CSSD is up and running, it seemed that the local OCR file used by cssd is corrupted due to service failure, which prevented oraclecsservice from starting and as a consequence it blocked the RDP connection.

And, When changed the logon owner of all oracler service, the insufficient privilege disappeared also and the database is up and running.

Then I've changed the listener.ora file and tnsnames.ora file with server address than server name and restarted the listener.