SDS Node to be stuck in Fast Recovery

Under: | |

SDS Node to be stuck in Fast Recovery

I thought it would be useful to share the following information as this is not very well described in in the IDS V11.5 Information Center.

The problem evolved after trying to start an SDS Node. The successful connect of the SDS Node was documented in the online.log of the Primary:

20:51:21  SDS Server gepard_sds1 - state is now connected

The onstat -g sds command executed on the Primary confirmed that:

IBM Informix Dynamic Server Version 11.50.FC2DE -- On-Line (Prim) -- Up 00:04:35 -- 149188 Kbytes

Local server type: Primary
Number of SDS servers:1

SDS server information

SDS srv      SDS srv      Connection        Last LPG sent        Supports
name         status       status            (log id,page)        Proxy Writes
gepard_sds1  Active       Connected              77,317          Y

However I was not able to execute SQL statements on the SDS Node.

echo "select sysdate from sysdual" | dbaccess sysmaster

27002: No connections are allowed in quiescent mode.

The SDS Node seemed to stuck in Fast Recovery and didn't proceed further:

IBM Informix Dynamic Server Version 11.50.FC2DE -- Fast Recovery (SDS) -- Up 00:04:08 -- 149188 Kbytes

After analyzing the online.log on the SDS Node, I realized the following message:

20:51:21  Restartable Restore has been ENABLED
20:51:21  Recovery Mode
20:51:23  Logical Recovery Started.
20:51:23  10 recovery worker threads will be started.
20:51:23  Start Logical Recovery - Start Log 77, End Log ?
20:51:23  Starting Log Position - 77 0x13a018
20:51:23  B-tree scanners disabled.
20:51:24  DR: SDS secondary server operational
20:51:24  Started processing open transactions on secondary during startup
20:51:25  Checkpoint Completed:  duration was 0 seconds.

I searched thru the documentation but was only able to find the following sentence 'hidden' in a documentation section called Isolation Levels on Secondary Servers:


After starting a secondary server, client applications connect to the server only when all transactions open at the startup checkpoint have either committed or rolled back.

A few minutes later the following message had been written to the online.log of the SDS node:

20:55:15  Finished processing open transactions on secondary during startup

I checked the status of the SDS node using onstat -:

IBM Informix Dynamic Server Version 11.50.FC2DE -- Updatable (SDS) -- Up 00:50:05 -- 149188 Kbytes

After that I was able to connecto to the SDS node and execute SQL statements:

echo "select sysdate from sysdual" | dbaccess  sysmaster

Database selected.

(expression)

2008-10-09 20:56:24.00000

1 row(s) retrieved.

I think this behaviour should be more clearly documented or it should at least be documented in the appropriate section (Setting up an SD Secondary Server) of the documentation.

So if your SDS node seems to stuck in Fast Recovery, inspect your online.log and search for the 'open transactions' message.