Skip to Content

Informix in the Cloud - Part 1: Storing your ontape Backups on Amazon S3

In this new Informix in the Cloud Series I'll focus on how to setup, configure and deploy the Informix database in the Cloud. Based on it's sophisticated replication technology and excellent embeddability, Informix is the optimal choice for deploying a database in the Cloud.

Most of the examples are based on the Amazon Web Services (AWS) Clould Computing platform. This is currently the most mature Cloud platform on the market with a sophisticated set of different services and a bunch of visual and command line management tools as well as an excellent API. However we'll also explore IBM's SmartCloud offering that is also on the way to become an interesting choice for public Cloud deployments.

Amazon S3 Overview

In this first article I'll demonstrate how to leverage Amazon S3 - Simple Storage Service - to store Informix ontape backups. AFAIK Amazon S3 has been the first Cloud service offered by Amazon. It is a data storage service that stores data redundantly and is able to scale to handle large amounts of data requested concurrently by a large number of users. Amazon S3 provides 99.999999999% durability and 99.99% availability for stored objects. These are excellent numbers for a reasonable price. The Amazon prices at the moment of writing this article are:

  • 0.140 USD per GB stored
  • 0.01 USD per Request made
  • 0.100 USD per GB in
  • 0.201 USD per GB out (first GB free, additional discounts if amount exceeds 10 TB)

As you can see you not only need to calculate the raw storage cost, you also have to calculate the number of requests you'll make (put, copy, post, list, get) and the amount of data transferred in and out of the Amazon Cloud. It is quite common to all AWS infracstructure services that there multiple cost factors. However, in summary, I believe that the Amazon Cloud is still an excellent and cost efficient offering compared to buying, configuring, operating and maintaining your own hardware infrastructure.

Getting started

Since Informix 11.70 (the Panther release) you can now directly perform Backups and Restores thru ontape into or out of the Amazon S3 Cloud infrastructure service.

How to get started ?

First of all you need an Amazon AWS account. This is now an excellent opportunity to create one because Amazon currently offers the AWS Free Usage Tier. This is a one-year cost free offering to try and test the AWS infrastructure services like EC2 (Elastic Compute Cloud) or S3 (Simple Storage Service).

You can sign up for AWS here.

Configuring Informix

After you have signed up for AWS, the configuration steps for Informix are fairly easy. First you need to login into your AWS account and lookup your security credentials (AccessKey and SecretKey). You'll find them under Account -> Security Credentials -> Access Credentials. The SecretKey will only be shown after clicking the View button.

You take the AccessKey and the Secretkey and write them to the configuration file of the Informix Cloud Backup utility.

  • $INFORMIXDIR/etc/ifxbkpcloud.credentials
accessKey=xxx
secretKey=yyy

In order to use the Informix Cloud Backup Utility, you needs Java 6 Runtime or Developer Kit installed on your machine.

Now you create a Amazon S3 bucket. A bucket is a container that is able to store millions of distinct files of various types (text, HTML, binary, graphic...). Of course you are not limited to a single bucket. You could create various buckets (up to 100 per AWS account) to group your backup objects if you like.

I prefer to create the bucket thru the AWS Management Console. This is a comfortable web managing interface for configuring AWS. I've named my bucket ifxtape. You need to specify a different name because the name must be globally unique across Amazon S3.

Another point to consider when creating the bucket is the region where the data is about to be stored. The Amazon Cloud provides currently 5 regions:

  • US East (Northern Virginia)
  • US West (Northern California)
  • EU (Ireland)
  • Asia Pacific (Singapur)
  • Asia Pacific (Tokyo)

You should choose the region that is geographically the closest to your Informix server. Alternatively to the AWS Management Console, you could create the Backup Device thru the Informix Cloud Backup utility. Here is an example:

$INFORMIXDIR/bin/java -jar ifxbkpcloud.jar CREATE_DEVICE amazon
ifxtape EU

The above command would create a bucket named ifxtape in the EU region.

After the bucket has been created we need to make the following entries in $INFORMIXDIR/etc/$ONCONFIG:

TAPEDEV '/tmp/tapedev, keep = no, cloud = amazon, url = https://ifxtape.s3.amazonaws.com'
BACKUP_FILTER '/usr/bin/mcrypt --gzip --keyfile /home/informix/key.txt'
RESTORE_FILTER '/usr/bin/mcrypt --gzip --keyfile /home/informix/key.txt --decrypt'

In more detail:

  • TAPEDEV (Entries separated by comma)
    • The 1st entry specifies the local directory where the temporary backup file will be stored
      for backup as well as restore operations.
    • The 2nd entry specifies that we want to delete the temporary file after it has been
      successfully transfered to Amazon S3.
    • The 3rd entry specifies the Cloud provider, in our case 'Amazon'. I don't know if currently other
      Cloud providers are supported. I believe additional providers will be added in the future.
    • The 4th entry is the URL of the Amazon S3 bucket. You should specify the https protocol
      here in order to open a secure connection to the Amazon Cloud.

BACKUP_FILTER and RESTORE_FILTER will call the mcrypt utility to encrypt/decrypt the backup data before it is send to Amazon S3 (Backup) and after it has been retrieved by Amazon S3 (Restore). Remember that we also use the https protocol for secure transmission over the internet. In addition the --gzip option is specified to compress the backup data in order to save transmission and storage costs on Amazon S3.

Performing a Backup

Now we are ready to perform our first backup to Amazon S3. As usally this done via ontape:

informix@dbconsult:/opt/ibm/ids/IDS.11.70.FC2B2UW/etc$ ontape -s -L 0
Using the backup and restore filter /bin/gzip.
10 percent done.
20 percent done.
100 percent done.
Backing up file /tmp/tapedev/dbconsult_0_L0 to cloud storage location
 https://ifxtape.s3.amazonaws.com.
File created: /tmp/tapedev/dbconsult_0_L0

Please label this tape as number 1 in the arc tape sequence. 
This tape contains the following logical logs:

 9

Program over.

After this you should be able to see the new backup object in your thru TAPEDEV specified target bucket on Amazon S3. You can view the contents of the bucket thru the AWS Management Console.

Second Backup to Cloud Storage fails

Please be aware of the fact that the 2nd backup to Amazon S3 will probably fail. At least during my testing this happened. It looks like a bug to me. You'll receive the following error messages during the backup:

informix@dbconsult:~/instances/panther/data$ ontape -s -L 0
Using the backup and restore filter /usr/bin/mcrypt --gzip --keyfile /home/informix/key.txt.
10 percent done.
20 percent done.
100 percent done.
Stdin was encrypted.
Backing up file /tmp/tapedev/dbconsult_0_L0 to cloud storage location 
 https://ifxtape.s3.amazonaws.com.
The Cloud Client utility failed to run. Check the utility's message log 
for additional information.
Archive failed -

Program over.

This is a good point to make you aware of the logfile that is written by the Informix Cloud Backup utility:

  • $INFORMIXDIR/ifxbkpcloud.log

Here are what it says after the 2nd backup failed:

informix@dbconsult:/opt/ibm/ids/IDS.11.70.FC2B2UW$ cat ifxbkpcloud.log 
06.06.2011 07:18:34 com.amazonaws.http.HttpClient execute
INFO: Sending Request: GET https://ifxtape.s3.amazonaws.com / Headers: 
(Authorization: AWS 0VWQKAS11ZBYC1CJP982:KIzHTj3cLO0Om5dYPuso0ZCRj2Y=, Date: 
Mon, 06 Jun 2011 05:18:34 GMT, Content-Type: 
 application/x-www-form-urlencoded; charset=utf-8, ) 
06.06.2011 07:18:47 com.amazonaws.http.HttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 05BFD09136A34A30

The file dbconsult_0_L0 already exists at device ifxtape

06.06.2011 07:18:47 com.amazonaws.http.HttpClient execute
INFO: Sending Request: GET https://ifxtape.s3.amazonaws.com 
/dbconsult_0_L0 Headers: (Authorization: AWS
 0VWQKAS11ZBYC1CJP982:+8/a3pp+fmlqjT2EDwjczoRHV4U=, Date: Mon, 06 Jun 2011 
05:18:47 GMT, Content-Type: application/x-www-form-urlencoded;
charset=utf-8, ) 
06.06.2011 07:18:56 com.amazonaws.http.HttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 36F52C0646D2FCDE

Error renaming file ifxtape:dbconsult_0_L0

It seems that there is a problem when renaming the backup obect in the Amazon S3 bucket. However you could this easily do manually thru the AWS Management Console.

Restoring Level-0-Backup

Restoring your backup is easy as backing it up. Simply perform the usual ontape command and the backup object will be retrieved from Amazon S3 and stored locally for restoring:

informix@dbconsult:~/instances/panther/data$ ontape -r -d
Restoring file /tmp/tapedev/dbconsult_0_L0 from cloud storage location 
 https://ifxtape.s3.amazonaws.com.
Restore will use level 0 archive file /tmp/tapedev/dbconsult_0_L0. 
Press Return to continue ...
Using the backup and restore filter /usr/bin/mcrypt --gzip --
 keyfile /home/informix/key.txt --decrypt.

Archive Tape Information

Tape type:      Archive Backup Tape 
Online version: IBM Informix Dynamic Server Version 11.70.FC2B2UW 
Archive date:   Mon Jun  6 07:13:24 2011 
User id:        informix 
Terminal id:    /dev/pts/4 
Archive level:  0 
Tape device:    /tmp/tapedev/ 
Tape blocksize (in k): 32 
Tape size (in k): system defined for directory 
Tape number in series: 1 
Backup filter:	/usr/bin/mcrypt --gzip --keyfile /home/informix/key.txt 

...
...

The usual steps with questions for performing a restore follow. Unfortunately the -d parameter for performing a non-interactive restore doesn't seem to be recognized here, so you need to answer a few questions interactively (i.e. 'Do you want to backup logs ? etc).

Backup Object Size Limitation

There is a limitation on the size of ontape backup objects stored in S3. This limitation is currently 5 GB per single object. The 5 GB limit is an old limitation from Amazon S3 that has been lifted last year and the new limitation is 5 TB for obects stored in S3. I assume that the limitation will also be increased to the new 5 TB size for ontape in an upcoming Informix release.

Given the fact that ontape doesn't backup temporary dbspace, backups only used pages (pages allocated to a table extent) and that you could configure BACKUP_FILTER in your onconfig in order to send the data thru gzip in order to reduze the backup size, you should probably be able to backup a 30-50 GB Informix instance to Amazon S3.

However due to the bandwith limitation when backing up a local Informix instance to the Amazon Cloud (upload thru internet line), it does only make sense to backup (very) small instances this way. As soon as you move your entire Informix instance to the Cloud (put them on an EC2 instance), you could backup them directly to an S3 bucket in the same region without additional costs for the data transfer between EC2 and S3. You pay only for the S3 storage occupation. From this perspective it would be good if the current 5 GB ontape limit would be lifted from IBM.

Creating an Informix instance on EC2 will be covered in the next article of the Informix in the Cloud Series.

Copyrightblog |