I nearly forgot to post this rating that came out over 3 weeks ago!! Informix was rated the #1in customer satisfaction by VendorRate. Check out the report.

Sorry for the loooooong break! With the election fever and kids homework, things got really wound up :-). Now that one is out of the way, I have found some lost time! Today I want to touch on a powerful Informix capability that can either help solve some interesting problems for game developers or find new applications! This capability is known as Informix Extensions - a capability that allows users to extend the capabilities of the Informix data engine and thereby open the door to some really innovative solutions.

Informix provides extensions using it’s datablade technology. Extensions in the Informix context can be thought of as domain specific data-types and their associated routines that are treated as a standard data type by the engine. As an example, consider the Informix spatial extension. The Informix spatial extension consists of data types such as Point, Polygon and include manipulation and query functions such as Intersects(), Boundary(), Union() and so on. The power of extensions is obvious when you consider that they are loaded on demand , executed in the data engine’s process space and accessible via regular SQL, thereby providing both flexibility and performance.

Informix provides an API that allows users to define their own extensions. These extensions can then be registered with the Informix engine as user defined extensions. Let’s say you wanted to manage asset dependencies in your game (see visual below) - for instance, let’s assume that changes to a mesh would trigger changes to the compiled model, the texture page file and the final character. Tracking these dependency changes becomes complex and error prone if it’s done using application business logic. However, by creating an Informix Node extension that represents these dependencies and the actions to be performed on them drastically reduces the complexities and makes it so much easier to use by encapsulating the data type and providing functions such as GetParent(), Compare(), Ancestors() and so on.

If you are excited about the potential this holds and would like to dive into the details let me know. Also I recommend starting with the following material :

The IBM Informix datablade family overview

Developing Extensions to IDS (from the online Informix Information Center)

In the next post, I am going to lead you into another built-in extension in Informix that can be useful in MMOGs and Virtual Worlds.

The GDC 2008 kicked off Monday Sept 15. The exhibit floor seemed very familiar to what I saw in 2007 in terms of technology platforms and vendors, some of the prominent vendor exhibits being Icarus Studios, BigWorld, Simutronics, Multiverse and of course many of the service providers were present - like Vivox and Aria systems.

In terms of the overall roadmap and business models, many vendors continue to focus on “serious games” along with attracting MMO developers and publishers. Learning and education seemed to be primary areas for serious games. There is also an increased focus on the need to get more out of the data from gameplay - improved metrics and deeper data mining.

Icarus Studios exhibited “Fallen Earth” which is now content complete. I believe Fallen Earth would be the first MMO in production on Informix.

In the overall scheme of things, providing rich content and intriguing gameplay to keep customers challenged, happy and coming back still ranked as the top priority. So was there talk around and interest in database technologies ? Indeed yes if you talk to MMO developers who are serious about managing data efficiently and having flexible options to scale data - which by the way most of them are serious about. I am going to blog the database perceptions and challenges in this industry in a separate post - I invite any experiences from you as an MMO developer or publisher on this topic.

When I look at the online game industry and its dynamics, it reminds me somewhat of earlier innovations – pivoting around data management - that took place in other industries. Consider for example, the automation of reservation systems in airlines with SABRE, or the blinding competitive advantage that Wal*Mart build up (and enjoys even today) with its automated supply chain and real time inventory management. These were among other companies and products with visions about the potential power of data. Capturing and storing data is one thing, realizing the competitive advantage of data is another.

Ok, so what does it mean to the online gaming industry?

I have had many a conversation and I still see a lack of “thinking through the database strategy”. Some in the industry are now realizing that the data is strategic to their business and that in terms of game performance, the database engine needs to perform and scale like their game engines. Yet, many companies out there seem to settle on their decision with open source, specifically, MySQL, too quickly. Here are some of the things I usually get:

• It’s free
• It does “most” of what we want it to do
• Google, Wikipedia use MySQL – it scales for them!

Now I will say upfront that I have nothing personally against MySQL – there are scenarios where it meets the needs of certain business requirements adequately. But there’s a lot of omission of detail in much of these statements (MySQL does advertise that Google uses them) that is actually presenting a lot of misleading conclusions – I think it’s probably time to put things in the right perspective. So , you might want to grab that cup of coffee and a snack to bite on , this could be some long reading.

Let’s put cost in perspective – “It’s free” refers to the licensing cost of acquiring the open source community edition. Most people I know always sign up for some level of support which has to be paid for. You then need to install it, run it, design database schemas, load data and maintain it. That’s investment in time and DBA resources, aka total cost of ownership. Calculate those costs for at least three years out for all the instances that need to be deployed and managed (and the supporting hardware and people needed to manage it). Add to that any additional costs for upcoming requirements such as cluster support or high availability. That’s the price you are going to have to pay. That, my friends, does not spell FREE.

In terms of “it does most of what we want it to do” – the biggest risk you face there is working around capabilities that the database management system doesn’t provide (but ought to). Think about it, this can not only be a huge cost, but also sub-optimal in terms of performance and efficiency. It means:

• that prior to MySQL 5.0 (which is very recent), code that dbmses were efficient at executing such as stored procedures and triggers had to be written in application logic. This further means that business logic that needs to be shared, cannot be and have to be made redundant across applications
• that additional support components for functionality such as load balancing and failover have to be acquired or again coded in the application.

If you can see through that, you realize that those are hidden costs that are not accounted for. Another risk that I have seen materialize is when customers find that a critical capability they need is missing, they have to invest in porting a lot of their code  – adding to the cost bucket significantly.

Now on to some interesting stuff – Google and Wikipedia!

First, yes, (a) Google uses MySQL and yes (b) Google scales but concluding (b) happens because of (a) – er, that’s really stretching it and is not factual.

Google has tens of billions of web pages, over 8 billion indexes, Google Earth data and Google Analytics and of course Google’s AdWords application. The only publicly cited application of MySQL at Google is AdWords. To my knowledge, once you sign up for AdWords and fill in your ad and keywords, then it’s pretty much a read-only data model where the ads are looked up and placed in Google pages/results and websites that cater to AdSense. Google’s infrastructure is much more beyond the AdWords application.

So Google does not scale on MySQL. In factt I did some research and have come to learn that Google scalability is built on:

1. Google File System (distributed file system clusters)
2. BigTable ( a specific data management system for a specific need)
3. Chubby – Distributed Lock service
4. MapReduce

They need the combination of the above four to store and scale large amounts of data – essentially they needed to scale across hundreds of thousands of distributed servers that collectively store petabytes of data.

Without going into too much detail, (there’s lot of info already on the internet) BigTable is a persistent multi-dimensional sorted map – the map is indexed by a row key , a column key and timestamp. In order to perform parallel computations on the data stored in BigTable, Google uses a software framework called MapReduce. All this runs on top of Google’s distributed file system cluster.

As you can see, Google made huge investments in building their own technology to scale their infrastructure – including their own data management system – BigTable. So there you go.

Wikipedia is built on top of the MediaWiki application server. Again Wikipedia does not scale on MySQL alone, in fact, there is a rich set of supporting scaling infrastructure that sits above MySQL:

1. A complete content delivery and acceleration network front-ended with caching servers – squid servers, load-balancing with Linux Virtual service – this load balances between the CDN, Application (MediaWiki) and Search (Lucene)
2. PowerDNS for geographical distribution
3. MediaWiki Application Server
4. memcached for distributed object cache
5. A separate Fulltext search component using Lucene – because MySQL fulltext is only available on MySQL ISAM and does not perform or scale as well as Lucene
6. Separate Media storage and server for static files including images
7. And finally, MySQL to host the core database – one database per wiki – core data consists of metadata, users, article revision history, article references. 90% of the queries will be single table lookups on primary keys – there are roughly about 12 tables per core database. When an edit is submitted, mediawiki writes it to the database, without deleting previous versions.

So how does Wikipedia scale? Look closely it’s a combination of Squid caching servers + MediaWiki + Lucene+MySQL+memcached.

A lot of hits are serviced by the cache and they “avoid” expensive database queries.

So in summary this works for WikiPedia because:

1. of lots of cacheable content.
2. scaling users is almost completely handled at the application level – MediaWiki and powerDNS
3. database requirements are very simple

The beauty with Informix is that it does not require applications to “help” out and does not need to rely on a host of supporting software to get work done. On the other hand, if the enterprise uses third party solutions as part of their infrastructure, Informix works well and complements many third party software and hardware high availability and scalability solutions. So stay tuned for a look at how built in Informix capabilities work to maximize your applications performance, scaling and availability requirements.

I hope that gives things a little bit more perspective. And hopefully, you made it through without a refill of your coffee cup! At the end of the day, what works for Google works for Google’s business needs, what works for Wikipedia works for Wikipedia’s needs. Your business is your business and the infrastructure needs have to be thought through at every level.

Stay tuned for more !!

I had the opportunity to be gorgeous LA attending the VW Expo. This was my first at any of the VW conferences.  I frequent the GDC

more. Anway, the thing about this being my first event was that I didn’t really have any particular expectations set. I was to there to take in the technology, the business models and the trends. And from that standpoint, it was a reasonably valuable experience for me! My colleague Karen Keeter forwarded me a few photos from the expo that I post here!

The details of the location and track can be found at the VWExpo site. There were 5 tracks for sessions and i spent my time in the Enterprise, Futures and Technology tracks. The keynote sessions were interesting in that they were more of a Q & A format with an industry expert interviewing the “keynote” speaker. My main interests were really around the application of VW for business value. From that perspective let me first kind of highlight what my takeaways from the conference were:

• Virtual Worlds hold a lot a lot of potential, however, enterprises are taking a cautious approach by deploying small pilots around support functions
• Adoption has several challenges - security of assets, scalability, integration with enterprise applications
• Nobody thinks or expects there to be a killer app, but many panel experts suggested that if there was one it may come out of IBM
• A lot of interest in bringing custo mized virtual world hosting inside the firewall

Effective collaboration continues to be the area where VW deployments are providing benefits - examples of sales training, education, online conference rooms were frequently mentioned. IBM incidentally has a large mindshare in this industry. One of the keynote speakers was Colin Parris, the VP of IBM’s Digital Convergence business unit. Stay tuned for some Lotus innovations with integrated 3-D interaction capabilities.

In terms of potential I get pretty excited just thinking about all the avenues this opens up - think of the post Katrina University of New Orleans and their virtual campus as a risk / contigency management investment - there’s a direct application of VW for disaster recovery!!

There was an Informix demo around 3-D commerce. That’s right. Shopping carts in 3-D worlds! If you look today, while there are a lot of companies with islands inside of SL, much of it is targeted towards brand marketing. But what if you could visually model an item in the world itself for fit,review it’s looks and then add it to a shopping cart inside the 3-D world itself? Leveraging Informix’s spatial capabilities , Lance Feagan (an Informix R&D engineer) whipped up a framework on top of IDS to enable these aspects of commerce as well as get some data mining capabilities. Here is an article on this topic.

Then there was the IBM demo of virtualizing the whole data center for enhancing data center monitoring - adding the “situational awareness” aspect to remote monitoring. It also greatly enhances “what if” scenarios and capcity planning. Now if they could capture the live sensors in those data centers, one can get real-time visuals on elements such as energy consumption and temperature!! I told you I could drum up some potential applications :-).

Yes there were some really neat demos  - I personally liked the Qwaq demo - Qwaq has some impressive capabilities around virtual conference rooms - sharing and editing a variety of document formats inside of the conference room, meeting room and document persistence, collaborative editing, integrated voice, etc.

I also liked the ESC demo (Electric Sheep Company) and their concept of “2.5 D” - personally I didn’t think you could do a whole lot with Flash,but I was impressed with the amount of immersiveness it can produce. So with WebFlock (their flagship product), your virtual world can be inside of a browser - no client downloads. I think the 2.5 comes from the fact that to interact with an asset inside the world such as a document, it has to be done outside of the flash world.

Now I flat out will let you know that I am not an expert on all things about virtual worlds and MMOGs. But I do know a thing or two about databases :-). And I do know about game data and some of the challenges and requirements for their management. For starters about how Informix capabilities can help manage game data, check out this white paper I developed - Fun and Games: The Data Server for Interactive Entertainment.

I look forward to your feedback and comments!

If you are new to Informix, I recommend you visit the IBM site for Informix.

And if you still need some real life validation that database technology matters, read this article on why one company Icarus Studios, switched to Informix.

My colleague George Dolbier, who is a well-respected IBMer in this industry has this valuable article on sizing.

Also, here is a nice article by David Kennerly on aspects of game data mining. If you are an MMO developer, this article gives you good insights into why (and why not to !) , when and what data to mine from your game.

Wow, I finally got here ! As Buzz Lightyear would have said it - To blogosphere and beyond !!

First of all, thank you for visiting my blog! My intent with this blog is to post and discuss on how Informix capabilities are applicable to data management requirements in this exciting Interactive Entertainment industry, especially MMOGs and Virtual Worlds. I also sincerely hope to hear from MMOG and VW developers, platform developers and service providers in this industry on what challenges they see with data management as VW and MMOGs gain mainstream adoption.

Over the last year (and few months), I have been talking with several companies on this topic. No doubt, game-play and rich immersive experiences seem to be the most critical aspects of the business. However, the performance and management of the data that is being used to service these games as well as the data being generated and captured from MMOGs and VWs are drawing more attention from CTOs and chief game architects. Everything from performance of the database, to scaling and mining data for competitive advantage is on the minds of these top level executives.

My colleagues tell me that with my focus on this industry, I am literally mixing “business with pleasure”! But seriously, I have learnt that developing MMOGs or VWs or their platforms are nothing if not serious investments in time and resources. I am truly amazed by the technology that provides so much of intriguing gameplay (for MMOGs) and the rich immersive experiences that VW and MMOGs provide today.

In addition to blogging on database management in VWs and MMOGs, I will also be posting my personal impressions and takeaways from my visit to conferences related to this industry. From time to time I will also post updates on what’s happening on the technology front with Informix as well as upcoming events!

Of course my postings here reflect the personal views of Rajesh Nair and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management.