Using Redis to Access and Process ‘Big Data’ on the Mainframe

I recently attended a Meetup session where the guest speaker was Dr. Josiah Carlson, one of the leading authorities on Redis and the author of the book “Redis In Action”.  Being a long-time mainframe developer (yes, we do still exist), I was interested to learn about Redis – not only as a technology about which I knew next to nothing, but to see what it was used for and the types of applications that it would work best with.

Among the many applications that Dr. Carlson described as best to use Redis for included those with the following requirements (forgive my brevity):

  • Web cookies
  • Distributed locking
  • Rate limiting
  • Anything you are using memcache for
  • Database row caching
  • Web page caching
  • AutoComplete processes
  • Generalized voting and ranking
  • Unique visitor counting
  • Queues
  • Analytics
  • Statistics
  • As a customized search engine
  • As a generalized data store

He went on to describe the types of Redis requests that could be executed to assist in applications that service the above requirements, and stated that, in his opinion, Redis is becoming the de facto standard for certain types of applications that have the requirements described above.  He also described some of the challenges when designing any application that used Redis.

Being a mainframer, I began to wonder if there were ways to be able to use Redis to directly access information that was stored on the mainframe – after all, by some estimates, more than 80% of all corporate data is still stored there, and large corporate entities are more than happy to have it stay that way for various reasons (cost of converting applications to a non-mainframe environment, ease of access, security, etc.).  Being involved in the legacy integration business, I also knew that there were ways to do some of this, but many of the existing solutions involved application integration or transport of data from the mainframe to a server via FTP – not ideal if all you want to do is search data in real time and provide analytics, statistics, or other information based on what you found.   I was especially interested in being able to access IBM’s VSAM data via Redis and, in effect, just treat a mainframe as ‘another source of data’ using a universally accepted method of access.

So I dove into Dr. Carlson’s book and tried to teach an old dog (i.e. me) a new trick – and discovered that there were capabilities within Redis (in short, setting up strings and sorted sets and using lexicographical searches) that would work very well with existing VSAM capabilities.  Now if only an instance of Redis existing that ran on the mainframe and was identical to the one that is used for non-mainframe apps….

It turns out that such a thing exists!  It is provided by HostBridge Technology (a company that I have had a relationship with for a number of years) and is called ‘HostBridge Redis for z/OS’.    I’ve had a chance to work with this offering, and (allowing for the fact that I am a Redis novice) am convinced that the capabilities are there to expose mainframe data to any application – all by using the same access method!)

Am I dreaming? 🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s