How to Unlock Big Data’s Big Potential

The volume of data in the digital universe is growing exponentially.

The volume of data in the digital universe is growing exponentially. We already have over two zettabytes (two billion terabytes) of data, and by 2020 estimates are that we’re going to be generating over ten times that amount every year.

The technology that we have at our disposal to process, store, and analyze all of this data is also improving very quickly. Since 2004, when Google published its groundbreaking work on the MapReduce paradigm, the Big Data community has created Apache Hadoop and dozens of related products spanning everything from machine learning through high performance ETL.

Recently, however, a new challenge has emerged. The early adopters of this powerful Big Data technology were the familiar consumer Internet companies of Silicon Valley like Yahoo, Facebook, Amazon, and LinkedIn. These companies were created essentially from scratch, and so were able to infuse software like Apache Hadoop at the very core of their offerings.

Within the last few years, this Big Data technology has seen rapid adoption by more “grown-up” entities across every industry, including financial services, retail, healthcare, and city government. These businesses and governments already have massive platforms, such as the mainframe, that have been built up over decades to handle their data systems of record and incredibly large transaction volumes. The new Big Data technology isn’t very useful if it can’t ingest and process the most important data stored in these systems!

But how do you efficiently and securely sync mainframe data to Apache Hadoop without disrupting existing high-performance mainframe systems? And how does a large enterprise manage and secure all of the Big Data technology sprawl being deployed? The early adopters of Big Data technology didn’t have to confront this challenge, since they didn’t have mainframes and they had much less complex infrastructure environments — but for most enterprises it turns out that these are hard problems to solve.

As software vendors, we’ve made a tremendous amount of progress already, and there’s a lot more to come, so please stay tuned. I recently gave a talk at CA World about the state of big data and how we might tackle some of the challenges that lie ahead. Hit play to hear more.

This video is part of a new online video speaker series called Luminaries Take 5 which was kicked off at CA World 2013. The five minute video segments, featuring technology industry visionaries, offer a provocative perspective on how disruptive technologies will drive innovation in today’s organizations. Other Luminaries Take 5 video segments can be found on and YouTube.

Written by

Lonne Jaffe

On March 15, 2012, Lonne Jaffe joined CA Technologies as the senior vice president for…

Published in

View this topic
  • James Holland

    This is great. Hooray for Disney’s imagineers!

    become a new brand in the share market research with its accurate research. Proven
    itself always right whether market is bull or bear. Last week all paid clients
    booked handsome profit in NIFTY, BANKINIFTY & STOCKS. Now for the coming
    week we expect more correction can come in NIFTY as the IRAQ issue is getting
    more tense, If it happens more then you will see a sharp fall in all world marketNSE BSE, STOCK TIPSbecause as we know all world run on
    crude & most of the crude comes from IRAQ. So be ready for a sharp fall so
    sell will be the best strategy for next week also. Traders can make a sell
    position in NIFTY around 7600-7650 with stoploss 7750 for the target of
    7300-7200.One can also make a sell call NIFTY 50 stocks as per NIFTY levels. You
    can also take our two days free trial to check our accuracy. For further updates
    you can visit our website.



  • king lear

    testing comment functionality, please do not publish this

  • Love the personal pic 🙂

    • CAHighlight

      Thank you!

  • Plutora Inc

    This is a good case study. 2.3 sec’s off a login transaction is big.

  • While the analysts were hyping DevOps, I posted the oversight of not including security as part of that discussion as you are highlighting here. Instead of just talking DevOps, it should be DOS (what’s old is new again 🙂 – DevOpsSec. As a previous AppDev person, it’s the app, who’s using it, why and where rather than the device and having the service available.

    As you rightly point, out Security should be baked into the solution.

    Nice Post and Timely!


    • CAHighlight

      Thank you for your feedback Michele. Agreed – security cannot be overlooked. Appreciate your input!

  • Mitesh

    I would love a printed copy

  • Lars Johansson

    I love the idea of BYOID! This makes me choose if I am almost anonymous (with my Hotmail Nicname) or official with identity from an official organisation. My Identity Provider will attach identity with right level of LoA according to the need of the Service provider.

    • CAHighlight

      Thank you for your comment. BYOID has tangible benefits for end users and relying parties but it also has to be weighed in the balance with potential risks and liability concerns. It will be interesting to see how BYOID plays out in the enterprise.