RETHINKing - Big Data - Is hardware/software co-design the key to success?

Tuesday, September 22, 2015

During this year’s VLDB conference, Oracle and IBM presented their upcoming (and existing) solutions for Big Data processing. Codenamed M7 and Power8, respectively, they promise orders of magnitude better transaction processing performance than their competitors. 

So, what is their secret recipe?

At the heart of these systems is the concept of “co-design”. Both Oracle and IBM followed a vertical engineering approach. By architecting solutions at all layers of the computing stack, they manage to gain both tremendous performance improvements and power savings. Oracle on one hand controls the whole “SPARC-JVM-OracleDB” stack. Based on DB trace analysis over decades of experience, they have injected into their hardware vital software algorithmic features. Furthermore, by integrating numerous DB hardware accelerators (DAX), on-chip, they have enhanced performance of their stack while maintaining a low-power footprint. IBM follows the same route by vertically engineering their Power architecture which consists of AIX and IBM DB2 components enhanced with their new CAPI interface.

Such large vendors with their massive engineering capacity and expertise have just started delivering Big Data “co-designed” solutions in order to push the performance boundaries.

But what about Europe?

As Europe does not host such a vertical vendor (in terms of hardware and software), we have to follow a slightly different route. We could capitalize on EU-based power-efficient architectures such as ARM’s Aarch64 and Kalray’s MPPA by building Big Data ecosystems around them. Existing and new large EU-funded projects such as MontBlanc and EuroServer are already moving in this direction, but these projects focus on HPC rather than Big Data.

RETHINK big is asking if European industry sees the benefit of leveraging EU strengths by launching new initiatives that bring together the various elements of the Big Data stack - from Data, software frameworks and runtimes to hardware architectures and interconnects.

Europe has the momentum to penetrate the power-efficient microserver market. This market is an ideal fit for Big Data processing where raw CPU performance is not the only key to success. Building a Big Data ecosystem around micro-servers with the proper integration and vertical design, however, is. Europe has vast expertise at all layers of the Big Data stack; from data sources, to databases, analytics, runtimes and micro-architectures. The key missing part is unifying those teams under the common goal of solving Big Data challenges.

Christos-efthymios Kotselidis