Frameworks, Software Models, Algorithms and Data Structures, Visualization
The new and advanced hardware, system architectures, and networking technology described above are a prerequisite to boost Big Data management. However, more powerful hardware does not guarantee that "old" software and algorithms will perform better, meaning at the same rate that the hardware capabilities improved. Rather, a revised software infrastructure is required to exploit the full potential of new hardware architectures. Hardware vendors are expected to supply the efficient and effective software drivers for their hardware. However, with the hardware landscape growing more diverse and complex, ever deeper and more complex software layers will be required to provide unified system interfaces. These software stacks will become a performance and scalability bottleneck by themselves.
Moreover, existing data management and data analysis approaches are largely bound to considering all relevant data to produce complete and correct results. Consequently, the processing capacity is limited by the amount of hardware available, prohibiting strict interactive response times with data that grows much faster than (hardware) processing capabilities advance. In the task, we will analyze the entire software stack from low-level hardware drivers over operating systems, data management systems and data analysis applications, all the way up to visualization and user interfaces. The goal is to identify the bottlenecks that limit efficiency, performance and/or scalability or that might need to be eliminated. Based on our analysis and available reports, this task will also examine the EU industrial landscape to identify companies or sectors that should support the removal of such bottlenecks as a matter of competitive advantage.