DAPHNE - DAPHNE-EU

Integrated Data Analysis Pipelines for Large-Scale Data Management, HPC and Machine Learning

About DAPHNE

The DAPHNE project aims to define and build an open and extensible system infrastructure for integrated data analysis pipelines, including data management and processing, high-performance computing (HPC), and machine learning (ML) training and scoring. The acronym DAPHNE relates to the title "integrated Data Analysis Pipelines for large-scale data management, High-performance computing, and machiNE learning". To develop a comprehensive framework, the project is organized in the following areas:

System Architecture

Improve productivity for integrated data pipelines with open APIs and a domain-specific language (DaphneDSL)

Hardware

Run workloads on state-of-the-art heterogeneous hardware devices and improve utilization of existing computer cluster resources

Scheduling

Improve scheduling and task planning and leverage interesting data characteristics such as the sorting order, degree of redundancy, and matrix/tensor sparsity

Benchmarks

Testing the system against existing benchmarks as well as a new benchmark developed as part of the DAPHNE project

Use cases

Evaluate the developed framework in real-life use cases with large-scale datasets and improve runtime and accuracy

Open Source

Publish an open, extensible reference implementation of the necessary compiler and runtime infrastructure to simplify the integration of current and future state-of-the-art methods

Daphne
/ˈdæfni/
Figure in Greek mythology associated with
fountains, wells, springs

Use Cases

Earth Observation

Understanding the local climate is essential for societies facing climate change. DAPHNE develops a deep learning pipeline for local climate zone classification, based on 4 PB of satellite images.

Material Degradation

This use case focuses on understanding and modeling material degradation during operation of semiconductor devices.

Automotive Vehicle Development

Regarding the automotive development process, we will investigate a closed loop high dimensional optimization problem supported by physics-based simulations and behavioral modeling.

Semiconductor Manufacturing

In this area, we will conduct an exploratory use case study about optimizing implantation equipment stability and utilization. Ion implanters generate a multitude of sensor readings – perfect for ML algorithms to learn from.