====== Lehrveranstaltung: High-Performance Data Analytics ======

Data-driven science requires the handling of large volumes of data in a quick period of
time. Executing efficient workflows is challenging for users but also for systems. This
module introduces concepts, principles, tools, system architectures, techniques, and
algorithms toward large-scale data analytics using distributed and parallel computing.
We will investigate the state-of-the-art of processing data of workloads using solutions in
High-Performance Computing and Big Data Analytics.

===== Key information =====

|| Contact || [[about:people:julian_kunkel|Julian Kunkel]], [[about:people:jonathan_decker|Jonathan Decker]] ||
|| Location || [[https://meet.gwdg.de/b/jul-35w-lfd-xfv|Virtual]] ||
|| Time || Monday 16:15-17:45 (lecture), Monday 12:15-13:45 (lunch exercise, starts 1 week later) ||
|| Language || English ||
|| Module || Modul B.Inf.1712: Vertiefung Hochleistungsrechnen, Module M.Inf.1236: High-Performance Data Analytics  ||
|| SWS || 4 ||
|| Credits || 6 ||
|| Contact time || 56 hours ||
|| Independent study || 124 hours ||
|| Exam || Written date: 14.02.2025 - 14:00-16:00, 2nd exam: 21.03.2025 - 14:00-16:00, both in [[https://lageplan.uni-goettingen.de/?ident=1487_1_EG_0.101|0.101 Provisorium]] ||

Please note that we plan to record sessions (lectures and seminar talks) with the intent of providing the recordings
via BBB to other students but also to publish and link the recordings on YouTube for future terms.
If you appear in any of the recordings via voice, camera or screen share, we need your consent to publish the recordings.
See also this {{ :teaching:templates:dataprivacy_student_notice_slide.pdf |Slide}}.

===== Topics =====

Topics cover:
  * Challenges in high-performance data analytics
  * Use-cases for large-scale data analytics
  * Performance models for parallel systems and workload execution
  * Data models to organize data and (No)SQL solutions for data management
  * Industry relevant processing models with tools like Hadoop, Spark, and Paraview
  * System architectures for processing large data volumes
  * Relevant algorithms and data structures
  * Visual Analytics
  * Parallel and distributed file systems

Weekly laboratory practicals and tutorials will guide students to learn the concepts
and tools. In the process of learning, students will form a learning community and
integrate peer learning into the practicals. Students will have opportunities to present
their solutions to the challenging tasks in the class. Students will develop presentation
skills and gain confidence in the topics.


===== Learning Objectives =====

  * Assign big data challenges to a given use-case
  * Outline use-case examples for high-performance data analytics
  * Estimate performance and runtime for a given workload and system
  * Create a suitable hardware configuration to execute a given workload within a deadline
  * Construct suitable data models for a given use-case and discuss their pro/cons
  * Discuss the rationales behind the design decisions for the tools
  * Describe the concept of visual analytics and its potential in scientific workflows
  * Compare the features and architectures of NoSQL solutions to the abstract concept of a parallel file system
  * Appraise the requirements for designing system architectures for systems storing and processing data
  * Apply distributed algorithms and data structures to a given problem instance and illustrate their processing steps in pseudocode
  * Explain the importance of hardware characteristics when executing a given workload

===== Examination =====

Written (90 Min.) or oral (ca. 30 Min.) -> depends on the number of attendees (typically written).

See the learning objectives.


===== Agenda =====

  * 21.10.24 - **Lecture Overview. Use Cases.** \\ {{ :teaching:autumn_term_2024:hpda:hpda-01.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-01.pdf |Exercises}}
    * Exercise: Discussion of use cases covering business/industry and science. Sketching the analytics pipeline for a use case.
  * 28.10.24 - **Data Models and Data Processing Strategies** \\ {{ :teaching:autumn_term_2024:hpda:hpda-02.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-02.pdf |Exercises}}
    * Exercise: Developing data models for selected use cases. Researching performance for HPDA. Python Word-Count.
  * 04.11.24 - **Databases and Data Warehouses** \\ {{ :teaching:autumn_term_2024:hpda:hpda-03.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-03.pdf |Exercises}}
    * Exercise: Developing a database schema and SQL queries.
  * 11.11.24 - **Distributed Storage and Processing with Hadoop** \\ {{ :teaching:autumn_term_2024:hpda:hpda-04.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-04.pdf |Exercises}}
    * Exercise: MapReduce processing with Python. Sketching the difference between SQL running via Hadoop (and Hive) vs. a traditional relational database vs. a data warehouse
  * 18.11.24 - **Dataflow Computation and Big Data SQL using Hive** \\ {{ :teaching:autumn_term_2024:hpda:hpda-05.pdf |Slides}}{{ :teaching:autumn_term_2024:hpda:hpda-06.pdf |Slides 2}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-05.pdf |Exercises}}
    * Exercise: MapReduce via Streaming in Hadoop. Developing a dataflow system in Python.
  * 25.11.24 - **Columnar Access and Document Storage** \\ {{ :teaching:autumn_term_2024:hpda:hpda-07.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-06.pdf |Exercises}}
    * Exercise: Managing data using MongoDB
  * 02.12.24 - **In-Memory Computation** \\ {{ :teaching:autumn_term_2024:hpda:hpda-08.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-07.pdf |Exercises}}
    * Exercise: Data processing using Spark
  * 09.12.24 - **Stream Processing ** \\ {{ :teaching:autumn_term_2024:hpda:hpda-09.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-08.pdf |Exercises}}
    * Exercise: Streaming concepts and crime data
  * 16.12.24 - **Designing Distributed Systems and Performance Modelling** \\ {{ :teaching:autumn_term_2024:hpda:hpda-10.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-09.pdf |Exercises}}
    * During the exercise, we discuss any questions you may have.
    * Exercise: RESTful Services. Consistent Hashing. Performance analysis of I/O mappings of use cases to systems.
  * 06.01.25 - **Visual Analytics and Large-Scale Data Analysis** \\ {{ :teaching:autumn_term_2024:hpda:hpda-11.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-10.pdf |Exercises}}
    * Exercise: Performance analysis and OpenAPI
  * 13.01.25 - **Data Storage** \\ {{ :teaching:autumn_term_2024:hpda:hpda-12.pdf |Slides}} {{ :teaching:autumn_term_2024:hpda:hpda-exercise-11.pdf |Exercises}}
    * Exercise: NetCDF data model and benchmarking.
  * 20.01.25 - **The Apache Ecosystem and Beyond** \\ //This slide deck is optional and not subject to examination// \\ {{ :teaching:autumn_term_2024:hpda:hpda-13.pdf |Slides}}
    * Exercise: discussion of last week's exercises
  * 27.01.25 - **Summary** \\ {{ :teaching:autumn_term_2024:hpda:hpda-14.pdf |Slides}}
    * Exercise: Q&A Session

===== Links =====
  * Example Scripts: https://github.com/JulianKunkel/hpda-samples