Heat: scaling the Python scientific stack to HPC systems

Memory bottleneck in scientific computing (4 minutes) - Limitations of single-node libraries - Complexity of existing workarounds: trade-offs between manual MPI programming (high developer effort) and task-parallel frameworks - The data-parallel alternative: performing uniform operations on distributed slices of a global tensor.

Architecture and implementation (8 minutes) - The DNDarray structure: Technical breakdown of the distributed n-dimensional array, which provides a global logical view while managing local physical storage across MPI ranks. - The split axis concept: How data is partitioned along specific dimensions (e.g., rows or columns) to optimize communication for different mathematical operations. - Backend synergy: - PyTorch as the compute engine for high-performance local tensor operations and GPU acceleration. - mpi4py for communication in cluster environments. - Hardware interoperability: Transparent execution across CPUs and GPUs, including NVIDIA (CUDA) and AMD (ROCm) accelerators.

Algorithmic building blocks for distributed memory (8 minutes) - Communication-aware linear algebra: Distributed matrix-matrix multiplication and its communication costs. Advanced matrix decomposition methods, such as hierarchical and randomized SVD (hSVD), for massive datasets. - Scalable machine learning and statistics: Example: clustering (K-Means) and Principal Component Analysis (PCA) on distributed arrays. - Temporal analysis using Dynamic Mode Decomposition (DMD) on large-scale scientific data like global wind speeds.

Performance and scaling efficiency (7 minutes) - Scaling methodologies: strong scaling (speedup for a fixed problem size) and weak scaling (efficiency as both problem size and resources grow). - Memory wall removal: Utilizing the cumulative RAM of many cluster nodes to process datasets that are otherwise impossible to load. - Case studies: Reviewing performance results from large-scale runs

Summary and project roadmap (3 minutes) - Key takeaways - Upcoming features - Open-source community