8 Advanced parallelization - Deep Learning with JAX

Using easy-to-revise parallelism with xmap() · Compiling and automatically partitioning functions with pjit() · Using tensor sharding to achieve parallelization with XLA · Running code in multi-host configurations

Lecture 2: Development Infrastructure & Tooling - The Full Stack

Tutorial 6 (JAX): Transformers and Multi-Head Attention — UvA DL

Compiler Technologies in Deep Learning Co-Design: A Survey

Efficiently Scale LLM Training Across a Large GPU Cluster with

A Brief Overview of Parallelism Strategies in Deep Learning

Intro to JAX for Machine Learning, by Khang Pham

Scaling Language Model Training to a Trillion Parameters Using

Scaling Language Model Training to a Trillion Parameters Using

What is Google JAX? Everything You Need to Know - Geekflare

Training Deep Networks with Data Parallelism in Jax