How to Differentiate Low Level Code for GPUs and TPUs

Array Cast - July 21, 2023
Show Notes

Thanks to Bob Therriault, Conor Hoekstra, and Adam Paszke for gathering these links:

[01] 00:01:49 J on AWS

https://code.jsoftware.com/wiki/System/Installation/Cloud

              KXcon '23 videos https://kx.com/kx-con-23-recordings/

              KXcon '23 ArrayCast review episode https://www.arraycast.com/episodes/episode54-kxcon23

[02] 00:04:10 PyTorch https://pytorch.org/
              Dex Programming Language

https://github.com/google-research/dex-lang/

              Coursera https://www.coursera.org/

[03] 00:07:48 Torch7 https://github.com/torch/torch7
              Torch

https://en.wikipedia.org/wiki/Torch_(machine_learning)

              Lua Programming Language https://www.lua.org/

              Python Programming Language https://www.python.org/

              Alpha Go documentary https://en.wikipedia.org/wiki/AlphaGo_(film)

              Tensorflow Programming Language https://www.tensorflow.org/

              LuaJIT interpreter  https://luajit.org/

[04] 00:15:43 Automatic differention https://en.wikipedia.org/wiki/Automatic_differentiation

[05] 00:18:25 GPU https://en.wikipedia.org/wiki/Graphics_processing_unit

              TPU https://en.wikipedia.org/wiki/Tensor_Processing_Unit

4
[06] 00:20:56 Machine Learning https://en.wikipedia.org/wiki/Machine_learning

[07] 00:29:30 AutoGrad Library https://autograd.readthedocs.io/en/latest/
              JAX https://github.com/google/jax

              Vectorization https://en.wikipedia.org/wiki/Vectorization

[08] 00:33:10 Dex White Paper 4 pages https://openreview.net/pdf?id=rJxd7vsWPS

              Dex White Paper 20 pages https://arxiv.org/pdf/2104.05372.pdf

              Dex Tutorial https://google-research.github.io/dex-lang/examples/tutorial.html

              MatLab Programming Language https://www.mathworks.com/products/matlab.html

[09] 00:35:40 Rank Polymorphism https://en.wikipedia.org/wiki/Parametric_polymorphism#Higher-rank_polymorphism

[10] 00:37:42 Futhark Programming Language https://futhark-lang.org/

              Dex and Futhark https://futhark-lang.org/blog/2020-12-28-futhark-and-dex.html

              ArrayCast Episode with Troels Henriksen https://www.arraycast.com/episodes/episode37-futhark

[11] 00:39:00 Haskell Programming Language https://www.haskell.org/

[12] 00:40:30 Ad Hoc Polymorphism https://en.wikipedia.org/wiki/Ad_hoc_polymorphism

              Parametric Polymorphism https://en.wikipedia.org/wiki/Parametric_polymorphism

              NumPy Python Library https://numpy.org/

[13] 00:46:15 CUDA GPU Language https://en.wikipedia.org/wiki/CUDA

[14] 00:53:13 XLA JAX compiler https://hpc.nih.gov/apps/JAX.html

[15] 00:56:57 Triton OpenAI https://openai.com/research/triton

[17] 01:01:44 Languish language tracker https://tjpalmer.github.io/languish/

              LLMs https://en.wikipedia.org/wiki/Large_language_model

[18] 01:05:43 github/google/Jax
              github/google-research/dex-lang

[19] 01:08:00 Nix https://docs.haskellstack.org/en/stable/nix_integration/

[20] 01:09:25 Adam's secret project - TBA

[21] 01:10:28 Contact AT ArrayCast DOT com