A wide ranging talk that includes a lot of the new work on ExecuTorch for on-device support, including the torch.export stack. The talk really goes through the whole flow from graph capture, lowering and optimizations so its a great way to get a sense for what is happening within PyTorch!
https://blog.christianperone.com/2023/12/pytorch-2-internals-talk/
https://www.hpcwire.com/2023/10/05/how-amd-may-get-across-the-cuda-moat/
CUDA is a huge advantage for NVidia, and is really baked in to a lot of workflows (PyTorch being a part of that!)
Having a good quality ROCm backend makes porting significantly easier – AMD have made significant efforts on testing and support too. Also, they have generally structured their software to mirror CUDA, which makes switching fairly seamless. It’ll be interesting to see how folks reading to the MI300, and the opportunities given that much HBM per card!
The PyTorch composability meeting is one of the most interesting discussions of how the various parts of PyTorch fit together, particularly in the context of the PyTorch compiler and associated technologies. The meetings themselves are public, and so are the meeting notes, which are a great reference point.
https://docs.google.com/document/u/0/d/1QTR3t3KdRu5JT1lvAuJLsPd3LruCfv0LecpsgR8eWhg/
The kernel dispatch time eats a lot of performance on GPU – CUDA graphs let you chain a bunch of kernels together, and they’re now more accessible from PyTorch:
CUDA Graphs, which made its debut in CUDA 10, let a series of CUDA kernels to be defined and encapsulated as a single unit, i.e., a graph of operations, rather than a sequence of individually-launched operations. It provides a mechanism to launch multiple GPU operations through a single CPU operation, and hence reduces the launching overheads.
https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/