diff --git a/deep-learning/20231115.FP-precision-notes.md b/deep-learning/20231115.FP-precision-notes.md new file mode 100644 index 0000000..d76bb26 --- /dev/null +++ b/deep-learning/20231115.FP-precision-notes.md @@ -0,0 +1,28 @@ +Notes on Floating Point Precisions in Deep Learning Computations +================================================================ + +ECCV 2020 Tutorial on Accelerating Computer Vision with Mixed Precision +----------------------------------------------------------------------- + +https://nvlabs.github.io/eccv2020-mixed-precision-tutorial/ + +Topics of the tutorial: + + * Training Neural Networks with Tensor Cores + * PyTorch Performance Tuning Guide + * Mixed Precision Training for Conditional GANs + * Mixed Precision Training for FAZE: Few-shot Adaptive Gaze Estimation + * Mixed Precision Training for Video Synthesis + * Mixed Precision Training for Convolutional Tensor-Train LSTM + * Mixed Precision Training for 3D Medical Image Analysis + +Has PDF of the slides and the videos. + +Q&A: + +**What's the difference between FP32 and TF32 modes?** +FP32 cores perform scalar instructions. TF32 is a Tensor Core mode, +which performs matrix instructions - they are 8-16x faster and more +energy efficient. Both take FP32 as inputs. TF32 mode also rounds +those inputs to TF32. +