Jun 24Understanding the energy spectrum“Anything that moves by itself without external stimuli contains life energy — karvai” The unknown origin The universe is filled with energy whose source of origin is unknown. It is now understood that the matter we observe across the universe such as planets, stars, meteors, and nebulas were created from the same energy…4 min read4 min read
May 17From Exascale, towards building Zettascale general purpose & AI SupercomputersRecently, I saw an interesting slide (shown above) presented by AMD’s CTO Mark Papermaster, showing a need for a 40x improvement in energy efficiency in computing by 2035. …8 min read8 min read
May 15How long will it take to train an LLM model like GPT-3?Teraflop/s is 10¹² computations per second, Petaflop/s is 10¹⁵ computations per second, Exaflop/s is 10¹⁸ computations per second, and Zetaflop/s 10²¹ computations per second. See below a table, in reference to the original GPT-3 paper, showing the time complexity of LLM models.2 min read2 min read
May 15Building an Exaflop Machine for Deep LearningIn this blog, we’ll see how to build an Exaflop machine using Nvidia’s GPU H100 based on its latest Hopper architecture. Nvidia H100 is manufactured using a TSMC 4nm node, 814 sq mm, with 700 W TDP, peak performance of 4 PFLOPs (int8/fp8) with energy efficiency at 5.7 TFLOPs/watt. 256…2 min read2 min read
