Learn the basics

Learn how AI runs on GPUs

Eighteen short lessons that follow one request from the words you type to the answer you read, and show how a GPU does the work along the way.

Open the animated version

Start here

  1. 1Why use a GPU?A CPU runs your app. A GPU helps with lots of similar calculations at once.
  2. 2Where the numbers are keptMemory stores numbers; how much it holds and how quickly it moves them are two different things.
  3. 3A chip, a computer or a whole rack?A chip is one part, a computer combines parts, and a rack holds a much larger group of equipment.

Your words become numbers

  1. 4How your text becomes piecesBefore a model sees your text, software splits it into pieces called tokens and swaps each piece for a number.
  2. 5Numbers that carry meaningEach token ID is swapped for a learned list of numbers, so pieces used in similar ways end up with similar lists.
  3. 6The model's knowledge and its working notesThe model reuses its learned numbers while keeping separate working notes for the text it is processing.

How a reply is built

  1. 7Read the prompt, then build the answerThe model reads your prompt to choose the first answer piece, then uses each new piece to choose another.
  2. 8Mixing information from earlier textAttention mixes information from allowed text positions, giving each position a calculated share in the result.
  3. 9Working on one piece at a timeAfter attention mixes information between pieces, a second part of the block reworks each piece on its own.
  4. 10The same block, many times overThe same shape of block repeats many times, each with its own learned numbers, and every piece passes through all of them.
  5. 11Multiplying grids of numbersAI models often multiply grids of numbers, and Tensor Cores are GPU parts built for supported versions of this math.

Choosing the words

  1. 12Choosing the next pieceEvery piece in the vocabulary gets a score, the scores become shares, and a setting decides which piece is taken.
  2. 13Numbers become words againEach chosen piece is turned back into characters and sent on straight away, so an answer appears while it is still being made.

Look a little closer

  1. 14How a GPU groups its workA GPU groups small pieces of work together, then runs those groups on parts of the chip called SMs.
  2. 15Using less space for each numberQuantization saves space by using fewer bits, the tiny units of computer data, to store each number.
  3. 16More replies or a shorter wait?A server can produce more total output while each person still waits longer for a reply.
  4. 17When GPUs need to share resultsGPUs sharing one model's work may need to send partial answers to one another.
  5. 18A big speed number needs a real testA big hardware speed number does not tell you how quickly a real chat app will answer.