Dot product example on p 58 fig 3.8

[richlevinson-design-2]

I bought this book based on reviews and believe it is highly regarded.
I have an MS in Physics and know what a dot product is.
On p 58 fig 3.8 there are 3 dot products shown x1.x2, x2.x2, x3.x2.
I cannot get any of them to match the result.
For example x1.x2 should = .4x.5 + .1x.8 + .8x.6 = .20 + .08 + .48 = .76 = w21
but answer in fig 3.8 says w21=0.9, which does not equal 0.76.

I know this is a nit, but I am new to LLM's and trying to understand this stuff for the first time and if I can't understand how the dot products are calculated it puts me on a very unfirm foundation for continuing to read the material.

Thanks,
Rich

[rasbt]

Hi Rich,
thanks for your interest in the book.

The quick answer here is that the numbers in the figures are truncated so that it could all fit into the figure with a reasonable font size.

I remember considering / trying 3 different options:

1. The full numbers (figures were too crammed and font size was too small)
2. Truncating (has the issue that numbers in the figures can only be used for referencing the code, but you get different results if you manually multiply those, like what you observed in your case)
3. Rounding (it made it harder to cross-reference the numbers and also had the same issue as the truncation)

I hope this helps, and please let me know in case you have any follow-up questions.

[richlevinson-design-2]

Thank you so much for getting back to me on this.
I am extremely interested on learning how these LLM's work.
I will spend some time reviewing your reply and try to get some python code going to repeat the examples and see if I can get the attention weights to come out (and try to understand how the normalization process works).
What really threw me was that I couldn't get any of the 3 dot products to work, and I did read on p57 about the rounding and couldn't get that to work on x1.x2 even w rounding, so I was very unsure if I was understanding correctly.
Again, thank you very much for your quick response.
I will get back to you after I have a chance to work through this a little more.

Rich

[rasbt]

Happy to help!

So the key tip here is, if you want to calculate the dot products by hand in this chapter, don't use the values shown in the figure to do the calculations. Instead, use the numbers in the text. The numbers in the figure are just showing the truncated values and would give you the wrong results (due to the truncation) if you multiply and add them.

Anyways, I hope you enjoy the book!

[richlevinson-design-2]

Hi Sebastian (I assume it's you :) ),

Thanks for your tips on the dot product in your book (I was reading ahead
to try to get a quick take on the "attention" concepts).
I have begun ch 2 and it's time to get serious about the python.

I have begun reading App A, and kind of got stuck on whether to do the
CPU or GPU version of PyTorch. I have an older pc (~2017, I think), but
I did go to task manager and found that it has on the Performance: GPU 0
page that it has an "Intel HD Graphics 620" card, which shows:

"Shared GPU/Memory" = 0.3/15.9 GB w utilization = 1%.
Also my python version is 3.13.9, Oct 14 2025.

Is that enough info for you to suggest that I use the GPU version of PyTorch,
as it indicates you recommend on top of p 255 in App A?

Thanks,
Rich

[rasbt]

Hi Rich,

yes, that's me (Sebastian) :).

Regarding your case, that would definitely be the CPU version. The PyTorch GPU version is actually pretty specific to NVIDIA CUDA GPUs. (Although, AMD cards are now also getting better support.)

I.e., the PyTorch GPU version includes a bunch of CUDA code and CUDA itself (which is why it's almost 1 GB in size). Since CUDA code runs only on NVIDIA GPUs, downloading the PyTorch GPU version would still work on your computer (it would just run in the default CPU mode), but it would be a bit wasteful in terms of storage space.

[richlevinson-design-2]

Hi Sebastian,
I am still working my way thru your book. Just starting ch 5, but have reviewed ahead and App A for neural networks etc.
I think the book is terrific and it is my anchor for learning about LLM's.
One question I have is about the embedding vectors and if there are actually 2 classes of vectors that are used in the book.
To clarify: on p 95, where the frequently used configuration, GPT_CONFIG_124M, is defined, there are 2 values:
"vocab_size": 50257
"emb_dim": 768
These 2 values keep reappearing in the book and I am finding some ambiguity between them.
On p95, "emb_dim" is described as "embedding size, transforming each token into a 768-dimensional vector".
That's ok, but then, on p99, we have: model outputs: logits, followed by explanation:
"Each text sample consists of four tokens; each token is a 50,257-dimensional vector, which matches the size of the tokenizer's vocabulary."
It goes on: "The embedding has 50,257 dimensions because ...".
So, we appear to have 2 classes of embeddings: 768 dim and 50257 dim corresponding to the two config variables above.

What I am taking from all this is the 768 seems ot apply to the token embeddings used in the neural network part of the architecture, and the 50,257 seems to apply primarily when the input is first encoded from text to tokens then embeddings, and when the output is finally decoded from embeddings to tokens then text.

Where the ambiguity comes in for me is I don't see any clear distinction where the 50,257 initial encoding switches to the 768 for neural net processing, and on the output side where the 768 dim vectors get transitioned back to the 50,257 vectors that are appended to the output string.

I can see the 50,257 has value because it represents the probability that the next token will be one of the 50,257 words or subwords in the vocab: i.e. all the probs add up to 1 and the largest of these is a good candidate for the output token to add to the response.

So, generally, my question is, for example, could someplace like p94, fig 4.4, be used to show a box where 768-dim tokens get transformed to 50,257 on output, and another box where the 50,257-dim tokens get transitioned to 768 for input to the neural net stack?

Thanks,
Rich

[richlevinson-design-2]

Note: started new discussion for above topic, because I'm a little unfamiliar w the mechanics of using this discussion app.

[rasbt]

No worries at all. Let me link the other discussion here for reference: #1034