• We explored model evaluation across two domains
• Traditional ML metrics: Accuracy, precision, recall
• The accuracy paradox: 99.9% accuracy can be useless!
• Overfitting: When models memorise instead of learn
• Cross-validation: More robust evaluation
• LLM evaluation: Perplexity, LLM-as-a-Judge, G-Eval
• Hallucinations and RAG for grounding answers
• Today: How do LLMs actually understand language? 🤔

Part 1: How LLMs “See” Text

• The translation problem: Text to numbers
• The processing pipeline

Part 2: Tokens and Tokenisation

• What tokens are (hint: not always words!)
• Tokenisation methods and quirks
• Token limits and API costs

Part 3: Embeddings

• Words as vectors in space
• Semantic similarity and the king-queen example in detail

Part 4: Parameters

• What are the billions of parameters?
• Embeddings, weights, and biases
• Temperature and creativity dials

Source: https://www.anthropic.com/research/AI-assistance-coding-skills

• Here’s something you probably know by now: Computers only understand numbers
• When you type “Hello, how are you?”, the LLM doesn’t see letters!
• It sees something like: [15496, 11, 703, 527, 499, 30]
• This creates a translation problem:
  • How do we convert text into numbers?
  • How do we preserve meaning in those numbers?
  • How do we capture relationships between words?
• The solution involves two key concepts:
  • Tokenisation: Breaking text into pieces
  • Embeddings: Converting pieces into meaningful vectors

Understanding the pipeline helps you:

• Write better prompts: Know what the model “sees”
• Understand costs: API pricing is per token, not per word
• Avoid surprises: Some words use more tokens than others
• Debug issues: Why did the model misunderstand?
• Use context wisely: Token limits are real constraints

• A token is the basic unit that an LLM reads
• Tokens are NOT always words!
• A token can be:
  • A full word: “hello” → 1 token
  • Part of a word: “unhappiness” → “un” + “happiness” (2 tokens)
  • A single character: “!” → 1 token
  • A space: ” ” → often included with the next word
• Rule of thumb: 1 token ≈ 4 characters in English
• Or roughly: 100 tokens ≈ 75 words
• Other languages often use more tokens per word
• Let’s try it out with “Hello, it’s Danilo here!”
• We’ll use OpenAI’s tokenizer for this example

Three approaches to breaking up text:

• Modern LLMs use Byte Pair Encoding (BPE)
• Common subwords become single tokens
• Rare words are split into known pieces
• This is why “ChatGPT” → [“Chat”, “G”, “PT”]
• How does this save space?
  • Example: “unbelievable”, “unhappy”, “undo”, and “unknown” all share the same “un”
  • The model only needs to store “un” once!

OpenAI’s Tokeniser (try it!): platform.openai.com/tokenizer

The number of tokens can change from one version to another!

Some surprising examples:

Notice: Capitalisation, spacing, and punctuation all affect tokenisation!

• Every LLM has a context window: Maximum tokens it can process
• This limit includes both your prompt AND the response!
• Output tokens typically cost 2-4x more than input tokens!

• If your prompt uses 3,500 tokens and the limit is 4,096…
• You only have 596 tokens left for the response!
• Exceeding limits → Error or truncated output
• Curiosity: Non-English languages use more tokens
  • Chinese, Japanese, Korean: ~2-3x more tokens per concept
  • This means higher costs and shorter effective context

• Once text is tokenised, each token gets converted to an embedding
• An embedding is a vector: A list of numbers
• Typically 768 to 4,096 numbers per word!
• Example: “cat” → [0.23, -0.45, 0.12, -0.89, ... (4,096 numbers)]
• These numbers capture the meaning of the word
• Similar words have similar vectors
  • “cat” and “dog” will be close together
  • “cat” and “aeroplane” will be far apart
• The model learns these representations during training
• This is where the “understanding” happens!
• Remember \(\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}\)

• Word2Vec (2013, Google’s Mikolov et al.) revolutionised NLP
• Two training approaches:
  • CBOW (Continuous Bag of Words): Predict word from context
  • Skip-gram: Predict context from word
• Key insight: Words appearing in similar contexts have similar meanings
  • “I love my _____” → cat, dog, hamster are all likely
  • So cat, dog, hamster should be near each other!
• Trained on billions of words from the web
• Created the embedding approach modern LLMs use
• 🎬 Watch: Word2Vec Explained (10 min)

• Imagine we have the word embeddings for a few words
• The colours represent numbers from -2 to +2, obtained from our model
• See how “woman” and “girl” are similar to each other, but “queen” is further away?
• This tell us something!
• Although we don’t know what each dimension codes for, we can see some patterns
• There are clear places where “king” and “queen” are similar to each other and distinct from all the others
• The model has learned something about gender and royalty!
• Note that “water” has few connections to other words

How embeddings power modern AI:

• Semantic search 🔍
  • Search by meaning, not just keywords
  • “cheap flights” finds “budget airfare”
• Recommendations 📚
  • Find similar products, articles, or content
  • “Users who liked X also liked Y”
• Clustering 📊
  • Group similar documents automatically
  • Topic modelling without labels
• RAG (Retrieval-Augmented Generation)
  • Find relevant documents for LLM context
  • Power behind ChatGPT + browsing/search

• Think back to algebra: \(y = 2x + 3\)
• The numbers 2 and 3 are parameters
• They determine the behaviour of the function
• In an LLM, parameters are numbers that:
  • Get set during training
  • Control how the model processes text
  • Determine what the model “knows”
• GPT-3: 175 billion parameters
• GPT-4: estimated 1+ trillion parameters
• Each parameter is adjusted thousands of times during training
• Training = finding the right numbers to make predictions accurate

1. Embedding parameters

• Store the learned vector for each token
• Vocabulary (~50,000) × dimensions (~4,096)
• = ~200 million parameters just for embeddings!

2. Weight parameters

• Control connections between neurons
• Determine how strongly different parts influence each other
• Like volume knobs: amplify or diminish signals

3. Bias parameters

• Adjust thresholds for neuron activation
• Help detect subtle patterns that might otherwise be missed
• Like a baseline adjustment

TL;DR: Weights and biases together determine how the model processes information

The scaling hypothesis:

• More parameters = more capability (generally)
• But diminishing returns set in
• A 10x bigger model isn’t 10x smarter

Small models fighting back:

• Efficient training: Better data, longer training
• Distillation: Teach small models from big ones
• Quantisation: Reduce precision (32-bit → 4-bit)
• Mixture of Experts: Only use relevant parts

Not all parameters are learned!

Hyperparameters are settings YOU control:

• Temperature 🌡️
  • Controls randomness/creativity
  • Low (0.1-0.3): Focused, deterministic
  • High (0.7-1.0): Creative, varied
  • 0.0: Always picks most likely word
• Top-p (nucleus sampling)
  • Only consider words in top p% probability
  • 0.9: Consider top 90% of likely words
• Top-k
  • Only consider top k most likely words
  • k=50: Choose from 50 most likely words

• Attention is the key innovation of transformers
• Every word “looks at” every other word
• Determines which words are relevant to each other

Example:

“The animal didn’t cross the street because it was too tired”

• What does “it” refer to?
• Attention helps the model connect “it” to “animal”
• Not all words matter equally for understanding

How it works:

• Each word asks: “Who should I pay attention to?”
• Creates weighted connections between all words
• Allows understanding of long-range dependencies

Now that you understand the internals:

• ✅ Be concise: Fewer tokens = lower cost, more room for response
• ✅ Use clear language: Common words tokenise efficiently
• ✅ Provide context: Help the attention mechanism
• ✅ Be specific: The model uses your exact words
• ✅ Test different temperatures: Match creativity to task
• ✅ Mind the context limit: Plan for both prompt AND response

When things go wrong:

• Model confused? → Try rephrasing with different words
• Model doesn’t understand? → Provide more context
• Running out of context? → Summarise earlier content