Embeddings
Date Published
Sometimes the models might need to see huge amounts of documents for certain use-cases but are limited by their resources even with a huge context window. This is were Embeddings come into play. Instead of storing words as texts, embeddings are used to translate them to vectors. Two sentences using different words might have the same meaning so they might have the same embedding. This is about semantic similarity. LLMs don’t understand words — they understand relationships between numbers. They compare embedding distances, not keywords.
These sentences would have very similar embeddings:
“I love machine learning”
“I enjoy studying ML”
“Machine learning is fascinating to me”
Even though the words differ, the meaning is similar, so their vectors are close.
Meanwhile:
“I love machine learning”
“I hate waiting in traffic”
→ embeddings are far apart.
🔹 Handle ambiguity (context!)
The word “bank” has different embeddings depending on context:
“I sat by the river bank”
“I went to the bank to withdraw money”
Same word → different embeddings, because context reshapes meaning.
This is crucial for LLM reasoning.
Embeddings + context window (together)
Inside an LLM:
Text → tokens
Tokens → embeddings
Embeddings are placed inside the context window
Attention layers compare embeddings to decide:
what’s important
what relates to what
Without embeddings:
the model sees only symbols
no meaning, no similarity, no structure
Key Takeaway:
Embeddings are how language becomes geometry — meaning is represented by distance, not definitions.