The word “bank” means something completely different in “river bank” and “bank account”. The classic Word2Vec assigns the same vector.
Word2Vec learns a single fixed representation for each word in its vocabulary. It captures broad semantic similarity well - “king” is close to “queen” (sorry, cliche example) - but it collapses all meanings of a word into one point in vector space.
The embedding itself never changes based on the sentence where the word appears. But words change their meaning as per the context in which they are used.
Recurrent Neural Networks were the first major step toward fixing this. Then came Transformers.
Transformers use self-attention, allowing every token to interact directly with every other token in the sentence. The representation of “bank” is computed using relationships across the entire sequence, enabling much richer contextual understanding.
BERT put this idea to work at scale and almost became a default for NLP systems.
Modern embedding models extend this idea even further. OpenAI embeddings and other modern ones are also built on Transformer foundations, but are trained specifically to produce high-quality semantic representations for search, retrieval, recommendation, and RAG.
i.e., words with similar meaning are mapped close together in vector space, even when the wording is very different.
“How do I bake bread?” and “What is the process for making a loaf?” may share a few keywords, yet modern embeddings place them near each other because they express nearly the same intent.
Static embeddings represent words. Modern embeddings represent meaning in context.