Word Embeddings
What are word embeddings and why use them?
Embeddings map each word to a dense vector where geometry encodes meaning: similar words sit close, and directions encode relationships. Famously, king − man + woman ≈ queen.
The vector from man→woman is parallel to king→queen — the embedding learned a consistent “gender” direction. Relationships become arithmetic.
- “How is similarity measured?” — Cosine similarity — the angle between vectors. (See Embeddings & Vector DBs.)
How does Word2Vec learn them?
From the distributional hypothesis: words in similar contexts have similar meanings. Word2Vec trains a shallow network to predict context words, and the learned weights become the embeddings.
- Predicts the center word from its context.
- Faster; better on frequent words.
- Predicts the context from the center word.
- Slower; better on rare words and small data.
- “What is negative sampling?” — A speedup that trains against a few random “wrong” words instead of the whole vocabulary.
Word2Vec vs GloVe?
Both produce static vectors with similar quality. Word2Vec is predictive (local context windows); GloVe is count-based, factorizing a global word co-occurrence matrix.
- “What is FastText?” — It embeds sub-word character n-grams, so it handles rare and out-of-vocabulary words.
- “Which to choose?” — In practice, pretrained contextual embeddings now beat both for most tasks.
What's the limitation of static embeddings?
Each word gets one fixed vector, so polysemy breaks: “bank” (river) and “bank” (money) share the same embedding. Contextual models (BERT, GPT) give a word a different vector per sentence.
- “What makes embeddings contextual?” — Self-attention blends each token with its neighbors, so the vector depends on context.
- “Are static embeddings obsolete?” — Still great when you need speed and a fixed per-word lookup (e.g. classic classifiers).