This is part of an ongoing series where I revisit influential papers in NLP. Each post will briefly summarize the core ideas, highlight what I found interesting (or confusing), and sometimes include small experiments. The goal isn’t to be exhaustive.

Attention is All You need (Vaswani et al., 2017)

Written in Oct 30, 2025

Many people consider this the landmark paper that ignited the Transformer revolution.

This is the mechanism that started it all. Much have changed since then: I will focus on what I found interesting when reading the paper again.

About Attention Mechanism

The two most commonly used attention functions are additive attention [2], and dot-product (multiplicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor of \(\sqrt{d_k}\). Additive attention computes the compatibility function using a feed-forward network with a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.

This shows that the idea of attention itself was not new. Previous paper also introduced attention mechanism. It was used as a middle layer connecting encoder and decoder RNNs. The novelty of this paper was showing that attention mechanism alone can be used to replace RNNs.

That said, I think that the title “Attention Is All You Need” is slightly misleading. When I first read the paper, I assumed attention parameters dominated model size. In reality, most parameters come from the feed forward layers (FFN) within each attention block. Many LLMs use 4 times dimension expansion in the FFN layer, with some recent MoE models even more aggressively expanding the FFN dimensions. (95%+ of parameters in case of OpenAI OSS models).

Parameter share numbers are estimates if we use 4x dimension expansion in FFN layers with standard attention.

Furthermore, works have focused on the calculation of the attention weights, which is \((O(n^2 \cdot d))\). However, the feed-forward layers also have complexity of \((O(n \cdot d^2))\), Since in the earlier days of transformers, \(d\) was often larger than \(n\), the feed-forward layers were more computationally expensive on simple tasks.

About Positional Embeddings

The paper introduced sinusodial positional embedding. That is,

where pos is the position and i is the dimension. That is, each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 · 2π. We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, \(PE_{pos+k}\) can be represented as a linear function of \(PE_{pos}\)

So what does this mean? Simply, put, we can design a rotational matrix \(\mathbf{R}^k\) such that

Such rotational matrix can be created by the following block diagonal matrix.

This also means position embedding difference between two tokens are determined by its relative distance.

I personally find this formulation elegant but slightly lacking. We can express this as:

However, the attention mechanism is not additive. It uses dot product attention, and capturing additive positional difference seems unaligned to me. I would consider the following to be more intutive.

\(P(x, pos)\) here is a function that encodes the position \(pos\) and token embedding \(x\) together. A position encoding should have some function \(\mathbf{F}\) that allows shifting the position by \(k\) when applied. This too, however, is counte to the linear additive nature of the transformers. Thus one might want a position embedding that keeps the latent representation additive while making attention mechanism rotational. Rotary Position Embeddings (RoPE) satisfy this property.

The paper also notes that replacing the sinusoidal embedding with a learnable one barely affects performance.

Simple Experiments

1. Sinusodial Embedding vs Learnable.
2. Additive (Bahdanau-style) vs Dot product attention.

For the base transformer implementation, I used the Andrej Karpathy’s minGPT and evaluated based on the perplexity of the Shakespeare dataset. The code can be found here.

There was a minor difficulty in implementing the Bahdanau Attention. The original attention mechanism in the paper (Which connects encoder with decoder) uses the following

where \(v_a \in \mathbb{R}^n\), \(W_a \in \mathbb{R}^{n \times m}\), \(U_a \in \mathbb{R}^{n \times 2m}\). More detail can be found in the original paper.

To fit this implementation to Transformers, I modified the equation a bit.

Let head size be denoted \(h\). Then, we have \(v \in \mathbb{R}^{h}\), \(W_q \in \mathbb{R}^{h \times h}\), \(W_k \in \mathbb{R}^{h \times h}\).

Notice that \(v\) here is a 1d vector instead of a matrix as in transformer. This was a design choice to be closer to the original Bahdanau attention. The \(q,k\) vectors are same as the original transformer.

But this approach takes up too much memory in parallel computation. We need to materialize matrix of (B, T, T, h) tensor to parallelize adding the \(q,k\) vectors. So I used an alternative implementation.

This way, we do not need to materialize the full (B, T, T, h) tensor. The two implemtations are not equivalent, but I think this is a reasonable approximation.

Results

All models produced similar perplexity curves — confirming that both learnable vs. sinusoidal embeddings and additive vs. dot-product attention behave comparably on small datasets.

Takeaways

• Attention wasn’t new — the innovation was using attention-based blocks.
• FFN layers dominate parameters, and sometimes, compute — attention is important, but not the whole story.
• Sinusoidal embeddings are fixed position embedding which allow relative position encoding via rotation.

Chain of Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022)

Written on Nov 1, 2025

This paper is widely regarded as the beginning of the “reasoning” era in large language models. Chain-of-Thought (CoT) prompting demonstrated that simply adding step-by-step reasoning in prompts dramatically improves performance on arithmetic, commonsense, and symbolic tasks. CoT directly inspired later frameworks such as ReAct (reasoning + acting) and motivated the development of models explicitly trained on reasoning traces.
Building on this, researchers began using reinforcement learning on reasoning processes to create “thinking” or “reasoning” models—systems that learn to generate intermediate reasoning steps before producing a final answer. OpenAI’s o1 was one of the first marketed LLMs to incorporate these ideas.

Core Idea

Prompting LLMs with few-shot step-by-step reasoning examples significantly improves performance on complex reasoning tasks. The authors showed that this works on arithmetic, commonsense, and symbolic reasoning benchmarks.

This method worked better than standard few-shot prompting, especially as the model size increased. Models with size less than 10B often degraded in performance with CoT while larger models such as GPT-3 and PaLM 540B showed improved performance.

The authors hypothesize that the ability to reason might be an emergent behavior with model size.

Additional Tests

In order to see which aspect of the CoT prompting helped, the authors designed several prompt variants.

For GSM8k, they provided models with

1. Standard Prompting (few-shot)
2. Equations only, no reasoning steps.
3. Variable compute only, where the llm was given sequence of dots (…) equal to the original CoT.
4. Chain of Thought After the Answer.
5. Chain of Thought Before the Answer (original CoT).

The results show that only the original CoT prompting significantly improves performance. This suggests that the step-by-step reasoning process itself is crucial.

Personal Thoughts

I wonder if their claim about model size and emergence still holds true in 2025. We have different training pipelines: training data, model architecture and SFT -> RL is different from 2020. One major problem is that the comparison is not an apples to apples comparison, as recent training data often includes the reasoning rollout. I do think that retesting the claim with curated data, MoE model would be interesting.

Since CoT and its variants are popular methods for prompting, I assumed that some organization would have kept a record of the results. With this, I would have been able to compare the accuracy increase claims on newer models. I couldn’t find it, and decided to test on smaller models.

After being unhappy on my own CoT prompts and answer extractor, I used ElutherAI’s LM Evaluation Harness to prompt and evalute LLM responses from GSM8k. As a side note, I think the uncentralized evaluation for LLMs is quite problematic: Some of the small improvements can be due to a more broad regex extractor, or from the randomness of the evaluative LLMs.

The 0-shot Chain of Thought version here isn’t the popular Zero shot CoT (Often referred as just CoT in some context) but prompt with zero examplars. The left bars could be seen as standard prompting with 0-shot standard and Chain-of-Thought prompting can be seen as directly asking the question without any context. (The difference between the two should be the variance).

We see that CoT prompting hurts performance for Llama 3.2 1B model, while slightly improving Llama 3.2 3B and Qwen3 4B model. Furthermore, we see that standard prompting outperform the CoT version when prompted with the Llama 3.2 1B and Qwen3 4B models. The model performance with and without CoT is model dependent in this experiment, and probably is dataset dependent.

Closing Remarks

Chain of Thought had a great impact on LLMs, affecting the prompting strategies and fine tuning pipelines. This can be directly seen in LLMs. Earlier tended to output only the final answer for mathematical and reasoning tasks. Now, even small models tend to explain their steps and output the final answers.

This is an instance where human understanding of the world have improved ML model. Prompting and training models on how humans think resulted in a better model performance, counter to the spirit of bitter lesson, which argues that progress in AI comes from building system that learn on that own, not from encoding what we know.

Maybe Chain of Thought style training will hit a wall in the near future. Of course, there are alternatives such as teaching the models to reason in latent space.

ROFORMER: ENHANCED TRANSFORMER WITH ROTARY POSITION EMBEDDING

Written in Nov 21, 2025

Rotary Positional Embedding (RoPE) is practically used in all modern LLMs. Llama, Qwen, and even the recent Olmo3 all use it. This paper introduced this embedding.

What is RoPE?

Simply put, instead of adding a sinusodial position embedding to the \(x\) vector, as done in the original transformer paper, The rotation is applied to the \(q,k\) vectors.

This creates some interesting properties.

Theoretical Explanation of RoPE

Section 3.4 explains the validity of the RoPE. It requires the unnormalized attention values \(q_m k_n\) to

1. Be same if the distance (\(n-m\)) are the same.
2. Be the original values of \(q,k\) if \(m, n\) are zero.

The paper shows that their formulation of Rotary Position embedding satisfies this property.

They also show an upper bound for this value to be \(\left( \max_i \lvert h_{i+1} - h_i \rvert \right) \sum_{i=0}^{\frac{d}{2}-1} \lvert S_{i+1} \rvert\), showing that the dot product reduces with more distance.

Paper’s Performance Results of RoPE

The paper shows effectiveness of RoPE by comparing its performance with sinnusodial position embedding in the original attention paper. The result is as follows.

The right graph compares RoFormer to a standard BERT baselline (using learned absolute embedding). The graph on the left uses PerFormer, a linear attention.

This result is the one that surprised me the most when reading this paper. I expected the loss to be more drastic, as RoPE is currently used in virtually all of Transformers. The results show a measly increase in performance, an increase where in some eyes, might be just because of luck instead of actual improvement. Same can be said for result on downstream tasks, where learned positional embedding perform better than RoFormer (Their postional embedded transformer) in some tasks.

Additional Experiment.

The paper argues that their method have a long distance decay. They argue that by proving that the expression is bounded above by \(\left( \max_i \lvert h_{i+1} - h_i \rvert \right) \sum_{i=0}^{\frac{d}{2}-1} \lvert S_{i+1} \rvert\). However, their graph only lasts up to around position ~260. What happens for longer sequencs? Does the relative upper bound go to zero, or does it stay stable? In some chance, the upper bound can increase after some time.

The graph suggest that the upper bounda stays stable (for d = 64) from 2 to 6 for 2000+ positions.

Personal Thoughts

When I decided to read this paper, I was expecting a monumental increase in performance, or a quality, such as decreased performance degradation over time, for Rotary Position Embedding. In fact, there should be a reason behind why all transformers use RoPE. However, this wasn’t the case. The results showed that their embedding is extrapolatable to longer distance, but didn’t do a short context learning -> long context training, or show a drastic increase in performance compared to previous embedding mechanism.

As an accepted author myself, I decided to explore other papers that examined my paper’s topic: LLM Sycophancy. Below are a few bullet points for each paper.

Self-Augmented Preference Alignment for Sycophancy Reduction in LLMs

• Created SAA dataset, which contains queries with user pressumption, correct answer, and incorrect alternatives.
  • The pressumptions are sometimes correct and sometimes incorrect.
• Trained LLM with SAA using Supervised Fine Tuning and Proximal Policy Optimization. Results show that in both scenarios, the rate of sycophancy decreased.
• I wonder if model trained with SAA works with out of domain pressumptions. the examples in the dataset are simple: they are single turn and the prompts pertain to simple questions.

Advancing Oversight Reasoning across Languages for Audit Sycophantic Behaviour via X-Agent

• Created a framework called X-Agent, an agent that checks for sycophancy in LLM responses.
• Their X-Agent successfully detects sycophantic responses,
  • Defined sycophancy as when LLM accepts incorrect user presupposition.
  • Overall accuracy with X-Agent increased in most LLMs.
• The models fine tuned on the X-Agent rollout also improved performance for both accuracy and sycophancy

Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories

• Created a SMART (Sycophancy Mitigate Through Adaptive Reasoning Trajectories) Framework
  • Smart uses UA-MCTS(Uncertainty-Aware Adaptive Monte Carlo Tree Search) to generate high quality reaoning trajectory
  • In each reasoning step, future rollouts are created and scored based on the improvement of answer correctness probability compared to the last step.
• The rollouts with scores are then used to train Large Language Models.
  • The implementation outperforms other approaches.
  • Dense Reinforcement Learning on the rollouts outperforms SFT.

Pointing to a Llama and Call it a Camel On the Sycophancy of Multimodal Large Language Models

• Multimodal Large Language Models are more susceptible to Sycophancy compared to Large Language Models.
• Created a pipeline called SRT (Sycophancy Reflective Tuning).
  • This pipeline consists of image textualization -> reflection -> Summarization.
• Fine tuning on the SRT rollout resulted in better overall score compared to original prompting.
  • Correction Rate (original answer incorrect, user suggestion correct) was reduced slightly, but Sycophancy Rate (original answer correct, user suggestion incorrect) increased considerably.
  • Removing reasoning in SRT results in low Correction Rate.

Measuring Sycophancy of Language Models in Multi-turn Dialogues

• Authors proposed a benchmark called SYCON-BENCH, a benchmark for evaluating sycophantic behavior in multi-turn, free-form conversational settings
• Alignment Tuning increases sycophancy
• Model scaling reduces sycophancy
• Reasoning Models are More Robust
• Prompting in third person, creating a Non-sycophantic prompt can mitigate sycophancy
• Models can identify sycophancy when asked to, but are often sycophantic. This shows that sycophancy is not driven by ignorance.

Challenging the Evaluator: LLM Sycophancy Under User Rebuttal

• Authors explored the discrepancy: LLMs are good as evaluative agents, but are sycophantic in follow-up questions. Since two actions are similar: choosing the better of the two different argument, what prompts an LLM to have this difference
• Found that conversational framework makes LLM more sycophantic
• More reasoning results in better persuasion, even if the reasoning is incorrect
• LLMs are more likely to be persuaded by casual prompts than by reasoned prompts.
• While casual prompt results in the best persuasion rate, the best correction rate is given when LLM is prompted as a judge.

Echoes of Agreement: Argument Driven Sycophancy in Large Language Models

• LLMs are likely to echo user’s presented stance
• LLMs tend to agree more when supporting arguments were provided and disagree more when refuting arguments were provided.
• LLMs are likely to change their stance both in single and multi turn settings.
• LLMs are more likely to change their stance for arguments with “better” strength.

Conducted quick experiments to find what actually speeds up PyTorch code on Apple Silicon (MPS).

Based on this video.

tl;dr

Keep tensors on GPU from the start, vectorize everything, avoid dynamic shapes. Code can be found here.

What’s tested

• Creating tensors: CPU to GPU vs creating directly on GPU
• Runtime constants: Dynamic tensor creation vs register_buffer
• Masking: Different masking approaches (indexing vs vectorized ops)
• DataLoader: pin_memory and non_blocking flags

Quick findings

1. Create tensors directly on device - Up to 80x speedup for larger tensors for creation and basic ops.

   # Slow
   tensor = torch.randn(size, size).to(device)
   
   # Fast
   tensor = torch.randn(size, size, device=device)
   

2. Use register_buffer for constants - Up to 8x speedup.

   self.register_buffer('constant', torch.tensor(..., device=device))
   

3. Vectorize masking operations - Massive speedup (50-90x)

   # Slow - creates irregular tensor
   result = attention_map[attention_map > .5].mean()
   
   # Fast - stays vectorized
   mask = (attention_map > .5).float()
   result = (attention_map * mask).sum() / mask.sum().clamp(min=1e-9)
   

4. Use non_blocking=True - Small improvement (~5%)

   data = data.to(device, non_blocking=True)
   

   Note: pin_memory doesn’t help on MPS (not supported yet)

In machine learning, many scores or values are difficult to interpret in siolation. For example, take BERTScore. While we know that higher scores indicate greater similarity, the raw values lack intuitive meaning. Is the difference between 0.7 and 0.8 comparable to the difference between 0.8 and 0.9? What constitutes a “good” BERTScore?

Furthermore, in Retrieval Augumented Generation (RAG) systems, consine similarity between the query and document embeddings are often used to determine relevant documents. However, the cosine similarity itself is not interpretable. A cosine similarity of 0.7 might seem reasonable, but how much better is 0.8? Without context, these numbers are difficult to assess.

More fundamentally, matrix multiplication is simply a parallelized version of a dot product, which is scaled cosine similarity. I noticed that I lack the intuition of cosine similarity values in high dimensional spaces. This motivated me to investigate the distribution of cosine similarities between random vectors in a unit hypersphere.

Experiment Starting with 2D

I To build intuition, I started with a simple 2D unit circle. Before you read further, try to guess what the cosine similarity distribution would look like.

import math
import random
import matplotlib.pyplot as plt
def get_random_point_on_circle():
    angle = random.uniform(0, 2 * math.pi)
    x = math.cos(angle)
    y = math.sin(angle)
    return (x, y)

random_points = [get_random_point_on_circle() for _ in range(10000)]

first_item = random_points[0]
cosine_similarities = []
for i in range(1, len(random_points)):
    dot_product = first_item[0] * random_points[i][0] + first_item[1] * random_points[i][1]
    cosine_similarities.append(dot_product)

plt.hist(cosine_similarities, bins=50)
plt.title("Histogram of Cosine Similarities of Unit Vectors in 2D")
plt.xlabel("Cosine Similarity")
plt.ylabel("Frequency")
plt.show()

The histogram looks like below.

Cosine similarities are more likely to be around extreme values of -1 and 1 and less likely to be around 0! This is quite counter-intuitive. I asked three people about this and all three guessed that the distribution would be uniform.

Understanding the 2D Distribution

Why does this happen? Let’s build geometric intuition.

We can fix the first vector to \((1,0)\) without loss of generality. If we look at the first quadrant of the unit circle, we see that \(x = y\) line bisects the circumference of the quadrant. However, the cosine similarity of the midpoint is \(\frac{\sqrt{2}}{2} \approx 0.707\). Since cosine of 90 degrees is 0, and cosine of 0 degrees is 1, we expect to see the same number of points with cosine similarity between 0 and 0.707 and cosine similarity between 0.707 and 1. This explains why we see more points around extreme values.

Another way to look at it would be to look at the slope. The slope of the circle at \((1,0)\) is perpendicular, while at \((0,1)\) it is horizontal. Thus a change in angle near \((1,0)\) results in a small change in cosine similarity, while a change in angle near \((0,1)\) results in a large change in cosine similarity.

Higher Dimensions

For higher dimensions, manually coding the angular rotations becomes error-prone. there exists an elegant approach which leverages the properties of the multivariate normal distribution: sample from a standard normal distribution, and then normalize. The reason is that multivariate normal distribution has spherically symmetric distribution, so normalizing the sampled vector results in a uniform distribution on the unit hypersphere.

import numpy as np
def random_point_on_unit_sphere(n: int) -> np.ndarray:
    point = np.random.randn(n)
    point = point / np.linalg.norm(point)

    return point

Using this function, I plotted the histogram of cosine similarities in 3D, 10D, 100D, 1000D below.

As dimension increases, the cosine similarities are more concentrated around 0. For 100D, there is no sample with cosine similarity greater than .3 among 10,000 samples.

Another interesting observation is that in 3 dimension, the distribution looks uniform. GPT5 shows that it is indeed uniform in 3D case. In fact, it showed that the distribution of the cosine similarity of dimension \(d\) follows a Beta distribution with parameters \((\frac{d-1}{2}, \frac{d-1}{2})\) link.

Implications for Embeddings

Text embedding models often assign cosine similarities above 0.7 to similar documents. In high-dimensional spaces (common embeddings use 384, 768, or 1536 dimensions), such high cosine similarities are extremely rare between random vectors. This means that semantically similar texts must have embeddings that are highly aligned in the embedding space, far from the random baseline.

I have created a simple attention visualizer for transformer models. It is available at this link. It can

1. Visualize all attention heads for a specific layer
2. Show average attention for each layer.
3. Single heatmap averaging all layers and heads.

The code should work for any causal LLMs.

More Details in the Repository.

Visualization Examples

Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions showed that LLMs, when prompted with questions which have false medical presuppositions, does not readily detect these presuppositions. There are there are two possible points of failure.

1. LLM does not know about the false medical presupposition.
2. LLM blindly follows user input, sign of sycophancy.

Problem (1) would indicate that LLM’s lack medical knowledge, while problem (2) would suggest sycophancy.

Dataset

The Cancer-Myth paper includes a public dataset. Sampled 20 first questions for this experiment. Each items in the dataset includes question, incorrect presuppostions, and other informations.

Experiment

Implemented a basic presupposition checker. Made LLMs

1. Deconstruct user query into presumptions
2. Check the factuality of each presumptions
3. Condense the report to ones that are shown false
4. Compare with the ground truth information

Tested with Anthropic’s Claude 4 Sonnet, and 3.5 Haiku models.

Results

On the first 20 questions, Both LLMs almost always found the correct response. Using the scoring rubric of [-1, 0, 1] (Provided in the paper), we have

Limitations

• Tested on only 20 questions.
• Didn’t compare the result to simple LLM QA (Might be the case that the chosen 20 questions were easy)
• Deconstructing and validating might be too over the board.
  • Simple QA of prompting LLM to detect incorrect presummptions might have been good enough.
• evaluator, deconstructor, and presumption checker all used same model. Would have been better if different LLM were used for evaluator.
• Only tested on Anthropic’s Claude models. Would be interesting to see how other models behave.

github link

How many phones can pyramid of Giza charge?

Today in class, as a ice breaker, we were to discuss the “best” invention in human history. The main point, I guess, other than a quick ice breaker, was to get us to think of the metric or a standard for “best”ness.

After some discussion, I thought maybe, the metric of “best” can be described as how much movement the invention has caused. For example, the internet would have caused a lot of movement in the world as online shopping, international trade would have happened less in the counterfactual world.

Then, the train of thought had led me to think of method to measure the movement. The easiest method (which is not really accurate in this example) would be to calculate the energy used to move item.

The easiest calcuation of such energy would be potential energy. This made me think of potential energy of the pyramid of Giza.

Calculation of potential energy.

Potential energy is given by \(PE = mgh\). We would require mass, dimension of the pyramid to calculate this. From quick internet search

• Hieght: 146.7 meters
• Base: 230.6 meters
• Density (limestone): 2.7 g/cm^3

source: wikipedia

The height of the pyramid differ from source and time of measurement. I am using the one that is said to be the original dimension.

Pyramid of Giza is mostly full of limestone. Thus we can use its shape to estimate the potential energy.

For each \(h\) height, the area of the horizontal cross section would be given by \((230.6 - 230.6 \frac{h}{146.7})^2\). Thus we can calculate the potential energy by

Which, according to Wolfram Alpha is \(2.52 \times 10^9 kJ\).

Now, an Iphone 16 has battery of 3561mAH with 3.85V. Thus the total energy inside the battery would be 13.7Wh. In Joules, that would be \(13.7 \times 3600 = 49320 J\). Thus the total number of phone charges would be \(2.52 \times 10^{12} / 49320 = 5.11 \times 10^7\). That is 51.1 million charges. That looks like a lot, but considering that there are around that many people in South Korea, the potential energy of the pyramid is equal to energy used to fully charge South Korean people’s phones once. That’s Kind of depressing.

What is the lower bound of land required to feed one person?

I live around the Yangjae river in Seoul. During the summer, there is a small plot of land that is used to grow rice. (This place doubles as a ice rink in the winter.) I used to wonder how much rice grows from that plot of land.

Furthermore, rice is known to be relatively calorie dense compared to other crops. It is more calorie dense than wheat, barley, and oats but less than potatoes. So having a rough estimate of amount of calories that can be produced from a unit of land could be used to estimate the lower bound of land required to feed one person. Since people don’t live on carbs alone, we would assume that lower bound would be higher than amount of calories that can be produced by rice.

According to United States Department of Agriculture, the country with best milled rice yield per hectare is Australia, with 10MT / hectare. Northeast Asia, Japan, South Korea, and China, has yield of 7 MT / Hactare. For the sake of simplicity, let’s estimate using 7MT / Hactare.

People typically consume 2000kcal per day. Also, milled rice has around 365Kcal per 100g. Thus, we can see that

• 1 hectare of rice creates \(7 \times 10^3 \times 3.65 \times 10^3 = 2.56 \times 10^7 kcal\)
• A Hectare of land can feed 12800 people per day.
• In order to sustain one person, we would need \((100000 / 12800) \simeq .78m^2\) of land.

So in caloric perspective, we only need a little less than one square meter to feed a person per day. We need a little less than 300 square meters to feed a person annually.

With the help of Huggingface’s Transformers, the model architecture and weights of open source models are available to the public.

Methods on Mechanistic Interpretability by Welch Labs made me think of a feature in LLM that I was not aware of. The experiments done in the video presupposes that token embedder (tokens to vector translator) and the LM Head (final vector to token logits) are identical.

I checked if this was the case for LMs as I was not familiar with this change. I thought that naturally, the LM head layer and the embedder layer were different. Turns out, it is a widely used method called weight tying.

Because of this characteristic, we can “estimate” the token of the intermediate layer by multiplying intermediate layer’s output with the LM head’s weight matrix, then taking the argmax of the result. Pretty useful trick for mechanistic interpretability.

So, I played around with this idea. The first question I had was if the LM head’s weights were L2 normalized. If the LM head’s weights are L2 normalized, then it follows that given token \(t\), and the head matrix \(W\), \(t = \text{argmax}(WW^T(t))\). If this were true, then it would give some solid baseline on future mechanistic interpretability experiments.

Experiment with Llama 3.2 on L2 Norm

However, the LM heads are not L2 normalized for Llama 3.2 models.

tokenizer_1b = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
model_1b = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
tokenizer_3b = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model_3b = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

model_1b_embedding = model_1b.model.embed_tokens
lm_head_1b = model_1b.lm_head

model_3b_embedding = model_3b.model.embed_tokens
lm_head_3b = model_3b.lm_head

def find_embedding_l2_info(model_embedding):
    model_embedding_weights = model_embedding.weight
    l2_norms = torch.norm(model_embedding_weights, p=2, dim=1)
    return l2_norms

def find_max_min_var(l2_norms):
    max_val = torch.max(l2_norms)
    min_val = torch.min(l2_norms)
    variance = torch.var(l2_norms)
    return max_val, min_val, variance

def find_max_min_indices(l2_norms):
    max_val = torch.max(l2_norms)
    min_val = torch.min(l2_norms)
    max_indices = torch.where(l2_norms == max_val)[0].item()
    min_indices = torch.where(l2_norms == min_val)[0].item()
    return max_indices, min_indices

l2_norms_1b = find_embedding_l2_info(model_1b_embedding)
l2_norms_3b = find_embedding_l2_info(model_3b_embedding)

max_val_1b, min_val_1b, var_1b = find_max_min_var(l2_norms_1b)
max_val_3b, min_val_3b, var_3b = find_max_min_var(l2_norms_3b)

print(f"Max value 1b: {max_val_1b}")
print(f"Min value 1b: {min_val_1b}")
print(f"Variance 1b: {var_1b}")
print(f"Max value 3b: {max_val_3b}")
print(f"Min value 3b: {min_val_3b}")
print(f"Variance 3b: {var_3b}")

max_indices_1b, min_indices_1b = find_max_min_indices(l2_norms_1b)
max_indices_3b, min_indices_3b = find_max_min_indices(l2_norms_3b)

print(f"Max indices 1b: {max_indices_1b}")
print(f"Min indices 1b: {min_indices_1b}")
print(f"Max indices 3b: {max_indices_3b}")
print(f"Min indices 3b: {min_indices_3b}")

Max value 1b: 1.3203290700912476
Min value 1b: 0.5214595794677734
Variance 1b: 0.008283221162855625
Max value 3b: 1.4694581031799316
Min value 3b: 0.6221248507499695
Variance 3b: 0.012782144360244274
Max indices 1b: 58996
Min indices 1b: 72710
Max indices 3b: 58996
Min indices 3b: 81259

As a fun note, the max, min index tokens are

Max
1b: yourselves
3b: yourselves

Min
1b: -->
3b:


artisanlib

Yes. The newlines and spaces are included in the tokens. Now, this means that there might be some tokens that does not map to itself after function \(\text{argmax}(WW^T())\).

Does tokens map to itself?

So, I checked if the tokens map to itself.

def find_mismatch_indices(tot_token_count, model_embedding, lm_head, batch_size=167): #set to 167 since 167 | 128256
    original_different_indices = []
    mismatched_indices = []
    for i in tqdm(range(tot_token_count // batch_size)):
        inputs = [j + i * batch_size for j in range(batch_size)]
        inputs = torch.tensor(inputs)
        embedding_out = model_embedding(inputs)
        head_out = lm_head(embedding_out)
        returned_items = torch.argmax(head_out, dim=-1)
        if not torch.all(returned_items == inputs):
            original_different_indices.extend(inputs[returned_items != inputs].tolist())
            mismatched_indices.extend(returned_items[returned_items != inputs].tolist())
    return original_different_indices, mismatched_indices


original_different_indices_1b, mismatched_indices_1b = find_mismatch_indices(128256, model_1b_embedding, lm_head_1b)
original_different_indices_3b, mismatched_indices_3b = find_mismatch_indices(128256, model_3b_embedding, lm_head_3b)

embedding_1b_correct_rate = (128256 - len(original_different_indices_1b)) / 128256
embedding_3b_correct_rate = (128256 - len(original_different_indices_3b)) / 128256

print(len(original_different_indices_1b))
print(len(original_different_indices_3b))

print(f"1b embedding correct rate: {embedding_1b_correct_rate}")
print(f"3b embedding correct rate: {embedding_3b_correct_rate}")

614
602
1b embedding correct rate: 0.9952126996007984
3b embedding correct rate: 0.9953062624750499

Less than .5 percent of the tokens does not map to itself. Then, which tokens does the unmatched tokens map to?

def create_count_dict(mismatched_indices):
    count_dict = {}
    for index in mismatched_indices:
        if index in count_dict:
            count_dict[index] += 1
        else:
            count_dict[index] = 1
    return count_dict

count_dict_1b = create_count_dict(mismatched_indices_1b)
count_dict_3b = create_count_dict(mismatched_indices_3b)

print(count_dict_1b)
print(count_dict_3b)

{122456: 477, 528: 1, 738: 1 ...}
{66325: 482, 528: 1, 738: 1 ...}

Surprisingly, the majority of the mismatched tokens map to the same token. In the case of 1b, 477 tokens map to token 122456 and in the case of 3b, 482 tokens map to token 66325.

Token 122456: организа
Token 66325: ♪

When I manually skimmed through the tokens mapped to 122456 and 66325, most of them were special tokens programming words, non-english words. Nevertheless, I found the result interesting. Why do the majority of the unmaatched tokens map to the same token?

If you have any ideas, please let me know. Thanks for reading!

Code can be found https://github.com/blindTissue/embedding_matrix_experiment

The Louvain method can be broken into two phases:

1. maximization of modularity: The algorithm tries to maximize the modularity of the graph by moving nodes between communities.
2. aggregation: The algorithm aggregates the nodes in the same community into a single node and creates a new graph.

The two phases are repeated until no further increase in modularity is possible.

First Phase: Maximization of Modularity

The modularity of a graph is defined as

Where

• \(A_{ij}\) is the weight of the edge between node \(i\) and node \(j\)
• \(k_i\) and \(k_j\) are the sum of the weights of the edges attached to nodes \(i,j\) respectively
• \(m\) is the sum of all the edge weights in the graph
• \(c_i\) and \(c_j\) are the communities to which the nodes \(i\) and \(j\) belong
• \(\delta\) is the Kronecker delta function:
  • \(\delta(x,y) = 1\) if \(c_i\) and \(c_j\) are the same cluster and 0 otherwise

Modularity lies within range \([-1/2, 1]\).

The modularity above can be written as sum of modularity in a cluster. A modularity of a cluster can be simplified into

and the total modularity is the sum of the modularity of all clusters.

Let’s start with basic example of modularity to gain some intuition.

For a graph without any edges, the modularity would be 0.

Consider the graph above. Say that we made two clusters, \((A, B, C, D)\) and \((E, F, G, H)\). Then modularity would be

If we define cluster such into \((A,E), (D, H), (B,F), (C,G)\), Then the modularity would be

Now, If we say that the entire graph is one big cluster, we have

Finding the optimal clustering is NP-complete hard. so the Louvain method uses a greedy approach. The algorithm does not guarantee the optimal clustering, but it is fast and often gives good results.

The algorithm is as follows:

1. Start with each node in its own cluster.
2. For each node, calculate the change in modularity if the node is moved to a neighboring cluster.
3. Move the node to the cluster that gives the maximum increase in modularity.
4. Repeat steps 2 and 3 until no further increase in modularity is possible.

After the algorithm ends, it is time to move to the second step.

Second Phase: Aggregation

In the second phase, the algorithm aggregates the nodes in the same community into a single node and creates a new graph. The edges between the communities are the sum of the weights of the edges between the nodes in the communities.

Code

What better way to understand the algorithm than to implement it? I have created a simple implementation of the Louvain method in Python. You can find the code here.

Souces

• Louvain Method
• On Modudularity Clustering
• Machine Learning with Graphs Lecture 13.3

1. Base case: Prove that the statement is true for the smallest possible value.
2. Inductive step: Assume that the statement is true for some value, and prove that the statement is true for the next value.
3. Conclusion: The statement is true for all values.

A simple example is to prove that

1. Base case: For \(n=1\), \(\sum_{i=1}^{1} i = 1 = \frac{1(1+1)}{2}\)
2. Inductive step: Assume that the statement is true for some \(n=k\). Then, \(\sum_{i=1}^{k} i = \frac{k(k+1)}{2}\). We want to prove that the statement is true for \(n=k+1\).
3. Conclusion: The statement is true for all \(n\).

For 2 we see that

So, the statement is true for all \(n\).

For induction examples shown in elementary discrete math, it is often the case that you can start from the \(n = k\) case and show that the statement is true for \(n=k+1\). However, for some problems, viewing the inductive step as a reduction problem can conceptually simplify your reasoning

General Outline of Reduction

In general, Induction can be used to prove that all elements of set \(S\) satisfies a mathematical statement with free variable. Let \(f\) be such mathematical statement that takes an element of set \(S\) and returns a boolean value. Then, induction can be used to prove that \(f(x)\) is true for all \(x \in S\). For simplicity say that \(S_k\) is a subset of \(S\). Similary, let’s say that \(f(S_k) = 1\) if and only if \(f(x) = 1\) for all \(x \in S_k\) and 0 otherwise.

For induction, we divide the set \(S\) into indexable subsets \(S_1, S_2, \cdots\) such that their union is \(S\). Then we prove that \(f(S_k) = 1\) for all \(k\) by the induction step above.

In our inductive step, mentally thinking the problem as a reduction is often easier. Instead of starting from items of \(S_i\), and expand to items of \(S_{i+1}\), we should treat the problem as finding reduction from items of \(S_{i+1}\) to items of \(S_i\). This is advantageous as for justification of expanding \(S_i\) to \(S_{i+1}\), we must show that the map from \(S_{i+1} spans all elements of\)S_{i+1}$$.

This is often hard. There are edge cases that one has to consider. Below is a simple example.

Problem

A binary tree is a rooted tree in which each node has at most two children. Using induction, show that in any binary tree the number of nodes with two children is exactly one less than the number of leaves.

Proof

For this problem, the set \(S\) is the set of all binary trees. \(f\) is statement that in \(x\), the number of nodes with two children is exactly one less than the number of leaves.

We would split the set by depth with \(S_k\) being binary trees with depth \(k\).

For the base case, single node tree has 1 node and 0 edges, so the statement is true.

Now look at case of \(S_{k+1}\). we look at nodes in depth \(k+1\). If the parent of the node has two children, then deleting the children of that parent would reduce the number of nodes and edges by 2. If the parent has a single child, then deleting the child would reduce the number of nodes and edges by 1. Deleting all parents of nodes of depth \(k+1\) would reduce the number of nodes and edges by same amount, and result in a tree of depth \(k\). So if \(f(S_k) = 1\), then \(f(S_{k+1}) = 1\).

So, the statement is true for all binary trees.

We have shown the inductive step by reducing the problem. If we were to do the opposite, we would need to show that our “addition” of nodes from depth \(k\) spans all trees of depth \(k+1\). Using the reduction method, we can avoid this problem.

Reduction is a strong tool in computer science. Dynamic Programming and Turing Reduction also simplifies the problem by reducing it to a problem we already know.