• Part 0: Setup
• Part A: Sampling Loops
  • 1.1 Implementing the Forward Process
  • 1.2 Classical Denoising
  • 1.3 One-Step Denoising
  • 1.4 Iterative Denoising
  • 1.5 Diffusion Model Sampling
  • 1.6 Classifier-Free Guidance (CFG)
  • 1.7 Image-to-Image Translation
  • 1.8 Visual Anagrams
  • 1.9 Hybrid Images
• Part B: Flow Matching from Scratch!
  • 1. Single-Step Denoising UNet
  • 2. Training a Diffusion Model

TLDR

In this project, I explore techniques for sampling from pretrained diffusion models (Part A) and train my own class-conditioned flow matching model from scratch using a UNet (Part B).

Here’s an example of an image sampled from the model I trained in part B, which can generate images of any handwritten digit between 0 and 9:

Part 0: Setup

In this part, I experimented with the DeepFloyd IF diffusion model. This is a text-to-image model that operates in two stages:

1. Stage 1: Generates a 64x64 resolution image from the text prompt.
2. Stage 2: Upscales the image to 256x256 and adds details.

I generated images for three different prompts using 20 inference steps. The random seed used for all parts of this project is 100.

“An oil painting of a snowy mountain village”

“A photo of a cat”

“A photo of a temple”

The outputs here are pretty good in my opinion, which is a testament to the quality of Deepfloyd IF’s training process.

Comparison with More Inference Steps

I also generated the temple image with 100 inference steps. This lets us see if the quality improves with more denoising iterations.

The image turned out a little bit oversaturated, which makes me believe that more inference steps were not necessary, at least for this specific prompt. That said, it didn’t ruin the image either, so my conclusion is that the number of inference steps has to be chosen qualitatively and depends on the prompt and model being used.

Part 1: Sampling Loops

1.1 Implementing the Forward Process

The forward process in diffusion models adds noise to a clean image $x_0$ to produce a noisy image $x_t$ at timestep $t$. The process is defined by the equation:

Which allows us to sample $x_t$ directly:

Here are the results of the forward process on the Campanile image at different noise levels ($t \in {250, 500, 750}$):

1.2 Classical Denoising

I first attempted to remove the noise using classical Gaussian blurring. As expected, this simple technique fails to recover the details, blurring out both the noise and the high-frequency content of the image.

No amount of Gaussian blurring can bring back parts of the image that were already lost. We need something more sophisticated.

1.3 One-Step Denoising

Using a pretrained diffusion model, we can try to recover $x_0$ in a single step. The model is trained to estimate the noise $\epsilon$ in a noisy image $x_t$. Given the estimate $\epsilon_\theta(x_t, t)$, we can approximate $x_0$ by inverting the forward process equation:

The model does a much better job than Gaussian blur, but for high noise levels (t=750), the one-step reconstruction is blurry and lacks fine detail. This is because the initial assumption of mapping directly to $x_0$ is difficult when the signal is heavily corrupted.

1.4 Iterative Denoising

To get high-quality images, we denoise iteratively. Starting from pure noise or a noisy image, we repeatedly apply the update step:

which effectively steps from $t$ to $t’$ by removing a fraction of the predicted noise and adding new variance. In practice, we don’t iteratively denoise over all 1000 timesteps, as that would be too costly. Instead, we iterate over a strided subset of timesteps (for this assignment, I set the stride to 30)

Here is the progression of iterative denoising (using strided sampling):

Comparison:

The iterative result is significantly sharper than the one-step estimation.

1.5 Diffusion Model Sampling

We can generate new images by running the iterative denoising loop starting from pure Gaussian noise ($x_T \sim \mathcal{N}(0, \mathbf{I})$) with the prompt “a high quality photo”.

1.6 Classifier-Free Guidance (CFG)

To improve image quality and prompt adherence, I implemented Classifier-Free Guidance, also known as CFG. We compute two noise estimates: one conditional on the text prompt ($\epsilon_{cond}$) and one unconditional ($\epsilon_{uncond}$). The final noise estimate is:

where $\gamma > 1$ is the guidance scale. This pushes the image towards the prompt.

The generated images are sharper and more clearly defined compared to the unguided samples.

1.7 Image-to-Image Translation

By taking a real image, adding noise to it up to a certain timestep $t$, and then running the iterative denoising process from there, we can edit images. This allows us to balance maintaining the original structure (via the starting noisy image) and generating new details (via the denoising loop).

Below I show edits of the Campanile image at noise levels [1, 3, 5, 7, 10, 20] with the conditional text prompt “a high quality photo”. I also show two edits of my own test images, which are captioned “original” in the below image:

1.7.1 SDEdit

Artificial Image Transition (SDEdit):

Hand-Drawn Image Transition:

1.7.2 Inpainting

We can use a mask to keep parts of the image constant while denoising the rest. At each step of the backward process, we force the pixels outside the mask to match the noisy version of the original image, while letting the model hallucinate content inside the mask.

1.7.3 Text-Conditional Image-to-Image Translation

We can guide the SDEdit process with specific text prompts to change the style or content of the image. Results for all three images are below:

Here, the prompt was “a rainy day”. Notice now the images gradually have more and more “rain-like” features. For example, the campanile turns into a bolt of lightning for time step 5 of the first row. Similarly, the happy face drawing starts to show small droplets of rain on its sides.

1.8 Visual Anagrams

Visual anagrams are images that look like one thing when upright and another when flipped. I implemented this by averaging the noise estimates for two different prompts, one computed on the upright image and one on the flipped image:

1.9 Hybrid Images

Hybrid images combine the low frequencies of one image with the high frequencies of another. We can generate these with diffusion by combining noise estimates:

Part B: Flow Matching from Scratch!

In this part, we implement a diffusion model from scratch using the MNIST dataset. We start with a simple single-step denoiser and then move on to a full diffusion model with time conditioning. Unless otherwise stated, the random seed used for all subparts was 100.

1. Single-Step Denoising UNet

1.1 Architecture

The backbone of our denoiser is a UNet. At its core, a UNet is just an autoencoder with a twist: it compresses the image into a bottleneck to capture global context (like “this is a digit 8”) and then expands it back to the original size. The “twist” is the skip connections—wires that bypass the bottleneck and plug the detailed, high-resolution features from the encoder directly into the decoder. This lets the network reconstruct fine details (like edges and noise) that would otherwise be lost in compression.

1.2 Noising Process Visualization

The noising process adds Gaussian noise to a clean image $x$. \(z = x + \sigma \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)\)

Here is the effect of varying $\sigma$ on a clean image:

1.2.1 Training the Denoiser

I trained a UNet to denoise images with $\sigma = 0.5$. The objective is to minimize the L2 distance between the denoised image and the original clean image: \(L = \mathbb{E}_{z,x} \|D_{\theta}(z) - x\|^2\)

Training Loss Curve:

Denoising Results (Epoch 1 vs Epoch 5): The model learns to remove the noise effectively after just a few epochs.

1.2.2 Out-of-Distribution Testing

The model was trained only on $\sigma=0.5$. Here I tested it on other noise levels. It performs reasonably well on lower noise levels but struggles when the noise level is much higher than what it was trained on (e.g., $\sigma=1.0$).

$\sigma=0.0$	$\sigma=0.2$
$\sigma=0.4$	$\sigma=0.5$
$\sigma=0.6$	$\sigma=0.8$
$\sigma=1.0$

1.2.3 Denoising Pure Noise

Here, I trained the model to denoise pure noise (i.e., mapping $\mathcal{N}(0, I)$ to MNIST digits).

Interestingly, the model manages to generate digit-like shapes, but they are often blurry or hybrids of multiple digits. This is because the mapping from pure noise to a specific digit is one-to-many and highly ambiguous, so the L2 loss forces the model to output the “average” of all possible digits, resulting in blurry blobs.

2. Training a Diffusion Model

Now we move to a proper diffusion model (Time-Conditioned UNet), where we iteratively denoise the image.

2.1 Adding Time Conditioning

To perform iterative denoising, the model needs to know the current noise level (or timestep $t$). We inject this information into the UNet using fully connected blocks (FCBlocks).

The scalar $t$ is fed into two fully connected blocks (fc1_t, fc2_t) to produce scaling coefficients. These coefficients are then used to modulate the feature maps at specific points in the UNet.

Specifically, $t$ is used to scale the activations after the unflatten step ($t_1$) and after the first upsampling block ($t_2$):

# fc1_t and fc2_t are small MLPs that project the scalar t to channel dimensions
t1 = fc1_t(t)
t2 = fc2_t(t)

# Modulate the unflattened features
unflatten = unflatten * t1

# ... intermediate layers ...

# Modulate the first upsampling block
up1 = up1 * t2

Training involves picking a random image $x_1$, a random timestep $t$, adding noise to get $x_t$, and training the network. The loss at every time step is calculated based on how well the model prediction conditioned on noisy image $x_t$ and time step $t$ matches $x_1 - x_0$: the clean image minus the random noise.

2.2 Time-Conditioned UNet Training

I trained the UNet conditioned on the timestep $t$.

2.3 Time-Conditioned Sampling

Sampling starts from pure noise $x_0 \sim \mathcal{N}(0, 1)$ and iteratively refines it to a clean image $x_1$.

Here are the sampling results at different epochs and different seeds (100, 101, and 102):

2.4 Adding Class-Conditioning to UNet

To improve the generation quality and gain control over the output, we condition the UNet on both the timestep $t$ and the digit class $c$. This allows us to ask the model for a “5” or a “7” specifically.

Architectural Changes

Similar to time conditioning, we inject the class information $c$ (a one-hot vector) into the network. We add two more FCBlocks (fc1_c, fc2_c) to process the class vector.

The class conditioning is added to the time conditioning, meaning the modulation signal becomes a combination of both:

# c is a one-hot vector for the digit class
c1 = fc1_c(c)
c2 = fc2_c(c)

# Combine with time embedding and modulate
unflatten = (c1 * unflatten) + t1
# ...
up1 = (c2 * up1) + t2

We also use dropout on the class conditioning (setting it to a null token with $p=0.1$) to enable Classifier-Free Guidance later.

2.5 Training the UNet

We train the class-conditioned UNet using the same process as before, but with the added class labels.

2.6 Sampling from the UNet

We use Classifier-Free Guidance (CFG) during sampling to improve quality. The final noise estimate is a combination of the conditional and unconditional estimates: \(\epsilon = \epsilon_{uncond} + \gamma (\epsilon_{cond} - \epsilon_{uncond})\)

By guiding the model with class labels, we can generate specific digits. Here are the results over 10 epochs using Classifier-Free Guidance ($\gamma=5.0$).

Epoch 1

Epoch 5

Epoch 10

Can we get rid of the annoying learning rate scheduler?

I tried training the model with a constant learning rate of 1e-4 instead of using an exponential decay scheduler. To account for the fact that the learning rate no longer decreases, I used AdamW with a weight decay of 1e-4. As shown in the loss curve, the training was still stable.

The sampling results are also comparable to the scheduled version, suggesting that for this specific task and architecture, a well-tuned constant learning rate is sufficient.

Epoch 1

Epoch 5

Epoch 10

Beneath the visualizations are explanations of key concepts that might be unfamiliar to readers: Grouped Query Attention, QK-Norm, and Rotary Positional Embeddings (RoPE). Note that a full explanation of transformers is out of scope for this post. For that, I recommend the Wikipedia article on Transformers. I would also recommend checking out Sebastian Raschka's LLM Architecture Comparison, which is what inspired this post.

In this project, I implemented NeRFs (Neural Radiance Fields) from scratch. NeRFs allow you to create 3D “reconstructions” of images, like the one below, using just ordinary photos that you take with your camera.

Basically, you take a bunch of photos of an object from various angles and it reconstructs a 3D radiance field of the object.

Background Info

This Fall 2025 semester, I’m taking a computer vision class, titled CS 180, taught by Professors Alexei Efros and Angjoo Kanazawa at UC Berkeley. The task for this project is to implement NeRFs, which are basically a way to synthesize a 3D scene from a set of 2D images. This post will walk you through my whole process of training a NeRF¹, from the dataset collection and camera calibration to the actual training of the neural net.

Part 0: Camera Calibration and 3D Scanning

The first task of the project is to take photos of our object from various angles, and place an object called an ARUCO tag next to them. (We are collecting the training data that will be used to train the NeRF later on.)

We also take photos of the camera setup without the object in frame. This is important because it allows us to recover the camera intrinsics. Once we have the camera intrinsics and a bunch of photos of our object, we pass it into a method that undistorts the images, computes camera-to-world (c2w) matrices, and splits the photos into train, test, and validation images.

Here is a visualization of the camera frustums recovered by my code from two different angles:

One thing I’d like to note is that setting the tag length to 60mm, which is its actual length, does not give good results because the markers end up too far apart. For this reason, my code assumes tag_length = 0.6 instead of tag_length = 60.

Part 1: Fitting a Neural Field to a 2D Image

Our first, warmup task is to fit a neural field to a 2D image. This isn’t used to create our NeRFs, but it’s a helpful exercise to get us familiar with the libraries that we will be using in part 2. If you can understand the 2D case well, then it’s easier to generalize to 3D. For the single-image regression task I kept the starter MLP that predicts RGB values from positional encodings of (x, y) coordinates.

This is the model architecture:

I trained my network using Adam as the optimizer with a learning rate of 1e-2 for 1,000 iterations with a batch size of 10k, as stated in the assignment instructions.

All other hyperparameters were the same as the ones described in the spec.

I also evaluate PSNR every 50 iterations and put that in a separate graph which you will see later in the post.

Training progression

The provided “monster” target converges quickly. This makes sense, as the image was relatively simple to begin with.

The fox image shows a similar pattern:

Effect of positional encoding frequency and width

Here we try various values of L and the hidden layer width and put them in a 2x2 grid. This helps us understand how varying the hyperparams affects the overall predictions of the model.

PSNR over training

The PSNR curve shows rapid gains for the first ~200 iterations. This makes sense because the model is learning the lower-frequency, coarse structure of the image. Then it gets smaller improvements as it corrects the higher frequency errors.

Here is the PSNR curve for the custom image (monster.jpg):

Overall, even this simple MLP can overfit a single view pretty well as long as you choose the right hyperparameters.

Other Notes

One optimization that I added is that we precompute the positional encoding of all the coordinates sampled during renders. Recall that we render a model prediction (as seen in the training progression diagram), you have to evaluate the model at many image coordinates to reconstruct its prediction. This requires computing the positional encoding for each coordinate. But since these positional encodings are not learned parameters, we can compute the positionally-encoded version of every pixel in the render ahead of time, which speeds things up a bit.

There are definitely more optimizations we could have done for this part, but as Part 1 is not the main focus on the assignment, I moved on as soon as I got it to work with a reasonable runtime.

Part 2: Fitting a Neural Radiance Field from Multi-view Images

How I implemented each part

With the calibrated cameras from Part 0 in hand, I reconstructed a full NeRF volume for the Lego scene.

Here is the architecture that I used for my neural network, as per the assignment spec:

Parameters: The layers and activation functions are all the same as the architecture described in the spec. I set the positional encoding for the X dimension to have L=10, and for the depth r_d, to L=4

Vectorized Code

I feel like the most crucial part of this section is correctly implementing the core methods: volumetric rendering, sampling points along rays, etc: implement these methods correctly and the training loop is generally straightforward (and doesn’t take too much added code).

At first I implemented some of these methods with numpy, but I found that this made training too slow and was too complicated. It required moving too many objects back and forth between my MPS backend and the CPU.

Instead, I implemented all the methods for sampling, dataloading, etc. in PyTorch, and I added support for batched operations. This made my implementation cleaner and easier to understand and debug.

Dataloader Class

I also found it helpful to create a Dataloader class to abstract away some of the complexities of sampling rays from images. My dataloader init’s method precomputes ray origins and directions for pixels u, v in each image. This is similar to what we did in part 1: it speeds up training and cuts down on redundant computations.

Other Abstractions

I created helper methods such as nerf_forward_pass, render_full_image and make_renders and housed them in a separate file called pt2_utils.py. Given the overall complexity of this project, I found it necessary to separate the code into multiple files and maintain helper methods to avoid code repetition.

Visualizing rays and samples

As per the assignment spec, I visualize rays and sample points as a sanity check to make sure my code worked:

Here is a visualization for just one camera:

where the darker dots along the lines represent points at which we sample.

As mentioned, we insert a small amount of noise to the sampling intervals in order to improve the model’s predictions. This is somewhat noticeable in the visualizations above; the gap between individual sampling points is not perfectly constant - and that’s by design.

Training progression

The snapshots below capture the predicted RGB image for a validation camera every ~200 iterations.

Validation curve

I average the PSNR across six validation images to compute the validation PSNR.

The final PSNR that I achieved was 23.51.

Here is the PSNR curve on that validation set:

Here’s the training loss curve too:

Spherical rendering

After convergence I evaluate the network across a 360 degree camera path using the provided rendering code. I use a held-out C2W matrix from the test split (c2ws_test) for this part.

Here is my spherical rendering:

If the rendering isn’t showing, you can also view it at this link: https://www.vkethana.com/assets/images/nerf/pt2/lego/final_gif.gif

Honestly, I thought the results were pretty good given the limited amount of compute / train time we used for this assignment!

Training with my own data (Part 2.6)

After succeeding on the LeGO image, I moved on to making a NeRF with my own set of images.

Unfortunately, this part of the assignment did not go as well :(

Here is the best spherical rendering that I could achieve:

If the rendering isn’t showing, you can also view it at this link: https://www.vkethana.com/assets/images/nerf/pt2/my_data/final_gif.gif

The model definitely learned something, and you can kind of make out distinct features of the scene in which I took the images, but overall, I would say it was unsuccessful.

Training and PSNR Plots

Here is my PSNR curve, which reached a maximum of value of 10.74 during training.

On the bright side, the model did at least show convergence as seen in the below training loss graph:

This suggests that the loss landscape just wasn’t shaped properly, i.e. that we need to take better photos or change something about the model architecture.

Training progression

The snapshots below capture the predicted RGB image for a validation camera every ~200 iterations.

Code / Hyperparameter changes made

• I downsized my images from 1500 by 2000 to 150 by 200 in order to prevent OOM errors and speed up training.
• Recall that the original LEGO images were 200x200, so it makes sense for these images to be in roughly the same neighborhood of size
• I use a different set of hyperparameters: I set the positional encoding for X to have L = 40 and the positional encoding for d to have L = 15. I chose these experimentally.
• Additionally, I set near and far (which determine the distances at which we sample points on the rays) to 0.1 and 2.0 respectively. I made this change by looking at the rays/frustum visualization and trying different values until the length / spacing of the sampled points looked reasonable.

Speculation on why my model didn’t work for the custom image: The Data

My guess for why the model did not work very well is that the input images simply weren’t good enough: I could have taken photos from more angles to remediate this.

Additionally, I think it would have been good to shoot images that were closer up, given the small size of the object.

Using a small, black-and-white object may have also been a mistake, because many other objects in the scene were white:

• the white tabletop that I placed it on
• the white piece of paper beneath it
• the white wall / couch in the background of some of the images

Conclusion

One key takeaway I got from the project was the importance of datasets in machine learning. The model architecture here was relatively straightforward to implement based on the assignment instructions. Most of the debugging effort went toward the dataloader and the sampling algorithms, not the model itself. Ultimately, I believe it was the dataset and not the model architecture that determined the performance of the NeRF on the images that I took.

Overall, this was a time consuming and challenging project, but it was rewarding and taught me a lot about computer vision and implementing ML models in practice.

This is my third project for CS 180 (Intro to Computer Vision) at UC Berkeley. The project covers image warping, image mosaic generation, and automatic feature detection for image stitching.

Part A.1: Shoot the Pictures

The first part of the assignment is to show at least 2 sets of images with projective transformations between them (fixed center of projection, rotate camera). The reason we are doing this is because we can then use these images to construct mosaics later on in the assignment.

Set 1:

Set 2:

Notice how the camera position is the same in both sets of images; all we’re doing is rotating it. Eventually, we will stitch these images together to create an image mosaic (see part A.4!).

Part A.2: Recover Homographies

A homography is a 3x3 projective transformation that maps points from one image to another when both images capture the same scene.

For this exercise, consider the below two images:

Observe that I manually selected correspondence points between the two images. These are necessary to align the images properly. Only four points are strictly necessary for the algorithm to work, but in practice it makes sense to select many more than four for redundancy.

In homogeneous coordinates: p2 = Hp1, where p1 = [x, y, 1]^T, p2 = [x', y', w']^T, and H is:

[ h11 h12 h13
h21 h22 h23
h31 h32 h33 ]

We convert the system to Cartesian coordinates by dividing by the ‘scaling factor’ introduced by the Homogeneous coordinates:

x' = (h11*x + h12*y + h13) / (h31*x + h32*y + h33)
y' = (h21*x + h22*y + h23) / (h31*x + h32*y + h33)

Rearranging gives two equations per correspondence:

x*h11 + y*h12 + 1*h13 - x*x'*h31 - y*x'*h32 = x'*h33
x*h21 + y*h22 + 1*h23 - x*y'*h31 - y*y'*h32 = y'*h33

Setting h33 to 1 as discussed in class, we arrive at the system of equations:

x' = x*h11 + y*h12 + 1*h13 - x*x'*h31 - y*x'*h32 
y' = x*h21 + y*h22 + 1*h23 - x*y'*h31 - y*y'*h32 

Stacking all pairs yields the linear system Ah = b with
h = [h11 h12 h13 h21 h22 h23 h31 h32]^T. I solved the resulting system of equations using least-squares (because we have >4 correspondences, the system is overdetermined).

This is the homography matrix H which I recovered:

[[ 1.46568340e+00 -1.20053461e-01 -4.98410322e+02]
[ 2.29064746e-01  1.24290079e+00 -1.30462059e+02]
[ 5.01352998e-04 -9.85491138e-05  1.00000000e+00]]

Part A.3: Image Warping and Rectification

In this section, I use the recovered homography H to warp images toward a reference view using inverse warping.
Inverse warping maps each pixel in the output image back into the input image coordinate system. This prevents holes or “gaps” in the outputted image.

1) Warp Functions

After we inverse-warp the image, there’s a problem: the resulting coordinate values are not integers! There are many kinds of interpolation methods we can apply to address this problem. Here I discuss two interpolation methods that I implemented from scratch.

Nearest Neighbor Interpolation: Round coordinates to the nearest pixel value. Runs relatively quickly but the results aren’t as good as bilinear interpolation.

Bilinear Neighbor Interpolation: Use a weighted average of four neighboring pixels. Takes longer but gives better-quality results.

2) Results

In the below image, zoom in on the words “vegetarian” and “vegan” on the sign. Notice how the bordering around the letters is less jagged and pixelated in the bilinear interpolation image, compared to nearest-neighbor.

Here’s another example:

For these examples, there is no ‘‘second image’’: instead, we warp the first image to a predetermined shape (a rectangle).

For the campanile example, I set the second image’s coordinates to that of a rectangle with height four times that of the width. Here’s what the coordinates of that rectangle looked like, for reference:

  "im2Points": [
    [0, 0],
    [1, 0],
    [1, 4],
    [0, 4]
  ]

For the sign image, I used a rectangle with height 1.58 times the width. I determined these values through manual measurement of the image and trial and error.

Part A.4 Blend the Images into a Mosaic

In this next part, I developed an algorithm to stitch images together into panoramas!

Procedure (one-shot, inverse warping)

1. Choose reference: Leave one image unwarped and warp the other into its projection using the computed homography.

2. Create alpha masks: For each image, define an alpha mask that indicates which pixels are valid, post-warping.

   • Pixels filled during warping are set to 1 (valid).
   • Empty or outside regions remain 0. These masks tell you which parts of the canvas belong to which image, and they decide how blending works in areas where the images overlap.

3. Apply weighted averaging to reduce artifacts: To smooth the overlap between the two images, I gave each pixel a weight based on how far it is from the image edge. To accomplish this, I define variables weight_1 and weight_2, where each is computed from the distance transform of its image’s alpha mask:

   dist_1 = distance_transform_edt(alpha_mask_1) # distance transform method imported from scipy
   weight_1 = dist_1 / (np.max(dist_1) + 1e-8)
   # dist_2, weight_2 defined the same way
   

   Pixels near the center of each image have higher weights, and edge pixels have lower weights. When blending, areas of overlap are averaged accordingly:

   mosaic = (weight_1 * image_1 + weight_2 * image_2) / (weight_1 + weight_2)
   

   This produces a soft, seamless transition between the two images. (In practice, the averaging is applied separately to each color channel, not all at once.)

Results

Here are the mosaics with the original images:

Here are the mosaics on their own for reference:

Part B: Automatic Feature Detection and Matching

In Part A, I manually selected correspondence points to create mosaics. This was tedious and error-prone. In Part B, I implemented an automatic system that detects features, matches them between images, and creates mosaics without any manual input.

The algorithm is based on the paper “Multi-Image Matching using Multi-Scale Oriented Patches” by Brown et al.

Part B.1: Harris Corner Detection

The first step is to detect interesting points (corners) in the image automatically. I used the Harris corner detector to find points with strong intensity gradients in multiple directions.

Here’s the result on a test image with all detected Harris corners:

As you can see, there are way too many corners detected.

How can we extract the most important corners and filter out the rest? That’s where ANMS comes in clutch!

Adaptive Non-Maximal Suppression (ANMS)

We use Adaptive Non-Maximal Suppression (ANMS) to get a more uniform spatial distribution of interest points. Each point’s suppression radius depends on its corner strength: points are kept only if they are local maxima within a radius where no stronger neighbor exists. The minimum suppression radius for a point x_i is defined as the distance to the nearest point x_j with significantly higher strength, f(x_i) < c_robust * f(x_j), using c_robust = 0.9.

Here’s the same image after applying ANMS to filter the bad corners.

The selected corners are much more evenly distributed and correspond to distinctive features in the image.

Part B.2: Feature Descriptor Extraction

Once I have corner locations, we need to describe what each corner looks like so we can match them between images.

For each corner, I extracted a 40×40 patch around it, convolved it with a Gaussian filter to blur it (using code from project 2), then downsampled it to 8×8. I normalized each descriptor to have zero mean and unit variance, as is standard practice.

Here are some of the 8×8 feature descriptors I extracted. Each small patch captures the local appearance around a corner point.

Part B.3: Feature Matching

To match features between two images, I compared their descriptors. For each feature in image 1, I found its closest match in image 2 by computing the distance between descriptors.

Of course, not all the matches are good. I used Lowe’s ratio test to filter out bad matches:

• For each feature, find the two closest matches
• Keep the match only if the closest is significantly better than the second-closest
• Ratio threshold: if distance_to_best / distance_to_second < 0.85, keep it

The intuition here is that if you are torn between two options, chances are that neither of them are very good. There should be one option that is vastly better than all the others.

Here are the feature matches I found between two library images:

You might have noticed that some of these matches are not good, even with Lowe’s ratio test. But don’t worry! In the next subpart, we will fix this issue.

Part B.4: RANSAC and Automatic Mosaics

The code we just wrote has a problem - some feature matches are still wrong. We refer to this bad matches as outliers. Conversely, desirable matches (ones that are correct) are called inliers. We want to compute the optimal homography using just the inlier points, not the outliers. That’s where we use RANSAC: Random Sample Consensus, an interesting algorithm that actually has many applications beyond just image processing.

How RANSAC Works:

1. Randomly sample 4 feature correspondences
2. Compute a homography from these 4 points
3. Count how many other matches agree with this homography (or are reasonably close)
4. Keep largest set of inliers
5. Recompute the final homography, using just the inliers

Here are the matches after RANSAC filtering (only inliers shown):

Automatic vs Manual Mosaics

Now, we can create mosaics automatically, without painstakingly labeling correspondences. Here’s a comparison of manual stitching (Part A) versus automatic stitching (Part B) on the same image pairs:

Library Mosaic:

Northside Mosaic:

Road Mosaic:

One key takeaway here is that RANSAC achieves pretty good results! The manual mosaics are better in a few spots, but given that RANSAC doesn’t have access to the manual correspondences, I think it did well.

What I Learned

The biggest takeaway I got from this project was the importance of filtering heuristics: algorithms / techniques for turning a large, messy set of features into something actually useful. An example of a “filtering heuristic” would be Lowe’s ratio test, as well as ANMS. The intuitions behind why we filter out certain points and keep others are definitely something that generalizes beyond image processing. As I continue in my ML journey, I will keep in mind these broad takeaways.

This is my second project for CS 180 (Intro to Computer Vision) at UC Berkeley. The project goes over how filters work in image processing and how we can manipulate images in the frequency domain to get interesting effects like edge detection, sharpening, and multiresolution blending. Here’s an example of a blended image I made as part of this project:

Part 1: Filters and Edges

Part 1.1: Finite Difference Operator

I implemented a zero-padded 2D convolution operation in NumPy, then compared it with scipy.signal.convolve2d.

Implementation Details

I created two versions:

• 4-loop version: Explicitly loops through every kernel element
• 2-loop version: Uses vectorized operations like np.dot

Both implementations produce the same results though the two-loop version is more efficient. One difference from scipy is boundary handling: my implementation uses zero-padding, but scipy.signal.convolve2d has various other options (fill, wrap, symm).

Runtime Comparison

The 2-loop implementation is faster than the 4-loop version due to vectorization. However, scipy.signal.convolve2d is still much faster, probably because it utilizes advanced techniques like FFT to bring down the runtime from O(n^2) - which is my algorithm’s runtime - to O(n log n).

Code Snippets

Two-loop convolution:

def convolve(im, kernel):
    pad_x = kernel.shape[1] // 2
    pad_y = kernel.shape[0] // 2

    padded_im = np.zeros((2 * pad_y + im.shape[0], 2 * pad_x + im.shape[1]))
    padded_im[pad_y : pad_y+im.shape[0], pad_x : pad_x+im.shape[1]] = im

    kernel_flat = kernel.flatten()

    new_image = []
    for i in range(im.shape[0]):
        new_row = []
        for j in range(0,im.shape[1]):
            curr_chunk = padded_im[i : i + kernel.shape[0], j : j + kernel.shape[1]].flatten()
            new_pixel = np.dot(kernel_flat, curr_chunk)
            new_row.append(new_pixel)
        new_image.append(new_row)
    new_image = np.array(new_image)

    assert im.shape == new_image.shape, "Shape mismatch detected!"
    return new_image

Four-loop convolution:

def convolve_slow(im, kernel):
    pad_x = kernel.shape[1] // 2
    pad_y = kernel.shape[0] // 2

    padded_im = np.zeros((2 * pad_y + im.shape[0], 2 * pad_x + im.shape[1]))
    padded_im[pad_y : pad_y+im.shape[0], pad_x : pad_x+im.shape[1]] = im

    kernel_flat = kernel.flatten()

    new_image = np.zeros_like(im, dtype=float)
    for i in range(im.shape[0]):
        new_row = []
        for j in range(0,im.shape[1]):
            res = 0.0
            for m in range(kernel.shape[0]):
                for n in range(kernel.shape[1]):
                    res += kernel[m, n] * padded_im[i+m, j+n]
            new_image[i, j] = res

    assert im.shape == new_image.shape, "Shape mismatch detected!"
    return new_image

Results: Finite Difference Filters

I applied finite difference operators Dx = [1, 0, -1] and Dy = [1, 0, -1]^T:

Top: Dx (horizontal gradients). Bottom: Dy (vertical gradients)

I also tested a 9x9 box filter for smoothing:

Top: My implementation (zero-padding). Bottom: Scipy (default boundary)

Part 1.2: Finite Difference Operator

Using the cameraman image, I computed partial derivatives and edge detection with finite differences.

Partial derivatives, gradient magnitude, and binarized edges (threshold=0.20)

The edge image shows significant noise, which is why I had to apply the threshold of 0.20. I chose 0.20 because lower values recorded too much noise in areas like the sky, while higher values removed important edge details in areas like the tripod. A better approach to this problem is to apply Gaussian smoothing first.

Part 1.3: Derivative of Gaussian (DoG) Filter

Background Info

To create the Gaussian kernel, I select a value of sigma, by default 2.0, and then compute the ksize, which I set as 6 times sigma plus one. I then invoke the cv2.getGaussianKernel with these two parameters. In this approach, because ksize is uniquely determined by sigma, I only specify the sigma values when listing the parameters.

Method 1: Gaussian → Derivative

First blur the image with a Gaussian, then apply finite difference operators:

Gaussian blur (sigma=2.0) then derivatives (threshold=0.10) - much cleaner edges!

I chose a threshold of 0.10 by trial and error: I found that it reduced noise without removing too much valuable information.

Method 2: Single DoG Convolution

By the property of convolution associativity, we can combine the Gaussian and derivative filters first, then apply to the image in one pass. Using the combined DoG filter gives identical results to applying the two filters separately. The Gaussian filter here uses sigma = 2.0 just like in method 1.

The DoG filters:

For example, here’s the edge image generated by the combined filter. This produces identical results to Method 1 and demonstrates the associativity property of convolution: (Image ⊗ Gaussian) ⊗ Derivative = Image ⊗ (Gaussian ⊗ Derivative). Results:

Part 2: Applications

Part 2.1: Image “Sharpening”

The unsharp mask filter enhances edges by emphasizing high frequencies. The process:

1. Blur the image to get low frequencies
2. Subtract blurred from original to isolate high frequencies
3. Add scaled high frequencies back: sharpened = original + α × (original - blurred)

Taj Mahal Results

Original, blurred, high frequencies, and sharpened (α=2.0)

Varying Alpha Parameter

α = 0.5, 1.0, 2.0, 5.0 (more α = more sharpening, but also artifacts)

Additional Example

Original and sharpened (α=40.0)

Blur → Sharpen Experiment

Can sharpening recover a blurred image?

Original sharp vs. blurred then sharpened - some detail lost forever

Sharpening helps, but there’s no way to fully recover lost information. Blurring is irreversible.

Part 2.2: Hybrid Images

Hybrid images take advantage of how the human eye perceives frequencies at different distances. The technique combines the low-frequency components of one image with the high-frequency components of another. Low frequencies capture broad shapes and overall structure, which dominate our perception from far away. High frequencies encode fine details and edges, which we perceive up close. To create a hybrid image, I apply a Gaussian low-pass filter to extract smooth, large-scale features from the first image, then subtract a Gaussian-blurred version from the second image to obtain only its high-frequency details. When these are combined, viewers see different images depending on their distance from the image: the high-frequency image is seen from close up and the low-frequency one from far away.

Example 1: Derek + Nutmeg

Frequency Analysis for this example:

Filtered Results:

For this example, the low-pass Gaussian filter had a sigma value of 15.0 whereas the high-pass Gaussian filter had a sigma value of 6.0. The images were already aligned well, so I performed no manual alignment. (As for the ksize, it is determined by the formula ksize = int(sigma*6.0 + 1).)

Example 2: Tennis Ball + Monster

Example 3: Dog + Cat

Part 2.3: Gaussian and Laplacian Stacks

Gaussian stacks progressively blur images, while Laplacian stacks capture details at each frequency band.

Apple Stack (for Oraple blending)

Orange Stack (for Oraple blending)

Part 2.4: Multiresolution Blending

Using Laplacian stacks, we blend images seamlessly across frequency bands.

Oraple (Apple + Orange)

Blending Process:

High frequencies show fine details, low frequencies show structure. Here I recreate the outcomes of Figure 3.42 from Szelski

Final Result:

The classic Oraple!

Additional Example 1: Day/Night Blend

Additional Example 2: Irregular Mask + Portal (from Minecraft)

Here I use a circular mask to insert another image inside of the purple portal you see below.

Circular mask creates a portal effect between dimensions

Here’s how the circular mask works:

def make_circular_mask(h, w, center, radius, feather=20):
    Y, X = np.ogrid[:h, :w]
    dist = np.sqrt((X - center[0])**2 + (Y - center[1])**2)
    mask = np.clip((radius + feather - dist) / feather, 0, 1)

    return mask

Most Important Thing I Learned

The most important thing I learned in this project is that classical methods are still very useful in 2025. These days neural network-driven approaches are very popular in the CS world, and rightfully so, but this project shows us that sometimes it doesn’t hurt to go old-school. For example, we could have used generative AI to merge the orange and apple photos, but there’s no reason to do that when the multiresolution blending approach can get the same result with a fraction of the resources.

I implemented, from scratch, an image colorization algorithm that can turn this:

Into this:

Part 0: Background Information

This Fall 2025 semester, I’m taking a computer vision class, titled CS 180, taught by Professors Alexei Efros and Angjoo Kanazawa at UC Berkeley. The first assignment is to colorize some images from the Prokudin-Gorskii photo collection, an example of which you saw above. Basically, the photographer took three exposures of every scene onto a glass plate using blue, green, and red filters. The “blue” image is the one on the top, the “green” the one in the middle, and the “red” the one at the bottom.

Our task is to turn the stack of three grayscale images into a single color image.

Part 1: Single-Scale Colorization

The first approach you might try is to naively divide the image into three. Take this image for example:

If we naively stack the three thirds of the image on top of each other, here’s what you end up with:

We need a better approach - a way to somehow align the three channels. That is, we want to find translation (dx, dy) to apply to the green and red channels such that they differ as little as possible from the blue image.

Translation Scoring Metric

There are many ways you can measure the “best translation” between two images. I used the Normalized Cross-Correlation (NCC) metric: I flatten both images into 1-D column vectors, then normalize them and compute their dot product. In pseudocode, we want to find: dot_product(image1./||image1|| , image2./||image2||). The higher the resulting dot product, the better the match between the two images.

Another option is to use the L2 norm, which can deliver reasonably good results as well. However, the assignment instructions recommended NCC, and I found that it worked well for me.

Translation Search Algorithm

Now that we know how to measure a good displacement, how can we actually calculate which displacement is optimal? One idea is to brute-force try all possible translations within a particular range: I used the interval [-15, 15]. This algorithm¹ is reasonably performant, but it had a problem: it was too slow, taking upwards of 50 seconds for some images. We can do better by being smarter about where we search.

Instead of checking every possible displacement, I start with a larger step size to get a rough estimate, and then progressively refine the search with smaller step sizes until I get to pixel-level accuracy. This way, we quickly narrow down the promising areas of the search space.

For example, let’s say that we set the max number of search steps to 4: I first try displacements in increments of 4 pixels. Once I find the best candidate, I reduce the step size to 2 and search in that neighborhood, and finally reduce the step size to 1 to lock in the exact displacement.

Here’s an example of what the algorithm does to a sample image. Notice how neatly aligned the channels are compared to before:

Part 2: Using an Image Pyramid

The single-scale approach we just discussed works well for lower-resolution JPEG images. But for higher-resolution TIFF images, we need a separate approach: the optimal displacement might be a lot more than 15 pixels in any direction. This is where the image pyramid comes in. Here’s how the image pyramid approach works:

• Build a pyramid for each channel by repeatedly downsampling by 2 for several levels. I found that having 5 levels worked well for me.
• Start at the coarsest level and align red/green to blue.
• Propagate the found displacement to the next level, scaling by 2. We scale by two because the new image has twice the resolution in both dimensions.
• Refine our estimation at the new level using the single-scale approach described earlier.
• Repeat until you reach the original resolution.

This drastically improves results for some images. For example, the Emir of Bukhara image below:

Without the pyramid, it looks like this:

Conversely, with the pyramid it looks much better:

Getting the Emir of Bukhara image to align properly was a big challenge. I had to patiently tweak the pyramid levels and search range to get it right. Even then, my result isn’t perfect because the red channel’s brightness values are very different from those of the blue and green images. I think there is a limit to how good the reconstruction can be using just NCC and (x, y) translations. A more advanced technique like edge detection might be helpful here.

Misc. Implementation Details

• I crop 12.5% off the borders of the image before computing the NCC, as we don’t want border artifacts to influence the final alignment.
• As a post-processing step, I crop 10% off the borders of the final images; the borders of the image contain artifacts from color alignment and are not visually appealing.
• Resizing of images is done via cv2.resize as opposed to manually applying a Gaussian filter.

Gallery of Pyramid Results

Here are some of the best results I obtained using the image pyramid approach. My displacements are in the form (dx, dy).

cathedral Green channel offset: (2, 5) Red channel offset: (3, 12)

church Green channel offset: (4, 25) Red channel offset: (-4, 58)

emir Green channel offset: (24, 49) Red channel offset: (57, 103)

harvesters Green channel offset: (17, 60) Red channel offset: (14, 124)

icon Green channel offset: (17, 41) Red channel offset: (23, 89)

italil Green channel offset: (21, 38) Red channel offset: (35, 77)

melons Green channel offset: (10, 81) Red channel offset: (13, 178)

monastery Green channel offset: (2, -3) Red channel offset: (2, 3)

self_portrait Green channel offset: (29, 79) Red channel offset: (37, 176)

siren Green channel offset: (-6, 49) Red channel offset: (-25, 96)

three_generations Green channel offset: (14, 53) Red channel offset: (11, 111)

tobolsk Green channel offset: (3, 3) Red channel offset: (3, 6)

lastochikino Green channel offset: (-2, -3) Red channel offset: (-9, 75)

lugano Green channel offset: (-16, 41) Red channel offset: (-29, 93)

Gallery of Additional Images from the Prokudin-Gorskii Collection

I selected these additional images from the original collection to test my algorithm:

Green channel offset: (3, 29) Red channel offset: (-7, 71)

Green channel offset: (-12, 39) Red channel offset: (-27, 86)

Green channel offset: (-18, 66) Red channel offset: (-34, 146)

MSE Policy

The Mean-Squared Error (MSE) policy is a straightforward approach to imitation learning with action chunking. In this setup, the policy \(\pi_\theta(o_t)\) maps the current observation \(o_t\) directly to a sequence of future actions \(A_t = (a_t, a_{t+1}, \dots, a_{t+K-1})\), where \(K\) is the chunk size. We use action chunking (sampling K actions from the policy at once) because it reduces the frequency of policy queries and often leads to smoother trajectories.

Architecture

For the MSE policy, I used a Multi-Layer Perceptron (MLP) architecture. The model takes the 5-dimensional state as input and outputs a flattened vector of size \(K \times \text{action\_dim}\). (Here, the action dimension is two, corresponding to the agent’s target coordinates.)

• Hidden Layers: 3 layers with 512 units each.
• Activation: ReLU for hidden layers, no activation for the output layer.
• Input Dimension: 5 (Push-T state).
• Output Dimension: \(K \times 2\) (Chunk size \(\times\) 2D actions).

Loss Function

The policy is trained by minimizing the L2 distance between the predicted action chunk and the expert’s action chunk:

Flow Matching Policy

While the MSE policy is simple, it can struggle with multimodal distributions (where the expert might take multiple different valid paths). Flow matching addresses this by learning to transform a simple noise distribution into the expert’s action distribution.

How it Works

Flow matching learns a conditional vector field \(v_\theta(o_t, A_{t,\tau}, \tau)\) that “pushes” noise samples toward the data distribution. We define a linear interpolation between a noise sample \(A_{t,0} \sim \mathcal{N}(0, I)\) and the expert action chunk \(A_t\):

The network is then trained to predict the velocity \((A_t - A_{t,0})\) that moves the interpolated sample toward the target:

Inference

At test time, we sample \(A_{t,0} \sim \mathcal{N}(0, I)\) and integrate the learned vector field from \(\tau=0\) to \(\tau=1\) using Euler integration:

This iteratively refines the action starting from pure noise. The exact number of denoising steps (which determines the value of $\Delta\tau$) is 30 for my training runs.

Results

MSE Policy Results

Training Curves

Performance Videos

Flow Matching Policy Results

Training Curves

Performance Videos

Qualitative Comparison

Qualitatively, we observe several key differences between the MSE and Flow Matching policies in the Push-T environment:

• Initialization and Early Training: At the start of training, the flow matching policy appears “jittery” and erratic. This is because it is sampling from a noise distribution and the vector field is not yet well-defined. In contrast, the untrained MSE policy tends to output a mean action that often results in the agent staying static or moving in an unhelpful direction.
• Final Performance: By the end of training, the flow matching policy significantly outperforms the MSE policy. The flow matching agent demonstrates clearer evidence of “planning”. It (re-)orients itself correctly relative to the T-block and makes purposeful pushes toward the goal.
• The “Mean Action” Problem: The performance gap is likely due to the training objective. An MSE policy learns to predict the average expert action for a given state. In multimodal scenarios (e.g., if an expert sometimes goes left and sometimes goes right to push the block), the MSE policy will predict the average, which may be straight into the block—leading to suboptimal behavior. Flow matching, being a generative model, can represent these multiple modes effectively, allowing it to “commit” to one valid path.

Conclusion

This assignment was fun. My next steps are to train policies for harder environments than Push-T, which will naturally necessitate the use of techniques more sophisticated than mere imitation learning. More on that coming soon!

“Le président Emmanuel Macron assure le peuple canadien que le gouvernement français va continuer à défendre le Canada contre la menace américain.”

Even if you don’t speak French, you can probably understand, or at least get the gist of, the above sentence: the French president Emmanuel Macron is assuring the “peuple canadien” (Canadian people) about something involving the “gouvernment français” (French government). Imagine reading thousands of sentences like this and gradually acquiring French through cognates you already know. This is a type of comprehensible input, a language learning technique popularized by linguist Stephen Krashen in the 1980s.

Comprehensible input is a good language learning method, but creating it is very hard. A native speaker has to painstakingly create thousands of sentences that speakers of another language can understand, gradually increasing the difficulty as the learner progresses. Resources like this actually do exist for a number of languages. For example, learners of Latin can use Lingua Latina per se Illustrata, a book that teaches you Latin using exclusively sentences in Latin. However, writing this text took years of effort on the part of Hans Ørberg, a linguist who dedicated a large part of his life to teaching Latin. Ørberg carefully wrote the sentences in his book to only use cognates an English speaker could understand and to avoid any complicated syntax an English speaker would have a hard time understanding.

What if there was a way to automate the process of generating “cognateful” sentences using large language models? Language models like GPT-4o and o1 have good enough reasoning ability to tell what words in a foreign language are and aren’t cognate with English. They can reliably generate sentences in other languages like French without any additional training. In short, this is an ideal use case for LLMs.

What I did

I made a simple interactive language-learning app, hosted at cognateful.vkethana.com. It teaches you French using stories written exclusively in French.

Users are provided with a brief, ten-sentence story consisting of French sentences. The user’s task is to read the sentences and translate them into English. The app tells you whether or not your translation is correct using GPT-4o-powered scoring. Based on your performance on the exercises, it assigns you a “difficulty” score, which goes up and down depending on your performance. Users then are served sentences at appropriate levels of difficulty based on their performance. For those who are curious, the source code for this project can be found here.

Caveats: This is just a minimum viable product. The number of sentences in the app is limited. I don’t speak French (yet), so sentences may contain mistakes. But the interface, scoring system, and sentence generation are all functional, and I think they will work at scale. The biggest hurdle to improving the app is increasing the number of sentences while not compromising on sentence quality.

How I generated and scored the sentences

Scoring

I have to explain sentence scoring before sentence generation because the scoring system influenced the prompt used to generate the sentences. I give o1-preview a sentence and ask it to score its difficulty on a scale of 0 to 3, 0 being very hard to understand for a monolingual English speaker and 3 being very easy.

Score 0: Completely unintelligible to English speakers. Example: “Je veux manger du pain.”

Score 1: Contains some cognate words, but contains words unintelligible to an English speaker. The cognates might allow them to guess the general topic but not the main idea or actual meaning. Example: “Le maître savant utilise beaucoup de livres.” (Has cognates like “savant” but key verbs/objects aren't cognates)

Score 2: Contains many cognate words. An English speaker might guess the main idea but would miss important details or nuances that change the meaning. Example: “Le patient refuse absolument de prendre ses médicaments malgré les protestations constantes du docteur.” ¹

Score 3: Fully understandable through cognates. Use almost exclusively cognate words except for basic connectors. Example: “Le président Emmanuel Macron assure le peuple canadien que le gouvernement français va continuer à défendre le Canada contre la menace américain.”

(Side Note: I found that using o1-preview is necessary for good-quality scoring. Other models – 4o, 4o-mini, and o1-mini – had a hard time determining what a cognate was. Also, they were too lenient, often assigning scores of 3 to sentences that in my opinion an English speaker wouldn’t be able to fully understand. Even o1-preview sometimes uses words an English speaker wouldn’t understand.)

Generation

To save time and money, I pregenerate all sentences on the site using GPT-4o and o1-preview. Unsurprisingly, o1 made much higher quality sentences, but 4o did a pretty good job too, and any bad-quality sentences are flagged by the scoring system anyway. So it’s OK to cut costs and use a cheaper model when generating sentences, but not when scoring them. To generate sentences, I made a function generate_story that takes in a target difficulty and then asks GPT-4o to generate a story consisting of sentences at that difficulty. This allows me to create a variety of sentences at different difficulty levels to suit the user’s needs.

To make the final set of sentences seen on the site, my script repeatedly calls, and saves the output of, generate_story with the target difficulty set to a randomly-generated integer between 0 and 3, inclusive. Here’s a breakdown of how many sentences the site currently has per difficulty level (recall that 1 story = 10 sentences).

# Source code: https://github.com/vkethana/cognate_sentences
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
language_codes = {
    'fr': 'French'
}
SENTENCE_GENERATION_MODEL = 'gpt-4o'
SENTENCE_SCORING_MODEL = 'o1-preview' # 'o1' doesn't work for some reason
 
def generate_story(lang_code, num_sentences, target_difficulty):
    system_prompt = f"""
    You are a fluent speaker of both {language_codes[lang_code]} and English.
    Generate exactly {num_sentences} {language_codes[lang_code]} sentences that:
    1. Form a coherent narrative where each sentence follows from the previous one
    2. Target difficulty level {target_difficulty} using these criteria:

        Level 0: Completely unintelligible to English speakers.
        Example: "Je veux manger du pain."

        Level 1: Contains some cognate words, but is largely unintelligible to an English speaker. The cognates might allow them to guess the general topic but not the actual meaning.
        Example: "Le maître savant utilise beaucoup de livres." (Has cognates like "savant" but key verbs/objects aren\'t cognates)

        Level 2: Contains many cognate words. An English speaker could understand the main idea but would miss important details or nuances that change the meaning.
        Example: "Le patient refuse absolument de prendre ses médicaments malgré les protestations constantes du docteur."
        An English speaker would get "patient refuses absolutely to take medications" and "constant protestations doctor" but might miss "his" and "despite", changing their understanding of whose medications and the relationship between the refusal and protestations.

        Level 3: Fully understandable through cognates. Use almost exclusively cognate words except for basic connectors.
        Example: "Le président Emmanuel Macron assure le peuple canadien que le gouvernement français va continuer à défendre le Canada contre la menace américain."

        DIFFICULTY TARGETING STRATEGIES:
        Difficulty 0: Use basic, high-frequency native vocabulary, avoid international words
        Difficulty 1: Use 25-30% cognates in non-crucial positions. Has cognates but leaves major meaning gaps.
        Difficulty 2: Use 50-60% cognates in main concept positions. Sentence is mostly understandable but has subtle meaning changes due to missed words\n
        Difficulty 3: Use 80-90% cognates, especially for key meaning-bearing words. Any small connecting words (le, que, etc.) can be ignored without losing meaning. Should be assigned sparingly - only when missed words don\'t change meaning\n

    Format your response as a JSON array of {num_sentences} objects:
    {{
        "sentence": "<Generated sentence>",
        "target_difficulty": {target_difficulty},
        "reasoning": "<Why this sentence matches difficulty. If this is not the first sentence, also explain why this continues the story from the previous sentence in this JSON array.>",
        "cognate_words": [<List of cognates used>]
    }}

    Important: Each sentence must directly follow from the previous one to form a coherent story.
    Generate {num_sentences} sentences meeting these criteria (difficulty level and story continuation).
    Note: Please do not include Markdown formatting tags (```) in your response, as my parser will not be able to interpret them.
    """
    
    response = client.chat.completions.create(
        model=SENTENCE_GENERATION_MODEL,
        messages=[{'role': 'user', 'content': system_prompt}],
        temperature=1.0
    )
    
    # Parse generated sentences
    return json.loads(response.choices[0].message.content)

def gpt_scored_rubric_batch(sentences):
    '''
    Score multiple French sentences at once using GPT-4.

    Args:
        sentences: List of sentences to score
    Returns:
        List of scoring results
    '''

    system_prompt = f"""
    You are an expert in French to English translation. I will give you {len(sentences)} sentences in French, and I want you to score each of them on a scale from 0-3 using the following rubric:

    0: Completely unintelligible to English speakers.
    Example: "Je veux manger du pain."

    1: Contains some cognate words, but contains words unintelligible to an English speaker. The cognates might allow them to guess the general topic but not the main idea or actual meaning.
    Example: "Le maître savant utilise beaucoup de livres." (Has cognates like "savant" but key verbs/objects aren\'t cognates)

    2: Contains many cognate words. An English speaker might guess the main idea but would miss important details or nuances that change the meaning.
    Example: "Le patient refuse absolument de prendre ses médicaments malgré les protestations constantes du docteur."
    An English speaker would get "patient refuses absolutely to take medications" and "constant protestations doctor" but might miss "his" and "despite", changing their understanding of whose medications and the relationship between the refusal and protestations.

    3: Fully understandable through cognates. Use almost exclusively cognate words except for basic connectors.
    Example: "Le président Emmanuel Macron assure le peuple canadien que le gouvernement français va continuer à défendre le Canada contre la menace américain."

    Important scoring notes:
    - Score 0 sentences have little to no cognates
    - Score 1 sentences have cognates but leave major meaning gaps
    - Score 2 sentences are mostly understandable but have subtle meaning changes due to missed words
    - Score 3 should be assigned sparingly - only when missed words don’t change meaning

    For each sentence, provide a JSON object with these fields:
    {{
      "sentence": "<Sentence>",
      "cognate_words": [<List of Cognate Words>],
      "reasoning": "<Reasoning for the score>",
      "score": <Numerical for the Sentence (0-3)>
    }} 

    Please format your response as a JSON array of these objects. You should have {len(sentences)} objects in your array.

    Here are the sentences to score:
    {json.dumps(sentences, ensure_ascii=False)}
    Note: Please do not include Markdown formatting tags (```) in your response, as my parser will not be able to interpret them.
    """

    completion = client.chat.completions.create(
        model=SENTENCE_SCORING_MODEL,
        messages=[
            {'role': 'user', 'content': system_prompt}
        ],
        temperature=1
    )
    
    response_text = completion.choices[0].message.content.strip()
    try:
        results = json.loads(response_text)
        return results
    except json.JSONDecodeError:
        print("Error: Failed to decode JSON from the response.")
        raise

Approaches that didn’t work

• Sentence starters: I was initially worried that repeatedly asking the model to generate sentences would result in the same stories being generated over and over. To deal with this, I modified my prompt to randomly pick a sentence starter from a hardcoded list of unfinished French sentences. I then asked the model to generate sentences which continued off the sentence starter. This works, but I eventually got rid of it and found that the sentences were still diverse enough.
• Live generation: Rather than pre-generating the sentences, I originally thought about generating them on the spot and feeding the model with information about the user’s past performance. But pre-generating sentences is cheaper, and we can still adapt to the user’s performance using the scoring system.
• Cognate ratios: Originally, I scored sentences using a weighted combination of GPT-4’s judgments and the percentage of cognate words in the sentence. This is a bad idea because it treats all cognate words equally, leading to inaccurate scoring. For example, “ouvre” and “technologie” are both cognates, but the latter is much easier to understand. A possible fix for this problem is to take into account the age of acquisition (AoA) of the cognate words. I plan to return to this idea, using a system that gives better scores to some cognate words.

Some optimizations I made

• Chain of Thought Prompting: I tell the model to reason through its scoring and generation process. This substantially reduces hallucinations and improves the output quality of weaker models. For example, my prompt for sentence scoring tells the LM to use the following output in its response:

    {
      "sentence": "<Sentence>",
      "cognate_words": "[<List of Cognate Words>]",
      "reasoning": "<Reasoning for the score>",
      "score": "<Numerical for the Sentence (0-3)>"
    }

• Batching LLM calls to reduce inference costs: Sentences are generated and scored in batches of 10, which brings down the cost and time of generating and scoring stories a lot.
• Require JSON outputs: I wasted a lot of time trying to get the LM to output in a format that was easy to parse in Python. Eventually I realized that JSON outputs were perfect for this situation. Anecdotally, it feels like formatting-related hallucinations are less common when the model is tasked with outputting JSON and not an ad-hoc, user-defined format.

Findings

With the right prompting and search strategy, LLMs can definitely generate “cognateful” sentences. I think all three of these examples can be mostly understood by an English speaker:

Netflix est une plateforme de streaming vidéo qui offre une large sélection de films, séries télévisées et documentaires.

L’intellectuel français Voltaire a dit: “La tolérance est un ingrédient essentiel de la civilisation.”

En 1310, le navigateur italien Marco Polo a traversé l’océan Indien.

Some cognate words have a stronger association with high-scoring sentences than others. For example, université and enthousiasme have average scores of 3.00, whereas recherches and ouvre have average scores of 1.67. These findings might seem obvious at first glance, but it’s proof that the scoring function is doing something right! Cognates that are very easy to understand receive high scores. More difficult or obscure cognates receive lower scores. Here’s a non-exhaustive table of some cognates and the average scores of the sentences containing them.

Cognate Words Sorted by Average Score:
université: 3.00
importante: 3.00
spectateurs: 3.00
applaudissent: 3.00
découverte: 3.00
dans: 3.00
urgente: 3.00
révèle: 3.00
admire: 3.00
sculptures: 3.00
équipe: 3.00
directeur: 3.00
technique: 3.00
exposition: 3.00
inclut: 3.00
beaucoup: 3.00
annonce: 3.00
étudiant: 3.00
présenter: 3.00
communauté: 3.00
anciennes: 3.00
invitation: 3.00
accepte: 3.00
enthousiasme: 3.00
applaudit: 3.00
renommée: 3.00
Le: 3.00
révolutionnaire: 3.00
la: 3.00
invite: 3.00
technologie: 3.00
immédiatement: 3.00
globales: 3.00
planifier: 3.00
vaisseau: 3.00
spatial: 3.00
atteint: 3.00
contacte: 3.00
agent: 3.00
crée: 3.00
reconnaissance: 3.00
collection: 3.00
acclamation: 3.00
encouragé: 3.00
peintures: 3.00
modernes: 3.00
objet: 3.00
propre: 3.00
exposer: 3.00
idée: 3.00
potentiel: 3.00
énorme: 3.00
vision: 3.00
nationale: 3.00
éducation: 3.00
système: 3.00
théories: 3.00
gagne: 3.00
talent: 3.00
acceptent: 3.00
énergie: 3.00
artistique: 3.00
peut: 3.00
ville: 3.00
Daniel: 3.00
physique: 3.00
Leur: 3.00
thème: 3.00
Londres: 3.00
Marie: 3.00
hôtel: 3.00
glaciers: 3.00
internationale: 2.93
succès: 2.90
célèbre: 2.87
présente: 2.83
entier: 2.83
Paris: 2.82
musée: 2.80
musicien: 2.80
moderne: 2.80
nouvelle: 2.79
organise: 2.78
principal: 2.75
problème: 2.75
académique: 2.75
économique: 2.75
voyager: 2.75
secret: 2.75
performance: 2.75
formule: 2.75
Les: 2.75
incroyable: 2.75
monde: 2.73
professeur: 2.72
conférence: 2.72
situation: 2.71
propose: 2.71
reçoit: 2.71
innovante: 2.67
amis: 2.67
étranger: 2.67
marche: 2.67
grande: 2.67
inspiration: 2.67
plan: 2.67
action: 2.67
ambitieux: 2.67
planète: 2.67
diffusent: 2.67
explorer: 2.67
avancée: 2.67
internationales: 2.67
prestigieux: 2.67
document: 2.67
résultats: 2.67
réalise: 2.67
autorités: 2.67
visiter: 2.67
positive: 2.67
œuvre: 2.67
discutent: 2.67
collaborer: 2.67
arrivent: 2.67
diplomates: 2.67
inspire: 2.60
découvrent: 2.60
finalement: 2.60
spectaculaire: 2.60
projet: 2.60
attention: 2.60
article: 2.60
artiste: 2.59
événement: 2.57
médias: 2.57
mission: 2.57
Finalement: 2.57
scientifique: 2.53
étudiants: 2.50
extraordinaire: 2.50
apparaît: 2.50
solution: 2.50
décide: 2.50
habitants: 2.50
réunion: 2.50
ancienne: 2.50
documents: 2.50
président: 2.50
gouvernement: 2.50
détails: 2.50
experts: 2.50
impact: 2.50
solutions: 2.50
Europe: 2.50
décident: 2.50
concert: 2.50
traditionnelle: 2.50
information: 2.50
gouvernements: 2.50
astronautes: 2.50
commencent: 2.50
spatiale: 2.50
animaux: 2.50
exotiques: 2.50
entrée: 2.50
grotte: 2.50
inscriptions: 2.50
hésite: 2.50
dangereuse: 2.50
fans: 2.50
soir: 2.50
admiration: 2.50
David: 2.50
palais: 2.50
innovation: 2.50
prix: 2.50
exceptionnelles: 2.50
excitation: 2.50
collaborent: 2.50
ingénieurs: 2.50
refuge: 2.50
important: 2.50
national: 2.50
menace: 2.50
nation: 2.50
étudier: 2.50
retourne: 2.50
réalité: 2.50
Avec: 2.50
scène: 2.50
style: 2.50
tableau: 2.50
unique: 2.50
New York: 2.50
histoire: 2.50
développement: 2.50
collègues: 2.50
presse: 2.50
locales: 2.50
universités: 2.50
visiteurs: 2.50
organiser: 2.50
mondiale: 2.50
intelligence: 2.50
interrompt: 2.50
recherche: 2.50
inspiré: 2.50
attire: 2.50
international: 2.50
discuter: 2.50
climatique: 2.50
œuvres: 2.50
nouveau: 2.50
complexes: 2.50
film: 2.50
mer: 2.50
participer: 2.50
démonstration: 2.50
très: 2.50
réunions: 2.50
Lucie: 2.50
voyage: 2.46
est: 2.45
commence: 2.43
art: 2.43
découvre: 2.40
public: 2.40
musique: 2.40
peinture: 2.40
incident: 2.33
docteur: 2.33
magnifiques: 2.33
continue: 2.33
nouvelles: 2.33
aventure: 2.33
explorent: 2.33
célèbres: 2.33
extraterrestre: 2.33
change: 2.33
humanité: 2.33
expédition: 2.33
rencontre: 2.33
Pierre: 2.33
internationaux: 2.33
curieux: 2.33
intérêt: 2.33
critique: 2.33
révélation: 2.33
exprime: 2.33
rapidement: 2.30
initiative: 2.25
nombreux: 2.25
grand: 2.25
participants: 2.25
galerie: 2.22
offre: 2.20
scientifiques: 2.20
arrive: 2.17
visite: 2.17
mystérieuse: 2.14
présentation: 2.00
brillante: 2.00
village: 2.00
enfants: 2.00
avec: 2.00
alarme: 2.00
affirme: 2.00
citoyens: 2.00
mesures: 2.00
résoudre: 2.00
critiques: 2.00
changements: 2.00
prépare: 2.00
historiques: 2.00
souvenirs: 2.00
observe: 2.00
groupe: 2.00
lettre: 2.00
étrange: 2.00
soupe: 2.00
plats: 2.00
France: 2.00
pour: 2.00
cabane: 2.00
analysent: 2.00
informations: 2.00
incroyables: 2.00
idées: 2.00
espion: 2.00
aide: 2.00
couvre: 2.00
encore: 2.00
certains: 2.00
innovantes: 2.00
conférences: 2.00
invité: 2.00
magnifique: 2.00
française: 2.00
Isabelle: 2.00
acteurs: 2.00
paysage: 2.00
jour: 2.00
significative: 2.00
mystérieux: 1.83
Après: 1.80
impatience: 1.75
forêt: 1.67
ensemble: 1.67
ouvre: 1.67
contribution: 1.67
recherches: 1.67
chat: 1.67
Thomas: 1.67
cuisine: 1.60
ami: 1.50
trésor: 1.50
secrète: 1.50
femme: 1.50
table: 1.50
monte: 1.50
le: 1.50
étranges: 1.50
porte: 1.50
lit: 1.50
sombre: 1.50
projets: 1.50
suit: 1.50
support: 1.50
professeurs: 1.50
long: 1.50
Luc: 1.40
soleil: 1.33
mais: 1.33
entre: 1.33
livre: 1.33
cherche: 1.25
maison: 1.00
famille: 1.00
part: 1.00
arbre: 1.00
et: 1.00
matin: 1.00

Features I plan to add

• Scale up the number of sentences in the app.
• Bring back beam search for sentence generation: Currently I’m making stories by generating 10 sentences at once. A better, but slower and more costly, way to get high-scoring sentences is to generate many options, expand the highest-scoring ones, and discard the rest, gradually building up the stories.
• Remove all English from the UI. Instead, express UI functions using images and icons. Any words which appear on the screen should be in the target language, not English, in order to immerse the user as much as possible.
• Come up with better heuristics for bumping up and down the user’s difficulty score based on their performance. Right now, we simply decrement / increment the user’s difficulty by 0.10 for each correct or incorrect answer. (Note that lower difficulty values = harder, not easier, sentences)
• Improve sentence scoring: I think that this is the hardest part of the project and that the sentence scoring has a lot of room for improvement. For example, I could modify the scoring system to use a weighted combination² of two things: GPT-4 judgement scoring and the presence of certain high-scoring cognate words (see “Findings” above).
• Add support for languages other than English.

How you can help

If you’re familiar with NLP, linguistics, or software development, you can help out by suggesting solutions to the following blockers that I’m currently facing.

• Cheaper and faster scoring: Is there a cheaper, more scalable way to score sentences than what I’ve described here? To recap, using reasoning models like o1 is effective but slow and expensive. Using models other than o1 results in bad quality sentences. Using non LLM-powered scoring misses the nuances of what makes a sentence easy or hard to understand.
• More intuitive UI: Users should be able to understand how the app works without reading an entire blog post about it. How can we engineer the UI so that it’s immediately obvious how to use the app and why it will teach you French?
• Better gameplay loop: Right now, all the user does is read sentences, translate them, and watch their score go up or down. How can we make the app more fun?

Thanks for reading my post! If you liked reading it or have thoughts on how to improve the project, please reach out over email or leave a comment below.

Here’s what I learned from the talk:

Background info: What is Clever?

Tyler started off the speaker event by explaining that Clever is an educational technology company that provides single sign-on, software setup help, and many other technology integrations for schools. With a team size of 210 people, Clever is used by over half of K-12 schools in the US. Even more surprising is that of the 100 largest school districts in America, 97 use Clever. It was funded by Y Combinator in 2012, and in 2021 it was acquired by Kahoot.

Their most well-known product is the single sign-on platform, which I remember using a lot in high school.

1. An interesting monetization model

American and Canadian schools aren’t charged anything for using Clever’s platform unless they specifically opt-in for some extra add-on features. The company instead generates revenue by asking other ed-tech tools, e.g. Google Classroom or Aeries (a tool used to track student grades and attendance) to pay to be integrated into Clever. It’s a win-win for all parties involved since schools pay nothing, Clever makes money, and the apps integrated into Clever get more users.

2. The team had domain expertise

One of the co-founders of Clever, Dan Carroll, worked as a teacher and later as an IT Director at a school in Denver, Colorado. As a result, the team had a clear sense of what kind of product teachers would actually use. This also prevented the founders from creating a product that sounds good on paper but wouldn’t actually work. In other words, “specific knowledge” played a crucial role in the development of Clever’s product. Coined by entrepreneur and angel investor Naval Ravikant, specific knowledge is a type of intuition that is gained through real-world employment or apprenticeships and is crucial to launching startups in domain-specific areas like ed-tech.

One other thing: a common mistake Tyler warned about is assuming that because most of us have gone through school as students, we don’t need any additional experience to differentiate between good and bad ed-tech ideas.

3. International acquisitions = lots of bureaucracy

During the pandemic, Kahoot offered to acquire Clever for $500 million. There was just one issue: bureaucracy. Since Kahoot is a Norweigan company, the US government was not happy that students’ (potentially sensitive) educational data was being handed over to other countries. For a while, the founders were concerned the deal wasn’t going to happen. But then one day, the team randomly got a call from the company lawyer, saying that the acquisition had been OKed.

The bureaucracy surrounding international acquisitions is very complex, and the history behind all of this is pretty interesting. It started in 1980s when there was widespread fear that American industries would get acquired by companies in other countries, especially Japan. For example, Fujitsu offered to acquire Fairchild Semiconductors, a pretty notable company at the time. This caused a little bit of panic in the United States, and in response, the Exxon-Florio amendment was passed. The law allowed the U.S. President to block any “foreign investment” deemed a threat to national security. It also expanded the power of the Committee on Foreign Investment in the United States, or CFIUS, the same organization that later oversaw the Kahoot acquisition.²

Another thing I’m curious about is, what kind of regulations exist for ed-tech companies in America, and how do they affect the kinds of ideas that founders are willing to take a risk on?

4. High valuation isn’t always a good thing

Clever’s initial fundraising rounds were very successful. Their Series A, for example, raised over $10 million USD through Sequoia capital. Investors were impressed and the team was enthusiastic. A few years after launching, though, the company ran into a problem: because they’d already raised so much money, their valuation was “too high to raise again”. (If anyone can find a news article or website which more thoroughly explains how Clever’s valuation was too high to raise again, how they dealt with this issue, and the long-term implications for the company, please reach out!)

Related: some platforms avoid seed funding altogether but end up successful anyway. For example, Dingboard was successfully bootstrapped even though the product is a compute-intensive, (presumably very expensive) AI-powered image editing tool.

5. Downtime bad

In The Social Network, there’s a scene where Mark Zuckerberg panicks over Facebook’s servers going down. He yells at his cofounder for causing the site to go down, complaining that even a short window of downtime would irreparably destroy The Facebook’s reputation and make it look “uncool.”

The Social Network is a dramatized movie, but I don’t think fictional-Zuckerberg was far off when he warned about the dangers of downtime. In the case of Clever, going down for just three days was enough to cause a mini-disaster for the company, complete with phone calls from upset school principals and long hours spent debugging technical issues. To make matters worse, the downtime happened during standardized testing season.

Luckily, the incident was resolved and from then on, it was referred to as “The Great Three-Day Outage”. (You can actually see this phrase at the bottom of the image from about how high valuation can hurt startups.)

6. A startup can be a “better mousetrap”

Clever’s signature product, the single sign-on platform, didn’t require a scientific breakthrough to implement - instead, the founders connected together existing technologies with a straightforward, easy-to-use interface.

There are many other examples of successful platforms which boil down to connecting other technologies together in a seamless, well-integrated way. Some examples of this include ChatGPT plugins, GPT wrappers (which, despite much negative press, can be profitable) and low-code platforms for computer vision (e.g. Datature). Here’s another cool example: one of my classmates at Berkeley is working on a startup, ezML, to make it fast and cheap to deploy computer vision apps.

Conclusion

The talk lasted less than two hours, but I felt like a got a lot of out of it. Moving forward, I’d like to better understand what makes technology businesses succeed and fail. Besides the actual product a company produces, what else determines its success or failure?

Thanks for reading my second blog post! If you loved (or hated) reading it, please reach out over email or leave a comment below.

2) How can we assess a language model’s performance at tasks in which scoring is arguably subjective, e.g. summarization? For example, if I ask an LLM to summarize a piece of text, how do you determine whether the summary is good or bad? How do you quantify this sort of question? One suggestion I’ve heard is to try breaking down the task into something more achievable. One resource I was looking at suggests drawing an analogy to flashcards - suppose that the summary of the text consists of a bunch of “flashcards” and evaluate every flash card individually. Go through every flash card and ask questions like, are the dates right? Does it mention the relevant key words? Are there any key words in this flash card which shouldn’t be there? (source: “How to Build Terrible AI Systems”)

3) Is it possible to generate a constructed language using AI? If the language was more “concise” than English (e.g. it takes 150 characters to express a thought that would take 200 characters in English), would there be any practical value to it over English? (Douglas Hofstader alludes to this idea in Godel, Escher, Bach when he talks about translation between languages by means of an intermediate langauge as opposed to dictionary lookup.)

4) Will it ever be possible to extend large language model context windows to infinite length? Some solutions to this problem that I’ve researched are MemGPT (which uses a memory hierarchy similar to how OSes work) and Grouped Attention.

5) How can we get language models to achieve “superhuman” performance on tasks that that even humans can’t do? For example, given a grammar for an arbitrary language, can we get models to output grammatical sentences in that language? More generally, if I give an LLM a detailed specification of some system (be it a language, a writing system like VJScript, or something else altogether) which has dozens, hundreds, or even thousands of rules, how can I get a language model to produce outputs which adhere to all these rules without giving it a lot of examples?

6) Given a silent MRI video of somebody talking, is it possible to train an ML model to detect what language they are speaking?

7) Suppose there exist two languages, language X and language Y. X and Y are sufficiently different from each other to be considered separate languages, but they still have a lot of shared vocabulary (e.g. English/French, Spanish/Italian). What is the most efficient way to generate sentences in language X that have high mutual intelligiblity for speakers of language Y?

For example, the French sentence below is mostly intelligible to a speaker of English:

“Le président Emmanuel Macron assure le peuple canadien que le gouvernement français va continuer à défendre le Canada contre la menace américain.”

Even if you didn’t catch any word, you can get the gist of it – the French president Emmanuel Macron is assuring the “peuple canadien” (Canadian people) about something involving the “gouvernment français” (French government). Imagine reading thousands of sentences like this – it would be a great way to “backdoor” into a new language using cognates you already know. Solving this problem will probably involve NLP, statistics, and some kind of cognate detection tool. I’ve made a simple demo of this concept here.

8) Is it possible to design a writing system that combines English consonant letters with Abugida-style vowel diacritics? For example, the letter “B” would be written “B” and the letter “BA” would be written “Bा. “BI”, “BO”, and “BU” would be “िB” “Bो”, and “Bु” respectively. Here’s an example:

Here’s another example, with diacritics exclusively on top of the words:

Update: I ended up writing a blog post about this, see my “writings” tab for more info.