How to make a panorama

I hiked Mount Pilatus, near Lucerne, Switzerland. At the summit, I walked around the Dragon Trail, so-named because people used to believe dragons lived at the top of the mountain. I took a few photos of the absolutely gorgeous view of houses, lakes, greenery, and other mountains.

Here are the original images:

Mount Pilatus left image
Mount Pilatus left image
Mount Pilatus right image
Mount Pilatus right image

You can see that the two images overlap. I would like to merge the images. Here’s how that can be done:

  1. Find keypoints (points of interest) in each image.
  2. Encode the keypoints in each image. Compare the encodings to determine if a keypoint in the left image matches a keypoint in the right image.
  3. From the matching keypoints, calculate how the 2nd image needs to be altered in order to fuse the images together.
  4. Combine the images!

Let’s do this!

1. Find keypoints

For each image, we create a Gaussian pyramid. At each level of the pyramid, we apply a Gaussian filter.

Next, we make a Difference of Gaussian pyramid. At each level of the Gaussian pyramid, we subtract the previous level.

Here’s what those pyramids look like. Click on the thumbnails to get a better view.

Left image - Gaussian pyramid
Left image – Gaussian pyramid
Left image - Difference of Gaussian pyramid
Left image – Difference of Gaussian pyramid
Right image - Gaussian pyramid
Right image – Gaussian pyramid
Right image - Difference of Gaussian pyramid
Right image – Difference of Gaussian pyramid

The Difference of Gaussian pyramids allow us to find corners, edges, and blobs in the image.

Next, we find possible interest points. Interest points are local extrema in space; they should have max or min pixel value compared to neighboring pixels. Interest points should also be extrema in scale; in the Difference of Gaussian pyramid, the point should be the max or min of the neighboring levels above and below.

We take all the points that are extrema and that have Difference of Gaussian magnitude above a certain threshold.

Finally, we filter out points that are edges. Edges make lousy keypoints,  because it’s difficult to tell where a point is on an edge, and edges are not as distinctive as corners and blobs. To get rid of edges, we calculate the principal curvature ratio for each of the candidate points. Edges have principal curvature that is much larger in one direction than the other direction. We can filter out points that do not meet a certain threshold for principal curvature.

Now we’ve got our keypoints for each image:

Left image keypoints
Left image keypoints
Right image keypoints
Right image keypoints

2. Encode the keypoints

We can use a BRIEF (Binary Robust Independent Elementary Features) descriptor to encode keypoints. At each keypoint, we take a 9×9 pixel patch centered at that keypoint. Then we generate a vector of bits by randomly sampling and comparing pixels in the patch.

We compare keypoints in the left image and the right image by taking the Hamming distance of the BRIEF descriptors. The Hamming distance is the number of bits that are different.

Here’s what those matches look like. If there is a keypoint in the left image that matches one to the right image, there is a red line.

Keypoint matches between left image and right image
Keypoint matches between left image and right image

There are a ton of matches! Some of them are outliers and obviously erroneous matches.

3. Calculate planar homography

We need to figure out how the left image and right image are related to each other. So we need to calculate the 3×3 homography matrix that tells us how an image taken from one camera view relates to an image taken from a different camera view. We can do this by finding the matrix that minimizes the least squares error.

But we ran into a problem: a lot of the matches are outliers, so they will skew the homography matrix with disastrous results. We solve this problem with RANSAC, random sample consensus. First, we pick 4 random matching pairs (we need at least 4 pairs of points to solve for the homography matrix). We calculate a homography matrix from these random pairs. Next, we apply the matrix to all the matching points in one image, then compare them with the matching points in the other. If the difference is within a certain tolerance level, the point pair is an inlier. We iterate this process, keeping the homography matrix that resulted in the most inliers.

4. Combine the images

Now we can combine the two images. Using the homography matrix, we warp one image into the reference frame of the other. We make adjustments so that both images fit into the panorama without clipping. Where the images overlap, we blend them together.

Panorama - left and right image combined
Panorama – left and right image combined

And here’s a panorama with three photos. The photo at the top of this blog post is used on the far left. Click to see it in its full glory:

Panorama with three images
Panorama with three images

In this panorama, there is clipping so that there is no blank space in the image.

Skogskyrkogården (The Woodland Cemetery)

I visited Skogskyrkogården, which, despite it being a relatively new 20th-century construction, is a UNESCO World Heritage site.

The Woodland Crematorium
The Woodland Crematorium

The buildings had different architectural styles— the Chapel of Resurrection had clean lines with a Greek column entrance, the Woodland Chapel blended in with the forest with its earthy colors and pyramid roof.

The granite cross
The granite cross

From the entrance, I walked up a hill into a grove of trees, a place of meditation. Then I walked for a long time on a gravel pathway (Seven Springs Way) that cut through a forest to the Resurrection Chapel. Graves were arranged in neat rows in the forest, and a soft light filtered through the trees. The intent is to reflect on nature as you contemplate life and death.

Near the Woodland Chapel was the grave of Greta Garbo. I had previously never heard of the famous actress, but her named cropped up everywhere on my trip. It turns out she was one of the foremost actresses of her generation. First, I saw her on the 100 kronor. Next, in one of the hotels we stayed at, we were assigned to the “Garbo room.” Pictures of her lined every wall. The hotel bed was a canopy bed that she had once used. And the largest portrait of all, of Greta’s confident, piercing gaze, was hung over the bed. When I first arrived at the hotel room, I found this shrine to her unsettling, and was unsurprised that no one had specifically requested this room. But then I got used to Greta staring through me and came to appreciate the room’s quirk. At Skogskyrkogården, her grave was a prominent circular plot with a wide berth of grass, her signature in gold carved into a red marble stone. The plot, isolated from the others, felt rather lonely.

Across the street of the entrance to Skogskyrkogården was another cemetery. Each plot was clinically labeled with a number and which maintenance group it was in. One grave was covered in flowers and stuffed toys. There was an etching of a young boy’s face, and the years showed he was only five.

Geocaching in Pittsburgh

There are a fair number of geocaches around Carnegie Mellon.

Geocaches. Let me show you.

Geocache in rock wall
Geocache in rock wall
Geocache on campus' south border
Geocache on campus’ south border
Geocache outside of Gates
Geocache outside of Gates

I found geocaches in Schenley Park. I was walking in a forest trail, when I noticed there was a geocache on the adjacent golf course. So I stepped out of the trees and someone on the green let me putt. I thought it was funny, that some random woman comes out of the trees and your first instinct is to offer her a chance to hit a golf ball.

Geocache on golf course
Geocache on golf course
Geocache disguised as rock in tree trunk
Geocache disguised as rock in tree trunk
Another geocache in a tree trunk
Another geocache in a tree trunk

This geocache was tricky to find. I had to scale a wall then feel around the hollow parts of a sign.

Geocache in Schenley Park sign
Geocache in Schenley Park sign

There are geocaches hidden at UPitt too.

Geocache under a picnic table
Geocache under a picnic table
Geocache in a parking garage
Geocache in a parking garage

Geocaching part 11

Will it ever end? No.

This nondescript geocache in Woodland Park was a clever hide, completely hidden underground. Pulling on the circular stopper revealed the bison.

Geocache in manhole
Geocache in manhole

There was a geocache hidden in a tree by the dog park too.

Geocache in Woodland Park tree
Geocache in Woodland Park tree

This geocache off of Princeton’s Nassau Street looked like a wad of gum.

Fake gum geocache
Fake gum geocache

A geocache was embedded into a park’s stone sign.

Geocache in park sign
Geocache in park sign

I found some geocaches in Stockholm. As a city with lots of foot traffic, the good old metal plate geocache was popular.

Geocache near Kungsträdgården
Geocache near Kungsträdgården
Geocache near waterfront
Geocache near waterfront
Geocache in busy shopping area
Geocache in busy shopping area

There were a few caches off of the islands of Skeppsholmen and Kastellholmen. This geocache was found in a rock nestled into the corner of a footbridge.

Geocache disguised as rock
Geocache disguised as rock
Geocache inside rock
Geocache inside rock

A geocache was in a rabbit hole near the museum of modern art. The string to pull up the cache was hidden under a pile of dead grass in a tree trunk.

Geocache near Moderna Museet
Geocache near Moderna Museet

A geocache was hidden in one of these mailboxes. The cache owner posted a hint as to which one (it was the white mailbox on the left).

Geocache mailbox
Geocache mailbox

There was a magnetic geocache on the Skeppsholmsbron bridge.

Geocache on bridge
Geocache on bridge

A geocache was brazenly hidden right in front of the Royal Palace (the “GC” magnet near the bottom of the parking ticket dispenser).

Geocache outside Stockholm's Royal Palace
Geocache outside Stockholm’s Royal Palace

And finally, there was a geocache in Old Town. Each tourist was doing his own thing, so no one noticed notice that I was reaching into a rain gutter.

Geocache in Gamla Stan
Geocache in Gamla Stan

Pittsburgh

I’ve been in Pittsburgh for about a month now, so I’ve had time to explore a small corner of this city of rivers and penguins.

Breakneck Rocks

Breakneck Rocks is about an hour’s drive from the city in the Southwestern Highlands of Pennsylvania. I went climbing there with CMU’s club. Top roping outdoors presented challenges that I was not accustomed to from my time in sterile indoor bouldering environments. Whereas when climbing indoors, the route’s holds are clear and sparse, outdoors there are plenty of holds but they do not offer a solid grip. There was a lot of groping around to find something suitable to latch on to. Also, the bugs were annoying. A bee buzzed around the hold I wanted to use for a good minute, so I had to pause climbing. A daddy long legs crawled precariously close to my hand. Sometimes I would stick my hand in a hold and feel spider webs. Mosquitoes constantly accosted me, and I left with 10 bug bites. But otherwise, the climbs were fun. I’m amazed that humans can scale near-vertical rock faces.

Schenley Park

Schenley Park is on the southern edge of CMU. I walked along several trails, such as the upper and lower Panther trails and bridle trail. There were a lot of chipmunks and deer.

I saw the native paw paw trees. They are something of an anomaly, with a fruit tasting like something out of the tropics, yet they grow in Pennsylvania and Ohio. They are not farmed though, because the fruit is only ripe for a short period of time before quickly going bad.

Phipps Conservatory

Phipps Conservatory
Phipps Conservatory

I walked around the Phipps Conservatory a couple times because it is just a few blocks from where I live. The greenhouses are sprawling, with rooms dedicated to tropical fruits, succulents, ferns, butterflies. Chipmunks scurry about indoors, and there are frogs populating the sustainable gardens outside. There is impressive flower glasswork throughout. I enjoyed smelling the aromatic flowers.

Carnegie Museum of Art and Natural History

The Carnegie Museum of Art has an impressive collection of art from a variety of artists, mediums, and eras. Like the Seattle Art Museum, the Carnegie Museum of Art houses pieces made by big-name artists. The pieces are representative of their styles but not their famous masterworks.

Carnegie Museum of Natural History plaster fabrications
Carnegie Museum of Natural History plaster fabrications

The Carnegie Museum of Natural History has a large collection of taxidermy, geodes, and my childhood favorite, dinosaur bones. The museum feels old-school, with lots of static displays but few interactive exhibits.

One room had facades of famous buildings. Initially, it was mind-boggling to think that they could transport those huge stone faces intact, but then I read that they were plaster models of the originals, meant to give the locals a taste of European culture back when travelling internationally was prohibitively expensive.