My classmate said there are two seasons in Pittsburgh: construction and winter. Pittsburgh has had a long and unusually warm construction season; up until last week, temperatures were in the 70’s. So I’ve been trying to get outside, and luckily, the university organizes outdoor excursions.

Laurel Caverns

I went caving in Laurel Caverns, located in southwest Pennsylvania. We descended lower and lower, until just our flashlights and headlamps illuminated the pathway. The tour guide asked us to turn off all our lights, and we experienced total darkness. Our guide explained that in the absence of visual stimuli, our brains made up visual information. For example, when we waved our hands in front of our faces, we saw our hands moving, but it was just our imagination. Also, there was some light in our peripheral vision. This, too, was our imagination.

There were two textures: hard rock, and soft sand with the consistency of a damp sand castle. Carbonic acid broke the rocks down into sand, creating the caves.

I had a lot of fun scrambling over rocks. There were some sections where we had to crawl under boulders, and the ground was muddy. So when we left we were covered in dirt.

Frick Park

I walked around Frick Park, a wooded area east of school. There were some elevation changes, akin to walking around downtown Seattle. I saw a pair of pet pigs. They were cute, munching the grass.

The Cathedral of Learning

The Cathedral of Learning is a 40-story tower with classrooms for the University of Pittsburgh, adjacent to the western edge of Carnegie Mellon. I took the elevator as high as I could, and there were panoramic views of the area. On the lower floors, there are Nationality Rooms. Each room is designed with the culture of some country or region.

I hiked Mount Pilatus, near Lucerne, Switzerland. At the summit, I walked around the Dragon Trail, so-named because people used to believe dragons lived at the top of the mountain. I took a few photos of the absolutely gorgeous view of houses, lakes, greenery, and other mountains.

Here are the original images:

You can see that the two images overlap. I would like to merge the images. Here’s how that can be done:

Find keypoints (points of interest) in each image.

Encode the keypoints in each image. Compare the encodings to determine if a keypoint in the left image matches a keypoint in the right image.

From the matching keypoints, calculate how the 2nd image needs to be altered in order to fuse the images together.

Combine the images!

Let’s do this!

1. Find keypoints

For each image, we create a Gaussian pyramid. At each level of the pyramid, we apply a Gaussian filter.

Next, we make a Difference of Gaussian pyramid. At each level of the Gaussian pyramid, we subtract the previous level.

Here’s what those pyramids look like. Click on the thumbnails to get a better view.

The Difference of Gaussian pyramids allow us to find corners, edges, and blobs in the image.

Next, we find possible interest points. Interest points are local extrema in space; they should have max or min pixel value compared to neighboring pixels. Interest points should also be extrema in scale; in the Difference of Gaussian pyramid, the point should be the max or min of the neighboring levels above and below.

We take all the points that are extrema and that have Difference of Gaussian magnitude above a certain threshold.

Finally, we filter out points that are edges. Edges make lousy keypoints, because it’s difficult to tell where a point is on an edge, and edges are not as distinctive as corners and blobs. To get rid of edges, we calculate the principal curvature ratio for each of the candidate points. Edges have principal curvature that is much larger in one direction than the other direction. We can filter out points that do not meet a certain threshold for principal curvature.

Now we’ve got our keypoints for each image:

2. Encode the keypoints

We can use a BRIEF (Binary Robust Independent Elementary Features) descriptor to encode keypoints. At each keypoint, we take a 9×9 pixel patch centered at that keypoint. Then we generate a vector of bits by randomly sampling and comparing pixels in the patch.

We compare keypoints in the left image and the right image by taking the Hamming distance of the BRIEF descriptors. The Hamming distance is the number of bits that are different.

Here’s what those matches look like. If there is a keypoint in the left image that matches one to the right image, there is a red line.

There are a ton of matches! Some of them are outliers and obviously erroneous matches.

3. Calculate planar homography

We need to figure out how the left image and right image are related to each other. So we need to calculate the 3×3 homography matrix that tells us how an image taken from one camera view relates to an image taken from a different camera view. We can do this by finding the matrix that minimizes the least squares error.

But we ran into a problem: a lot of the matches are outliers, so they will skew the homography matrix with disastrous results. We solve this problem with RANSAC, random sample consensus. First, we pick 4 random matching pairs (we need at least 4 pairs of points to solve for the homography matrix). We calculate a homography matrix from these random pairs. Next, we apply the matrix to all the matching points in one image, then compare them with the matching points in the other. If the difference is within a certain tolerance level, the point pair is an inlier. We iterate this process, keeping the homography matrix that resulted in the most inliers.

4. Combine the images

Now we can combine the two images. Using the homography matrix, we warp one image into the reference frame of the other. We make adjustments so that both images fit into the panorama without clipping. Where the images overlap, we blend them together.

And here’s a panorama with three photos. The photo at the top of this blog post is used on the far left. Click to see it in its full glory:

In this panorama, there is clipping so that there is no blank space in the image.

I visited Skogskyrkogården, which, despite it being a relatively new 20th-century construction, is a UNESCO World Heritage site.

The buildings had different architectural styles— the Chapel of Resurrection had clean lines with a Greek column entrance, the Woodland Chapel blended in with the forest with its earthy colors and pyramid roof.

From the entrance, I walked up a hill into a grove of trees, a place of meditation. Then I walked for a long time on a gravel pathway (Seven Springs Way) that cut through a forest to the Resurrection Chapel. Graves were arranged in neat rows in the forest, and a soft light filtered through the trees. The intent is to reflect on nature as you contemplate life and death.

Near the Woodland Chapel was the grave of Greta Garbo. I had previously never heard of the famous actress, but her named cropped up everywhere on my trip. It turns out she was one of the foremost actresses of her generation. First, I saw her on the 100 kronor. Next, in one of the hotels we stayed at, we were assigned to the “Garbo room.” Pictures of her lined every wall. The hotel bed was a canopy bed that she had once used. And the largest portrait of all, of Greta’s confident, piercing gaze, was hung over the bed. When I first arrived at the hotel room, I found this shrine to her unsettling, and was unsurprised that no one had specifically requested this room. But then I got used to Greta staring through me and came to appreciate the room’s quirk. At Skogskyrkogården, her grave was a prominent circular plot with a wide berth of grass, her signature in gold carved into a red marble stone. The plot, isolated from the others, felt rather lonely.

Across the street of the entrance to Skogskyrkogården was another cemetery. Each plot was clinically labeled with a number and which maintenance group it was in. One grave was covered in flowers and stuffed toys. There was an etching of a young boy’s face, and the years showed he was only five.

There are a fair number of geocaches around Carnegie Mellon.

Geocaches. Let me show you.

I found geocaches in Schenley Park. I was walking in a forest trail, when I noticed there was a geocache on the adjacent golf course. So I stepped out of the trees and someone on the green let me putt. I thought it was funny, that some random woman comes out of the trees and your first instinct is to offer her a chance to hit a golf ball.

This geocache was tricky to find. I had to scale a wall then feel around the hollow parts of a sign.

This nondescript geocache in Woodland Park was a clever hide, completely hidden underground. Pulling on the circular stopper revealed the bison.

There was a geocache hidden in a tree by the dog park too.

This geocache off of Princeton’s Nassau Street looked like a wad of gum.

A geocache was embedded into a park’s stone sign.

I found some geocaches in Stockholm. As a city with lots of foot traffic, the good old metal plate geocache was popular.

There were a few caches off of the islands of Skeppsholmen and Kastellholmen. This geocache was found in a rock nestled into the corner of a footbridge.

A geocache was in a rabbit hole near the museum of modern art. The string to pull up the cache was hidden under a pile of dead grass in a tree trunk.

A geocache was hidden in one of these mailboxes. The cache owner posted a hint as to which one (it was the white mailbox on the left).

There was a magnetic geocache on the Skeppsholmsbron bridge.

A geocache was brazenly hidden right in front of the Royal Palace (the “GC” magnet near the bottom of the parking ticket dispenser).

And finally, there was a geocache in Old Town. Each tourist was doing his own thing, so no one noticed notice that I was reaching into a rain gutter.