Style transfer using convolutional neural nets

Using deep neural nets, it is possible to change a photo or video to mimic the style of a piece of art. We have the original image, the style source image, and the pastiche. Here’s an example of J at Jack Block Park using the style of Rain Princess by Leonid Afremov. You can see the pastiche appears to be made of colorful oil strokes.

Jack Block Park beach
Jack Block Park beach
Rain Princess by Leonid Afremov
Rain Princess by Leonid Afremov
Image with style from Rain Princess
Image with style from Rain Princess

We run the original image through a neural net. I initialized the neural net with weights from a VGG19 model trained on ImageNet. By doing so, we already have a good basis for features important to object categorization and invariance to translation, rotation, etc.

The neural net’s loss function seeks to create a pastiche that minimizes content loss (the content of the original image) and the style loss (the style of the style source image). By adjusting weights in our loss function, we can change the importance of minimizing either content or style loss.

Here’s a picture of Cherry Creek Falls in the style of Vincent van Gogh’s The Starry Night. In the first pastiche, minimizing content loss is more important. In the second pastiche, minimizing style loss is more important.

Cherry Creek Falls
Cherry Creek Falls
Cherry Creek Falls weighted to minimize content loss
Cherry Creek Falls weighted to minimize content loss
Cherry Creek Falls weighted to minimize style loss
Cherry Creek Falls weighted to minimize style loss

And here’s a video of me floating in a pool. Even on a 10-second 240×134 video, the style transfer process was slow, because I trained a neural net on each frame. At 60fps, there are hundreds of frames, so hundreds of corresponding neural nets that were trained.

Cherry Creek Falls

J and I started the new year with an early hike to Cherry Creek Falls. I enjoyed the drive through Duvall and its bucolic meadows and farms. There were disused barns made up of old and rotten wood, with gaping holes where planks were missing. Other barns were new and massive, walls and roofing in vibrant colors. Brown horses nibbled grass, staying warm in their quilted blue turnout blankets. In the Duvall/Carnation area, there is a lot of wide open acreage. So the area hosts obstacle races and mud runs, which are rather trendy right now.

The hike was relatively easy, only 6 miles round trip with 500 ft. elevation gain. The trail was wide and well-maintained. There were multiple forks, but it was always clear which trail was the main path.

The forest was so green. Trees were covered in emerald moss and the ground was carpeted by ferns. Sun filtered through trees and light fog to provide a warm, diffuse glow.

Near the beginning, we passed by the silver chassis of a car, and I was surprised that a car could crash so deep in the woods. But then I thought about it some more, and the roads in Duvall are winding and shaded, so black ice could form easily. And King County doesn’t salt its roads and is slow to clear snow and does not have the equipment nor the budget for winter conditions compared to, say, Chicago. And when there’s only an inch of snow, everyone works from home because snow drifts form at the bases of hills, and vehicles on inclines get trapped or uncontrollably slide down snow and ice. And it seems whenever it rains or snows, there are many car accidents and commute times become ridiculous, and so perhaps it is not so surprising to see a car wreck in the middle of a forest. Towards the end of the hike, we passed by yet another car wreck, this one more intact. The car is facing the hiking trail, and the trail actually curves around the car. The car still has its yellow shell and brown leather seats. Someone drew graffiti on a seat. The shell was rusting with brown circles on its doors, giving the appearance of bullet holes.

We had to cross multiple streams and large puddles, and only one crossing had a bridge. For the others, we used stepping stones and fallen branches to make our way across and stay dry. There was one creek that was raging from the winter snow melt, and we knew there would be no way to cross without getting our legs soaked, but luckily, there was a downed log by the trail.

The falls were beautiful, three wide cascades feeding into calm waters. It would be a great place to swim in the summer.

The Boys in the Boat book review

I read The Boys in the Boat: Nine Americans and Their Epic Quest for Gold at the 1936 Berlin Olympics, by Daniel James Brown. The US crew team was made up of undergraduates from the University of Washington. Most of the oarsmen were from rough-cut families, sons of lumberjacks, farmers, and other laborers. The main focus was on rower Joe Rantz. As early as age 10, he was left by his parents to fend for himself. Joe had to split wood, illegally poach fish and steal alcohol, do janitorial and construction work, and do any other odd jobs he could find to support himself and pay for his tuition. So Joe’s story starts out with him being abandoned and alone, determining that he could trust no one but himself. But through crew, he learns to trust his teammates, finds a sense of belonging, and reconciles with his family and starts a family of his own. I don’t know anything about crew, but this book gave detailed, reverent, even spiritual descriptions of the sport. The author waxes on about the difficulty and precision required to be in sync, the need to be strong individually and yet subsume oneself to the team, the transcendental feeling of togetherness when the oarsmen move as one unit. The book culminates in the crew’s suspenseful come-from-behind victory in their gold medal race, overcoming an unfair start and bad lane position to become world champions. Interspersed in the story of the crew, there are tangents on the significance of the Olympics to Nazi Germany, the preparations and propaganda for the purpose of deceiving the world into believing that Hitler’s Reich was a powerful but benevolent force. The US crew’s victory is all the more meaningful knowing the global, historical context.

Oftentimes, the US crew team’s strategy was to start at a slow pace, then finish off the race with a sprint. Likewise, this book started off slow. So slow, in fact, that I read one chapter last year, then shelved the book out of boredom. But when I was back in Seattle, there the book was on the shelf, and I thought I’d give it another go. The book is descriptive, verbose to a fault. But as I read more and more chapters, it had me hooked. I developed an emotional connection to Joe and wanted to know what happened next, even though I knew from the book’s cover that the crew team would end up with Olympic gold. Their journey, their camaraderie— it was inspiring, something I wish everyone could have the opportunity to experience. And I was so impressed by the resolve the oarsmen showed in their lives, to succeed despite their modest upbringings.

I learned a bit of trivia too. For example, this week I drove on Royal Brougham Way, by Safeco Field. From the book, I learned that Royal Brougham was an enthusiastic Seattle sportswriter, and among the stories he covered was the University of Washington crew team’s ascent. I also happened to drive by Carnegie Lake this week, and I learned from the book that it was built so that Princeton’s crew could practice there instead of having to row in the Delaware and Raritan canal with all the commercial boats.

The Crows

I was walking home to my apartment. Across the street from where I live, there is a Catholic church with Gothic architecture. On this particular night, there were at least a hundred crows buzzing around the church in a great swirling mass of black. Others perched on trees. The branches were at maximum capacity, drooping under the weight of the crows. The scene was straight out of a horror story, except the stories fail to mention that the birds shit on the sidewalk until the ground is covered like a Jackson Pollock painting. They were instinctively restless from the imminent arrival of winter.

I walked upstairs deeply affected by what I had just seen. And so I made a sketch of a crow.

Quid pro crow
Quid pro crow

I told J about the pun, “quid pro crow.”

“How about quid pro fro?” J responded.

“Nah, that’s a bit of a stretch.”