Understanding Decisions and Tradeoffs of Neural Style Transfer

Abstract

A Neural Algorithm of Artistic Style introduced an artificial system to create artistic images. It does so by generating a set of feature maps from a content image and a style image. These feature maps are used to generate a loss, which is applied iteratively to a noisy image via gradient descent. This process results in the generated image having features which represent that of the target content and the target style images. The goal of this project is to explore the original research decisions, and explicitly document the tradeoffs being made. The original research used the Visual Geometry Group’s 19 weight-layer model. Many properties of this method were not fully documented - the optimization method is not specified (only broadly referred to as gradient descent), the architecture is altered to “improve gradient flow” without documenting how it was improved, no other models are evaluated, and many properties of the gradient descent process were not explored. This project aims to explore additional models, document various methods of gradient descent, explain characteristics of the loss, and propose solutions to improve the gradient flow.