Photographers beware; the impact of AI on the profession is likely to be much greater than previously assumed. While the ability of AI to sort and find images has long been recognized–and cheered on by businesses, photo editors, and individuals consumers alike–the ability of AI to replace not only the skills, but the photographer herself, is slowly coming down the pipeline. It’s high time to reflect.

#thinking

Image Sorting, Suggested Tagging, and Facial Recognition

“We don’t like this reflection [of ourselves], so we blame the mirror”

For a little while now, computers have been able to look at our images and “see” what we want them to “see.” For the most part, this has been a welcome development that most of us engage with every day. Some examples: reverse image search, Facebook’s suggested auto-tagging, and the software organizing our favorite stock image sites–saving editors literally thousands of hours of sorting work.

The drawbacks to this tech, however, have been widely noted, the most glaring example being Google Photos labelling an African American couple “gorillas.” Notice this is not an random, otherworldly pairing of object and title, but rather a highlight of some of the most racist undercurrents in our society. As noted on The Verge: “the prejudices and inequalities of society often become hard-coded in AI.” This includes language software which assumes doctors are male, and nurses female.

It does what??!

While this is highly disturbing, it is not an unfixable problem (Google launched the PAIR program in July with the aim of making sure that AI “benefits and empowers everyone”). Our indignation aimed at the researchers behind this tech is analogous to what Emily Parker writes in the NYT regarding our blaming social media for the recent election results: “We don’t like this reflection [of ourselves], so we blame the mirror.”

Remember, AI only “knows” what we teach it. It “learns” the contents of a photo by being shown thousand of photos along with the corresponding description of their object. If it thinks doctors are always male, or can’t recognize a race of people as properly homo sapien, we are left to wonder what it is in our society that would lead a recent entrant to it, like these machines, to such conclusions. 

And this is a conversation worth having.

Now that it’s out in the open–let’s talk about it.

Image Improvement

“some flaws are inevitable”

More recently, perhaps unsurprisingly, AI has taken the leap into into improving image quality. A prominent development in this field has been the work of researchers Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey and Luc Van Gool. In a paper entitled “DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks,” presented at the 2017 International Conference on Computer Vision, this team outlined a nifty algorithm they developed to yes, take DSRL-Quality Photos on Mobile Devices.

Image Courtesy: Computer Vision Laboratory

If it sounds too good to be true, well, it sorta is. The tech isn’t perfect, and while it certainly brightens and sharpens an image, it often (in my experience and others) leaves a funny tint. As the researchers themselves concede, “Since the proposed enhancement process is fully-automated, some flaws are inevitable. Two typical artifacts that can appear on the processed images are color deviations…and too high contrast levels.” Nonetheless, it is leading the way and, as the first iteration of its sort, it’s pretty impressive.

Image Courtesy: Computer Vision Laboratory

To develop their AI, they showed it “a large-scale dataset that consists of real photos captured from three different phones and one high-end reflex camera,” teaching it to translate the former to the latter. By doing so, while focusing on specific parameters, the AI learns what it “takes,” to turn a cellphone photo into a DSLR photo. For those of you interested in testing it out, they’ve set up a website where you can upload a photo for AI-enhancement. “

The only thing it can’t do,” Engadget’s Jon Fingas writes, “is add details that weren’t already there,” which brings us to our next topic:

 

Adding Details

“There are a lot of applications for this”

Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input,” or, in other words, using that enhance tool they always talk about on crime shows. While for the longest time this has been no more than a myth (one which photographers love to point out to fans of CSI and the like), researchers at the Max Planck Institute have made great strides turning in turning this into a reality.

Typically the approach taken towards “super-resolution” is trying to get the highly-pixelated original image to look as if it had more definition by seeking to reconstruct, based upon clues, what the original image would look like had it been shot in a higher resolution. This accords with our CSI-type notions that there is an original reality to be found. However, this team of researchers decided to flip the process on its head, and to great success.

On the left is typical image-smoothing, on the right is their AI-powered algorithm. Image Courtesy: Max Planck Institute for Intelligent Systems

Seeking to recover pixels that aren’t there, while intuitively the proper way to go about enhancing resolution, simply “correlate[s] poorly with the human perception of image quality,” leading to “over-smoothed images that lack high-frequency textures and do not look natural.” Instead, Mehdi S. M. Sajjadi and his team focused on the textures present in photographs, seeking to teach AI to recreate them.

To do so, they fed their AI thousands of samples of textures in low-quality, asking it to upsample them. Afterwards, they compared these to the original, high-quality shots of the textures and, for any mistakes, the researchers retweaked the algorithm until it achieved a quality they found to be believable. The results are fairly astounding.

From left to right: low quality image, traditional image-smoothing, AI-algorithm, actual image. Image courtesy: Max Planck Institute for Intelligent Systems

As the head researcher notes, and is quite obvious, “There are a lot of applications for this…from upsampling old movies to 4K quality” to “restoring old family photographs that are too blurry when you want to get a large print.” Also: catching that hard-to-find perpetrator.

 

Taking Photos

“we mimic the workflow of a landscape photographer”

So AI can tag your photos, sort your photos, improve their quality, and up their resolution, adding details that were never there to begin with. All of these abilities come into play in post-production, with AI acting as a tool for the human photographer. But what if AI were to replace the photographer altogether, acting as its own image maker from start finish? Google researchers Hui Fang and Meng Zhang sought to do just that.

Image courtesy: Google Creatism

The Creatism system, outlined in a July 11th paper, crops, adjusts saturation, adjusts high dynamic range (HDR) and alters lighting in order to produce a professional-quality photograph from a Google Street View frame fed to it. Some of the resulting images fooled the pros. Of 173 photos taken by the Creatism system and evaluated by professional photographers (on a scale of 1-4):

  • 41% were ranked at or above a semi-professional level (a score of 3 or greater)
  • 13% percent of the photos scored at or above 3.5
  •  Only 45% of actual professional photos received a score of  3.5 or higher.

Image courtesy: Google Creatism

Much like other AI, Creatism was informed by being shown 15,000 photos of top-ranking landscape photographers. It added lighting or saturation to each input, producing four versions of each. Then, from the four, a group of professional photographers selected one as the best, verifying the algorithm used to create it as the proper method. From there, Creatism was given full panoramas, and allowed to crop as it liked, “mimic[king] the workflow of a landscape photographer.”

Image courtesy: Google Creatism

For any photographers who want to Creatism a run for its money, a full gallery of its work can be found here.

 

What’s It All Mean?

“What happens when machines start to see?”

In his new exhibit A Study of Invisible Images,” artist Trevor Paglen urges us to answer just that. As these “images” come to be the dominant point of view on our planet, dictating the way we live (self-driving cars) to how we view ourselves (AI-powered image editing) we need to ask ourselves not only how they work but how we want them to work.

Image Courtesy Metro Pictures, Trevor Paglen