Hi, I'm Adam Steinberger, a Computer Science and Music student here at Skidmore. This summer, Prof.

Mike Eckmann and I have been researching Digital Image Source Identification.

Slide 1 of 14

Cameras often insert data into digital photos that state make, model, time and place, among other things. This extra data is easily spoofed using modern software, so we can't trust it. When given an image, we want to determine which camera took it.

Slide 2 of 14

Since we can't trust the text data cameras already provide, we need something else to trust. We want to create a fingerprint per camera based on the pixels of digital photos. Certain data in the pixels will hopefully allow us to uniquely determine the source cameras of a given photo. Some examples of image data include sensor dust markings, noise and white balancing. These have been used to create almost unique fingerprints and are a lot harder to spoof than text data.

Slide 3 of 14

Given a digital image, we want to determine what camera took it. We created a classifier built up from image attributes. The way we created our classifier was by training it using the attributes from about half of our images. By training the classifier we mean that it creates a fingerprint for each camera. The attributes from the other half of our images are used to test the classifier to determine its accuracy. We took the pictures ourselves, along with our colleagues, from 25 different cameras. So obviously we know which camera each image belongs to. The attributes the classifier uses relates to the pixel colors in images. These attributes also relate to the demosaicing process in cameras, which I will describe shortly.

Slide 4 of 14

Digital images are composed of pixels, each containing a red, green and blue component. When a digital photo is taken, light from the outside world hits sensors inside the camera that make up a Color Filter Array. Each of these sensors could capture red, green or blue. To keep costs down, cameras capture only one color per pixel. The other two colors are computed using neighborhoods of pixels. For example, a pixel corresponding to a blue sensor. (show blue sensor in middle) needs some way to obtain red and green values. Cameras will estimate a red and green component by some computation of the neighboring red and green sensors. (show different neighborhoods) Differing weights of sensor outputs and different sized neighborhoods are used in different camera makes and models. These differing processes can help us create fingerprints for cameras.

Slide 5 of 14

The process I just described, where cameras compute the two missing colors for each pixel from some neighborhood around it, is called demosaicing. Typically cameras use unique proprietary demosaicing processes. Also, a pixel in a smooth area is usually processed differently than pixels on edges. (show smooth and edge pixels) Different makes and models of cameras use different proprietary demosaicing processes.

Slide 6 of 14

Just a head's up: We're approaching some details worth mentioning about our attributes. I've developed original software to generate attributes from photos related to the demosaicing process inside cameras. I've written code that can tell the difference between pixels in smooth areas versus those on an edge. The next couple slides are details to give you a sense of how some of these attributes were computed. The implementation of this next process is a small sample of the stuff I worked on this summer.

Slide 7 of 14

Here we have one color for a single pixel. This is the red channel of a pixel, which by the way also has a green and blue channel. Since I developed code that worked on red separately from green and blue, I'm showing you an example here where we are just focusing on the red channel. But this works for green and blue channels, too. (click) The neighborhood of a smooth pixel might look like this, where the values here are within a small range. This is a defining characteristic of smooth areas. (click) To get an error, we start with the origin pixel. (click) Then we sort the pixel's neighbors. (click) Removing the outer pixels, we find the median neighbor. (click) Subtracting the origin pixel with its median neighbor gives us an error value. (click)

Slide 8 of 14

Now let's consider an edge pixel. (click) This is what the neighborhood of an edge might look like. Notice the pixels on the left have significantly smaller values than those on the right. (click) Start with the origin pixel. (click) We remove the edge that doesn't belong. (click) Then sort the remaining neighbors. (click) Next, we get the median of these neighbors. (click) Finally, subtract the origin pixel with its median neighbor to get an error value. (click)

Slide 9 of 14

We compute errors for 3 neighborhood sizes, and all 3 color channels all separately. That's 9 different groups of data that are all handled separately. For each group of data, we compute the mean, standard deviation, skewness, kurtosis, energy and entropy of the pixel errors. (Show what each statistics moment is)

Slide 10 of 14

So from what I just described, the attributes come from 4 statistics moments, energy and entropy for 3 colors and 3 neighborhood sizes for both edge and smooth pixel types. That plus 18 unrelated attributes, that due to time constraints and since they're unrelated to the demosaicing process I won't include in this talk, adds up to 126 total attributes. The classifier uses these attributes to create a fingerprint for each camera. The attributes will also fit into our collaborator's larger system, which includes data about noise in images.

Slide 11 of 14

For all 25 cameras, our original classifier used smooth errors attributes to trace images to their cameras with 34.96% accuracy. When we added attributes for edge errors, our accuracy rate rose to 36.72%. So, processing edge pixels differently and adding the 18 other attributes produced only marginally better results. We created an additional classifier for a subset of 3 iPhones to see if the attributes can distinguish between different instances of the same make and model of cameras. This classifier's accuracy was 79.1%, so it seems that these can differentiate between the same camera models.

Slide 12 of 14

The attributes we designed for the classifier clearly are capturing some good information, as our classifier beats random choice easily. But they don't do well enough on their own. We plan to incorporate our attributes along with our colleagues attribute sets to improve overall accuracy rates. We also plan to extract the top N camera choices for each test image in our classifier. Hopefully, we'll be able to predict with 99% accuracy the correct top N camera choices for an image. And we plan to implement a few common demosaicing processes that cameras use and compute attributes from these to try to increase our classifier's accuracy.

Slide 13 of 14

Please feel free to contact me by visiting my website at amsteinberger.com. Now, I'd like to open the floor for questions.

Slide 14 of 14