Camera model identification based on forensic traces extracted from homogeneous patches

3.1. Overview

A high-level overview of the proposed methodology is presented in Fig. 4. The pipeline highlights three major steps. Firstly, we determine the homogeneous regions in an image. This is done by dividing the input image into overlapping blocks of size 128 × 128 pixels. These blocks are then subjected to the proposed homogeneity criteria, in order to filter out non-homogeneous patches. In the second step, a pre-trained ConvNet is used to determine the camera brand for each of the homogeneous patches. These predictions are then combined using a majority vote to determine the camera brand. Finally, based on the determined brand the corresponding model-level classifier is used to determine the specific camera model for each of the homogeneous patches. The patch-level predictions are then combined by performing a majority vote which determines the source camera model for the given input image. We begin by describing the methodology for patch selection.

1. Download : Download high-res image (235KB)
2. Download : Download full-size image

Fig. 4. The flow chart depicts a high-level pipeline of the proposed methodology, highlighting the three major steps. Firstly, the input image is divided into overlapping blocks of size 128 × 128 pixels, which are further filtered based on our proposed homogeneity criteria, resulting in homogeneous patches. Secondly, a ConvNet is used to determine the camera brand for each patch, which is followed by a majority vote on the predictions. Finally, based on the brand-level prediction the corresponding model-level classifier is used to determine the camera model for each homogeneous patch. The model-level predictions are then combined by performing a majority vote to determine the source camera model for the given input image.

3.2. Homogeneous patch selection and extraction

As described earlier, every image consists of both scene details and camera processing noise. Image regions can be classified into regions with high-level scene details (that is, regions containing edges, high texture details, etc.) and low-level scene details (that is, regions that are homogeneous, where a group of neighbouring pixels have almost similar intensity values). The latter type of region contains camera processing noise that is least distorted by high-level scene details. We now describe the methodology to extract homogeneous patches from such image regions.

In order to extract patches from an image we tile the input image with blocks of size 128 × 128 pixels. Using a larger block size, such as 256 × 256 pixels, would result in fewer blocks, and the chances of a block being homogeneous are reduced. Using a smaller block size, such as 32 × 32 pixels, or 64 × 64 pixels, would result in more image patches that are homogeneous, however, extracting camera-specific features from smaller image patches is more difficult. To account for this trade-off, we sample the input image using a tile of size 128 × 128 pixels, and with a stride of 0.25 times the block size. This effectively increases the number of extracted image regions substantially, while retaining a sufficiently large image block.

Each block is then examined for homogeneity by determining the standard deviation of its pixel values. Since there are three colour channels, we determine three standard deviations, one for each channel. Fig. 5 depicts a sample input image in part (a) and the corresponding standard deviations of each image block are shown in part (b). Note that for the sake of clarity, we show the standard deviations for the red colour channel and use non-overlapping patches with a stride of 1 block. The standard deviation values of the homogeneous regions are smaller than the others. We subjectively determined a high threshold with a value of 0.02 to exclude non-homogeneous patches with significant scene level details. This threshold was determined after manually examining hundreds of extracted patches. We also set a low threshold of 0.005 to eliminate saturated patches. A saturated patch is one that has true loss of data due to having many pixel values clipped to the maximum intensity (i.e. 255), thereby overriding the camera noise. This occurs when there is very high incoming light intensity to the sensor or when certain operations (e.g. denoising) are used by editing software. In the latter case, such operations may result in a few regions where the difference between the neighbouring pixels is close to zero. Finally, for a patch to be considered homogeneous, all three standard deviations must independently adhere to the threshold limits defined above. Sample image patches for saturated, homogeneous, and non-homogeneous are shown in Fig. 5(c–e).

1. Download : Download high-res image (492KB)
2. Download : Download full-size image

Fig. 5. An illustration of patch selection using standard deviation. (a) A sample image of size 2176 × 3968 pixels. (b) The heat map of standard deviations for non-overlapping patches of size 128 × 128 pixels from the red colour channel. Examples of (c) saturated patches ($\text{std}<0.005$), (d) homogeneous patches ($0.005\le \text{std}\le 0.02$), and (e) non-homogeneous patches ($\text{std}>0.02$). Note that, for the sake of visual clarity, in (b) we only show non-overlapping tiles of size 128 × 128 pixels. In our experiments, however, we use a stride of 0.25 times the block size to sample image patches. Furthermore, this is done for all three colour channels and the std criterion in (c), (d), and (e) must hold for all three colour channels separately.

In our experiments, described in Section 4, we determine the number of homogeneous patches to be extracted from each image as follows. Suppose a given number of $p$ patches need to be extracted. If the number of homogeneous patches for an image is more than $p$, then we uniformly sample $p$ patches so that they are evenly distributed among the homogeneous regions across the whole image. On the other hand, when there are fewer than $p$ patches per image, we choose all the homogeneous patches and choose the remaining patches from the saturated and non-homogeneous patches with the lowest standard deviations. Thus, we ensure that homogeneous regions are given priority during the patch extraction. Finally, we subtract the per-colour-channel mean from the respective colour channel of the input image. This is done to minimize the colour information being inadvertently learnt for classification. Fig. 6 illustrates this pre-processing step and demonstrates that the effect of brightness in the input patches is reduced by this operation. Thereby, the classifier focuses on the noise, which is our matter of interest. Fig. 7 depicts the ratio of the average number of homogeneous to non-homogeneous to saturated patches for each camera model in the Dresden data set. Further details concerning the camera models are presented in Table 1.

The patch selection approach that we propose enable us to process images of different dimensions. In contrast, methods that use a full image (Bennabhaktula et al., 2020) may have to do resizing first, which can lead to unintended modifications to the forensic traces. Moreover, our approach of choosing patches gives the ability to choose regions that are less affected by the scene content. Next, we describe the details of the ConvNet architecture used for patch classification.

1. Download : Download high-res image (201KB)
2. Download : Download full-size image

Fig. 6. An illustration of the pre-processing step. (a) Examples of homogeneous patches and (b) the corresponding pre-processed patches after per-channel-mean subtraction. The resulting images are in the range $[-1,+1]$. For visualization purposes the resulting images are shifted to the range $[0,1]$ by adding 1 and dividing by 2.

1. Download : Download high-res image (288KB)
2. Download : Download full-size image

Fig. 7. A depiction of the ratio between the average number of homogeneous, non-homogeneous, and saturated patches for the 18 camera models in the Dresden data set. This plot is generated by considering the $400$ patches (of size 128 × 128 pixels) with the lowest standard deviations from each image.

Table 1. List of camera models considered for our experiments from the Dresden data set.

Sr. no.	Camera model name	# Devices	Total # images
1	Canon_Ixus70	3	522
2	Casio_EX-Z150	5	850
3	FujiFilm_FinePixJ50	3	630
4	Kodak_M1063	5	2314
5	Nikon_CoolPixS710	5	846
6	Nikon_D200	2	673
7	Nikon_D70	4	676
8	Olympus_mju_1050SW	5	965
9	Panasonic_DMC-FZ50	3	931
10	Pentax_OptioA40	4	638
11	Praktica_DCZ5.9	5	942
12	Ricoh_GX100	5	854
13	Rollei_RCP-7325XS	3	544
14	Samsung_L74wide	3	641
15	Samsung_NV15	3	599
16	Sony_DSC-H50	2	541
17	Sony_DSC-T77	4	906
18	Sony_DSC-W170	2	405

3.3. Patch classification

For each patch to be classified into its source camera brand and source camera model, we use a ConvNet for feature extraction that gives input to a fully connected neural network for classification. The ConvNet architecture that we use is inspired by the MISLNet, which was proposed by Bayar and Stamm (2018a). The overall design of the proposed architecture is shown in Fig. 8. The ConvNet architecture is divided into seven blocks. The dimensions of the input layer, 128 × 128 × 3, correspond to the patch size used in our experiments. This is followed by four convolutional blocks and three fully connected blocks. Bayar and Stamm (2018a) used a constrained convolutional layer in the design of the MISLNet to suppress the high-level scene details. As our approach relies on the homogeneous regions in the images, we skip the constrained convolutional layer. Note that we make use of all three input colour channels (RGB) to extract better forensic traces instead of relying on monochrome images as done by Bayar and Stamm (2018a).

As shown in Fig. 8, the first block consists of $96$ convolutional filters each of size 7 × 7 × 3, which are configured to use a stride of 2 × 2 and a valid zero padding. This convolutional layer is followed by a batch normalization layer (Ioffe & Szegedy, 2015), which in turn is followed by a Rectified Linear Unit (ReLU) activation to introduce non-linearity.

Unlike MISLNet, we use ReLU activations for the convolutional layers instead of the tanh activations. The activation layer is followed by a max-pooling layer of size 2 × 2, which effectively reduces the spatial dimensions by a factor of $4$. We follow a similar pattern for blocks 1–4, where each block consists of a convolution layer, batch normalization, and ReLU transformation followed by a max-pooling layer. The exact details of these blocks are given in Fig. 8. In the proposed ConvNet, blocks 1–4 constitute the feature extraction part of the network, while blocks 5–7 represent the classifier.

1. Download : Download high-res image (374KB)
2. Download : Download full-size image

Fig. 8. The proposed ConvNet architecture consisting of convolutional (Conv) and fully connected (FC) layers. The first four Conv blocks represent the feature extraction part of the network. Each Conv layer is followed by a batch normalization (BN), ReLU activations and a max-pooling (MaxPool) layer with a filter of size 2 × 2 pixels along with a stride of 2 × 2. The convolutional layers transform the input image of size 128 × 128 × 3 pixels into a feature vector of size $2048$. The final three fully connected layers classify the extracted feature vector into a source camera class label. The number of outputs in the FC3 layer is determined upon the sub-problem (brand or model classification) and the type of available devices. The blocks are not drawn to scale and, therefore, the labelled dimensions must be used for the exact details.

The output of the feature extractor block is flattened to obtain a $2048-$element feature vector. We use a fully connected neural network with three layers to classify these features into the desired camera class. In the case of brand classification, the output of the classifier is the camera brand (for example Sony, Canon, Nikon, and so on), while in the case of model classification the number of units in the output layer is set to match the number of camera models (for example Nikon_CoolPixS710, Nikon_D200, or Nikon_D70, among others). In order to avoid overfitting and achieve regularization, we use a dropout layer with a dropout factor of 0.3.

The learning parameters, loss function, and data set used for the experiments are described in Section 4.

3.4. Majority voting

An image may contain many homogeneous regions. Based on the patch selection scheme numerous homogeneous patches are extracted from the image. For each patch in a given image, we apply the concerned ConvNet model which gives us a label. All such patch-level predictions are then combined by taking a majority vote to determine the label for the given test image. As demonstrated in Section 4 this scheme of majority voting significantly decreases the error rate for image-level prediction in comparison to patch-level predictions.

3.5. Hierarchical classification scheme

To classify an input image to its source camera model, we propose to follow a two-level hierarchical classification approach as shown in Fig. 1. The first level is concerned with brand classification while the second one is for model classification. The training of these classifiers at both brand and model level are performed independently of each other. The trained classifiers can then be used in a hierarchical fashion to perform predictions on images. In order to predict the source camera model for a single image, firstly, homogeneous patches are extracted. Secondly, the brand level classifier determines the brand for each homogeneous patch. In case the majority vote on the predictions corresponds to only one camera model, the brand classification trivially determines the corresponding model-level classification. On the other hand, if the predicted brand corresponds to multiple camera models, then a second-level classifier is used to determine the class labels for the concerned patches. This step is followed by a majority vote to determine the predicted camera model for the given image.

The hierarchical scheme takes the advantage of training with fewer camera devices for each classifier. Since each classifier in the hierarchical scheme accounts for only a limited set of camera models, the training time to learn each classifier is reduced considerably. This also makes it possible to independently train each classifier in parallel. Moreover, with this modular approach, issues with any brand-specific classifier do not impact other brand-specific classifiers, and the classification scheme can be extended to include additional camera devices without having to retrain all classifiers.

4.1. Data set

To conduct our experiments we used the publicly available benchmark Dresden data set (Gloe & Böhme, 2010). It consists of more than 14,000 images captured by $66$ camera devices, $18$ camera models, and $13$ major camera brands. The images are divided into several subsets, namely ‘JPEG’, ‘natural’, ‘flat-field’, and ‘dark-field’, among others. We conducted our experiments on the ‘natural’ subset of the data set, which contains $74$ camera devices that were used to capture $84$ different indoor and outdoor scenes. This high number of diverse scenes makes the ‘natural’ subset the most challenging and realistic one, unlike the other subsets which contain images from at most two different scenes. The methods that we compare our results with also use the same ‘natural’ subset of Dresden. Of the $74$ camera devices, $8$ devices were present with a single instance of a camera model. In order to perform a fair evaluation for camera model identification, we decided to consider devices that have at least two instances from the same camera model. This requirement allows us to keep one instance aside for testing while using the rest for training. We, therefore, conduct experiments on the remaining $66$ devices. Together, they represent $18$ camera models, whose distribution is listed in Table 1. As suggested by Kirchner and Gloe (2015), we combined the models Nikon_D70 and Nikon_D70s into a single camera model (Nikon_D70) as they correspond to the same model with a different lens.

The natural images can be further classified into $84$ sub-categories, based on the scene content. In order to perform a reliable evaluation, Bondi et al. (2016) and Rafi, Wu, and Hasan (2020) used a subset of scenes for testing, while the rest were used for training and validation. That approach ensured that the model’s accuracy is scene independent. In our experiments we do not need this setting as the selection of homogeneous patches ensures that the scene-specific details are omitted.

4.2. Data balancing

In order to train a robust machine learning model it is necessary to ensure that the data imbalance is appropriately handled during the training phase. This becomes essential for our study as the distribution of the number of images per camera model varies significantly between classes. We refer the reader to Table 1 for the exact details of image distribution per camera model. Traditional data balancing techniques such as over-sampling and under-sampling of images from camera models may not result in a representative data set. More specifically, over-sampling causes repetition in the training data which can lead to model overfitting. Random under-sampling of data, on the other hand, can lead to missing out on potentially useful data and, therefore, we propose a systematic top-down scheme for patch balancing.

Consider the data imbalance problem for brand classification. Let the total number of image patches that we would like to train be denoted by $k$. The goal is to distribute these $k$ patches evenly across the training data set. The value of $k$ is an estimate that will drive the rest of the patch balancing algorithm. Suppose that brand classification is an $n_b$-class classification problem, where $n_b$ represents the total number of camera brands. Without loss of generality, let $k_b$ represents the number of patches to be sampled from any specific brand $b$. In order to construct a balanced data set, $k_b$ can be determined as: (1)$k_b=\frac{k}{n_b}⌉$where $k$, $k_b$, $n_b\in \mathbb{N}$, and $\lfloor \cdot \rceil$ denotes the standard rounding function in Python.² Let $n_m$ denotes the number of models representing the brand $b$. Without loss of generality, let $k_m$ denotes the number of patches to be sampled from a particular model $m$. In order to keep the number of patches the same across different models of the same brand, $k_m$ can be set to: (2)$k_m=\frac{k}{n_m\cdot n_b}⌉$where $k_m$, $n_m\in \mathbb{N}$. We continue this process to determine the number of patches to be sampled at the device-level $k_d$, for all the devices $n_d$ of model $m$. The value of $k_d$ is given by: (3)$k_d=\frac{k}{n_d\cdot n_m\cdot n_b}⌉$where $k_d$, $n_d\in \mathbb{N}$. Finally, we determine the number of patches to be sampled from each image $i$, captured by the device $d$ as: (4)$k_i=\frac{k}{n_i\cdot n_d\cdot n_m\cdot n_b}⌉$where $n_i$ represents the number of images in the data set belonging to the device $d$. Thus, for an initial estimate of $k$, the number of patches that need to be extracted from an image $i$ is determined by $k_i$, where $d$, $m$, and $b$ correspond to the device, model, and brand of the image, respectively. It should be noted that these $k_i$ patches are chosen by following the patch selection algorithm presented in Section 3.2.

The proposed method for patch selection ensures that the patches are evenly distributed at all levels of the hierarchy. Fig. 9 illustrates the patch balancing algorithm. Based on the choice of $k$, there might be very minor differences in the number of patches between different classes which is caused due to the rounding function. For our setting, this minor difference in class distribution is acceptable and is not considered a data imbalance.

1. Download : Download high-res image (321KB)
2. Download : Download full-size image

Fig. 9. An illustration of the patch balancing algorithm, where the patches are balanced at each level of the hierarchy. At the first level, an initial estimate of $k=260,000$ patches is distributed equally among the $13$ camera brands, resulting in $k_b=\text{20,000}$ patches per brand. Thereafter, in the second level, 20,000 patches are evenly distributed among the $3$ Nikon models, resulting in $k_m=6666$ patches per model, and likewise for Canon models. In the same fashion, patches are equally distributed at device and image levels.

4.3. Data set split

In line with Kirchner and Gloe (2015) we perform a 5-fold cross-validation, where we leave out one device per model for the test and use the remaining for training. For each fold, the device to be considered as a test device is determined in a round-robin fashion. As shown in Table 1, the number of camera devices per model is not the same. The maximum number of devices per model is $5$, and for such models, each device appears exactly once in the test set across the $5$ folds. For models where there are $4$ or fewer devices, we repeat the devices in the test set following a round-robin approach across the $5$ folds. This strategy of cross-validation accounts for the data set bias and provides us with reliable results. We report the results for all the folds, along with the global average.

4.4. Brand classification

As a first step, we perform camera brand classification. The data set that we consider consists of $n_c=13$ camera brands. Firstly, we balance the patches by setting $\hat{k}=260,000$, which gives us 20,000 patches per class.

The training loss is determined by computing the categorical cross-entropy between the target and the predicted output of the network. Let $f({\mathbf{x}}_{\mathbf{i}};\mathbf{w})$ represents the network with weights $\mathbf{w}$, input example ${\mathbf{x}}_{\mathbf{i}}$, and the corresponding softmax output as ${\mathbf{y}}_{\mathbf{i}}$. The categorical cross-entropy is a multi-class logistic loss function which is defined as: (5)$\mathcal{L}(f({\mathbf{x}}_{\mathbf{i}};\mathbf{w});{\mathbf{t}}_i)=-log\left({\mathbf{y}}_i\right)\cdot {\mathbf{t}}_i$(6)$\mathcal{J}\left(\mathbf{w}\right)=\frac{1}{s}\sum _{i=1}^s{l}_i$ where $\mathcal{L}$ denotes the loss function, ${\mathbf{t}}_{\mathbf{i}}\in {\{0,1\}}^{n_c}$ denotes the one-hot encoded target vector for the input ${\mathbf{x}}_{\mathbf{i}}$, and $\mathcal{J}$ is the empirical loss for the mini-batch of size $s$. In our experiments, we set the batch size $s=512$.

We use the technique of stochastic gradient descent (SGD) for the optimization of the model parameters with a learning rate of 0.1 and momentum of 0.8. We were able to set such a high initial learning rate because of the batch normalization layers in the ConvNet. Furthermore, an exponential learning rate decay is used with a multiplicative factor of 0.9 for achieving model convergence. The learning rate is decayed at the end of every epoch. In order to avoid overfitting, we used l2-regularization by setting the weight decay factor to 0.0005. With these settings, the model was trained until convergence. We perform early stopping when the loss converges, and it does not fluctuate more than 0.02 for $5$ consecutive epochs. Fig. 10 shows the convergence in the loss that was achieved with these hyper-parameters. The best epoch is then determined with the maximum validation accuracy for each fold and used during the evaluation phase of the brand classifier. The proposed ConvNet model consists of $2,585,149$ trainable parameters (for $n_c=13$).

Fig. 11 shows the confusion matrix for the classification of $13$ camera brands. We show the confusion matrix only for the first fold (refer to Fig. 11) and report a summary of the remaining folds in the first row of Table 2. Note that the results in Table 2 correspond to image-level classification accuracy and not at the level of patches. During the training phase, the training data set is balanced in a hierarchical fashion as shown in Fig. 9. This is, however, not the case during model evaluation. For a given test image, a set of $200$ homogeneous patches is extracted. The trained model is then used to predict class labels for all $200$ homogeneous patches. Finally, a majority vote is taken to determine the predicted class label for the given image. Taking the majority vote on patch-level predictions decreased the average error rate by almost $50$ percent for image-level predictions in comparison to individual patch-level predictions. Since the test data is skewed between classes, in addition to accuracy we also report the macro F1 score. It is defined as the empirical average of class-wise F1 scores, with the F1 score for each class defined as: (7)$\text{F1}=\frac{2\cdot \text{TP}}{2\cdot \text{TP}+\text{FP}+\text{FN}}$where TP, FP, and FN stand for true positives, false positives, and false negatives, respectively. The accuracy values reported in our experiments have been determined using the following equation: (8)$\text{Accuracy}=\frac{\text{\# correct predictions}}{\text{total \# predictions}}$

1. Download : Download high-res image (329KB)
2. Download : Download full-size image

Fig. 10. The plots depict the convergence of the loss for fold 1 (out of 5). On the left are three stacked plots each representing the respective training loss plot for Nikon, Samsung, and Sony classifiers. The loss plot on the right corresponds to the training of the brand-level classifier. In all plots the dashed vertical lines indicate the epoch where the highest validation accuracy was achieved. The trained model for each classifier was chosen based on that epoch.

We achieved an average accuracy of $0.9941\pm 0.0010$ percent and an average macro F1 score of $0.992\pm 0.001$ for brand-level classification. Having trained and tested the brand classifier, we now discuss similar aspects for model-level classifiers.

1. Download : Download high-res image (223KB)
2. Download : Download full-size image

Fig. 11. The confusion matrix of the results (fold 1) for the classification of $13$ camera brands. The $13$ camera brands are 1. Canon, 2. Casio, 3. FujiFilm, 4. Kodak, 5. Nikon, 6. Olympus, 7. Panasonic, 8. Pentax, 9. Pratica, 10. Ricoh, 11. Rollei, 12. Samsung, and 13. Sony. Note that each brand in the test set has a different number of images, which were all used during the evaluation.

Table 2. Classification accuracy for the proposed approach on classifying 200 homogeneous patches from each image in the test set.

Classification	fold 1	fold 2	fold 3	fold 4	fold 5	Average
Brands	0.995	0.992	0.994	0.994	0.995	$0.9941\pm 0.0010$
Nikon	0.997	0.997	0.999	0.997	0.997	$0.9973\pm 0.0006$
Samsung	1.000	0.995	0.998	1.000	0.995	$0.9976\pm 0.0022$
Sony	0.970	0.989	0.965	0.980	0.972	$0.9751\pm 0.0085$
Hierarchical	0.991	0.991	0.989	0.990	0.989	$0.9899\pm 0.0007$

4.5. Model classification

In order to train model-level classifiers, we consider only those camera brands which have multiple camera models. Of the $13$ camera brands, only $3$ brands have multiple camera models, namely Nikon, Samsung, and Sony. We use the same ConvNet architecture for training model-level classifiers as described in Section 4.4. In order to create a level-balanced training data set for model classification, we need to determine the number of patches to be extracted from each image. This can be determined using Eq. (4) and by setting $n_b=1$ (since in model-level classification we are dealing with $1$ brand per model). Therefore, for model-level classification, the number of patches to be extracted from each image $i$ is determined by: (9)$k_i=\frac{k}{n_m\cdot n_d\cdot n_i}⌉$

For the training of the Nikon, Samsung, and Sony classifiers, we choose $k=\text{60,000}$, 40,000, and 60,000 patches, respectively. We set these values with the idea of extracting 20,000 patches from each camera model. Based on the mentioned values for $k$, the data sets are prepared for training the respective classifiers. We continue to perform $5$-fold cross-validation and ensure that the distribution of camera devices remains consistent between the folds of the brand and model-level classification.

Moreover, the training of each model-level and the brand-level classifiers can be performed in parallel, as there is no dependency during the training phase. Using the same hyperparameters, which were used to train the brand classifier, we train each model-level classifier for about $40$ epochs. The model-level classifiers that achieve the highest validation accuracy rates are chosen as the final ones.

Fig. 12 shows the confusion matrix for each model-level classifier. They are generated on the test data for the first fold. The Nikon classifier achieves an average accuracy of $0.9973\pm 0.0006$ percent across all $5$ folds. Similarly, the Samsung and Sony classifiers achieve an average accuracy of $0.9976\pm 0.0022$ and $0.9751\pm 0.0085$ percent, respectively. As in the case of brand classification, taking a majority vote on patch-level predictions to determine the image-level prediction reduces the error rate by almost 60 percent. For complete details regarding the reduction in error rates, refer to Table 4. Finally, we determine the macro F1 score to account for the imbalance in the test data set. The Nikon, Samsung, and Sony classifiers achieve an average F1 score of $0.997\pm 0.001$, $0.998\pm 0.002$, and $0.976\pm 0.008$, respectively. We refer the reader to Table 3 for complete details concerning the macro F1 scores. The end-to-end hierarchical evaluation pipeline is described in the following sub-section.

1. Download : Download high-res image (209KB)
2. Download : Download full-size image

Fig. 12. Confusion matrices achieved by the Nikon, Samsung, and Sony model-level classifiers. They are generated by evaluating the test data in fold $1$ (out of $5$).

4.6. Hierarchical classification

Having trained the brand-level and model-level classifiers, we now use them in a hierarchical fashion to predict the source camera model for a given image. We begin by extracting $200$ homogeneous patches from a given image. At the top of the hierarchy is the brand-level classifier, which selects the brand for each of the $200$ patches. A majority vote is performed to determine the source camera brand. In the trivial case, when there is only one camera model for a particular brand, the brand classification directly determines the model classification. Otherwise, based on the predicted brand, the corresponding model-level classifier is used to determine the source camera model. In our case, we trained three different model-level classifiers, one each for the Nikon, Samsung, and Sony brands. A majority vote is once again performed on all $200$ patch predictions that determine the source camera model for the given image.

The first fold results of the hierarchical evaluation are presented as a confusion matrix in Fig. 13. We achieve an overall classification accuracy of $0.9899\pm 0.0007$ and an overall macro F1 score of $0.990\pm 0.001$ across all five folds. Table 2, Table 3 summarize the accuracy and the macro F1 score achieved for each fold, respectively.

In order to compare our method with the state-of-the-art, we choose methods that have been evaluated on the same ‘natural’ subset of the Dresden data set. The summary of this comparison is presented in Table 5. Similar to the works in Bondi et al., 2016, Rafi, Tonmoy, et al., 2020, Rafi, Wu, and Hasan, 2020 we follow the leave-one-device-out strategy for cross-validation, and show the results in Table 5. Our strategy of homogeneous patch selection ensures to ignore scene details from the image, which allows us to avoid inadvertent classification of the scene content. It was, therefore, not necessary to employ the scene-independent test set strategy proposed by Bondi et al. (2016). Marra et al. (2017) performed experiments on $25$ camera models of which $7$ models have only one device per model in the Dresden data set. In their setting, it is not possible to test the generalizability of the trained classifier for those $7$ devices.

1. Download : Download high-res image (284KB)
2. Download : Download full-size image

Fig. 13. Confusion matrix for classifying $18$ camera models, based on the hierarchical pipeline. The result shown here is generated on the test data for fold $1$ of $5$. The $18$ camera models correspond to the list of camera models in Table 1. Note that each brand in the test set has a different number of images and all were used during the evaluation.

Table 3. Macro F1 scores obtained on classifying 200 homogeneous patches from each image in the test set.

Classification	fold 1	fold 2	fold 3	fold 4	fold 5	Average
Brands	0.993	0.992	0.992	0.992	0.993	$0.992\pm 0.001$
Nikon	0.997	0.997	0.999	0.997	0.997	$0.997\pm 0.001$
Samsung	1.000	0.995	0.998	1.000	0.995	$0.998\pm 0.002$
Sony	0.971	0.989	0.967	0.981	0.973	$0.976\pm 0.008$
Hierarchical	0.989	0.991	0.988	0.990	0.990	$0.990\pm 0.001$

Table 4. Reduction in error rate (percent) with respect to the accuracy for image-level predictions (majority vote of 200 homogeneous patches) in comparison to patch-level predictions.

Classification	fold 1	fold 2	fold 3	fold 4	fold 5	Average (%)
Brands	49.2	40.8	47.1	50.9	52.7	$48.17\pm 03.75$
Nikon	30.3	77.1	65.7	70.2	35.2	$55.69\pm 17.47$
Samsung	100	29.4	25.5	100	31.1	$57.20\pm 31.95$
Sony	68.0	88.5	66.6	79.5	69.4	$74.43\pm 07.66$

5.1. Number of patches

The results reported thus far were achieved by using $200$ homogeneous patches per image during the test phase. In order to understand the number of patches required for evaluation, we conducted a series of experiments which are summarized in Fig. 14. As can be seen, certain brands can work with a few patches while others require more patches to obtain high accuracy. This behaviour is evident with the Sony classifier, which significantly improves in performance with an increasing number of patches. Interestingly, the performance of the Samsung classifier slightly decreases when the number of patches is increased from $1$ to $10$. This minor deviation from the expected trend is, however, so small that it is probably due to chance. In general, classification becomes progressively robust with increasing number of randomly selected homogeneous patches, up to a certain point.

The extraction of too many patches would lead us into the regime of non-homogeneous patches. Therefore, for the Dresden data set, we recommend performing prediction using $200$ patches. Based on the homogeneous patch distribution shown in Fig. 7 we can conclude that most of the images (from all camera models) contain at least $200$ homogeneous patches of size 128 × 128 pixels. In fact, in Fig. 14 one can observe that the classification accuracy of the Sony classifier decreases on considering $400$ patches when compared to using only $200$ patches. This decrease in accuracy may be due to the inclusion of non-homogeneous patches when we extract $400$ patches (Fig. 7). As the Dresden data set constitutes images belonging to diverse scenes, it is reasonable to assume that most natural images contain about $200$ homogeneous patches.

1. Download : Download high-res image (249KB)
2. Download : Download full-size image

Fig. 14. The image-level accuracy achieved with varying numbers of homogeneous patches. Each accuracy point corresponds to the average across the $5$-folds.

It is also important to account for the time taken to extract such patches. In our experiments, the extraction of patches from an image of size 3072 × 2304 pixels with patches of size 128 × 128 pixels took approximately 2.16 seconds when measured on a single core of Intel Xeon E5-2680 v3 CPU (2.5 GHz). Because we use the integral image for patch selection and extraction the execution time primarily depends on the image size and not on the number of homogeneous patches.

5.2. Hierarchical vs flat approach

One of our hypotheses was that a hierarchical approach is better than using a single classifier (flat approach). In order to evaluate this hypothesis, we conducted experiments by training all the $18$ camera models using a single classifier. For a fair comparison, the ConvNet parameters were kept unchanged and were set to match our earlier experiments. We evaluated the flat ConvNet by using $200$ homogeneous patches and performed a $5$-fold cross-validation. The results of these experiments are summarized in Table 6. We obtained an average classification accuracy of $0.9851\pm 0.0033$. In comparison to the hierarchical approach, the average error rate increases by more than $50$ percent for the flat approach.

The flat classifiers also took more epochs to converge when compared to their hierarchical counterparts. This difficulty in model learning is due to the mixing up of different camera models and brands into a single classifier. When using the hierarchical approach the brand classifier accounts for inter-brand noise variation, while the model-level classifiers account for the intra-brand noise variations. This separation of noise variations allows the hierarchical classifiers to learn better features for classification. Notable is also the fact that the results obtained by the flat approach are also better than existing works (Table 5).

Certain brands can achieve high accuracy with fewer patches. For instance, referring to Fig. 14 we could use only $10$ patches per image to determine the camera brand with around 99.5 percent accuracy. This is also the case with Nikon and Samsung classifiers. The Sony classifier, however, achieves the highest classification accuracy with $200$ patches. One may, therefore, exploit this result to use the optimal number of patches per model in order to also maximize the speed of the evaluation process, something that would not be possible with a single-classifier approach.

5.3. Future work

One direction for future research is the evaluation of the proposed approach to varying levels of image quality. In such evaluation one may consider no-reference image quality measures (Gu et al., 2017, Gu et al., 2016, Mittal et al., 2012), which can be used to determine the visual quality of images, among others (Gu, Li, et al., 2017, Gu, Zhou, et al., 2017). Using this information, the relationship between the input image quality and the resulting performance for SCI can be determined. This information could be used to assess whether a given image has sufficient quality for a reliable determination of its camera model.

Another direction would be to extend this work for device-level identification, which presents further challenges than those of the model-level identification at hand. Such an approach would be particularly useful when LEAs need to discriminate between two devices of the same camera model.