Fresh and Rotten Fruits - Product inspection with Convolutional Neural Networks

The objective of this analysis is to develop a deep learning model that can accurately classify images of fresh and rotten fruits. We explored three models: a baseline Convolutional Neural Network (CNN), an enhanced CNN with Batch Normalization, and a VGG16-based model with pretrained layers. The analysis aims to automate fruit quality assessment, which is traditionally done manually and is both labor-intensive and error-prone. Automation is expected to reduce costs and improve efficiency for stakeholders like fruit distributors and retailers. The study utilized a diverse and realistic dataset, including augmented images, to ensure robust performance across various real-world conditions. Among the models tested, the CNN with Batch Normalization using Augmented Data achieved the highest validation accuracy of 95 Percent, making it the best-performing model in this study. This model not only provided the best accuracy but also balanced precision and recall, making it the most reliable for distinguishing between fresh and rotten fruits. The model's success demonstrates the effectiveness of augmentation and Batch Normalization in improving model performance while maintaining computational efficiency, making it well-suited for deployment in real-world fruit quality assessment tasks.

1. Objective of the Analysis

The primary goal of this analysis is to develop a robust deep learning model that can accurately classify images of fresh and rotten fruits. We will utilize a Convolutional Neural Network (CNN) due to its proven effectiveness in image classification tasks as base model. We will then extend the base model by Batch Normalization layers as a second model. As third model we include a pretrained model VGG16 with pretrained layers extented by customized final layers. This analysis is intended to assist stakeholders, including fruit distributors and retailers, by automating the process of assessing fruit quality. This automation can significantly reduce labor costs and minimize the errors associated with manual inspections.

Value of the Analysis

• Current Challenges: Typically, the task of distinguishing fresh from rotten fruits is performed manually, which is not only inefficient but also costly for fruit farmers and vendors. Therefore, creating an automated classification model is essential to reduce human effort, lower costs, and expedite production in the agriculture industry.
• Comprehensive Scope: Compared to other datasets, this one includes a wider range of fruit classes, which can help researchers achieve better performance and more accurate classifications.
• Realistic Data: The images in this dataset were collected from various fruit markets and fields under natural weather conditions, which include varying lighting conditions. This diversity in data collection makes it challenging to identify defects with the naked eye, enhancing the dataset's realism.
• Economic Impact: Early detection of fruit spoilage, as facilitated by this research, could enable farmers to produce larger quantities of quality fruit, thereby contributing positively to the economy.

2. Dataset Description

Dataset Overview

Fruits play a critical role in the economic development of many countries, and consumers demand fresh, high-quality produce. Since fruits naturally degrade over time, this can negatively impact the economy. It's estimated that about one-third of all fruits become rotten, leading to significant financial losses. Furthermore, spoiled fruits can harm public perception and health, which can reduce sales. This dataset provides a foundation for developing algorithms aimed at the early detection and classification of fresh versus rotten fruits, which is vital for mitigating these issues in the agricultural sector.

This dataset includes an extensive collection of fruit images across sixteen categories, including fresh and rotten variants of apples, bananas, oranges, grapes, guavas, jujubes, pomegranates, and strawberries. In total, there are 3,200 original images and an additional 12,335 augmented images. All images are uniformly sized at 512 × 512 pixels.

Citation:
Sultana, Nusrat; Jahan, Musfika; Uddin, Mohammad Shorif (2022), “Fresh and Rotten Fruits Dataset for Machine-Based Evaluation of Fruit Quality”, Mendeley Data, V1, doi: 10.17632/bdd69gyhv8.1.

Dataset Summary

Description of Fruit Classes

Here are brief descriptions of the covered fruit classes:

3. Data Exploration and Preprocessing

The dataset is well-balanced across the various fruit classes, which is beneficial for developing a reliable model. As we analyse images there are no further adjustments like for missing values or outliers needed. Below are some sample images from the dataset to illustrate the variety and quality of the data.

Sample pictures of fresh and rotten fruits in the dataset

4. Model Training

Model Selection and Training

We will experiment with three variations of a Convolutional Neural Network (CNN):

1. Baseline Model: A simple CNN with three convolutional layers.
2. Enhanced Model: A CNN with three convolutional layers, Batch Normalization layers and dropout for regularization.
3. VGG16 Pretrained Model: A CNN model based on the ImageNet pretrained VGG16, customized with a final Dense layer and an Output layer.

Each model is trained and evaluated for the Original Dataset and the augmented data set. These Image datasets are split in 80% training and 20% test data used vor validation.

CNN Baseline Model

The CNN Baseline Model consists of three Convolutional Blocks each with a Convolutional Layer and a Pooling layer. The total number of parameters is 4,762,928 from which all are trainable.

The table below summarizes the parameters in each layer, the architcture is hown in Appendix 1.

Below you can find the results for the original and augmented Dataset after 20 Epochs of training and a batch size of 20. Both scenarios show improved performance towards the end of the epochs. No signs of overfitting can be detected. The use of the larger augmented dataset results in improved performance.

Accuracy for CNN Basemodel

Loss for CNN Basemodel

Accuracy for CNN Basemodel with Augmented Dataset

Loss for CNN Basemodel with Augmented Dataset

CNN with Batch Normalization

The secon CNN Model adds Batch Normalization and Dropout layers to the Baseline Model. The total number of parameters is slightly increased to 4,764,400 from which all are trainable.

The table below summarizes the parameters in each layer, the architcture is hown in Appendix 2.

Below you can find the results for the original and augmented Dataset after 20 Epochs of training and a batch size of 20. Both scenarios show improved performance towards the end of the epochs. No signs of overfitting can be detected. As with the previous model the use of the larger augmented dataset results in improved performance.

Accuracy Loss for CNN Batch Normalization

Loss for CNN Batch Normalization

Accuracy for CNN Batch Normalization with Augmented Dataset

Loss for CNN Batch Normalization with Augmented Dataset

VGG16 Pretrained Model

The third mmodel is based on the pretrained VGG16. VGG16 is a deep convolutional neural network model that was trained on the ImageNet dataset, which contains millions of images across 1,000 classes. The model has 16 layers, including 13 convolutional layers, that are designed to extract hierarchical features from images, making it highly effective for various image classification tasks. Pretrained VGG16 on ImageNet is commonly used for transfer learning, allowing the model to be fine-tuned for specific tasks with relatively small amounts of new data. For customization we added 4 layers, a Pooling Layer, a Dense Layer an additional Dropout layer and the final Output layer. The total number of layers is 14,985,552 from which only 270,864 are trainable.

The table below summarizes the parameters in each layer, the architcture is hown in Appendix 3.

Below you can find the results for the original and augmented Dataset after 20 Epochs of training and a batch size of 20. Both scenarios show improved performance towards the end of the epochs. No signs of overfitting can be detected. As with the previous two model the use of the larger augmented dataset results in improved performance.

Figure 10: Accuracy for VGG Pretrained

Loss for VGG Pretrained

Accuracy for VGG Pretrained with Augmented Dataset

Loss for VGG Pretrained with Augmented Dataset

5. Model Evaluation and Recommendation

Model Evaluation

Each model's performance was evaluated using accuracy, precision, recall, F1 score and AUC. The table below summarizes the results:

Model Recommendation

The CNN Batch Normalized Model with Augmented Data shows the best validation accuracy and also shows great performance for all other metrics including a balance between precision and recall. This model is well-suited for the task of classifying fresh and rotten fruits.

We show the resulting confusion matrix for the augmented dataset and a validation size of 2461 images.

Confusion matrix for CNN Batch Normalized Model with Augmented Data

The Confusion matrix confirms the high accuracy of the model. The only outlier would be the 31 cases of Rotten Apples which were classified as Rotten Pomegranates. Rotten Oranges on the other hand were perfectly classified.

Below we present 15 examples of misclassification, one for each class of fruit (except for Rotten Oranges as mentioned without misclassification).

Examples of misclassification for CNN Batch Normalized Model with Augmented Data

We can summarize the performance for each class as shown in the Classification Report Heatmap below. Rotten Oranges have a perfect Recall value. Perfect Precision values can be found for Rotten Guavas, Rotten Bananas and Rotten Strawberries.

Classification Report for CNN Batch Normalized Model with Augmented Data

6. Key Findings and Insights

Summary of Findings

• The CNN Batch Normalized Model with Augmented Data achieved the highest accuracy of 95%, making it the most effective model for this classification task.
• Data Augmentation played a crucial role in enhancing model performance by introducing more diversity in the training data.
• The Baseline Model provided a solid starting point but was outperformed by the more complex models.
• Enhanced Model showed improved results, demonstrating the benefits of Batch Normalization architectures.
• Overall, the CNN Batch Normalized Model with Augmented Data strikes a good balance between accuracy and model complexity, making it the optimal choice for this application.
• The Pretrained VGG Model with Augmented Data shows similar performance and could be considered as alternative, while being more complex and computational intensive.

Insights

• Impact of Augmentation: The application of data augmentation techniques significantly improved the model's generalization capability, suggesting that further exploration of augmentation strategies could yield even better results.
• Model Complexity: While increasing the depth of the network improved performance, it also increased training time and computational cost, highlighting the trade-offs involved in model selection.
• Future Directions: Exploring more advanced architectures such as ResNet or using ensemble methods could potentially lead to further improvements in classification accuracy.

7. Conclusion and Next Steps

Conclusion

This analysis demonstrates the effectiveness of using a Convolutional Neural Network (CNN) for classifying fruit quality based on images. Among the models tested, the CNN Batch Normalized Model with Augmented Data stands out as the most reliable, achieving the highest accuracy and offering a balanced performance in terms of precision and recall. This model is well-suited for the task of distinguishing between fresh and rotten fruits and can serve as a robust solution for automated fruit quality assessment.

Suggestions for Future Work

• Explore Additional Data: Incorporating more fruit varieties or expanding the dataset with additional images could enhance the model’s robustness and generalizability.
• Experiment with Advanced Architectures: Testing more complex architectures such as ResNet, Inception, or DenseNet might yield further improvements in classification accuracy and model performance.
• Ensemble Methods: Investigating the use of ensemble methods, where multiple models are combined, could potentially increase the reliability of the predictions.
• Real-World Application: Consider deploying the model in a real-world setting to automate the fruit quality assessment process in retail or agricultural environments. This could involve integrating the model into an existing supply chain management system.
• Optimize for Deployment: Further work could focus on optimizing the model for deployment, including reducing its size and improving inference speed without sacrificing accuracy.

Appendix

CNN Basemodel Layers

CNN Batch Normalization Model Layers

Appendix 3:VGG Pretrained Model Layers