TMAF-YOLO:温室番茄成熟度与缺陷果实轻量化实时检测模型

1. Introduction

Tomato is an important vegetable crop in protected agriculture. Its maturity directly affects harvesting time, transport management, postharvest ripening, quality grading, and commercial value. In greenhouse environments, tomato fruits are often affected by leaf occlusion, fruit overlap, uneven illumination, and scale variation. Manual maturity assessment is inefficient and easily influenced by subjective experience. Therefore, a rapid, stable, and non-contact detection method is needed to support automated harvesting, grading management, and robotic visual perception in protected agriculture [1,2,3].

Among deep learning detection methods, region-based convolutional neural network (R-CNN) models and their variants were among the earlier methods applied to fruit and vegetable detection and counting tasks. An improved Faster R-CNN model, MatDet, was proposed for multi-object tomato maturity detection in complex scenarios [1]. The model used a 50-layer residual network (ResNet-50), region-of-interest alignment (RoIAlign), and a path aggregation network (PANet) to improve detection under occlusion, overlap, and illumination variation, achieving a mean average precision (mAP) of 96.14%. Faster R-CNN has also been used for tomato-related detection tasks. Highly occluded immature tomatoes on plants were detected using Faster R-CNN, with an average precision of 87.83% [4]. Greenhouse tomato flowers were detected and counted using Faster R-CNN, achieving an average precision of 96.02% [5]. These methods have strong localization ability and can provide stable target region information. However, their model structures are relatively complex, and their inference speed and lightweight deployment ability remain limited.

With the development of Transformer architectures, end-to-end detectors such as Detection Transformer (DETR), DETR with Improved deNoising anchOr boxes (DINO), and Real-Time Detection Transformer (RT-DETR) have been introduced into object detection tasks. DETR formulates object detection as a set prediction problem [6]. DINO improves the convergence and detection performance of DETR-like models through denoising training and query optimization [7]. RT-DETR further improves the Transformer detection structure for real-time detection requirements [8]. In fruit and vegetable maturity recognition, PDSI-RTDETR was proposed for tomato maturity detection. Compared with the original RT-DETR, its mAP50 and mAP50-95 increased by 3.9 and 4.1 percentage points, respectively [9]. An improved RT-DETR was also applied to fruit ripeness detection, indicating the potential of Transformer detectors in global feature modeling [10]. However, these models usually have higher training costs, larger parameter sizes, and greater hardware requirements.

In contrast, You Only Look Once (YOLO) series models provide a better balance between detection accuracy and inference speed. They have become common methods for fruit and vegetable detection and maturity recognition. A lightweight improved YOLOv5 model was proposed for real-time tomato detection and mobile deployment [2]. Compared with the original YOLOv5s, the number of parameters and floating-point operations (FLOPs) were reduced by 78% and 84.15%, respectively, while maintaining high detection accuracy. RGB-D information was combined with YOLOv5 to detect and count greenhouse tomato clusters [3]. MTD-YOLO was proposed for multitask detection of cherry tomato fruits, clusters, and maturity [11]. The model achieved an overall detection score of 86.6% and an average inference time of 4.9 ms. An improved YOLOX model was proposed for tomato ripeness and stem recognition, achieving an mAP of 92.17% and improving recognition under sample imbalance and scale variation [12]. YOLOv8 was combined with a lightweight Swin Transformer for strawberry ripeness detection [13]. Compared with YOLOv5s, CenterNet, and the Single Shot MultiBox Detector (SSD), the mAP50 increased by 1.6%, 33.5%, and 3.4%, respectively. These results indicate that improved YOLO models are suitable for fine-grained fruit maturity recognition. In addition, recent YOLO-based studies on tomato detection and ripening recognition have improved detection performance using attention mechanisms, lightweight structures, multiscale feature fusion, and postprocessing optimization [14,15,16,17].

Existing studies have verified the effectiveness of deep learning methods in fruit and vegetable detection and maturity recognition. However, further improvement is still needed for in situ greenhouse tomato detection. First, the color transition between semi-mature and mature tomatoes is continuous, making fine-grained classification difficult. Second, leaf occlusion, fruit overlap, and illumination variation weaken maturity-related feature representation. Third, defective samples are relatively limited, and conventional classification losses may underemphasize minority and hard samples. Previous studies have shown that Focal Loss and Class-Balanced Loss can adjust the loss weights of hard and imbalanced samples, thereby improving the learning ability for long-tailed classes and difficult samples [18,19]. To address these issues, this study proposes TMAF-YOLO based on YOLOv8n.

The main contributions of this study are as follows.

(1)
A lightweight tomato maturity detection model was constructed for complex greenhouse scenarios. Inspired by the low-cost feature generation strategy in GhostNet, an LGhostConv module was designed and embedded in key positions of the Backbone and Neck [20]. This design reduces the number of parameters and computational cost while maintaining detection performance.

(2)
A tomato maturity-aware aggregation fusion module, termed TMAF, was proposed. This module was designed to address continuous color transitions between adjacent maturity stages, weak local texture differences, and insufficient feature representation caused by occlusion and overlap. By integrating local texture enhancement and channel-spatial recalibration, TMAF strengthens maturity-related feature representation.

(3)
MA-CB Focal Loss was developed to improve learning under class imbalance and hard-sample conditions. By combining class-balanced weights with the Focal Loss modulation term, the loss enhances learning for minority and difficult samples, including defective fruits and adjacent maturity stages.

2. Materials and Methods

2.1. Overall Method Design

This study focuses on in situ tomato maturity detection before greenhouse harvesting. An overall application workflow was constructed, including greenhouse image acquisition, fruit detection and category recognition, harvesting decision-making, and postharvest grading management. As shown in Figure 1, tomato images were first acquired under natural growth conditions in the greenhouse. Then, the developed detection model was used to identify fruit locations, maturity stages, and defective fruits. Finally, the detection results were used to support harvesting time determination, defective fruit screening, and postharvest grading and quality management. It should be noted that this study mainly focuses on greenhouse tomato maturity and defective fruit detection. The harvesting, grading, and management processes in the figure are used to illustrate the potential application scenarios of the proposed model.

Figure 1. Overall workflow of greenhouse tomato maturity detection.

2.2. Dataset Construction and Annotation

The self-collected images used in this study were acquired in a tomato greenhouse at the Tianjin Academy of Agricultural Sciences, China, from 10 to 17 January 2026. Image acquisition was conducted between 9:00 a.m. and 12:00 p.m. using a HUAWEI Mate 60 smartphone (Huawei Device Co., Ltd., Dongguan, China) in automatic shooting mode. Exposure, focus, ISO sensitivity, and shutter speed were adjusted automatically by the built-in camera system. The shooting distance ranged from approximately 0.5 to 2.0 m, and the image resolution was 1280 × 760 pixels. Images were captured under naturally varying greenhouse illumination; illuminance was not measured instrumentally.

To represent practical greenhouse conditions, images were acquired from different viewing angles and distances. The collected scenes included fruits at different maturity stages and scales, together with leaf and branch occlusion, fruit overlap, dense fruit distribution, small targets, complex backgrounds, and local image blur. Representative examples of self-collected images are shown in Figure 2.

Figure 2. Representative tomato images acquired under typical greenhouse conditions: (a) mature fruits; (b) adjacent fruits at different maturity stages; (c) fruit overlap and foliage interference; (d) leaf and branch occlusion; (e) small targets in a relatively long-range view; and (f) local image blur.

A total of 596 valid images were collected from the greenhouse. Because defective fruits were relatively scarce, 33 additional defective-fruit images were selected from the Tomato Fruit Diseases Dataset available on Kaggle (https://www.kaggle.com/datasets/profnourasemary/tomato-dataset, accessed on 1 July 2026). The additional images mainly consisted of tomato fruits exhibiting visible disease symptoms, together with a smaller number showing physical damage, decay, cracking, or other abnormal surface characteristics. The selected public images were annotated according to the same criteria as the self-collected images.

The 596 self-collected images were divided at the original-image level into training, validation, and test subsets containing 470, 63, and 63 images, respectively. The 33 public images were added exclusively to the training set to increase the number and diversity of defective-fruit samples. Consequently, the final training, validation, and test sets contained 503, 63, and 63 images, respectively. The validation and test sets therefore consisted entirely of self-collected greenhouse images.

The annotated tomato instances were divided into four categories: immature, semi-mature, mature, and defective. The first three categories represent successive normal maturity stages based primarily on fruit color and visible appearance. The defective category includes fruits with visible disease spots, physical damage, decay, cracking, or other abnormal surface characteristics and was treated as an independent appearance category rather than as a normal maturity stage.

All tomato instances were manually annotated with bounding boxes using LabelMe v6.0.0. The original annotations were stored in JSON format and subsequently converted into the format required by the YOLO detection framework. Partially occluded or locally blurred fruits were retained when their categories and principal visible boundaries could be reliably determined. Instances were excluded only when severe occlusion, insufficient visible information, or ambiguous appearance prevented consistent annotation. All annotations were reviewed according to uniform class definitions to ensure labeling consistency.

The final dataset contained 629 images and 2505 annotated tomato instances, including 1170 immature, 788 semi-mature, 304 mature, and 243 defective instances. Data augmentation was performed after dataset partitioning and was applied only to the training set. The validation and test sets were not augmented. The training set was used for model parameter optimization, the validation set for training monitoring and model selection, and the test set exclusively for final performance evaluation. The image and instance distributions before data augmentation are summarized in Table 1.

Table 1. Distribution of images and annotated tomato instances before data augmentation.

2.3. Construction of the TMAF-YOLO Model

2.3.1. TMAF-YOLO: An Improved YOLOv8n Model

The proposed TMAF-YOLO uses YOLOv8n as the baseline model. It is improved from three aspects: lightweight feature extraction, maturity-related feature fusion, and classification loss optimization. The overall architecture is shown in Figure 3. After the original image is input into the network, multiscale features are first extracted by the Backbone, in which selected convolutional structures are replaced by LGhostConv. Then, features at different scales are fused in the Neck, and TMAF modules are added to the P3/8 and P4/16 branches. Finally, the model outputs prediction results through three detection heads at P3/8, P4/16, and P5/32 scales. During training, the model is optimized using a joint loss function composed of WIoU, DFL, and MA-CB Focal Loss. The specific designs of LGhostConv, TMAF, and MA-CB Focal Loss are introduced in the following sections.

Figure 3. Overall architecture of TMAF-YOLO.

2.3.2. Lightweight Feature Extraction Module: LGhostConv

To reduce redundant computation during feature extraction and multiscale feature fusion, a lightweight Ghost convolution module, termed LGhostConv, was introduced into selected downsampling layers of the backbone and neck. Similar to GhostConv, LGhostConv generates output features through primary and complementary feature branches. However, it replaces the dense spatial convolution in the primary branch with a depthwise-separable operation and employs directional depthwise convolutions in the complementary branch.

Given an input feature map $X$ with $C_{i n}$ channels, LGhostConv first applies a $k \times k$ depthwise convolution with stride s for spatial feature extraction and downsampling, followed by a $1 \times 1$ pointwise convolution for channel interaction. This process generates approximately half of the required output channels as the primary features. The remaining channels are generated by dividing the primary features into two groups and applying $1 \times 3$ and $3 \times 1$ depthwise convolutions to capture horizontal and vertical information, respectively. The directional features are then fused with the primary features through a parameter-free residual connection and concatenated to form the final output.

Ignoring bias terms, batch-normalization parameters, and activation operations, the number of parameters in LGhostConv is expressed as follows:

$$
P_{L G h o s t C o n v} = k^{2} C_{i n} + C_{i n} m + 3 n
$$

(1)

where $m$ = ⌈$C_{o u t}$/2⌉ and $n$ = $C_{o u t}$ − $m$ denote the numbers of primary and complementary feature channels, respectively. In comparison, a standard $k \times k$ convolution requires $k^{2} C_{i n} C_{o u t}$ parameters. Therefore, LGhostConv reduces model complexity by replacing dense spatial channel interactions with depthwise spatial filtering, pointwise channel projection, and low-cost directional feature generation. Its effects on detection accuracy, model size, computational complexity, and inference speed are evaluated in Section 3.3.

2.3.3. Tomato Maturity-Aware Aggregation Fusion Module: TMAF

To strengthen the representation of maturity- and defect-related cues in the fused neck features, a tomato maturity-aware aggregation fusion module, termed TMAF, was designed. TMAF is embedded in the P3 and P4 feature fusion paths of the YOLOv8n neck, where relatively high-resolution features are retained for fine-grained maturity and defect detection. The original P5 path is preserved to limit the additional computational cost. As illustrated in Figure 4, TMAF comprises a lightweight texture aggregation branch and a TMCAttention submodule.

Figure 4. Structure of the TMAF module.

Let the input feature be $X$, with a size of $X \in R^{B \times C \times H \times W}$, where $B$, $C$, $H$, and $W$ denote the batch size, number of channels, feature map height, and feature map width, respectively. The input feature first passes through a $1 \times 1$ convolution for channel mapping. It is then split along the channel dimension into a retained branch and a texture branch:

$$
X_{1} , X_{2} = S p l i t \left(\right. C o n v_{1 \times 1} \left(\right. X \left.\right) \left.\right)
$$

(2)

where $X_{1}$ is the retained branch, which maintains the direct transmission of input information. $X_{2}$ is the texture branch, which passes through three LTMBlocks in sequence to obtain texture-enhanced features at different levels, denoted as $T_{1}$, $T_{2}$, and $T_{3}$. Each LTMBlock consists of a $1 \times 1$ convolution, a $3 \times 3$ depthwise convolution, a $5 \times 5$ depthwise convolution, and a $1 \times 1$ convolution. The depthwise convolutions are used to supplement local texture and edge details. The $1 \times 1$ convolutions are used for channel adjustment and feature fusion.

Then, the retained branch $X_{1}$, and the texture branch outputs $T_{1}$, $T_{2}$, and $T_{3}$ are concatenated along the channel dimension. A $1 \times 1$ convolution is then used to complete feature fusion and obtain the fused feature $F$:

$$
= C o n v_{1 \times 1} \left(\right. C o n c a t \left(\right. X_{1} , T_{1} , T_{2} , T_{3} \left.\right) \left.\right)
$$

(3)

To further adjust maturity-related feature responses, TMCAttention is applied after feature fusion. This structure contains three branches: channel attention, spatial attention, and residual connection. Channel attention is used to adjust the importance of different feature channels. It helps the model focus on color, texture, and defect responses related to maturity discrimination. Spatial attention is used to highlight fruit region responses and reduce interference from irrelevant background features. The outputs of the two attention branches are denoted as $F_{c}$ and $F_{s}$, respectively.

Finally, the outputs of the channel attention branch and the spatial attention branch are fused and added to the residual branch. The output feature of the TMAF module is obtained as follows:

$$
Y = F + C o n v_{1 \times 1} \left(\right. F_{c} + F_{s} \left.\righ