A deep learning perception framework for farm digital twins: weed species classification, semantic segmentation, and gradient-based visual interpretability

Abstract

The emergence of Farm Digital Twins (Farm-DT) as a transformative paradigm in smart agriculture demands robust, real-time perception modules capable of continuous plant-level monitoring, predictive analytics, and automated decision support. A critical bottleneck in operationalising Farm-DTs is the absence of interpretable, species-level weed identification engines that can feed spatially precise weed-distribution data into the virtual farm replica for simulation, yield forecasting, and optimized herbicide scheduling. This paper presents a tri-component deep learning framework designed explicitly as a perception and interpretability layer for Farm Digital Twin architectures, evaluated on a five-species balanced subset of the Moving Fields Weed Dataset (MFWD)—a publicly available benchmark of 94K high-resolution images of 28 weed species. Eleven ImageNet pre-trained architectures spanning convolutional neural networks (CNNs) and vision transformers were benchmarked under a unified stratified 80/10/10 holdout protocol. Swin Transformer v1 achieved the highest test accuracy of 97.3% [F 1 = 0.964; 95% CI: (95.9, 98.4)], and EfficientNetV2-S reached 95.5% [F 1 =0.956; 95% CI: (93.8, 96.9)]; pairwise McNemar tests confirm these advantages are statistically significant (p< 0.01, Bonferroni-corrected). Gradient-weighted Class Activation Mapping (Grad-CAM) applied to the top CNN models confirmed that 94.3% of high-activation pixels overlap with ground-truth foliage annotations, providing quantitative validation that classification decisions are driven by botanically meaningful morphological features. The SegFormer-B3 semantic segmentation module achieved mIoU = 0.8961, enabling precise pixel-level weed delineation that directly populates the DT spatial model for variable-rate herbicide application and robotic weeding simulation. Together, these three components map directly onto the Farm-DT architecture: the classifier feeds the species inventory, the segmentation module populates the spatial weed model, and Grad-CAM provides the trust layer required for agronomist acceptance of DT-automated decisions—collectively advancing real-time monitoring and decision-support for Digital Twin-driven urban and peri-urban agriculture.

1 Introduction

The convergence of accelerated urbanization, climate change, and food security pressures has elevated urban agriculture (UA) from a marginal practice to a cornerstone of sustainable urban development. Within this landscape, Digital Twin (DT) technology has emerged as a transformative enabler: by creating a dynamic, continuously updated virtual replica of a physical farm system, DTs allow stakeholders to simulate, forecast, and optimize the full food production lifecycle within a risk-free environment. A fundamental prerequisite for any functional Farm Digital Twin is a high-fidelity, real-time perception layer that can translate physical field conditions—plant health, species composition, pest pressure—into structured data streams feeding the virtual model (; ).

Weed management represents one of the most pressing challenges in this context. Weeds compete directly with crops for light, water, and nutrients, and species-level accurate weed identification is the essential first step toward effective integrated pest management within a DT framework (). Precise species identification permits the DT to simulate the consequences of specific herbicide interventions, predict resistance development (), and schedule variable-rate applications—all without physical trial and error. Yet current approaches suffer from two critical limitations that prevent their integration into operational Farm-DT architectures: (1) most models lack the interpretability required for agronomist trust and regulatory acceptance; and (2) the majority focus on classification alone, without providing the pixel-level spatial data that the DT spatial model requires for site-specific robotic intervention ().

The emergence of deep learning, particularly the use of CNNs and, to some extent, transformer inspired architectures, has significantly changed automated plant phenotyping and pest detection. These capabilities make it possible to perform feature extraction hierarchically directly from the high-dimensional image data (; ). CNNs excel at capturing fine-grained spatial structures and complex plant patterns through locally receptive fields. On the other hand, vision transformers apply self-attention to understand long-range spatial dependencies and global contextual relationships, thereby surpassing purely convolutional representations in capturing global context (). Yet, even with those changes, two issues continue to limit large-scale adoption: (1) models typically lack sufficient interpretability for agronomists to verify that classification decisions are grounded in actual plant morphology rather than data artifacts; and (2) Most of the methods focus on either classification or segmentation and do not provide the level of pixel-wise spatial detail required by robotic weeding devices and variable-rate sprayer systems.

To address these challenges, we develop a tri-component deep learning framework with three distinct objectives. The first goal is to develop a multi-class species classifier capable of distinguishing morphologically similar weed species under varied acquisition conditions The second goal is to use explainable artificial intelligence (XAI) techniques to validate model decisions and check whether the features that the model has learned correspond to the botanical features of the plants. The third goal is to accurately generate pixel-level weed delineations by semantic segmentation, which in turn allows for site-specific robotic intervention. To achieve these goals, we systematically evaluate eleven ImageNet pre-trained architectures: GoogLeNet, DenseNet-121, AlexNet, ConvNeXt-Tiny, ShuffleNetV2, SqueezeNet, Inceptionv3, ResNet-50, EfficientNetV2-S, MobileNetV2, and Swin Transformer v1. We chose these models to cover a wide variety of architectural paradigms, such as residual networks, lightweight mobile-optimized networks, attention-based transformers, and inception modules with multi-scale feature extraction designs.

Experiments were conducted using the publicly available Moving Fields Weed Dataset (MFWD) (), which consists of 94, 321 high-resolution RGB images covering 28 weed species commonly appearing in German maize and sorghum environments. These images were captured in a fully automated, high-throughput indoor phenotyping facility with controlled, uniform illumination. A stratified sampling process was carried out leading to a five-species balanced subset (PULDY, match, cheal, galap, SOLNI) with a total of 7, 954 images to reduce classification bias and ensure fair class representation. Class-discriminative heatmaps were created through Gradient-weighted Class Activation Mapping (Grad-CAM) () applied to the top-performing CNNs (EfficientNetV2-S, Inception v3), and quantitative analysis confirmed that 94.3% of high-activation pixels correspond to ground-truth foliage regions. The SegFormer-B3 model (), a transformer-based encoder–decoder architecture that paradoxically melds hierarchically multi-scale feature extraction with a very simple mlp decoder, was trained for semantic segmentation on 7,954 semi-automatically annotated binary masks, initially generated through hsv color-space thresholding and subsequently verified by domain experts. A mean Intersection over Union (mIoU) of 0.8961 was achieved.

The main contributions of this work are as follows:

• Systematic benchmark with statistical validation. Eleven pre-trained architectures spanning CNNs and vision transformers are evaluated under a unified 80/10/10 stratified holdout protocol with McNemar pairwise significance tests (Bonferroni-corrected), paired t-tests across five random seeds, and 95% confidence intervals. Swin Transformer v1 achieved the highest test accuracy of 97.3% and EfficientNetV2-S achieved 95.5%, confirmed as statistically superior to all CNN baselines (p< 0.01).

• Quantitative XAI validation. Grad-CAM () applied to EfficientNetV2-S and Inception v3 confirms that 94.3% of high-activation pixels overlap with ground-truth foliage annotations, providing evidence that classification decisions are driven by botanical morphological features rather than background artifacts or spurious correlations.

• Integrated segmentation pipeline. A semi-automated HSV-thresholding and expert-review annotation pipeline produced 7,954 pixel-wise binary masks. SegFormer-B3 () trained on these masks obtains mIoU = 0.8961 (precision = 0.8139; recall = 0.8204) which allows reliable weed-patch delineation for robotic weeding and targeted herbicide application.

• Ablation study and efficiency analysis. Three controlled ablation experiments have been used to measure the independent effects of ImageNet transfer learning (+15%–17% accuracy), data augmentation technique (+4.2% for Swin v1), and segmentation loss mixture (+0.035 mIoU), together with a study of accuracy-efficiency balance to help decide on models that would work within limited resource environments.

In this work, the classification module serves as the DT species inventory feed, the segmentation module populates the DT spatial model with pixel-level weed maps, and the Grad-CAM module provides the interpretability layer required for agronomist trust in DT-automated decisions.

The remaining sections are organized as follows. In Section 2, we summarize the state-of-the-art deep learning-based methods for weed detection and semantic segmentation. Sections 3, 4 provide a detailed methodology that includes dataset preparation, model architectures, training configurations, and evaluation protocols. Section 5 compiles a full set of experiments with cross-comparison of different methods, statistical significance, and ablation analysis. Section 6 discusses the limitations of the proposed framework. Section 7 summarizes the findings and outlines future research directions for interpretable precision agriculture systems.

2 Literature review

Combining computer vision with deep learning for agricultural weed management is rapidly becoming a major topic that is not only enhancing sustainable farming practices but also limiting the inappropriate use of herbicides. This section surveys image-based weed classification, transformer architectures, semantic segmentation frameworks, and explainable AI techniques, followed by a structured gap analysis (Table 1).

Table 1

Study	Dataset	Cls.	Best Acc.	C + T	CV	Aug	XAI(Q)	Seg	Stat	Abl	CI	DT-Ready
	Sugarcane	7	96.6%	×	×	✓	×	×	×	×	×	×
	Mixed crop	2	87.2%	×	×	✓	×	×	×	×	×	×
	Custom weed	10	98.6%	°	✓	✓	°	×	×	×	×	°
	MFWD-UAV	2	89% F 1	×	×	✓	×	✓	×	×	×	×
	Sorghum	5	99.1%	×	✓	✓	°	×	×	°	×	×
	Grape leaf	4	99.4%	×	×	✓	°	×	×	×	×	°
	Crop/weed	6	96.1%	°	×	✓	×	×	×	×	×	×
	Custom	3	94.5%	×	×	✓	×	×	×	×	×	×
	Potato	2	92.4%	×	×	✓	×	✓	×	×	×	×
	Straw./pea	2	96.8%	×	×	✓	×	×	×	×	×	×
Proposed	MFWD (	5+seg	97.3%	✓	°	✓	✓	✓	✓	✓	✓	✓

Structured comparison of representative weed detection studies across nine evaluation dimensions.

C + T, both CNN and Transformer families evaluated; CV, cross-validation; Aug, data augmentation; XAI(Q), quantitative XAI evaluation; Seg, semantic segmentation component; Stat, statistical significance tests reported; Abl, ablation study included; CI, confidence intervals reported. ✓ = present; × = absent; ° = partial. Bold values denote results for the proposed method, shown for comparison against the reviewed literature.

2.1 CNN-based weed classification

CNNs have proven to be more effective than traditional ML techniques by utilizing hand-crafted features to identify weeds (; ). To identify sugarcane weeds, developed a deep learning framework using 5,660 images. Six architectures—DarkNet-53, AlexNet, GoogLeNet, Inception v3, ResNet-50, and Xception—were evaluated across two batch sizes and three training durations at a fixed learning rate of 0.001. DarkNet-53 achieved the best performance with an F 1-score of 0.99 and 96.6% test accuracy at batch size 16 and 20 epochs, classifying 98% of weed cases with high confidence and thereby reducing the need for precision herbicide applicators.

examined transfer learning strategies for weed–crop discrimination using MobileNet variants (V1–V3) and DenseNet variants (121/169/201). MobileNetV3 achieved higher classification accuracy than all DenseNet variants with substantially lower computational cost, demonstrating the practical advantage of lightweight architectures in mixed-crop scenarios.

2.2 Vision transformer architectures

Vision transformers utilize multi-head self-attention to obtain spatial relationships over large distances and global contextual relationships that are not available in purely convolutional representations (). used ImageNet pre-trained weights and AutoAugment data augmentation for fine-tuning Swin Transformer v2 resulting in a 98.61% top-1 classification accuracy on a multi-species weed dataset. Grad-CAM++ visualization showed class-discriminative regions appropriate for robotic intervention guidance, illustrating the transformer's validity for fine-grained botanical classification. evaluated various deep learning architectures including transformer variants for crop and weed identification and found that attention-based models surpass CNN baselines on morphologically complex species pairs.

2.3 Semantic segmentation for weed mapping

Pixel-wise segmentation enables exact spatial localization and is essential for site-specific weed management since robotic weeding and variable-rate herbicide application require it (). built a UNet-like encoder–decoder with a ResNet-34 backbone for classifying weeds at an early growth stage in uav -sorghum images. Multi-scale feature aggregation and skip connections allowed the model to handle occlusion and motion blur, achieving an 89% F 1-score for intra-row weed detection. tried to solve the problem of weed detection in heavily occluded potato field settings and demonstrated that deep learning remains robust under dense canopy conditions. introduced SegFormer, which couples a hierarchical Mix Transformer (mit) encoder with a lightweight all-mlp decoder, achieving competitive segmentation without positional encodings across standard benchmarks.

2.4 Explainable AI in agricultural systems

Interpretability is essential for building agronomist confidence and regulatory acceptance, as it allows practitioners to verify that model decisions reflect genuine plant morphology rather than dataset artifacts (). introduced Gradient-weighted Class Activation Mapping (Grad-CAM), which produces class-discriminative localizationmaps by backpropagating class scores through the final convolutional layer without modifying the network architecture. Grad-CAM enhances weakly supervised localization, improves adversarial robustness, and provides spatial explanations for all stakeholders.

proposed an interpretable deep learning method for sorghum weed classification based on a specially designed DenseNet-169 supplemented by Dimensional Convolutional Feature Fusion (dcff) and Deformable Multi-Kernel Channel Fusion (dmkcf) modules for extracting features at different scales. Grad-CAM and Local Interpretable Model-agnostic Explanations (lime) were used for the interpretability at the global and instance levels respectively. The method achieved 99.07% accuracy and 100% precision for broad-leaf weed detection.

Karim et al. employed MobileNetV3-Large with custom dropout-dense layers for real-time grape leaf disease detection on the NVIDIA Jetson Nano edge platform. The model achieved 99.66% training accuracy and 99.42% test accuracy. Grad-CAM was used to localize diseased regions, enabling automated recognition of both real-time and stored images through a custom graphical interface.

2.5 Digital twins in agriculture: sensing requirements

Farm Digital Twins require a real-time perception layer that converts field imagery into structured data streams for simulation and decision support. Deep learning-based weed detection fulfils this role by providing species inventory, spatial weed maps, and confidence scores that the DT simulation engine consumes directly. The absence of integrated, interpretable, and spatially precise weed detection frameworks represents a key gap in current Farm-DT implementations, which this study addresses.

2.6 Structured comparison and research gaps

Table 1 shows a detailed comparison, covering nine dimensions, of major weed detection studies. There are still four significant issues that the existing literature has not addressed.

• Most of the current research focuses on assessing traditional CNN families (such as ResNet, VGG, Inception) and in many cases do not perform systematic comparisons between these and recent transformer-based models (Swin, EfficientNetV2, ConvNeXt) on the same benchmark under exactly the same experimental conditions.

• Performance evaluations are limited to scalar measures (accuracy, F 1-score) without any quantitative analysis that would verify that classification decisions actually rely on biologically significant characteristics instead of background correlations or dataset artifacts.

• Classification and segmentation are handled separately; no end-to-end pipelines incorporating both modalities for full weed management decision support.

• Most studies employ laboratory-acquired or synthetically augmented datasets that fail to capture the phenotypic diversity and imaging variability present in operational agricultural environments.

This study addresses each of the four research gaps identified in the literature, as follows: (1) A thorough performance comparison of eleven CNN and transformer-based models with McNemar significance tests, paired t-tests, and 95% confidence intervals; (2) Quantitative Grad-CAM results show that 94.3% of the top-activated pixels are indeed ground-truth foliage areas; (3) an integrated classification-segmentation model capable of providing species identification together with spatial localization at the pixel level; (4) the highly realistic MFWD dataset collected in indoor phenotyping laboratory conditions (); and (5) a three-experiment ablation study that quantified the independent contributions of transfer learning, data augmentation, and loss function design.

3 Materials

3.1 Moving fields weed dataset (MFWD)

3.1.1 Dataset characterization

All experiments use the publicly available Moving Fields Weed Dataset (MFWD) (), comprising 94,321 high-resolution rgb images of 28 weed species common in German maize and sorghum production systems (DOI: 10.14459/2023md1717366). Images were acquired in a fully automated, high-throughput indoor phenotyping facility—not under natural field conditions— using a conveyor-mounted imaging platform that photographs individual microplot trays (40 × 30 × 22 cm) under controlled, uniform illumination at high temporal resolution. This setup tracks more than 5,000 individual plants across all developmental stages, from cotyledon emergence to reproductive phases, providin