Multimodal Spatio-Temporal Domain Adaptation Untuk Klasifikasi Fase Fenologi Padi Pada Pertanian Vertikal

Abstract

Penelitian ini dilatarbelakangi oleh semakin terbatasnya lahan pertanian produktif untuk budidaya padi, yang disebabkan oleh alih fungsi lahan, urbanisasi, serta perubahan iklim. Kondisi ini mendorong kebutuhan akan inovasi sistem budidaya alternatif yang mampu menjamin keberlanjutan produksi pangan, salah satunya melalui pendekatan vertical farming. Meskipun sistem ini telah banyak diterapkan pada komoditas hortikultura, implementasinya pada tanaman padi masih menghadapi berbagai tantangan, khususnya dalam pemantauan fase pertumbuhan secara akurat akibat kompleksitas interaksi antara morfologi tanaman dan dinamika lingkungan mikro. Tantangan ini menjadi semakin penting pada pengembangan padi fungsional (functional rice) atau padi dengan karakteristik khusus (misalnya tinggi nutrisi atau adaptif terhadap kondisi tertentu), yang membutuhkan kontrol lingkungan secara presisi. Permasalahan ini menjadi semakin penting pada pengembangan padi fungsional yang membutuhkan pengendalian lingkungan secara presisi dan berbasis data. Penelitian ini bertujuan mengembangkan model klasifikasi fase pertumbuhan tanaman padi berbasis multimodal deep learning yang mampu mengintegrasikan data citra tanaman dan data sensor lingkungan, sekaligus memiliki kemampuan adaptasi terhadap perbedaan distribusi data antara lahan sawah konvensional dan lingkungan vertical farming. Model yang diusulkan menggunakan pendekatan Multimodal Spatio-Temporal Domain Adaptation (MST-DA) dengan menggabungkan fitur spasial dari citra tanaman menggunakan CNN dan fitur temporal dari data sensor menggunakan LSTM. Beberapa backbone CNN yang digunakan meliputi ResNet50, MobileNet, VGG19, dan Xception. Model melakukan klasifikasi terhadap delapan kelas fase fenologi tanaman padi, yaitu persemaian, vegetatif awal, anakan maksimum, inisiasi malai, bunting, pengisian gabah, pematangan, dan Pra-Panen. Proses klasifikasi memanfaatkan fitur spasial yang diekstraksi dari citra tanaman, meliputi karakteristik morfologi, warna daun, tekstur, bentuk tajuk, serta kerapatan vegetasi. Selain itu, model juga memanfaatkan fitur temporal yang berasal dari data sensor lingkungan, meliputi suhu udara, kelembapan udara, intensitas cahaya, kelembapan tanah, pH air, pH tanah yang direkam secara berkala selama proses pertumbuhan tanaman. Untuk meningkatkan kualitas integrasi multimodal, penelitian ini mengimplementasikan mekanisme Cross-Modal Attention yang memungkinkan model memberikan bobot perhatian adaptif terhadap fitur sensor yang relevan dengan fitur citra. Mekanisme ini terbukti mampu memperkuat hubungan antara perubahan morfologi tanaman dan variasi parameter lingkungan, sehingga menghasilkan representasi fitur yang lebih informatif dan diskriminatif. Selain itu, pendekatan MST-DA diterapkan melalui penggunaan Gradient Reversal Layer (GRL) dan Maximum Mean Discrepancy (MMD) untuk meminimalkan domain shift antara domain sumber dan target. Hasil eksperimen menunjukkan bahwa model MST-DA mampu mengatasi perbedaan distribusi data antara domain sumber dan domain target secara efektif. Pada konfigurasi optimal ?MMD = 0,1 dan a = 20, model mencapai accuracy sebesar 91,48%, F1-score sebesar 83,74%, dan Mean Intersection over Union (Mean IoU) sebesar 73,77%. Selama proses pelatihan, nilai loss menurun secara konsisten yang menunjukkan konvergensi model yang baik, sementara domain accuracy pada domain sumber dan domain target menurun hingga mendekati distribusi acak (˜0,10–0,15), mengindikasikan bahwa model berhasil mempelajari representasi fitur yang semakin domain-invariant. Hasil tersebut menunjukkan bahwa pendekatan MST-DA mampu meningkatkan kemampuan generalisasi model dalam melakukan klasifikasi fase fenologi padi pada lingkungan vertical farming. Temuan utama dari penelitian ini menunjukkan bahwa integrasi antara Cross-Modal Attention dan domain adaptation mampu meningkatkan kualitas representasi fitur serta memperkuat kemampuan model dalam menghadapi heterogenitas data pertanian lintas domain. Kebaruan penelitian ini terletak pada pengembangan arsitektur MST-DA yang mengintegrasikan pembelajaran spasial-temporal, mekanisme perhatian lintas modalitas, dan domain adaptation dalam satu kerangka terpadu untuk klasifikasi fase pertumbuhan padi berbasis IoT vertical farming. Dari sisi ilmiah, penelitian ini memperluas penerapan domain adaptation pada bidang pertanian cerdas, khususnya pada data multimodal yang bersifat dinamis dan heterogen. Dari sisi implementasi, model yang dikembangkan berpotensi diterapkan pada sistem smart farming berbasis IoT untuk mendukung pemantauan pertumbuhan tanaman secara otomatis, adaptif, dan berbasis data dalam lingkungan vertical farming.

This research was motivated by the increasing scarcity of productive agricultural land for rice cultivation due to land-use conversion, urbanization, and climate change. These challenges have created an urgent need for innovative alternative cultivation systems capable of ensuring sustainable food production, one of which is vertical farming. Although this cultivation system has been widely implemented for horticultural crops, its application to rice cultivation remains challenging, particularly in accurately monitoring rice growth stages because of the complex interactions between plant morphology and microenvironmental dynamics. These challenges become even more critical in the development of functional rice, which requires precise and data-driven environmental control throughout the cultivation process. This study aims to develop a multimodal deep learning model for rice growth stage classification by integrating plant image data and environmental sensor data while adapting to distribution discrepancies between conventional paddy fields and vertical farming environments. The proposed Multimodal Spatio-Temporal Domain Adaptation (MST-DA) framework combines spatial feature extraction from plant images using Convolutional Neural Networks (CNNs) with temporal feature learning from environmental sensor data using Long Short-Term Memory (LSTM) networks. Four CNN backbone architectures, namely ResNet50, MobileNet, VGG19, and Xception, were employed to evaluate their feature extraction capabilities. The proposed model classifies rice plants into eight phenological stages: seedling, early vegetative, maximum tillering, panicle initiation, booting, grain filling, ripening, and pre-harvest. The classification process utilizes spatial features extracted from plant images, including plant morphology, leaf color, texture, canopy structure, and vegetation density. In addition, temporal features are extracted from environmental sensor data, including air temperature, relative humidity, light intensity, soil moisture, water pH, and soil pH, which are continuously recorded throughout the plant growth cycle. To enhance multimodal feature integration, this study incorporates a Cross-Modal Attention mechanism that enables the model to assign adaptive attention weights to environmental sensor features according to their relevance to visual features extracted from plant images. This mechanism effectively strengthens the relationship between changes in plant morphology and variations in environmental conditions, thereby producing more informative and discriminative feature representations. Furthermore, the proposed MST-DA framework integrates a Gradient Reversal Layer (GRL) and Maximum Mean Discrepancy (MMD) to minimize domain shift between the source and target domains. Experimental results demonstrate that the proposed MST-DA model effectively addresses differences in data distribution between the source and target domains. Under the optimal parameter configuration (?MMD = 0.1 and a = 20), the model achieved an accuracy of 91.48%, an F1-score of 83.74%, and a Mean Intersection over Union (Mean IoU) of 73.77%. During the training process, the loss consistently decreased, indicating stable model convergence, while the domain classification accuracy in both the source and target domains decreased to values close to a random distribution (˜0,10–0,15), demonstrating that the proposed framework successfully learned increasingly domain-invariant feature representations. These results indicate that the MST-DA approach effectively improves the model's generalization capability for rice phenological stage classification in vertical farming environments. The findings of this study demonstrate that the integration of Cross-Modal Attention and domain adaptation enhances the quality of feature representations and improves the model's robustness in handling heterogeneous agricultural data across different domains. The primary novelty of this research lies in the development of a unified MST-DA framework that integrates spatiotemporal learning, cross-modal attention, and domain adaptation into a single architecture for IoT-based rice phenological stage classification in vertical farming. From a scientific perspective, this study extends the application of domain adaptation in smart agriculture, particularly for dynamic and heterogeneous multimodal data. From a practical perspective, the proposed model has strong potential for implementation in IoT-enabled smart farming systems to support automated, adaptive, and data-driven monitoring of rice growth in vertical farming environments.