Batch Upload of Crop Dataset to Qdrant for AI Analysis

This workflow automates the process of importing a dataset of agricultural crop images from Google Cloud Storage, generating vector embeddings using Voyage AI API, and uploading the data in batches to a Qdrant vector database. It begins with a manual trigger to start the process. The workflow checks if the designated Qdrant collection exists, creates it if necessary, and sets up a payload index on the ‘crop_name’ field for optimized searches. It fetches the image URLs from Google Cloud Storage, constructs public links, and groups images into batches while generating UUIDs for each point. It then creates vector embeddings for each batch via Voyage AI API and uploads these to the Qdrant collection. Additional nodes prepare and filter the data, such as excluding images of ‘tomato’ crops for anomaly detection testing, and organize metadata payloads. This setup is useful for machine learning tasks like anomaly detection and KNN classification on large image datasets, enabling efficient nearest-neighbor search and clustering in Qdrant.