Automated Dataset Upload and Embedding to Qdrant

This workflow automates the process of uploading an image dataset to Qdrant for advanced image analysis tasks such as anomaly detection and KNN classification. It starts with a manual trigger, then checks for existing Qdrant collection, and creates a new collection if necessary. The workflow fetches images from Google Cloud Storage, constructs public links, and groups images into batches. It uses Voyage AI’s multimodal API to generate embeddings for each image, which are then uploaded in batches to Qdrant. Throughout, it includes setup steps like creating payload indices and filtering images to test models without certain classes like ‘tomato’. This setup is ideal for machine learning projects involving image similarity, anomaly detection, or classification, especially when managing large datasets that require efficient processing and storage.