Automated Testing and Analysis of Local LLM Models

This workflow streamlines the process of testing and analyzing multiple Local Language Models (LLMs) with LM Studio, integrating data collection, response analysis, and result tracking. It is ideal for developers or data scientists aiming to evaluate model performance based on readability, response quality, and response time.

The process begins with setting up LM Studio and configuring the desired models, which can be dynamically fetched via a local API request. A webhook listens for chat inputs, triggering the workflow when a message is received. The workflow captures start and end timestamps to measure response times, then sends prompts to different models based on extracted IDs.

Responses from the models are analyzed using custom JavaScript to calculate metrics such as word count, sentence count, average sentence length, readability score, and average word length. These metrics are then optionally logged into a Google Sheet for easy review and comparison of different models and prompts.

The workflow is useful for testing model responses under different configurations, fine-tuning response parameters like temperature and top P, and tracking performance metrics over time, all in an automated manner.