![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/](\"https:\/\/www.enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Evaluating-large-language-models.jpg\")
<\/p>\r\n<\/p>\r\n
As Artificial Intelligence (AI) becomes more useful in many areas, large language models like GPT-4, Claude-3, Llama-2, Gemini, Falcon, Mistral and others have become central to tasks like writing assistance, information retrieval, customer support, and problem-solving.<\/p>\r\n\r\n\r\n\r\n
Think of evaluation metrics<\/mark><\/strong> as a report card for language models. Just as exams assess a student\u2019s grasp of a subject, metrics measure a model\u2019s understanding and proficiency in language tasks. For example, we may wonder:<\/p>\r\n\r\n\r\n\r\n Evaluation metrics help us answer these questions, offering insights into a model’s quality, accuracy, and reliability. They help us determine whether the model\u2019s output aligns with human standards or is just the result of random guesses. Automatic metrics are computed directly from the model\u2019s output. They provide a quick and standardized way to measure a model\u2019s performance, making them highly useful for large-scale evaluation.<\/p>\r\n\r\n\r\n\r\n Confidence level metrics help determine a model’s reliability in its predictions, identifying whether it\u2019s \u201csure\u201d about its answers and how accurate that certainty is.<\/p>\r\n\r\n\r\n\r\n Beyond numbers, qualitative metrics focus on ethical and functional aspects like fairness and robustness.<\/p>\r\n\r\n\r\n\r\n Human judgment offers an irreplaceable layer of insight, assessing aspects like fluency, coherence, and relevance. Although more resource-intensive, human evaluation often uncovers nuances automated metrics may miss.<\/p>\r\n\r\n\r\n\r\n Recent research explores using advanced language models, like GPT-4, to judge outputs from other models. By carefully crafting prompts, GPT-4 can evaluate coherence, relevance, and overall quality, providing sophisticated feedback similar to human judgment. For example, a prompt might ask GPT-4 to rate the coherence of a generated summary on a 1\u20135 scale based on whether it flows logically from sentence to sentence. When comparing different models, statistical tests help establish if differences in performance are genuine or due to chance.<\/p>\r\n\r\n\r\n\r\n Additional tests, like linguistic analysis, provide further insights into a model\u2019s language capabilities:<\/p>\r\n\r\n\r\n\r\n\r\n
<\/p>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Evaluation-Metrics-and-Methods.png\")
<\/figure>\r\n\r\n\r\n\r\n\r\n
1. Automatic Evaluation Metrics<\/strong><\/h3>\r\n\r\n\r\n\r\n\r\n
The Exact Match score calculates the percentage of generated texts that precisely match reference texts, measuring word-for-word correctness. For example, if a model generates 100 sentences and 20 are identical to reference sentences, the EM score is 20%
<\/li>\r\n\r\n\r\n\r\n
The F1 Score considers both precision and recall at the token level, balancing the proportion of correct words in the generated text:
Precision<\/strong>: Proportion of correctly generated tokens.
Recall<\/strong>: Proportion of reference tokens captured in the output.
For instance, if the model generates the sentence “The quick brown fox jumps over the lazy dog,” and the reference sentence is “A quick brown fox jumps over the lazy dog,” precision and recall yield an F1 score of about 0.778.
<\/li>\r\n\r\n\r\n\r\n
BLEU is commonly used for evaluating machine translation. It calculates n-gram precision (matching sequences of words) and applies a brevity penalty to prevent favoring shorter outputs. BLEU scores closer to 1 indicate higher accuracy.
<\/li>\r\n\r\n\r\n\r\n
ROUGE is especially valuable for summarization tasks. It measures the overlap of n- grams between generated and reference texts. Variants include:
ROUGE-N: Recall of specific n-grams.
ROUGE-L: Longest common subsequence, which helps capture sentence structure.
ROUGE-S: Skip-bigram co-occurrence, focusing on sequences with gaps.
<\/li>\r\n\r\n\r\n\r\n
METEOR evaluates machine translation and text generation by considering exact matches, stemming, synonyms, and paraphrases. It balances precision and recall, penalizing fragmented phrases and preferring longer, coherent matches.
<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nAutomatic](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Automatic-Evaluation-Metrics-Fine-Tuing-LLM.png\")
<\/figure>\r\n\r\n\r\n\r\n
2. Confidence Level Metrics<\/h3>\r\n\r\n\r\n\r\n\r\n
ECE calculates the difference between a model\u2019s confidence level and actual accuracy. By partitioning predictions into bins, it measures how well confidence aligns with correctness across different ranges.
<\/li>\r\n\r\n\r\n\r\n
ECE calculates the difference between a model\u2019s confidence level and actual accuracy. By partitioning predictions into bins, it measures how well confidence aligns with correctness across different ranges.
<\/li>\r\n<\/ul>\r\n<\/div>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nConfidence](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Confidence-Level-Metrics-Fine-tuning-LLM.png\")
<\/figure>\r\n\r\n\r\n\r\n\r\n
3. Qualitative Metrics<\/h3>\r\n\r\n\r\n\r\n\r\n
Fairness is critical to prevent biases. Metrics for counterfactual fairness and equalized odds are used to identify any demographic biases, ensuring the model treats all groups equitably.
<\/li>\r\n\r\n\r\n\r\n
Robustness measures a model\u2019s ability to handle variations in data without a decline in performance. It tests model responses to perturbed or adversarial inputs, revealing its resilience under real-world conditions.
<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nQualitative](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Qualitative-Metrics-Fine-Tuning-LLM.png\")
<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n\r\n
4. Human Evaluation<\/h2>\r\n\r\n\r\n\r\n\r\n
This method rates the model’s output on qualities like fluency and relevance, usually on a scale from 1 to 5, capturing human perspectives on output quality.
<\/li>\r\n\r\n\r\n\r\n
Here, outputs from different models are compared directly by judges, who select the best result based on subjective criteria.
<\/li>\r\n\r\n\r\n\r\n
In A\/B testing, users interact with different versions of the model in real scenarios and provide feedback on their experience, which helps validate the model\u2019s real- world utility.
<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nMethods](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Methods-Human-Evaluation-Fine-Tuning-LLM-Enkefalos.png\")
<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n\r\n
5. Leveraging LLM as an Evaluation Tool<\/h3>\r\n\r\n\r\n\r\n
<\/p>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nLeveraging](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Leveraging-LLM-as-an-Evaluation-Tool-Fine-tuning-LLM-Enkefalos.png\")
<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n\r\n
6. Statistical Significance Tests<\/h3>\r\n\r\n\r\n\r\n\r\n
McNemar\u2019s test is useful for binary classification tasks, identifying significant differences between two models by examining cases where their predictions disagree.
<\/li>\r\n\r\n\r\n\r\n
For continuous metrics like BLEU, the Wilcoxon Signed-Rank Test evaluates paired differences, helping to compare models based on ranked scores.
<\/li>\r\n\r\n\r\n\r\n
Bootstrapping involves sampling from the test set with replacement to generate multiple subsamples. This technique provides confidence intervals and helps determine if a model’s performance is robust.
<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nStatistical](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Statistical-Significance-Tests-Fine-tuning-LLM-Enkefalos.png\")
<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n\r\n
7. Other Evaluation Methods<\/h2>\r\n\r\n\r\n\r\n\r\n
Tools evaluate grammaticality and coherence to assess if sentences make logical sense and maintain structural flow, as seen in the Entity Grid or Discourse Coherence Model.
<\/li>\r\n\r\n\r\n\r\n
Diversity metrics, such as Self-BLEU or Distinct-N, ensure the model doesn\u2019t simply memorize training data, encouraging varied and creative output.
<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nOther](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Other-Evaluation-Methods-Fine-Tuning-LLM-Enkefalos.png\")
<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n
![\"https:\/\/enkefalos.com\/blog\/blog\/large-language-models\/evaluating-fine-tuned-large-language\/\r\nContinuous](\"https:\/\/enkefalos.com\/blog\/wp-content\/uploads\/2024\/11\/Continuous-Evaluation-Cycle-fine-tuning-LLM-Enkefalos-1.png\")
<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n