From the course: Learning XAI: Explainable Artificial Intelligence

Unlock this course with a free trial

Join today to access over 26,000 courses taught by industry experts.

HELM

HELM

From the course: Learning XAI: Explainable Artificial Intelligence

Start my 1-month free trial Buy for my team

HELM

“

- Imagine evaluating 30 different AI models, each tested on different scenarios using different metrics under different conditions. How could you possibly compare them fairly? This was the challenge that Stanford University tackled when they created HELM, the holistic evaluation of language models. Let's explore how this groundbreaking framework is transforming how we evaluate AI systems for bias and fairness. HELM measures seven critical metrics for every model it evaluates. One, accuracy. How often is the model correct? Two, calibration. How well does it know what it knows? Three, robustness. How stable are its outputs? Four, fairness. Does it treat all groups equally? Five, bias. Does it show systematic prejudices? Six, toxicity. Does it generate harmful content? And lastly, efficiency. How resource intensive is it? Before HELM, models were typically only evaluated on a fraction of these metrics, making it more difficult to understand their full impact. What makes HELM…

Contents