Cloud Computing

Amazon SageMaker Make clear makes it simpler to judge and choose basis fashions (preview)

Spread the love


Voiced by Polly

I’m comfortable to share that Amazon SageMaker Make clear now helps basis mannequin (FM) analysis (preview). As a knowledge scientist or machine studying (ML) engineer, now you can use SageMaker Make clear to judge, examine, and choose FMs in minutes based mostly on metrics comparable to accuracy, robustness, creativity, factual information, bias, and toxicity. This new functionality provides to SageMaker Make clear’s current potential to detect bias in ML information and fashions and clarify mannequin predictions.

The brand new functionality gives each computerized and human-in-the-loop evaluations for giant language fashions (LLMs) anyplace, together with LLMs obtainable in SageMaker JumpStart, in addition to fashions educated and hosted exterior of AWS. This removes the heavy lifting of discovering the appropriate mannequin analysis instruments and integrating them into your improvement setting. It additionally simplifies the complexity of making an attempt to undertake educational benchmarks to your generative synthetic intelligence (AI) use case.

Consider FMs with SageMaker Make clear
With SageMaker Make clear, you now have a single place to judge and examine any LLM based mostly on predefined standards throughout mannequin choice and all through the mannequin customization workflow. Along with computerized analysis, you may also use the human-in-the-loop capabilities to arrange human critiques for extra subjective standards, comparable to helpfulness, inventive intent, and magnificence, by utilizing your individual workforce or managed workforce from SageMaker Floor Fact.

To get began with mannequin evaluations, you should utilize curated immediate datasets which can be purpose-built for frequent LLM duties, together with open-ended textual content technology, textual content summarization, query answering (Q&A), and classification. You can even prolong the mannequin analysis with your individual customized immediate datasets and metrics to your particular use case. Human-in-the-loop evaluations can be utilized for any job and analysis metric. After every analysis job, you obtain an analysis report that summarizes the leads to pure language and consists of visualizations and examples. You’ll be able to obtain all metrics and experiences and in addition combine mannequin evaluations into SageMaker MLOps workflows.

In SageMaker Studio, you’ll find Mannequin analysis below Jobs within the left menu. You can even choose Consider immediately from the mannequin particulars web page of any LLM in SageMaker JumpStart.

Evaluate foundation models with Amazon SageMaker Clarify

Choose Consider a mannequin to arrange the analysis job. The UI wizard will information you thru the choice of computerized or human analysis, mannequin(s), related duties, metrics, immediate datasets, and evaluate groups.

Evaluate foundation models with Amazon SageMaker Clarify

As soon as the mannequin analysis job is full, you possibly can view the leads to the analysis report.

Evaluate foundation models with Amazon SageMaker Clarify

Along with the UI, you may also begin with instance Jupyter notebooks that stroll you thru step-by-step directions on the way to programmatically run mannequin analysis in SageMaker.

Consider fashions anyplace with the FMEval open supply library
To run mannequin analysis anyplace, together with fashions educated and hosted exterior of AWS, use the FMEval open supply library. The next instance demonstrates the way to use the library to judge a customized mannequin by extending the ModelRunner class.

For this demo, I select GPT-2 from the Hugging Face mannequin hub and outline a customized HFModelConfig and HuggingFaceCausalLLMModelRunner class that works with causal decoder-only fashions from the Hugging Face mannequin hub comparable to GPT-2. The instance can also be obtainable within the FMEval GitHub repo.

!pip set up fmeval

# ModelRunners invoke FMs
from amazon_fmeval.model_runners.model_runner import ModelRunner

# Further imports for customized mannequin
import warnings
from dataclasses import dataclass
from typing import Tuple, Non-compulsory
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

@dataclass
class HFModelConfig:
    model_name: str
    max_new_tokens: int
    normalize_probabilities: bool = False
    seed: int = 0
    remove_prompt_from_generated_text: bool = True

class HuggingFaceCausalLLMModelRunner(ModelRunner):
    def __init__(self, model_config: HFModelConfig):
        self.config = model_config
        self.mannequin = AutoModelForCausalLM.from_pretrained(self.config.model_name)
        self.tokenizer = AutoTokenizer.from_pretrained(self.config.model_name)

    def predict(self, immediate: str) -> Tuple[Optional[str], Non-compulsory[float]]:
        input_ids = self.tokenizer(immediate, return_tensors="pt").to(self.mannequin.gadget)
        generations = self.mannequin.generate(
            **input_ids,
            max_new_tokens=self.config.max_new_tokens,
            pad_token_id=self.tokenizer.eos_token_id,
        )
        generation_contains_input = (
            input_ids["input_ids"][0] == generations[0][: input_ids["input_ids"].form[1]]
        ).all()
        if self.config.remove_prompt_from_generated_text and never generation_contains_input:
            warnings.warn(
                "Your mannequin doesn't return the immediate as a part of its generations. "
                "`remove_prompt_from_generated_text` does nothing."
            )
        if self.config.remove_prompt_from_generated_text and generation_contains_input:
            output = self.tokenizer.batch_decode(generations[:, input_ids["input_ids"].form[1] :])[0]
        else:
            output = self.tokenizer.batch_decode(generations, skip_special_tokens=True)[0]

        with torch.inference_mode():
            input_ids = self.tokenizer(self.tokenizer.bos_token + immediate, return_tensors="pt")["input_ids"]
            model_output = self.mannequin(input_ids, labels=input_ids)
            chance = -model_output[0].merchandise()

        return output, chance

Subsequent, create an occasion of HFModelConfig and HuggingFaceCausalLLMModelRunner with the mannequin data.

hf_config = HFModelConfig(model_name="gpt2", max_new_tokens=32)
mannequin = HuggingFaceCausalLLMModelRunner(model_config=hf_config)

Then, choose and configure the analysis algorithm.

# Let's consider the FM for FactualKnowledge
from amazon_fmeval.fmeval import get_eval_algorithm
from amazon_fmeval.eval_algorithms.factual_knowledge import FactualKnowledgeConfig

eval_algorithm_config = FactualKnowledgeConfig("<OR>")
eval_algorithm = get_eval_algorithm("factual_knowledge", eval_algorithm_config)

Let’s first take a look at with one pattern. The analysis rating is the proportion of factually right responses.

model_output = mannequin.predict("London is the capital of")[0]
print(model_output)

eval_algo.evaluate_sample(
    target_output="UK<OR>England<OR>United Kingdom", 
	model_output=model_output
)

the UK, and the UK is the biggest producer of meals on the planet.

The UK is the world's largest producer of meals on the planet.
[EvalScore(name="factual_knowledge", value=1)]

Though it’s not an ideal response, it consists of “UK.”

Subsequent, you possibly can consider the FM utilizing built-in datasets or outline your customized dataset. If you wish to use a customized analysis dataset, create an occasion of DataConfig:

config = DataConfig(
    dataset_name="my_custom_dataset",
    dataset_uri="dataset.jsonl",
    dataset_mime_type=MIME_TYPE_JSONLINES,
    model_input_location="query",
    target_output_location="reply",
)

eval_output = eval_algorithm.consider(
    mannequin=mannequin, 
    dataset_config=config, 
    prompt_template="$characteristic", #$characteristic is changed by the enter worth within the dataset 
    save=True
)

The analysis outcomes will return a mixed analysis rating throughout the dataset and detailed outcomes for every mannequin enter saved in an area output path.

Be a part of the preview
FM analysis with Amazon SageMaker Make clear is out there immediately in public preview in AWS Areas US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Eire). The FMEval open supply library is out there on GitHub. To study extra, go to Amazon SageMaker Make clear.

Get began
Log in to the AWS Administration Console and begin evaluating your FMs with SageMaker Make clear immediately!

— Antje

Leave a Reply

Your email address will not be published. Required fields are marked *