Mistral Small 3: Complete Benchmark Performance Guide

mistral small-3 2025-01-29 Apache 2.0

In-depth analysis of Mistral Small 3 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Versionsmall-3
Release Date2025-01-29
Context Window32768 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://docs.mistral.ai/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU76.9%2025-01-295-shotview
HUMANEVAL74.4pass@12025-01-29—view
GSM8K79.6%2025-01-290-shot CoTview
MATH39.2%2025-01-290-shot CoTview
BBH71.3%2025-01-293-shot CoTview
GPQA30.1%2025-01-290-shotview
IFEVAL76.9%2025-01-29prompt_strictview
ARC91.8%2025-01-29challengeview
MUSR46%2025-01-290-shotview
WINOGRANDE78.8%2025-01-290-shotview

Pricing

TierPriceCurrency
Input$0.2 / MtokUSD
Output$0.6 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2025-01-29

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Mistral Small 3

Model Genel Bakışı

Mistral Small 3 开源模型, 24B 参数, 32K 上下文, 性能接近 Mistral Large 2, Apache 2.0 可商用。

Temel Özellikler

SatıcıSürümYayın TarihiBağlam PenceresiGiriş ModlarıÇıkış ModlarıLisans
Mistralsmall-32025-01-2932KtexttextApache 2.0

Benchmark Performansı

BenchmarkPuanBirimNotlar
MMLU (Massive Multitask Language Understanding)76.9%5-shot
HumanEval74.4pass@1—
GSM8K (Grade School Math 8K)79.6%0-shot CoT
MATH39.2%0-shot CoT
BBH (BIG-Bench Hard)71.3%3-shot CoT
GPQA30.1%0-shot
IFEval76.9%prompt_strict
ARC91.8%challenge
MUSR46.0%0-shot
WinoGrande78.8%0-shot

Fiyatlandırma

GirişÇıkışÖnbellek OkumaÖnbellek Yazma
————

milyon token başına

Güçlü Yönler

  • 可靠的通用模型。

Zayıf Yönler

  • 闭源专有模型，不支持自托管。

Kullanım Senaryoları

  • 代码生成与调试

Referanslar