Llama 3.1 8B: Complete Benchmark Performance Guide

meta 3.1-8b 2024-07-23 Llama 3 Community License

In-depth analysis of Llama 3.1 8B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormeta
Version3.1-8b
Release Date2024-07-23
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseLlama 3 Community License
Documentationhttps://llama.meta.com/docs/

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU73%2024-07-235-shotview
HUMANEVAL63.4pass@12024-07-23—view
GSM8K58.8%2024-07-230-shot CoTview
MATH31.1%2024-07-230-shot CoTview
BBH67.1%2024-07-233-shot CoTview
GPQA34.8%2024-07-230-shotview
IFEVAL62.1%2024-07-23prompt_strictview
ARC87.4%2024-07-23challengeview
MUSR41.7%2024-07-230-shotview
WINOGRANDE77.6%2024-07-230-shotview

Pricing

TierPriceCurrency
Input$0.18 / MtokUSD
Output$0.18 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.meta.com/blog/ · as of 2024-07-23

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Llama 3.1 8B

Model Genel Bakışı

Meta Llama 3.1 8B 经济型开源模型, 128K 上下文, 8B 参数, 适合边缘部署与高吞吐量场景。

Temel Özellikler

SatıcıSürümYayın TarihiBağlam PenceresiGiriş ModlarıÇıkış ModlarıLisans
Meta3.1-8b2024-07-23128KtexttextLlama 3 Community License

Benchmark Performansı

BenchmarkPuanBirimNotlar
MMLU (Massive Multitask Language Understanding)73.0%5-shot
HumanEval63.4pass@1—
GSM8K (Grade School Math 8K)58.8%0-shot CoT
MATH31.1%0-shot CoT
BBH (BIG-Bench Hard)67.1%3-shot CoT
GPQA34.8%0-shot
IFEval62.1%prompt_strict
ARC87.4%challenge
MUSR41.7%0-shot
WinoGrande77.6%0-shot

Fiyatlandırma

GirişÇıkışÖnbellek OkumaÖnbellek Yazma
————

milyon token başına

Güçlü Yönler

  • 可靠的通用模型。

Zayıf Yönler

  • 闭源专有模型，不支持自托管。

Kullanım Senaryoları

  • 通用对话与问答

Referanslar