ServerlessLLMChat
Meta’s Llama 3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks. Llama 3 70B Instruct (HF Version) is the original, FP16 version of Llama 3 70B Instruct whose results should be consistent with the official Hugging Face implementation.
Llama 3 70B Instruct (HF version) is available via Fireworks' serverless API, where you pay per token. There are several ways to call the Fireworks API, including Fireworks' Python client, the REST API, or OpenAI's Python client.
See below for easy generation of calls and a description of the raw REST API for making API requests. See the Querying text models docs for details.
Generate a model response using the chat endpoint of llama-v3-70b-instruct-hf. API reference
import requests import json url = "https://api.fireworks.ai/inference/v1/chat/completions" payload = { "model": "accounts/fireworks/models/llama-v3-70b-instruct-hf", "max_tokens": 1024, "top_p": 1, "top_k": 40, "presence_penalty": 0, "frequency_penalty": 0, "temperature": 0.6, "messages": [ { "role": "user", "content": "Hello, how are you?" } ] } headers = { "Accept": "application/json", "Content-Type": "application/json", "Authorization": "Bearer <API_KEY>" } requests.request("POST", url, headers=headers, data=json.dumps(payload))
Llama 3 70B Instruct (HF version) can be fine-tuned on your data to create a model with better response quality. Fireworks uses low-rank adaptation (LoRA) to train a model that can be served efficiently at inference time.
See the Fine-tuning guide for details.
On-demand deployments allow you to use Llama 3 70B Instruct (HF version) on dedicated GPUs with Fireworks' high-performance serving stack with high reliability and no rate limits.
See the On-demand deployments guide for details.