Evaluation Examples
Examples overview
Reward Kit Examples
This page provides an overview of and links to documentation for various examples demonstrating the capabilities of the Reward Kit. All documentation for these examples is self-contained within thedocs/ folder.
Many examples use Hydra for configuration. Please refer to the specific documentation page for each example for execution instructions.
Available Examples
-
Accuracy Length Example:
- Demonstrates combined accuracy and length rewards.
- View Documentation
-
APPS Coding Example:
- Illustrates evaluation for coding problems from the APPS dataset.
- View Documentation (New file to be created)
-
E2B (Code Execution Sandbox) Examples:
- Covers various E2B integration scenarios for sandboxed code execution.
- View Documentation
-
GCP Cloud Run Deployment Example:
- Shows how to deploy a Reward Kit application on GCP Cloud Run.
- View Documentation (New file to be created)
-
Math Example (GSM8K):
- Focuses on evaluating math word problems using the GSM8K dataset.
- View Documentation
-
Math with Formatting Example:
- Extends math evaluation to handle specific formatting requirements.
- View Documentation (New file to be created)
-
Tool Calling Example:
- Demonstrates evaluation for models that use tools or function calls.
- View Documentation (New file to be created)
-
TRL Integration Example:
- Shows how to integrate reward-kit functions with the TRL library.
- View Documentation
examples/metrics/ and examples/test_tasks/ directories in the root examples/ folder contain supporting resources and are not standalone documented examples here.
General Guides for Examples
While the pages above cover specific examples, these general guides might also be useful:- Reward Functions Overview (To be reviewed/updated)
- Basic Reward Function Concepts (To be reviewed/updated)
- Advanced Reward Function Concepts (To be reviewed/updated)
Next Steps
- See the Developer Guide for comprehensive information.
- Check the Tutorials for step-by-step guides.
- Refer to the API Reference for detailed documentation.