Compare models on Sift's news-processing tasks.
A Python evaluation harness with categorization and summarization tasks, model adapters, and scripted agent regression tests. Built to support a future comparison of local models and API models for Sift.
First result: Llama 3.1 8B matched 79.8% of Sift's stored category labels on 400 real development articles, with macro-F1 0.7598. These labels are model-assigned and have not been independently reviewed by humans. The 100-item holdout remains unevaluated.
319/400 labels matched across ten reference categories. This measures agreement with existing model-assigned labels, not human-validated accuracy.
Replay fixtures exercise retrieval, a transient tool failure, and a document injection. They test the harness; they do not measure a live model's reliability.
The reserved set is unevaluated. Independent label review, a final held-out comparison, more model runs, and measured costs remain pending.
What's on this site
- Methodology — implemented scoring, study design, and limitations.
- Development results — measured categorization agreement, sample scope, category breakdown, and a downloadable aggregate artifact.
- Executive summary — what is built and what remains to run.