Categorization results
Llama 3.1 8B Instruct Q4_K_M completed 400 real development articles across 10 reference categories. This first run measures agreement with Sift's stored labels.
- Stored-label agreement
- 79.8%
- 319 of 400 labels matched
- Macro-F1 against stored labels
- 0.7598
- Descriptive 95% bootstrap interval: 0.7100–0.8035
- Execution coverage
- 400/400
- 0 errors, 0 retries, 0 parse failures
- Reserved holdout
- 100 articles
- Not used in this development run
Per-category agreement
Every row compares the model's output with the stored category. The largest disagreement pairs were world → politics (14) and general → politics (10). These cases have not been adjudicated.
| Stored category | Items | Matches | F1 vs stored labels |
|---|---|---|---|
| business | 31 | 23 | 0.780 |
| energy | 16 | 14 | 0.800 |
| entertainment | 66 | 54 | 0.850 |
| general | 23 | 4 | 0.296 |
| health | 28 | 21 | 0.808 |
| politics | 103 | 95 | 0.812 |
| science | 21 | 19 | 0.776 |
| sports | 35 | 31 | 0.886 |
| technology | 31 | 29 | 0.866 |
| world | 46 | 29 | 0.725 |
Energy has only 16 development items; its class-level estimate is particularly uncertain.
Sample and scope
From 962 unique source-matched inputs, a fixed-seed stratified sample selected 500 articles: 400 development and 100 reserved holdout. Original RSS titles, sources, and bodies were matched to a read-only snapshot of production labels. Generated summaries and stored category fields were excluded from article input.
- 55 outlets; 205 fresh-capture and 195 saved original inputs, published October 10, 2024–October 24, 2026. This is not a random recent-production sample.
tophad only 14 unique captured inputs, below the 20-item minimum, and was excluded from reference sampling. All eleven Sift labels remained valid outputs; macro-F1 uses the ten sampled reference categories. All 400 development items are counted.- A disagreement may reflect a model error, a noisy stored label, or an ambiguous category boundary. Independent label review is still needed.
- One development run does not establish a model ranking or justify a production routing change. Final held-out and cost comparisons remain pending.
Runtime and provenance
Local macOS ARM64, Ollama 0.40.10; temperature 0, seed 20261024, maximum 10output tokens. The categorization-only prompt uses Sift's checked-out category rules, rather than the production batch summarization operation.
Median generation HTTP latency: 0.663s; p95: 3.618s. These timings exclude the separate digest check and are not a DGX Spark benchmark or a measured cost comparison.
The Ollama manifest was checked before each generation. Its SHA-256 is 46e0c10c039e019119339687c3c1757cc81b9da49709a3b3924863ba87ca666e. This identifies the Ollama artifact; no Hugging Face commit is asserted.
The run started from commit 6e37a11 with uncommitted run-enablement changes. Executed sources were snapshotted and hashed; that base commit alone is not the complete run code. Matching source files and hashes are published with this baseline. Article text and per-item artifacts remain local.
Download aggregate results and provenance (JSON) · Read the run report
See the methodology and project summary for scoring and the broader study. Agent scenarios remain scripted fixtures, separate from this real categorization run.