We build the expert human-data layer behind frontier coding models — SFT trajectories, RLHF preference data, agentic SWE evaluations, and verifiable benchmarks. Every artifact is reviewed by senior engineers, not crowdworkers.
Powering data pipelines for frontier labs & AI-native teams
Each engagement is staffed by working software engineers and calibrated against a strict, versioned rubric. No bulk crowdsourcing.
High-signal supervised fine-tuning examples: multi-file edits, bug-fix pairs, test authoring, and CLI/terminal traces — each with a verifiable solution.
Calibrated pairwise & ranked preferences over model completions, with structured rationales engineers can actually audit and replay.
End-to-end grading of coding agents on real repositories — tool use, multi-step trajectories, and environment outcomes scored against ground truth.
Private, contamination-resistant benchmarks tailored to your stack — so you measure progress on the code that actually matters to you.
Acceptance rate on lab QA
Vetted senior engineers
Languages & frameworks
Median pilot turnaround
A four-stage pipeline with measurement baked into every step — the same rigor you'd expect from a senior code review.
We co-author the rubric with you, run a calibration batch, and lock the spec before scaling.
Working engineers produce and execute solutions in sandboxed environments with full trace capture.
Independent senior reviewers grade each item; disagreements are adjudicated and tracked as κ.
Automated test harness confirms correctness; data ships with full provenance and metrics.
Every dataset ships with a transparent scorecard. Track capability-level performance over time, compare model versions, and catch regressions before they ship.
Contamination-resistant. Held-out tasks are leakage-checked against public corpora.
Reproducible. A versioned harness reruns the exact same suite on demand.
Explainable. Drill from headline score into per-task traces and reviewer notes.
SWE-Agent Benchmark v3
held-out · 480 tasks
Frontier buyers are uncompromising on security, IP, and quality. So are we.
Audited controls, encryption at rest and in transit, and least-privilege access throughout.
Per-project NDAs, segregated workspaces, and full chain-of-custody on every artifact.
Every annotator passes a live coding screen. No anonymous crowd labor, ever.
Every label carries authorship, review history, timestamps, and rubric version.
Predictable weekly delivery with live dashboards — no black-box timelines.
Anything below the agreed quality bar is reworked at no cost. Aligned incentives.
We're hiring working software engineers to author, solve, and review high-signal data for frontier coding models. Flexible, remote, and paid per accepted contribution — with a clear path to senior reviewer.
Premium pay. Competitive per-task rates that scale with difficulty and your review tier.
Fully flexible. Work from anywhere, on your own hours. Take batches when you want them.
Interesting work. Real repos, real bugs, agentic trajectories — not toy snippets.
Tell us your model, your stack, and the capability you're trying to move. We'll scope a calibration batch and send you a sample scorecard.
No spam. We reply to every serious inquiry within 24 hours.