Frontier Data Efforts

Data markets need to be split apart and industrialized. A new class of problems exist with the normalization of post-training knowledge and how current frontier data is being curated. I'm building something new to tackle:

1. Evals aren't aligned properly to real world tasking
2. Contributor incentives are misaligned and we lack processes and programs to accurately collect process data
3. Reward Rubrics are increasingly hard to define for RL env companies because there are too many people involved in its curation who aren't domain experts.
4. The shape of good data is subjective and hard to standardize. We need to redefine the example of an eval itself and build dynamic evals for increasingly continuously trained models.

I'm working on solutions here that ignore scaling issues, prioritizing quality over quantity. These happen to do with issues regarding enterprise interactions with frontier RL problems - reward model alignment with business KPIs, maintainable envs, and evals for "data realism" as it pertains to real economic value.

Reach out to cr4sean@gmail.com with proposals/thoughts. Much of our work is based on my observations in human data markets from spending time with top RL env companies and human data pioneers.