Too Early to Say
Independent applied economics research. AI-augmented analysis with full methodology transparency.
Analysis Spotlight
Logistic regression beats LLM readouts on survey prediction
On a real public-health prediction task, a plain logistic regression on seven demographic facts outpredicted a language-model activation pipeline built on the same facts, and steering the model's internals changed nothing about the ranking while making the probabilities less trustworthy. Part 3 of our series on classical statistics in the age of AI.
The Head-to-Head
- Logistic regression: AUC 0.769 vs the GPT-2 readout's 0.747. DeLong p = 0.003, on 937 held-out respondents.
- The gradient-boosted tree ties the readout. Difference -0.003, p = 0.73.
- Steering moves probabilities, not ranks. AUC stays at 0.747-0.746 while calibration error roughly triples.
- A 56x larger model lands at 0.722. Below both the small model and the regression.
Latest Research
Matching in Python: a balanced covariate table doesn't make the estimate valid
Difference-in-differences in Python: why the TWFE coefficient can mislead
How to tell whether a double machine learning estimate is right
Start Here: AI for Applied Researchers
A five-step sequence for putting an AI coding agent into a real applied economics study, from the first literature scan to the replication package.
- Step 1Literature reviewPull the benchmark numbers before any code.
- Step 2Code generationA methods paragraph becomes a verified pipeline.
- Step 3Data cleaningClassify noisy categories, report the error rate.
- Step 4Quality assuranceBreak the result before you report it.
- Step 5DocumentationTrace every number back to the code.
Methodology tutorials
Step-by-step methods with real data and open code.
- How to estimate difference-in-differences in PythonEvent-study estimation with the parallel-trends diagnostics.
- When the parallel-trends test fails on one leadTWFE difference-in-differences on SNAP BBCE with a Goodman-Bacon decomposition.
- How to interpret a classifier with SHAP valuesWhat SHAP importance does and does not measure.
- One context file, zero re-explanationsA CLAUDE.md that teaches an AI assistant your research project.
Methods
How we do research: AI-integrated workflows, validation routines, and reproducible pipelines with open code. The starter kit that sets up an economist's first week in Claude Code is cloneable from the companion repository on GitHub.
Browse methodology →Research by Topic
Browse by subject area
Food Security
Food access, SNAP enrollment, grocery store distribution, and the complex relationship between geography and food insecurity.
Transit Equity
How transit access shapes economic opportunity. Mobility deserts, accessibility measurement, and transportation barriers.
Education Policy
School funding, special education budgets, and how districts respond to fiscal constraints and stimulus funding.
AI-Integrated Methodology
How we do research: data validation, transit routing, API collection, and AI-integrated workflows with code examples.
About Too Early to Say
Too Early to Say is an independent applied-economics research lab founded by Victoria Cholette, PhD. We publish reproducible policy analysis on food security, transit equity, education finance, and health economics, each paired with open code and a transparent, AI-augmented methodology log. Read about the lab →
Need applied economics research for a specific decision? Advisory engagements available →
New research in your inbox
Analyses and the methods behind them: food security, transit, education, and health.
Subscribe for Free