Research
Stanford-linked paper argues the real evaluation problem in AI factor discovery is the system, not the factor
"Agentic Empirical Asset Pricing", posted September 1 by Pan, Ding and Giesecke, proposes evaluating the autonomous research process that finds a trading factor, not just the factor's backtested return, and shows single-metric rankings don't hold up under rolling re-execution.
A paper posted to arXiv on September 1 by Yingjian Pan, Xiaowei Ding and Kay Giesecke introduces what it calls Agentic Empirical Asset Pricing (AEAP): systems built to autonomously run the scientific discovery process itself, in this case the search for return-predicting factors, rather than simply executing a fixed research pipeline a human designed. The framing matters for readers of this site's research thread on LLM agents versus quant. Most of the factor-discovery agents covered here to date (AQuA, XALPHA, AlphaCrafter, QRAFTI) are judged on the factors they output: does the strategy backtest well, is the Sharpe ratio competitive, does the signal survive out-of-sample. Giesecke's group argues that framing quietly assumes the discovery system itself is a black box worth ignoring once it has produced a result, and that assumption is where the paper's real contribution sits.
The methodological claim is that a discovered factor and the system that discovered it need to be evaluated separately, because a system can produce one good factor by chance, overfitting, or evaluation leakage in ways a single backtest won't reveal. The paper's proposed fix is a reference architecture for these systems plus evaluation standards that examine performance across multiple axes at once, on the finding that no single metric consistently ranks competing discovery systems against each other. That is a direct challenge to how most agentic-finance papers currently get compared in the literature, which tends to lean on one or two headline numbers (return, Sharpe, hit rate) to declare a winner.
The more consequential piece is what the authors call rolling re-execution: rerunning the discovery process itself at successive points in time to test whether the process reliably finds good factors again, not whether one factor it found once still performs. That distinction is the difference between validating a result and validating a research method, and it's the second one that actually matters if a firm intends to let an agentic system run unsupervised on a recurring basis rather than treat a single factor as a one-off deliverable to hand off to a human portfolio manager. Static backtesting, standard practice across nearly every agentic-trading paper covered on this site so far, can't distinguish a discovery process that will keep working from one that got lucky once.
Notably, the paper documents negative findings and evaluation pitfalls specific to AEAP systems rather than only reporting favourable results, a habit still rare in a literature that skews toward publishing whichever configuration beat its benchmark. For firms evaluating whether to let an agent run their own factor research pipeline rather than just execute trades it's already been told to make, the paper's practical implication is blunt: a vendor's backtest of one discovered factor tells you almost nothing about whether their discovery system is trustworthy to run again next quarter, and that's a materially harder thing to demonstrate.
Read the original: arXiv (Pan, Ding, Giesecke) - Stanford-linked paper argues the real evaluation problem in AI factor discovery is the system, not the factor. Commentary is the independent editorial view of Share Trading; the original article is credited to its publisher.