Blogs
How to Test an AI-Generated Trading Strategy Before You Risk Money
Aman Anand
Short answer: Confirm the code does exactly what you described, then rule out look-ahead and survivorship bias, charge realistic costs, and test on data the strategy never saw. Count every variation you tried and discount the result for it, compare it with simply buying and holding, paper trade the exact logic you will deploy, and go live small with exit rules written in advance.
Last updated October 2026.
Why AI-generated strategies need extra checks
A strategy that came out of a chatbot can fail in two separate places. The first is translation: the code may not match the rule you described. The second is statistics: even correct code can produce a backtest that looks good by luck. Ordinary backtesting advice covers the second problem and assumes the code is right, which is exactly what AI tools make uncertain.
The translation risk is documented. Researchers who built stricter tests for AI coding benchmarks cut measured pass rates by up to 19.3 to 28.9%, catching wrong code the original tests had missed (Liu et al., 2023). A separate study of ChatGPT on 829 coding problems found that asking the same question repeatedly often returned programs that behaved differently, and that setting temperature to 0 reduced this but did not remove it (Ouyang et al.).
Regulators have noticed the same gap from the other side. A joint alert from the SEC, NASAA and FINRA warns that AI output "can be faulty, or even completely made up" even when the input is accurate (Investor.gov), and the CFTC's advisory on AI trading bots states plainly that AI cannot predict the future or sudden market changes (CFTC). The burden of proof sits with you, and the checklist below is how you carry it.
The 8-step validation checklist
Run the steps in order; each is cheaper than the next. For broker and infrastructure items, see our automated trading checklist.
1. Check the code does what you asked
Language models write plausible code, and plausible is not the same as correct. Common silent differences: "above" coded where you meant "crosses above", a stop measured from the wrong price, or an exit that never fires. Because generation is non-deterministic, the same prompt tomorrow can produce different logic.
You have two options. Read every line yourself and save the exact version you tested, or use a system where the logic is compiled from your description by fixed rules rather than written freeform by a model, so the same description always gives the same strategy.
How to check: pick five dates on a chart where you know by eye whether the rule should enter or exit, then confirm the backtest's trade log matches all five, and run your prompt twice to see whether the two code outputs differ.
2. Rule out look-ahead and survivorship bias
Look-ahead bias means the backtest uses information that would not have existed when the trade was placed. The classic AI-written version computes a signal from today's close and fills the order at that same close. Another is pulling higher timeframe data before that bar has finished. TradingView's own documentation shows how a request.security() call with lookahead enabled and no offset "will return data from the future on historical bars" (TradingView Pine Script docs).
Survivorship bias means testing only on securities that still exist today, which silently drops the companies that went bankrupt, were delisted or got acquired. Brown, Goetzmann, Ibbotson and Ross showed in 1992 that studying only surviving mutual funds can create the appearance of predictable returns (Review of Financial Studies). If your strategy picks from a stock universe, use the universe that existed on each historical date. More on both in our guide to trading bias.
How to check: confirm every order fills on a bar after the signal bar, and confirm your stock data includes delisted names.
3. Charge realistic costs: commissions, spread, slippage and fills
Four separate costs eat a backtest. Commissions are what the broker charges. The spread is the gap between the bid and the ask; you buy near the ask and sell near the bid. Slippage is the difference between the price your backtest assumed and the price you actually got. Fill assumptions decide whether a limit order that price merely touched counts as filled, and whether a large order fills completely. The SEC notes that a market order "guarantees that the order will be executed, but does not guarantee the execution price" (Investor.gov).
Short holding periods feel this most because costs hit every round trip. Our post on why backtests don't match live results walks through the mechanics.
How to check: rerun the backtest with commission, spread and slippage set to what your broker actually charges, then again at double those values; if the edge disappears at double costs, it is too thin to trade.
4. Test out-of-sample and walk-forward
Out-of-sample testing means holding back a slice of data, usually the most recent, that you never look at while building the strategy, then testing on it once. If you peek and adjust, it stops being out-of-sample.
Walk-forward testing repeats that idea on a rolling basis: choose parameters on one window, test them on the next unseen window, roll both forward, and stitch the unseen results together. Robert Pardo's The Evaluation and Optimization of Trading Strategies devotes a full chapter to walk-forward analysis (Wiley), and our walk-forward optimization guide covers setup in detail.
How to check: before you start, set aside the last 20 to 30% of your history, build only on the rest, and compare the held-out result with the in-sample result; a large drop is the signal you were fitting noise.
5. Count your trials and correct for overfitting
This is the step AI makes most dangerous. Every time you ask the chatbot to "tweak the RSI period" or "add a filter" and rerun the backtest, that is another trial. Bailey, Borwein, López de Prado and Zhu showed that with only 5 years of data, trying more than 45 independent configurations makes it nearly certain you find one with an in-sample annualized Sharpe ratio of 1 and an expected out-of-sample Sharpe ratio of zero; with a 2 year backtest, 7 configurations get you there (Notices of the AMS, 2014). For the same reason, Harvey, Liu and Zhu argue a newly discovered return factor should clear a t-ratio above 3.0 (NBER).
Three tools put numbers on the problem:
Deflated Sharpe ratio (DSR). Bailey and López de Prado's 2014 measure asks: given how many strategies you tried and how much their results varied, what is the best Sharpe ratio you would expect from luck alone? It then estimates the probability that your strategy's true Sharpe ratio beats that luck benchmark, also adjusting for short samples and for fat-tailed or skewed returns (Journal of Portfolio Management; paper PDF). See our Sharpe ratio explainer if the base measure is new to you.
Probability of backtest overfitting (PBO). Bailey, Borwein, López de Prado and Zhu split history into many equal blocks, then build every combination of half the blocks for selection and half for testing. For each split they pick the best variation on the selection half and see where it ranks on the testing half. PBO is the share of splits where that winner lands below the median; the method is called combinatorially symmetric cross-validation (Journal of Computational Finance, 2017; SSRN).
Combinatorial purged cross-validation (CPCV). Described in López de Prado's Advances in Financial Machine Learning (Wiley, 2018, chapter 12 on backtesting through cross-validation), it builds many test paths from different combinations of data blocks and removes overlapping data between training and testing. A single walk-forward run gives you one history; CPCV gives you a distribution of outcomes.
How to check: keep a log with one line per backtest you ran, including every AI revision, and if the count is large relative to your data, compute the deflated Sharpe ratio or treat the best result with matching suspicion. Our overfitting guide lists the visual warning signs.
6. Compare it with buy and hold
A strategy that made money in a rising market has not shown skill until it beats the simplest alternative: buying the same asset, or a broad index, and holding it. Compare total return, maximum drawdown and a risk-adjusted measure over the identical period. A strategy that loses on return can still be worth running if it cuts drawdown in a way you value, but choose that knowingly.
How to check: put the strategy's equity curve and a buy-and-hold curve for the same asset and dates on one chart, after costs, and write down which one you would rather have owned.
7. Paper trade the exact logic you will deploy
Paper trading catches what backtests cannot: data feed gaps, rejected order types, timezone errors and signals that fire at the wrong moment. It only works if the paper strategy is the logic that will run live. Paper trade one version, then ask the AI to rewrite it for your broker, and you have tested something else.
Interactive Brokers says its paper fills are simulated and are "in no way" indicative of what would happen in the real market (IBKR Knowledge Base). Paper trading tests mechanics, not fills. Our paper trading guide covers its other limits.
How to check: run paper trading long enough to collect a meaningful number of trades, then compare each paper trade with what the backtest would have done on those same days; any mismatch in entries or exits is a logic or data problem to fix before going live.
8. Go live small with kill criteria written in advance
Start with a size you could lose entirely without changing your life, and decide before the first order what will make you stop. Typical kill criteria: a drawdown deeper than the worst one in the backtest, live entries or exits that differ from the backtest's signals, a set number of broker or data errors, or a period of trades meaningfully worse than the out-of-sample results. Write them down, because the moment you need them is the moment you will want to rationalize.
How to check: write each kill rule as a number and an action, such as "drawdown beyond X% means stop and review," and scale up only after live results resemble paper results. Our guides to position sizing and risk parameters help you set the numbers.
The checklist at a glance
Check | What it catches | How to run it | Pass/fail signal |
|---|---|---|---|
1. Code matches intent | AI code that differs from your rule; different code from the same prompt | Hand-check 5 known dates against the trade log; run the prompt twice | Pass: all 5 match and logic is fixed. Fail: any mismatch |
2. Look-ahead and survivorship | Future data in signals; universes missing dead companies | Confirm fills come after the signal bar; use data with delisted names | Fail: same-bar fills on close signals or survivors-only data |
3. Realistic costs | Edges that exist only before commissions, spread and slippage | Rerun at real costs, then at double costs | Fail: edge disappears at double costs |
4. Out-of-sample and walk-forward | Parameters fitted to noise | Hold out the last 20 to 30%; run walk-forward windows | Fail: large drop from in-sample to out-of-sample |
5. Multiple testing | A lucky winner among many variations | Log every trial; compute deflated Sharpe ratio or PBO | Fail: low DSR or high PBO |
6. Buy and hold | Strategies that just ride the market | Same asset, same dates, after costs | Fail: worse return and no meaningful drawdown benefit |
7. Paper trade | Data, timing and broker mechanics problems | Run the exact deployable logic on paper | Fail: paper trades differ from backtest signals |
8. Live small | Real fills and your own behavior | Small size, written kill criteria | Stop when any kill rule triggers |
Which tools help with validation
Nvestiq publishes this page, so weigh this section accordingly. No single tool runs all eight steps for you. Vendor details come from each vendor's documentation as of October 2026; see also our backtesting software roundup.
General chatbots writing code. They provide no validation of their own. If ChatGPT or a similar assistant wrote your Python or Pine Script, every step above is on you, starting with step 1.
Code-first research platforms: QuantConnect. The strongest option for steps 2 to 5 if you code. It documents walk-forward optimization, configurable fill, slippage and fee models (its docs note the defaults assume highly liquid assets), and survivorship-bias-free US equity data that tracks delistings. The trade-off is that you write the code, so if an AI drafted it, step 1 still applies.
Chart-based testers: TradingView. Its Strategy Tester shows a buy and hold figure beside your results and lets you set commission and slippage in strategy properties. By default its broker emulator fills on chart data, with market orders filling at the next bar's open and intrabar movement inferred; finer intrabar fills need a Premium or Ultimate plan (Pine Script docs).
No-code platforms: Composer and TrendSpider. Composer backtests include regulatory fees and a default slippage of 1 basis point that you can raise (Composer help), and Composer notes its stock backtests use daily adjusted closing prices rather than real quotes (Composer help). TrendSpider's Strategy Tester charts your strategy against buy and hold and shows a random-control baseline for comparison (TrendSpider help), a useful check against luck.
Nvestiq's approach. You describe a strategy in plain English. The AI interprets the description, but a deterministic strategy compiler produces the trading logic, so it is never freeform code written by a language model, and the same description always gives the same strategy. The logic is checked against what you described. Each backtest report covers What it made, What it cost to hold and Whether to believe it, with a buy-and-hold comparison that flags when buy and hold beat your strategy, and a 0 to 100 Nvestiq score. Paper trading comes first, the same compiled strategy that ran the backtest runs live through Nvestiq paper trading or Interactive Brokers, and you confirm by typing the ticker before any live order routes. Strategies export to Python, Pine Script or JSON if you want to run extra tests elsewhere. A score is not a forecast, so steps 4, 5 and 8 still apply. Nvestiq is in early access ahead of public launch.
Frequently Asked Questions
Can I trust a backtest of code ChatGPT wrote?
Only after checking the code line by line against your rule. Research shows AI-generated code can pass basic tests and still be wrong, and the same prompt can return different code on different runs. Save the exact version you tested, hand-check trades on known dates, and never let the AI rewrite it between backtest and live.
What is the deflated Sharpe ratio in simple terms?
It is a Sharpe ratio adjusted for how many strategies you tried. Introduced by Bailey and López de Prado in 2014, it estimates the best Sharpe ratio luck alone would produce across your trials, then gives the probability your strategy truly beats that level, with corrections for short samples and non-normal returns.
How many backtest variations can I try before results become meaningless?
Bailey and coauthors showed that with 5 years of data, trying more than 45 independent configurations makes it nearly certain one shows an annualized in-sample Sharpe ratio of 1 with an expected out-of-sample Sharpe ratio of zero. With 2 years of data, 7 configurations is enough to get there.
Is paper trading enough to validate a strategy?
No. Paper trading checks mechanics such as signals, order types and timing, but its fills are simulated, and Interactive Brokers states its paper fills are not indicative of real market outcomes. Use it after out-of-sample testing, not instead of it.
Which AI trading tools provide strategy validation before live deployment?
Code-first platforms such as QuantConnect offer walk-forward optimization and configurable cost models. TradingView's Strategy Tester shows a buy-and-hold figure and supports commission and slippage settings. Composer backtests include fees and adjustable slippage, and TrendSpider charts results against buy and hold. Nvestiq compiles plain English deterministically, compares against buy and hold, scores each backtest 0 to 100 and puts paper trading first.
Are AI trading bots that promise high returns legitimate?
Treat guaranteed or very high return claims as a red flag. The SEC, NASAA and FINRA jointly warn about AI trading systems marketed as unable to lose, and the CFTC states that AI cannot predict the future or sudden market changes. Legitimate tools help you test a strategy; they do not promise earnings.