FREUD is a reinforcement-learning trading agent, designed and measured piece by piece since 2023. Before every training run, we compute what the best possible policy would earn with the information the agent has. Every hypothesis becomes an experiment, with a protocol and a threshold written before the run. This page sets out our design principles, our method, what our experiments confirmed or refuted, and the research that guided them.
Six choices shape the agent. Each was measured before being kept.
For each market we first compute what the best possible policy would earn, with the available information and real costs. No training starts without that ceiling: it says whether the effort can succeed, and which market to target.
The optimal strategy is characterized by two principles: 1) aim in front of the target and 2) trade partially towards the current aim.
Gârleanu et al. (2013)
A perception module condenses the market into a small number of quantities; the agent decides on that summary. On our measurements, this split captures twice as much information as a network trained end to end on raw data.
The agent estimates a distribution of outcomes rather than an average. It knows its plausible worst case before acting, and its conviction shows in the gap between the actions it compares.
That is, we examine methods of learning the value distribution instead of the value function.
Dabney et al. (2018)
Risk limits define what the agent is allowed to do: maximum size, available exits. A penalty can be negotiated; a missing action cannot.
The agent reassesses its position at regular intervals but only pays the cost when it changes it. It chooses the length of its trades, rather than a deadline set in advance.
At every decision the agent writes down what it perceived, the action it preferred, the runner-up, the gap between them and its worst case, before knowing the outcome. Every choice can be checked afterwards.
The rules that apply to every experiment, without exception.
“Hundreds of papers and hundreds of factors attempt to explain the cross-section of expected returns.”Harvey, C. R., Liu, Y. & Zhu, H. (2016). … and the Cross-Section of Expected Returns. Review of Financial Studies, 29(1), 5–68. Read the paper ↗
Bailey, D. H., Borwein, J., López de Prado, M. & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance. Read the paper ↗
Eleven questions, asked in the order in which each conditions the next.
| Question | What was tested | Result | Verdict |
|---|---|---|---|
| 3.1Is there an exploitable signal, and at what horizon? | Forecasts from 5 minutes to 21 days, every year from 2016 to 2024 | A short-term signal, positive every year. Beyond a few hours, nothing verifiable. | ConfirmedT55 · T84 · T104 · Figure 1 |
| 3.2Which architecture captures it best? | Four model families, same data, same judge | A dedicated perception captures twice as much information as an end-to-end network. | EstablishedT64 · T65 · T78 · Figure 2 |
| 3.3Does the signal pay for its execution cost? | Best computable policy, ten cost levels, two markets | EUR/USD: yes, at a raw account’s cost. Gold: at no cost. | EstablishedT101 · T102 · Figure 3 |
| 3.4Does the result survive out of sample? | Reserved period 2024-2026, read once, verdict declared in advance | Positive gain, still too short to tell from chance. | Not establishedT109 |
| 3.5Does other market information add anything? | 36 macroeconomic series, order flow, 80 proposals from external reviewers | None does better than a random control of the same size. | RefutedT80 · T83 |
| 3.6Does the long-term trend still pay? | Twelve-month trend, 2000-2024 | Yes until 2015; nothing since, even without costs. | RefutedT86 |
| 3.7Does a discovery replicate? | Walk-forward over five successive periods | A three-day edge, seen in discovery, vanishes in confirmation. | RefutedT57 |
| 3.8Does the learning loop compute what it claims? | Independent recomputation of every step, against the published formulas | Exact to within 10⁻⁷, after two improvements. | ConfirmedT72 · T99 |
| 3.9Do the agent’s design choices hold? | Anchoring the value level (four variants); minimum holding time | Anchoring: the cause lies elsewhere, being measured. Minimum holding: removed for EUR/USD (+23% information learned). | OngoingD1 · T110 |
| 3.10Are the learning settings right? | 60 training runs: 15 settings, 3 seeds, two datasets, rules fixed in advance | Two settings matter: the learning pace, which left its range from the 51st episode on, and how fast the target is updated. The others bring nothing. | EstablishedT123 · T124 · T126 |
| 3.11Does exploration make the agent live through what it will do once judged? | Two ways of exploring, same values, same market, compared with the judged behaviour | No: at the end of exploration, only one decision in five was the one it would take in validation. With an explicit trial, nine in ten. | EstablishedT125 · T127 |
Each verdict reads on a scale of evidence: from left to right, refuted, not established, established, confirmed. A scale without a mark: measurement ongoing.
At thirty minutes the information clearly exceeds the noise. At two hours it meets it. At one day it would have to be seventeen times stronger to be verifiable: beyond a few hours, nothing can be proven on a single market.
Noise threshold ≈ 2/√n, where n is the number of independent observations at that horizon: overlapping forecasts count once.
The network trained end to end on raw data captures half the information of the dedicated perception: on a weak signal, it mostly learns noise. A compact head placed on the perception keeps almost everything.
T64 and T78, same protocol. Model families are not named.
The lower the cost, the stronger EUR/USD’s gain: a real signal, held back by cost. On gold, lowering the cost changes nothing.
Data 2016 → February 2024. Optimal policy computed on out-of-sample forecasts, walk-forward (T102, 28 Sept 2026). Newey-West statistic on daily gains.
| Cost (pips) | strength |
|---|---|
| 0.50 | +3.99 |
| 0.60 | +3.94 |
| 0.70 | +2.70 |
| 0.80 | +1.94 |
| 0.84 | +1.80 |
| 1.00 | +1.79 |
| 1.20 | +0.49 |
| 1.40 | +0.52 |
| 1.63 | −1.13 |
| 1.80 | −0.58 |
Each note states what was measured, what the agent gained from it, and the research that sheds light on it. Insufficient versions are included: measuring them is how the agent progressed.
During training, the agent alternates its own decisions with trials. Its exploration setting read like a probability of trying; measured, it was not. At the end of exploration, only one decision in five was the one the agent would take once judged, and its positions lasted an hour at the median: it was learning from conduct it would never have. With an explicit trial, nine decisions in ten are its own, and positions last two and a half times longer. On a short run, fifteen episodes of pure trials at the start are enough to cover 95% of situations by the 18th episode.
What the agent gainedExploration becomes an explicit trial: its setting finally says what it does. A plateau of pure trials opens the run. Its effect on learning will be measured on the run itself.
The learning pace, how fast the target is updated, the batch size and the learning frequency came from earlier research, never redone on the current agent. A study written before computing varied them one at a time: 45 training runs, then 15 confirmation runs on ticks, three seeds each. The learning pace left its useful range from the 51st episode on: at the end-of-run pace, the agent learned 27% less information. The inherited rule “update the target every 80 steps” brings nothing and is among the most unstable.
What the agent gainedThe pace now follows the length of the run and ends within its range. A slower target cuts policy reversals by 40%. Batch size and frequency stay unchanged: changing them brings nothing, or loses information.
As training went on, the average level of the learned values kept rising while the gaps between actions did not follow. We compared four ways to anchor it, drawn from the average-reward literature, on the same real data and two seeds each. None pins it down in 10,500 learning steps, and the learned information is the same in all four cases.
What the agent gainedNo setting is changed, as the rule fixed in advance required. The drift comes from a bias in the targets, which we are now measuring.
“All of our algorithms are based on using the temporal-difference error rather than the conventional error when updating the estimate of the average reward.”Wan, Y., Naik, A. & Sutton, R. S. (2021). Learning and Planning in Average-Reward Markov Decision Processes. ICML. Read the paper ↗
One of the four variants compared.
“The popular Q-learning algorithm is known to overestimate action values under certain conditions.”van Hasselt, H., Guez, A. & Silver, D. (2016). Deep Reinforcement Learning with Double Q-learning. AAAI. Read the paper ↗
The lead we are now measuring.
A minimum holding time, calibrated on gold, forced the agent to live through long positions. Measured on EUR/USD in three ways: it halves the best policy’s gain at a raw account’s cost; it multiplies long positions in the training data by ten; and yet the agent learns 23% more information without it.
What the agent gainedThe minimum holding time becomes market-specific: removed for EUR/USD, kept for gold.
The cost of a decision went from 0.83 s in the first episode to 3.89 s in the seventh, with no change in the data. A dedicated measurement isolated the cause: a step of the agent’s memory was rebuilt at every insertion.
What the agent gainedThe step is now built once, and a test guarantees it. Lightening the rest of the step is under way.
Our diagnostic tool gave the wrong direction to 89% of the trades closed by a reversal: its alerts described the tool, not the agent. Checked against the environment’s ground truth, it was rebuilt.
What the agent gainedEvery instrument is now validated against ground truth before it is read.
March 2024 → September 2026 had never been looked at. Everything was frozen beforehand: perception, protocol, a cost of 0.82 basis points, the declared verdict. Over 31,183 decisions the forecast beats its 20 permutations and the timing gain is positive (+871 pips). The signal survives out of sample, weakened by more than half; two and a half years remain too short to tell it from chance.
What the agent gainedThe period is now spent. The next proof will come from new data: the agent’s log, timestamped before each outcome.
Bailey, D. H., Borwein, J., López de Prado, M. & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance. Read the paper ↗
Why a reserved period is not re-read: every new trial on the same data raises the probability of overfitting.
Removing a single point at the end of the series was enough to move EUR/USD’s result markedly: the draw of training points weighed as much as the effect. Of five single draws, three stayed below the proof threshold; their combination crosses it.
What the agent gainedThe agent’s perception combines several draws, and no result near a threshold is read on a single one any more.
Same perception, same control, ten costs. On EUR/USD the gain is proven at 0.5 pip, vanishes from 1.2 pips and turns into a loss at the average cost of our former simulator. On gold, lowering the cost reveals nothing, even at $0.10/oz.
What the agent gainedThe agent trains on EUR/USD at raw-account costs.
Before asking an agent to learn, we compute the optimal policy for the information it receives, the way one knows the value of perfect play before training a player. With transaction costs, that policy leaves zones where it is better not to move. On gold, at the full futures contract’s cost, its gain is indistinguishable from zero; at a CFD’s cost it never takes a position.
What the agent gainedNo GPU training starts without a computed ceiling.
“The optimal strategy is characterized by two principles: 1) aim in front of the target and 2) trade partially towards the current aim.”Gârleanu, N. & Pedersen, L. H. (2013). Dynamic Trading with Predictable Returns and Transaction Costs. Journal of Finance, 68(6), 2309–2340. Read the paper ↗
The theoretical frame of optimal trading with predictable signals and costs.
The literature documents a trend effect across many markets. We found it on gold from 2000 to 2015, with a Sharpe ratio of +0.46 in futures. Over 2016-2024 it earns nothing, even with no costs at all.
What the agent gainedThe long trend serves as a control: a bench that failed to find it over 2000-2015 would be faulty.
“We find that the past 12-month excess return of each instrument is a positive predictor of its future return.”Moskowitz, T. J., Ooi, Y. H. & Pedersen, L. H. (2012). Time Series Momentum. Journal of Financial Economics. Read the paper ↗
The effect we find until 2015, and not after.
Same data, same judge, four algorithms. The 30-minute forecast is positive every year from 2016 to 2024. Beyond a few hours, nothing, whatever the algorithm (figure 1).
What the agent gainedThe agent decides at the pace where the information exists, on a deliberately small state.
On the same data, our perception module captures twice as much information as a network trained end to end (+0.035 against +0.018). A compact head placed on that module keeps almost everything (+0.031): the large network was re-learning the perception from raw data, and overfitting it (figure 2).
What the agent gainedThe agent decides on the summary produced by the perception, no longer on raw data.
With a very long horizon, every learned value carries a huge common term: a hundred thousand times the average reward. The network spent its effort following that level instead of telling the actions apart.
What the agent gainedThe average reward is removed before learning. On 3 October we checked that this is not enough to pin the level (note D1).
“In addition, we show that if a problem's rewards are shifted by a constant, then standard methods perform much worse, whereas methods with reward centering are unaffected.”Naik, A., Wan, Y., Tomar, M. & Sutton, R. S. (2024). Reward Centering. Reinforcement Learning Conference (RLC). Read the paper ↗
The method adopted.
The memory that replays the most instructive experiences first had two shortcomings: recent experiences were under-represented, and a single non-numeric value could suspend learning with no visible error. Every step of the loop is now recomputed independently, to within 1.5 × 10⁻⁷.
What the agent gainedExact priorities, non-numeric values isolated, and a bench that recomputes the whole learning loop.
“In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently.”Schaul, T., Quan, J., Antonoglou, I. & Silver, D. (2016). Prioritized Experience Replay. ICLR. Read the paper ↗
The memory mechanism we measured.
A regime analysis in the environment read the thirty minutes after the decision: a shortcoming invisible on learning curves, which makes every result optimistic.
What the agent gainedThe analysis now reads only the past, and two automated checks verify it.
A fixed deadline closed any position after 15 hours; winning trades lasted longer, and all were cut before they paid.
What the agent gainedThe policy chooses its horizon and pays the cost once per trade.
The simulator counted the spread in the execution price and again in the fees: the agent saw twice the real cost, and quite logically learned that trading does not pay.
What the agent gainedThe spread is now counted once, in the execution price.
With a weak signal, a method that averages many models beats a sequential method by 27%, on nine time folds out of ten (t = +3.60, paired test). What limits the forecast here is noise, and averaging reduces it.
What the agent gainedThe agent’s perception is built to reduce variance.
A first bench found an edge at three days (t = +2.52). Redone walk-forward, over five successive periods, it is positive on only one or two: the result came from a favourable split of the data.
What the agent gainedEvery result is tested walk-forward, with a purge between training and test, and read against a stricter threshold than t = 2.
“Hundreds of papers and hundreds of factors attempt to explain the cross-section of expected returns.”Harvey, C. R., Liu, Y. & Zhu, H. (2016). … and the Cross-Section of Expected Returns. Review of Financial Studies, 29(1), 5–68. Read the paper ↗
After so much research on the same data, the authors propose requiring t > 3.0 from a new factor.
A shaping component of the reward paid +0.527 per step to a position down €100: it encouraged holding losses. We exposed it by deriving the formula, then rebuilt it.
What the agent gainedThe final reward is logarithmic, a Kelly criterion, and every component is audited line by line.
Ng, A. Y., Harada, D. & Russell, S. (1999). Policy invariance under reward transformations: theory and application to reward shaping. ICML.
The theorem that makes shaping harmless to the optimal policy assumes the same discount factor throughout: the condition the insufficient version did not meet.
A first market environment, a first reinforcement-learning agent, and the pipeline that turns ticks into training data. Many architecture trials, most of them set aside.
31 revisionsUncertainty modelling, realistic brokerage fees, GPU training in the cloud. The first long runs show that an agent can look as if it learns while learning nothing: you need instruments to tell.
34 revisionsWe stop judging the agent by its curves. Every component is run and measured; every shortcoming becomes an improved version, and a test that guards it.
24 revisionsEvery hypothesis becomes an experiment, with its protocol, its controls and a threshold fixed in advance. Is there a signal? At what horizon? What does exploiting it cost? Then computing the best possible policy steers the agent towards EUR/USD.
154 revisionsThe review of the first EUR/USD training settles two design choices by measurement and lightens the training step. 22 years of EUR/USD ticks, downloaded again at millisecond resolution, form the new training set, checked table by table. A study written in advance, 60 training runs, sets the next training’s settings.
ongoingPassages in quotation marks are copied from the abstracts published by their authors.