State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jesper Klicks, Sander Vržina, Vincent François-Lavet
• Venue: arXiv: cs.LG
• Published: 2026-06-25
• Collected: 2026-06-26
• URL: http://arxiv.org/abs/2606.27032v1
• PDF: https://arxiv.org/pdf/2606.27032v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers study the impact of state representation on a reinforcement learning agent in energy trading decisions, focusing on a pumped-storage arbitrage environment. They find that combining absolute price information with relative and forecast features leads to significantly better results than relying on any single feature family.

What the Document Says

• The study focuses on the significance of state representation for a reinforcement learning agent in energy trading, particularly in a pumped-storage arbitrage environment named HydroDam.
• The research uses a fixed Double DQN agent and keeps the environment, action space, reward function, network, and training protocol consistent while altering only the market features.
• Four types of features are compared: absolute price/calendar features, relative features comparing current prices with recent market history, forecast features, and combinations of these three families.
• Training and policy selection are done using 2007-2011 Belgian day-ahead prices, and the results are evaluated on a later same-market test set from 2012-2025 and 39 other ENTSO-E market zones.
• Absolute features perform poorly, achieving only 28.8% on the test set and a median of 5.7% across zones.
• Relative-only and forecast-only states also underperform a rolling price-score heuristic in the cross-zone median.
• Combining feature families leads to better results: absolute + relative achieves 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%.
• The study concludes that state representation is crucial in storage-trading RL, requiring the combination of price scale, recent relative price context, and short-horizon forecast information rather than relying on any single feature family.

Key Concepts Mentioned

• reinforcement learning
• energy trading
• state representation
• pumped-storage arbitrage
• Double DQN

Claims Made by the Document

• State representation is an important design choice in energy trading.
• Combining absolute, relative, and forecast features leads to better results than relying on any single feature family.
• Robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information.

Evidence Used

• Results from training and evaluating policies using different feature combinations in a pumped-storage arbitrage environment.
• Comparisons between the performance of absolute, relative, forecast features, and combined feature sets.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Energy trading decisions depend not only on current market prices, but also on expected future market conditions, and operational constraints. This makes the state representation given to a reinforcement learning agent an important design choice. We study this in HydroDam, a pumped-storage arbitrage environment, using a fixed Double DQN agent. The environment, action space, reward function, network, and training protocol are kept fixed; only the market features are changed. We compare absolute price/calendar features, relative features that compare current prices with recent market history, forecast features, and all combinations of these three feature families. Policies are trained and selected using 2007--2011 Belgian day-ahead prices and evaluated on two test settings: a later same-market test set from 2012--2025 and 39 other ENTSO-E market zones. Absolute features only reaches 28.8% on the test set and a median 5.7% across zones. Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median. Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%. These results suggest that state representation is not a minor preprocessing choice in storage-trading RL, but a central part of the policy design: robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.