State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jesper Klicks, Sander Vržina, Vincent François-Lavet
• Venue: arXiv: cs.LG
• Published: 2026-06-25
• Collected: 2026-06-26
• URL: http://arxiv.org/abs/2606.27032v1
• PDF: https://arxiv.org/pdf/2606.27032v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers compared different methods for representing market conditions in reinforcement learning (RL) models for energy trading. They found that combining multiple types of features led to stronger performance compared to using a single type.

What the Document Says

• The study focuses on the impact of state representation in deep reinforcement learning (DRL) for energy trading.
• The research is conducted in a pumped-storage arbitrage environment called HydroDam, using a fixed Double DQN agent.
• Four types of features are compared: absolute price/calendar features, relative features, forecast features, and combinations thereof.
• Training and selection of policies were done with 2007-2011 Belgian day-ahead prices and tested on two settings: a later same-market test set from 2012-2025 and 39 other ENTSO-E market zones.
• Results show that using absolute features only achieves 28.8% on the test set and a median of 5.7% across zones.
• Relative-only, forecast-only states also underperform a rolling price-score heuristic in the cross-zone median.
• Combining feature families leads to stronger performance: absolute + relative achieves 49.9% on the test set and a 39.8% cross-zone median.
• The strongest combination, absolute + relative + forecast, reaches 55.6% and 47.5% respectively.

Key Concepts Mentioned

• deep reinforcement learning
• energy trading
• state representation
• HydroDam
• Double DQN
• market features
• reward function

Claims Made by the Document

• State representation is an important design choice in deep RL for energy trading.
• Combining price scale, recent relative price context, and short-horizon forecast information leads to stronger performance compared to relying on a single feature family.

Evidence Used

• Results from the study comparing different feature combinations

Methods / Approach

• Comparison of multiple state representation methods in a deep RL model for energy trading

Limitations Stated by the Authors

• The study only considers one specific environment (HydroDam) and one type of agent (fixed Double DQN)

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The model does not account for potential limitations in the generalizability of the results to other environments, agents, or reward functions.

Open Questions

• What is the impact of these findings on RL models for energy trading in different environments and with different agents?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper does not discuss the potential implications of their findings for other areas beyond energy trading.

Raw Abstract / Excerpt

Energy trading decisions depend not only on current market prices, but also on expected future market conditions, and operational constraints. This makes the state representation given to a reinforcement learning agent an important design choice. We study this in HydroDam, a pumped-storage arbitrage environment, using a fixed Double DQN agent. The environment, action space, reward function, network, and training protocol are kept fixed; only the market features are changed. We compare absolute price/calendar features, relative features that compare current prices with recent market history, forecast features, and all combinations of these three feature families. Policies are trained and selected using 2007--2011 Belgian day-ahead prices and evaluated on two test settings: a later same-market test set from 2012--2025 and 39 other ENTSO-E market zones. Absolute features only reaches 28.8% on the test set and a median 5.7% across zones. Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median. Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%. These results suggest that state representation is not a minor preprocessing choice in storage-trading RL, but a central part of the policy design: robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.