State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jesper Klicks, Sander Vržina, Vincent François-Lavet
• Venue: arXiv: cs.LG
• Published: 2026-06-25
• Collected: 2026-06-26
• URL: http://arxiv.org/abs/2606.27032v1
• PDF: https://arxiv.org/pdf/2606.27032v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers studied the impact of state representation in deep reinforcement learning (DRL) for energy trading decisions, using a fixed Double DQN agent in the HydroDam environment. They found that combining absolute price/calendar features, relative features comparing current prices with recent market history, and forecast features significantly outperformed relying on any single feature family, suggesting that state representation is crucial in storage-trading RL.

What the Document Says

• The paper focuses on energy trading decisions and the impact of state representation on a reinforcement learning agent. The study was conducted in the HydroDam environment using a fixed Double DQN agent.
• They compared four feature families: absolute price/calendar features, relative features comparing current prices with recent market history, forecast features, and combinations of these three feature families.

Key Concepts Mentioned

• Energy trading
• Deep Reinforcement Learning (DRL)
• HydroDam
• Double DQN agent
• State representation
• Feature families

Claims Made by the Document

• Robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.

Evidence Used

• Absolute features only reaches 28.8% on the test set and a median 5.7% across zones.
• Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median.
• Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Energy trading decisions depend not only on current market prices, but also on expected future market conditions, and operational constraints. This makes the state representation given to a reinforcement learning agent an important design choice. We study this in HydroDam, a pumped-storage arbitrage environment, using a fixed Double DQN agent. The environment, action space, reward function, network, and training protocol are kept fixed; only the market features are changed. We compare absolute price/calendar features, relative features that compare current prices with recent market history, forecast features, and all combinations of these three feature families. Policies are trained and selected using 2007--2011 Belgian day-ahead prices and evaluated on two test settings: a later same-market test set from 2012--2025 and 39 other ENTSO-E market zones. Absolute features only reaches 28.8% on the test set and a median 5.7% across zones. Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median. Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%. These results suggest that state representation is not a minor preprocessing choice in storage-trading RL, but a central part of the policy design: robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.