Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

Source

arXiv

arxiv.org/abs/2606.27032v1

Read the source

Source

Plain-Language Brief

Researchers studied how the representation of a state given to a reinforcement learning (RL) agent impacts energy trading decisions. They used a fixed Double DQN agent in a pumped-storage arbitrage environment and compared different feature families for state representations. Absolute features only performed poorly, while combining feature families led to significantly better results, suggesting that state representation is an important part of the policy design in storage-trading RL.

What the Document Says

  • The study was conducted on HydroDam, a pumped-storage arbitrage environment using a fixed Double DQN agent.
  • The environment's action space, reward function, network, and training protocol remained constant. Only the market features were changed for comparison.
  • Four feature families were compared: absolute price/calendar features, relative features, forecast features, and combinations of these three.
  • Absolute features only reached 28.8% on the test set and a median of 5.7% across zones.
  • Relative-only and forecast-only states also underperformed a rolling price-score heuristic in the cross-zone median.
  • Combining feature families resulted in better performance: absolute + relative reached 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reached 55.6% and 47.5% respectively.
  • The findings suggest that state representation is crucial in storage-trading RL, requiring a combination of price scale, recent relative price context, and short-horizon forecast information.

Key Concepts Mentioned

  • Reinforcement Learning
  • Deep Q-Network (DQN)
  • Pumped-storage arbitrage
  • State representation
  • Energy trading
  • Feature families

Claims Made by the Document

  • Absolute features only performed poorly in energy trading RL.
  • Combining feature families led to significantly better results in energy trading RL.
  • State representation is an important part of the policy design in storage-trading RL.

Evidence Used

  • Results from testing various feature families on HydroDam's pumped-storage arbitrage environment.
  • Performance metrics such as test set scores and cross-zone medians.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

  • The study only examined a fixed Double DQN agent and a specific environment (HydroDam).
  • Other feature combinations or agents may yield different results.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The model may not generalize to other reinforcement learning problems beyond energy trading.
  • The model's performance could be influenced by the quality and representativeness of the training data.

Open Questions

  • What other environments, agents, or feature combinations could be explored in this context?
  • How does the performance of these methods compare to other RL approaches for energy trading?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The research focuses on the impact of state representation in a specific environment (HydroDam) and agent (fixed Double DQN).
  • The study examines the effectiveness of different feature families for state representation.
  • Performance is measured by test set scores and cross-zone medians.

Raw Abstract / Excerpt

Energy trading decisions depend not only on current market prices, but also on expected future market conditions, and operational constraints. This makes the state representation given to a reinforcement learning agent an important design choice. We study this in HydroDam, a pumped-storage arbitrage environment, using a fixed Double DQN agent. The environment, action space, reward function, network, and training protocol are kept fixed; only the market features are changed. We compare absolute price/calendar features, relative features that compare current prices with recent market history, forecast features, and all combinations of these three feature families. Policies are trained and selected using 2007--2011 Belgian day-ahead prices and evaluated on two test settings: a later same-market test set from 2012--2025 and 39 other ENTSO-E market zones. Absolute features only reaches 28.8% on the test set and a median 5.7% across zones. Relative-only and forecast-only states also stay below a rolling price-score heuristic in the cross-zone median. Combining feature families is much stronger: absolute + relative reaches 49.9% on the test set and a 39.8% cross-zone median, while absolute + relative + forecast reaches 55.6% and 47.5%. These results suggest that state representation is not a minor preprocessing choice in storage-trading RL, but a central part of the policy design: robust transfer requires combining price scale, recent relative price context, and short-horizon forecast information, rather than relying on any single feature family.