Contributed Talk "Challenges of Online Decision-Making in Energy Systems" at "New Horizons of Online Learning" workshop at ACML 2023
Date: 11th November 2023, 17:20-17:35
Room: A203, Conf. Center, Acıbadem University, İstanbul, Turkey
Workshop Organizers: Kohei Hatano, Eiji Takimoto, Atsuyoshi Nakamura and Junya Honda
Abstract
As energy systems undergo transformative changes, driven by the integration of renewables and by ever-increasing energy demand, it becomes challenging to control decentralized interconnected renewables, storage units, and loads. In private households or within companies, the interconnection of such systems forms small-scale energy networks, called decentralized micro-grids. This challenge stems from the vast space of possible decisions that must be made promptly due to the uncertain and non-stationary nature of micro-grids. Uncertainty and non-stationarity result from intermittent renewables, fluctuating loads, and the potential for energy trading with the utility grid.
Furthermore, decision-making is confined to feasible operations that may change over time, to ensure system constraints. For instance, the space of allowed battery charging and discharging operations depends on the current state of the battery. Incorporating multiple objectives (e.g., reducing the carbon footprint, or minimizing operating costs) adds further complexity. Consequently, human micro-grid operators cannot make such decisions, necessitating the use of automated online decision-making strategies.
Existing decision-making strategies in energy grids rely on traditional control theory, rule sets based on domain knowledge, time series forecasting, operational research, or reinforcement learning (RL). RL, in particular, has shown promising performance in modeling energy grids as Markov Decision Processes (MDP). However, existing work falls short in addressing the non-stationarity and uncertainty of decentralized micro-grids while achieving near-optimal and timely decisions.
This talk reviews the state of the art in online decision-making for micro-grids, emphasizing both single- and multi-objective contexts. We discuss how principles from RL can help addressing these challenges. Then, we present results from our ongoing work, where we framed the problem of minimizing operational costs as an MDP with continuous and interdependent action space. We adapted Proximal Policy Optimization, a well-known Deep RL method, to this MDP by incorporating a projection layer and soft penalties to ensure the feasibility of actions. We demonstrate the superior performance of our approach in small-scale micro-grids over existing methods based on real-world data. Specifically, our approach offers more fine-granular control than methods with discrete action spaces, resulting in a smaller deviation from the global optimal solution.
Moreover, we will talk about other practical use cases and applications of (Deep) RL in energy systems.
