Link to Colorado State University Home Page

Robust Reinforcement Learning Control for Building Energy Systems

Contents

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

This material is based upon work supported by the National Science Foundation under Grant No. 0245291. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

People

Current Project Members: Past Project Members:

Back to Contents

Funding

This work has been funded by NSF Grants

Back to Contents

Project Summary

Most control problems involve systems that are either partially unknown or whose true nature varies over time. Modern robust control methods offer theory and techniques for analyzing and designing controllers with guarantees of stability, but such robust controllers are by necessity less aggressive than they could be if full knowledge of the system were available.

Our approach is to add to the robust control framework a family of reinforcement learning algorithms based on artificial neural networks and designed to optimize the performance of a controller through experience with the actual system. We have showed through analysis and experiments that stability can be guaranteed even as the agent learns. This is some of the first work we know of to address K. Narendra's call for new approaches: "It is precisely in problems where the system has to adapt to large uncertainty that controllers based on neural networks will be needed in practical applications. For such problems, new concepts and methods based on stability theory will have to be explored." (Narendra, 1990)

The objectives of our current work are the following:

Our work will ultimately lead to procedures by which an initial robust controller is designed with as much knowledge of the system to be controlled as is available, but whose performance is optimized through interactions with the actual system, complete with guarantees of stability while learning! The significance to the reinforcement learning field is one of practicality. very important to control engineers who require guarantees that a learning approach will not injure people or systems. Also of practical importance is the initial robust controller which alleviates a common complaint about the tremendously long times needed for a reinforcement learning agent to acquire a good policy. Our learning agent must only learn the unknown or changing parts of the system. The significance to the control of energy systems in buildings is that robust MIMO controller design results in control strategies that vary multiple actuators to minimize energy cost while maximizing tracking performance in ways that are impossible with the single-input, single-output (SISO) approach common in modern HVAC systems. For an energy system of complexity adequate for a building, it is impossible to have a complete model of the system, so an approach like our robust reinforcement learning approach is necessary to fine-tune the control strategy for the true system and to track changes over time.

We are an interdisciplinary team consisting of a specialist in robust control from the Electrical & Computer Engineering Department, a specialist in reinforcement learning for neural networks from the Department of Computer Science, and a specialist in design, modeling and control of HVAC systems from the Mechanical Engineering Department. This interdisciplinary approach will further advance the state-of-the-art in the theory of robust reinforcement learning control design, demonstrate these new methods on an experimental HVAC system and provide much needed improved methods for controlling HVAC systems in buildings. Eventual wide spread implementation of these schemes in buildings around the world will reduce energy consumption, improve comfort, and extend equipment life.

Back to Contents

Publications

2008

2007

2006

2005

2004

2002

2001

2000

1999

1997

1996

Back to Contents

Patents

Back to Contents

Talks

Back to Contents

HVAC Physical Testbed

This is a photograph of the physical heating system we have constructed for running experiments. This is located at the Solar Energy Applications Lab at Colorado State University, Fort Collins, CO.
Here is a close-up of the control hardware, including the PC that drives the system.

Back to Contents

Other On-Line Resources

IQC

Convex Optimization

LMI

General Control Theory

Reinforcement Learning

Reinforcement Learning with Control

Back to Contents