Sultan Yerumbayev's blog

SARSA controller setup After we figured out the environment, it is important to understand how to apply RL concept on it. RL works in continuous as well as in discrete time state-space environments. MLE+ dictates the conditions in which we will operate: we will construct discrete state-space using variation of three parameters – temperature set point, actual temperature of a zone and electric heater. I did it in straightforward way – having the vectors of values that can be possible for each variable (e.g. heater can have loads between 0 and 12000KW); made combinations of all three vectors to define all state-spaces within which our building model can operate (more importantly we are trying to not define state-spaces that not possible for a real building systems). Along creation of states it creates Q matrix for three actions meaning increase, decrease and don't change temperature set point using reward function described bellow: As you may notice reward function appreciates...

So, how to model an office building to simulate the work of our controller? In short, I have used the following list of programs: Matlab, EnergyPlus and MLE+ in tandem. First things first, EnergyPlus - is an building simulation engine that will allow you to modulate maybe not all but most of the physical phenomena running inside real physical structures, including heat transfer and temperature spread. Even though it is a quite a hard to understand how to use EnergyPlus if you are not expert (maybe even for civil engineers), you can always download already designed models of a buildings like I did), which can be found on energy.gov site. Therefore, I took one floor three office medium building EnergyPlus model that comes with and .idf and weather files. Basically, the building has three rooms with electric radiant heating floors and one window and some ventilation system. Simple schematics shown on a picture below: As you may guess, I want to be able to test some controllers on th...

How Machine Learning and Reinforcement Learning applied for physical systems like building heating? Some intro ... Many civil engineering and computer science researchers are pushing forward the idea of applying new intelligent methods of control of energy consuming systems such as heating, ventilation and air conditioning(HVAC) in non-residential or commercial buildings. The motivation behind improvement of performance of these systems is evident, they account for almost 40% out of total building energy demand. Since this was a question of my university study I have already done some literature review on this topic and outline the main research directions. Even though some research directions may not be seem directly related, but they are have potential to used for our purpose. Thus, the following are main research topics related to the improvement of energy and comfort performance of HVAC systems: Statistical methods. Record previous loads, building statistical models trying ...