How to Turn RL Into Supervised Learning?

The end game for creating intelligent agents like robots or any RL agent is to clone behavior. The only thing that you need to go beyond just cloning is also understand what it means to solve a task. One way to turn RL into supervised learning was using the upside down reinforcement learning formulation. But then that was not general enough, so I combined these two insights together to get the idea.