The speaker was frustrated with his work for the past 20 years because he didn't understand why he was getting frustrated, despite loving his work. This led to a conversation with others and the realization that he was not working in his genius. He often found himself doing tasks outside of his genius, which drained him of energy and joy.