The speaker reflects on giving up a significant number of books during past moves, mentioning that the number of books they let go was around five times the amount they currently own. They express sentimentality towards the lost books, emphasizing the stories and lessons they contained, and how those books represented future possibilities and personal growth. The speaker acknowledges that if they had kept every book they ever owned, it could have filled a library.