The Importance of World Building in Fantasy

Pratchett has kind of gone on the record as disappointed that the world has kind of relegated fantasy and by extension sci-fi off to the side of literary fiction. It pisses me off that fantasy is unregarded as a literary form. When you think about it fantasy is the oldest form of fiction. What were the storytellers of old doing when they talked about the beginnings of the world they were weaving fantasies. I do not have a working knowledge but like I don't care about Dota too.