The Yom Kippur Sacrifices

The Yom Kippur sacrifices are described as a toning for the sins, trespasses, and iniquities of the people. But on an annual basis there was this special sacrifice where two goats were brought, one was sacrificed, one was taken outside the camp. It's almost as though their sins are piling up here in the tabernacle court. And then once a year they're taken out. Is that legitimate?