The Brothers and Sisters of the Lord Are Not Sons and Daughters of Mary.

The only son of mary is jesus ah. So that's why jesus is even called the son of mARY. But just because jesus was adopted by joseph, it doesn't follow that mary subsequently adopted these children. They were never technically her sons and daughters. Just because she was married to their father, that did not automatically make them her sons and daughter.