How to Debug a Debugger?

The first version, the daliminy, maybe we were lucky. Somehow it walked well, pretty fast. But there were some burgs that i spent two weeks or more to to understand what was happening. Training those large models is very, very hard. You have peaks that happen randomly. And then sudn you have like big piks, and i go, you fin, i'm going to restort before and i coue nowit went through bit, like five minutes later, another peak. It's not very s because one model does n't necessarily fit on one device,. so you neet to spread the weights across the other devices. Thers onlya sait