AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Code Book Vectors and a Vector Quantizing Hack
The DVA is like another discrete auto encoder thing but it's a little bit different than the VQVA. The hack is what's called the gumbell softmax which is sort of a trick for approximating a discrete distribution with a continuous distribution because the problem is you can't really differentiate through sampling from a discrete distribution so they try to approximate it. And yeah if I my end would be dead so now you have like a probability distribution over your code book vector. So sounds a bit hacky but I trusted the networks.