Mar 27, 2025
Spike-and-slab prior is a popular choice for sparse parametric models \[ \theta \sim p_0 \delta_0 + (1-p_0) p_\theta(\theta;\eta_1) \quad \text{and} \quad X \sim p_X(X;\theta) \]
We commonly approximate the posterior by \[ Z \sim \mathrm{Binomial}(p_0) \; \Rightarrow \; \theta \mid Z \sim p_\theta(\theta;\eta_Z) \]
\[ \mathrm{minimize}_q \; \mathrm{KL}(q_\theta(\theta)q_Z(Z) \mid p(\theta, Z \mid X)) \] assuming \(\theta \perp Z \mid X\)
This work shows that \(p(\theta \mid X)\) and \(p(\theta)\) both belongs to the same type of exponential family
Then, solves the following optimization \[ \mathrm{minimize}_q \; \mathrm{KL}(q(\theta) \mid p(\theta \mid X)) \] where \(q\) belongs to that exponential family
This avoids the over-restrictive assumption that \(\theta \perp Z \mid X\)