By work of Looijenga and others, one understands the relationship between Geometric Invariant Theory (GIT) and Baily–Borel compactifications for the moduli spaces of degree-$2$ $K3$ surfaces, cubic fourfolds, and a few other related examples. The similar-looking cases of degree-$4$ $K3$ surfaces and double Eisenbud–Popescu–Walter (EPW) sextics turn out to be much more complicated for arithmetic reasons. In this paper, we refine work of Looijenga in order to handle these cases. Specifically, in analogy with the so-called Hassett–Keel program for the moduli space of curves, we study the variation of log canonical models for locally symmetric varieties of Type IV associated to $D$-lattices. In particular, for the $19$-dimensional case, we conjecturally obtain a continuous one-parameter interpolation between the GIT and Baily–Borel compactifications for the moduli of degree-$4$ $K3$ surfaces. The analogous $18$-dimensional case, which corresponds to hyperelliptic degree-$4$ $K3$ surfaces, can be verified by means of Variation of Geometric Invariant Theory (VGIT) quotients.