Climate models have substantial biases in the climatological latitude of the Southern Hemisphere eddy-driven jet and the time scale of annular mode variability and disagree on the jet response to climate change. Zonally symmetric dry dynamical cores are often used for idealized modeling of the jet response to forcing and its sensitivity to model setup changes. The limits to which these models represent the key mechanisms that control the jet in complex models or the real world have not been systematically investigated. Here we show that substantial intermodel differences in jet latitude and strength can arise from differences in dynamical cores and resolved topography. Including topography and a more realistic surface drag in a dry model substantially alters the jet response to changes in drag strength. Using real-world maps, enhanced drag over land shifts the jet poleward, whereas enhanced drag over the ocean leads to an equatorward shift. No universal relationship between annular mode time scale and forced response emerges in the dry model with topography. These results suggest that zonally symmetric models with Rayleigh drag lack important mechanisms that control the behavior of the midlatitude jet in coupled climate models. A dry model with topography and quadratic surface drag can fill this gap in the model hierarchy.