The joint space-time structure of climate variability across a wide range of scales is investigated from a spectral perspective. Specifically, global near-surface temperature fields - taken from the TraCE-21ka paleo-climate simulation, from CMIP5 historically forced and control simulations, as well as from the NOAA 20th Century Reanalysis - are decomposed into spherical harmonic and frequency components. The obtained spectral space-time scaling behaviour is discussed and compared to that of simple stochastic-diffusive energy balance models, focusing on (a) the timescale-dependence of the effective number of spatial degrees of freedom, and (b) the impact of external climate forcing on the space-time scaling behaviour of the variability.