Eight atmospheric regional climate models (RCMs) were run for the period September 1997 to October 1998 over the western Arctic Ocean. This period was coincident with the observational campaign of the Surface Heat Budget of the Arctic Ocean (SHEBA) project. The RCMs shared common domains, centred on the SHEBA observation camp, along with a common model horizontal resolution, but differed in their vertical structure and physical parameterizations. All RCMs used the same lateral and surface boundary conditions. Surface downwelling solar and terrestrial radiation, surface albedo, vertically integrated water vapour, liquid water path and cloud cover from each model are evaluated against the SHEBA observation data. Downwelling surface radiation, vertically integrated water vapour and liquid water path are reasonably well simulated at monthly and daily timescales in the model ensemble mean, but with considerable differences among individual models. Simulated surface albedos are relatively accurate in the winter season, but become increasingly inaccurate and variable in the melt season, thereby compromising the net surface radiation budget. Simulated cloud cover is more or less uncorrelated with observed values at the daily timescale. Even for monthly averages, many models do not reproduce the annual cycle correctly. The inter-model spread of simulated cloud-cover is very large, with no model appearing systematically superior. Analysis of the co-variability of terms controlling the surface radiation budget reveal some of the key processes requiring improved treatment in Arctic RCMs. Improvements in the parameterization of cloud amounts and surface albedo are most urgently needed to improve the overall performance of RCMs in the Arctic. © Springer-Verlag 2007.