Here we evaluate five atmospheric reanalyses in an Arctic gateway during late summer. The reanalyses include ERA5, ERA-Interim, Japanese 55 year Re-Analysis (JRA-55), Climate Forecasting System Reanalysis-version 2 (CFSv2), and Modern Era Retrospective analysis for Research and Applications-version 2 (MERRA-2). We use observations from 50 radiosondes launched in the Fram Strait around 79-80°N, between 25 August and 11 September 2017. Crucially, data from 27 radiosondes were not transmitted to the Global Telecommunications System and therefore not assimilated into any reanalysis. In most reanalyses, the magnitude of wind speed and humidity errors is similar for profiles with and without data assimilation. In cases without data assimilation, correlation coefficients (R) exceed 0.88 for temperature, wind speed, and specific humidity, in all reanalyses. Overall, the newly released ERA5 has higher correlation coefficients than any other reanalyses as well as smaller biases and root-mean-square errors, for all three variables. The largest improvements identified in ERA5 are in its representation of the wind field, and temperature profiles over warm water.