To fully exploit the spectral information of modern geostationary satellites, this work proposes a deep learning framework using convolutional neural networks (CNNs) and attention mechanism for 5-min ground-level global horizontal irradiance (GHI) and direct normal irradiance (DNI) estimations. The inputs are spectral satellite images with the target ground station in the center, and the labels are irradiance measurements normalized by their clear-sky estimations. The use of CNNs and attention mechanism aims to better extract the spatial information for estimating ground-level solar irradiance. To improve the modeling efficiency, only a subset of spectral bands is selected based on correlation analysis, which has comparable performance with the usage of all satellite bands. The results show that the proposed method produces GHI estimation with a normalized root mean squared error (nRMSE) of 20.57% and a normalized mean bias error (nMBE) of −2.04%, and the DNI estimation has an nRMSE of 23.63% and the nMBE is 0.36%. Compared with the national solar radiation database (NSRDB), GHI and DNI estimations of the proposed method has the nRMSE reduction of 5.15% and 13.77%, respectively. Meanwhile, the proposed models generally yield better GHI and DNI estimations under different intervals of clear-sky index than NSRDB. The combination of deep learning and remote sensing shows potential in better extracting the cloud information via multispectral satellite images, which can better support solar resource assessment, especially for cloudy conditions.