Accurate solar irradiance forecasting is essential for improving the economics and stability of building energy systems. Although existing multimodal deep learning methods have improved forecasting accuracy by integrating meteorological data with satellite cloud images, they still suffer from insufficient cloud feature extraction, inadequate modeling of temporal dependencies, and limited capability of uncertainty quantification under complex weather conditions. To address these issues, this study proposed an end-to-end multimodal deep learning framework for intra-day solar irradiance forecasting by integrating meteorological data with satellite cloud images. Specifically, Mamba blocks were employed to efficiently capture complex temporal dependencies in meteorological sequences, while a global-local dual-branch image encoder based on frequency-mixed and feature-enhanced convolutional neural networks (CNNs) was developed to extract multi-scale cloud features. Furthermore, a cross-modal interaction mechanism and an encoder-decoder-based future modeling strategy were introduced to enable deep multimodal fusion. Dual-output heads are further employed to jointly generate point forecasts and prediction intervals. Experimental results show that the proposed method consistently outperforms recent advanced multimodal benchmarks across different forecasting horizons and weather conditions, demonstrating strong robustness and generalization capability. In particular, the proposed model achieves relative reductions of 16.39%–21.18% in MAPE and 10.11%–14.31% in nRMSE in point forecasting, while achieving a favorable balance between reliability and sharpness in interval forecasting. Ablation studies and cross-site experiments further verify the effectiveness, robustness, and applicability of the proposed model. This work provides an effective and robust solution for high-accuracy intra-day solar irradiance forecasting.