The inherent uncertainty of the weather system frequently causes solar ramp events, limiting the efficient integration of solar power into the grid. Although the use of multi-modal data sources with deep learning models has improved solar forecasting accuracy as measured by statistical error metrics, the common reliance on Euclidean distance loss functions often fails to accurately extract the characteristics of ramp events, which is essential for grid operators to integrate solar power. Therefore, we introduced the Relaxed Shape Alignment and Temporal Distortion Loss (RSTLoss), a novel differentiable loss function designed for deep learning-based solar prediction models. The effectiveness of RSTLoss is thoroughly verified through extensive experiments, showing improvements of up to 7.48% in the Ramp Score and 13.17% in the Temporal Distortion Index compared to existing loss functions. These findings indicate that RSTLoss significantly enhances ramp prediction accuracy while also exhibiting excellent model compatibility.