In order to meet the requirements for the accuracy and efficiency of the insulator skirt damage detection in the field of UAV power inspection, a lightweight insulator skirt damage detection algorithm based on a double-layer pyramid structure is proposed in this paper. To avoid the filtering of useful information, the proposed method utilized both high-level and low-level semantic features from images. Simultaneously, multi-channel feature extraction is used to increase the feature expression ability of the model. In addition, deep separable convolution is employed to optimize the performance of the convolution kernel for real-time operation on a low-power embedded front-end. Experimental results show that the proposed algorithm has good accuracy, real-time performance and robustness for the insulator skirt damage detection under different working conditions.