With the rapid development of Internet online shopping, automatic classification of product images has become an interesting research topic. In this paper, an accurate classification method for women dress images are investigated. Firstly, 40 000 product images were crawled from the Vipshop online shopping websites, which all are annotated by several experts. Then, several baselines using deep convolutional networks were provided. Finally, a new loss function was introduced and the multi-task learning method was used to improve the classification accuracy. With the comparison of different strategies, the experimental results show that the proposed method can obtain higher classification accuracy.