To reduce the occurrence of traffic accidents caused by driver distraction, a behavior detection method based on histogram of oriented gradient (HOG) and support vector machine (SVM) was proposed in this paper. In the algorithm, interesting region of driver was detected first from the video images. Then the image was enhanced, smoothed and normalized. The histogram of oriented gradient was used to extract the feature of the target image. The cross-validation method was used to optimize the SVM parameters, and then used for the classification of driver behaviors. In the experiments, the proposed method was compared with classical SVM and the local binary patterns feature based SVM algorithms. The results show that, the proposed method can obtain better classification accuracy.