Real-time detection of driver behaviors, fundamental for autonomous vehicles, is crucial for preventing accidents and enhancing traffic safety. Traditional methods, relying on manual observations or sensor-based monitoring, are increasingly being replaced by automated solutions using machine learning and computer vision technologies. This study aims to improve the classification of driver behaviors through the integration of deep learning models with LSTM layers. A multi-class driver behavior dataset, including images of safe driving, phone conversations, texting, turning, and other distractions, was used. Data processing involved cross-validation to ensure reliable performance evaluations. Various deep learning models such as VGG19, ResNet50, MobileNetV2, InceptionV3, DenseNet201, and InceptionResNetV2 were employed, each integrated with LSTM layers to create hybrid architecture. LSTM’s ability to capture temporal dependencies enabled more accurate behavior classification. Model performances were evaluated using accuracy, precision, recall, F1-Score, Log Loss, and ROC-AUC metrics. Experimental results demonstrated that LSTM integration significantly enhanced classification performance. InceptionResNetV2 and MobileNetV2 also achieved strong results with LSTM, while DenseNet201 was the most accurate at 94.77%. Road safety applications and real-time monitoring systems can benefit from these findings. In addition, this study contributes to the development of driver monitoring systems based on machine learning, which has the potential to enhance safety in autonomous vehicles.