In order to increase the market value and quality of wheat, it is important to separate different types and determine the amount of foreign matter using the visual properties of durum and bread wheat. In this study, the extreme learning machine (ELM) algorithm, which is often preferred in real-time applications, was used to make classifications using features obtained from images containing the wheat kernel and foreign matter. The feature selection process was applied to remove the irrelevant ones from the obtained 236 features. In addition, the Harris hawks’ optimizer (HHO), a novel method in the literature, and the particle swarm optimizer (PSO), one of the well-known algorithms, were used to improve the ELM model. As part of this study, new models called HHO-ELM and PSO-ELM were created and compared with the original ELM model and other artificial neural networks (ANNs) studies published in the literature. As a result, in comparison with other models, the optimized ELM models demonstrated good stability and accuracy, having 99.32% in binary classification and 95.95% in multi-class classification.