In this study, bisphenol A-type epoxy (ER) and epoxy phenol novolac resins (EPN) with two amine-type curing agents (KH 816 and IPOX EH 2041) having different total amine values were selected. The mechanical properties and density of the EPN resin were found higher in the case of both curing agents. EPN was modified with polystyrene (EPN-PS) and used as a matrix using an IPOX hardener. Nano- and hybrid composites were prepared with pristine and modified with tetramethylammonium chloride nano montmorillonite (NC and MNC), and red mud waste (RMW). The chemical structure was elucidated by the Fourier transform infrared (FT-IR). The composites were characterized by scanning electron microscopy and x-ray diffraction. Appropriate PS and NC ratio was accepted as 4% and 2% by weight, respectively. Curing degrees calculated from the FT-IR spectra of ER, EPN, and EPN-PS were 99.3%, 99.9%, and 86.5%, respectively. A comparative study of the two binary systems; (i) 2 wt% NC-(15–35) wt% RMW; and (ii) 2 wt% of MNC-(15–35) wt% of RMW on the mechanical, thermal, and flammability properties of EPN-PS matrix was carried out. The appropriate RMW ratio was 30 wt%. Hybrid composites with 25–30 wt% RMW can be considered self-extinguishing materials.