In iron-based superconductors, a spin-density-wave (SDW) magnetic order is suppressed with doping, and unconventional superconductivity appears in close proximity to the SDW instability. The optical response of the SDW order shows clear gap features, substantial suppression in the low-frequency optical conductivity, alongside a spectral weight transfer from low to high frequencies. Here, we study the detailed temperature dependence of the optical response in three different series of the Ba122 system [Ba1−xKxFe2As2, Ba(Fe1−xCox )2As2, and BaFe2(As1−xPx )2]. Intriguingly, we find that the suppression of the low-frequency optical conductivity and spectral weight transfer appear at a temperature T ∗ much higher than the SDW transition temperature TSDW. Since this behavior has the same optical feature and energy scale as the SDW order, we attribute it to SDW fluctuations. Furthermore, T ∗ is suppressed with doping, closely following the doping dependence of the nematic fluctuations detected by other techniques. These results suggest that the magnetic and nematic orders have an intimate relationship, in favor of the magnetic-fluctuation-driven nematicity scenario in iron-based superconductors.