Attenuators do have an impact, and the components and cables used influence that impact, but in the audio band their use should not generally be noticeable except for the reduced signal level. Of course, other combinations of source, attenuator, cable, and load impedances will yield different results, and some combinations might have much greater impact, but those would have to be pretty special cases IME/IMO.

The Signal Chain: How do noise and distortion propagate through my system?
https://www.audiosciencereview.com/forum/index.php?threads/the-signal-chain-how-do-noise-and-distortion-propagate-through-my-system.33358/#post-1165118