To add to this, the compile time is usually in the range of a few ms each but often when you need to compile a new shader it's because a new entity appeared and there are likely several new shaders associated with that entity that you have to compile at once, so you could see a 100-200ms delay on the rendering for larger jobs.

Often times if you change settings you'll trigger recompilation of some shaders. This can result in worse performance which people often attribute to their setting change (so they revert it/change it some more, triggering more shader compiling). So, make sure you play a bit once you've changed settings before judging the performance.

Using something like MangoHud which displays CPU/GPU load and frame time graphs (and about a million other customizable things) you can generally see when you're having shader stuttering. Some games (Path of Exile 2, for example) have in-game performance monitor tools which will specifically tell you when shaders are compiling and how many are in the queue.