Consider the feedback composition with two LTI systems:
Assume the frequency response of S1 is H1, of S2 is H2, and of S is H. Then assume that
x = exp(i w t).
The output must be
y = H(w)x
Since this is itself a complex exponential, it must be true that
z = H2(w)y = H2(w)H(w)x
Hence
u = x - z = x - H2(w)H(w)x = (1 - H2(w)H(w))x
which is also a complex exponential. Since y = H1(w)u, it must be that
y = H1(w)(1 - H2(w)H(w))x
Since y = H(w)x,
H(w)x = H1(w)(1 - H2(w)H(w))x
Eliminate x and solve for H to get
H(w) = H1(w)/(1 - H2(w)H1(w))
when (1 - H2(w)H1(w)) is not zero. This gives the frequency response of the feedback system in terms of those of the component systems.