Even if it did not penetrate the clothing, would you still not get the impact from the bullet hitting you?

Much the same way the vest does not allow the bullet to penetrate but you still get the impact, but spread over a wider area.

I still believe that it would stop, if only temporaily, the bad guy.