Why does my ping fluctuate so much, and how can I fix it?

Whenever I do a ping test multiple times online it will always fluctuate like crazy. On average my ping sits somewhere around 40ms and my jitter is usually good as well, but sometimes my ping and jitter will be in the hundreds. And this directly reflects my online gaming performance, because sometimes I'll have a good solid 4bar connection and other times it will fluctuate between a 3 and 4bar like crazy. And I know it's not the host because I'll be the only one doing it. So why is this happening, and how can I fix it?