When I use my cane, my shoulders are really lopsided

So basically, I occasionally use a cane when my SI joint is being stupid or if I'm having bad nerve pains in my hips/legs. I noticed today that when I'm using it (usually in my right hand), my shoulder is a good 3-4 inches higher than the other. My cane hits right at my wrist, so I thought it was the right size, but I guess I could be mistaken. Has anyone else had this problem and/or know how to fix it?