Rule Question

Good Day, I was wondering if it is a foul if while addressing the white I dig my fingers into the cloth while addressing the white thus moving the cloth which caused the green to move quite a bit because it fell into a spotting hole in the cloth. I called a foul because I seen the green move, he said I did not touch the green, I said it did not move on its own. I tried to find the answer in the rules but could not. Thank You,,,

Rule Question