Twist anticlockwise and it will release. Its held in by 3 metal tangs. And there is a trim piece that just clips in.
My wife pulled my mirror off (again) a couple of weeks without even twisting it. It had got loose with us both using the car and changing the mirror position over and over again for many years (it was her car).