In this case, instead of defending yourself at your end immediately go to the head of your college or talk about this with your parents. Make sure your parents raise their voices against this activity and protect your image and yourself.
If the matter still doesn’t come under control go to the police and ask for their help and protection.