No removing from the registry entry manually would not help because I have tried this before and exe if running in the background would be constantly monitoring that and as soon as they are not there would write it down again.

Now stopping the exe would not be possible and if you kill the exe it would start the process before getting stopped and so that would again keep the exe running.