What if someone know's / has contacts who work at your ISP to which your IP belongs then surely they can find out the personal details right?

I remember this person saying they have contacts in certain ISP, but trying to be clever that person did not reveal which one. Obviously from the WHOIS u can find the ISP details to a particular IP address.