I think you are using send and receive functions in you socket program.Instead try with sendto ,recvfrom functions you will get the address and port of the other informations in the packet .

man recvfrom
man sendto