Let's simplify by using just one level of indirection. Let's also represent integers and pointers in two bytes. Suppose you have in memory:

1000 00 42
1002 00 31
1004 10 00

representing 42 at 1000 and 31 at 1002, and 1000 at 1004.
1000 and 1002 are integers and 1004 is an int*, let's call it p and initialise it to point at 1000.

The compiler knows that p points at an int, and that sizeof(int)=2.
So when you do p++, the compiler thinks aha, he doesn't just want to add 1 to the pointer, that would be useless, because it points you half way up an integer and if you were to dereference it, you would get 4200 instead of 0031. What he wants is to point to the next integer in memory, which is at p+sizeof(int), which is 1002.

So if you increment p by 1, this means it increments p by the size of one integer, which is actually 2 bytes. So after p++, memory will contain:

1000 00 42
1002 00 31
1004 10 02

note 1002 rather than 1001 at 1004.

So let's say 1006 is another int*, let's call it p1, that still points at the first int:

1000 00 42
1002 00 31
1004 10 02
1006 10 00

Now you want to do p-p1. The compiler sees that you're manipulating int pointers and so the subtraction is done within the context of the fact that you are operating on integers, NOT on individual bytes of memory. The actual difference in bytes between p and p1 is of course 2, but because sizeof(int)=2 this means that the difference between those two pointers is actually one complete integer, which is 2 bytes. So the compiler divides the result(2) by sizeof(int)(also 2) which gives 1.

So in a sense your statement "the point is the difference between ptr & p in my compiler is 2" is incorrect. The difference between two adjacent integers is one integer, and that's the context in which the language is operating. If you want to deal with individual bytes of memory that's fine; you just have to use pointers that point at things that are only 1 byte long (char*, for example).

The other point about adding sizeof(thing) to a pointer when you do p++ is that you don't necessarily know the size of the thing you're using. If you have a horribly complicated structure with nested structures and unions all over the place then it could take you ages to figure out how big they are, and you're completely stuffed if the structure size changes in development because you'd have to go through all those structure size constants in your application so that pointer arithmetic still works, whereas if the compiler handles all this for you then all you have to do is "p++;" and whatever p is pointing at, after that statement it will point to the next one in memory. Similarly if you know a pile of memory contains contiguous objects then to find out how many objects are there you just do p-p0 where p is the pointer to your object and p0 is the pointer to the start of that memory, and the answer is the number of objects, rather than some big number that's completely useless unless you know the size of the object.