return b[size-1][size-1]; does NOT return the whole of array b, it just returns the single integer at b[size-1][size-1], i.e. b[2][2], which is in fact undefined because only b[0..1][0..1] are defined. The four integers at b[0..1][0..1] (in the case that size=3), are not returned, and so b[size-1][size-1]=cofactor(a,m,n); only assigns a single integer to b[2][2], which is in fact the famous buffer overflow bug you may have heard so much about, because the memory at b[2][2] is not part of b but belongs to some other part of your program.
My recommendation therefore is to pass b into cofactor() rather than trying to return it, i.e.:
Code:
void cofactor(int a[size][size], int m, int n,int b[size-1][size-1])
{
...
}
and in main:
cofactor(a,m,n,b);
With this change I get the following output:
Code:
Enter row index=1 Enter column index=1 The matrix is 1 2 3 4 5 6 7 8 9 Cofactor of a[1][1] is 1 3 7 9

