It's no problem, the questions are designed to help, but perhaps you're not going to spot this one without a little direct help.

return b[size-1][size-1]; does NOT return the whole of array b, it just returns the single integer at b[size-1][size-1], i.e. b[2][2], which is in fact undefined because only b[0..1][0..1] are defined. The four integers at b[0..1][0..1] (in the case that size=3), are not returned, and so b[size-1][size-1]=cofactor(a,m,n); only assigns a single integer to b[2][2], which is in fact the famous buffer overflow bug you may have heard so much about, because the memory at b[2][2] is not part of b but belongs to some other part of your program.

My recommendation therefore is to pass b into cofactor() rather than trying to return it, i.e.:

Code:

void cofactor(int a[size][size], int m, int n,int b[size-1][size-1])
{
...
}
and in main:
cofactor(a,m,n,b);

Then cofactor() updates the b in main directly, rather than trying to update its own copy. The return statement only returns a single value.

With this change I get the following output:

Code:

Enter row index=1
Enter column index=1
The matrix is
1 2 3
4 5 6
7 8 9
Cofactor of a[1][1] is
1 3
7 9

which is what you were expecting.