> .help me in debugging the program

Yes, to that end I've asked some questions which so far you haven't answered.
- What output did you expect? (Pointing me to a tutorial on cofactors doesn't answer this)

If adding debugging code didn't help then try adding more. Those dodgy values must be coming from SOMEWHERE.
- What output did the code I suggested display?
- Did you try adding equivalent code to the other loops?

What exactly are you expecting "return b[size-1][size-1];" to return to the caller ( using test matrix {{1,2,3},{4,5,6},{7,8,9}} ) ?