Figured out my mistake....

Right before I load the ZipStream with the Memory Stream, I have to reset the position of the memory stream...

ms.Position = 0;


Then I can load it into the ZipInput Stream....