You can't read less than a byte from a file. If you could, it would still be terribly inefficient because file operations are orders of magnitude slower than memory operations. You'll need to stick with shifts, masks, or modulus. Modulus is generally slower than shifts, but your compiler will usually optimize it to a shift, where that would be faster.
You CAN read more than one byte, using fread. It doesn't really gain you anything but a larger number of bits in a larger variable. In any reasonably modern system, the system will read an entire sector, minimum, at a time. It may read much more, if it has available cache. This will then be parceled out to your program, where your language's IO stuff will further adapt it to the type of read you asked for.