istringstream .eof() mystery

Hello, using the code posted below, the eof() condition on the line_stream object will terminate the corresponding while loop after reading only 1 word from the stream. If there is more than 1 word in the line_stream they will not be accounted for. However removing the eof() condition, such that the condition is simply line_stream >> word, alleviates the problem. However the cause of the eof()s dysfunction is still a mystery to me. Can anyone solve this?

Code:

typedef map<string, list<int> > word_line_map;

istream& fill_map(istream& input_mechanism, word_line_map& wlm){
	
string line;
int line_count(1);

	while(getline(input_mechanism, line),!input_mechanism.eof()){
		istringstream line_stream(line);	
	
		string word;
		while(line_stream >> word, !line_stream.eof()){
			list<int> line_count_list;
			line_count_list.push_back(line_count);

			pair<word_line_map::iterator, bool> insert_result = wlm.insert(make_pair(word, line_count_list));
			word_line_map::iterator &loc(insert_result.first);
			bool &inserted(insert_result.second);

			if(!inserted){
				list<int> &line_list(loc->second);
				line_list.push_back(line_count);
			}
		}
		++line_count;
	}
	return input_mechanism;
}

 

Simplified testcase:

Code:

void linestream()
{
	istringstream line_stream("The quick brown fox jumps over the lazy dog");
	string word;
	while(line_stream >> word, !line_stream.eof())
	{
		printf("%s\n",word.c_str());
	}
}

This is a common bug. If you pull out the last word, which sets eof TRUE, then test eof and exit the loop if it's true, then the loop won't process the last word.

Output from the above is:
The
quick
brown
fox
jumps
over
the
lazy

 

So, what is the trick to using the .eof() test and being able to process the last word?

 

No trick as such, just don't assume that eof means processing is complete. In other words, don't use eof as the loop condition unless you're absolutely certain eof means processing really is over.

 

A stream goes into EOF state whenever the end of stream is seen, i.e. a character past the end has been read. As operator» and getline normally keep reading characters until the end of token (until whitespace, invalid characters, line terminator or EOF) it is possible that the stream EOF flag gets set even though the token was read correctly. Conversely, the stream does not go into EOF state if there happens to be any whitespace after the last token, but trying to read another token will still fail.

Therefore, the EOF flag cannot be used as a test in a loop intended to read all stream contents until EOF.