I think that for most people, it is a case of trying to build a good quality site, and then doing SEO for it, i.e. it isn't an either/or option.
Unless techniques are being used on a site that stray too far away from their guidelines, it isn't going to be affected.
Also in the recording is an explanation that Google is not against SEO and reference is made to Matt Cutts video on the subject in which he explains the benefits of it:
http://www.youtube.com/watch?v=BS75vhGO-kk
I dont think it is a matter of "worrying less" about SEO, but about using it to the best advantage without over doing it.