Acutally it's 14 billion, so the universe is only three times older than Earth itself.
That's called Fermi's Paradox (http://en.wikipedia.org/wiki/Fermi_paradox) and I agree with you, it is very fascinating. There is a section in the wiki-article that lists about two dozen hypothetical solutions, you can take a look at that if you like (it's quite long).