My sympathies for your situation, this one hits close to my heart because it's so much like the story of my parents. My mother divorced my father when I was around 5, out of, from what I deduce, vague dissatisfaction and boredom. It was the beginning of a long and nasty struggle for my father to stay in our lives. My mother did all she could to keep him out of our lives, and for many years we could only see him every other weekend, etc. My mother moved us around several times to different states, once not even informing my father. He showed up to pick us up one weekend and the apartment was empty. My mother even had my siblings and I convinced that he was insane and everything he did (constant court battles, etc.) to try to stay in our lives was making our lives worse. His committment was unwaivering, though, and by the time we got to high school he was finally able to get half-time with us when me moved to a new jurisdiction. It's only been in the last couple of years (I'm in my mid 20s now) that I learned to see through my mother's bullshit, and can fully appreciate what he did for us, and have a father that I actually respect and look up to, as a role model. My relationship with him now is stronger than it's ever been. I shudder to think what type of person I might have become if he had simply given up all those years ago and I did not have him in my life. It makes me very emotional to write this.
Don't underestimate what a powerful force you are, and can potentially be, for your child! You are his single greatest asset. Don't ever doubt that.