The exposure does need to stop. Have you come to the same conclusion?
I think the fact that you're scared to talk to Bob's parents is very important. That fact alone would indicate to me that your kids shouldn't be hanging out. You're letting a man whom you're afraid of coach your son. By avoiding Bob's parents, you're modeling that behavior to your son. And I think that's exactly how you would "pass-down" your tendency to be bullied to your son (you've said it's already showing up with Bob). He would be mirroring your passivity and conflict-avoidance with aggressive, scary people.
I'd be scared to talk to those parents too, it's never a fun conversation, and you're not crazy for assuming it might "cause a war" so to speak. They could yell at you, they could involve the school (although I'm not sure how that hurts you), and they could have revenge on the soccer field. All the more reason to get clear about what to expect, what you want, and plan and rehearse how you're going to get it.
I think it's important to model a more proactive, assertive position here, and if you've been a victim of bullying, maybe that doesn't come naturally to you. So practice! Figure out exactly what it is that you want from this situation. (Don't assume you're going to be able to change these people). If you're going to have a discussion with them, figure out the few bullet points you want to get across, pick a time and place, and do it. If they want to get ugly about it, that's their choice, and will give you yet another reason to cut them out. In my experience, confronting bullies is usually a lot less scary than I imagine it will be, and is usually sufficient to get them to back down. So the message could simply be "We don't like this influence, we don't want our son to be around yours." and if you're feeling neighborly/generous you could offer to help them with their troubles.
Standing up for your son this way, you can think of it as your chance to redo your own childhood, and use your experience to shield him from the damaging relationships you suffered with. What do you wish your parents did for your when you were a kid?