I understand completely: being behind the wheel feels totally different than being in the passenger seat, and I too was a nervous wreck . . . it also doesn't help too that just the slightest little pressure you put on the gas pedal sometimes makes the car feel like it's just going blast off on its own at a break-neck speed . . . heck, even 20mph felt like zooming to me in the beginning.
Even now, one of the things I hate the most is driving on a highway or the interstate, because you're trying to drive the speed limit (whether it be 55, 65, or even 70 in some cases), but everybody's swerving around you acting like you're impeding traffic. Even worse is when you're driving off of an exit and you're trying to merge into traffic on such a major road, but there's so much on-coming traffic, and nobody is willing to let you merge, so you find yourself basically either having to drive alongside the shoulder, or just pull over altogether until the traffic's clear enough for you to merge onto the road. Big rigs are especially bad about this, since they like to feel like they own the road.
It took me years, but I'll admit I've pretty much gotten over all of the fears and anxieties of driving - but at the same time, I still have some apprehension when I'm driving. I also still get nervous whenever I have passenger(s), because that adds to my apprehension knowing I'm responsible for other people's lives.