This is the way I'd do it:
Get about an 18-inch length of PVC pipe, a tennis ball. a 45-degree PVC coupling, and a hole saw the same diameter as the end of the coupling. You'll also need a board that's at least a 1/2 inch thick and a couple of small wood screws and a drill bit about the diameter of the wood screws. It think 1-inch PVC is about the best for this -- you could try smaller, but I think you'll need it that big when you attach the coupling to the board.
1. drill the large hole in the board all the way through. try to get the hold as snug to the diameter of the PVC pipe coupling as possible.
2. insert the coupling into the wood so that the flat end is snug with the backside of the board. Also make sure the 45 degree bend is pointing "up."
3. Now, the tricky part: drill some pilot holes through the inside of the coupling into the wood. Don't drill though the face out the other side, though. You basically want to drill through the coupling and maybe a 1/4 inch into the wall of the wood part of the hole. You may want to experiment on a piece of scrap until you get it figured out.
4. finger tighten the screw in the hole until it's snug. It doesn't really have to be all the way in -- it's just there to keep the coupling from spinning. For added support, you could put in a couple, but I don't it's necessary unless your puppet is particularly heavy.
5. Insert the PVC pipe into the coupling on the front. You can paint it and glue it if you like, but I don't think it's necessary.
6. Cut a "+" shape in the tennis ball about an inch around (you may have seen a similar technique on the bottom of older people's walkers so the legs don't scuff the floors). Pop the tennis ball onto the end of the PVC pipe to make keep any sharp edges from messing up the inside of your puppet.
The arm rods can just hand there, or if they're removable, you can take them out when the puppet's on display.
I haven't made something like this, but it's the way I'd attack it if I was doing something like this. Feel free to improvise or come up with your own way of doing it!