The seat is an intimate decision, dictated by the shape of your butt. You'll want to try a few out, but thinner, sleeker seats generally weigh less and cost more. Gender-specific seats take anatomical differences into account, so try a few. Seatposts are just weight. So you're paying for a light seatpost that can hold things at the proper level.
The stem connects the top of the forks to the handlebars. Different lengths and angles equate to different rider positions. Your choice of material translates into feel and durability. Cast aluminum is the weakest, machined aluminum is the strongest and stiffest, and carbon fiber is the lightest (but weight difference in stems is negligible and really only applies to podium chasers).
Your choice of handlebars will have a critical effect on your riding comfort. Riser bars allow you to sit more upright, while flat bars will put you in a more aggressive, maneuverable position. Take advantage of your local bike shop's demo program and try a few bikes before deciding. Bar ends were once thought to make climbing hills on a bike more efficient but have fallen out of favor in pursuit of shaving ounces.
This is the set of bearings that the handlebar, stem and forks mount through for steering control. You'll see 1Þ-inch, 1½-inch and tapered headsets. The last two are found on higher-end bikes, which give a stiffer ride, more control and longer bearing life. Spending money here means your headset will last longer and require fewer adjustments. Almost all decent headsets will have sealed bearings. The lowest-end, toy store bikes will have unsealed bearings that will need to be greased every few seasons to prevent them from seizing up.
Shifters: Tubular grip shifters have fallen out of favor in performance circles, replaced by trigger shifters that come in a wide range of trim levels. Aim for middle-of-the-road — Shimano SLX and SRAM X7 series — and adjust fire to fit your budget.
Brake levers: Integrated shifters and brake levers may save a few bucks up front, but stand-alone brake levers allow you to recover from a smash-up without forking out for new shifters, too. And because the brake levers are integrated into the shifter body in an integrated system, you can't adjust the angle of the brake levers individually.