Rims: Carbon fiber is the lightest rim material and stiffest, but the most expensive. Most rims are made from different alloys of aluminum, and the differences between them come down to stiffness and durability. You're looking for a rim that's stiff, but not brittle. The stiffness also is affected by the number of spokes and the shape of the rim. While 26 inches is the standard diameter for an adult bike, 29 inches (known as 29ers) have become popular in the hardcore XC and all-mountain disciplines because they are thought to roll over obstacles more easily.
Hubs: Almost all modern hubs are sealed bearings with aluminum construction. The difference between high and low end is weight and the number of engagement teeth on the freehub in the rear wheel. The more engagement points, the less distance you have to pedal to engage the hub from a stop or when coasting.
Tires: Knobby tires come standard on mountain bikes. They offer more traction but also more resistance. Shallower treads will perform better on roads or gravel.
Rigid vs. suspension: Rigid forks are the least expensive and will suit budget-conscious recreational riders and commuters. That said, if you even think you want to go off road at some point, you'll want a suspension fork.
Air vs. oil: Cross-country and all-mountain bikes use air as their means of suspension. Higher-end forks can absorb larger bumps without upsetting the handling and have more adjustment and controls that allow you to lock the suspension out for more efficient riding. Freeride and downhill bikes generally have oil forks that use fluid technology similar to what you'd find in a motorcycle fork they can take more abuse, but they're more expensive, heavier and harder to maintain.
Full suspension vs. hardtail: Full-suspension frames give a more forgiving ride while hardtails are faster and more responsive and cost less. Recreational riders will find that a hardtail offers enough performance for bike paths, unpaved roads and worn-in singletrack. FS bikes are at home on technical singletrack with lots of roots, rocks and hazards.
Geometry: The key here is the angle of the head tube and the distances between the seat tube, handlebars and wheels. These angles and distances dictate the handling a steeper head tube angle means faster steering, and a shorter distance between the wheels contributes to an agile feel. A shallower head tube angle and longer wheelbase make for a stable-feeling ride. The location and angle of the seat tube affect your position as you ride. If you're a big guy, for example, you want some space between the seat tube and the handlebars.
Material: Carbon fiber gives a softer ride by absorbing the vibration or "feedback" you would normally get from a metal frame. It's the lightest frame material and the most expensive. Aluminum frames are stiff and transmit all the feedback from the trail to your body. Chromoly steel is the heaviest frame material, but it's the most durable and does offer a little dampening of the ride. A lightweight frame is important only to the degree that a lighter bike is more efficient, but a bike that's too light for a given trail will bounce all over and be no fun to ride.