Honestly? Unless you enjoy the process of tweaking itself I'd advise against going too far down that path. The manufacturers of things like memory and CPUs are pretty good at "binning", or testing their parts to see how well they run at certain speeds. Parts that test higher are sold as higher-performance parts, and of course at higher-performance prices. The chances that you will buy a lower-spec part that you can get radically higher performance out of are pretty slim. Also, doing things like pushing the clock rate higher will have negative results even if you manage significant gains. Heat output will go up for instance, so you will need to have a better - and more expensive - cooling solution. Heat will also affect the lifetime of the part(s). And of course if you push things too far you can get into instability and reliability problems. If you're the kind of person who just wants your computer to be a reliable platform for email, web, games, etc. then it is best to stick with stock settings.
That said, you can often get a 10-20% increase in performance without too much trouble. Manufacturers do bin their parts, but they also leave good margins so that their parts work reliably no matter what system they find themselves running in. BIOSs these days often have features and settings that allow the owner to fiddle with things to overclock or otherwise improve performance. Don't expect to be able to get much out of a cost-conscious build created from low-end parts, but if your motherboard has good power and your BIOS gives you the control then you can often tweak things to get better performance out of your system's particular collection of parts without using up all the margin left by the manufacturers.
For myself I usually spend a little bit more up front to make sure I have as much margin as possible. Especially for things like power supplies that I can reuse over several builds I think it makes sense to get the one that has higher efficiency, produces more stable power, and gives me more "headroom" to use if I need it. Even a low-end motherboard + RAM + CPU + stock cooler will work fine at stock speeds, at least if you stick to the well-known brands. But I'd rather have the one with better VRMs and a BIOS that gives me the option to adjust clock rates, voltages, and RAM timings, even if it is few $$ more. A button on the motherboard to reset to *your* last known-good settings rather than a jumper to reset to factory-specs is also handy, as is a built-in POST-code display. I'll also go for the heat sink that's slightly bigger than necessary and maybe an extra case fan or two. Maybe even a nice water cooler. Stuff like that. It'll cost extra, but if you don't go too crazy spending money - or if you can justify the extra expense in terms of enjoying the tweaking process itself - then that's probably fine. As long as it doesn't break your budget of course.
Even then I often stick with pretty much stock settings to start with. I do somewhat enjoy the tweaking process, but I also use my computer for a lot of other things as well. I prefer a stable platform for those things. It is only after I've had the system for a while that I start messing with pushing the performance envelope. Usually what pushes me over the edge is hearing about the next generation's performance. Then I'll get in there and see if I can get an extra 10 or 20% out of what I've got. At that point I've usually had a couple/few years of reliable computing on the system anyway. If I'm already starting to think about upgrading parts, or even getting a whole new system, then pushing the limits of the old parts/system can help me avoid that expense for a while longer. I never get TOO crazy, but even breaking things typically isn't going to cause much of a problem at that point. There's a lot less stress in breaking something you were kind of thinking of replacing anyway than letting the magic smoke out of that expensive bit of kit you just bought.