In programming, the actual "code" programmers create is merely written text, and programs are little more than text files. As a result, the most important tool anyone can have is a solid, reliable editor. Sure, you can use Notepad to whip up a batch file, or TextEdit on the Mac, but most projects require more than Notepad provides, while the rich text features of Wordpad or TextEdit will only get in the way.
The nice thing is that to get started you don't have to invest a lot of money in fancy IDE's (integrated development environments) or hours downloading the hundreds of megabytes of XCode or its equivalent. The Eclipse IDE for Java - while fairly large still smaller than Xcode - is free, and most web or Perl/Python/etc. development can be done quite nicely with a broad selection of free editors. Outside of Java and Cocoa (Mac) programming, all of my coding is done in BBEdit.
On the Mac-only side, I recommend getting started with Textwrangler - the "lite" version of BBEdit. Despite missing some advanced features it has the critical ones - syntax color hilights, multiple-file searches, regular expression searches that let you search by pattern definitions, and my favorite file comparison tool for displaying the difference between two files.
For Windows, check out Notepad++.
Finally, for free, ridiculously powerful, and cross-platform (including Linux) there's Vim and Emacs. Both have all the features you need and then some, come in command-line and GUI variants, and have fanatical adherents who will tell you at the drop of a hat why Vim/Emacs is awesome and the other one sucks. No matter which one you choose, you'll end up with an editor powerful enough to handle your needs for years (One of the guys handling FX for the movie 2012 uses Emacs for handling 3D render programming...).
For more suggestions, take a look at this lifehacker page on the best text editors.