Buying a handheld recorder can bag you a surprising variety of sounds if you carry it around every day. A bog-standard MP3 notetaker can be had for a mere fiver whilst the real quality starts at around £100. Tascam do something nice at around that price.
http://www.ebay.co.uk/itm/250822490030
Failing that, your laptop and a USB/Firewire mike preamp could do you the power of good. If you already have a corking mike or two then you need spend only on the recording kit itself. Mind, the goody-bag will be bulkier should you decide to sneak a quick sample of one of the latest keyboards or top-end pianos in a music shop. But if you have a carry bag then you can still swing by anywhere. (My tip is to rush into a church and quickly record your free vocal or percussion overdub in a place with killer accustics!)
Cheap but it can't be connected to a battery:
http://www.ebay.co.uk/itm/ART-Tube-MP-O ... 1513wt_905
And, thinking about it, I suppose there's also the telly. Plenty of scope for lifting noises from programmes and videos - tracked in the finest quality - but you didn't hear that from me. Though with the mindblowing array of cheap sound effect discs on the market now, you wouldn't need to waste your days wading through all the old pony the stations broadcast, just on the off-chance of grabbing maybe a bell or a dog.
BUYERS' GUIDE:
http://www.bhphotovideo.com/find/newsLe ... -Guide.jsp
I would fancy a handheld with phantom power XLR input.