Many geocachers have a bag they take with them while geocaching.

Below is a list of what a carry in my bag. It's a messenger bag I found on clearance at Meijer (a Michigan based chain that pioneered the format used by Wal-Mart Supercenters), Originally a $15 bag, I snatched it up on clearance for under $4.

Cache Repair Kit
First Aid Kit
Tweezers and Neelenose Pliers
For when logs et stuck in micro and nano caches.
Trail Maps
Many of the parks around me have extensive trails.
Pencil Sharpener
If the pen fails and pencil breaks Murphy really doesn't like you.
Trade Items
Items not pictured but normally carried.
Notebook
If I discover a potential hiding place for a future cache I might place, I jot down the coordinates and check when I get home to see if it is a spot further than 528 feet from any other cache.
Pens
Pencils
As a back up to when ink pens don't work-plan on Murphy's Law.