Only a limited amount of climbing can be done at Stone Mountain. The mountain can only be climbed on the approved hiking trails. All allowed routes are walkable with no climbing skills or equipment required. However, walking to the top on the Walk-Up Trail is very good exercise.
While walking off-trail on the mountain is prohibited, people can often be seen doing so. It's not a good idea and there are plenty of allowed alternatives. You can walk the Cherokee Trails and the many Connecting Trails. You can take the Walk-Up Trail, and at the summit you can explore a large area without crossing the fence.
Remember that on the trail or off, the mountain can be very slippery, especially when wet or where there is pine straw, sand gravel, etc.