Funny this thread should come up. I have recently been thinking the same question.
The cemetery across from my house recently re-graveled all their roads with a nice, fine, white crushed rock. Naturally, they had a little left over at the end of the job and left two huge piles for the ground crews for repairs (filling pot holes, etc.). After a quick little stroll and some grave robbin' later, I had little bucket full of sample material.
I used a big strainer to get the bigger rocks out. They'll work great for rip-raff and what not. I thought at first it would good O Scale ballast, but after checking it on a section of track, I think it's too big.
After a the big rocks were out I ran it all through a fine siv to get all the dust and thing too small to use out. Then I picked up a cheap flour sifter from Walmart and broke down the remaining into two smaller sizes. The former will work great for O Scale, the later is perfect for HO:
Still a touch dusty, but not too bad, and it seems to disappear when with wet water and glue.
So, fine crushed white stone, some sifters and a little bit of time and voila - instant ballast. Real rock, too.