You can find "salt and pepper" ballast, mostly where they ballasted with one color from one pit and then reballasted from another pit.
Commonly ballast is some shade of grey (from a dark grey to a blue grey to near white), or pink (red/pink granite) or a dark brown color (dark rock or slag).
The color also depends on the era. Older roads used seashells or oyster shells or whatever the local rock was on the main and cinders for the subroadbed or yards. Cinders were usually a darker color, blackish to grey. Modern roads use ballast for everything, they just use different sizes of ballast, really big on concrete ties, big on mains, smaller in yards and chat on walkways.
Dirty ballast was the color of, well dirt. I mix a little of the latex paint I use for the dirt color into my glue mix I dribble over the ballast. Just enough to give a hint of color. That adds a dirty dusty effect to the ballast and unifies the colors.