Yes, the prototype does use different sizes of track ballast.
If you grab your copy of the AREMA Manual and check out Chapter 1, Part 2 you'll see the various gradations. What's that you say? You don't have a copy of the AREMA Manual for Railway Engineering? It's a great reference for model (or real) railroaders. You can pick one up for the low, low price of $1470. ($920 if you're an AREMA member.) Yes, it is by far the most expensive book I've ever purchased, and thankfully work paid for it. It's actually several books, each quite large, and takes up a couple feet of shelf space. It's also rather dry reading for the most part.
OK, so you probably don't have a copy handy. You can see the common sizes in this chart:
http://armytransportation.tpub.com/TI-850-02/Table-6-8-Recommended-Ballast-Gradations-64.htm
Short version?
Size 4A is 2" to 3/4" This the big stuff, used on high speed mainlines with concrete ties.
Size 4 is 1-1/2" to 3/4" This is the standard track ballast. Basically normal crushed rock, no fines.
Size 5 is 1" to 3/8 This walkway rock and yard ballast. Used for areas where trainmen are walking.
Some of the other comments are correct, a lot of the railroad ballast model railroaders use is too big. On the other hand, it does have a texture to it, and that's what you're trying to capture.
You will often see various colors in the rocks. This is a good way to get the impression of ballast without having to use big chunks. A speckled texture will highlight the fact it's made up of various pieces.
Color varies, sometimes a lot. I seem to recall that the D&RGW out of Denver was known for pink ballast.
Here's an example of a typical modern mainline. You can definitely see texture in the rock as well as some color variation.
Like the "nail holes" that many model railroaders put in structures (some of which are more like bullet holes in size), sometimes you need to go a bit oversize for effect. The best approach is to work from photos and then take photos of the the model and compare.