There are both rocky deserts and the typical endless sand dune deserts. The trick is the scale. Rocks can be both jagged or semi smooth and these can be found in your garden, local gardening store or a construction site. Just make sure it doesn't seem too big when stacked against your vehicles/figs. As for sand, beach sand comes close but you'll have to check it off for scale against your model to make sure its fine enough. Not sure if you can get hold of this but it should be available in gardening store. Its actually a fertilizer called Rock Phosphate or sometimes known as Christmas Island Rock Phosphate, as it came from Christmas Island.
Its basically a finely, ground down rock and it hardly dissolves in water so no problem sticking it down with water based glues. Its normally a nice shade of brown coming close to desert sand but its made advantage is the texture is really fine.
Not sure if this will help but the little dio I did a long time ago uses natural pebbles and fine soil that has been sifted. The scrub was from a lichen like material coated with tea leaves. It features the Ramcke brigade in Libya.


