That really depends on where you get them and the sort of collection you want to start. If you want a collection of well formed crystals ($$$), tumbled stuff ($$) or unconventionally stuff ($). There are all sorts of places you can get rocks, places like gift shops and well-being places(crystal power) all the way to professional and amateur dealers at gem shows.
Then there’s collecting which can be as free as the price of transportation, tools, and dig fees/permission. This is of course my favourite.
Sometimes though, you may get lucky and find an estate sale or even a local willing to give you a deal.
I’m not sure how much interest you have in Earth’s minerals, but I’m going to take a wild guess that this snapshot of a dilithium crystal will. Also there are surprisingly few images, must be a Federation secret or something.