Naturally radioactive minerals are not very radioactive! This is because they are formed from naturally occurring radioactive isotopes which, by necessity, have extremely long lifetimes - comparable or greater than the age of the Earth itself. The radioactivity of an isotope is strongly dependent upon its half-life. Tritium (hydrogen-3) has a half-life of only 11.2 years, meaning that half of it decays every 11 years. It is highly radioactive - but not nearly as radioactive as some isotopes used in medicine, such as Technetium-99 with a half-life of only 6 hours. A unit of Technetium-99 emits 16,000 times as much radiation per second than does Tritium, which in turn is 7,000,000 times as radioactive as Plutonium!
There are no plutonium minerals. Consider Plutonium-244 (the most stable isotope of plutonium). It has a half-life of 82 million years - and consequently is not very radioactive. Yet in the 4.5 billion year existence of the Earth, the total number of remaining atoms dropped in half 550 times. There simply is none left to form minerals.
There are four main radioactive isotopes that have lasted as long as the Earth:
That means that essentially all radioactive minerals contain either uranium or thorium. The Rare Earth Elements typically include a percentage of uranium and/or thorium as trace elements, rendering many of them slightly radioactive. Note that both uranium and thorium have decay chains involving many short-lived isotopes (they consequently emit much more radiation than a simple isotope such as tritium), but these are present in tiny traces and typically don't exist long enough to become concentrated and form minerals.
There are other relatively common radioactive nuclides: Radium-226 and Radon-222 are both produced during the decay of Uranium. Radon is considered the most hazardous, because as a gas we breathe it into our lungs where it can attach until it decays. Luckily, Radon only has a 4-day half-life, which limits the build-up in a basement or from a concrete wall. Unluckily, Radon only has a 4-day half-life, which means it is extremely radioactive.
Another well-known radioactive isotope is Carbon-14, produced as a measurable trace by cosmic rays acting on atmospheric CO2, and with a half-life of 5732 years. Since it is carbon, it is readily incorporated into all living things. Indeed, our bodies are naturally radioactive, to a degree, from all of these sources (Potassium-40 contributes the most, then Carbon-14, then Uranium and its decay products).
Minerals containing Thorium (including two minerals also containing Uranium):
Minerals containing Uranium (radioactivity is proportional to the fraction of uranium, and alpha particles are primarily emitted from the surface):
Rare Earth Oxides (REOs) and Trash Can Minerals (which tend to concentrate the leftovers after common minerals have formed, and will generally include traces of Uranium and/or Thorium: "traces" means they are less radioactive):
Note that my statement that Uranium and Thorium are not very radioactive does NOT mean they aren't dangerous! There are different types of ionizing radiation with different hazards. Beta radiation (high-energy electrons), Gamma rays (high-energy photons), Neutrons, and Alpha particles (high-energy helium nuclei) have different effects and respond differently to shielding. The primary type of radiation emitted by Uranium and Thorium is in the form of alpha particles which are extremely potent but easily shielded. Placing a specimen beneath a glass or plastic dome will block more than 99% of the radiation. Even a sheet of paper will block it, or the outer (dead) layers of your skin. The danger from alpha-emitters is in breathing dust (or gas such as Radon), which places radioactive particles in contact with live tissue which is easily damaged by the radiation.
For safety, ALWAYS keep radioactive minerals in a display case and/or dome, MINIMIZE handling which can release dust particles, IMMEDIATELY WASH hands and clothing that comes into contact with radioactive minerals, HOLD YOUR BREATH while handling specimens (or opening containers which might retain radon), and VACUUM or WASH to clean - do not blow dust off. All of these precautions are intended to prevent inhaling (or eating) radioactive particles. Again, simply touching a uranium or thorium mineral with your hand is not intrinsically dangerous.
Don't worry about sending trash to an incinerator, as they release far more radiation into the air than from other materials than from the dust in your packaging! In fact, many times more radiation has been released into the environment in the form of uranium as a trace component of coal ash than from nuclear power plants; the ash is not very radioactive, but there is a lot of it (especially from older power plants), and while the waste from a nuclear power plant is extremely radioactive (due to the fission products of uranium and/or plutonium atoms), it is well monitored. After all, the workers in a nuclear power plant are worried about radiation and work hard to keep it contained, but not the workers in a coal-burning power plant.