I'm always charmed by museums that feel like this sort of "weird collection". The trend seems to be firmly away from it, however. During the time I lived in London, the Horniman museum reorganized itself and became "respectible" in the process. While I was living in Exeter the RAMM did the same thing. If you have a similar taste, I strongly recommend visiting the Pitt-Rivers museum in Oxford: it's a truly bizarre collection of absolutely amazing objects - or it was the last time I was there; if it's since been re-curated along modern lines, then you'll have missed a uniquely odd experience.
There must be something instinctive about collecting oddities, or perhaps it's a social aspect of extream curiousity, where after examining an object, a desire to get another opinion, requires storing the object, and so it begins.
A friend who has turned pro, makes his living partialy through the buying and selling of oddities, often whole collections that are bieng sold/disposed of by exasperated relatives, his pragmatic approach to his own desire to have things he calls "catch and release", and so getting to enjoy some interesting thing, but sell it forward, and aquire the next item.
It is a wierd world though, as many people develope specific fetishes and kinks around objects , or just simpler less troublesome obsessions, and then of course movie people, decorators, advertisers, artists, bibliophiles, etc.
If you're in London, check out John Soane's museum. And then, on the other side of the square, the Hunterian museum where the original collection has been presented in a more modern style. Some Hunterian artifacts turned out to be useful in the development of modern medical artifacts. Soane's gives you an experience closer to the original. Hunterian shows the evolution described in the article.
The pathological museum in Vienna (also called "Narrenturm" because it used to be an asylum) has a section with actual historic curiosity cabinets. It's a very interesting museum to visit, but it's not for the faint of heart. They have a paramedic on site because visitors regularly get sick :-D