Fair warning......plan on spending some significant money as you hobby grows. Yes, you will be purchasing molds, colorants, plastisol, etc. but also plan on (providing you have a significant other) flowers, dinner dates, vacation in some tropical place......Just kidding! Enjoy the new hobby! 
...wait- I could afford those things with the wife only before I got into this "hobby".

Where did I go wrong?

Agreed on the passion for this!!! COOLNESS to the fullest. My advice to you however, is if you have a green thumb, are a great cook, and already live in a tropical place, you can do just as knifemaker says! Otherwise, your passion for this "hobby" may include taking a break from those things for a bit.
OK the communication and honesty with the wife/husband is one place where we are failing miserably is this world today and is a huge passion of mine (as it should be for any Christian man) so forgive me here but- All joking aside, I've learned (again if you have a significant other) it is best to sit down with her and show her some costs, and discuss the plan (maybe it is a hobby or maybe it is a 'business' like for some of us around here), and come up with a budget. You may be shooting single cavity molds for a while, but at some point when you decide to add molds and start changing gears, you can re-evaluate the plan.
I know some say "it's easier to ask for forgiveness than permission" but which one feels better?
Plus- sharing this with my wife is turning out to be more rewarding than I could have ever imagined. There's my two cents now I'll shut up.

-Shawn