The prima guide of lies states that gems/ingots can contribute to a beautifully decorated moodlet which is worth +40. A nicely decorated room (one without gems etc. I suppose) is worth +25.
You can hit +40 without the gems, just with a good number of high-environment items. So, presumably, gems are just another decorative thing to toss around. Thanks Prima Guides - not wrong, but not overly right, either.