Yes, the smaller the file, the more less the quality gets. And it's also a fact that conversion from one format to another will have some change in quality depending on the settings.
I always convert at the standard bitrate which is 32Kbps, I use the "normalize volume" option so that all songs are standard in volume, I set the frequency at 44100Hz, and lastly I set the channel to stereo. Setting it a bit higher (let's say 40Kbps or 64Kbps at 44100Hz) will have better quality but adds about 400-800 kb and still a lot less than the original large mp3 file.
It's the same setting as you used. So far in this settings my music files sounds fine to me. Maybe if you'd compare them through playing the original and then the converted file you'll find some difference in quality.
Setting a higher bitrate and frequency of a wma file can greatly boost sound quality but at the same time makes the file much larger BUT still a lot less than the original size. Tweak the settings to what suits your taste.