Er... No. PS Vita TV is, from what I know, at the time, strictly a "PS4 extender" and compatible with SOME PSP and PSOne titles released by Sony. It doesn't have (yet?) access to the full library of those consoles. So, you'd be actually paying to buy "a machine" that a) works in conjuction with another console you don't have (the PS4) and b) has to show as "its gaming library" only one tiny bit of the collections of two older consoles, with decade-old graphics and sound, that in many cases you'll have to re-pay to get them.
If you already have a collection of PSP games you could re-licence on it (I don't know how Sony implemented the "play your PSP games on it" function) OR ultra-fast internet to be able to use Sony's streaming service to get access to some old-but-great PSOne titles, hit it.
I'd go for the PS3 as you can guess. It's got an awesome library of titles, but... But even that makes the choice seem a bit off today. After the PS4 and Xbox One, it was like it became instantly apparent how old the PS3 and Xbox 360 were. So, maybe it would be even better keeping your money for a PC upgrade or a PS4 some time later. Especially since the first PS3 emulator for PCs has also started running some commercial stuff, but let's not talk about it here 'cause "questionable legality" blah-blah-blah.
As for what to expect if you pick a PS3 to a Vita TV, instead of playing 10 year old games on a 480i (but upscaled) resolution you'll be mostly playing 5-year titles on a 720p resolution. It is "a bit" better. A huge "bit". And among the titles you'll be able to play, Uncharted, The Last of Us and, yeah, even Crysis 2 - that I loved on my PS3 even more than on my PC (thanks to its 3D support that was awesome combined with my Panasonic Viera TV).