I don't believe mobile games will kill off handhelds for the same reason computers haven't killed off consoles. At the end of the day it's a lot more convenient to have a dedicated handheld to gaming that's effortless to use and has some well-known quality control backing it. There's always going to be a upgrade arms race, bluetooth controllers that may or may not configure properly with your device and non-existent quality control on smartphones and tablets unless something seriously changes. My current Android phone and tablet are both a little old so it can play some games on the market but not all games.
When you buy into a console or handheld you're buying into simplicity and knowing that what you're getting will play anything that is brought to it. A Nintendo console with have Nintendo games that you pop in and play with virtually no effort at all. People always argue that computers will always be way more powerful then console gamers and you know what, they're right. Look at how many games are getting cut from the 1080p, 60fps treatment because the Xbone/PS4 can't handle it. Despite that, you don't have to put the machine together, spend tons of money to get something that will hopefully play everything for the next few years, run into driver problems, run into hardware problems, run into issues with your OS if you stray from Windows for any reason, etc. Consoles just don't have that problem. While tablets/smartphones don't have as many problems for the average user problems like quality control, hardware upgrades, OS upgrades you may or may not get and other things are still a concern.
Despite people feeling mobile games are taking from the handheld buying demographic, the Nintendo DS is the 2nd best selling gaming system of all time and the 3DS is making its way to 50 million sold. Maybe the Vita isn't doing too hot, but neither did the Sega Gamegear about 25 years ago despite Sega being at their prime and this being before mobile games were a thing. I think mobile games have the potential to kill off handhelds but I believe they've got a ways to go before that happens unless Sony or Nintendo release the Nintendo/Sony Tablet or Phone.