Sounds like you need to decrypt the WIF first, then import it.
Also, I'm pretty sure there are still bugs with electrum and hardware wallets,
so you might want to choose different software to manage the keys on the hardware wallet.
You'll want to do this on the machine that has the wallet.
Go...