Correct.
Crypto isn't stored on an individual device, rather it's ownership is stored on the block chain. Wallets, both hardware and software, only store your keys that protect your on-chain assets and allow you to sign transactions on the chain.
Hardware wallets are exceptional here because they have special chips that are designed to safely store and protect your keys (ledger for example uses "Secure Elements"), which make reading and stealing said keys very difficult - even if you connect a hardware wallet to an infected computer, your keys remain inaccessible and protected.
A USB or software wallet doesn't have the ability to do this, and the keys are accessible to malicious software when connected. While you can disconnect a USB drive from your computer making it safe when not in use, it becomes vulnerable as soon as you connect it to use it, so while it is a physical device, it is little more secure than a software wallet.