The delegated tokens are not accessible by the validator or anyone, just by the owner, so in case of an attack to the validator the delegated tokens are safe.
The most common risk for a delegator is that the validator they're delegated with is penalized by the network for fraudulent actions (also known as slashing) such as double signing blocks, attacks to the network or big downtimes of their servers.