As long as your data is properly encrypted, they are safe. But in theory, if a hostile person will get a physical access to your device, he can inject a script into recovery, that will hijack your data in the moment you get your device back and try unlock it first time. The solution: if someone really suspicious took your phone and then gave it back to you - without decryption perform a full wipe, restore it to factory defaults in all aspects (including OS, recovery and bootloader re-locking), and then setup it from scratch as a totally new device.