I got stuck here for quite some time before I found some documentation on the SailfishOS site.
Short answer: yes, you need some drivers (different from the ADB ones).
BUT, you need drivers not from those listed there on the developers site as XA2. Rather, per the instructions at https://jolla.com/sailfishx-windows-instructions-xa2/ , you’ll need drivers for the Xperia X. The driver is called Sony sa0114. The instructions at Jolla/Sailfish will guide you through the rest of the process.
Good luck. It took me a bit of playing around, and you may well need to re-flash the stock Sony ROM again if things go fubar, but the “Emma” flashtool available from Sony will be able to let you do that quite easily.