Ok, so the USB connection is working at least, that’s something.
The adb command doesn’t work in Fastboot Mode, so that’s to be expected, but the fastboot command should work.
Just to make sure … The phone really is in Fastboot mode when you get this? Here’s an example picture from [HOW-TO] Flash /e/-OS on Fairphone 3 using Debian based GNU/Linux … displayed data may vary …
I experienced Fastboot as being more robust than ADB so far, so it’s a bit odd to see it reported the other way around. If on Windows, there could still be driver trouble which could be avoided.