Cant get into factory reset keeps going into download mode with S8

Using /e/OS Setup
11

On Windows, Odin is good for the job. Check out this HOWTO [HOWTO] Install /e/ on a Samsung smartphone with Windows easily

Later Edit Good to see that this suggestion worked out for you, Samsung S8 install via ODIN works. Installer does not.