You are talking about a supply chain attack, where a security vulnerability is introduced intentionally during the manufacturing process. Backdoors built into the Android firmware have been found before although not by manufacturers such a Samsung specifically.
Firmware is one important piece of software that is commonly overlooked when dealing with security and privacy. It’s what makes the physical components work correctly such as the phone camera, bluetooth or speakers. But it’s in pretty much everything like home routers which are usually left unpatched for years.
So there is a real possibility that your smartphone is spying on you at the hardware level even after installing a new version of Android such as LineageOS or /e/. Unless they take it upon themselves to reverse-engineer the device’s components but I don’t think this is the case.
With many drivers being proprietary it makes it difficult to examine the code to verify it does what it says it does. This is not to say that open-source software is automatically “better” or “more secure” either, but unmaintained software is doomed to become not secure.
It is up to the manufacturer of your device to provide firmware updates and in many cases this support disappears after a couple of years, leaving you with the only choice of buying a new phone. And it is understandable to a certain degree the difficulty involved in maintaining up to date every single device ever produced, but this is where open source it could come very handy for the community.
Firmware is not the only source of potential vulnerabilities at the hardware level. On mobiles, SIM cards can be exploited as well.
EDIT: A much more in depth study about security on mobile devices: https://securephones.io/