(you’re right, it can’t be deleted)
the recently-used Address book is provided via Raw z-app-generated--contactsinteraction--recent URI visible to user · Issue #2187 · nextcloud/contacts · GitHub → disabling the plugin will remove the addressbook
if you run your own nextcloud
./occ app:disable contactsinteraction