memory as in “disk space”? in any case, you’d need to free up space in userdata, not system
in an older thread I speculate “… the invalid arg error is originating from fs [recovery-] kernel module / mount-option incompatibility”, see How to mount encrypted /data partition in recovery mode? - #51 by tcecyk
Can you try suppling some of the mount options (oriole → raviole → gs101) of userdatas
fstab entry
(it’s f2fs)
mnt_flags / options:
noatime,nosuid,nodev,discard,reserve_root=32768,resgid=1065,fsync_mode=nobarrier,inlinecrypt,atgc,checkpoint_merge
fs_mgr_flags:
latemount,wait,check,quota,formattable,sysfs_path=/dev/sys/block/bootdevice,checkpoint=fs,reservedsize=128M,fileencryption=@fileencryption@,metadata_encryption=@metadata_encryption@,keydirectory=/metadata/vold/metadata_encryption,fscompress,readahead_size_kb=128