I don't know about sizing the keyboard, but you can certainly try other keyboards in the app. I still have the choice of the Samsung keyboard instead of the "Gboard" keyboard that appears automatically selected in Google messages on my S24+.
Edit to add: Makes me also wonder if something like accessibility features on the phone could be messing up.
Second edit: Does it act the same if you rotate the screen 180 degrees from the failing scenario? Have you altered the settings for the navigation bar in the phone's Settings -> Display -> navigation bar?