You could try Rhasspy - https://github.com/rhasspy/rhasspy
Mike is a pretty cool guy. I know the repo is archived, but he might help you if you reach him.
Alternatively, what about a panic button? Both commercial solutions and privacy friendly local solutions exist. I know it’s intrusive and grandma will have to keep it on her person at all times. But… that’s kind of what they’re there for.