if I have a custom domain in my account, can I share it with other users in a family plan, and can they use it to create aliases in their own?

I think so but I only have an individual account with Mailbox so I've never tried this. See their help pages (under Resources), they have extensive documentation. Another option is to use the contact form on their site to ask.

I know they have this thing called "family accounts" that takes individual accounts and makes them part of a "family" so they can share stuff like files and calendars. I think it means they can also partake in custom domains. And of course each account can make their own aliases.

Keep in mind each account needs to be paid separately though. Not sure it's possible to have multiple mailboxes with different logins and self-managed aliases under the same account.

You can make free accounts and add them to a family but free accounts have some limitations, particularly about aliases.