Is it possible to configure per-domain quota in Dovecot?

Answer: No.

You may be tricked by default quota config file which suggest that it’s possible:

Multiple quota roots are also possible, for example this gives each user their own 100MB quota and one shared 1GB quota within the domain:

plugin {
#quota = dict:user::proxy::quota
#quota2 = dict:domain:%d:proxy::quota_domain
#quota_rule = *:storage=102400
#quota2_rule = *:storage=1048576
}

But in the end there is no piece of code that supports per-domain quota.

You can set “per domain quota” as in example configuration, but it won’t work as you expect.
It shares same name of quota for multiple users (domain name) but doesn’t properly calculate used storage as sum of all users (mailboxes) in domain.

This is status from 2020-04-06, it may change in future.