You may be tricked by default quota config file which suggest that it’s possible:
Multiple quota roots are also possible, for example this gives each user their own 100MB quota and one shared 1GB quota within the domain:
#quota = dict:user::proxy::quota
#quota2 = dict:domain:%d:proxy::quota_domain
#quota_rule = *:storage=102400
#quota2_rule = *:storage=1048576
But in the end there is no piece of code that supports per-domain quota.
You can set “per domain quota” as in example configuration, but it won’t work as you expect.
It shares same name of quota for multiple users (domain name) but doesn’t properly calculate used storage as sum of all users (mailboxes) in domain.
This is status from 2020-04-06, it may change in future.