infrastructure/mox
1
0
Fork 0
mirror of https://github.com/mjl-/mox.git synced 2025-07-12 18:24:35 +03:00
Code Issues Packages Projects Releases Wiki Activity

add a webapi and webhooks for a simple http/json-based api

Browse Source 
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.

this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.

unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...

matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.

a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.

messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.

suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.

submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".

to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.

admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.

new config options have been introduced. they are editable through the admin
and/or account web interfaces.

the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.

gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.

for issue #31 by cuu508
This commit is contained in:
Mechiel Lukkien
2024-04-15 21:49:02 +02:00
parent 8bec5ef7d4
commit 09fcc49223
87 changed files with 15556 additions and 1306 deletions
Download Patch File Download Diff File Expand all files Collapse all files

151
mox-/admin.go
View File

@ -899,6 +899,7 @@ func AddressAdd(ctx context.Context, address, account string) (rerr error) {
}
// AddressRemove removes an email address and reloads the configuration.
// Address can be a catchall address for the domain of the form "@<domain>".
func AddressRemove(ctx context.Context, address string) (rerr error) {
log := pkglog.WithContext(ctx)
defer func() {
@ -934,6 +935,52 @@ func AddressRemove(ctx context.Context, address string) (rerr error) {
if !dropped {
return fmt.Errorf("address not removed, likely a postmaster/reporting address")
}
// Also remove matching address from FromIDLoginAddresses, composing a new slice.
var fromIDLoginAddresses []string
var dom dns.Domain
var pa smtp.Address // For non-catchall addresses (most).
var err error
if strings.HasPrefix(address, "@") {
dom, err = dns.ParseDomain(address[1:])
if err != nil {
return fmt.Errorf("parsing domain for catchall address: %v", err)
}
} else {
pa, err = smtp.ParseAddress(address)
if err != nil {
return fmt.Errorf("parsing address: %v", err)
}
dom = pa.Domain
}
for i, fa := range a.ParsedFromIDLoginAddresses {
if fa.Domain != dom {
// Keep for different domain.
fromIDLoginAddresses = append(fromIDLoginAddresses, a.FromIDLoginAddresses[i])
continue
}
if strings.HasPrefix(address, "@") {
continue
}
dc, ok := Conf.Dynamic.Domains[dom.Name()]
if !ok {
return fmt.Errorf("unknown domain in fromid login address %q", fa.Pack(true))
}
flp, err := CanonicalLocalpart(fa.Localpart, dc)
if err != nil {
return fmt.Errorf("getting canonical localpart for fromid login address %q: %v", fa.Localpart, err)
}
alp, err := CanonicalLocalpart(pa.Localpart, dc)
if err != nil {
return fmt.Errorf("getting canonical part for address: %v", err)
}
if alp != flp {
// Keep for different localpart.
fromIDLoginAddresses = append(fromIDLoginAddresses, a.FromIDLoginAddresses[i])
}
}
na.FromIDLoginAddresses = fromIDLoginAddresses
nc := Conf.Dynamic
nc.Accounts = map[string]config.Account{}
for name, a := range Conf.Dynamic.Accounts {
@ -948,12 +995,16 @@ func AddressRemove(ctx context.Context, address string) (rerr error) {
return nil
}
// AccountFullNameSave updates the full name for an account and reloads the configuration.
func AccountFullNameSave(ctx context.Context, account, fullName string) (rerr error) {
// AccountSave updates the configuration of an account. Function xmodify is called
// with a shallow copy of the current configuration of the account. It must not
// change referencing fields (e.g. existing slice/map/pointer), they may still be
// in use, and the change may be rolled back. Referencing values must be copied and
// replaced by the modify. The function may raise a panic for error handling.
func AccountSave(ctx context.Context, account string, xmodify func(acc *config.Account)) (rerr error) {
log := pkglog.WithContext(ctx)
defer func() {
if rerr != nil {
log.Errorx("saving account full name", rerr, slog.String("account", account))
log.Errorx("saving account fields", rerr, slog.String("account", account))
}
}()
@ -966,6 +1017,8 @@ func AccountFullNameSave(ctx context.Context, account, fullName string) (rerr er
return fmt.Errorf("account not present")
}
xmodify(&acc)
// Compose new config without modifying existing data structures. If we fail, we
// leave no trace.
nc := c
@ -973,100 +1026,12 @@ func AccountFullNameSave(ctx context.Context, account, fullName string) (rerr er
for name, a := range c.Accounts {
nc.Accounts[name] = a
}
acc.FullName = fullName
nc.Accounts[account] = acc
if err := writeDynamic(ctx, log, nc); err != nil {
return fmt.Errorf("writing domains.conf: %v", err)
return fmt.Errorf("writing domains.conf: %w", err)
}
log.Info("account full name saved", slog.String("account", account))
return nil
}
// DestinationSave updates a destination for an account and reloads the configuration.
func DestinationSave(ctx context.Context, account, destName string, newDest config.Destination) (rerr error) {
log := pkglog.WithContext(ctx)
defer func() {
if rerr != nil {
log.Errorx("saving destination", rerr,
slog.String("account", account),
slog.String("destname", destName),
slog.Any("destination", newDest))
}
}()
Conf.dynamicMutex.Lock()
defer Conf.dynamicMutex.Unlock()
c := Conf.Dynamic
acc, ok := c.Accounts[account]
if !ok {
return fmt.Errorf("account not present")
}
if _, ok := acc.Destinations[destName]; !ok {
return fmt.Errorf("destination not present")
}
// Compose new config without modifying existing data structures. If we fail, we
// leave no trace.
nc := c
nc.Accounts = map[string]config.Account{}
for name, a := range c.Accounts {
nc.Accounts[name] = a
}
nd := map[string]config.Destination{}
for dn, d := range acc.Destinations {
nd[dn] = d
}
nd[destName] = newDest
nacc := nc.Accounts[account]
nacc.Destinations = nd
nc.Accounts[account] = nacc
if err := writeDynamic(ctx, log, nc); err != nil {
return fmt.Errorf("writing domains.conf: %v", err)
}
log.Info("destination saved", slog.String("account", account), slog.String("destname", destName))
return nil
}
// AccountAdminSettingsSave saves new account settings for an account only an admin can change.
func AccountAdminSettingsSave(ctx context.Context, account string, maxOutgoingMessagesPerDay, maxFirstTimeRecipientsPerDay int, quotaMessageSize int64, firstTimeSenderDelay bool) (rerr error) {
log := pkglog.WithContext(ctx)
defer func() {
if rerr != nil {
log.Errorx("saving admin account settings", rerr, slog.String("account", account))
}
}()
Conf.dynamicMutex.Lock()
defer Conf.dynamicMutex.Unlock()
c := Conf.Dynamic
acc, ok := c.Accounts[account]
if !ok {
return fmt.Errorf("account not present")
}
// Compose new config without modifying existing data structures. If we fail, we
// leave no trace.
nc := c
nc.Accounts = map[string]config.Account{}
for name, a := range c.Accounts {
nc.Accounts[name] = a
}
acc.MaxOutgoingMessagesPerDay = maxOutgoingMessagesPerDay
acc.MaxFirstTimeRecipientsPerDay = maxFirstTimeRecipientsPerDay
acc.QuotaMessageSize = quotaMessageSize
acc.NoFirstTimeSenderDelay = !firstTimeSenderDelay
nc.Accounts[account] = acc
if err := writeDynamic(ctx, log, nc); err != nil {
return fmt.Errorf("writing domains.conf: %v", err)
}
log.Info("admin account settings saved", slog.String("account", account))
log.Info("account fields saved", slog.String("account", account))
return nil
}

68
mox-/config.go
View File

@ -61,6 +61,8 @@ var (
Conf = Config{Log: map[string]slog.Level{"": slog.LevelError}}
)
var ErrConfig = errors.New("config error")
// Config as used in the code, a processed version of what is in the config file.
//
// Use methods to lookup a domain/account/address in the dynamic configuration.
@ -317,10 +319,11 @@ func (c *Config) allowACMEHosts(log mlog.Log, checkACMEHosts bool) {
// todo future: write config parsing & writing code that can read a config and remembers the exact tokens including newlines and comments, and can write back a modified file. the goal is to be able to write a config file automatically (after changing fields through the ui), but not loose comments and whitespace, to still get useful diffs for storing the config in a version control system.
// must be called with lock held.
// Returns ErrConfig if the configuration is not valid.
func writeDynamic(ctx context.Context, log mlog.Log, c config.Dynamic) error {
accDests, errs := prepareDynamicConfig(ctx, log, ConfigDynamicPath, Conf.Static, &c)
if len(errs) > 0 {
return errs[0]
return fmt.Errorf("%w: %v", ErrConfig, errs[0])
}
var b bytes.Buffer
@ -1272,8 +1275,52 @@ func prepareDynamicConfig(ctx context.Context, log mlog.Log, dynamicPath string,
}
acc.NotJunkMailbox = r
}
acc.ParsedFromIDLoginAddresses = make([]smtp.Address, len(acc.FromIDLoginAddresses))
for i, s := range acc.FromIDLoginAddresses {
a, err := smtp.ParseAddress(s)
if err != nil {
addErrorf("invalid fromid login address %q in account %q: %v", s, accName, err)
}
// We check later on if address belongs to account.
dom, ok := c.Domains[a.Domain.Name()]
if !ok {
addErrorf("unknown domain in fromid login address %q for account %q", s, accName)
} else if dom.LocalpartCatchallSeparator == "" {
addErrorf("localpart catchall separator not configured for domain for fromid login address %q for account %q", s, accName)
}
acc.ParsedFromIDLoginAddresses[i] = a
}
c.Accounts[accName] = acc
if acc.OutgoingWebhook != nil {
u, err := url.Parse(acc.OutgoingWebhook.URL)
if err == nil && (u.Scheme != "http" && u.Scheme != "https") {
err = errors.New("scheme must be http or https")
}
if err != nil {
addErrorf("parsing outgoing hook url %q in account %q: %v", acc.OutgoingWebhook.URL, accName, err)
}
// note: outgoing hook events are in ../queue/hooks.go, ../mox-/config.go, ../queue.go and ../webapi/gendoc.sh. keep in sync.
outgoingHookEvents := []string{"delivered", "suppressed", "delayed", "failed", "relayed", "expanded", "canceled", "unrecognized"}
for _, e := range acc.OutgoingWebhook.Events {
if !slices.Contains(outgoingHookEvents, e) {
addErrorf("unknown outgoing hook event %q", e)
}
}
}
if acc.IncomingWebhook != nil {
u, err := url.Parse(acc.IncomingWebhook.URL)
if err == nil && (u.Scheme != "http" && u.Scheme != "https") {
err = errors.New("scheme must be http or https")
}
if err != nil {
addErrorf("parsing incoming hook url %q in account %q: %v", acc.IncomingWebhook.URL, accName, err)
}
}
// todo deprecated: only localpart as keys for Destinations, we are replacing them with full addresses. if domains.conf is written, we won't have to do this again.
replaceLocalparts := map[string]string{}
@ -1423,6 +1470,25 @@ func prepareDynamicConfig(ctx context.Context, log mlog.Log, dynamicPath string,
}
}
// Now that all addresses are parsed, check if all fromid login addresses match
// configured addresses.
for i, a := range acc.ParsedFromIDLoginAddresses {
// For domain catchall.
if _, ok := accDests["@"+a.Domain.Name()]; ok {
continue
}
dc := c.Domains[a.Domain.Name()]
lp, err := CanonicalLocalpart(a.Localpart, dc)
if err != nil {
addErrorf("canonicalizing localpart for fromid login address %q in account %q: %v", acc.FromIDLoginAddresses[i], accName, err)
continue
}
a.Localpart = lp
if _, ok := accDests[a.Pack(true)]; !ok {
addErrorf("fromid login address %q for account %q does not match its destination addresses", acc.FromIDLoginAddresses[i], accName)
}
}
checkRoutes("routes for account", acc.Routes)
}

113
mox-/fill.go Normal file
View File

@ -0,0 +1,113 @@
package mox
import (
"reflect"
)
// FillNil returns a modified value with nil maps/slices replaced with empty
// maps/slices.
func FillNil(rv reflect.Value) (nv reflect.Value, changed bool) {
switch rv.Kind() {
case reflect.Struct:
for i := 0; i < rv.NumField(); i++ {
if !rv.Type().Field(i).IsExported() {
continue
}
vv := rv.Field(i)
nvv, ch := FillNil(vv)
if ch && !rv.CanSet() {
// Make struct settable.
nrv := reflect.New(rv.Type()).Elem()
for j := 0; j < rv.NumField(); j++ {
nrv.Field(j).Set(rv.Field(j))
}
rv = nrv
vv = rv.Field(i)
}
if ch {
changed = true
vv.Set(nvv)
}
}
case reflect.Slice:
if rv.IsNil() {
return reflect.MakeSlice(rv.Type(), 0, 0), true
}
n := rv.Len()
for i := 0; i < n; i++ {
rve := rv.Index(i)
nrv, ch := FillNil(rve)
if ch {
changed = true
rve.Set(nrv)
}
}
case reflect.Map:
if rv.IsNil() {
return reflect.MakeMap(rv.Type()), true
}
i := rv.MapRange()
for i.Next() {
erv, ch := FillNil(i.Value())
if ch {
changed = true
rv.SetMapIndex(i.Key(), erv)
}
}
case reflect.Pointer:
if !rv.IsNil() {
FillNil(rv.Elem())
}
}
return rv, changed
}
// FillExample returns a modified value with nil/empty maps/slices/pointers values
// replaced with non-empty versions, for more helpful examples of types. Useful for
// documenting JSON representations of types.
func FillExample(seen []reflect.Type, rv reflect.Value) reflect.Value {
if seen == nil {
seen = make([]reflect.Type, 100)
}
// Prevent recursive filling.
rvt := rv.Type()
index := -1
for i, t := range seen {
if t == rvt {
return rv
} else if t == nil {
index = i
}
}
if index < 0 {
return rv
}
seen[index] = rvt
defer func() {
seen[index] = nil
}()
switch rv.Kind() {
case reflect.Struct:
for i := 0; i < rv.NumField(); i++ {
if !rvt.Field(i).IsExported() {
continue
}
vv := rv.Field(i)
vv.Set(FillExample(seen, vv))
}
case reflect.Slice:
ev := FillExample(seen, reflect.New(rvt.Elem()).Elem())
return reflect.Append(rv, ev)
case reflect.Map:
vv := FillExample(seen, reflect.New(rvt.Elem()).Elem())
nv := reflect.MakeMap(rvt)
nv.SetMapIndex(reflect.ValueOf("example"), vv)
return nv
case reflect.Pointer:
nv := reflect.New(rvt.Elem())
return FillExample(seen, nv.Elem()).Addr()
}
return rv
}

31
mox-/localserve.go Normal file
View File

@ -0,0 +1,31 @@
package mox
import (
"strconv"
"strings"
"github.com/mjl-/mox/smtp"
)
func LocalserveNeedsError(lp smtp.Localpart) (code int, timeout bool) {
s := string(lp)
if strings.HasSuffix(s, "temperror") {
return smtp.C451LocalErr, false
} else if strings.HasSuffix(s, "permerror") {
return smtp.C550MailboxUnavail, false
} else if strings.HasSuffix(s, "timeout") {
return 0, true
}
if len(s) < 3 {
return 0, false
}
s = s[len(s)-3:]
v, err := strconv.ParseInt(s, 10, 32)
if err != nil {
return 0, false
}
if v < 400 || v > 600 {
return 0, false
}
return int(v), false
}