We normally check errors for all operations. But for some cleanup calls, eg
"defer file.Close()", we didn't. Now we also check and log most of those.
Partially because those errors can point to some mishandling or unexpected code
paths (eg file unexpected already closed). And in part to make it easier to use
"errcheck" to find the real missing error checks, there is too much noise now.
The log.Check function can now be used unconditionally for checking and logging
about errors. It adjusts the log level if the error is caused by a network
connection being closed, or a context is canceled or its deadline reached, or a
socket deadline is reached.
In store/search.go, we would make a copy of a byte array, but then still use
the original instead of the copy. Could result in search operations not finding
messages that do have the content, but under very unlikely conditions only.
We'll keep running ineffassign with "make check", useful enough.
The original config option stays, and we still use it for the common case where
we have a single separator. The "+" is configured by default. It is optional,
just like the new option "LocalpartCatchallSeparators" (plural).
When parsing the config file, we combine LocalpartCatchallSeparator and
LocalpartCatchallSeparators into a single list
LocalpartCatchallSeparatorsEffective, which we use throughout the code.
For issue #301 by janc13
Mox does not look up names from the /etc/hosts file, only through DNS. But
"localhost" may not resolve through DNS, or when offline a DNS server may not
even be available. We will want deliveries to work in "mox localserve" mode.
Found by dstotijn.
Keeping the message files around, and the message details in the database, is
useful for IMAP sessions that haven't seen/processed the removal of a message
yet and try to fetch it. Before, we would return errors. Similarly, a session
that has a mailbox selected that is removed can (at least in theory) still read
messages.
The mechanics to do this need keeping removed mailboxes around too. JMAP needs
that anyway, so we now keep modseq/createseq/expunged history for mailboxes
too. And while we're at it, for annotations as well.
For future JMAP support, we now also keep the mailbox parent id around for a
mailbox, with an upgrade step to set the field for existing mailboxes and
fixing up potential missing parents (which could possibly have happened in an
obscure corner case that I doubt anyone ran into).
DeliverMessage() is now MessageAdd(), and it takes a Mailbox object that it
modifies but doesn't write to the database (the caller must do it, and plenty
of times can do it more efficiently by doing it once for multiple messages).
The new AddOpts let the caller influence how many checks and how much of the
work MessageAdd() does. The zero-value AddOpts enable all checks and all the
work, but callers can take responsibility of some of the checks/work if it can
do it more efficiently itself.
This simplifies the code in most places, and makes it more efficient. The
checks to update per-mailbox keywords is a bit simpler too now.
We are also more careful to close the junk filter without saving it in case of
errors.
Still part of more upcoming changes.
logging of login attempts happens in the background, because we don't want to
block regular operation with disk since for such logging. however, when a line
is logged, we evaluate some attributes of a connection, notably the username.
but about when we do authentication, we change the username on a connection. so
we were reading and writing at the same time. this is now fixed by evaluating
the attributes before we pass off the logger to the goroutine.
found by the go race detector.
and don't do needless normalization for the username we got from scram: the
scram package would have failed if the name wasn't already normalized.
unicode may not be specified for sasl with imap (i'm not sure), but there's no
point in accepting it over smtpserver but not in imapserver.
and show them in the account and admin interfaces. this should help with
debugging, to find misconfigured clients, and potentially find attackers trying
to login.
we include details like login name, account name, protocol, authentication
mechanism, ip addresses, tls connection properties, user-agent. and of course
the result.
we group entries by their details. repeat connections don't cause new records
in the database, they just increase the count on the existing record.
we keep data for at most 30 days. and we keep at most 10k entries per account.
to prevent unbounded growth. for successful login attempts, we store them all
for 30d. if a bad user causes so many entries this becomes a problem, it will
be time to talk to the user...
there is no pagination/searching yet in the admin/account interfaces. so the
list may be long. we only show the 10 most recent login attempts by default.
the rest is only shown on a separate page.
there is no way yet to disable this. may come later, either as global setting
or per account.
to facilitate migrations from/to other mail setups.
a domain can be added in "disabled" mode (or can be disabled/enabled later on).
you can configure a disabled domain, but incoming/outgoing messages involving
the domain are rejected with temporary error codes (as this may occur during a
migration, remote servers will try again, hopefully to the correct machine or
after this machine has been configured correctly). also, no acme tls certs will
be requested for disabled domains (the autoconfig/mta-sts dns records may still
point to the current/previous machine). accounts with addresses at disabled
domains can still login, unless logins are disabled for their accounts.
an account now has an option to disable logins. you can specify an error
message to show. this will be shown in smtp, imap and the web interfaces. it
could contain a message about migrations, and possibly a URL to a page with
information about how to migrate. incoming/outgoing email involving accounts
with login disabled are still accepted/delivered as normal (unless the domain
involved in the messages is disabled too). account operations by the admin,
such as importing/exporting messages still works.
in the admin web interface, listings of domains/accounts show if they are disabled.
domains & accounts can be enabled/disabled through the config file, cli
commands and admin web interface.
for issue #175 by RobSlgm
useful when a catchall is configured, and messages to some address need to be
rejected.
would have been nicer if this could be part of a ruleset. but evaluating a
ruleset requires us to have the message (so we can match on headers, etc). but
we can't reject messages to individual recipients during the DATA command in
smtp. that would reject the entire delivery attempt.
for issue #156 by ally9335
Intended for future use with chatmail servers. Standard email ports may be
blocked on some networks, while the HTTPS port may be accessible.
This is a squashed commit of PR #255 by s0ph0s-dog.
mail clients will use these message from addresses also for smtp mail from, so
sending over smtp would fail for these cases. for the webmail and webapi they
already succeeded since we just took the "message from" address as "smtp mail
from" address.
for issue #266 by Robby-, thanks for reporting!
the field is optional. if absent, the default behaviour is currently to disable
session tickets. users can set the option if they want to try if delivery from
microsoft is working again. in a future version, we can switch the default to
enabling session tickets.
the previous fix was to disable session tickets for all tls connections,
including https. that was a bit much.
for issue #237
the imap & smtp servers now allow logging in with tls client authentication and
the "external" sasl authentication mechanism. email clients like thunderbird,
fairemail, k9, macos mail implement it. this seems to be the most secure among
the authentication mechanism commonly implemented by clients. a useful property
is that an account can have a separate tls public key for each device/email
client. with tls client cert auth, authentication is also bound to the tls
connection. a mitm cannot pass the credentials on to another tls connection,
similar to scram-*-plus. though part of scram-*-plus is that clients verify
that the server knows the client credentials.
for tls client auth with imap, we send a "preauth" untagged message by default.
that puts the connection in authenticated state. given the imap connection
state machine, further authentication commands are not allowed. some clients
don't recognize the preauth message, and try to authenticate anyway, which
fails. a tls public key has a config option to disable preauth, keeping new
connections in unauthenticated state, to work with such email clients.
for smtp (submission), we don't require an explicit auth command.
both for imap and smtp, we allow a client to authenticate with another
mechanism than "external". in that case, credentials are verified, and have to
be for the same account as the tls client auth, but the adress can be another
one than the login address configured with the tls public key.
only the public key is used to identify the account that is authenticating. we
ignore the rest of the certificate. expiration dates, names, constraints, etc
are not verified. no certificate authorities are involved.
users can upload their own (minimal) certificate. the account web interface
shows openssl commands you can run to generate a private key, minimal cert, and
a p12 file (the format that email clients seem to like...) containing both
private key and certificate.
the imapclient & smtpclient packages can now also use tls client auth. and so
does "mox sendmail", either with a pem file with private key and certificate,
or with just an ed25519 private key.
there are new subcommands "mox config tlspubkey ..." for
adding/removing/listing tls public keys from the cli, by the admin.
and add an alerting rule if the failure rate becomes >10% (e.g. expired
certificate).
the prometheus metrics includes a reason, including potential tls alerts, if
remote smtp clients would send those (openssl s_client -starttls does).
inspired by issue #237, where incoming connections were aborted by remote. such
errors would show up as "eof" in the metrics.
was encountered during smtp session. but could also happen for imapserver and
webmail.
in smtpserver, we now log error messages for smtp errors that cause us to print
a stack trace. would have made logging output more helpful (without having to
turn on trace-level logging).
hopefully solves issue #238 by mwyvr, thanks for reporting!
before this change, we were logging an empty string, which turned into "[]",
looking like an empty array. misleading and unhelpful.
this is fixed by making struct fields on type recipient "exported" so they can
get logged, and by changing the logging code to log nested
struct/pointer/interface fields if we would otherwise wouldn't log anything
(when only logging more basic data types).
we'll now get log lines like:
l=info m="deliver attempt to unknown user(s)" pkg=smtpserver recipients="[addr=bogus@test.example]"
for issue #232 by snabb, thanks for reporting!
we add various information while analysing an incoming message. like
dkim/spf/ip reputation. and content-based junk filter threshold/result and
ham/spam words used.
for issue #179 by Fell and #157 by mattfbacon
there is only an internet-draft about the required behaviour. it says clients
should ignore the strings. some clients do check the string. most servers
appear to use "Username:" and "Password:" as challenge. we'll follow them,
hoping to improve interoperability.
for issue #223 by gdunstone, and with analysis from wneessen of go-mail.
thanks!
for some errors during the scram authentication protocol, we would treat some
errors that a client connection could induce as server errors, printing a stack
trace and aborting the connection.
this change recognizes those errors and sends regular "authentication failed"
or "protocol error" error messages to the client.
for issue #222 by wneessen, thanks for reporting
we didn't announce starttls as capability, but clients can still try them. we
would try to do a handshake with a nil certificate, which would cause a
goroutine panic (which is handled gracefully, shutting down the connection).
found with code that was doing starttls unconditionally.
instead of failing the connection because no certificates are available.
this may improve interoperability. perhaps the remote smtp client that's doing
the delivery will decide they do like the tls cert for our (mx) hostname after
all.
this only applies to incoming smtp deliveries. for other tls connections
(https, imaps/submissions and imap/submission with starttls) we still cause
connections for unknown sni hostnames to fail. if case no sni was present, we
were already falling back to a cert for the (listener/mx) hostname, that
behaviour hasn't changed.
for issue #206 by RobSlgm
by sending the (encoded) string "User Name" as mentioned by the internet-draft,
https://datatracker.ietf.org/doc/html/draft-murchison-sasl-login-00#section-2.1
that document says clients should ignore the challenge (which is why were were
not doing any effort and sending an empty challenge). but it also says some
clients require the challenge "Username:" instead of "User Name", implying that
it's important to not send an empty challenge. we can't send both challenges
though...
for issue #51
it's the responsibility of the sender to use unique fromid's.
we do check if that's the case, and return an error if not.
also make it more clear that "unique smtp mail from addresses" map to the
"FromIDLoginAddresses" account config field.
based on feedback from cuu508 for #31, thanks!
the members must currently all be addresses of local accounts.
a message sent to an alias is accepted if at least one of the members accepts
it. if no members accepts it (e.g. due to bad reputation of sender), the
message is rejected.
if a message is submitted to both an alias addresses and to recipients that are
members of the alias in an smtp transaction, the message will be delivered to
such members only once. the same applies if the address in the message
from-header is the address of a member: that member won't receive the message
(they sent it). this prevents duplicate messages.
aliases have three configuration options:
- PostPublic: whether anyone can send through the alias, or only members.
members-only lists can be useful inside organizations for internal
communication. public lists can be useful for support addresses.
- ListMembers: whether members can see the addresses of other members. this can
be seen in the account web interface. in the future, we could export this in
other ways, so clients can expand the list.
- AllowMsgFrom: whether messages can be sent through the alias with the alias
address used in the message from-header. the webmail knows it can use that
address, and will use it as from-address when replying to a message sent to
that address.
ideas for the future:
- allow external addresses as members. still with some restrictions, such as
requiring a valid dkim-signature so delivery has a chance to succeed. will
also need configuration of an admin that can receive any bounces.
- allow specifying specific members who can sent through the list (instead of
all members).
for github issue #57 by hmfaysal.
also relevant for #99 by naturalethic.
thanks to damir & marin from sartura for discussing requirements/features.
instead of skipping on any smtp and delivering messages to accounts.
we dial the ip of the smtp listener, which is localhost:1025 by default.
the smtp server now uses a mock dns resolver during spf & dkim verification for
hosted domains (localhost by default), so they should pass.
the advantage is that we get regular full smtp server behaviour for delivering
in localserve, including webhooks, and potential first-time sender delays
(though this is disabled by default now).
incoming deliveries now go through normal address resolution, where before we
would always deliver to mox@localhost. we still accept email for unknown
recipients to mox@localhost.
this will be useful upcoming alias/list functionality.
localserve will now generate a dkim key when creating a new config. existing
users may wish to reset (remove) their localserve directory, or add a dkim key.
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.
this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.
unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...
matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.
a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.
messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.
suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.
submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".
to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.
admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.
new config options have been introduced. they are editable through the admin
and/or account web interfaces.
the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.
gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.
for issue #31 by cuu508
Squashed commit of the following:
commit 11c25d727f0fff72bfb2dde5b0121d65be5cdc09
Author: Laurent Meunier <laurent@deltalima.net>
Date: Sun Mar 31 12:37:09 2024 +0200
Fix style issue
commit c075a8cd8bb116dc1b8ecae9880a70656d362714
Author: Laurent Meunier <laurent@deltalima.net>
Date: Sun Mar 31 12:35:04 2024 +0200
Also check smtputf8 for submitted messages or when in pedantic mode
commit c02328f881c653c1e84448233f6b04a6bc30bc4f
Author: Laurent Meunier <laurent@deltalima.net>
Date: Sun Mar 31 12:33:20 2024 +0200
Calls to `newParser` should use `c.smtputf8`
commit a0bbd13afc17e5bd7eb845d2045b8bc156c19d25
Author: Laurent Meunier <laurent@deltalima.net>
Date: Sun Mar 31 12:32:12 2024 +0200
Improve SMTPUTF8 tests
commit 08735690f3682e96b7f91cae2a32eaba7dc8b1f9
Author: Laurent Meunier <laurent@deltalima.net>
Date: Sat Mar 30 17:22:33 2024 +0100
do earlier smtputf8-check
commit 3484651691cb3a78062e5c19d5ac7046a5dfba7b
Author: Laurent Meunier <laurent@deltalima.net>
Date: Thu Mar 28 17:47:11 2024 +0100
do not require the SMTPUTF8 extension when not needed
fix#145
an é (e with accent) can also be written as e+\u0301. the first form is NFC,
the second NFD. when logging in, we transform usernames (email addresses) to
NFC. so both forms will be accepted. if a client is using NFD, they can log
in too.
for passwords, we apply the PRECIS "opaquestring", which (despite the name)
transforms the value too: unicode spaces are replaced with ascii spaces. the
string is also normalized to NFC. PRECIS may reject confusing passwords when
you set a password.
transferring the data only once. we only do this when the recipient domains
are the same. when queuing, we now take care to set the same NextAttempt
timestamp, so queued messages are actually eligable for combined delivery.
this adds a DeliverMultiple to the smtp client. for pipelined requests, it will
send all RCPT TO (and MAIL and DATA) in one go, and handles the various
responses and error conditions, returning either an overal error, or per
recipient smtp responses. the results of the smtp LIMITS extension are also
available in the smtp client now.
this also takes the "LIMITS RCPTMAX" smtp extension into account: if the server
only accepts a single recipient, we won't send multiple.
if a server doesn't announce a RCPTMAX limit, but still has one (like mox does
for non-spf-verified transactions), we'll recognize code 452 and 552 (for
historic reasons) as temporary error, and try again in a separate transaction
immediately after. we don't yet implement "LIMITS MAILMAX", doesn't seem likely
in practice.
smtp & imap can only indicate which mechanisms the server software supports.
individual accounts may not have derived secrets for all those mechanisms. imap
& smtp cannot indicate that a client should try another (specific) mechanism.
but at least we shouldn't slow the connection down due to failed auth attempts
in that case.
heard from ben that this is a common source for trouble when setting up email
accounts.
the smtp extension, rfc 4865.
also implement in the webmail.
the queueing/delivery part hardly required changes: we just set the first
delivery time in the future instead of immediately.
still have to find the first client that implements it.
mox was already strict in its "\r\n.\r\n" handling for end-of-message in an
smtp transaction.
due to a mostly unrelated bug, sequences of "\nX\n", including "\n.\n" were
rejected with a "local processing error".
the sequence "\r\n.\n" dropped the dot, not necessarily a big problem, this is
unlikely to happen in a legimate transaction and the behaviour not
unreasonable.
we take this opportunity to reject all bare \r. we detect all slightly
incorrect combinations of "\r\n.\r\n" with an error mentioning smtp smuggling,
in part to appease the tools checking for it.
smtp errors are 500 "bad syntax", and mention smtp smuggling.
to get the security benefits (detecting mitm attempts), explicitly configure
clients to use a scram plus variant, e.g. scram-sha-256-plus. unfortunately,
not many clients support it yet.
imapserver scram plus support seems to work with the latest imtest (imap test
client) from cyrus-sasl. no success yet with mutt (with gsasl) though.
so a single user cannot fill up the disk.
by default, there is (still) no limit. a default can be set in the config file
for all accounts, and a per-account max size can be set that would override any
global setting.
this does not take into account disk usage of the index database. and also not
of any file system overhead.
