Erasing a message (removing the message file from the file system) was made a
separate step a few days ago. The verifydata command checks for consistency of
the data, but didn't correctly skip checking expunged-but-not-yet-erased
messages, leading to the warning.
A similar consistency check in store/account.go does check for that.
I was warned by my nightly backup+veridata script.
In case the precis check failed, our return of a nil account cleared acc, and
we were then trying to close it, returning in a nil pointer dereference.
Rewrite the return statements so we don't overwrite the named return variables.
Especially relevant when the name contains a comma, e.g. "lastname, firstname".
Or when it contains parentheses, e.g. "(organization)".
When sending to an address with a comma that isn't quoted, we would actually
interpret it as two addresses: One without an "@" before the comma, and the
second part after the comma with half of the name and the email addrss. This
resulted in an error message.
When sending to a recipient with unquoted parentheses in the name, those
parentheses would be interpreted as an generic email header comment, and left
out.
For issue #305 by mattfbacon.
The original config option stays, and we still use it for the common case where
we have a single separator. The "+" is configured by default. It is optional,
just like the new option "LocalpartCatchallSeparators" (plural).
When parsing the config file, we combine LocalpartCatchallSeparator and
LocalpartCatchallSeparators into a single list
LocalpartCatchallSeparatorsEffective, which we use throughout the code.
For issue #301 by janc13
Mox does not look up names from the /etc/hosts file, only through DNS. But
"localhost" may not resolve through DNS, or when offline a DNS server may not
even be available. We will want deliveries to work in "mox localserve" mode.
Found by dstotijn.
We'll need RSA DKIM keys for a long time to come because many systems don't
support ed25519 DKIM signatures. We've been adding both types of keys when
adding a new domain, and adding both two DKIM signatures to outgoing messages.
This works fine in practice, other mail servers are correctly ignoring the
ed25519 signature if they don't understand it. Unfortunately, it causess noise
in DMARC reports: Systems will warn that a DKIM check failed. Sometimes with a
vague message about a missing key, or a 0-bit key. Sometimes they leave the
selector out of the report, making it hard to understand what's going on. This
causes postmasters to investigate because they think something is wrong, only
to eventually find out it's all fine. So we're causing needless chores for
postmasters. By having only an RSA DKIM signature, we skip that noise. This
also reduces the number of DNS records postmasters have to add for a domain.
The small ed25519 DKIM DNS TXT records would make them preferrable over the
long multi-string RSA DKIM DNS TXT records (which are often hard to add
correctly through DNS operator web interfaces), but as mentioned, we'll have to
add the RSA DKIM keys anyway.
Another reason why RSA keys _may_ be preferrable over ed25519 keys is that with
RSA, signing is more computationally expensive than verifying, while it's the
other way around for ed25519 keys.
Admins can always add an ed25519 DKIM key to their domain. And we can always
switch back to adding them to new domains by default in the future.
For issue #299.
Just to be cautious. This hasn't happened yet in practice that I'm aware of.
But in theory, mox could crash after it has written the message file during
delivery, but before the database transaction was committed. In that case, a
message file for the "next message id" is already present. Any future delivery
attempts will get assigned the same message id by the database, but writing the
file will fail because there already is one, causing delivery to fail (until
the file is moved away).
When opening an account, we now check in the file system if newer files exist
than we expect based on the last existing message in the database. If so, we
adjust the message ID the database will assign next.
It still blocks on reading partial flushes from clients, preventing progress
and eventually timing out. The flate library needs more changes to make this
work.
Connections from iOS mail sometimes timed out, not always.
The extension is simply not announced, code is still present.
The Open call returns an errno ENAMETOOLONG. We didn't handle that specially,
so turned it into a "500 internal server error" response. When serving static
files, we should just return "404 file not found" errors. The file obviously
does not exist.
Saw a few overlong requests from bad bots not recognizing "data:" uri's inlined
in html files, trying to request them.
Broken since introducing LoginAttempts. The fuzzing functions didn't get the
store.Init() call, and would hang on trying to send to the loginattemptwriter.
Keeping the message files around, and the message details in the database, is
useful for IMAP sessions that haven't seen/processed the removal of a message
yet and try to fetch it. Before, we would return errors. Similarly, a session
that has a mailbox selected that is removed can (at least in theory) still read
messages.
The mechanics to do this need keeping removed mailboxes around too. JMAP needs
that anyway, so we now keep modseq/createseq/expunged history for mailboxes
too. And while we're at it, for annotations as well.
For future JMAP support, we now also keep the mailbox parent id around for a
mailbox, with an upgrade step to set the field for existing mailboxes and
fixing up potential missing parents (which could possibly have happened in an
obscure corner case that I doubt anyone ran into).
We normally recover from those situations, printing stack traces instead of
crashing the program. But during tests, we're not looking at the prometheus
metrics or all the output. Without these checks, such panics could go
unnoticed. Seems like a reasonable thing to add, unhandled panics haven't been
encountered in tests.
DeliverMessage() is now MessageAdd(), and it takes a Mailbox object that it
modifies but doesn't write to the database (the caller must do it, and plenty
of times can do it more efficiently by doing it once for multiple messages).
The new AddOpts let the caller influence how many checks and how much of the
work MessageAdd() does. The zero-value AddOpts enable all checks and all the
work, but callers can take responsibility of some of the checks/work if it can
do it more efficiently itself.
This simplifies the code in most places, and makes it more efficient. The
checks to update per-mailbox keywords is a bit simpler too now.
We are also more careful to close the junk filter without saving it in case of
errors.
Still part of more upcoming changes.
In the common case, it's the same as the previous delivery. That means we don't
have to try to create the directory (fewer syscalls) and that we can sync the
dir to disk.
This also tweaks the defer handling in case of a late failure.
We effectively held the account write-locked by using a writable transaction
while processing the FETCH command. We did this because we may have to update
\Seen flags, for non-PEEK attribute fetches. This meant other FETCHes would
block, and other write access to the account too.
We now read the messages in a read-only transaction. We gather messages that
need marking as \Seen, and make that change in one (much shorter) database
transaction at the end of the FETCH command.
In practice, it doesn't seem too sensible to mark messages as seen
automatically. Most clients probably use the PEEK-variant of attribute fetches.
Related to issue #128.
The previous commit fixed an array out of bounds access that resulted in a
panic on an smtpserver connection. The panic is recovered and marked as
"unhandled panic" in metrics and the connection closed.
Fix up a test or two. Simplify the XOR logic when we train the junk filter:
Only if junk or nonjunk is set, but not when both (or none are set). i.e. when
the values aren't the same.
Locking the account when we do consistency checks prevents spurious test
failures that may have been introduced in the previous commit.
Writing to a connection goes through the flate library to compress. That writes
the compressed bytes to the underlying connection. But that underlying
connection is wrapped to raise a panic with an i/o error instead of returning a
normal error. Jumping out of flate leaves the internal state of the compressor
in undefined state. So far so good. But as part of cleaning up the connection,
we could try to flush output again. Specifically: If we were writing user data,
we had switched from tracing of protocol data to tracing of user data, and we
registered a defer that restored the tracing kind and flushed (to ensure data
was traced at the right level). That flush would cause a write into the
compressor again, which could panic with an out of bounds slice access due to
its inconsistent internal state.
This fix prevents that compressor panic in two ways:
1. We wrap the flate.Writer with a moxio.FlateWriter that keeps track of
whether a panic came out of an operation on it. If so, any further operation
raises the same panic. This prevents access to the inconsistent internal flate
state entirely.
2. Once we raise an i/o error, we mark the connection as broken and that makes
flushes a no-op.
REPLACE can be used to update draft messages as you are editing. Instead of
requiring an APPEND and STORE of \Deleted and EXPUNGE. REPLACE works
atomically.
It has a syntax similar to APPEND, just allows you to specify the message to
replace that's in the currently selected mailbox. The regular REPLACE-command
works on a message sequence number, the UID REPLACE commands on a uid. The
destination mailbox, of the updated message, can be different. For example to
move a draft message from the Drafts folder to the Sent folder.
We have to do quite some bookkeeping, e.g. for updating (message) counts for
the mailbox, checking quota, un/retraining the junk filter. During a
synchronizing literal, we check the parameters early and reject if the replace
would fail (eg over quota, bad destination mailbox).
MULTIAPPEND modifies the existing APPEND command to allow multiple messages. it
is somewhat more involved than a regular append of a single message since the
operation (of adding multiple messages) must be atomic. either all are added,
or none are.
we check as early as possible if the messages won't cause an over-quota error.
