to work around clients, like the gmail smtp client, that tries to
authenticate with a webpki-issued certificate (which we don't know).
i tried specifying a list of accepted (subjects of) CA certs during the
tls handshake (with just 1 entry, with "xmox.nl" as common name), which
clients can use to influence their cert selection. however, the gmail
smtp client ignores it, so not a solution for the issue where this was
raised. also, specifying a list of accepted certs could cause other
clients to not send their client cert anymore, breaking existing setups.
i also considered only asking for tls client auth when at least one
account has a tls pubkey configured. but decided against it since any
account can add one on their own (without system admin interaction),
changing behaviour of the system and potentially breaking existing
submission/tls configs.
we now also print the "subject" and "issuer" of certs when tls client
auth fails, should be useful for future debugging.
for issue #359
The acc.Close() at the end of the fuzzing would find inconsistencies. For
example, message files on disk that aren't in the database file. I don't
understand what is happening there, the database file on disk does have those
messages, and it seems the database file is getting replaced. When running the
same code not as a fuzzing test but as a regular Go test doesn't show the
problem. So it seems to be some interaction with fuzzing. The problem is
"solved" (feels more like side-stepped), by starting each fuzz test with a
clean database. We still open & close the account in each fuzz test, and it
doesn't find consistency problems.
The imapclient needs more changes, like more strict parsing, before it can be a
generally usable IMAP client, these are a few steps towards that.
- Fix a bug in the imapserver METADATA responses for TOOMANY and MAXSIZE.
- Split low-level IMAP protocol handling (new Proto type) from the higher-level
client command handling (existing Conn type). The idea is that some simple
uses of IMAP can get by with just using these commands, while more intricate
uses of IMAP (like a synchronizing client that needs to talk to all kinds of
servers with different behaviours and implemented extensions) can write custom
commands and read untagged responses or command completion results
explicitly. The lower-level method names have clearer names now, like
ReadResponse instead of Response.
- Merge the untagged responses and (command completion) "Result" into a new
type Response. Makes function signatures simpler. And make Response implement
the error interface, and change command methods to return the Response as error
if the result is NO or BAD. Simplifies error handling, and still provides the
option to continue after a NO or BAD.
- Add UIDSearch/MSNSearch commands, with a custom "search program", so mostly
to indicate these commands exist.
- More complete coverage of types for response codes, for easier handling.
- Automatically handle any ENABLED or CAPABILITY untagged response or response
code for IMAP command methods on type Conn.
- Make difference between MSN vs UID versions of
FETCH/STORE/SEARCH/COPY/MOVE/REPLACE commands more clear. The original MSN
commands now have MSN prefixed to their name, so they are grouped together in
the documentation.
- Document which capabilities are needed for a command.
Initialize store and switchboard first, then open account, and close in reverse
order.
Replace all "CheckClosed" calls with "WaitClosed", future changings will keep
an account reference open for a bit after the last regular close, so we can't
know that an account should be closed during tests.
Remove one parameter from the (still too long) "start test server" function in
imapserver testing code.
Broken since introducing LoginAttempts. The fuzzing functions didn't get the
store.Init() call, and would hang on trying to send to the loginattemptwriter.
Keeping the message files around, and the message details in the database, is
useful for IMAP sessions that haven't seen/processed the removal of a message
yet and try to fetch it. Before, we would return errors. Similarly, a session
that has a mailbox selected that is removed can (at least in theory) still read
messages.
The mechanics to do this need keeping removed mailboxes around too. JMAP needs
that anyway, so we now keep modseq/createseq/expunged history for mailboxes
too. And while we're at it, for annotations as well.
For future JMAP support, we now also keep the mailbox parent id around for a
mailbox, with an upgrade step to set the field for existing mailboxes and
fixing up potential missing parents (which could possibly have happened in an
obscure corner case that I doubt anyone ran into).
to compress the entire IMAP connection. tested with thunderbird, meli, k9, ios
mail. the initial implementation had interoperability issues with some of these
clients: if they write the deflate stream and flush in "partial mode", the go
stdlib flate reader does not return any data (until there is an explicit
zero-length "sync flush" block, or until the history/sliding window is full),
blocking progress, resulting in clients closing the seemingly stuck connection
after considering the connection timed out. this includes a coy of the flate
package with a new reader that returns partially flushed blocks earlier.
this also adds imap trace logging to imapclient.Conn, which was useful for
debugging.
logging of login attempts happens in the background, because we don't want to
block regular operation with disk since for such logging. however, when a line
is logged, we evaluate some attributes of a connection, notably the username.
but about when we do authentication, we change the username on a connection. so
we were reading and writing at the same time. this is now fixed by evaluating
the attributes before we pass off the logger to the goroutine.
found by the go race detector.
for admins to open an imap connection preauthenticated for an account (by address), also when
it is disabled for logins.
useful for migrations. the admin typically doesn't know the password of the
account, so couldn't get an imap session (for synchronizing) before.
tested with "mox localserve" and running:
mutt -e 'set tunnel="MOXCONF=/home/mjl/.config/mox-localserve/mox.conf ./mox admin imapserve mox@localhost"'
may also work with interimap, but untested.
i initially assumed imap would be done fully on file descriptor 0, but mutt
expects imap output on fd 1. that's the default now. flag -fd0 is for others
that expect it on fd0.
for issue #175, suggested by DanielG
to facilitate migrations from/to other mail setups.
a domain can be added in "disabled" mode (or can be disabled/enabled later on).
you can configure a disabled domain, but incoming/outgoing messages involving
the domain are rejected with temporary error codes (as this may occur during a
migration, remote servers will try again, hopefully to the correct machine or
after this machine has been configured correctly). also, no acme tls certs will
be requested for disabled domains (the autoconfig/mta-sts dns records may still
point to the current/previous machine). accounts with addresses at disabled
domains can still login, unless logins are disabled for their accounts.
an account now has an option to disable logins. you can specify an error
message to show. this will be shown in smtp, imap and the web interfaces. it
could contain a message about migrations, and possibly a URL to a page with
information about how to migrate. incoming/outgoing email involving accounts
with login disabled are still accepted/delivered as normal (unless the domain
involved in the messages is disabled too). account operations by the admin,
such as importing/exporting messages still works.
in the admin web interface, listings of domains/accounts show if they are disabled.
domains & accounts can be enabled/disabled through the config file, cli
commands and admin web interface.
for issue #175 by RobSlgm
Intended for future use with chatmail servers. Standard email ports may be
blocked on some networks, while the HTTPS port may be accessible.
This is a squashed commit of PR #255 by s0ph0s-dog.
in the precis password change before the previous release, the password used in
fuzzing wasn't correct, triggering sleeps due to botched protocols often, which
made the tests run much longer.
an é (e with accent) can also be written as e+\u0301. the first form is NFC,
the second NFD. when logging in, we transform usernames (email addresses) to
NFC. so both forms will be accepted. if a client is using NFD, they can log
in too.
for passwords, we apply the PRECIS "opaquestring", which (despite the name)
transforms the value too: unicode spaces are replaced with ascii spaces. the
string is also normalized to NFC. PRECIS may reject confusing passwords when
you set a password.
we don't want external software to include internal details like mlog.
slog.Logger is/will be the standard.
we still have mlog for its helper functions, and its handler that logs in
concise logfmt used by mox.
packages that are not meant for reuse still pass around mlog.Log for
convenience.
we use golang.org/x/exp/slog because we also support the previous Go toolchain
version. with the next Go release, we'll switch to the builtin slog.
getting mox to compile required changing code in only a few places where
package "syscall" was used: for accessing file access times and for umask
handling. an open problem is how to start a process as an unprivileged user on
windows. that's why "mox serve" isn't implemented yet. and just finding a way
to implement it now may not be good enough in the near future: we may want to
starting using a more complete privilege separation approach, with a process
handling sensitive tasks (handling private keys, authentication), where we may
want to pass file descriptors between processes. how would that work on
windows?
anyway, getting mox to compile for windows doesn't mean it works properly on
windows. the largest issue: mox would normally open a file, rename or remove
it, and finally close it. this happens during message delivery. that doesn't
work on windows, the rename/remove would fail because the file is still open.
so this commit swaps many "remove" and "close" calls. renames are a longer
story: message delivery had two ways to deliver: with "consuming" the
(temporary) message file (which would rename it to its final destination), and
without consuming (by hardlinking the file, falling back to copying). the last
delivery to a recipient of a message (and the only one in the common case of a
single recipient) would consume the message, and the earlier recipients would
not. during delivery, the already open message file was used, to parse the
message. we still want to use that open message file, and the caller now stays
responsible for closing it, but we no longer try to rename (consume) the file.
we always hardlink (or copy) during delivery (this works on windows), and the
caller is responsible for closing and removing (in that order) the original
temporary file. this does cost one syscall more. but it makes the delivery code
(responsibilities) a bit simpler.
there is one more obvious issue: the file system path separator. mox already
used the "filepath" package to join paths in many places, but not everywhere.
and it still used strings with slashes for local file access. with this commit,
the code now uses filepath.FromSlash for path strings with slashes, uses
"filepath" in a few more places where it previously didn't. also switches from
"filepath" to regular "path" package when handling mailbox names in a few
places, because those always use forward slashes, regardless of local file
system conventions. windows can handle forward slashes when opening files, so
test code that passes path strings with forward slashes straight to go stdlib
file i/o functions are left unchanged to reduce code churn. the regular
non-test code, or test code that uses path strings in places other than
standard i/o functions, does have the paths converted for consistent paths
(otherwise we would end up with paths with mixed forward/backward slashes in
log messages).
windows cannot dup a listening socket. for "mox localserve", it isn't
important, and we can work around the issue. the current approach for "mox
serve" (forking a process and passing file descriptors of listening sockets on
"privileged" ports) won't work on windows. perhaps it isn't needed on windows,
and any user can listen on "privileged" ports? that would be welcome.
on windows, os.Open cannot open a directory, so we cannot call Sync on it after
message delivery. a cursory internet search indicates that directories cannot
be synced on windows. the story is probably much more nuanced than that, with
long deep technical details/discussions/disagreement/confusion, like on unix.
for "mox localserve" we can get away with making syncdir a no-op.
for conditional storing and quick resynchronisation (not sure if mail clients are actually using it that).
each message now has a "modseq". it is increased for each change. with
condstore, imap clients can request changes since a certain modseq. that
already allows quickly finding changes since a previous connection. condstore
also allows storing (e.g. setting new message flags) only when the modseq of a
message hasn't changed.
qresync should make it fast for clients to get a full list of changed messages
for a mailbox, including removals.
we now also keep basic metadata of messages that have been removed (expunged).
just enough (uid, modseq) to tell client that the messages have been removed.
this does mean we have to be careful when querying messages from the database.
we must now often filter the expunged messages out.
we also keep "createseq", the modseq when a message was created. this will be
useful for the jmap implementation.
this is quite common on the internet. the other side may be trying some other
protocol, e.g. http, or some common vulnerability. we don't want to spam our
own logs with multiple invalid lines. if the first command is valid, but later
are not, we'll keep trying to process them. so this only affects protocol
sessions that are very likely not smtp/imap.
also remove a few more sleeps during tests, making imapserver and smtpserver tests a bit faster.
