to work around clients, like the gmail smtp client, that tries to
authenticate with a webpki-issued certificate (which we don't know).
i tried specifying a list of accepted (subjects of) CA certs during the
tls handshake (with just 1 entry, with "xmox.nl" as common name), which
clients can use to influence their cert selection. however, the gmail
smtp client ignores it, so not a solution for the issue where this was
raised. also, specifying a list of accepted certs could cause other
clients to not send their client cert anymore, breaking existing setups.
i also considered only asking for tls client auth when at least one
account has a tls pubkey configured. but decided against it since any
account can add one on their own (without system admin interaction),
changing behaviour of the system and potentially breaking existing
submission/tls configs.
we now also print the "subject" and "issuer" of certs when tls client
auth fails, should be useful for future debugging.
for issue #359
another approach i looked at was enabling/disabling rate limiting per
web handler. but we want to apply the rate limit as early as possible
(not after we've already done quite some work for the request), and with
per-handler rate limits on/off the code would be sprinkled with calls to
rate limiting. this is probably good enough for now. other protocols are
less likely to need this.
we were always using the ip address of the connection for rate limiting.
but some setups have a reverse proxy in front. if any handler on a
http/https port is marked as "forwarded" (with a reverse proxy), we use
the ip address from the x-forwarded-for header (like we already did for
authentication requests over http).
for issue #346
Such as "-" when addresses are dmarc-reports@ and tls-reports@.
Existing configuration files can have these combinations. We don't allow them
to be created through the webadmin interface, as this is a likely source of
confusion about how addresses will be matched. We already didn't allow regular
addresses containing catchall separators.
The defaults for a new domain were dmarc-reports@ and tls-reports@. But some
setups use "-" as catchall separator, which currently would cause messages to
those addresses to be rejected with a "no such user" smtp error.
Better to prevent these issues in the future by using dmarcreports@ and
tlsreports@ localparts.
The config checks don't enforce that the DMARC and TLS reporting addresses
don't contain the localpart catchall separator. A next commit will fix
accepting incoming reports to such addresses.
With that recent change, we would keep track of Content-* headers of parsed
messages. We could ask admins to run a command to reparse messages for all
accounts. But instead we just do it automatically when opening the account. We
keep track whether we did the upgrade. And we do it in the background. Those
recent changes were to add optional fields to the IMAP fetch "bodystructure"
responses. There is a small chance that an IMAP client requests these fields
before they are properly populated with the reparse (only existing messages,
new incoming messages are parsed with the new code). We could try to detect
whether the upgrade has completed, and chance IMAP behaviour based on that. But
the complexity and long-term maintenance burden doesn't seem worth it. Worst
case, we'll temporarily claim some relatively unimportant headers aren't
present on a message. Most email clients won't even look at those fields, but
will parse them message themselves instead.
We now use "*string" for such header fields, for Content-* fields, as used in
the imapserver when responding to FETCH commands. We'll now return NIL for an
absent header, and "" (empty string) if the header value is empty.
The acc.Close() at the end of the fuzzing would find inconsistencies. For
example, message files on disk that aren't in the database file. I don't
understand what is happening there, the database file on disk does have those
messages, and it seems the database file is getting replaced. When running the
same code not as a fuzzing test but as a regular Go test doesn't show the
problem. So it seems to be some interaction with fuzzing. The problem is
"solved" (feels more like side-stepped), by starting each fuzz test with a
clean database. We still open & close the account in each fuzz test, and it
doesn't find consistency problems.
The imapclient needs more changes, like more strict parsing, before it can be a
generally usable IMAP client, these are a few steps towards that.
- Fix a bug in the imapserver METADATA responses for TOOMANY and MAXSIZE.
- Split low-level IMAP protocol handling (new Proto type) from the higher-level
client command handling (existing Conn type). The idea is that some simple
uses of IMAP can get by with just using these commands, while more intricate
uses of IMAP (like a synchronizing client that needs to talk to all kinds of
servers with different behaviours and implemented extensions) can write custom
commands and read untagged responses or command completion results
explicitly. The lower-level method names have clearer names now, like
ReadResponse instead of Response.
- Merge the untagged responses and (command completion) "Result" into a new
type Response. Makes function signatures simpler. And make Response implement
the error interface, and change command methods to return the Response as error
if the result is NO or BAD. Simplifies error handling, and still provides the
option to continue after a NO or BAD.
- Add UIDSearch/MSNSearch commands, with a custom "search program", so mostly
to indicate these commands exist.
- More complete coverage of types for response codes, for easier handling.
- Automatically handle any ENABLED or CAPABILITY untagged response or response
code for IMAP command methods on type Conn.
- Make difference between MSN vs UID versions of
FETCH/STORE/SEARCH/COPY/MOVE/REPLACE commands more clear. The original MSN
commands now have MSN prefixed to their name, so they are grouped together in
the documentation.
- Document which capabilities are needed for a command.
We were first getting UIDs in a transaction with a lock. Then getting the
changes and processing them in a special way. And then processing for qresync
in a new transaction. The special processing of changes is now gone, it seems
to have skipped adding/removing uids to the session, which can't be correct.
The new approach is just using a lock and transaction and process the whole
opening of the mailbox, and not processing any changes as part of the open, and
getting rid of the special "initial" mode processing a mailbox.
Once clients enable this extension, commands can no longer refer to "message
sequence numbers" (MSNs), but can only refer to messages with UIDs. This means
both sides no longer have to carefully keep their sequence numbers in sync
(error-prone), and don't have to keep track of a mapping of sequence numbers to
UIDs (saves resources).
With UIDONLY enabled, all FETCH responses are replaced with UIDFETCH response.
NOTIFY is like IDLE, but where IDLE watches just the selected mailbox, NOTIFY
can watch all mailboxes. With NOTIFY, a client can also ask a server to
immediately return configurable fetch attributes for new messages, e.g. a
message preview, certain header fields, or simply the entire message.
Mild testing with evolution and fairemail.
MIME returns the part headers. If 1 is a message, i.e. a message/rfc822 or
message/global, for example when top-level is a multipart/mixed, we were
returning the MIME headers from the message, not from the part.
We also shouldn't be returning a MIME-Version header or the separating newline
for MIME. Those are for MIME headers of a message, but the "MIME" fetch body
part is always about the part.
Found after looking into FETCH BODY handling for issue #327.
The gmail apps generate messages consisting of multipart/mixed, with text/html
referring to a sibling image/jpeg. We weren't resolving that cid before.
Related to issue #327.
The gmail iOS/Android app were showing mime image parts as (garbled) text
instead of rendering them as image. By returning all the optional fields in the
bodystructure fetch attribute, the gmail app renders the image as expected by
the user. So we now add all fields. We didn't before, because we weren't
keeping track of Content-MD5, Content-Language and Content-Location header
fields, since they aren't that useful.
Messages in mailboxes have to be reparsed:
./mox reparse
Without reparsing, imap responses will claim the extra fields
(content-disposition) are absent for existing messages, instead of not claiming
anything at all, which is what we did before.
Accounts and all/some mailboxes can get their "uid validity" bumped ("./mox
bumpuidvalidity $account [$mailbox]"), which should trigger clients to load all
messages from scratch, but gmail doesn't appear to notice, so it would be
better to remove & add the account in gmail.
For issue #327, also relevant to issue #217.
As required by RFC 2045 (MIME). The 78 byte lines work in practice, except that
SpamAssassin has rules that give messages with 78-byte lines spam points.
Mentioned by kjetilho on irc.
Before, we would only reconnect the SSE connection when the previous one lasted
10 minutes. For some reason, firefox disconnects SSE connections when there is
any network change. Running the docker integration tests changes the network a
few time in quick succession, prevent further automatic reconnects.
This changes the "stop reconnection automatically" period from 10 minutes to 5
seconds.
For long searches in big mailboxes, without any matches, we would previously
keep working and not say anything. Clients could interpret this silence as a
broken connection at some point. We now send a "we're still searching" untagged
OK responses with code INPROGRESS every 10 seconds while we're still searching,
to prevent the client from closing the connection. We also send how many
messages we've processed, and usually also how many we need to process in grand
total. Clients can use this to show a progress bar.
The default paths for the web interfaces, such as /admin/, /account/, /webmail/
and /webapi/ end with a slash. They should end with a slash because we use the
path when restricting cookies to just that web interface. You could configure
paths not ending with a slash, but due to using http.StripPrefix, and our
handler, some of those requests may not work properly.
We now warn if configured paths don't end with a trailing slash when parsing
the config file. We normally error out when such things happen, but users
probably have paths without trailing slashes configured, and we don't want to
break them on a future upgrade. We now use an effective path that includes the
trailing slash.
We would always redirect requests to the configured paths but without trailing
slash to the path with trailing slash, and that stays.
For issue #325 by odama626.
