We now use "*string" for such header fields, for Content-* fields, as used in
the imapserver when responding to FETCH commands. We'll now return NIL for an
absent header, and "" (empty string) if the header value is empty.
The acc.Close() at the end of the fuzzing would find inconsistencies. For
example, message files on disk that aren't in the database file. I don't
understand what is happening there, the database file on disk does have those
messages, and it seems the database file is getting replaced. When running the
same code not as a fuzzing test but as a regular Go test doesn't show the
problem. So it seems to be some interaction with fuzzing. The problem is
"solved" (feels more like side-stepped), by starting each fuzz test with a
clean database. We still open & close the account in each fuzz test, and it
doesn't find consistency problems.
The imapclient needs more changes, like more strict parsing, before it can be a
generally usable IMAP client, these are a few steps towards that.
- Fix a bug in the imapserver METADATA responses for TOOMANY and MAXSIZE.
- Split low-level IMAP protocol handling (new Proto type) from the higher-level
client command handling (existing Conn type). The idea is that some simple
uses of IMAP can get by with just using these commands, while more intricate
uses of IMAP (like a synchronizing client that needs to talk to all kinds of
servers with different behaviours and implemented extensions) can write custom
commands and read untagged responses or command completion results
explicitly. The lower-level method names have clearer names now, like
ReadResponse instead of Response.
- Merge the untagged responses and (command completion) "Result" into a new
type Response. Makes function signatures simpler. And make Response implement
the error interface, and change command methods to return the Response as error
if the result is NO or BAD. Simplifies error handling, and still provides the
option to continue after a NO or BAD.
- Add UIDSearch/MSNSearch commands, with a custom "search program", so mostly
to indicate these commands exist.
- More complete coverage of types for response codes, for easier handling.
- Automatically handle any ENABLED or CAPABILITY untagged response or response
code for IMAP command methods on type Conn.
- Make difference between MSN vs UID versions of
FETCH/STORE/SEARCH/COPY/MOVE/REPLACE commands more clear. The original MSN
commands now have MSN prefixed to their name, so they are grouped together in
the documentation.
- Document which capabilities are needed for a command.
We were first getting UIDs in a transaction with a lock. Then getting the
changes and processing them in a special way. And then processing for qresync
in a new transaction. The special processing of changes is now gone, it seems
to have skipped adding/removing uids to the session, which can't be correct.
The new approach is just using a lock and transaction and process the whole
opening of the mailbox, and not processing any changes as part of the open, and
getting rid of the special "initial" mode processing a mailbox.
Once clients enable this extension, commands can no longer refer to "message
sequence numbers" (MSNs), but can only refer to messages with UIDs. This means
both sides no longer have to carefully keep their sequence numbers in sync
(error-prone), and don't have to keep track of a mapping of sequence numbers to
UIDs (saves resources).
With UIDONLY enabled, all FETCH responses are replaced with UIDFETCH response.
NOTIFY is like IDLE, but where IDLE watches just the selected mailbox, NOTIFY
can watch all mailboxes. With NOTIFY, a client can also ask a server to
immediately return configurable fetch attributes for new messages, e.g. a
message preview, certain header fields, or simply the entire message.
Mild testing with evolution and fairemail.
MIME returns the part headers. If 1 is a message, i.e. a message/rfc822 or
message/global, for example when top-level is a multipart/mixed, we were
returning the MIME headers from the message, not from the part.
We also shouldn't be returning a MIME-Version header or the separating newline
for MIME. Those are for MIME headers of a message, but the "MIME" fetch body
part is always about the part.
Found after looking into FETCH BODY handling for issue #327.
The gmail apps generate messages consisting of multipart/mixed, with text/html
referring to a sibling image/jpeg. We weren't resolving that cid before.
Related to issue #327.
The gmail iOS/Android app were showing mime image parts as (garbled) text
instead of rendering them as image. By returning all the optional fields in the
bodystructure fetch attribute, the gmail app renders the image as expected by
the user. So we now add all fields. We didn't before, because we weren't
keeping track of Content-MD5, Content-Language and Content-Location header
fields, since they aren't that useful.
Messages in mailboxes have to be reparsed:
./mox reparse
Without reparsing, imap responses will claim the extra fields
(content-disposition) are absent for existing messages, instead of not claiming
anything at all, which is what we did before.
Accounts and all/some mailboxes can get their "uid validity" bumped ("./mox
bumpuidvalidity $account [$mailbox]"), which should trigger clients to load all
messages from scratch, but gmail doesn't appear to notice, so it would be
better to remove & add the account in gmail.
For issue #327, also relevant to issue #217.
As required by RFC 2045 (MIME). The 78 byte lines work in practice, except that
SpamAssassin has rules that give messages with 78-byte lines spam points.
Mentioned by kjetilho on irc.
Before, we would only reconnect the SSE connection when the previous one lasted
10 minutes. For some reason, firefox disconnects SSE connections when there is
any network change. Running the docker integration tests changes the network a
few time in quick succession, prevent further automatic reconnects.
This changes the "stop reconnection automatically" period from 10 minutes to 5
seconds.
For long searches in big mailboxes, without any matches, we would previously
keep working and not say anything. Clients could interpret this silence as a
broken connection at some point. We now send a "we're still searching" untagged
OK responses with code INPROGRESS every 10 seconds while we're still searching,
to prevent the client from closing the connection. We also send how many
messages we've processed, and usually also how many we need to process in grand
total. Clients can use this to show a progress bar.
The default paths for the web interfaces, such as /admin/, /account/, /webmail/
and /webapi/ end with a slash. They should end with a slash because we use the
path when restricting cookies to just that web interface. You could configure
paths not ending with a slash, but due to using http.StripPrefix, and our
handler, some of those requests may not work properly.
We now warn if configured paths don't end with a trailing slash when parsing
the config file. We normally error out when such things happen, but users
probably have paths without trailing slashes configured, and we don't want to
break them on a future upgrade. We now use an effective path that includes the
trailing slash.
We would always redirect requests to the configured paths but without trailing
slash to the path with trailing slash, and that stays.
For issue #325 by odama626.
Download as eml is useful with firefox, because opening the raw message in a
new tab, and then downloading it, causes firefox to request the url without
cookies, causing it to save a "403 - forbidden" response.
Exporting a selection is useful during all kinds of testing. Makes it easy to
an entire thread, or just some messages.
The export popover now has buttons for each combination of mbox/maildir vs
zip/tgz/tar. Before you may have had to select the email format and archive
format first, followed by a click. Now it's just a click.
Attackers scanning the internet can use it to easily create a database of
hosts, software and versions. Let's not make it too easy to find old versions
that may be vulnerable to potential bugs found in the future. We could try
hiding the name "mox" as well, but the banner will still be identifyable, so
there isn't much point, and the public knowing approximately which software is
running can be useful for debugging.
The ID command in IMAP is used by clients to announce their software and
version. We only respond with our version when the user is authenticated.
There are still ways to discover the version number. But they don't involve
standard banner scanning, so someone would have to specifically target mox. We
could tighten that in the future.
For issue #322, based on email. Thanks everyone for discussing.
We were already generating previews of plain text parts for the webmail
interface, but we didn't store them, so were generating the previews each time
messages were listed.
Now we store previews in the database for faster handling. And we also generate
previews for html parts if needed. We use the first part that has textual
content.
For IMAP, the previews can be requested by an IMAP client. When we get the
"LAZY" variant, which doesn't require us to generate a preview, we generate it
anyway, because it should be fast enough. So don't make clients first ask for
"PREVIEW (LAZY)" and then again a request for "PREVIEW".
We now also generate a preview when a message is added to the account. Except
for imports. It would slow us down, the previews aren't urgent, and they will
be generated on-demand at first-request.
We normally check errors for all operations. But for some cleanup calls, eg
"defer file.Close()", we didn't. Now we also check and log most of those.
Partially because those errors can point to some mishandling or unexpected code
paths (eg file unexpected already closed). And in part to make it easier to use
"errcheck" to find the real missing error checks, there is too much noise now.
The log.Check function can now be used unconditionally for checking and logging
about errors. It adjusts the log level if the error is caused by a network
connection being closed, or a context is canceled or its deadline reached, or a
socket deadline is reached.
In store/search.go, we would make a copy of a byte array, but then still use
the original instead of the copy. Could result in search operations not finding
messages that do have the content, but under very unlikely conditions only.
We'll keep running ineffassign with "make check", useful enough.
Adding to the queue is done in a transaction, the queue db file is mox-global.
Appending the message to the Sent folder, removing it from Drafts, marking the
original message as answered/forwarded, is done in a separate database
transaction that gets the ctx passed in. If the ctx was canceled in between,
the queueing was finished, but the rest wasn't completed.
Reported by mteege, thanks!
On reply, with too many Cc/Bcc, I usually hit ctrl+backspace a few time. I just
want to clear the addresses, but I practically always still want a To address.
When removing an address, we want to make sure any queued messages for the
account still still have their address associated with the account. E.g.
through a catchall address.
Before removing an account, we fail deliveries still in the queue for the
account. We remove any addresses on the suppression list (which are stored in
the queue database, not the account database file that is removed completely).
We also clear all sessions for the webmail/webaccount interfaces. For the
webmail, further operations will fail, and the reconnection attempt will cause
the login popup with a message about an unknown session token.
Example symptom, when deleting a message in the webmail (which moves to Trash):
l=error m="duplicating message in old mailbox for current sessions" err="link data/accounts/mjl/msg/I/368638 data/accounts/mjl/msg/J/368640: no such file or directory" pkg=webmail
Problem introduced a few weeks ago, where moving messages starting duplicating
the message first, and the copy is erased once all references (in IMAP
sessions) to the old mailbox have been removed.
The simplistic logging approach we've followed so far is to not log struct
fields that are themselves structs, which time.Time is. So we skipped, but do
log it now.
