NOTIFY is like IDLE, but where IDLE watches just the selected mailbox, NOTIFY
can watch all mailboxes. With NOTIFY, a client can also ask a server to
immediately return configurable fetch attributes for new messages, e.g. a
message preview, certain header fields, or simply the entire message.
Mild testing with evolution and fairemail.
For long searches in big mailboxes, without any matches, we would previously
keep working and not say anything. Clients could interpret this silence as a
broken connection at some point. We now send a "we're still searching" untagged
OK responses with code INPROGRESS every 10 seconds while we're still searching,
to prevent the client from closing the connection. We also send how many
messages we've processed, and usually also how many we need to process in grand
total. Clients can use this to show a progress bar.
We were already generating previews of plain text parts for the webmail
interface, but we didn't store them, so were generating the previews each time
messages were listed.
Now we store previews in the database for faster handling. And we also generate
previews for html parts if needed. We use the first part that has textual
content.
For IMAP, the previews can be requested by an IMAP client. When we get the
"LAZY" variant, which doesn't require us to generate a preview, we generate it
anyway, because it should be fast enough. So don't make clients first ask for
"PREVIEW (LAZY)" and then again a request for "PREVIEW".
We now also generate a preview when a message is added to the account. Except
for imports. It would slow us down, the previews aren't urgent, and they will
be generated on-demand at first-request.
We normally check errors for all operations. But for some cleanup calls, eg
"defer file.Close()", we didn't. Now we also check and log most of those.
Partially because those errors can point to some mishandling or unexpected code
paths (eg file unexpected already closed). And in part to make it easier to use
"errcheck" to find the real missing error checks, there is too much noise now.
The log.Check function can now be used unconditionally for checking and logging
about errors. It adjusts the log level if the error is caused by a network
connection being closed, or a context is canceled or its deadline reached, or a
socket deadline is reached.
MULTIAPPEND modifies the existing APPEND command to allow multiple messages. it
is somewhat more involved than a regular append of a single message since the
operation (of adding multiple messages) must be atomic. either all are added,
or none are.
we check as early as possible if the messages won't cause an over-quota error.
only with "return" including "metadata". so clients can quickly get certain
metadata (eg for display, such as a color) for mailboxes.
this also adds a protocol token type "mailboxt" that properly encodes to utf7
if required.
to compress the entire IMAP connection. tested with thunderbird, meli, k9, ios
mail. the initial implementation had interoperability issues with some of these
clients: if they write the deflate stream and flush in "partial mode", the go
stdlib flate reader does not return any data (until there is an explicit
zero-length "sync flush" block, or until the history/sliding window is full),
blocking progress, resulting in clients closing the seemingly stuck connection
after considering the connection timed out. this includes a coy of the flate
package with a new reader that returns partially flushed blocks earlier.
this also adds imap trace logging to imapclient.Conn, which was useful for
debugging.
we already supported special-use flags. settable through the webmail interface,
and new accounts already got standard mailboxes with special-use flags
predefined. but now the IMAP "CREATE" command implements creating mailboxes
with special-use flags.
it makes a new field available on stored messages. not when they they were
received (over smtp) or appended to the mailbox (over imap), but when they were
last "saved" in the mailbox. copy/move of a message (eg to the trash) resets
the "savedate" value. this helps implement "remove messages from trash after X
days".
this allows setting per-mailbox and per-server annotations (metadata). we have
a fixed maximum for total number of annotations (1000) and their total size
(1000000 bytes). this size isn't held against the regular quota for simplicity.
we send unsolicited metadata responses when a connection is in the idle
command and a change to a metadata item is made.
we currently only implement the /private/ namespace. we should implement the
/shared/ namespace, for mox-global metadata annotations. only the admin should
be able to configure those, probably through the config file, cli, or admin web
interface.
for issue #290
not generating it yet from imapserver because we don't have content-md5
available. we could send "nil" instead of any actual content-md5 header (and
probably no contemporary messages include a content-md5 header), but it would
not be correct. if no known clients have problems in practice with absent
extensible data, it's better to just leave the bodystructure as is, with
extensible data.
for issue #217 by danieleggert
we only have a "storage" limit. for total disk usage. we don't have a limit on
messages (count) or mailboxes (count). also not on total annotation size, but
we don't have support annotations at all at the moment.
we don't implement setquota. with rfc 9208 that's allowed. with the previous
quota rfc 2087 it wasn't.
the status command can now return "DELETED-STORAGE". which should be the disk
space that can be reclaimed by removing messages with the \Deleted flags.
however, it's not very likely clients set the \Deleted flag without expunging
the message immediately. we don't want to go through all messages to calculate
the sum of message sizes with the deleted flag. we also don't currently track
that in MailboxCount. so we just respond with "0". not compliant, but let's
wait until someone complains.
when returning quota information, it is not possible to give the current usage
when no limit is configured. clients implementing rfc 9208 should probably
conclude from the presence of QUOTA=RES-* capabilities (only in rfc 9208, not
in 2087) and the absence of those limits in quota responses (or the absence of
an untagged quota response at all) that a resource type doesn't have a limit.
thunderbird will claim there is no quota information when no limit was
configured, so we can probably conclude that it implements rfc 2087, but not
rfc 9208.
we now also show the usage & limit on the account page.
for issue #115 by pmarini
so a single user cannot fill up the disk.
by default, there is (still) no limit. a default can be set in the config file
for all accounts, and a per-account max size can be set that would override any
global setting.
this does not take into account disk usage of the index database. and also not
of any file system overhead.
for conditional storing and quick resynchronisation (not sure if mail clients are actually using it that).
each message now has a "modseq". it is increased for each change. with
condstore, imap clients can request changes since a certain modseq. that
already allows quickly finding changes since a previous connection. condstore
also allows storing (e.g. setting new message flags) only when the modseq of a
message hasn't changed.
qresync should make it fast for clients to get a full list of changed messages
for a mailbox, including removals.
we now also keep basic metadata of messages that have been removed (expunged).
just enough (uid, modseq) to tell client that the messages have been removed.
this does mean we have to be careful when querying messages from the database.
we must now often filter the expunged messages out.
we also keep "createseq", the modseq when a message was created. this will be
useful for the jmap implementation.
the warnings that remained were either unused code that i wanted to use in the
future, or other type's of todo's. i've been mentally ignoring them, assuming i
would get back to them soon enough to fix them. but that hasn't happened yet,
and it's better to have a clean list with only actual isses.
the mailbox select/examine responses now return all flags used in a mailbox in
the FLAGS response. and indicate in the PERMANENTFLAGS response that clients
can set new keywords. we store these values on the new Message.Keywords field.
system/well-known flags are still in Message.Flags, so we're recognizing those
and handling them separately.
the imap store command handles the new flags. as does the append command, and
the search command.
we store keywords in a mailbox when a message in that mailbox gets the keyword.
we don't automatically remove the keywords from a mailbox. there is currently
no way at all to remove a keyword from a mailbox.
the import commands now handle non-system/well-known keywords too, when
importing from mbox/maildir.
jmap requires keyword support, so best to get it out of the way now.
