as the recent Go patch release showed, textproto.ReadMIMEHeaders is parsing
http headers, strictly. too strict for email message headers. valid headers,
e.g. with a slash in them, were rejected by textproto.ReadMIMEHeaders.
the functions in Go's mail package handle RFC 2047 charset-encoded words in
address headers. it can do that because we tell it those headers are addresses,
where such encodings are valid. but that encoding isn't valid in all places in
all headers. for other cases, we must decode explicitly, such as for the
subject header.
with this change, some messages that could not be parsed before can now be
parsed (where headers were previously rejected for being invalid). and the
subject of parsed messages could now be properly decoded. you could run "mox
ensureparsed -all <account>" (while mox isn't running) to force reparsing all
messages. mox needs a subcommand to reparse while running...
it wasn't much of a problem before, because imap email clients typically do
their own parsing (of headers, including subject decoding) again. but with the
upcoming webmail client, any wrong parsing quickly reveals itself.
message.Part now has a ReaderUTF8OrBinary() along with the existing Reader().
the new function returns a reader of decoded content. we now use it in a few
places, including search. we only support the charsets in
golang.org/x/text/encoding/ianaindex.
search has also been changed to not read the entire message in memory. instead,
we make one 8k buffer for reading and search in that, and we keep the buffer
around for all messages. saves quite some allocations when searching large
mailboxes.
the space is explicitly mentioned as not valid in rfc 5321, but it clients do
send it, such as microsoft outlook 365 apps for enterprise. no need to punish
such users, we'll allow it. but only for submission, not regular smtp, because
it is normally a sign of a spammer. we still don't allow it in pedantic mode
(as used by localserve).
for issue #51 by hmfaysal, thanks for reporting and testing!
i always get bitten by some caching or missing checks when i use docker...
Dockerfile.moxmail doesn't exist anymore, but that doesn't matter, it doesn't
even look at it but will just use some image that is still around (based on the
name?) i suppose that means docker-compose also doesn't rebuild an image when
the dockerfile mentioned in the build changes.
accounts with a mailbox with this problem can be fixed by running the "mox
fixuidmeta <account>" command.
we were resetting the mailbox uidnext after delivering messages when we were
setting new keywords on the mailbox at the end of the import. so in a future
delivery attempt to that mailbox, a uid would be chosen that was already
present.
the fix is to fetch the updated mailbox from the database before setting the
new keywords.
http/import.go doesn't have this bug because it was already fetching the
mailbox before updating keywords (because it can import into many mailboxes,
so different code).
the "mox verifydata" command (recommended with backups) also warns about this
issue (but doesn't fix it)
found while working on new functionality (webmail).
at least the android gmail/mail app doesn't show messages without content-type
header. i believe missing content-type is meant to be interpreted as
text/plain, but doesn't hurt to be explicit.
for conditional storing and quick resynchronisation (not sure if mail clients are actually using it that).
each message now has a "modseq". it is increased for each change. with
condstore, imap clients can request changes since a certain modseq. that
already allows quickly finding changes since a previous connection. condstore
also allows storing (e.g. setting new message flags) only when the modseq of a
message hasn't changed.
qresync should make it fast for clients to get a full list of changed messages
for a mailbox, including removals.
we now also keep basic metadata of messages that have been removed (expunged).
just enough (uid, modseq) to tell client that the messages have been removed.
this does mean we have to be careful when querying messages from the database.
we must now often filter the expunged messages out.
we also keep "createseq", the modseq when a message was created. this will be
useful for the jmap implementation.
the warnings that remained were either unused code that i wanted to use in the
future, or other type's of todo's. i've been mentally ignoring them, assuming i
would get back to them soon enough to fix them. but that hasn't happened yet,
and it's better to have a clean list with only actual isses.
nowadays the quickstart will warn about this, but it may be missed/ignored. and
users that installed mox a few versions ago never got the warning. so now we
keep warning about it in the dns check.
based on feedback from Mendel on slack, thanks!
so mail user agents will show DSNs threaded/grouped with the original message.
we store the MessageID in the message queue, so we have the value within reach
when we need it.
i saw a references header in a DSN from gmail on a test account. makes sense to me.
when broadcasting a change, we would try to send the changes on a channel,
non-blocking. if we couldn't send (because there was no pending blocked
receive), we would wait until the potential receiver would explicitly request
the changes. however, the imap idle handler would not explicitly request the
changes, but do a receive on the changes channel. since there was no pending
blocked send on the channel, that receive would block. only when another event
would come in, would both the pending and the new changes be sent.
we now use a channel only for signaling there are pending changes. the channel
is buffered, so when broadcasting we can just set the signal by a non-blocking
send and continue with the next listener. the receiver will get the buffered
signal. it can then get the changes directly, but lock-protected.
found when looking at a missing/delayed new message notification in thunderbird
when two messages arrive immediately after each other. this doesn't fix that
problem though: it seems thunderbird just ignores imap untagged "exists"
messages (indicating a new message arrived) during the "uid fetch" command that
it issued after notifications from an "idle" command.
and add a bit more logging for unexpected failures when closing files.
and make tests pass with a TMPDIR on a different filesystem than the testdata directory.
an EHLO ipv4 address looks like this: "[1.2.3.4]". for ipv6, the syntax is:
"[IPv6🔡:1]". mail user agents aren't as careful in compliance as smtp
servers. for incoming messages from smtp servers, we want to be strict (we're
eager to find a reason not to accept spam messages, and not adhering to the
standards is usually a strong spam signal), but there is no reason to punish
authenticated users.
for the syntax requirements, see ABNF rule "address-literal" in rfc 5321.
for issue #48 by @bobobo1618, thanks!
if a window user visited beta.gobuilds.org, they would be redirected to the
windows build, which would fail. better point them to a working build that
shows links to the platform they may actually need.
the import was still processed, but the SSE connection to fetch progress did
not work since adding the loggingWriter.
found while working on other functionality that uses SSE.
it's still not great. better to automatically change domains.conf. but that
would currently rewrite the whole file, which may not be what admins that
manually edit expect, it would remove their comments. we need better
config-update code.
for issue #40
the trailing slash is commonly forgotten. in the default setup, for the admin
endpoint, this makes you end up at the account endpoint, which won't accept
your admin credentials. with this change, users won't get confused by that
anymore.
for issue #43
this doesn't really test the output of the ctl commands, just that they succeed
without error. better than nothing...
testing found two small bugs, that are not an issue in practice:
1. we were ack'ing streamed data from the other side of the ctl connection
before having read it. when there is no buffer space on the connection (always
the case for net.Pipe) that would cause a deadlock. only actually happened
during the new tests.
2. the generated dkim keys are relatively to the directory of the dynamic
config file. mox looked it up relative to the directory of the _static_ config
file at startup. this directory is typicaly the same. users would have noticed
if they had triggered this.
so mox doesn't have to be running when you run it.
will be useful for testing in the near future.
this also moves cpuprof and memprof cli flags to top-level flag parsing, so all
commands can use them.
we could make more types of delays configurable. the current approach isn't
great, as it results in an a default value of "0s" in the config file, while
the actual default is 15s (which is documented just above, but still).
renaming inbox is special. the mailbox isn't renamed, but its messages moved to
a new mailbox. we weren't updating the destination mailbox uidnext with the new
messages. the fix not only sets the uidnext correctly, but also renumbers the
uids, starting at 1.
this also adds a consistency check for message uids and mailbox uidnexts, and
for mailbox uidvalidity account nextuidvalidity in "mox verifydata".
this also adds command "mox fixuidmeta" (not listed) that fixes up mailbox uidnext
and account uidvalidity. and command "mox reassignuids" that will renumber the
uids for either one or all mailboxes in an account.
for a uid set, the syntax <num>:* must be interpreted as <num>:<maxuid>. a
wrong check turned the uid set into <maxuid>:<maxuid>. that check was meant for
the case where <num> is higher than <maxuid>, in which case num must be
replaced with maxuid.
this affected "uid expunge" with a uid set, possibly causing messages marked
for deletion not to be actually removed, and this affected "search" with the
uid parameter, possibly not returning all messages that were searched for.
found while writing tests for upcoming condstore/qresync extensions.
these could cause the parser to reject correct commands.
the first bug is about the allowed chars for an "atom", we were accepting too
many. this probably isn't easily triggered in practice.
the second bug is about how numbers (digits) are parsed. when gathering digits
to parse as number, we didn't consider only the directly upcoming digits that
make up the number, but continued looking for digits later on in the command.
then we tried to parse a string that was too long as a number, which would fail
because of additional characters. this could have been triggered with commands
containing two numbers. this is possible with e.g. "tag search or larger 123
smaller 123", the "or" takes two search keys again, each with a number. not too
common, but can happen.
found while writing tests for upcoming condstore/qresync implementation.
the mailbox select/examine responses now return all flags used in a mailbox in
the FLAGS response. and indicate in the PERMANENTFLAGS response that clients
can set new keywords. we store these values on the new Message.Keywords field.
system/well-known flags are still in Message.Flags, so we're recognizing those
and handling them separately.
the imap store command handles the new flags. as does the append command, and
the search command.
we store keywords in a mailbox when a message in that mailbox gets the keyword.
we don't automatically remove the keywords from a mailbox. there is currently
no way at all to remove a keyword from a mailbox.
the import commands now handle non-system/well-known keywords too, when
importing from mbox/maildir.
jmap requires keyword support, so best to get it out of the way now.
