implement message threading in backend and webmail

we match messages to their parents based on the "references" and "in-reply-to"
headers (requiring the same base subject), and in absense of those headers we
also by only base subject (against messages received max 4 weeks ago).

we store a threadid with messages. all messages in a thread have the same
threadid.  messages also have a "thread parent ids", which holds all id's of
parent messages up to the thread root.  then there is "thread missing link",
which is set when a referenced immediate parent wasn't found (but possibly
earlier ancestors can still be found and will be in thread parent ids".

threads can be muted: newly delivered messages are automatically marked as
read/seen.  threads can be marked as collapsed: if set, the webmail collapses
the thread to a single item in the basic threading view (default is to expand
threads).  the muted and collapsed fields are copied from their parent on
message delivery.

the threading is implemented in the webmail. the non-threading mode still works
as before. the new default threading mode "unread" automatically expands only
the threads with at least one unread (not seen) meessage. the basic threading
mode "on" expands all threads except when explicitly collapsed (as saved in the
thread collapsed field). new shortcuts for navigation/interaction threads have
been added, e.g. go to previous/next thread root, toggle collapse/expand of
thread (or double click), toggle mute of thread. some previous shortcuts have
changed, see the help for details.

the message threading are added with an explicit account upgrade step,
automatically started when an account is opened. the upgrade is done in the
background because it will take too long for large mailboxes to block account
operations. the upgrade takes two steps: 1. updating all message records in the
database to add a normalized message-id and thread base subject (with "re:",
"fwd:" and several other schemes stripped). 2. going through all messages in
the database again, reading the "references" and "in-reply-to" headers from
disk, and matching against their parents. this second step is also done at the
end of each import of mbox/maildir mailboxes. new deliveries are matched
immediately against other existing messages, currently no attempt is made to
rematch previously delivered messages (which could be useful for related
messages being delivered out of order).

the threading is not yet exposed over imap.

webmail/api.json

@@ -235,6 +235,46 @@
 			],
 			"Returns": []
 		},
+		{
+			"Name": "ThreadCollapse",
+			"Docs": "ThreadCollapse saves the ThreadCollapse field for the messages and its\nchildren. The messageIDs are typically thread roots. But not all roots\n(without parent) of a thread need to have the same collapsed state.",
+			"Params": [
+				{
+					"Name": "messageIDs",
+					"Typewords": [
+						"[]",
+						"int64"
+					]
+				},
+				{
+					"Name": "collapse",
+					"Typewords": [
+						"bool"
+					]
+				}
+			],
+			"Returns": []
+		},
+		{
+			"Name": "ThreadMute",
+			"Docs": "ThreadMute saves the ThreadMute field for the messages and their children.\nIf messages are muted, they are also marked collapsed.",
+			"Params": [
+				{
+					"Name": "messageIDs",
+					"Typewords": [
+						"[]",
+						"int64"
+					]
+				},
+				{
+					"Name": "mute",
+					"Typewords": [
+						"bool"
+					]
+				}
+			],
+			"Returns": []
+		},
 		{
 			"Name": "SSETypes",
 			"Docs": "SSETypes exists to ensure the generated API contains the types, for use in SSE events.",
@@ -288,6 +328,12 @@
 						"ChangeMsgFlags"
 					]
 				},
+				{
+					"Name": "msgThread",
+					"Typewords": [
+						"ChangeMsgThread"
+					]
+				},
 				{
 					"Name": "mailboxRemove",
 					"Typewords": [
@@ -394,6 +440,13 @@
 						"bool"
 					]
 				},
+				{
+					"Name": "Threading",
+					"Docs": "",
+					"Typewords": [
+						"ThreadMode"
+					]
+				},
 				{
 					"Name": "Filter",
 					"Docs": "",
@@ -783,7 +836,7 @@
 				},
 				{
 					"Name": "Subject",
-					"Docs": "",
+					"Docs": "Q/B-word-decoded.",
 					"Typewords": [
 						"string"
 					]
@@ -1319,8 +1372,9 @@
 				},
 				{
 					"Name": "MessageItems",
-					"Docs": "If empty, this was the last message for the request.",
+					"Docs": "If empty, this was the last message for the request. If non-empty, a list of thread messages. Each with the first message being the reason this thread is included and can be used as AnchorID in followup requests. If the threading mode is \"off\" in the query, there will always be only a single message. If a thread is sent, all messages in the thread are sent, including those that don't match the query (e.g. from another mailbox). Threads can be displayed based on the ThreadParentIDs field, with possibly slightly different display based on field ThreadMissingLink.",
 					"Typewords": [
+						"[]",
 						"[]",
 						"MessageItem"
 					]
@@ -1388,6 +1442,13 @@
 					"Typewords": [
 						"string"
 					]
+				},
+				{
+					"Name": "MatchQuery",
+					"Docs": "If message does not match query, it can still be included because of threading.",
+					"Typewords": [
+						"bool"
+					]
 				}
 			]
 		},
@@ -1630,19 +1691,62 @@
 				},
 				{
 					"Name": "MessageID",
-					"Docs": "Value of Message-Id header. Only set for messages that were delivered to the rejects mailbox. For ensuring such messages are delivered only once. Value includes \u003c\u003e.",
+					"Docs": "Canonicalized Message-Id, always lower-case and normalized quoting, without \u003c\u003e's. Empty if missing. Used for matching message threads, and to prevent duplicate reject delivery.",
+					"Typewords": [
+						"string"
+					]
+				},
+				{
+					"Name": "SubjectBase",
+					"Docs": "For matching threads in case there is no References/In-Reply-To header. It is lower-cased, white-space collapsed, mailing list tags and re/fwd tags removed.",
 					"Typewords": [
 						"string"
 					]
 				},
 				{
 					"Name": "MessageHash",
-					"Docs": "Hash of message. For rejects delivery, so optional like MessageID.",
+					"Docs": "Hash of message. For rejects delivery in case there is no Message-ID, only set when delivered as reject.",
 					"Typewords": [
 						"[]",
 						"uint8"
 					]
 				},
+				{
+					"Name": "ThreadID",
+					"Docs": "ID of message starting this thread.",
+					"Typewords": [
+						"int64"
+					]
+				},
+				{
+					"Name": "ThreadParentIDs",
+					"Docs": "IDs of parent messages, from closest parent to the root message. Parent messages may be in a different mailbox, or may no longer exist. ThreadParentIDs must never contain the message id itself (a cycle), and parent messages must reference the same ancestors.",
+					"Typewords": [
+						"[]",
+						"int64"
+					]
+				},
+				{
+					"Name": "ThreadMissingLink",
+					"Docs": "ThreadMissingLink is true if there is no match with a direct parent. E.g. first ID in ThreadParentIDs is not the direct ancestor (an intermediate message may have been deleted), or subject-based matching was done.",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "ThreadMuted",
+					"Docs": "If set, newly delivered child messages are automatically marked as read. This field is copied to new child messages. Changes are propagated to the webmail client.",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "ThreadCollapsed",
+					"Docs": "If set, this (sub)thread is collapsed in the webmail client, for threading mode \"on\" (mode \"unread\" ignores it). This field is copied to new child message. Changes are propagated to the webmail client.",
+					"Typewords": [
+						"bool"
+					]
+				},
 				{
 					"Name": "Seen",
 					"Docs": "",
@@ -1888,7 +1992,7 @@
 		},
 		{
 			"Name": "ChangeMsgAdd",
-			"Docs": "ChangeMsgAdd adds a new message to the view.",
+			"Docs": "ChangeMsgAdd adds a new message and possibly its thread to the view.",
 			"Fields": [
 				{
 					"Name": "MailboxID",
@@ -1927,9 +2031,10 @@
 					]
 				},
 				{
-					"Name": "MessageItem",
+					"Name": "MessageItems",
 					"Docs": "",
 					"Typewords": [
+						"[]",
 						"MessageItem"
 					]
 				}
@@ -2088,6 +2193,34 @@
 				}
 			]
 		},
+		{
+			"Name": "ChangeMsgThread",
+			"Docs": "ChangeMsgThread updates muted/collapsed fields for one message.",
+			"Fields": [
+				{
+					"Name": "MessageIDs",
+					"Docs": "",
+					"Typewords": [
+						"[]",
+						"int64"
+					]
+				},
+				{
+					"Name": "Muted",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				},
+				{
+					"Name": "Collapsed",
+					"Docs": "",
+					"Typewords": [
+						"bool"
+					]
+				}
+			]
+		},
 		{
 			"Name": "ChangeMailboxRemove",
 			"Docs": "ChangeMailboxRemove indicates a mailbox was removed, including all its messages.",
@@ -2382,6 +2515,27 @@
 		}
 	],
 	"Strings": [
+		{
+			"Name": "ThreadMode",
+			"Docs": "",
+			"Values": [
+				{
+					"Name": "ThreadOff",
+					"Value": "off",
+					"Docs": ""
+				},
+				{
+					"Name": "ThreadOn",
+					"Value": "on",
+					"Docs": ""
+				},
+				{
+					"Name": "ThreadUnread",
+					"Value": "unread",
+					"Docs": ""
+				}
+			]
+		},
 		{
 			"Name": "AttachmentType",
 			"Docs": "AttachmentType is for filtering by attachment type.",