infrastructure/mox
1
0
Fork 0
mirror of https://github.com/mjl-/mox.git synced 2025-07-12 17:04:39 +03:00
Code Issues Packages Projects Releases Wiki Activity

add a webapi and webhooks for a simple http/json-based api

Browse Source 
for applications to compose/send messages, receive delivery feedback, and
maintain suppression lists.

this is an alternative to applications using a library to compose messages,
submitting those messages using smtp, and monitoring a mailbox with imap for
DSNs, which can be processed into the equivalent of suppression lists. but you
need to know about all these standards/protocols and find libraries. by using
the webapi & webhooks, you just need a http & json library.

unfortunately, there is no standard for these kinds of api, so mox has made up
yet another one...

matching incoming DSNs about deliveries to original outgoing messages requires
keeping history of "retired" messages (delivered from the queue, either
successfully or failed). this can be enabled per account. history is also
useful for debugging deliveries. we now also keep history of each delivery
attempt, accessible while still in the queue, and kept when a message is
retired. the queue webadmin pages now also have pagination, to show potentially
large history.

a queue of webhook calls is now managed too. failures are retried similar to
message deliveries. webhooks can also be saved to the retired list after
completing. also configurable per account.

messages can be sent with a "unique smtp mail from" address. this can only be
used if the domain is configured with a localpart catchall separator such as
"+". when enabled, a queued message gets assigned a random "fromid", which is
added after the separator when sending. when DSNs are returned, they can be
related to previously sent messages based on this fromid. in the future, we can
implement matching on the "envid" used in the smtp dsn extension, or on the
"message-id" of the message. using a fromid can be triggered by authenticating
with a login email address that is configured as enabling fromid.

suppression lists are automatically managed per account. if a delivery attempt
results in certain smtp errors, the destination address is added to the
suppression list. future messages queued for that recipient will immediately
fail without a delivery attempt. suppression lists protect your mail server
reputation.

submitted messages can carry "extra" data through the queue and webhooks for
outgoing deliveries. through webapi as a json object, through smtp submission
as message headers of the form "x-mox-extra-<key>: value".

to make it easy to test webapi/webhooks locally, the "localserve" mode actually
puts messages in the queue. when it's time to deliver, it still won't do a full
delivery attempt, but just delivers to the sender account. unless the recipient
address has a special form, simulating a failure to deliver.

admins now have more control over the queue. "hold rules" can be added to mark
newly queued messages as "on hold", pausing delivery. rules can be about
certain sender or recipient domains/addresses, or apply to all messages pausing
the entire queue. also useful for (local) testing.

new config options have been introduced. they are editable through the admin
and/or account web interfaces.

the webapi http endpoints are enabled for newly generated configs with the
quickstart, and in localserve. existing configurations must explicitly enable
the webapi in mox.conf.

gopherwatch.org was created to dogfood this code. it initially used just the
compose/smtpclient/imapclient mox packages to send messages and process
delivery feedback. it will get a config option to use the mox webapi/webhooks
instead. the gopherwatch code to use webapi/webhook is smaller and simpler, and
developing that shaped development of the mox webapi/webhooks.

for issue #31 by cuu508
This commit is contained in:
Mechiel Lukkien
2024-04-15 21:49:02 +02:00
parent 8bec5ef7d4
commit 09fcc49223
87 changed files with 15556 additions and 1306 deletions
Download Patch File Download Diff File Expand all files Collapse all files

244
webapi/client.go Normal file
View File

@ -0,0 +1,244 @@
package webapi
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strings"
)
// Client can be used to call webapi methods.
// Client implements [Methods].
type Client struct {
BaseURL string // For example: http://localhost:1080/webapi/v0/.
Username string // Added as HTTP basic authentication if not empty.
Password string
HTTPClient *http.Client // Optional, defaults to http.DefaultClient.
}
var _ Methods = Client{}
func (c Client) httpClient() *http.Client {
if c.HTTPClient != nil {
return c.HTTPClient
}
return http.DefaultClient
}
func transact[T any](ctx context.Context, c Client, fn string, req any) (resp T, rerr error) {
hresp, err := httpDo(ctx, c, fn, req)
if err != nil {
return resp, err
}
defer hresp.Body.Close()
if hresp.StatusCode == http.StatusOK {
// Text and HTML of a message can each be 1MB. Another MB for other data would be a
// lot.
err := json.NewDecoder(&limitReader{hresp.Body, 3 * 1024 * 1024}).Decode(&resp)
return resp, err
}
return resp, badResponse(hresp)
}
func transactReadCloser(ctx context.Context, c Client, fn string, req any) (resp io.ReadCloser, rerr error) {
hresp, err := httpDo(ctx, c, fn, req)
if err != nil {
return nil, err
}
body := hresp.Body
defer func() {
if body != nil {
body.Close()
}
}()
if hresp.StatusCode == http.StatusOK {
r := body
body = nil
return r, nil
}
return nil, badResponse(hresp)
}
func httpDo(ctx context.Context, c Client, fn string, req any) (*http.Response, error) {
reqbuf, err := json.Marshal(req)
if err != nil {
return nil, fmt.Errorf("marshal request: %v", err)
}
data := url.Values{}
data.Add("request", string(reqbuf))
hreq, err := http.NewRequestWithContext(ctx, "POST", c.BaseURL+fn, strings.NewReader(data.Encode()))
if err != nil {
return nil, fmt.Errorf("new request: %v", err)
}
hreq.Header.Set("Content-Type", "application/x-www-form-urlencoded")
if c.Username != "" {
hreq.SetBasicAuth(c.Username, c.Password)
}
hresp, err := c.httpClient().Do(hreq)
if err != nil {
return nil, fmt.Errorf("http transaction: %v", err)
}
return hresp, nil
}
func badResponse(hresp *http.Response) error {
if hresp.StatusCode != http.StatusBadRequest {
return fmt.Errorf("http status %v, expected 200 ok", hresp.Status)
}
buf, err := io.ReadAll(&limitReader{R: hresp.Body, Limit: 10 * 1024})
if err != nil {
return fmt.Errorf("reading error from remote: %v", err)
}
var xerr Error
err = json.Unmarshal(buf, &xerr)
if err != nil {
if len(buf) > 512 {
buf = buf[:512]
}
return fmt.Errorf("error parsing error from remote: %v (first 512 bytes of response: %s)", err, string(buf))
}
return xerr
}
// Send composes a message and submits it to the queue for delivery for all
// recipients (to, cc, bcc).
//
// Configure your account to use unique SMTP MAIL FROM addresses ("fromid") and to
// keep history of retired messages, for better handling of transactional email,
// automatically managing a suppression list.
//
// Configure webhooks to receive updates about deliveries.
//
// If the request is a multipart/form-data, uploaded files with the form keys
// "inlinefile" and/or "attachedfile" will be added to the message. If the uploaded
// file has content-type and/or content-id headers, they will be included. If no
// content-type is present in the request, and it can be detected, it is included
// automatically.
//
// Example call with a text and html message, with an inline and an attached image:
//
// curl --user mox@localhost:moxmoxmox \
// --form request='{"To": [{"Address": "mox@localhost"}], "Text": "hi ☺", "HTML": "<img src=\"cid:hi\" />"}' \
// --form 'inlinefile=@hi.png;headers="Content-ID: <hi>"' \
// --form attachedfile=@mox.png \
// http://localhost:1080/webapi/v0/Send
//
// Error codes:
//
// - badAddress, if an email address is invalid.
// - missingBody, if no text and no html body was specified.
// - multipleFrom, if multiple from addresses were specified.
// - badFrom, if a from address was specified that isn't configured for the account.
// - noRecipients, if no recipients were specified.
// - messageLimitReached, if the outgoing message rate limit was reached.
// - recipientLimitReached, if the outgoing new recipient rate limit was reached.
// - messageTooLarge, message larger than configured maximum size.
// - malformedMessageID, if MessageID is specified but invalid.
// - sentOverQuota, message submitted, but not stored in Sent mailbox due to quota reached.
func (c Client) Send(ctx context.Context, req SendRequest) (resp SendResult, err error) {
return transact[SendResult](ctx, c, "Send", req)
}
// SuppressionList returns the addresses on the per-account suppression list.
func (c Client) SuppressionList(ctx context.Context, req SuppressionListRequest) (resp SuppressionListResult, err error) {
return transact[SuppressionListResult](ctx, c, "SuppressionList", req)
}
// SuppressionAdd adds an address to the suppression list of the account.
//
// Error codes:
//
// - badAddress, if the email address is invalid.
func (c Client) SuppressionAdd(ctx context.Context, req SuppressionAddRequest) (resp SuppressionAddResult, err error) {
return transact[SuppressionAddResult](ctx, c, "SuppressionAdd", req)
}
// SuppressionRemove removes an address from the suppression list of the account.
//
// Error codes:
//
// - badAddress, if the email address is invalid.
func (c Client) SuppressionRemove(ctx context.Context, req SuppressionRemoveRequest) (resp SuppressionRemoveResult, err error) {
return transact[SuppressionRemoveResult](ctx, c, "SuppressionRemove", req)
}
// SuppressionPresent returns whether an address is present in the suppression list of the account.
//
// Error codes:
//
// - badAddress, if the email address is invalid.
func (c Client) SuppressionPresent(ctx context.Context, req SuppressionPresentRequest) (resp SuppressionPresentResult, err error) {
return transact[SuppressionPresentResult](ctx, c, "SuppressionPresent", req)
}
// MessageGet returns a message from the account storage in parsed form.
//
// Use [Client.MessageRawGet] for the raw message (internet message file).
//
// Error codes:
// - messageNotFound, if the message does not exist.
func (c Client) MessageGet(ctx context.Context, req MessageGetRequest) (resp MessageGetResult, err error) {
return transact[MessageGetResult](ctx, c, "MessageGet", req)
}
// MessageRawGet returns the full message in its original form, as stored on disk.
//
// Error codes:
// - messageNotFound, if the message does not exist.
func (c Client) MessageRawGet(ctx context.Context, req MessageRawGetRequest) (resp io.ReadCloser, err error) {
return transactReadCloser(ctx, c, "MessageRawGet", req)
}
// MessagePartGet returns a single part from a multipart message, by a "parts
// path", a series of indices into the multipart hierarchy as seen in the parsed
// message. The initial selection is the body of the outer message (excluding
// headers).
//
// Error codes:
// - messageNotFound, if the message does not exist.
// - partNotFound, if the part does not exist.
func (c Client) MessagePartGet(ctx context.Context, req MessagePartGetRequest) (resp io.ReadCloser, err error) {
return transactReadCloser(ctx, c, "MessagePartGet", req)
}
// MessageDelete permanently removes a message from the account storage (not moving
// to a Trash folder).
//
// Error codes:
// - messageNotFound, if the message does not exist.
func (c Client) MessageDelete(ctx context.Context, req MessageDeleteRequest) (resp MessageDeleteResult, err error) {
return transact[MessageDeleteResult](ctx, c, "MessageDelete", req)
}
// MessageFlagsAdd adds (sets) flags on a message, like the well-known flags
// beginning with a backslash like \seen, \answered, \draft, or well-known flags
// beginning with a dollar like $junk, $notjunk, $forwarded, or custom flags.
// Existing flags are left unchanged.
//
// Error codes:
// - messageNotFound, if the message does not exist.
func (c Client) MessageFlagsAdd(ctx context.Context, req MessageFlagsAddRequest) (resp MessageFlagsAddResult, err error) {
return transact[MessageFlagsAddResult](ctx, c, "MessageFlagsAdd", req)
}
// MessageFlagsRemove removes (clears) flags on a message.
// Other flags are left unchanged.
//
// Error codes:
// - messageNotFound, if the message does not exist.
func (c Client) MessageFlagsRemove(ctx context.Context, req MessageFlagsRemoveRequest) (resp MessageFlagsRemoveResult, err error) {
return transact[MessageFlagsRemoveResult](ctx, c, "MessageFlagsRemove", req)
}
// MessageMove moves a message to a new mailbox name (folder). The destination
// mailbox name must already exist.
//
// Error codes:
// - messageNotFound, if the message does not exist.
func (c Client) MessageMove(ctx context.Context, req MessageMoveRequest) (resp MessageMoveResult, err error) {
return transact[MessageMoveResult](ctx, c, "MessageMove", req)
}

367
webapi/doc.go Normal file
View File

@ -0,0 +1,367 @@
// NOTE: DO NOT EDIT, this file is generated by gendoc.sh.
/*
Package webapi implements a simple HTTP/JSON-based API for interacting with
email, and webhooks for notifications about incoming and outgoing deliveries,
including delivery failures.
# Overview
The webapi can be used to compose and send outgoing messages. The HTTP/JSON
API is often easier to use for developers since it doesn't require separate
libraries and/or having (detailed) knowledge about the format of email messages
("Internet Message Format"), or the SMTP protocol and its extensions.
Webhooks can be configured per account, and help with automated processing of
incoming email, and with handling delivery failures/success. Webhooks are
often easier to use for developers than monitoring a mailbox with IMAP and
processing new incoming email and delivery status notification (DSN) messages.
# Webapi
The webapi has a base URL at /webapi/v0/ by default, but configurable, which
serves an introduction that points to this documentation and lists the API
methods available.
An HTTP POST to /webapi/v0/<method> calls a method.The form can be either
"application/x-www-form-urlencoded" or "multipart/form-data". Form field
"request" must contain the request parameters, encoded as JSON.
HTTP basic authentication is required for calling methods, with an email
address as user name. Use a login address configured for "unique SMTP MAIL
FROM" addresses, and configure a period to "keep retired messages delivered
from the queue" for automatic suppression list management.
HTTP response status 200 OK indicates a successful method call, status 400
indicates an error. The response body of an error is a JSON object with a
human-readable "Message" field, and a "Code" field for programmatic handling
(common codes: "user" or user-induced errors, "server" for server-caused
errors). Most successful calls return a JSON object, but some return data
(e.g. a raw message or an attachment of a message). See [Methods] for the
methods and and [Client] for their documentation. The first element of their
return values indicate their JSON object type or io.ReadCloser for non-JSON
data. The request and response types are converted from/to JSON. optional and
missing/empty fields/values are converted into Go zero values: zero for
numbers, empty strings, empty lists and empty objects. New fields may be added
in response objects in future versions, parsers should ignore unrecognized
fields.
An HTTP GET to a method URL serves an HTML page showing example
request/response JSON objects in a form and a button to call the method.
# Webhooks
Webhooks for outgoing delivery events and incoming deliveries are configured
per account.
A webhook is delivered by an HTTP POST with headers "X-Mox-Webhook-ID" (unique
ID of webhook) and "X-Mox-Webhook-Attempt" (number of delivery attempts,
starting at 1), and a JSON body with the webhook data. Webhook delivery
failures are retried at a schedule similar to message deliveries, until
permanent failure.
See [webhook.Outgoing] for the fields in a webhook for outgoing deliveries, and
in particular [webhook.OutgoingEvent] for the types of events.
Only the latest event for the delivery of a particular outgoing message will be
delivered, any webhooks for that message still in the queue (after failure to
deliver) are retired as superseded when a new event occurs.
Webhooks for incoming deliveries are configured separately from outgoing
deliveries. Incoming DSNs for previously sent messages do not cause a webhook
to the webhook URL for incoming messages, only to the webhook URL for outgoing
delivery events. The incoming webhook JSON payload contains the message
envelope (parsed To, Cc, Bcc, Subject and more headers), the MIME structure,
and the contents of the first text and HTML parts. See [webhook.Incoming] for
the fields in the JSON object. The full message and individual parts, including
attachments, can be retrieved using the webapi.
# Transactional email
When sending transactional emails, potentially to many recipients, it is
important to process delivery failure notifications. If messages are rejected,
or email addresses no longer exist, you should stop sending email to those
addresses. If you try to keep sending, the receiving mail servers may consider
that spammy behaviour and blocklist your mail server.
Automatic suppression list management already prevents most repeated sending
attempts. The webhooks make it easy to receive failure notifications.
To keep spam complaints about your messages a minimum, include links to
unsubscribe from future messages without requiring further actions from the
user, such as logins. Include an unsubscribe link in the footer, and include
List-* message headers, such as List-Id, List-Unsubscribe and
List-Unsubscribe-Post.
# Webapi examples
Below are examples for making webapi calls to a locally running "mox
localserve" with its default credentials.
Send a basic message:
$ curl --user mox@localhost:moxmoxmox \
--data request='{"To": [{"Address": "mox@localhost"}], "Text": "hi ☺"}' \
http://localhost:1080/webapi/v0/Send
{
"MessageID": "<kVTha0Q-a5Zh1MuTh5rUjg@localhost>",
"Submissions": [
{
"Address": "mox@localhost",
"QueueMsgID": 10010,
"FromID": "ZfV16EATHwKEufrSMo055Q"
}
]
}
Send a message with files both from form upload and base64 included in JSON:
$ curl --user mox@localhost:moxmoxmox \
--form request='{"To": [{"Address": "mox@localhost"}], "Subject": "hello", "Text": "hi ☺", "HTML": "<img src=\"cid:hi\" />", "AttachedFiles": [{"Name": "img.png", "ContentType": "image/png", "Data": "bWFkZSB5b3UgbG9vayE="}]}' \
--form 'inlinefile=@hi.png;headers="Content-ID: <hi>"' \
--form attachedfile=@mox.png \
http://localhost:1080/webapi/v0/Send
{
"MessageID": "<eZ3OEEA2odXovovIxHE49g@localhost>",
"Submissions": [
{
"Address": "mox@localhost",
"QueueMsgID": 10011,
"FromID": "yWiUQ6mvJND8FRPSmc9y5A"
}
]
}
Get a message in parsed form:
$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 424}' http://localhost:1080/webapi/v0/MessageGet
{
"Message": {
"From": [
{
"Name": "mox",
"Address": "mox@localhost"
}
],
"To": [
{
"Name": "",
"Address": "mox@localhost"
}
],
"CC": [],
"BCC": [],
"ReplyTo": [],
"MessageID": "<84vCeme_yZXyDzjWDeYBpg@localhost>",
"References": [],
"Date": "2024-04-04T14:29:42+02:00",
"Subject": "hello",
"Text": "hi \u263a\n",
"HTML": ""
},
"Structure": {
"ContentType": "multipart/mixed",
"ContentTypeParams": {
"boundary": "0ee72dc30dbab2ca6f7a363844a10a9f6111fc6dd31b8ff0b261478c2c48"
},
"ContentID": "",
"DecodedSize": 0,
"Parts": [
{
"ContentType": "multipart/related",
"ContentTypeParams": {
"boundary": "b5ed0977ee2b628040f394c3f374012458379a4f3fcda5036371d761c81d"
},
"ContentID": "",
"DecodedSize": 0,
"Parts": [
{
"ContentType": "multipart/alternative",
"ContentTypeParams": {
"boundary": "3759771adede7bd191ef37f2aa0e49ff67369f4000c320f198a875e96487"
},
"ContentID": "",
"DecodedSize": 0,
"Parts": [
{
"ContentType": "text/plain",
"ContentTypeParams": {
"charset": "utf-8"
},
"ContentID": "",
"DecodedSize": 8,
"Parts": []
},
{
"ContentType": "text/html",
"ContentTypeParams": {
"charset": "us-ascii"
},
"ContentID": "",
"DecodedSize": 22,
"Parts": []
}
]
},
{
"ContentType": "image/png",
"ContentTypeParams": {},
"ContentID": "<hi>",
"DecodedSize": 19375,
"Parts": []
}
]
},
{
"ContentType": "image/png",
"ContentTypeParams": {},
"ContentID": "",
"DecodedSize": 14,
"Parts": []
},
{
"ContentType": "image/png",
"ContentTypeParams": {},
"ContentID": "",
"DecodedSize": 7766,
"Parts": []
}
]
},
"Meta": {
"Size": 38946,
"DSN": false,
"Flags": [
"$notjunk",
"\seen"
],
"MailFrom": "",
"MailFromValidated": false,
"MsgFrom": "",
"MsgFromValidated": false,
"DKIMVerifiedDomains": [],
"RemoteIP": "",
"MailboxName": "Inbox"
}
}
Errors (with a 400 bad request HTTP status response) include a human-readable
message and a code for programmatic use:
$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 999}' http://localhost:1080/webapi/v0/MessageGet
{
"Code": "notFound",
"Message": "message not found"
}
Get a raw, unparsed message, as bytes:
$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 123}' http://localhost:1080/webapi/v0/MessageRawGet
[message as bytes in raw form]
Mark a message as read:
$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 424, "Flags": ["\\Seen", "custom"]}' http://localhost:1080/webapi/v0/MessageFlagsAdd
{}
# Webhook examples
A webhook is delivered by an HTTP POST, wich headers X-Mox-Webhook-ID and
X-Mox-Webhook-Attempt and a JSON body with the data. To simulate a webhook call
for incoming messages, use:
curl -H 'X-Mox-Webhook-ID: 123' -H 'X-Mox-Webhook-Attempt: 1' --json '{...}' http://localhost/yourapp
Example webhook HTTP POST JSON body for successful outgoing delivery:
{
"Version": 0,
"Event": "delivered",
"DSN": false,
"Suppressing": false,
"QueueMsgID": 101,
"FromID": "MDEyMzQ1Njc4OWFiY2RlZg",
"MessageID": "<QnxzgulZK51utga6agH_rg@mox.example>",
"Subject": "subject of original message",
"WebhookQueued": "2024-03-27T00:00:00Z",
"SMTPCode": 250,
"SMTPEnhancedCode": "",
"Error": "",
"Extra": {}
}
Example webhook HTTP POST JSON body for failed delivery based on incoming DSN
message, with custom extra data fields (from original submission), and adding address to the suppression list:
{
"Version": 0,
"Event": "failed",
"DSN": true,
"Suppressing": true,
"QueueMsgID": 102,
"FromID": "MDEyMzQ1Njc4OWFiY2RlZg",
"MessageID": "<QnxzgulZK51utga6agH_rg@mox.example>",
"Subject": "subject of original message",
"WebhookQueued": "2024-03-27T00:00:00Z",
"SMTPCode": 554,
"SMTPEnhancedCode": "5.4.0",
"Error": "timeout connecting to host",
"Extra": {
"userid": "456"
}
}
Example JSON body for webhooks for incoming delivery of basic message:
{
"Version": 0,
"From": [
{
"Name": "",
"Address": "mox@localhost"
}
],
"To": [
{
"Name": "",
"Address": "mjl@localhost"
}
],
"CC": [],
"BCC": [],
"ReplyTo": [],
"Subject": "hi",
"MessageID": "<QnxzgulZK51utga6agH_rg@mox.example>",
"InReplyTo": "",
"References": [],
"Date": "2024-03-27T00:00:00Z",
"Text": "hello world ☺\n",
"HTML": "",
"Structure": {
"ContentType": "text/plain",
"ContentTypeParams": {
"charset": "utf-8"
},
"ContentID": "",
"DecodedSize": 17,
"Parts": []
},
"Meta": {
"MsgID": 201,
"MailFrom": "mox@localhost",
"MailFromValidated": false,
"MsgFromValidated": true,
"RcptTo": "mjl@localhost",
"DKIMVerifiedDomains": [
"localhost"
],
"RemoteIP": "127.0.0.1",
"Received": "2024-03-27T00:00:03Z",
"MailboxName": "Inbox",
"Automated": false
}
}
*/
package webapi
// NOTE: DO NOT EDIT, this file is generated by gendoc.sh.

297
webapi/gendoc.sh Executable file
View File

@ -0,0 +1,297 @@
#!/bin/bash
set -euo pipefail
# this is run with .. as working directory.
# note: outgoing hook events are in ../queue/hooks.go, ../mox-/config.go, ../queue.go and ../webapi/gendoc.sh. keep in sync.
# todo: find some proper way to generate the curl commands and responses automatically...
cat <<EOF
// NOTE: DO NOT EDIT, this file is generated by gendoc.sh.
/*
Package webapi implements a simple HTTP/JSON-based API for interacting with
email, and webhooks for notifications about incoming and outgoing deliveries,
including delivery failures.
# Overview
The webapi can be used to compose and send outgoing messages. The HTTP/JSON
API is often easier to use for developers since it doesn't require separate
libraries and/or having (detailed) knowledge about the format of email messages
("Internet Message Format"), or the SMTP protocol and its extensions.
Webhooks can be configured per account, and help with automated processing of
incoming email, and with handling delivery failures/success. Webhooks are
often easier to use for developers than monitoring a mailbox with IMAP and
processing new incoming email and delivery status notification (DSN) messages.
# Webapi
The webapi has a base URL at /webapi/v0/ by default, but configurable, which
serves an introduction that points to this documentation and lists the API
methods available.
An HTTP POST to /webapi/v0/<method> calls a method.The form can be either
"application/x-www-form-urlencoded" or "multipart/form-data". Form field
"request" must contain the request parameters, encoded as JSON.
HTTP basic authentication is required for calling methods, with an email
address as user name. Use a login address configured for "unique SMTP MAIL
FROM" addresses, and configure a period to "keep retired messages delivered
from the queue" for automatic suppression list management.
HTTP response status 200 OK indicates a successful method call, status 400
indicates an error. The response body of an error is a JSON object with a
human-readable "Message" field, and a "Code" field for programmatic handling
(common codes: "user" or user-induced errors, "server" for server-caused
errors). Most successful calls return a JSON object, but some return data
(e.g. a raw message or an attachment of a message). See [Methods] for the
methods and and [Client] for their documentation. The first element of their
return values indicate their JSON object type or io.ReadCloser for non-JSON
data. The request and response types are converted from/to JSON. optional and
missing/empty fields/values are converted into Go zero values: zero for
numbers, empty strings, empty lists and empty objects. New fields may be added
in response objects in future versions, parsers should ignore unrecognized
fields.
An HTTP GET to a method URL serves an HTML page showing example
request/response JSON objects in a form and a button to call the method.
# Webhooks
Webhooks for outgoing delivery events and incoming deliveries are configured
per account.
A webhook is delivered by an HTTP POST with headers "X-Mox-Webhook-ID" (unique
ID of webhook) and "X-Mox-Webhook-Attempt" (number of delivery attempts,
starting at 1), and a JSON body with the webhook data. Webhook delivery
failures are retried at a schedule similar to message deliveries, until
permanent failure.
See [webhook.Outgoing] for the fields in a webhook for outgoing deliveries, and
in particular [webhook.OutgoingEvent] for the types of events.
Only the latest event for the delivery of a particular outgoing message will be
delivered, any webhooks for that message still in the queue (after failure to
deliver) are retired as superseded when a new event occurs.
Webhooks for incoming deliveries are configured separately from outgoing
deliveries. Incoming DSNs for previously sent messages do not cause a webhook
to the webhook URL for incoming messages, only to the webhook URL for outgoing
delivery events. The incoming webhook JSON payload contains the message
envelope (parsed To, Cc, Bcc, Subject and more headers), the MIME structure,
and the contents of the first text and HTML parts. See [webhook.Incoming] for
the fields in the JSON object. The full message and individual parts, including
attachments, can be retrieved using the webapi.
# Transactional email
When sending transactional emails, potentially to many recipients, it is
important to process delivery failure notifications. If messages are rejected,
or email addresses no longer exist, you should stop sending email to those
addresses. If you try to keep sending, the receiving mail servers may consider
that spammy behaviour and blocklist your mail server.
Automatic suppression list management already prevents most repeated sending
attempts. The webhooks make it easy to receive failure notifications.
To keep spam complaints about your messages a minimum, include links to
unsubscribe from future messages without requiring further actions from the
user, such as logins. Include an unsubscribe link in the footer, and include
List-* message headers, such as List-Id, List-Unsubscribe and
List-Unsubscribe-Post.
# Webapi examples
Below are examples for making webapi calls to a locally running "mox
localserve" with its default credentials.
Send a basic message:
\$ curl --user mox@localhost:moxmoxmox \\
--data request='{"To": [{"Address": "mox@localhost"}], "Text": "hi ☺"}' \\
http://localhost:1080/webapi/v0/Send
{
"MessageID": "<kVTha0Q-a5Zh1MuTh5rUjg@localhost>",
"Submissions": [
{
"Address": "mox@localhost",
"QueueMsgID": 10010,
"FromID": "ZfV16EATHwKEufrSMo055Q"
}
]
}
Send a message with files both from form upload and base64 included in JSON:
\$ curl --user mox@localhost:moxmoxmox \\
--form request='{"To": [{"Address": "mox@localhost"}], "Subject": "hello", "Text": "hi ☺", "HTML": "<img src=\"cid:hi\" />", "AttachedFiles": [{"Name": "img.png", "ContentType": "image/png", "Data": "bWFkZSB5b3UgbG9vayE="}]}' \\
--form 'inlinefile=@hi.png;headers="Content-ID: <hi>"' \\
--form attachedfile=@mox.png \\
http://localhost:1080/webapi/v0/Send
{
"MessageID": "<eZ3OEEA2odXovovIxHE49g@localhost>",
"Submissions": [
{
"Address": "mox@localhost",
"QueueMsgID": 10011,
"FromID": "yWiUQ6mvJND8FRPSmc9y5A"
}
]
}
Get a message in parsed form:
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 424}' http://localhost:1080/webapi/v0/MessageGet
{
"Message": {
"From": [
{
"Name": "mox",
"Address": "mox@localhost"
}
],
"To": [
{
"Name": "",
"Address": "mox@localhost"
}
],
"CC": [],
"BCC": [],
"ReplyTo": [],
"MessageID": "<84vCeme_yZXyDzjWDeYBpg@localhost>",
"References": [],
"Date": "2024-04-04T14:29:42+02:00",
"Subject": "hello",
"Text": "hi \u263a\n",
"HTML": ""
},
"Structure": {
"ContentType": "multipart/mixed",
"ContentTypeParams": {
"boundary": "0ee72dc30dbab2ca6f7a363844a10a9f6111fc6dd31b8ff0b261478c2c48"
},
"ContentID": "",
"DecodedSize": 0,
"Parts": [
{
"ContentType": "multipart/related",
"ContentTypeParams": {
"boundary": "b5ed0977ee2b628040f394c3f374012458379a4f3fcda5036371d761c81d"
},
"ContentID": "",
"DecodedSize": 0,
"Parts": [
{
"ContentType": "multipart/alternative",
"ContentTypeParams": {
"boundary": "3759771adede7bd191ef37f2aa0e49ff67369f4000c320f198a875e96487"
},
"ContentID": "",
"DecodedSize": 0,
"Parts": [
{
"ContentType": "text/plain",
"ContentTypeParams": {
"charset": "utf-8"
},
"ContentID": "",
"DecodedSize": 8,
"Parts": []
},
{
"ContentType": "text/html",
"ContentTypeParams": {
"charset": "us-ascii"
},
"ContentID": "",
"DecodedSize": 22,
"Parts": []
}
]
},
{
"ContentType": "image/png",
"ContentTypeParams": {},
"ContentID": "<hi>",
"DecodedSize": 19375,
"Parts": []
}
]
},
{
"ContentType": "image/png",
"ContentTypeParams": {},
"ContentID": "",
"DecodedSize": 14,
"Parts": []
},
{
"ContentType": "image/png",
"ContentTypeParams": {},
"ContentID": "",
"DecodedSize": 7766,
"Parts": []
}
]
},
"Meta": {
"Size": 38946,
"DSN": false,
"Flags": [
"\$notjunk",
"\\seen"
],
"MailFrom": "",
"MailFromValidated": false,
"MsgFrom": "",
"MsgFromValidated": false,
"DKIMVerifiedDomains": [],
"RemoteIP": "",
"MailboxName": "Inbox"
}
}
Errors (with a 400 bad request HTTP status response) include a human-readable
message and a code for programmatic use:
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 999}' http://localhost:1080/webapi/v0/MessageGet
{
"Code": "notFound",
"Message": "message not found"
}
Get a raw, unparsed message, as bytes:
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 123}' http://localhost:1080/webapi/v0/MessageRawGet
[message as bytes in raw form]
Mark a message as read:
\$ curl --user mox@localhost:moxmoxmox --data request='{"MsgID": 424, "Flags": ["\\\\Seen", "custom"]}' http://localhost:1080/webapi/v0/MessageFlagsAdd
{}
# Webhook examples
A webhook is delivered by an HTTP POST, wich headers X-Mox-Webhook-ID and
X-Mox-Webhook-Attempt and a JSON body with the data. To simulate a webhook call
for incoming messages, use:
curl -H 'X-Mox-Webhook-ID: 123' -H 'X-Mox-Webhook-Attempt: 1' --json '{...}' http://localhost/yourapp
EOF
for ex in $(./mox example | grep webhook); do
./mox example $ex
echo
done
cat <<EOF
*/
package webapi
// NOTE: DO NOT EDIT, this file is generated by gendoc.sh.
EOF

29
webapi/limitreader.go Normal file
View File

@ -0,0 +1,29 @@
package webapi
// similar between ../moxio/limitreader.go and ../webapi/limitreader.go
import (
"errors"
"io"
)
var errLimit = errors.New("input exceeds maximum size") // Returned by limitReader.
// limitReader reads up to Limit bytes, returning an error if more bytes are
// read. LimitReader can be used to enforce a maximum input length.
type limitReader struct {
R io.Reader
Limit int64
}
// Read reads bytes from the underlying reader.
func (r *limitReader) Read(buf []byte) (int, error) {
n, err := r.R.Read(buf)
if n > 0 {
r.Limit -= int64(n)
if r.Limit < 0 {
return 0, errLimit
}
}
return n, err
}

260
webapi/webapi.go Normal file
View File

@ -0,0 +1,260 @@
package webapi
import (
"context"
"io"
"time"
"github.com/mjl-/mox/webhook"
)
// todo future: we can have text and html templates, let submitters reference them along with parameters, and compose the message bodies ourselves.
// todo future: generate api specs (e.g. openapi) for webapi
// todo future: consider deprecating some of the webapi in favor of jmap
// Methods of the webapi. More methods may be added in the future. See [Client]
// for documentation.
type Methods interface {
Send(ctx context.Context, request SendRequest) (response SendResult, err error)
SuppressionList(ctx context.Context, request SuppressionListRequest) (response SuppressionListResult, err error)
SuppressionAdd(ctx context.Context, request SuppressionAddRequest) (response SuppressionAddResult, err error)
SuppressionRemove(ctx context.Context, request SuppressionRemoveRequest) (response SuppressionRemoveResult, err error)
SuppressionPresent(ctx context.Context, request SuppressionPresentRequest) (response SuppressionPresentResult, err error)
MessageGet(ctx context.Context, request MessageGetRequest) (response MessageGetResult, err error)
MessageRawGet(ctx context.Context, request MessageRawGetRequest) (response io.ReadCloser, err error)
MessagePartGet(ctx context.Context, request MessagePartGetRequest) (response io.ReadCloser, err error)
MessageDelete(ctx context.Context, request MessageDeleteRequest) (response MessageDeleteResult, err error)
MessageFlagsAdd(ctx context.Context, request MessageFlagsAddRequest) (response MessageFlagsAddResult, err error)
MessageFlagsRemove(ctx context.Context, request MessageFlagsRemoveRequest) (response MessageFlagsRemoveResult, err error)
MessageMove(ctx context.Context, request MessageMoveRequest) (response MessageMoveResult, err error)
}
// Error indicates an API-related error.
type Error struct {
// For programmatic handling. Common values: "user" for generic error by user,
// "server" for a server-side processing error, "badAddress" for malformed email
// addresses.
Code string
// Human readable error message.
Message string
}
// Error returns the human-readable error message.
func (e Error) Error() string {
return e.Message
}
type NameAddress struct {
Name string // Optional, human-readable "display name" of the addressee.
Address string // Required, email address.
}
// Message is an email message, used both for outgoing submitted messages and
// incoming messages.
type Message struct {
// For sending, if empty, automatically filled based on authenticated user and
// account information. Outgoing messages are allowed maximum 1 From address,
// incoming messages can in theory have zero or multiple, but typically have just
// one.
From []NameAddress
// To/Cc/Bcc message headers. Outgoing messages are sent to all these addresses.
// All are optional, but there should be at least one addressee.
To []NameAddress
CC []NameAddress
// For submissions, BCC addressees receive the message but are not added to the
// message headers. For incoming messages, this is typically empty.
BCC []NameAddress
// Optional Reply-To header, where the recipient is asked to send replies to.
ReplyTo []NameAddress
// Message-ID from message header, should be wrapped in <>'s. For outgoing
// messages, a unique message-id is generated if empty.
MessageID string
// Optional. References to message-id's (including <>) of other messages, if this
// is a reply or forwarded message. References are from oldest (ancestor) to most
// recent message. For outgoing messages, if non-empty then In-Reply-To is set to
// the last element.
References []string
// Optional, set to time of submission for outgoing messages if nil.
Date *time.Time
// Subject header, optional.
Subject string
// For outgoing messages, at least text or HTML must be non-empty. If both are
// present, a multipart/alternative part is created. Lines must be
// \n-separated, automatically replaced with \r\n when composing the message.
// For parsed, incoming messages, values are truncated to 1MB (1024*1024 bytes).
// Use MessagePartGet to retrieve the full part data.
Text string
HTML string
}
// SendRequest submits a message to be delivered.
type SendRequest struct {
// Message with headers and contents to compose. Additional headers and files can
// be added too (see below, and the use of multipart/form-data requests). The
// fields of Message are included directly in SendRequest. Required.
Message
// Metadata to associate with the delivery, through the queue, including webhooks
// about delivery events. Metadata can also be set with regular SMTP submission
// through message headers "X-Mox-Extra-<key>: <value>". Current behaviour is as
// follows, but this may change: 1. Keys are canonicalized, each dash-separated
// word changed to start with a capital. 2. Keys cannot be duplicated. 3. These
// headers are not removed when delivering.
Extra map[string]string
// Additional custom headers to include in outgoing message. Optional.
// Unless a User-Agent or X-Mailer header is present, a User-Agent is added.
Headers [][2]string
// Inline files are added to the message and should be displayed by mail clients as
// part of the message contents. Inline files cause a part with content-type
// "multipart/related" to be added to the message. Optional.
InlineFiles []File
// Attached files are added to the message and should be shown as files that can be
// saved. Attached files cause a part with content-type "multipart/mixed" to be
// added to the message. Optional.
AttachedFiles []File
// If absent/null, regular TLS requirements apply (opportunistic TLS, DANE,
// MTA-STS). If true, the SMTP REQUIRETLS extension is required, enforcing verified
// TLS along the delivery path. If false, TLS requirements are relaxed and
// DANE/MTA-STS policies may be ignored to increase the odds of successful but
// insecure delivery. Optional.
RequireTLS *bool
// If set, it should be a time in the future at which the first delivery attempt
// starts. Optional.
FutureRelease *time.Time
// Whether to store outgoing message in designated Sent mailbox (if configured).
SaveSent bool
}
type File struct {
Name string // Optional.
ContentType string // E.g. application/pdf or image/png, automatically detected if empty.
ContentID string // E.g. "<randomid>", for use in html email with "cid:<randomid>". Optional.
Data string // Base64-encoded contents of the file. Required.
}
// MessageMeta is returned as part of MessageGet.
type MessageMeta struct {
Size int64 // Total size of raw message file.
DSN bool // Whether this message is a DSN.
Flags []string // Standard message flags like \seen, \answered, $forwarded, $junk, $nonjunk, and custom keywords.
MailFrom string // Address used during SMTP "MAIL FROM" command.
MailFromValidated bool // Whether SMTP MAIL FROM address was SPF-validated.
MsgFrom string // Address used in message "From" header.
MsgFromValidated bool // Whether address in message "From"-header was DMARC(-like) validated.
DKIMVerifiedDomains []string // Verified domains from DKIM-signature in message. Can be different domain than used in addresses.
RemoteIP string // Where the message was delivered from.
MailboxName string
}
type SendResult struct {
MessageID string // "<random>@<domain>", as added by submitter or automatically generated during submission.
Submissions []Submission // Messages submitted to queue for delivery. In order of To, CC, BCC fields in request.
}
type Submission struct {
Address string // From original recipient (to/cc/bcc).
QueueMsgID int64 // Of message added to delivery queue, later webhook calls reference this same ID.
FromID string // Unique ID used during delivery, later webhook calls reference this same FromID.
}
// Suppression is an address to which messages will not be delivered. Attempts to
// deliver or queue will result in an immediate permanent failure to deliver.
type Suppression struct {
ID int64
Created time.Time `bstore:"default now"`
// Suppression applies to this account only.
Account string `bstore:"nonzero,unique Account+BaseAddress"`
// Unicode. Address with fictional simplified localpart: lowercase, dots removed
// (gmail), first token before any "-" or "+" (typical catchall separator).
BaseAddress string `bstore:"nonzero"`
// Unicode. Address that caused this suppression.
OriginalAddress string `bstore:"nonzero"`
Manual bool
Reason string
}
type SuppressionListRequest struct{}
type SuppressionListResult struct {
Suppressions []Suppression // Current suppressed addresses for account.
}
type SuppressionAddRequest struct {
EmailAddress string
Manual bool // Whether added manually or automatically.
Reason string // Free-form text.
}
type SuppressionAddResult struct{}
type SuppressionRemoveRequest struct {
EmailAddress string
}
type SuppressionRemoveResult struct{}
type SuppressionPresentRequest struct {
EmailAddress string
}
type SuppressionPresentResult struct {
Present bool
}
type MessageGetRequest struct {
MsgID int64
}
type MessageGetResult struct {
Message Message
Structure webhook.Structure // MIME structure.
Meta MessageMeta // Additional information about message and SMTP delivery.
}
type MessageRawGetRequest struct {
MsgID int64
}
type MessagePartGetRequest struct {
MsgID int64
// Indexes into MIME parts, e.g. [0, 2] first dereferences the first element in a
// multipart message, then the 3rd part within that first element.
PartPath []int
}
type MessageDeleteRequest struct {
MsgID int64
}
type MessageDeleteResult struct{}
type MessageFlagsAddRequest struct {
MsgID int64
Flags []string // Standard message flags like \seen, \answered, $forwarded, $junk, $nonjunk, and custom keywords.
}
type MessageFlagsAddResult struct{}
type MessageFlagsRemoveRequest struct {
MsgID int64
Flags []string
}
type MessageFlagsRemoveResult struct{}
type MessageMoveRequest struct {
MsgID int64
DestMailboxName string // E.g. "Inbox", must already exist.
}
type MessageMoveResult struct{}