Extensible version of Mutt
=-=-=-=-=-=-=-=-=-=-=-=-=-

This file describes an experimental integration of extension languages into
Mutt, why one might need that, and what have been the basic design ideas
behind that.

Currently, the only supported extension language is Scheme (Guile).
More details about the Guile side of Mutt may be found below, in part 2.

Mutt may still be compiled without extension language support (at
least theoritically), and even when compiled with extension language
support it remains 100% compatible with the regular Mutt.

This work is based on Mutt 1.5.4i.


1. Design approach
------------------

The idea was to allow Mutt to be extensible with _any_ extension language in a
way that makes it possible to use and customize internal mechanisms, without
being incompatible with the original Mutt.  However, we don't want Mutt to
become a heavy-weight program like GNU Emacs -- that's why we are not using
Gnus. :) 


1.1. Why adding an extension language?

Mutt already has some nice features that may be used to customize its
behavior, namely "hooks".  However, there are mainly three limitations to this
approach:

* hooks are pre-defined and even though there are already several of them in
  well-chosen places, one might some day want to add an hook to another event
  than those predefined;

* hooks are basic actions that can be executed either before or after a given
  operation and may not be used to change the behaviour of a given action;

* hooks can only execute what Mutt calls `commands' (e.g. "set from=ludo@net",
  "exec tag-pattern", "exec delete-message"); however functions that can be
  used are, again, pre-defined, and most of them are interactive which makes
  it impossible to automate some particular operation (for instance,
  tag-pattern would present a prompt and wait till the user entered a pattern;
  delete-message would not only mark a message as deleted but also jump to the
  next non-deleted message).

These limitations sometimes make the use of hooks quite frustrating.

Without the use of an extension language, it is almost impossible for the user
to:

  * customize a given feature (e.g. I don't like the default scoring method
    which rounds up negative scores to zero and I would like to have my own;
    I would like Mutt to warn me before displaying a message whose header
    contains 'X-Spam-Status: Yes'; etc.);

  * implement new features, new commands, which may be configured using new
    configuration variables (e.g. if `spam_learn' is "yes", then I would like
    Mutt to pipe messages through SpamAssassin once they have been displayed
    and recognized as spam/ham).

Using a simple interpreted language such as Scheme makes it possible to
quickly to this.


1.2. What should be extensible?  How should it be extended?

Ideally, any built-in feature, function or mechanism should be make
extensible via the extension language.  It should also be possible to
implement and use new functions programmed with the extension language.
Practically, this means that it should be possible to customize the way
messages are scored or the way messages are displayed.  This also means that
it should be possible to implement a function that counts the number of
messages coming from your best friend and call it through the standard Mutt
interface.

Muttrc files should still be available, but their use should become optional.
This means that built-in configuration variables, commands and functions have
to be made available to the extension language.  Executing functions or
commands written using the extension language requires that the
command/function execution mechanism be customisable itself.


1.3. Implementation details

Being able to change what a function does using the extension language
requires that Mutt calls functions written in that language rather than
built-in C functions.  Usually, extension languages (i.e. interpreted
languages) would allow to redefine a given function.  This way, any feature
can easily be modified.  For instance, instead of calling mutt_display_message
(), Mutt has to directly call the `display-message' Scheme function; in turn,
this Scheme function may choose to just the `builtin-display-message' Scheme
function which is just a wrapper for the original `mutt_display_message ()' C
function.

In the current implementation, the functions that can be extended in this way
are defined in the file extension.h and we call them `commands' (which is
different from what Mutt call `commands', functions that are called within a
muttrc file).  These commands declared in extension.h with the
`MUTT_DECLARE_COMMAND' macro are extensible features.  This macro declares
three functions: one with a leading `cmd_' which is the function to be used
throughout the code; one with a leading `smutt_cmd_' which is the Scheme
version of this command; one with a leading `builtin_' which is the original C
function for this feature.  The C code should only call the `cmd_' functions,
which in turn call the corresponding `smutt_cmd_' function if the extension
language is Guile (see extension.c).  The `builtin_' functions should be made
available to the extension language.  Here is what happens, by default, when
`cmd_display_message ()' is called:

  cmd_display_message (C) -> smutt_cmd_display_message (C) -------------.
								        |
  .--- builtin-display-message (Scheme) <- display-message (Scheme) <---'
  |
  `---> builtin_display_message (C)

Sometimes, features can be customized at different levels of abstraction.
Consider the execution of a command:

  + source_rc_file (".muttrc")
  |
  +--+ parse_rc_line ("set realname='ludo' ; exec delete-message")
     |
     +--+ execute_command ("set", "realname='ludo'")
     |  |
     |  +--+ set_option ("realname", "ludo")
     |
     +--+ execute_command ("exec", "delete-message")
        |
	+--+ execute_function ("delete-message")

Each one of these features (commands) may be customized (except
`source_rc_file').  For example, allowing transparent
query/modification of built-in and extension language variables may be
done by customizing the `set_option' command and make it check whether
the given variable name is the name of a built-in variable or not.
Transparent execution of functions written in the execution language
can be done by customizing the `execute_command' and/or the
`execute_function' command.

Making it possible required to split the `parse_set ()' C function into two
distinct functions: `builtin_set_option ()' which is used to set the value of
a built-in variable, and `builtin_query_option ()' which is used to display
the value of a variable.  The latter one gets called for instance when typing
"set ?realname".  The `mutt_parse_rc_line ()' function also had to be splitted
in a similar way, leading to the creation of `builtin_execute_command ()'.

As one might guess, integrating features one by one is not very scalable but
seems to be the only solution.  However, any built-in Mutt command may be
customized since command execution is customizable.


1.4. Things left to do

A lot of things are left to be done, mostly making more features extensible.
The key binding mechanism might need some rewrite in order to be extensible
(currenly, a key is bound to an `op' which is basically the ID of a function
to be called when the key is pressed).  It should be possible to create and
handle menus (shouldn't be too hard).

Also, currently only a few features may be customized (see `extension.h' for a
complete list of what may be customized) and it would be nice to have some
more.  However, some features are not easy to integrate because the manipulate
internal things which we do not want the extension language to know: for
instance mx_sync_mailbox () sets its `index_hint' argument to something that
may be used by the index; send_message () from send.c may not only return
`true' of `false' but also a third value in case a message needs to be
postponed.

The current execute-command function is just passed a command name and an
unparsed argument string.  It would be much cleaner if argument parsing could
be done *before* execute-command is called and then commands would just
receive and argc/argv list of arguments.  However, this would require
rewriting of all Mutt commands and it may not work for all of them (for
instance, would '=' be an argument for command 'set' when typing "set
realname=myself"?).

Try to add support for your favorite language (Python?).



2. Extending and Customizing Mutt with Guile
--------------------------------------------


2.1. Extending Mutt: Defining New Commands, Functions and Options

All the features described here are programmed in Scheme in the system-wide
`mutt.scm' file which should get installed in /etc.

Transparent call of Scheme functions through Mutt's command interface (i.e.
either in a Muttrc file or by typing `:' in the index menu) or through the
function interface (i.e. "exec <function>").

The following Scheme functions

  (define-command (my-command argstring)
    (ui-message argstring))

  (define-function (my-function)
    (ui-message "This is my function"))

can be called within a muttrc file (or using Mutt's `enter-command' function):

  my-command and its arguments
  exec my-function

Note that `define-command' is a macro defined in the system-wide mutt.scm
configuration file that defines the function and calls `register-command!' so
that `execute-command' knows that it is a valid Mutt command.  A new command
called `help' returns the documentation of Scheme-defined commands,
functions, options and procedures (there also exist a `make-doc'
module capable of automagically generating documentation for the new
commands, functions and options available in your Mutt).

New user-visible options may be defined from Scheme code using the
`register-option!', `register-boolean-option!', or
`register-integer-option!' macros.  Once a Scheme variable has been
registered as an option, it may be accessed using muttrc instructions
like any built-in option.  Here are a few examples:

  (define do-it? #t)
  (register-boolean-option! "do-it?"
                            "Tells whether it should be done.")

  (define how-much 7)
  (register-integer-option! "how-much"
                            "Tells how much of it should be stuffed.")

These variables may then be accessed naturally:

  :set do-it?      is equivalent to    (set! do-it? #t)
  :set nodo-it?    is equivalent to    (set! do-it? #f)
  :set ?how-much   is equivalent to    (ui-message (query-option "how-much"))
  :set how-much=2  is equivalent to    (set! how-much 2)
  :help how-much   shows help message for this option.

One can also execute Scheme code directly from within Mutt using the
new `scheme' command:

  :scheme '(/ (mailbox-unread-messages) (mailbox-messages))'

On the other hand, it may sometimes be helpful to embed muttrc
instructions into your mutt.scm file.  This can be done very easily by
starting the muttrc intruction sequence with `#[' and terminating it
with `]#'.  Note that there may be several lines in between these
delimiters.  For instance,

  (set-option "spoolfile" "/var/mail/myself")
  (set-option "copy"      "yes")

and

  #[ set copy
     set spoolfile = /var/mail/myself ]#

are equivalent.  Alternatively, a muttrc file may be loaded using the
`source-rc-file' procedure:

  (source-rc-file (string-append (getenv "HOME") "/.muttrc"))

By default, your muttrc file won't be loaded, unless you don't have a
mutt.scm file.  So if you do have a mutt.scm, then you may want to
insert the line above in there.

Finally, Scheme things may be debugged by setting the Scheme variable
`debug?' to "yes" (which can be done by typing `:set debug?'); the
`debug-time' variable may be set to the delay of debug messages.


2.2. Changing Mutt's Behavior

Some useful internal functions of Mutt may be rewritten in Scheme in
order to obtain a different behavior than the default one.  As
explained before, there exist a Scheme binding for each such
customizable function (e.g. `score-message', `pipe-message',
`save-alias', etc.).  By default, the system-wide `mutt.scm' file
binds all these functions to their built-in version:

  (define define-alias builtin-define-alias)
  (define save-alias builtin-save-alias)
  (define display-message builtin-display-message)
  (define make-forward-subject! builtin-make-forward-subject!)
  (define score-message builtin-score-message)
  (define save-message builtin-save-message)
  (define append-signature builtin-append-signature)

However, Scheme code may very well any of these functions so that it
behaves differently:

  (define (append-signature port message)
    (if (execute-pattern to-my-girlfriend message)
        (display "\n--\nXoXo.\n" port)
        (builtin-append-signature port message)))

Another example is what the `(mutt aliases)' module does: it basically
redefines the `save-alias' function so that aliases can be saved in a
schemey way.  For instance, while the standard way of saving aliases
is:

  alias ludo Ludovic Courtès <ludo@ludo.net>

the schemey way to do that is:

  (define-alias "ludo" (create-address "Ludovic Courtès <ludo@ludo.net>"))

This is not very useful and may be overriden in the user's `$HOME/.mutt.scm'
by simply adding the following line:

  (set! save-alias builtin-save-alias)

Looking at the files `$PKGDATADIR/guile/mutt.scm' (typically
`/usr/local/share/mutt/guile/mutt.scm') and
`guile/contrib/dotmutt.scm' from the source tree (my own `.mutt.scm'
configuration file) is probably the best way to get started.  Also, a
number of ready-to-be-used modules providing various features will get
installed in your `$PKGDATADIR/guile/mutt' directory.  These modules
may be used very easily just by adding something like `(use-modules
(mutt MODULE-NAME))' to your `.mutt.scm' file.  Please, read the
`guile/README' file from the source tree for more information.


2.3. Data types

The available data types available are:

* `mailbox', the equivalent of Mutt's CONTEXT, which may be used to get the
  number of messages (`mailbox-messages mailbox', etc.) and to traverse them
  (`mailbox-select-messages proc mailbox');

* `alias', alias objects that may be created using
   `create-alias name address';

* `address', address objects that may be created using
   `create-address realname email';

* `message' which represents messages (HEADER struct) and may be
   observed and modified using
  `message-delete!', `message-score', `message-envelope', etc.;

* `envelope' which represents a header's envelope and may be observed and
  modified using `envelope-from', `envelope-subject', etc.;

* `body' represents the body of a message (i.e. an header object);

* `pattern' which represent Mutt limitation patterns and may be used using
   `compile-pattern string', `execute-pattern header pattern';

* `menu' which is not really useful yet.

* `score', the equivalent of Mutt's SCORE which represents a message scoring
  rule (not very useful).

* `buffy', the equivalent of Mutt's BUFFY type which represents incoming
  mailboxes specified with the `mailboxes' command.


All these data types come with a number of accessor functions allowing to
access (and modify) their contents.  For example, given MESSAGE, a message
object, the following expression

  (address-mailbox (envelope-to (message-envelope message)))

returns the email address of MESSAGE's recipient.  Some accessors are
procedures with setter, ie. procedures that may be used also when modifying an
element with `set!':

  (set! (message-score message) (+ 3 (message-score message)))

For a more detailed example of how message objects can be created and
handled, look at the `(mutt helpers send)' module which provide
functions for creating message objects that may be sent.  The `(mutt
reminders)' module also contains an "attachment reminder" that
traverses message attachments before sending it to look for the
occurence of the word "attach".

There are also type predicates: for instance, `message?' returns true when its
argument is a message object.


2.4. Things left to do

Currently, not every internal function may be customized.  Ideally, we
would like to be able to program the following things using *only*
Guile:

  -  tiny ssmtp-like email relay (the `send-message' feature);
  -  email protocol (e.g. nntp);
  -  new user interface (e.g. X/Gtk, although this would not be so useful).

Currently, only the first item would be feasible.  The second item
needs folder (`mailbox') handling be made extensible.  The last one
needs all UI-related stuff (menus, messages, fields) be made extensible.



Sept. 2003.
Please send comments to Ludovic Courtès <ludovic.courtes@laas.fr>.
$Id: README.Extensible 1.12 Mon, 03 Nov 2003 17:23:23 +0100 ludo $
