Merged NEW_SUMMARY branch back to trunk, and resolved conflicts.

2000-05-08  NotZed  <NotZed@HelixCode.com>

	* Merged NEW_SUMMARY branch back to trunk, and resolved conflicts.

	* providers/mbox/camel-mbox-summary.c (camel_mbox_summary_update):
	Return status.

	* camel-stream-filter.c (do_close): We NEED a stream close.

svn path=/trunk/; revision=2906

17 files changed, 1580 insertions, 1626 deletions

diff --git a/camel/ChangeLog b/camel/ChangeLog
index d66cb86975..afe10fe07c 100644
--- a/camel/ChangeLog
+++ b/camel/ChangeLog
@@ -1,3 +1,12 @@
+2000-05-08  NotZed  <NotZed@HelixCode.com>
+
+	* Merged NEW_SUMMARY branch back to trunk, and resolved conflicts.
+	
+	* providers/mbox/camel-mbox-summary.c (camel_mbox_summary_update):
+	Return status.
+
+	* camel-stream-filter.c (do_close): We NEED a stream close.
+
 2000-05-07  Dan Winship  <danw@helixcode.com>
 
 	Make camel not leak like a sieve.
@@ -65,6 +74,132 @@
 	* providers/smtp/camel-smtp-provider.c: Note in the description
 	that this provider is not yet tested.
 
+2000-05-08    <notzed@helixcode.com>
+
+	* camel-mime-part.c (write_to_stream): Free the filter stream when
+	done.
+
+	* camel-mime-parser.c (folder_seek): Make sure we add the \n
+	terminal when we seek as well (frob!).
+
+	* camel-mime-utils.c (header_decode_addrspec): Plug minor memleak.
+
+	* camel-mime-part.c (finalize): Free header tables once finished.
+
+	* camel-folder-summary.c (camel_folder_summary_remove): Dont try
+	to access info after its free'd.
+
+2000-05-07  NotZed  <NotZed@HelixCode.com>
+
+	* camel-mime-part.c (write_to_stream): Apply encoding to content
+	part, when writing to a stream *sigh*.
+
+	* camel-stream-filter.c (do_write): implement write for the
+	filtering stream.  Writes shouldn't be mixed with reads.
+	(do_flush): Implemented flush.  Again write/flush shouldn't be
+	mixed with reads.  Only flushes if the last op was write.
+	(do_close): Force flush on close.
+
+	* camel-mime-filter.c (filter_run): Oops, make sure we include the
+	backlen in the total length before passing onto the filter.
+
+	* camel-mime-filter-from.c: New filter, munges 'From ' lines into
+	'>From ', for mbox.
+
+	* camel-mime-parser.c (camel_mime_parser_header_remove): New
+	function to remove the parser's raw header, rather than
+	manipulating the header directly (wich doesn't work with
+	mempools).
+
+	* camel-mime-utils.c (header_address_list_clear): Fixed some
+	broken(tm) logic, which would leak entries on multivalued lists.
+
+	* providers/mbox/camel-mbox-summary.c (camel_mbox_summary_load):
+	Use ibex_save() to save the ibex.  Makes a big difference to
+	startup times for very large mailboxes.
+	(camel_mbox_summary_expunge): Dum de dum, reimplemented.  Designed
+	to be much more robust, and to stop immediately if anything awry
+	happens.
+	(copy_block): Utility function to copy n bytes from one fd to
+	another.
+	(header_write): Utility function to write out raw headers to an
+	fd.
+	(camel_mbox_summary_update): Incremental summary updater.
+
+	* providers/mbox/camel-mbox-folder.c (mbox_get_message_by_uid):
+	Dont unref the stream, because of the broken(tm) ref model of gtk
+	widget that for some odd reason is being perpetuated in camel.	
+	(mbox_expunge): Reenable expunge again.
+	(mbox_append_message): Removed the optimised mbox append.  If its
+	an issue, it can go back later.  Cleaned up a lot, checks error
+	returns, and automagically translates 'From ' into '>From' as
+	necessary.
+
+2000-05-07    <notzed@helixcode.com>
+
+	* camel-mime-filter.c (filter_run): Oops, forgot to add the
+	backlen to the pre-buffer (*poof*).
+
+2000-05-07  NotZed  <NotZed@HelixCode.com>
+
+	* camel-mime-message.c (construct_from_parser): Allow
+	HSCAN_FROM_END to terminate the processing of a message.
+
+	* camel-folder-summary.c (perform_content_info_load): Ick, dont
+	try and append a node onto its own list.
+	(camel_folder_summary_clear): Actually clear the indexes after
+	we've removed the messages.
+	(camel_folder_summary_clear): Set dirty if it changes.
+	(camel_folder_summary_load): Clear dirty.
+	(camel_folder_summary_save): Only save if dirty.
+
+	* providers/mbox/camel-mbox-summary.c (summary_header_load): Oops,
+	remember to call that parent class first ...
+	(summary_header_save): Here too.
+	(camel_mbox_summary_load): Do more checking to verify the index
+	contents as well as teh summary contents, against the mbox
+	contents.
+	(camel_mbox_summary_load): Removed some fo that checking, it needs
+	more code to work reliably.
+
+2000-05-07    <notzed@helixcode.com>
+
+	* providers/mbox/camel-mbox-summary.c (camel_mbox_summary_load):
+	Set the size and mtime of the mbox we indexed once done.
+
+	* camel-folder-summary.c (camel_folder_summary_set_index): Dont
+	write the index if it changes - let the claler fix it (uh, kind of
+	impacts performance).
+	(camel_folder_summary_load): close in.
+
+	* camel-folder-summary.c (summary_format_string): Check header
+	exists before trying to strip its leading spaces.
+
+2000-05-06  NotZed  <NotZed@HelixCode.com>
+
+	* camel-folder.h: Removed summary info from here, and include
+	camel-folder-summary.h as well.
+	
+	* camel-mime-parser.c (camel_mime_parser_step): Allow it to accept
+	a NULL databuffer.
+
+	* providers/mbox/camel-mbox-summary.c: Totally new file, now
+	subclasses camel-folder-summary.
+
+	* camel-folder-summary.c (message_info_load): Load the uid as a
+	string.
+	(message_info_save): And save too.
+	(camel_folder_summary_clear): New function, clears the contents of
+	the summary.
+
+	* providers/mbox/camel-mbox-folder.c: Fixes for	summary changes.
+	(mbox_get_message_by_uid): Completely redone.  Now cross-checks
+	the summary information to make sure we get a real message.
+	(mbox_append_message): Disabled the copy version of append for
+	now.
+	(mbox_expunge): Temporarily disabled the expunge function, until
+	it is put back in camel-mbox-summary.c
+
 2000-05-05  NotZed  <NotZed@HelixCode.com>
 
 	* camel-folder-summary.c: And same here ...
@@ -72,9 +207,16 @@
 	return -1 on error ..
 	(camel_folder_summary_decode_fixed_int32): Neither deos fread.
 	(camel_folder_summary_encode_token): Fix here too.
+	(summary_build_content_info): Use start-headers to get the pos of
+	the message, not parser_tell(), which might not be what we
+	expected because of parser_unstep().
+	(camel_folder_summary_encode_token): Use bserch() to tokenise the
+	values, rather than a linear search.
 
 	* camel-mime-utils.c: Defined out some memory profiling stuff I
 	left there by mistake.
+	(header_decode_mailbox): Dont try to append the word part of a
+	local address if we ran out of words.
 
 	* camel-mime-parser.c (folder_scan_content): Apply the fix from
 	the header scanner to here too.
diff --git a/camel/Makefile.am b/camel/Makefile.am
index c24113df45..3b9eee05de 100644
--- a/camel/Makefile.am
+++ b/camel/Makefile.am
@@ -38,6 +38,7 @@ libcamel_la_SOURCES = 				\
 	camel-exception.c			\
 	camel-folder.c				\
 	camel-folder-search.c			\
+	camel-folder-summary.c			\
 	camel-medium.c				\
 	camel-marshal-utils.c			\
 	camel-mime-message.c			\
@@ -71,20 +72,19 @@ libcamel_la_SOURCES = 				\
 	camel-mime-filter-save.c		\
 	camel-mime-filter-charset.c		\
 	camel-mime-filter-index.c		\
+	camel-mime-filter-from.c		\
 	camel-stream-filter.c			\
 	camel-address.c				\
 	camel-internet-address.c		\
 	$(pthread_SRC)
 
-#	camel-folder-summary.c			\
-#	camel-folder-summary.h			\
-
 libcamelinclude_HEADERS =			\
 	camel.h					\
 	camel-data-wrapper.h			\
 	camel-exception.h			\
 	camel-folder.h				\
 	camel-folder-search.h			\
+	camel-folder-summary.h			\
 	camel-marshal-utils.h			\
 	camel-medium.h				\
 	camel-mime-message.h			\
@@ -119,6 +119,7 @@ libcamelinclude_HEADERS =			\
 	camel-mime-filter-save.h		\
 	camel-mime-filter-charset.h		\
 	camel-mime-filter-index.h		\
+	camel-mime-filter-from.h		\
 	camel-stream-filter.h			\
 	camel-address.h				\
 	camel-internet-address.h		\
@@ -133,19 +134,19 @@ EXTRA_DIST =					\
 	$(libcamel_extra_sources)		\
 	README 
 
-noinst_PROGRAMS =				\
-	camel-folder-summary
-
-camel_folder_summary_SOURCES = 			\
-	camel-folder-summary.c
-
-camel_folder_summary_LDADD = 			\
-	../camel/libcamel.la			\
-	../e-util/libeutil.la			\
-	../libibex/libibex.la			\
-	$(GNOME_LIBDIR)				\
-	$(GNOMEUI_LIBS)				\
-	$(INTLLIBS)				\
-	$(PTHREAD_LIB)				\
-	$(EXTRA_GNOME_LIBS)
+#noinst_PROGRAMS =				\
+#	camel-mime-filter-from
+#
+#camel_mime_filter_from_SOURCES = 		\
+#	camel-mime-filter-from.c
+#
+#camel_mime_filter_from_LDADD = 		\
+#	../camel/libcamel.la			\
+#	../e-util/libeutil.la			\
+#	../libibex/libibex.la			\
+#	$(GNOME_LIBDIR)				\
+#	$(GNOMEUI_LIBS)				\
+#	$(INTLLIBS)				\
+#	$(PTHREAD_LIB)				\
+#	$(EXTRA_GNOME_LIBS)
 
diff --git a/camel/camel-folder-summary.c b/camel/camel-folder-summary.c
index de66646483..bdca6e677c 100644
--- a/camel/camel-folder-summary.c
+++ b/camel/camel-folder-summary.c
@@ -24,6 +24,7 @@
 #include <ctype.h>
 #include <string.h>
 #include <errno.h>
+#include <stdlib.h>
 
 #include "camel-folder-summary.h"
 
@@ -34,7 +35,11 @@
 #include <camel/camel-mime-filter-basic.h>
 #include "hash-table-utils.h"
 
+/* this should probably be conditional on it existing */
+#define USE_BSEARCH
+
 #define d(x)
+#define io(x)			/* io debug */
 
 #if 0
 extern int strdup_count, malloc_count, free_count;
@@ -216,8 +221,6 @@ void camel_folder_summary_set_index(CamelFolderSummary *s, ibex *index)
 {
 	struct _CamelFolderSummaryPrivate *p = _PRIVATE(s);
 
-	if (p->index)
-		ibex_write(p->index);
 	p->index = index;
 }
 
@@ -275,8 +278,8 @@ perform_content_info_load(CamelFolderSummary *s, FILE *in)
 	for (i=0;i<count;i++) {
 		part = perform_content_info_load(s, in);
 		if (part) {
-			my_list_append((struct _node **)&ci->childs, (struct _node *)ci);
-			ci->parent = ci;
+			my_list_append((struct _node **)&ci->childs, (struct _node *)part);
+			part->parent = ci;
 		} else {
 			g_warning("Summary file format messed up?");
 		}
@@ -293,27 +296,39 @@ camel_folder_summary_load(CamelFolderSummary *s)
 
 	g_assert(s->summary_path);
 
+	printf("loading summary\n");
+
 	in = fopen(s->summary_path, "r");
 	if ( in == NULL ) {
 		return -1;
 	}
 
+	printf("loading header\n");
+
 	if ( ((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->summary_header_load(s, in) == -1) {
 		fclose(in);
 		return -1;
 	}
 
+	printf("loading content\n");
+
 	/* now read in each message ... */
 	/* FIXME: check returns */
 	for (i=0;i<s->saved_count;i++) {
 		mi = ((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->message_info_load(s, in);
 
 		if (s->build_content) {
-			mi->content = content_info_load(s, in);
+			mi->content = perform_content_info_load(s, in);
 		}
+
+		camel_folder_summary_add(s, mi);
 	}
+	
+	if (fclose(in) == -1)
+		return -1;
+
+	s->flags &= ~CAMEL_SUMMARY_DIRTY;
 
-	/* FIXME: check error return */
 	return 0;
 }
 
@@ -344,6 +359,13 @@ camel_folder_summary_save(CamelFolderSummary *s)
 
 	g_assert(s->summary_path);
 
+	printf("saving summary?\n");
+
+	if ((s->flags & CAMEL_SUMMARY_DIRTY) == 0) {
+		printf("nup\n");
+		return 0;
+	}
+
 	fd = open(s->summary_path, O_RDWR|O_CREAT, 0600);
 	if (fd == -1)
 		return -1;
@@ -353,6 +375,8 @@ camel_folder_summary_save(CamelFolderSummary *s)
 		return -1;
 	}
 
+	io(printf("saving header\n"));
+
 	if ( ((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->summary_header_save(s, out) == -1) {
 		fclose(out);
 		return -1;
@@ -369,7 +393,11 @@ camel_folder_summary_save(CamelFolderSummary *s)
 			perform_content_info_save(s, out, mi->content);
 		}
 	}
-	return fclose(out);
+	if (fclose(out) == -1)
+		return -1;
+
+	s->flags &= ~CAMEL_SUMMARY_DIRTY;
+	return 0;
 }
 
 void camel_folder_summary_add(CamelFolderSummary *s, CamelMessageInfo *info)
@@ -392,17 +420,19 @@ retry:
 	s->flags |= CAMEL_SUMMARY_DIRTY;
 }
 
-void camel_folder_summary_add_from_header(CamelFolderSummary *s, struct _header_raw *h)
+CamelMessageInfo *camel_folder_summary_add_from_header(CamelFolderSummary *s, struct _header_raw *h)
 {
-	CamelMessageInfo *info;
+	CamelMessageInfo *info = NULL;
 
 	info = ((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->message_info_new(s, h);
 	camel_folder_summary_add(s, info);
+
+	return info;
 }
 
-void camel_folder_summary_add_from_parser(CamelFolderSummary *s, CamelMimeParser *mp)
+CamelMessageInfo *camel_folder_summary_add_from_parser(CamelFolderSummary *s, CamelMimeParser *mp)
 {
-	CamelMessageInfo *info;
+	CamelMessageInfo *info = NULL;
 	char *buffer;
 	int len;
 	struct _CamelFolderSummaryPrivate *p = _PRIVATE(s);
@@ -437,6 +467,68 @@ void camel_folder_summary_add_from_parser(CamelFolderSummary *s, CamelMimeParser
 
 		camel_folder_summary_add(s, info);
 	}
+	return info;
+}
+
+static void
+perform_content_info_free(CamelFolderSummary *s, CamelMessageContentInfo *ci)
+{
+	CamelMessageContentInfo *pw, *pn;
+
+	pw = ci->childs;
+	((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->content_info_free(s, ci);
+	while (pw) {
+		pn = pw->next;
+		perform_content_info_free(s, pw);
+		pw = pn;
+	}
+}
+
+void
+camel_folder_summary_clear(CamelFolderSummary *s)
+{
+	int i;
+
+	if (camel_folder_summary_count(s) == 0)
+		return;
+
+	for (i=0;i<camel_folder_summary_count(s);i++) {
+		CamelMessageInfo *mi = camel_folder_summary_index(s, i);
+		CamelMessageContentInfo *ci = mi->content;
+
+		((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->message_info_free(s, mi);		
+		if (s->build_content && ci) {
+			perform_content_info_free(s, ci);
+		}
+	}
+
+	g_ptr_array_set_size(s->messages, 0);
+	g_hash_table_destroy(s->messages_uid);
+	s->messages_uid = g_hash_table_new(g_str_hash, g_str_equal);
+	s->flags |= CAMEL_SUMMARY_DIRTY;
+}
+
+void camel_folder_summary_remove(CamelFolderSummary *s, CamelMessageInfo *info)
+{
+	CamelMessageContentInfo *ci = info->content;
+
+	g_hash_table_remove(s->messages_uid, info->uid);
+	g_ptr_array_remove(s->messages, info);
+	((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->message_info_free(s, info);
+	if (s->build_content && ci) {
+		perform_content_info_free(s, ci);
+	}
+}
+
+void camel_folder_summary_remove_uid(CamelFolderSummary *s, const char *uid)
+{
+        CamelMessageInfo *oldinfo;
+        char *olduid;
+
+        if (g_hash_table_lookup_extended(s->messages_uid, uid, (void *)&olduid, (void *)&oldinfo)) {
+		camel_folder_summary_remove(s, oldinfo);
+		g_free(olduid);
+        }
 }
 
 int
@@ -444,6 +536,8 @@ camel_folder_summary_encode_uint32(FILE *out, guint32 value)
 {
 	int i;
 
+	io(printf("Encoding int %u\n", value));
+
 	for (i=28;i>0;i-=7) {
 		if (value >= (1<<i)) {
 			unsigned int c = (value>>i) & 0x7f;
@@ -469,6 +563,9 @@ camel_folder_summary_decode_uint32(FILE *in, guint32 *dest)
 		return 01;
 	}
 	*dest = value | (v&0x7f);
+
+	io(printf("Decoding int %u\n", *dest));
+
         return 0;
 }
 
@@ -520,10 +617,12 @@ static char * tokens[] = {
 	"octet-stream",
 	"parallel",
 	"plain",
+	"postscript",
 	"quoted-printable",
+	"related",
 	"rfc822",
 	"text",
-	"us-ascii",		/* 23 words */
+	"us-ascii",		/* 25 words */
 };
 
 #define tokens_len (sizeof(tokens)/sizeof(tokens[0]))
@@ -534,9 +633,20 @@ static char * tokens[] = {
     >=32 string, length = n-32
 */
 
+#ifdef USE_BSEARCH
+static int
+token_search_cmp(char *key, char **index)
+{
+	d(printf("comparing '%s' to '%s'\n", key, *index));
+	return strcmp(key, *index);
+}
+#endif
+
 int
 camel_folder_summary_encode_token(FILE *out, char *str)
 {
+	io(printf("Encoding token: '%s'\n", str));
+
 	if (str == NULL) {
 		return camel_folder_summary_encode_uint32(out, 0);
 	} else {
@@ -545,16 +655,23 @@ camel_folder_summary_encode_token(FILE *out, char *str)
 
 		if (len <= 16) {
 			char lower[32];
+			char **match;
 
 			for (i=0;i<len;i++)
 				lower[i] = tolower(str[i]);
 			lower[i] = 0;
+#ifdef USE_BSEARCH
+			match = bsearch(lower, tokens, tokens_len, sizeof(char *), (int (*)(void *, void *))token_search_cmp);
+			if (match)
+				token = match-tokens;
+#else
 			for (i=0;i<tokens_len;i++) {
 				if (!strcmp(tokens[i], lower)) {
 					token = i;
 					break;
 				}
 			}
+#endif
 		}
 		if (token != -1) {
 			return camel_folder_summary_encode_uint32(out, token+1);
@@ -573,8 +690,11 @@ camel_folder_summary_decode_token(FILE *in, char **str)
 {
 	char *ret;
 	int len;
+
+	io(printf("Decode token ...\n"));
 	
 	if (camel_folder_summary_decode_uint32(in, &len) == -1) {
+		g_warning("Could not decode token from file");
 		*str = NULL;
 		return -1;
 	}
@@ -604,6 +724,8 @@ camel_folder_summary_decode_token(FILE *in, char **str)
 		ret[len]=0;
 	}
 
+	io(printf("Token = '%s'\n", ret));
+
 	*str = ret;
 	return 0;
 }
@@ -613,6 +735,8 @@ camel_folder_summary_encode_string(FILE *out, char *str)
 {
 	register int len;
 
+	io(printf("Encoding string: '%s'\n", str));
+
 	if (str == NULL)
 		return camel_folder_summary_encode_uint32(out, 0);
 
@@ -631,6 +755,8 @@ camel_folder_summary_decode_string(FILE *in, char **str)
 	int len;
 	register char *ret;
 
+	io(printf("Decode string ...\n", str));
+
 	if (camel_folder_summary_decode_uint32(in, &len) == -1) {
 		*str = NULL;
 		return -1;
@@ -639,6 +765,7 @@ camel_folder_summary_decode_string(FILE *in, char **str)
 	len--;
 	if (len < 0) {
 		*str = NULL;
+		io(printf("String = '%s'\n", *str));
 		return -1;
 	}
 
@@ -649,6 +776,8 @@ camel_folder_summary_decode_string(FILE *in, char **str)
 		return -1;
 	}
 
+	io(printf("String = '%s'\n", ret));
+
 	ret[len] = 0;
 	*str = ret;
 	return 0;
@@ -698,6 +827,8 @@ summary_header_load(CamelFolderSummary *s, FILE *in)
 
 	fseek(in, 0, SEEK_SET);
 
+	io(printf("Loading header\n"));
+
 	if (camel_folder_summary_decode_fixed_int32(in, &version) == -1
 	    || camel_folder_summary_decode_fixed_int32(in, &flags) == -1
 	    || camel_folder_summary_decode_fixed_int32(in, &nextuid) == -1
@@ -722,6 +853,8 @@ summary_header_save(CamelFolderSummary *s, FILE *out)
 {
 	fseek(out, 0, SEEK_SET);
 
+	io(printf("Savining header\n"));
+
 	camel_folder_summary_encode_fixed_int32(out, s->version);
 	camel_folder_summary_encode_fixed_int32(out, s->flags);
 	camel_folder_summary_encode_fixed_int32(out, s->nextuid);
@@ -788,6 +921,21 @@ summary_format_address(struct _header_raw *h, const char *name)
 	return ret;
 }
 
+static char *
+summary_format_string(struct _header_raw *h, const char *name)
+{
+	const char *text;
+
+	text = header_raw_find(&h, name, NULL);
+	if (text) {
+		while (isspace(*text))
+			text++;
+		return header_decode_string(text);
+	} else {
+		return NULL;
+	}
+}
+
 static CamelMessageInfo *
 message_info_new(CamelFolderSummary *s, struct _header_raw *h)
 {
@@ -795,7 +943,7 @@ message_info_new(CamelFolderSummary *s, struct _header_raw *h)
 
 	mi = g_malloc0(s->message_info_size);
 
-	mi->subject = header_decode_string(header_raw_find(&h, "subject", NULL));
+	mi->subject = summary_format_string(h, "subject");
 	mi->from = summary_format_address(h, "from");
 	mi->to = summary_format_address(h, "to");
 	mi->date_sent = header_decode_date(header_raw_find(&h, "date", NULL), NULL);
@@ -808,13 +956,13 @@ message_info_new(CamelFolderSummary *s, struct _header_raw *h)
 static CamelMessageInfo *
 message_info_load(CamelFolderSummary *s, FILE *in)
 {
-	guint32 tmp;
 	CamelMessageInfo *mi;
 
 	mi = g_malloc0(s->message_info_size);
 
-	camel_folder_summary_decode_uint32(in, &tmp);
-	mi->uid = g_strdup_printf("%u", tmp);
+	io(printf("Loading message info\n"));
+
+	camel_folder_summary_decode_string(in, &mi->uid);
 	camel_folder_summary_decode_uint32(in, &mi->flags);
 	camel_folder_summary_decode_uint32(in, &mi->date_sent);	/* warnings, leave them here */
 	camel_folder_summary_decode_uint32(in, &mi->date_received);
@@ -830,7 +978,9 @@ message_info_load(CamelFolderSummary *s, FILE *in)
 static int
 message_info_save(CamelFolderSummary *s, FILE *out, CamelMessageInfo *mi)
 {
-	camel_folder_summary_encode_uint32(out, strtoul(mi->uid, NULL, 10));
+	io(printf("Saving message info\n"));
+
+	camel_folder_summary_encode_string(out, mi->uid);
 	camel_folder_summary_encode_uint32(out, mi->flags);
 	camel_folder_summary_encode_uint32(out, mi->date_sent);
 	camel_folder_summary_encode_uint32(out, mi->date_received);
@@ -875,8 +1025,14 @@ content_info_load(CamelFolderSummary *s, FILE *in)
 	guint32 count, i;
 	struct _header_content_type *ct;
 
+	io(printf("Loading content info\n"));
+
 	ci = g_malloc0(s->content_info_size);
 
+	camel_folder_summary_decode_uint32(in, &ci->pos);
+	camel_folder_summary_decode_uint32(in, &ci->bodypos);
+	camel_folder_summary_decode_uint32(in, &ci->endpos);
+
 	camel_folder_summary_decode_token(in, &type);
 	camel_folder_summary_decode_token(in, &subtype);
 	ct = header_content_type_new(type, subtype);
@@ -908,6 +1064,12 @@ content_info_save(CamelFolderSummary *s, FILE *out, CamelMessageContentInfo *ci)
 	struct _header_content_type *ct;
 	struct _header_param *hp;
 
+	io(printf("Saving content info\n"));
+
+	camel_folder_summary_encode_uint32(out, ci->pos);
+	camel_folder_summary_encode_uint32(out, ci->bodypos);
+	camel_folder_summary_encode_uint32(out, ci->endpos);
+
 	ct = ci->type;
 	if (ct) {
 		camel_folder_summary_encode_token(out, ct->type);
@@ -952,20 +1114,21 @@ summary_build_content_info(CamelFolderSummary *s, CamelMimeParser *mp)
 	char *buffer;
 	CamelMessageContentInfo *info = NULL;
 	struct _header_content_type *ct;
-	int start, body;
+	int body;
 	int enc_id = -1, chr_id = -1, idx_id = -1;
 	struct _CamelFolderSummaryPrivate *p = _PRIVATE(s);
 	CamelMimeFilterCharset *mfc;
 	CamelMessageContentInfo *part;
 
+	d(printf("building content info\n"));
+
 	/* start of this part */
-	start = camel_mime_parser_tell(mp);
 	state = camel_mime_parser_step(mp, &buffer, &len);
 	body = camel_mime_parser_tell(mp);
 
 	info = ((CamelFolderSummaryClass *)((GtkObject *)s)->klass)->content_info_new_from_parser(s, mp);
 
-	info->pos = start;
+	info->pos = camel_mime_parser_tell_start_headers(mp);
 	info->bodypos = body;
 
 	switch(state) {
@@ -975,17 +1138,23 @@ summary_build_content_info(CamelFolderSummary *s, CamelMimeParser *mp)
 		if (p->index && header_content_type_is(ct, "text", "*")) {
 			char *encoding;
 			const char *charset;
-				
+			
+			d(printf("generating index:\n"));
+			
 			encoding = header_content_encoding_decode(camel_mime_parser_header(mp, "content-transfer-encoding", NULL));
 			if (encoding) {
 				if (!strcasecmp(encoding, "base64")) {
+					d(printf(" decoding base64\n"));
 					if (p->filter_64 == NULL)
 						p->filter_64 = camel_mime_filter_basic_new_type(CAMEL_MIME_FILTER_BASIC_BASE64_DEC);
 					enc_id = camel_mime_parser_filter_add(mp, (CamelMimeFilter *)p->filter_64);
 				} else if (!strcasecmp(encoding, "quoted-printable")) {
+					d(printf(" decoding quoted-printable\n"));
 					if (p->filter_qp == NULL)
 						p->filter_qp = camel_mime_filter_basic_new_type(CAMEL_MIME_FILTER_BASIC_QP_DEC);
 					enc_id = camel_mime_parser_filter_add(mp, (CamelMimeFilter *)p->filter_qp);
+				} else {
+					d(printf(" ignoring encoding %s\n", encoding));
 				}
 				g_free(encoding);
 			}
@@ -994,7 +1163,7 @@ summary_build_content_info(CamelFolderSummary *s, CamelMimeParser *mp)
 			if (charset!=NULL
 			    && !(strcasecmp(charset, "us-ascii")==0
 				 || strcasecmp(charset, "utf-8")==0)) {
-				d(printf("Adding conversion filter from %s to utf-8\n", charset));
+				d(printf(" Adding conversion filter from %s to utf-8\n", charset));
 				mfc = g_hash_table_lookup(p->filter_charset, charset);
 				if (mfc == NULL) {
 					mfc = camel_mime_filter_charset_new_convert(charset, "utf-8");
@@ -1020,6 +1189,7 @@ summary_build_content_info(CamelFolderSummary *s, CamelMimeParser *mp)
 		camel_mime_parser_filter_remove(mp, idx_id);
 		break;
 	case HSCAN_MULTIPART:
+		d(printf("Summarising multipart\n"));
 		while (camel_mime_parser_step(mp, &buffer, &len) != HSCAN_MULTIPART_END) {
 			camel_mime_parser_unstep(mp);
 			part = summary_build_content_info(s, mp);
@@ -1032,6 +1202,7 @@ summary_build_content_info(CamelFolderSummary *s, CamelMimeParser *mp)
 		}
 		break;
 	case HSCAN_MESSAGE:
+		d(printf("Summarising message\n"));
 		part = summary_build_content_info(s, mp);
 		if (part) {
 			part->parent = info;
@@ -1049,10 +1220,54 @@ summary_build_content_info(CamelFolderSummary *s, CamelMimeParser *mp)
 
 	info->endpos = camel_mime_parser_tell(mp);
 
+	d(printf("finished building content info\n"));
+
 	return info;
 }
 
-#if 1
+static void
+content_info_dump(CamelMessageContentInfo *ci, int depth)
+{
+	char *p;
+
+	p = alloca(depth*4+1);
+	memset(p, ' ', depth*4);
+	p[depth*4] = 0;
+
+	if (ci == NULL) {
+		printf("%s<empty>\n", p);
+		return;
+	}
+
+	printf("%sconent-type: %s/%s\n", p, ci->type->type, ci->type->subtype);
+	printf("%sontent-transfer-encoding: %s\n", p, ci->encoding);
+	printf("%scontent-description: %s\n", p, ci->description);
+	printf("%sbytes: %d %d %d\n", p, (int)ci->pos, (int)ci->bodypos, (int)ci->endpos);
+	ci = ci->childs;
+	while (ci) {
+		content_info_dump(ci, depth+1);
+		ci = ci->next;
+	}
+}
+
+static void
+message_info_dump(CamelMessageInfo *mi)
+{
+	if (mi == NULL) {
+		printf("No message?\n");
+		return;
+	}
+
+	printf("Subject: %s\n", mi->subject);
+	printf("To: %s\n", mi->to);
+	printf("From: %s\n", mi->from);
+	printf("UID: %s\n", mi->uid);
+	printf("Flags: %04x\n", mi->flags & 0xffff);
+	content_info_dump(mi->content, 0);
+}
+
+
+#if 0
 int main(int argc, char **argv)
 {
 	CamelMimeParser *mp;
@@ -1060,6 +1275,7 @@ int main(int argc, char **argv)
 	CamelFolderSummary *s;
 	char *buffer;
 	int len;
+	int i;
 	ibex *index;
 
 	gtk_init(&argc, &argv);
@@ -1109,12 +1325,34 @@ int main(int argc, char **argv)
 		}
 	}
 
+	printf("Printing summary\n");
+	for (i=0;i<camel_folder_summary_count(s);i++) {
+		message_info_dump(camel_folder_summary_index(s, i));
+	}
+
 	printf("Saivng summary\n");
 	camel_folder_summary_set_filename(s, "index.summary");
 	camel_folder_summary_save(s);
 
-	gtk_object_unref(GTK_OBJECT(mp));
-	gtk_object_unref(GTK_OBJECT(s));
+	{
+		CamelFolderSummary *n;
+
+		printf("\nLoading summary\n");
+		n = camel_folder_summary_new();
+		camel_folder_summary_set_build_content(n, TRUE);
+		camel_folder_summary_set_filename(n, "index.summary");
+		camel_folder_summary_load(n);
+
+		printf("Printing summary\n");
+		for (i=0;i<camel_folder_summary_count(n);i++) {
+			message_info_dump(camel_folder_summary_index(n, i));
+		}
+		gtk_object_unref(n);		
+	}
+
+
+	gtk_object_unref(mp);
+	gtk_object_unref(s);
 
 	printf("summarised %d messages\n", camel_folder_summary_count(s));
 #if 0
diff --git a/camel/camel-folder-summary.h b/camel/camel-folder-summary.h
index c8948a405c..71b7cab7cb 100644
--- a/camel/camel-folder-summary.h
+++ b/camel/camel-folder-summary.h
@@ -97,7 +97,7 @@ struct _CamelFolderSummary {
 	guint32 flags;		/* flags */
 	guint32 nextuid;	/* next uid? */
 	guint32 saved_count;	/* how many were saved/loaded */
-	time_t time;		/* timestamp for this summary */
+	time_t time;		/* timestamp for this summary (for implementors to use) */
 
 	/* sizes of memory objects */
 	guint32 message_info_size;
@@ -148,11 +148,16 @@ int camel_folder_summary_save(CamelFolderSummary *);
 
 /* add a new raw summary item */
 void camel_folder_summary_add(CamelFolderSummary *, CamelMessageInfo *info);
-void camel_folder_summary_add_from_header(CamelFolderSummary *, struct _header_raw *);
-void camel_folder_summary_add_from_parser(CamelFolderSummary *, CamelMimeParser *);
 
-/* removes a summary item, fixes offsets? */
-void camel_mbox_summary_remove_uid(CamelFolderSummary *s, const char *uid);
+/* build/add raw summary items */
+CamelMessageInfo *camel_folder_summary_add_from_header(CamelFolderSummary *, struct _header_raw *);
+CamelMessageInfo *camel_folder_summary_add_from_parser(CamelFolderSummary *, CamelMimeParser *);
+
+/* removes a summary item, doesn't fix content offsets */
+void camel_folder_summary_remove(CamelFolderSummary *s, CamelMessageInfo *info);
+void camel_folder_summary_remove_uid(CamelFolderSummary *s, const char *uid);
+/* remove all items */
+void camel_folder_summary_clear(CamelFolderSummary *s);
 
 /* lookup functions */
 int camel_folder_summary_count(CamelFolderSummary *);
diff --git a/camel/camel-folder.h b/camel/camel-folder.h
index c8ffdd52ac..eac1898028 100644
--- a/camel/camel-folder.h
+++ b/camel/camel-folder.h
@@ -35,7 +35,7 @@ extern "C" {
 #endif /* __cplusplus }*/
 
 #include <camel/camel-object.h>
-#include <time.h>
+#include <camel/camel-folder-summary.h>
 
 #define CAMEL_FOLDER_TYPE     (camel_folder_get_type ())
 #define CAMEL_FOLDER(obj)     (GTK_CHECK_CAST((obj), CAMEL_FOLDER_TYPE, CamelFolder))
@@ -54,49 +54,6 @@ typedef enum {
 	FOLDER_OPEN_RW      = 3    /* folder is read/write        */ 
 } CamelFolderOpenMode;
 
-
-typedef struct {
-	gchar *name;
-	gint nb_message;	/* ick, these should be renamed to something better */
-	gint nb_unread_message;
-	gint nb_deleted_message;
-} CamelFolderInfo;
-
-/* A tree of message content info structures
-   describe the content structure of the message (if it has any) */
-typedef struct _CamelMessageContentInfo {
-	struct _CamelMessageContentInfo *next;
-
-	struct _CamelMessageContentInfo *childs;
-	struct _CamelMessageContentInfo *parent;
-
-	struct _header_content_type *type;
-	char *id;
-	char *description;
-	char *encoding;
-
-	guint32 size;
-
-} CamelMessageContentInfo;
-
-/* information about a given object */
-typedef struct {
-	/* public fields */
-	gchar *subject;
-	gchar *to;
-	gchar *from;
-
-	gchar *uid;
-	guint32 flags;
-
-	time_t date_sent;
-	time_t date_received;
-
-	/* tree of content description - NULL if it is not available */
-	CamelMessageContentInfo *content;
-} CamelMessageInfo;
-
-
 typedef void (*CamelFolderAsyncCallback) ();
 
 struct _CamelFolder
@@ -119,8 +76,6 @@ struct _CamelFolder
 	gboolean has_search_capability:1;
 };
 
-
-
 typedef struct {
 	CamelObjectClass parent_class;
 
diff --git a/camel/camel-mime-filter-from.c b/camel/camel-mime-filter-from.c
new file mode 100644
index 0000000000..065b15e656
--- /dev/null
+++ b/camel/camel-mime-filter-from.c
@@ -0,0 +1,242 @@
+/*
+ *  Copyright (C) 2000 Helix Code Inc.
+ *
+ *  Authors: Michael Zucchi <notzed@helixcode.com>
+ *
+ *  This program is free software; you can redistribute it and/or 
+ *  modify it under the terms of the GNU General Public License as 
+ *  published by the Free Software Foundation; either version 2 of the
+ *  License, or (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+ *  USA
+ */
+
+#include "camel-mime-filter-from.h"
+#include <string.h>
+
+#define d(x)
+
+struct _CamelMimeFilterFromPrivate {
+};
+
+#define _PRIVATE(o) (((CamelMimeFilterFrom *)(o))->priv)
+
+static void camel_mime_filter_from_class_init (CamelMimeFilterFromClass *klass);
+static void camel_mime_filter_from_init       (CamelMimeFilterFrom *obj);
+static void camel_mime_filter_from_finalise   (GtkObject *obj);
+
+static CamelMimeFilterClass *camel_mime_filter_from_parent;
+
+enum SIGNALS {
+	LAST_SIGNAL
+};
+
+static guint signals[LAST_SIGNAL] = { 0 };
+
+guint
+camel_mime_filter_from_get_type (void)
+{
+	static guint type = 0;
+	
+	if (!type) {
+		GtkTypeInfo type_info = {
+			"CamelMimeFilterFrom",
+			sizeof (CamelMimeFilterFrom),
+			sizeof (CamelMimeFilterFromClass),
+			(GtkClassInitFunc) camel_mime_filter_from_class_init,
+			(GtkObjectInitFunc) camel_mime_filter_from_init,
+			(GtkArgSetFunc) NULL,
+			(GtkArgGetFunc) NULL
+		};
+		
+		type = gtk_type_unique (camel_mime_filter_get_type (), &type_info);
+	}
+	
+	return type;
+}
+
+struct fromnode {
+	struct fromnode *next;
+	char *pointer;
+};
+
+static void
+complete(CamelMimeFilter *mf, char *in, size_t len, size_t prespace, char **out, size_t *outlen, size_t *outprespace)
+{
+	*out = in;
+	*outlen = len;
+	*outprespace = prespace;
+}
+
+/* Yes, it is complicated ... */
+static void
+filter(CamelMimeFilter *mf, char *in, size_t len, size_t prespace, char **out, size_t *outlen, size_t *outprespace)
+{
+	CamelMimeFilterFrom *f = (CamelMimeFilterFrom *)mf;
+	register char *inptr, *inend;
+	int left;
+	int midline = f->midline;
+	int fromcount = 0;
+	struct fromnode *head= NULL, *tail = (struct fromnode *)&head, *node;
+	char *outptr;
+
+	inptr = in;
+	inend = inptr+len;
+
+	d(printf("Filtering '%.*s'\n", len, in));
+
+	/* first, see if we need to escape any from's */
+	while (inptr<inend) {
+		register int c=-1;
+
+		if (midline)
+			while (inptr<inend && (c=*inptr++) != '\n')
+				;
+
+		if (c=='\n' || !midline) {
+			left = inend-inptr;
+			if (left > 0) {
+				midline = TRUE;
+				if (left<5) {
+					if (inptr[0]=='F') {
+						camel_mime_filter_backup(mf, inptr, left);
+						midline = FALSE;
+						inend = inptr;
+						break;
+					}
+				} else {
+					if (!strncmp(inptr, "From ", 5)) {
+						fromcount++;
+						/* yes, we do alloc them on the stack ... at most we're going to get
+						   len / 7 of them anyway */
+						node = alloca(sizeof(*node));
+						node->pointer = inptr;
+						node->next = NULL;
+						tail->next = node;
+						tail = node;
+						inptr+=5;
+					}
+				}
+			} else {
+				/* \n is at end of line, check next buffer */
+				midline = FALSE;
+			}
+		}
+	}
+
+	f->midline = midline;
+
+	if (fromcount > 0) {
+		camel_mime_filter_set_size(mf, len + fromcount, FALSE);
+		node = head;
+		inptr = in;
+		outptr = mf->outbuf;
+		while (node) {
+			memcpy(outptr, inptr, node->pointer-inptr);
+			outptr+=node->pointer-inptr;
+			*outptr++='>';
+			inptr = node->pointer;
+			node = node->next;
+		}
+		memcpy(outptr, inptr, inend-inptr);
+		outptr += inend-inptr;
+		*out = mf->outbuf;
+		*outlen = outptr-mf->outbuf;
+		*outprespace = mf->outbuf-mf->outreal;
+
+		d(printf("Filtered '%.*s'\n", *outlen, *out));
+	} else {
+		*out = in;
+		*outlen = inend - in;
+		*outprespace = prespace;
+		
+		d(printf("Filtered '%.*s'\n", *outlen, *out));
+	}
+}
+
+static void
+camel_mime_filter_from_class_init (CamelMimeFilterFromClass *klass)
+{
+	GtkObjectClass *object_class = (GtkObjectClass *) klass;
+	CamelMimeFilterClass *filter_class = (CamelMimeFilterClass *) klass;
+	
+	camel_mime_filter_from_parent = gtk_type_class (camel_mime_filter_get_type ());
+
+	object_class->finalize = camel_mime_filter_from_finalise;
+
+	filter_class->filter = filter;
+	filter_class->complete = complete;
+
+	gtk_object_class_add_signals (object_class, signals, LAST_SIGNAL);
+}
+
+static void
+camel_mime_filter_from_init (CamelMimeFilterFrom *obj)
+{
+	struct _CamelMimeFilterFromPrivate *p;
+
+	p = _PRIVATE(obj) = g_malloc0(sizeof(*p));
+	obj->midline = FALSE;
+}
+
+static void
+camel_mime_filter_from_finalise (GtkObject *obj)
+{
+	((GtkObjectClass *)(camel_mime_filter_from_parent))->finalize((GtkObject *)obj);
+}
+
+/**
+ * camel_mime_filter_from_new:
+ *
+ * Create a new CamelMimeFilterFrom object.
+ * 
+ * Return value: A new CamelMimeFilterFrom widget.
+ **/
+CamelMimeFilterFrom *
+camel_mime_filter_from_new (void)
+{
+	CamelMimeFilterFrom *new = CAMEL_MIME_FILTER_FROM ( gtk_type_new (camel_mime_filter_from_get_type ()));
+	return new;
+}
+
+#if 0
+
+#include <stdio.h>
+
+int main(int argc, char **argv)
+{
+	CamelMimeFilterFrom *f;
+	char *buffer;
+	int len, prespace;
+
+	gtk_init(&argc, &argv);
+
+
+	f = camel_mime_filter_from_new();
+
+	buffer = "This is a test\nFrom Someone\nTo someone. From Someone else, From\n From blah\nFromblah\nBye! \nFrom ";
+	len = strlen(buffer);
+	prespace = 0;
+
+	printf("input = '%.*s'\n", len, buffer);
+	camel_mime_filter_filter(f, buffer, len, prespace, &buffer, &len, &prespace);
+	printf("output = '%.*s'\n", len, buffer);
+	buffer = "";
+	len = 0;
+	prespace =0;
+	camel_mime_filter_complete(f, buffer, len, prespace, &buffer, &len, &prespace);
+	printf("complete = '%.*s'\n", len, buffer);
+	
+
+	return 0;
+}
+
+#endif
diff --git a/camel/camel-mime-filter-from.h b/camel/camel-mime-filter-from.h
new file mode 100644
index 0000000000..2a97bcc0f0
--- /dev/null
+++ b/camel/camel-mime-filter-from.h
@@ -0,0 +1,50 @@
+/*
+ *  Copyright (C) 2000 Helix Code Inc.
+ *
+ *  Authors: Michael Zucchi <notzed@helixcode.com>
+ *
+ *  This program is free software; you can redistribute it and/or 
+ *  modify it under the terms of the GNU General Public License as 
+ *  published by the Free Software Foundation; either version 2 of the
+ *  License, or (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+ *  USA
+ */
+
+#ifndef _CAMEL_MIME_FILTER_FROM_H
+#define _CAMEL_MIME_FILTER_FROM_H
+
+#include <gtk/gtk.h>
+#include <camel/camel-mime-filter.h>
+
+#define CAMEL_MIME_FILTER_FROM(obj)         GTK_CHECK_CAST (obj, camel_mime_filter_from_get_type (), CamelMimeFilterFrom)
+#define CAMEL_MIME_FILTER_FROM_CLASS(klass) GTK_CHECK_CLASS_CAST (klass, camel_mime_filter_from_get_type (), CamelMimeFilterFromClass)
+#define IS_CAMEL_MIME_FILTER_FROM(obj)      GTK_CHECK_TYPE (obj, camel_mime_filter_from_get_type ())
+
+typedef struct _CamelMimeFilterFrom      CamelMimeFilterFrom;
+typedef struct _CamelMimeFilterFromClass CamelMimeFilterFromClass;
+
+struct _CamelMimeFilterFrom {
+	CamelMimeFilter parent;
+
+	struct _CamelMimeFilterFromPrivate *priv;
+
+	int midline;		/* are we between lines? */
+};
+
+struct _CamelMimeFilterFromClass {
+	CamelMimeFilterClass parent_class;
+};
+
+guint		camel_mime_filter_from_get_type	(void);
+CamelMimeFilterFrom      *camel_mime_filter_from_new	(void);
+
+#endif /* ! _CAMEL_MIME_FILTER_FROM_H */
diff --git a/camel/camel-mime-filter.c b/camel/camel-mime-filter.c
index e402ba0ff2..27cee890fe 100644
--- a/camel/camel-mime-filter.c
+++ b/camel/camel-mime-filter.c
@@ -139,7 +139,7 @@ static void filter_run(CamelMimeFilter *f,
 	  have the pre-space required.  We make a buffer that does ...
 	*/
 	if (prespace < f->backlen) {
-		int newlen = len+prespace;
+		int newlen = len+prespace+f->backlen;
 		p = _PRIVATE(f);
 		if (p->inlen < newlen) {
 			/* NOTE: g_realloc copies data, we dont need that (slower) */
@@ -157,6 +157,7 @@ static void filter_run(CamelMimeFilter *f,
 	if (f->backlen > 0) {
 		memcpy(in-f->backlen, f->backbuf, f->backlen);
 		in -= f->backlen;
+		len += f->backlen;
 		prespace -= f->backlen;
 		f->backlen = 0;
 	}
diff --git a/camel/camel-mime-message.c b/camel/camel-mime-message.c
index db42a3e599..d6fdc2c0a6 100644
--- a/camel/camel-mime-message.c
+++ b/camel/camel-mime-message.c
@@ -463,7 +463,7 @@ camel_mime_message_set_user_flag	(CamelMimeMessage *m, const char *name, gboolea
 	char *oldname;
 	gboolean oldvalue;
 
-	there = g_hash_table_lookup_extended(m->user_flags, name, &oldname, &oldvalue);
+	there = g_hash_table_lookup_extended(m->user_flags, name, (void *)&oldname, (void *)&oldvalue);
 
 	if (value && !there) {
 		g_hash_table_insert(m->user_flags, g_strdup(name), (void *)TRUE);
@@ -519,8 +519,13 @@ construct_from_parser(CamelMimePart *dw, CamelMimeParser *mp)
 
 	/* ... then clean up the follow-on state */
 	state = camel_mime_parser_step(mp, &buf, &len);
-	if (!(state == HSCAN_MESSAGE_END || state == HSCAN_EOF)) {
-		g_error("Bad parser state: Expecing MESSAGE_END or EOF, got: %d", camel_mime_parser_state(mp));
+	switch (state) {
+	case HSCAN_EOF: case HSCAN_FROM_END: /* this doesn't belong to us */
+		camel_mime_parser_unstep(mp);
+	case HSCAN_MESSAGE_END:
+		break;
+	default:
+		g_error("Bad parser state: Expecing MESSAGE_END or EOF or ROM, got: %d", camel_mime_parser_state(mp));
 		camel_mime_parser_unstep(mp);
 		return -1;
 	}
diff --git a/camel/camel-mime-parser.c b/camel/camel-mime-parser.c
index 97a43e44d7..ef4434e97a 100644
--- a/camel/camel-mime-parser.c
+++ b/camel/camel-mime-parser.c
@@ -33,6 +33,7 @@
 #include <unicode.h>
 
 #include <regex.h>
+#include <ctype.h>
 
 #include <glib.h>
 #include "camel-mime-parser.h"
@@ -46,10 +47,17 @@
 #define c(x)
 #define d(x)
 
+/*#define PURIFY*/
+
 #define MEMPOOL
 
 #define STRUCT_ALIGN 4
 
+#ifdef PURIFY
+int inend_id = -1,
+  inbuffer_id = -1;
+#endif
+
 #if 0
 extern int strdup_count;
 extern int malloc_count;
@@ -166,6 +174,7 @@ void mempool_free(MemPool *pool)
 		g_free(pool);
 	}
 }
+
 #endif
 
 
@@ -266,6 +275,9 @@ static struct _header_scan_stack *folder_scan_header(struct _header_scan_state *
 static int folder_scan_skip_line(struct _header_scan_state *s);
 static off_t folder_seek(struct _header_scan_state *s, off_t offset, int whence);
 static off_t folder_tell(struct _header_scan_state *s);
+#ifdef MEMPOOL
+static void header_append_mempool(struct _header_scan_state *s, struct _header_scan_stack *h, char *header, int offset);
+#endif
 
 static void camel_mime_parser_class_init (CamelMimeParserClass *klass);
 static void camel_mime_parser_init       (CamelMimeParser *obj);
@@ -323,7 +335,9 @@ static void
 finalise(GtkObject *o)
 {
 	struct _header_scan_state *s = _PRIVATE(o);
-
+#ifdef PURIFY
+	purify_watch_remove_all();
+#endif
 	folder_scan_close(s);
 
 	((GtkObjectClass *)camel_mime_parser_parent)->finalize (o);
@@ -464,7 +478,7 @@ camel_mime_parser_header(CamelMimeParser *m, const char *name, int *offset)
  * until the next call to parser_step(), or parser_drop_step().
  * 
  * Return value: The raw headers, or NULL if there are no headers
- * defined for the current part or state.
+ * defined for the current part or state.  These are READ ONLY.
  **/
 struct _header_raw *
 camel_mime_parser_headers_raw(CamelMimeParser *m)
@@ -603,7 +617,8 @@ void camel_mime_parser_drop_step(CamelMimeParser *m)
  * camel_mime_parser_step:
  * @m: 
  * @databuffer: Pointer to accept a pointer to the data
- * associated with this step (if any).
+ * associated with this step (if any).  May be #NULL,
+ * in which case datalength is also ingored.
  * @datalength: Pointer to accept a pointer to the data
  * length associated with this step (if any).
  * 
@@ -630,9 +645,17 @@ camel_mime_parser_step(CamelMimeParser *m, char **databuffer, int *datalength)
 
 	d(printf("OLD STATE:  '%s' :\n", states[s->state]));
 
-	if (s->unstep <= 0)
+	if (s->unstep <= 0) {
+		char *dummy;
+		int dummylength;
+
+		if (databuffer == NULL) {
+			databuffer = &dummy;
+			datalength = &dummylength;
+		}
+			
 		folder_scan_step(s, databuffer, datalength);
-	else
+	} else
 		s->unstep--;
 
 	d(printf("NEW STATE:  '%s' :\n", states[s->state]));
@@ -791,7 +814,10 @@ folder_read(struct _header_scan_state *s)
 
 	if (s->inptr<s->inend-s->atleast)
 		return s->inend-s->inptr;
-
+#ifdef PURIFY
+	purify_watch_remove(inend_id);
+	purify_watch_remove(inbuffer_id);
+#endif
 	/* check for any remaning bytes (under the atleast limit( */
 	inoffset = s->inend - s->inptr;
 	if (inoffset>0) {
@@ -812,7 +838,10 @@ folder_read(struct _header_scan_state *s)
 	}
 
 	g_assert(s->inptr<=s->inend);
-
+#ifdef PURIFY
+	inend_id = purify_watch(&s->inend);
+	inbuffer_id = purify_watch_n(s->inend+1, SCAN_HEAD-1, "rw");
+#endif
 	r(printf("content = %d '%.*s'\n", s->inend - s->inptr,  s->inend - s->inptr, s->inptr));
 	/* set a sentinal, for the inner loops to check against */
 	s->inend[0] = '\n';
@@ -848,6 +877,10 @@ folder_seek(struct _header_scan_state *s, off_t offset, int whence)
 	} else {
 		newoffset = lseek(s->fd, offset, whence);
 	}
+#ifdef PURIFY
+	purify_watch_remove(inend_id);
+	purify_watch_remove(inbuffer_id);
+#endif
 	if (newoffset != -1) {
 		s->seek = newoffset;
 		s->inptr = s->inbuf;
@@ -856,11 +889,16 @@ folder_seek(struct _header_scan_state *s, off_t offset, int whence)
 			len = camel_stream_read(s->stream, s->inbuf, SCAN_BUF);
 		else
 			len = read(s->fd, s->inbuf, SCAN_BUF);
-		if (len>=0)
+		if (len>=0) {
 			s->inend = s->inbuf+len;
-		else
+			s->inend[0] = '\n';
+		} else
 			newoffset = -1;
 	}
+#ifdef PURIFY
+	inend_id = purify_watch(&s->inend);
+	inbuffer_id = purify_watch_n(s->inend+1, SCAN_HEAD-1, "rw");
+#endif
 	return newoffset;
 }
 
@@ -1086,6 +1124,8 @@ retry:
 				s->midline = FALSE;
 			}
 
+			g_assert(inptr<=s->inend);
+
 			header_append(s, start, inptr);
 
 			h(printf("outbuf[0] = %02x '%c' oubuf[1] = %02x '%c'\n",
@@ -1230,6 +1270,8 @@ retry:
 			} else {
 				s->midline = FALSE;
 			}
+
+			g_assert(inptr<=s->inend);
 		}
 
 		/* *sigh* so much for the beautiful simplicity of the code so far - here we
diff --git a/camel/camel-mime-parser.h b/camel/camel-mime-parser.h
index baefbfe67f..2283ec6f98 100644
--- a/camel/camel-mime-parser.h
+++ b/camel/camel-mime-parser.h
@@ -97,9 +97,10 @@ enum _header_state camel_mime_parser_state(CamelMimeParser *);
 /* get content type for the current part/header */
 struct _header_content_type *camel_mime_parser_content_type(CamelMimeParser *);
 
-/* get a raw header by name */
+/* get/change raw header by name */
 const char *camel_mime_parser_header(CamelMimeParser *, const char *, int *offset);
-/* get all raw headers */
+
+/* get all raw headers. READ ONLY! */
 struct _header_raw *camel_mime_parser_headers_raw(CamelMimeParser *);
 
 /* add a processing filter for body contents */
diff --git a/camel/camel-mime-part.c b/camel/camel-mime-part.c
index 7d08c6a5b1..1688168737 100644
--- a/camel/camel-mime-part.c
+++ b/camel/camel-mime-part.c
@@ -34,6 +34,8 @@
 #include <ctype.h>
 #include "camel-mime-parser.h"
 #include "camel-stream-mem.h"
+#include "camel-stream-filter.h"
+#include "camel-mime-filter-basic.h"
 
 #define d(x)
 
@@ -184,6 +186,8 @@ finalize (GtkObject *object)
 
 	if (mime_part->content_input_stream) gtk_object_unref (GTK_OBJECT (mime_part->content_input_stream));
 
+	header_raw_clear(&mime_part->headers);
+
 	GTK_OBJECT_CLASS (parent_class)->finalize (object);
 }
 
@@ -503,9 +507,37 @@ write_to_stream (CamelDataWrapper *data_wrapper, CamelStream *stream)
 
 	content = camel_medium_get_content_object (medium);
 	if (content) {
+		/* I dont really like this here, but i dont know where else it might go ... */
+#define CAN_THIS_GO_ELSEWHERE
+#ifdef CAN_THIS_GO_ELSEWHERE
+		CamelMimeFilter *filter = NULL;
+		CamelStreamFilter *filter_stream = NULL;
+
+		switch(mp->encoding) {
+		case CAMEL_MIME_PART_ENCODING_BASE64:
+			filter = (CamelMimeFilter *)camel_mime_filter_basic_new_type(CAMEL_MIME_FILTER_BASIC_BASE64_ENC);
+			break;
+		case CAMEL_MIME_PART_ENCODING_QUOTEDPRINTABLE:
+			filter = (CamelMimeFilter *)camel_mime_filter_basic_new_type(CAMEL_MIME_FILTER_BASIC_QP_ENC);
+			break;
+		default:
+			break;
+		}
+		if (filter) {
+			gtk_object_ref((GtkObject *)stream);
+			filter_stream = camel_stream_filter_new_with_stream(stream);
+			camel_stream_filter_add(filter_stream, filter);
+			stream = (CamelStream *)filter_stream;
+		}
+
+#endif
 		if ( (count = camel_data_wrapper_write_to_stream(content, stream)) == -1 )
-			return -1;
-		total += count;
+			total = -1;
+		else
+			total += count;
+
+		if (filter_stream)
+			gtk_object_unref((GtkObject *)filter_stream);
 	} else {
 		g_warning("No content for medium, nothing to write");
 	}
diff --git a/camel/camel-mime-utils.c b/camel/camel-mime-utils.c
index 22b252116b..65bbdf32e8 100644
--- a/camel/camel-mime-utils.c
+++ b/camel/camel-mime-utils.c
@@ -1293,6 +1293,7 @@ header_decode_addrspec(const char **in)
 			word = header_decode_domain(&inptr);
 			if (word) {
 				addr = g_string_append(addr, word);
+				g_free(word);
 			} else {
 				w(g_warning("Invalid address, missing domain: %s", *in));
 			}
@@ -1392,8 +1393,10 @@ header_decode_mailbox(const char **in)
 		inptr++;
 		g_free(pre);
 		pre = header_decode_word(&inptr);
-		addr = g_string_append_c(addr, '.');
-		addr = g_string_append(addr, pre);
+		if (pre) {
+			addr = g_string_append_c(addr, '.');
+			addr = g_string_append(addr, pre);
+		}
 		header_decode_lwsp(&inptr);
 	}
 	g_free(pre);
@@ -2274,12 +2277,13 @@ void header_address_list_append(struct _header_address **l, struct _header_addre
 void header_address_list_clear(struct _header_address **l)
 {
 	struct _header_address *a, *n;
-	a = (struct _header_address *)l;
-	while (a && a->next) {
+	a = *l;
+	while (a) {
 		n = a->next;
-		a = n->next;
-		header_address_unref(n);
+		header_address_unref(a);
+		a = n;
 	}
+	*l = NULL;
 }
 
 static void
diff --git a/camel/camel-stream-filter.c b/camel/camel-stream-filter.c
index 500bf22846..e468435cb6 100644
--- a/camel/camel-stream-filter.c
+++ b/camel/camel-stream-filter.c
@@ -35,6 +35,8 @@ struct _CamelStreamFilterPrivate {
 
 	char *filtered;		/* the filtered data */
 	size_t filteredlen;
+
+	int last_was_read;	/* was the last op read or write? */
 };
 
 #define READ_PAD (64)		/* bytes padded before buffer */
@@ -130,6 +132,7 @@ camel_stream_filter_init (CamelStreamFilter *obj)
 	_PRIVATE(obj) = p = g_malloc0(sizeof(*p));
 	p->realbuffer = g_malloc(READ_SIZE + READ_PAD);
 	p->buffer = p->realbuffer + READ_PAD;
+	p->last_was_read = TRUE;
 }
 
 /**
@@ -216,6 +219,8 @@ static	gint      do_read       (CamelStream *stream, gchar *buffer, gint n)
 	int size;
 	struct _filter *f;
 
+	p->last_was_read = TRUE;
+
 	if (p->filteredlen<=0) {
 		int presize = READ_SIZE;
 
@@ -253,16 +258,52 @@ static	gint      do_read       (CamelStream *stream, gchar *buffer, gint n)
 	return size;
 }
 
-static	gint      do_write      (CamelStream *stream, const gchar *buffer, gint n)
+static	gint      do_write      (CamelStream *stream, const gchar *buf, gint n)
 {
-	/* what semantics *should* this have?? */
-	g_warning("Writing to a non-writable stream");
-	return -1;
+	CamelStreamFilter *filter = (CamelStreamFilter *)stream;
+	struct _CamelStreamFilterPrivate *p = _PRIVATE(filter);
+	struct _filter *f;
+	int presize;
+	char *buffer = (char *)buf;
+
+	p->last_was_read = FALSE;
+
+	f = p->filters;
+	presize = 0;
+	while (f) {
+		camel_mime_filter_filter(f->filter, buffer, n, presize, &buffer, &n, &presize);
+		f = f->next;
+	}
+
+	return camel_stream_write(filter->source, buffer, n);
 }
 
 static	void      do_flush      (CamelStream *stream)
 {
-	/* NO OP */
+	CamelStreamFilter *filter = (CamelStreamFilter *)stream;
+	struct _CamelStreamFilterPrivate *p = _PRIVATE(filter);
+	struct _filter *f;
+	char *buffer;
+	int len, presize;
+
+	if (p->last_was_read) {
+		g_warning("Flushing a filter stream without writing to it");
+		return;
+	}
+
+	buffer = "";
+	len = 0;
+	presize = 0;
+	f = p->filters;
+	while (f) {
+		camel_mime_filter_complete(f->filter, buffer, len, presize, &buffer, &len, &presize);
+		f = f->next;
+	}
+	if (camel_stream_write(filter->source, buffer, len) == -1) {
+		g_warning("Flushing filter failed to write, no way to signal failure ...");
+	}
+
+	return camel_stream_flush(filter->source);
 }
 
 static	gboolean  do_eos        (CamelStream *stream)
@@ -276,6 +317,19 @@ static	gboolean  do_eos        (CamelStream *stream)
 	return camel_stream_eos(filter->source);
 }
 
+static	void      do_close      (CamelStream *stream)
+{
+	CamelStreamFilter *filter = (CamelStreamFilter *)stream;
+	struct _CamelStreamFilterPrivate *p = _PRIVATE(filter);
+
+	if (p->last_was_read == 0) {
+		camel_stream_flush(stream);
+	}
+
+	p->filteredlen = 0;
+	camel_stream_close(filter->source);
+}
+
 static	void      do_reset      (CamelStream *stream)
 {
 	CamelStreamFilter *filter = (CamelStreamFilter *)stream;
diff --git a/camel/providers/mbox/camel-mbox-folder.c b/camel/providers/mbox/camel-mbox-folder.c
index a0a593a00f..255753b140 100644
--- a/camel/providers/mbox/camel-mbox-folder.c
+++ b/camel/providers/mbox/camel-mbox-folder.c
@@ -42,7 +42,8 @@
 #include "camel-mbox-summary.h"
 #include "camel-data-wrapper.h"
 #include "camel-mime-message.h"
-
+#include "camel-stream-filter.h"
+#include "camel-mime-filter-from.h"
 #include "camel-exception.h"
 
 #define d(x)
@@ -207,26 +208,37 @@ static void
 mbox_open (CamelFolder *folder, CamelFolderOpenMode mode, CamelException *ex)
 {
 	CamelMboxFolder *mbox_folder = CAMEL_MBOX_FOLDER (folder);
+	int forceindex;
+	struct stat st;
 
 	/* call parent class */
 	parent_class->open (folder, mode, ex);
 	if (camel_exception_get_id(ex))
 		return;
 
+	/* if we have no index file, force it */
+	forceindex = stat(mbox_folder->index_file_path, &st) == -1;
+
+	printf("loading ibex\n");
 	mbox_folder->index = ibex_open(mbox_folder->index_file_path, O_CREAT|O_RDWR, 0600);
+	printf("loaded ibex\n");
 	if (mbox_folder->index == NULL) {
-		g_warning("Could not open/create index file: %s: indexing will not function",
+		/* yes, this isn't fatal at all */
+		g_warning("Could not open/create index file: %s: indexing not performed",
 			  strerror(errno));
 	}
 
+	/* no summary (disk or memory), and we're proverbially screwed */
+	printf("loading summary\n");
 	mbox_folder->summary = camel_mbox_summary_new(mbox_folder->summary_file_path, mbox_folder->folder_file_path, mbox_folder->index);
-	if (mbox_folder->summary == NULL) {
+	if (mbox_folder->summary == NULL
+	    || camel_mbox_summary_load(mbox_folder->summary, forceindex) == -1) {
 		camel_exception_set (ex, 
 				     CAMEL_EXCEPTION_FOLDER_INVALID, /* FIXME: right error code */
 				     "Could not create summary");
 		return;
 	}
-	camel_mbox_summary_load(mbox_folder->summary);
+	printf("summary loaded\n");
 }
 
 static void
@@ -246,15 +258,17 @@ mbox_close (CamelFolder *folder, gboolean expunge, CamelException *ex)
 		ibex_close(mbox_folder->index);
 		mbox_folder->index = NULL;
 	}
-	camel_mbox_summary_save (mbox_folder->summary);
-	camel_mbox_summary_unref (mbox_folder->summary);
-	mbox_folder->summary = NULL;
-	if (mbox_folder->search)
+	if (mbox_folder->summary) {
+		camel_folder_summary_save ((CamelFolderSummary *)mbox_folder->summary);
+		gtk_object_unref((GtkObject *)mbox_folder->summary);
+		mbox_folder->summary = NULL;
+	}
+	if (mbox_folder->search) {
 		gtk_object_unref((GtkObject *)mbox_folder->search);
-	mbox_folder->search = NULL;
+		mbox_folder->search = NULL;
+	}
 }
 
-
 static void
 mbox_expunge (CamelFolder *folder, CamelException *ex)
 {
@@ -269,7 +283,6 @@ mbox_expunge (CamelFolder *folder, CamelException *ex)
 	gtk_signal_emit_by_name((GtkObject *)folder, "folder_changed", 0);
 }
 
-
 /* FIXME: clean up this snot */
 static gboolean
 mbox_exists (CamelFolder *folder, CamelException *ex)
@@ -690,131 +703,83 @@ mbox_get_message_count (CamelFolder *folder, CamelException *ex)
 	g_assert (folder);
 	g_assert (mbox_folder->summary);
 	
-	return camel_mbox_summary_message_count(mbox_folder->summary);
+	return camel_folder_summary_count((CamelFolderSummary *)mbox_folder->summary);
 }
 
-/*
-  This is a lazy append.
-
-  Basically, messages are appended to the end of the mbox, and probably assigned
-  a new uid (they wont be if copying from a source folder which doesn't have
-  a uid - which wont happen with the current summariser).
-
-  Indexing/summarising happens when the mbox is next queried.
-
-  Should this set a flag up for subsequent updating??
-*/
-
 /* FIXME: this may need some tweaking for performance? */
-/* FIXME: MUST check all sytem call return codes MUST MUST */
 static void
 mbox_append_message (CamelFolder *folder, CamelMimeMessage *message, CamelException *ex)
 {
-	CamelMboxFolder *mbox_folder = CAMEL_MBOX_FOLDER (folder), *source_folder;
-	CamelStream *output_stream;
+	CamelMboxFolder *mbox_folder = CAMEL_MBOX_FOLDER (folder);
+	CamelStream *output_stream = NULL, *filter_stream = NULL;
+	CamelMimeFilter *filter_from;
 	struct stat st;
-	off_t seek;
+	off_t seek = -1;
 	char *xev;
 	guint32 uid;
 
-	if (stat(mbox_folder->folder_file_path, &st) != 0) {
-		camel_exception_setv (ex, 
-				      CAMEL_EXCEPTION_FOLDER_INSUFFICIENT_PERMISSION, /* FIXME: what code? */
-				      "Cannot append to mbox file: %s", strerror (errno));
-		return;
-	}
-
-	/* are we coming from an mbox folder?  then we can optimise somewhat ... */
-	if (message->folder && IS_CAMEL_MBOX_FOLDER(message->folder)) {
-		CamelMboxMessageInfo *info;
-		int sfd, dfd;
-		off_t pos;
-
-		/* FIXME: this is pretty ugly - we lookup the message info in the source folder, copy it,
-		   then go back and paste in its real uid. */
-		source_folder = (CamelMboxFolder *)message->folder;
-		info = camel_mbox_summary_uid(source_folder->summary, message->message_uid);
-
-		d(printf("Copying message directly from %s to %s\n", source_folder->folder_file_path, mbox_folder->folder_file_path));
-		d(printf("start = %d, xev = %d\n", ((CamelMboxMessageContentInfo *)info->info.content)->pos, info->xev_offset));
-
-		sfd = open(source_folder->folder_file_path, O_RDONLY);
-		dfd = open(mbox_folder->folder_file_path, O_RDWR|O_CREAT, 0600);
-		if (lseek(dfd, st.st_size, SEEK_SET) != st.st_size) {
-			camel_exception_setv (ex, 
-					      CAMEL_EXCEPTION_FOLDER_INSUFFICIENT_PERMISSION, /* FIXME: what code? */
-					      "Cannot append to mbox file: %s", strerror (errno));
-			close(sfd);
-			close(dfd);
-			return;
-		}
-		write(dfd, "From - \n", strlen("From - \n"));
-		camel_mbox_summary_copy_block
-			(sfd, dfd, ((CamelMboxMessageContentInfo *)info->info.content)->pos,
-			 ((CamelMboxMessageContentInfo *)info->info.content)->endpos - ((CamelMboxMessageContentInfo *)info->info.content)->pos);
-		if (info->xev_offset != -1) {
-			pos = st.st_size + (info->xev_offset - ((CamelMboxMessageContentInfo *)info->info.content)->pos) + strlen("From - \n");
-			d(printf("Inserting new uid at %d\n", (int)pos));
-			if (pos != lseek(dfd, pos, SEEK_SET)) {
-				ftruncate(dfd, st.st_size);
-				camel_exception_setv (ex, 
-						      CAMEL_EXCEPTION_FOLDER_INSUFFICIENT_PERMISSION, /* FIXME: what code? */
-						      "Cannot append to mbox file: %s", strerror (errno));
-				close(sfd);
-				close(dfd);
-				return;
-			}
-			uid = camel_mbox_summary_next_uid(mbox_folder->summary);
-			xev = g_strdup_printf("X-Evolution: %08x-%04x", uid, 0);
-			write(dfd, xev, strlen(xev)); /* FIXME: check return */
-			d(printf("header = %s\n", xev));
-			g_free(xev);
-		}
-		close(sfd);
-		close(dfd);
-		return;
-	}
+	if (stat(mbox_folder->folder_file_path, &st) != 0)
+		goto fail;
 
-	/* its not an mbox folder, so lets do it the slow way ... */
-	output_stream = camel_stream_fs_new_with_name (mbox_folder->folder_file_path, O_CREAT|O_RDWR, 0600);
-	if (output_stream == NULL) {
-		camel_exception_setv (ex, 
-				      CAMEL_EXCEPTION_FOLDER_INSUFFICIENT_PERMISSION, /* FIXME: what code? */
-				      "Cannot append to mbox file: %s", strerror (errno));
-		return;
-	}
+	output_stream = camel_stream_fs_new_with_name (mbox_folder->folder_file_path, O_RDWR, 0600);
+	if (output_stream == NULL)
+		goto fail;
 
 	seek = camel_seekable_stream_seek((CamelSeekableStream *)output_stream, st.st_size, SEEK_SET);
-	if (seek != st.st_size) {
-		camel_exception_setv (ex, 
-				      CAMEL_EXCEPTION_FOLDER_INSUFFICIENT_PERMISSION, /* FIXME: what code? */
-				      "Cannot seek to position in mbox file: %s", strerror (errno));
-		gtk_object_unref ((GtkObject *)output_stream);
-		return;
-	}
+	if (seek != st.st_size)
+		goto fail;
 
-	/* assign a new x-evolution header */
-	/* FIXME: save flags? */
+	/* assign a new x-evolution header/uid */
 	camel_medium_remove_header((CamelMedium *)message, "X-Evolution");
-	uid = camel_mbox_summary_next_uid(mbox_folder->summary);
-	xev = g_strdup_printf("%08x-%04x", uid, 0);
+	uid = camel_folder_summary_next_uid((CamelFolderSummary *)mbox_folder->summary);
+	xev = g_strdup_printf("%08x-%04x", uid, message->flags & 0xffff);
 	camel_medium_add_header((CamelMedium *)message, "X-Evolution", xev);
 	g_free(xev);
 
-	camel_stream_write_string (output_stream, "From - \n");
-	/* FIXME: does this return an error?   IT HAS TO FOR THIS TO BE RELIABLE */
-	camel_data_wrapper_write_to_stream (CAMEL_DATA_WRAPPER (message), output_stream);
-
-	/* TODO: update the summary so it knows a new message is there to summarise/index */
-	/* This is only a performance improvement, the summary is *only* a cache */
+	/* we must write this to the non-filtered stream ... */
+	if (camel_stream_write_string (output_stream, "From - \n") == -1)
+		goto fail;
+
+	/* and write the content to the filtering stream, that translated '\nFrom' into '\n>From' */
+	filter_stream = (CamelStream *)camel_stream_filter_new_with_stream(output_stream);
+	filter_from = (CamelMimeFilter *)camel_mime_filter_from_new();
+	camel_stream_filter_add((CamelStreamFilter *)filter_stream, filter_from);
+	if (camel_data_wrapper_write_to_stream (CAMEL_DATA_WRAPPER (message), filter_stream) == -1)
+		goto fail;
+
+#warning WE NEED A STREAM CLOSE OR THIS WILL FAIL TO WORK
+#warning WE NEED A STREAM CLOSE OR THIS WILL FAIL TO WORK
+#warning WE NEED A STREAM CLOSE OR THIS WILL FAIL TO WORK
+#warning WE NEED A STREAM CLOSE OR THIS WILL FAIL TO WORK
+
+	/* FIXME: stream_close doesn't return anything */
+/*	camel_stream_close (filter_stream);*/
+	gtk_object_unref (GTK_OBJECT (filter_stream));
+
+	/* force a summary update - will only update from the new position, if it can */
+	camel_mbox_summary_update(mbox_folder->summary, seek);
+	return;
+
+fail:
+	camel_exception_setv (ex, 
+			      CAMEL_EXCEPTION_FOLDER_INSUFFICIENT_PERMISSION, /* FIXME: what code? */
+			      "Cannot append to mbox file: %s", strerror (errno));
+	if (filter_stream) {
+		/*camel_stream_close (filter_stream);*/
+		gtk_object_unref ((GtkObject *)filter_stream);
+	} else if (output_stream)
+		gtk_object_unref ((GtkObject *)output_stream);
 
-	camel_stream_flush (output_stream);
-	gtk_object_unref (GTK_OBJECT (output_stream));
+	/* make sure the file isn't munged by us */
+	if (seek != -1) {
+		int fd = open(mbox_folder->folder_file_path, O_WRONLY, 0600);
+		if (fd != -1) {
+			ftruncate(fd, st.st_size);
+			close(fd);
+		}
+	}
 }
 
-
-
-
 static GList *
 mbox_get_uid_list (CamelFolder *folder, CamelException *ex) 
 {
@@ -823,9 +788,9 @@ mbox_get_uid_list (CamelFolder *folder, CamelException *ex)
 	int i, count;
 
 	/* FIXME: how are these allocated strings ever free'd? */
-	count = camel_mbox_summary_message_count(mbox_folder->summary);
+	count = camel_folder_summary_count((CamelFolderSummary *)mbox_folder->summary);
 	for (i=0;i<count;i++) {
-		CamelMboxMessageInfo *info = camel_mbox_summary_index(mbox_folder->summary, i);
+		CamelMboxMessageInfo *info = (CamelMboxMessageInfo *)camel_folder_summary_index((CamelFolderSummary *)mbox_folder->summary, i);
 		uid_list = g_list_prepend(uid_list, g_strdup(info->info.uid));
 	}
 	
@@ -840,7 +805,7 @@ mbox_get_message_by_number (CamelFolder *folder, gint number, CamelException *ex
 
 	g_warning("YOUR CODE SHOULD NOT BE GETTING MESSAGES BY NUMBER, CHANGE IT");
 
-	info = camel_mbox_summary_index(mbox_folder->summary, number);
+	info = (CamelMboxMessageInfo *)camel_folder_summary_index((CamelFolderSummary *)mbox_folder->summary, number);
 	if (info == NULL) {
 		camel_exception_setv (ex, CAMEL_EXCEPTION_FOLDER_INVALID,
 				      "No such message %d in folder `%s'.",
@@ -855,10 +820,16 @@ mbox_get_message_by_number (CamelFolder *folder, gint number, CamelException *ex
 static void
 message_changed(CamelMimeMessage *m, int type, CamelMboxFolder *mf)
 {
+	CamelMessageInfo *info;
+
 	printf("Message changed: %s: %d\n", m->message_uid, type);
 	switch (type) {
 	case MESSAGE_FLAGS_CHANGED:
-		camel_mbox_summary_set_flags_by_uid(mf->summary, m->message_uid, m->flags);
+		info = camel_folder_summary_uid((CamelFolderSummary *)mf->summary, m->message_uid);
+		if (info)
+			info->flags = m->flags | CAMEL_MESSAGE_FOLDER_FLAGGED;
+		else
+			g_warning("Message changed event on message not in summary: %s", m->message_uid);
 		break;
 	default:
 		printf("Unhandled message change event: %d\n", type);
@@ -866,62 +837,80 @@ message_changed(CamelMimeMessage *m, int type, CamelMboxFolder *mf)
 	}
 }
 
-
 static CamelMimeMessage *
 mbox_get_message_by_uid (CamelFolder *folder, const gchar *uid, CamelException *ex)
 {
 	CamelMboxFolder *mbox_folder = CAMEL_MBOX_FOLDER (folder);
-	CamelStream *message_stream;
+	CamelStream *message_stream = NULL;
 	CamelMimeMessage *message = NULL;
-	CamelStore *parent_store;
 	CamelMboxMessageInfo *info;
-
-	/* get the parent store */
-	parent_store = camel_folder_get_parent_store (folder, ex);
-	if (camel_exception_get_id (ex)) {
-		return NULL;
-	}
+	CamelMimeParser *parser = NULL;
+	char *buffer;
+	int len;
 
 	/* get the message summary info */
-	info = camel_mbox_summary_uid(mbox_folder->summary, uid);
+	info = (CamelMboxMessageInfo *)camel_folder_summary_uid((CamelFolderSummary *)mbox_folder->summary, uid);
 
 	if (info == NULL) {
-		camel_exception_setv (ex, 
-				     CAMEL_EXCEPTION_FOLDER_INVALID_UID,
-				     "uid %s not found in the folder",
-				      uid);
-		return NULL;
+		errno = ENOENT;
+		goto fail;
 	}
 
 	/* if this has no content, its an error in the library */
 	g_assert(info->info.content);
+	g_assert(info->frompos != -1);
+
+	/* where we read from */
+	message_stream = camel_stream_fs_new_with_name (mbox_folder->folder_file_path, O_RDONLY, 0);
+	if (message_stream == NULL)
+		goto fail;
+
+	/* we use a parser to verify the message is correct, and in the correct position */
+	parser = camel_mime_parser_new();
+	camel_mime_parser_init_with_stream(parser, message_stream);
+	camel_mime_parser_scan_from(parser, TRUE);
+
+	camel_mime_parser_seek(parser, info->frompos, SEEK_SET);
+	if (camel_mime_parser_step(parser, &buffer, &len) != HSCAN_FROM) {
+		g_warning("File appears truncated");
+		goto fail;
+	}
+
+	if (camel_mime_parser_tell_start_from(parser) != info->frompos) {
+		g_warning("Summary doesn't match the folder contents!  eek!");
+		errno = EINVAL;
+		goto fail;
+	}
 
-	/* FIXME: more checks below */
-        /* create a stream bound to the message position/size */
-	message_stream = camel_stream_fs_new_with_name_and_bounds (mbox_folder->folder_file_path, O_RDONLY, 0,
-								   ((CamelMboxMessageContentInfo *)info->info.content)->pos,
-								   ((CamelMboxMessageContentInfo *)info->info.content)->endpos);
 	message = camel_mime_message_new();
-	if (camel_data_wrapper_construct_from_stream((CamelDataWrapper *)message, message_stream) == -1) {
-		gtk_object_unref((GtkObject *)message);
-		gtk_object_unref((GtkObject *)message_stream);
-		camel_exception_setv (ex, 
-				     CAMEL_EXCEPTION_FOLDER_INVALID_UID, /* FIXME: code */
-				      "Could not create message for uid %s: %s", uid, strerror(errno));
-		return NULL;
+	if (camel_mime_part_construct_from_parser((CamelMimePart *)message, parser) == -1) {
+		g_warning("Construction failed");
+		goto fail;
 	}
-	gtk_object_unref((GtkObject *)message_stream);
 
-	/* init other fields? */
+	/* we're constructed, finish setup and clean up */
 	message->folder = folder;
 	gtk_object_ref((GtkObject *)folder);
 	message->message_uid = g_strdup(uid);
 	message->flags = info->info.flags;
-	printf("%p flags = %x = %x\n", message, info->info.flags, message->flags);
-
 	gtk_signal_connect((GtkObject *)message, "message_changed", message_changed, folder);
 
+	gtk_object_unref((GtkObject *)parser);
+
 	return message;
+
+fail:
+	camel_exception_setv (ex, 
+			      CAMEL_EXCEPTION_FOLDER_INVALID_UID,
+			      "Cannot get message: %s", strerror(errno));
+	if (parser)
+		gtk_object_unref((GtkObject *)parser);
+	if (message_stream)
+		gtk_object_unref((GtkObject *)message_stream);
+	if (message)
+		gtk_object_unref((GtkObject *)message);
+
+	return NULL;
 }
 
 /* get message info for a range of messages */
@@ -931,10 +920,10 @@ GPtrArray *summary_get_message_info (CamelFolder *folder, int first, int count)
 	int i, maxcount;
 	CamelMboxFolder *mbox_folder = (CamelMboxFolder *)folder;
 
-        maxcount = camel_mbox_summary_message_count(mbox_folder->summary);
+        maxcount = camel_folder_summary_count((CamelFolderSummary *)mbox_folder->summary);
 	maxcount = MIN(count, maxcount);
 	for (i=first;i<maxcount;i++)
-		g_ptr_array_add(array, g_ptr_array_index(mbox_folder->summary->messages, i));
+		g_ptr_array_add(array, camel_folder_summary_index((CamelFolderSummary *)mbox_folder->summary, i));
 
 	return array;
 }
@@ -945,7 +934,7 @@ mbox_summary_get_by_uid(CamelFolder *f, const char *uid)
 {
 	CamelMboxFolder *mbox_folder = (CamelMboxFolder *)f;
 
-	return (CamelMessageInfo *)camel_mbox_summary_uid(mbox_folder->summary, uid);
+	return camel_folder_summary_uid((CamelFolderSummary *)mbox_folder->summary, uid);
 }
 
 static GList *
@@ -959,7 +948,8 @@ mbox_search_by_expression(CamelFolder *folder, const char *expression, CamelExce
 
 	camel_folder_search_set_folder(mbox_folder->search, folder);
 	if (mbox_folder->summary)
-		camel_folder_search_set_summary(mbox_folder->search, mbox_folder->summary->messages);
+		/* FIXME: dont access summary array directly? */
+		camel_folder_search_set_summary(mbox_folder->search, ((CamelFolderSummary *)mbox_folder->summary)->messages);
 	camel_folder_search_set_body_index(mbox_folder->search, mbox_folder->index);
 
 	return camel_folder_search_execute_expression(mbox_folder->search, expression, ex);
diff --git a/camel/providers/mbox/camel-mbox-summary.c b/camel/providers/mbox/camel-mbox-summary.c
index fedd9f6411..1fbbbad075 100644
--- a/camel/providers/mbox/camel-mbox-summary.c
+++ b/camel/providers/mbox/camel-mbox-summary.c
@@ -3,1502 +3,700 @@
  *
  *  Authors: Michael Zucchi <notzed@helixcode.com>
  *
- *  This program is free software; you can redistribute it and/or
- *  modify it under the terms of the GNU Library General Public License
- *  as published by the Free Software Foundation; either version 2 of
- *  the License, or (at your option) any later version.
+ *  This program is free software; you can redistribute it and/or 
+ *  modify it under the terms of the GNU General Public License as 
+ *  published by the Free Software Foundation; either version 2 of the
+ *  License, or (at your option) any later version.
  *
  *  This program is distributed in the hope that it will be useful,
  *  but WITHOUT ANY WARRANTY; without even the implied warranty of
  *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- *  GNU Library General Public License for more details.
+ *  GNU General Public License for more details.
  *
- *  You should have received a copy of the GNU Library General Public
- *  License along with this program; if not, write to the Free Software
- *  Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+ *  USA
  */
 
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <fcntl.h>
-#include <unistd.h>
-#include <sys/uio.h>
-
-#include <stdio.h>
-#include <string.h>
-
-#include <gtk/gtk.h>
-
-#include <camel/camel-mime-message.h>
-
-#include <camel/camel-mime-parser.h>
-#include <camel/camel-mime-filter.h>
-#include <camel/camel-mime-filter-basic.h>
-#include <camel/camel-mime-filter-charset.h>
-#include <camel/camel-mime-filter-index.h>
-
-#include <camel/camel-mime-utils.h>
-
 #include "camel-mbox-summary.h"
+#include <camel/camel-mime-message.h>
 
+#include <sys/stat.h>
+#include <sys/uio.h>
+#include <unistd.h>
 #include <errno.h>
-#include <ctype.h>
-#include <netinet/in.h>
+#include <string.h>
+#include <stdlib.h>
 
+#define io(x)
 #define d(x)
 
-#define CAMEL_MBOX_SUMMARY_VERSION 3
-
-static int safe_write(int fd, char *buffer, size_t towrite);
-static void camel_mbox_summary_add(CamelMboxSummary *s, CamelMboxMessageInfo *info);
-
-/*
-  Disk file format?
-
-  message uid
-message-block
-  date:
-  date received?
-
-  subject: (unicode encoded)
-  from: (unicode encoded)
-  to: (unicode)
+#define CAMEL_MBOX_SUMMARY_VERSION (0x1000)
 
-  content-block
-
-content-block
-  content-type: ; params;
-  content-id:
-  content-description:
-  content-transfer-encoding:
-  message-start:
-  header-size:
-  body-size:
-
-  message-block
-  multipart-block
-
- */
-
-/* pah, i dont care, its almost no code and it works, dont need a glist */
-struct _node {
-	struct _node *next;
+struct _CamelMboxSummaryPrivate {
 };
 
-static struct _node *
-my_list_append(struct _node **list, struct _node *n)
-{
-	struct _node *ln = (struct _node *)list;
-	while (ln->next)
-		ln = ln->next;
-	n->next = 0;
-	ln->next = n;
-	return n;
-}
+#define _PRIVATE(o) (((CamelMboxSummary *)(o))->priv)
 
-static int
-my_list_size(struct _node **list)
-{
-	int len = 0;
-	struct _node *ln = (struct _node *)list;
-	while (ln->next) {
-		ln = ln->next;
-		len++;
-	}
-	return len;
-}
-
-/* low-level io functions */
-static int
-encode_int (FILE *out, gint32 value)
-{
-	int i;
+static int summary_header_load(CamelFolderSummary *, FILE *);
+static int summary_header_save(CamelFolderSummary *, FILE *);
 
-	for (i=28;i>0;i-=7) {
-		if (value >= (1<<i)) {
-			unsigned int c = (value>>i) & 0x7f;
-			if (fputc(c, out) == -1)
-				return -1;
-		}
-	}
-	return fputc(value | 0x80, out);
-}
+static CamelMessageInfo * message_info_new(CamelFolderSummary *, struct _header_raw *);
+static CamelMessageInfo * message_info_new_from_parser(CamelFolderSummary *, CamelMimeParser *);
+static CamelMessageInfo * message_info_load(CamelFolderSummary *, FILE *);
+static int		  message_info_save(CamelFolderSummary *, FILE *, CamelMessageInfo *);
+/*static void		  message_info_free(CamelFolderSummary *, CamelMessageInfo *);*/
 
-static gint32
-decode_int (FILE *in)
-{
-        gint32 value=0, v;
+static void camel_mbox_summary_class_init (CamelMboxSummaryClass *klass);
+static void camel_mbox_summary_init       (CamelMboxSummary *obj);
+static void camel_mbox_summary_finalise   (GtkObject *obj);
 
-        /* until we get the last byte, keep decoding 7 bits at a time */
-        while ( ((v = fgetc(in)) & 0x80) == 0 && v!=EOF) {
-                value |= v;
-                value <<= 7;
-        }
-        value |= (v&0x7f);
-        return value;
-}
+static CamelFolderSummaryClass *camel_mbox_summary_parent;
 
-static int
-encode_fixed_int (FILE *out, gint32 value)
-{
-	guint32 save;
-
-	save = htonl(value);
-	return fwrite(&save, sizeof(save), 1, out);
-}
-
-static gint32
-decode_fixed_int (FILE *out)
-{
-	guint32 save;
-
-	if (fread(&save, sizeof(save), 1, out) != -1) {
-		return ntohl(save);
-	} else {
-		return -1;
-	}
-}
-
-/* should be sorted, for binary search */
-/* This is a tokenisation mechanism for strings written to the
-   summary - to save space.
-   This list can have at most 31 words. */
-static char * tokens[] = {
-	"7bit",
-	"8bit",
-	"alternative",
-	"application",
-	"base64",
-	"boundary",
-	"charset",
-	"filename",
-	"html",
-	"image",
-	"iso-8859-1",
-	"iso-8859-8",
-	"message",
-	"mixed",
-	"multipart",
-	"name",
-	"octet-stream",
-	"parallel",
-	"plain",
-	"quoted-printable",
-	"rfc822",
-	"text",
-	"us-ascii",		/* 23 words */
+enum SIGNALS {
+	LAST_SIGNAL
 };
 
-#define tokens_len (sizeof(tokens)/sizeof(tokens[0]))
-
-/* baiscally ...
-    0 = null
-    1-tokens_len == tokens[id-1]
-    >=32 string, length = n-32
-*/
+static guint signals[LAST_SIGNAL] = { 0 };
 
-static int
-encode_string (FILE *out, char *str)
-{
-	if (str == NULL) {
-		return encode_int(out, 0);
-	} else {
-		int len = strlen(str);
-		int i, token=-1;
-
-		if (len <= 16) {
-			char lower[32];
-
-			for (i=0;i<len;i++)
-				lower[i] = tolower(str[i]);
-			lower[i] = 0;
-			for (i=0;i<tokens_len;i++) {
-				if (!strcmp(tokens[i], lower)) {
-					token = i;
-					break;
-				}
-			}
-		}
-		if (token != -1) {
-			return encode_int(out, token+1);
-		} else {
-			if (encode_int(out, len+32) == -1)
-				return -1;
-			return fwrite(str, len, 1, out);
-		}
-	}
-	return 0;
-}
-
-static char *
-decode_string (FILE *in)
+guint
+camel_mbox_summary_get_type (void)
 {
-	char *ret;
-	int len;
+	static guint type = 0;
 	
-	len = decode_int(in);
-
-	if (len<32) {
-		if (len <= 0) {
-			ret = NULL;
-		} else if (len<= tokens_len) {
-			ret = g_strdup(tokens[len-1]);
-		} else {
-			g_warning("Invalid token encountered: %d", len);
-			ret = NULL;
-		}
-	} else if (len > 10240) {
-		g_warning("Got broken string header length: %d bytes", len);
-		ret = NULL;
-	} else {
-		len -= 32;
-		ret = g_malloc(len+1);
-		if (fread(ret, len, 1, in) == -1) {
-			g_free(ret);
-			return NULL;
-		}
-		ret[len]=0;
-	}
-
-	return ret;
-}
-
-
-
-/* allocation functions */
-
-static void
-body_part_dump(CamelMboxMessageContentInfo *bs, int depth)
-{
-	CamelMboxMessageContentInfo *c;
-	char *prefix;
-
-	if (bs == NULL)
-		return;
-
-	prefix = alloca(depth*2+1);
-	memset(prefix, ' ', depth*2);
-	prefix[depth*2]=0;
-	printf("%scontent-range: %d %d %d\n", prefix, (int)bs->pos, (int)bs->bodypos, (int)bs->endpos);
-	printf("%scontent-type: %s/%s\n", prefix, bs->info.type?bs->info.type->type:"?", bs->info.type?bs->info.type->subtype:"?");
-	printf("%scontent-id: %s\n", prefix, bs->info.id);
-	printf("%scontent-description: %s\n", prefix, bs->info.description);
-	printf("%scontent-transfer-encoding: %s\n", prefix, bs->info.encoding);
-	c = (CamelMboxMessageContentInfo *)bs->info.childs;
-	while (c) {
-		printf("%s -- \n", prefix);
-		body_part_dump(c, depth+1);
-		c = (CamelMboxMessageContentInfo *)c->info.next;
+	if (!type) {
+		GtkTypeInfo type_info = {
+			"CamelMboxSummary",
+			sizeof (CamelMboxSummary),
+			sizeof (CamelMboxSummaryClass),
+			(GtkClassInitFunc) camel_mbox_summary_class_init,
+			(GtkObjectInitFunc) camel_mbox_summary_init,
+			(GtkArgSetFunc) NULL,
+			(GtkArgGetFunc) NULL
+		};
+		
+		type = gtk_type_unique (camel_folder_summary_get_type (), &type_info);
 	}
+	
+	return type;
 }
 
 static void
-message_struct_dump(CamelMboxMessageInfo *ms)
-{
-	char *tmp;
-
-	if (ms == NULL) {
-		printf("Empty message?\n");
-		return;
-	}
-
-	printf("Subject: %s\n", ms->info.subject);
-	printf("From: %s\n", ms->info.from);
-	printf("To: %s\n", ms->info.to);
-	tmp = header_format_date(ms->info.date_sent, 0);
-	printf("Date: %s\n", tmp);
-	g_free(tmp);
-	tmp = header_format_date(ms->info.date_received, 0);
-	printf("Date-Received: %s\n", tmp);
-	g_free(tmp);
-	printf("UID: %08x-%04x\n", atoi(ms->info.uid), ms->info.flags);
-	printf(" -- content ->\n");
-	body_part_dump((CamelMboxMessageContentInfo *)ms->info.content, 1);
-}
-
-static CamelMboxMessageContentInfo *
-body_part_new(CamelMimeParser *mp, CamelMboxMessageContentInfo *parent, int start, int body)
+camel_mbox_summary_class_init (CamelMboxSummaryClass *klass)
 {
-	CamelMboxMessageContentInfo *bs;
-
-	bs = g_malloc0(sizeof(*bs));
-
-	bs->info.parent = (CamelMessageContentInfo *)parent;
-
-	bs->info.type = camel_mime_parser_content_type(mp);
-	header_content_type_ref(bs->info.type);
+	GtkObjectClass *object_class = (GtkObjectClass *) klass;
+	CamelFolderSummaryClass *sklass = (CamelFolderSummaryClass *) klass;
+	
+	camel_mbox_summary_parent = gtk_type_class (camel_folder_summary_get_type ());
 
-	bs->info.id = header_msgid_decode(camel_mime_parser_header(mp, "content-id", NULL));
-	bs->info.description = header_decode_string(camel_mime_parser_header(mp, "content-description", NULL));
-	bs->info.encoding = header_content_encoding_decode(camel_mime_parser_header(mp, "content-transfer-encoding", NULL));
+	object_class->finalize = camel_mbox_summary_finalise;
 
-	/* not sure what to set here? */
-	bs->pos = start;
-	bs->bodypos = body;
-	bs->endpos = -1;
+	sklass->summary_header_load = summary_header_load;
+	sklass->summary_header_save = summary_header_save;
 
-	if (parent)
-		my_list_append((struct _node **)&parent->info.childs, (struct _node *)bs);
+	sklass->message_info_new  = message_info_new;
+	sklass->message_info_new_from_parser = message_info_new_from_parser;
+	sklass->message_info_load = message_info_load;
+	sklass->message_info_save = message_info_save;
+	/*sklass->message_info_free = message_info_free;*/
 
-	return bs;
+	gtk_object_class_add_signals (object_class, signals, LAST_SIGNAL);
 }
 
-static CamelMboxMessageInfo *
-message_struct_new(CamelMimeParser *mp, CamelMboxMessageContentInfo *parent, int start, int body, off_t xev_offset)
+static void
+camel_mbox_summary_init (CamelMboxSummary *obj)
 {
-	CamelMboxMessageInfo *ms;
-	struct _header_address *addr;
-	const char *text;
-
-	ms = g_malloc0(sizeof(*ms));
-
-	/* FIXME: what about cc, sender vs from? */
-	ms->info.subject = header_decode_string(camel_mime_parser_header(mp, "subject", NULL));
-	text = camel_mime_parser_header(mp, "from", NULL);
-	addr = header_address_decode(text);
-	if (addr) {
-		ms->info.from = header_address_list_format(addr);
-		header_address_list_clear(&addr);
-	} else {
-		ms->info.from = g_strdup(text);
-	}
-	text = camel_mime_parser_header(mp, "to", NULL);
-	addr = header_address_decode(text);
-	if (addr) {
-		ms->info.to = header_address_list_format(addr);
-		header_address_list_clear(&addr);
-	} else {
-		ms->info.to = g_strdup(text);
-	}
+	struct _CamelMboxSummaryPrivate *p;
+	struct _CamelFolderSummary *s = (CamelFolderSummary *)obj;
 
-	ms->info.date_sent = header_decode_date(camel_mime_parser_header(mp, "date", NULL), NULL);
-	ms->info.date_received = 0;
+	p = _PRIVATE(obj) = g_malloc0(sizeof(*p));
 
-	ms->info.content = (CamelMessageContentInfo *)body_part_new(mp, parent, start, body);
-	ms->xev_offset = xev_offset;
+	/* subclasses need to set the right instance data sizes */
+	s->message_info_size = sizeof(CamelMboxMessageInfo);
+	s->content_info_size = sizeof(CamelMboxMessageContentInfo);
 
-	return ms;
+	/* and a unique file version */
+	s->version = CAMEL_MBOX_SUMMARY_VERSION;
 }
 
 static void
-body_part_free(CamelMboxMessageContentInfo *bs)
+camel_mbox_summary_finalise (GtkObject *obj)
 {
-	CamelMboxMessageContentInfo *c, *cn;
-
-	c = (CamelMboxMessageContentInfo *)bs->info.childs;
-	while (c) {
-		cn = (CamelMboxMessageContentInfo *)c->info.next;
-		body_part_free(c);
-		c = cn;
-	}
-	g_free(bs->info.id);
-	g_free(bs->info.description);
-	g_free(bs->info.encoding);
-	header_content_type_unref(bs->info.type);
-	g_free(bs);
+	((GtkObjectClass *)(camel_mbox_summary_parent))->finalize((GtkObject *)obj);
 }
 
-static void
-message_struct_free(CamelMboxMessageInfo *ms)
+/**
+ * camel_mbox_summary_new:
+ *
+ * Create a new CamelMboxSummary object.
+ * 
+ * Return value: A new CamelMboxSummary widget.
+ **/
+CamelMboxSummary *
+camel_mbox_summary_new (const char *filename, const char *mbox_name, ibex *index)
 {
-	g_free(ms->info.subject);
-	g_free(ms->info.to);
-	g_free(ms->info.from);
-	body_part_free((CamelMboxMessageContentInfo *)ms->info.content);
-	g_free(ms);
+	CamelMboxSummary *new = CAMEL_MBOX_SUMMARY ( gtk_type_new (camel_mbox_summary_get_type ()));
+	if (new) {
+		/* ?? */
+		camel_folder_summary_set_build_content((CamelFolderSummary *)new, TRUE);
+		camel_folder_summary_set_filename((CamelFolderSummary *)new, filename);
+		new->folder_path = g_strdup(mbox_name);
+		new->index = index;
+	}
+	return new;
 }
 
 
-/* IO functions */
-static CamelMboxMessageContentInfo *
-body_part_load(FILE *in)
+static int summary_header_load(CamelFolderSummary *s, FILE *in)
 {
-	CamelMboxMessageContentInfo *bs = NULL, *c;
-	struct _header_content_type *ct;
-	char *type;
-	char *subtype;
-	int i, count;
-
-	d(printf("got content-block\n"));
-	bs = g_malloc0(sizeof(*bs));
-	bs->pos = decode_int(in);
-	bs->bodypos = bs->pos + decode_int(in);
-	bs->endpos = bs->pos + decode_int(in);
-
-	/* do content type */
-	d(printf("got content-type\n"));
-	type = decode_string(in);
-	subtype = decode_string(in);
-
-	ct = header_content_type_new(type, subtype);
-	bs->info.type = ct;
-	count = decode_int(in);
-	d(printf("getting %d params\n", count));
-	for (i=0;i<count;i++) {
-		char *name = decode_string(in);
-		char *value = decode_string(in);
-		
-		d(printf(" %s = \"%s\"\n", name, value));
-		
-		header_content_type_set_param(ct, name, value);
-		/* FIXME: do this so we dont have to double alloc/free */
-		g_free(name);
-		g_free(value);
-	}
+	CamelMboxSummary *mbs = (CamelMboxSummary *)s;
 
-	d(printf("got content-id\n"));
-	bs->info.id = decode_string(in);
-	d(printf("got content-description\n"));
-	bs->info.description = decode_string(in);
-	d(printf("got content-encoding\n"));
-	bs->info.encoding = decode_string(in);
-
-	count = decode_int(in);
-	d(printf("got children, %d\n", count));
-	for (i=0;i<count;i++) {
-		c = body_part_load(in);
-		if (c) {
-			my_list_append((struct _node **)&bs->info.childs, (struct _node *)c);
-			c->info.parent = (CamelMessageContentInfo *)bs;
-		} else {
-			printf("Cannot load child\n");
-		}
-	}
+	if (((CamelFolderSummaryClass *)camel_mbox_summary_parent)->summary_header_load(s, in) == -1)
+		return -1;
 
-	return bs;
+	return camel_folder_summary_decode_uint32(in, &mbs->folder_size);
 }
 
-static int
-body_part_save(FILE *out, CamelMboxMessageContentInfo *bs)
+static int summary_header_save(CamelFolderSummary *s, FILE *out)
 {
-	CamelMboxMessageContentInfo *c, *cn;
-	struct _header_content_type *ct;
-	struct _header_param *hp;
-
-	encode_int(out, bs->pos);
-	encode_int(out, bs->bodypos - bs->pos);
-	encode_int(out, bs->endpos - bs->pos);
-
-	ct = bs->info.type;
-	if (ct) {
-		encode_string(out, ct->type);
-		encode_string(out, ct->subtype);
-		encode_int(out, my_list_size((struct _node **)&ct->params));
-		hp = ct->params;
-		while (hp) {
-			encode_string(out, hp->name);
-			encode_string(out, hp->value);
-			hp = hp->next;
-		}
-	} else {
-		encode_string(out, NULL);
-		encode_string(out, NULL);
-		encode_int(out, 0);
-	}
-	encode_string(out, bs->info.id);
-	encode_string(out, bs->info.description);
-	encode_string(out, bs->info.encoding);
+	CamelMboxSummary *mbs = (CamelMboxSummary *)s;
 
-	encode_int(out, my_list_size((struct _node **)&bs->info.childs));
-
-	c = (CamelMboxMessageContentInfo *)bs->info.childs;
-	while (c) {
-		cn = (CamelMboxMessageContentInfo *)c->info.next;
-		body_part_save(out, c);
-		c = cn;
-	}
+	if (((CamelFolderSummaryClass *)camel_mbox_summary_parent)->summary_header_save(s, out) == -1)
+		return -1;
 
-	return 0;
+	return camel_folder_summary_encode_uint32(out, mbs->folder_size);
 }
 
-static CamelMboxMessageInfo *
-message_struct_load(FILE *in)
-{
-	CamelMboxMessageInfo *ms;
-
-	ms = g_malloc0(sizeof(*ms));
-
-	ms->info.uid = g_strdup_printf("%u", decode_int(in));
-	ms->info.flags = decode_int(in);
-	ms->info.date_sent = decode_int(in);
-	ms->info.date_received = decode_int(in);
-	ms->xev_offset = decode_int(in);
-	ms->info.subject = decode_string(in);
-	ms->info.from = decode_string(in);
-	ms->info.to = decode_string(in);
-	ms->info.content = (CamelMessageContentInfo *)body_part_load(in);
+static int
+header_evolution_decode(const char *in, guint32 *uid, guint32 *flags)
+{
+        char *header;
+        if (in
+            && (header = header_token_decode(in))) {
+                if (strlen(header) == strlen("00000000-0000")
+                    && sscanf(header, "%08x-%04x", uid, flags) == 2) {
+                        g_free(header);
+                        return *uid;
+                }
+                g_free(header);
+        }
 
-	return ms;
+        return -1;
 }
 
-static int
-message_struct_save(FILE *out, CamelMboxMessageInfo *ms)
+static char *
+header_evolution_encode(guint32 uid, guint32 flags)
 {
-	encode_int(out, strtoul(ms->info.uid, NULL, 10));
-	encode_int(out, ms->info.flags);
-	encode_int(out, ms->info.date_sent);
-	encode_int(out, ms->info.date_received);
-	encode_int(out, ms->xev_offset);
-	encode_string(out, ms->info.subject);
-	encode_string(out, ms->info.from);
-	encode_string(out, ms->info.to);
-	body_part_save(out, (CamelMboxMessageContentInfo *)ms->info.content);
-
-	return 0;
+	return g_strdup_printf("%08x-%04x", uid, flags & 0xffff);
 }
 
-static unsigned int
-header_evolution_decode(const char *in, unsigned int *uid, unsigned int *flags)
+static CamelMessageInfo * message_info_new(CamelFolderSummary *s, struct _header_raw *h)
 {
-	char *header;
-	if (in
-	    && (header = header_token_decode(in))) {
-		if (strlen(header) == strlen("00000000-0000")
-		    && sscanf(header, "%08x-%04x", uid, flags) == 2) {
-			g_free(header);
-			return *uid;
-		}
-		g_free(header);
-	}
+	CamelMessageInfo *mi;
 
-	return ~0;
-}
+	mi = ((CamelFolderSummaryClass *)camel_mbox_summary_parent)->message_info_new(s, h);
+	if (mi) {
+		const char *xev;
+		guint32 uid, flags;
+		CamelMboxMessageInfo *mbi = (CamelMboxMessageInfo *)mi;
 
-static int
-safe_write(int fd, char *buffer, size_t towrite)
-{
-	size_t donelen;
-	size_t len;
-
-	donelen = 0;
-	while (donelen < towrite) {
-		len = write(fd, buffer + donelen, towrite - donelen);
-		if (len == -1) {
-			if (errno == EINTR || errno == EAGAIN)
-				continue;
-			return -1;
+		xev = header_raw_find(&h, "X-Evolution", NULL);
+		if (xev
+		    && header_evolution_decode(xev, &uid, &flags) != -1) {
+			g_free(mi->uid);
+			mi->uid = g_strdup_printf("%u", uid);
+			mi->flags = flags;
+		} else {
+			/* to indicate it has no xev header? */
+			mi->flags |= CAMEL_MESSAGE_FOLDER_FLAGGED|CAMEL_MESSAGE_FOLDER_NOXEV;
+			mi->uid = g_strdup_printf("%u", camel_folder_summary_next_uid(s));
 		}
-		donelen += len;
+		mbi->frompos = -1;
 	}
-	return donelen;
+	return mi;
 }
 
-static int
-header_write(int fd, struct _header_raw *header, unsigned int uid, unsigned int flags)
+static CamelMessageInfo * message_info_new_from_parser(CamelFolderSummary *s, CamelMimeParser *mp)
 {
-	struct iovec iv[4];
-	int outlen = 0;
+	CamelMessageInfo *mi;
+	CamelMboxSummary *mbs = (CamelMboxSummary *)s;
 
-	iv[1].iov_base = ":";
-	iv[1].iov_len = 1;
-	iv[3].iov_base = "\n";
-	iv[3].iov_len = 1;
+	mi = ((CamelFolderSummaryClass *)camel_mbox_summary_parent)->message_info_new_from_parser(s, mp);
+	if (mi) {
+		CamelMboxMessageInfo *mbi = (CamelMboxMessageInfo *)mi;
 
-	while (header) {
-		if (strcasecmp(header->name, "x-evolution")) {
-			int len;
+		mbi->frompos = camel_mime_parser_tell_start_from(mp);
 
-			iv[0].iov_base = header->name;
-			iv[0].iov_len = strlen(header->name);
-			iv[2].iov_base = header->value;
-			iv[2].iov_len = strlen(header->value);
-
-			do {
-				len = writev(fd, iv, 4);
-			} while (len == -1 && errno == EINTR);
-
-			if (len == -1)
-				return -1;
-			outlen += len;
+		/* do we want to index this message as we add it, as well? */
+		if (mbs->index_force
+		    || (mi->flags & CAMEL_MESSAGE_FOLDER_FLAGGED) != 0
+		    || !ibex_contains_name(mbs->index, mi->uid)) {
+			camel_folder_summary_set_index(s, mbs->index);
+		} else {
+			camel_folder_summary_set_index(s, NULL);
 		}
-		header = header->next;
 	}
-
-	return outlen;
+	return mi;
 }
 
-/* returns -1 on error, else number of bytes written */
-int
-camel_mbox_summary_copy_block(int fromfd, int tofd, off_t readpos, size_t bytes)
+static CamelMessageInfo * message_info_load(CamelFolderSummary *s, FILE *in)
 {
-	char buffer[4096];
-	int written = 0;
-	off_t pos, newpos;
-
-	pos = lseek(fromfd, 0, SEEK_CUR);
-	if (pos == -1)
-		return -1;
+	CamelMessageInfo *mi;
 
-	newpos = lseek(fromfd, readpos, SEEK_SET);
-	if (newpos == -1 || newpos != readpos)
-		goto error;
-
-	d(printf("oldpos = %d;  copying %d from %d\n", (int)pos, (int)bytes, (int)readpos));
+	io(printf("loading mbox message info\n"));
 
-	while (bytes>0) {
-		int toread, towrite, donelen;
+	mi = ((CamelFolderSummaryClass *)camel_mbox_summary_parent)->message_info_load(s, in);
+	if (mi) {
+		CamelMboxMessageInfo *mbi = (CamelMboxMessageInfo *)mi;
 
-		toread = bytes;
-		if (bytes>4096)
-			toread = 4096;
-		else
-			toread = bytes;
-	reread:
-		towrite = read(fromfd, buffer, toread);
-		if (towrite == -1) {
-			if (errno == EINTR || errno == EAGAIN)
-				goto reread;
-			goto error;
-		}
-
-		/* check for 'end of file' */
-		if (towrite == 0)
-			break;
-
-		if ( (donelen = safe_write(tofd, buffer, towrite)) == -1)
-			goto error;
-
-		written += donelen;
-		bytes -= donelen;
+		camel_folder_summary_decode_uint32(in, &mbi->frompos);
 	}
+	return mi;
+}
 
-	d(printf("written %d bytes\n", written));
+static int		  message_info_save(CamelFolderSummary *s, FILE *out, CamelMessageInfo *mi)
+{
+	CamelMboxMessageInfo *mbi = (CamelMboxMessageInfo *)mi;
 
-	newpos = lseek(fromfd, pos, SEEK_SET);
-	if (newpos == -1 || newpos != pos);
-		return -1;
+	io(printf("saving mbox message info\n"));
 
-	return written;
+	((CamelFolderSummaryClass *)camel_mbox_summary_parent)->message_info_save(s, out, mi);
 
-error:
-	lseek(fromfd, pos, SEEK_SET);
-	return -1;
+	return camel_folder_summary_encode_uint32(out, mbi->frompos);
 }
 
-#define SAVEIT
-
-/* TODO: Allow to sync with current summary info, without over-writing flags if they
-   already exist there */
-/* TODO: Lazy sync, make this read-only, and dont write out xev headers unless we need
-   to? */
-static int index_folder(CamelMboxSummary *s, int startoffset)
+static int
+summary_rebuild(CamelMboxSummary *mbs, off_t offset)
 {
 	CamelMimeParser *mp;
 	int fd;
-	int fdout;
-	int state;
-
-	int toplevel = FALSE;
-	const char *xev;
-	char *data;
-	int datalen;
-
-	int enc_id=-1;
-	int chr_id=-1;
-	int idx_id=-1;
-	struct _header_content_type *ct;
-	int doindex=FALSE;
-	CamelMimeFilterCharset *mfc = NULL;
-	CamelMimeFilterIndex *mfi = NULL;
-	CamelMimeFilterBasic *mf64 = NULL, *mfqp = NULL;
-
-	CamelMboxMessageContentInfo *body = NULL, *parent = NULL;
-	CamelMboxMessageInfo *message = NULL;
-
-	int from_end = 0;	/* start of message */
-	int from = 0;		/* start of headers */
-	int last_write = 0;	/* last written position */
-	int eof;
-	int write_offset = 0;	/* how much does the dest differ from the source pos */
-	int old_offset = 0;
-
-	guint32 newuid, newflags;
-	off_t xevoffset = -1;
-
-	char *tmpname;
-
-	printf("indexing %s (%s) from %d\n", s->folder_path, s->summary_path, startoffset);
-
-	fd = open(s->folder_path, O_RDONLY);
-	if (fd==-1) {
-		perror("Can't open folder");
-		return -1;
-	}
+	int ok = 0;
 
-	tmpname = g_strdup_printf("%s.tmp", s->folder_path);
-
-	fdout = open(tmpname, O_WRONLY|O_CREAT|O_TRUNC, 0600);
-	if (fdout==-1) {
-		perror("Can't open output");
-		g_free(tmpname);
-		return -1;
-	}
+	printf("(re)Building summary from %d (%s)\n", (int)offset, mbs->folder_path);
 
+	fd = open(mbs->folder_path, O_RDONLY);
 	mp = camel_mime_parser_new();
 	camel_mime_parser_init_with_fd(mp, fd);
 	camel_mime_parser_scan_from(mp, TRUE);
-
-	/* FIXME: cleaner fail code */
-	if (startoffset > 0) {
-		if (camel_mime_parser_seek(mp, startoffset, SEEK_SET) != startoffset) {
-			g_free(tmpname);
-			gtk_object_unref((GtkObject *)mp);
-			return -1;
-		}
-	}
-
-	mfi = camel_mime_filter_index_new_ibex(s->index);
-
-	while ( (state = camel_mime_parser_step(mp, &data, &datalen)) != HSCAN_EOF ) {
-		switch(state) {
-		case HSCAN_FROM: /* starting a new message content */
-			/* save the current position */
-			d(printf("from = %d\n", (int)camel_mime_parser_tell(mp)));
-			toplevel = FALSE;
-			from = camel_mime_parser_tell(mp);
-			break;
-
-		case HSCAN_FROM_END:
-			d(printf("from-end = %d\n", (int)camel_mime_parser_tell(mp)));
-			d(printf("message from %d to %d\n", from_end, (int)camel_mime_parser_tell(mp)));
-			from_end = camel_mime_parser_tell(mp);
-			break;
-
-		case HSCAN_MESSAGE:
-		case HSCAN_MULTIPART:
-		case HSCAN_HEADER: /* starting a new header */
-			newuid=~0;
-			newflags=0;
-			if (!toplevel) {
-				char name[32];
-				unsigned int olduid, oldflags;
-				int headerlen;
-				int docopy = FALSE;
-
-				/* check for X-Evolution header ... if its there, nothing to do (skip content) */
-				xev = camel_mime_parser_header(mp, "x-evolution", (int *)&xevoffset);
-				if (xev) {
-					d(printf("An x-evolution header exists at: %d = %s\n", xevoffset + write_offset, xev));
-					xevoffset = xevoffset + write_offset;
-					if (header_evolution_decode(xev, &olduid, &oldflags) != ~0) {
-						d(printf(" uid = %d = %x\n", olduid, olduid));
-						newuid = olduid;
-						newflags = oldflags;
-#if 0
-						while (camel_mime_parser_step(mp, &data, &datalen) != HSCAN_FROM_END)
-							;
-						break;
-#endif
-					} else {
-						printf("Invalid xev header?  I need to write out a new one ...\n");
-					}
-				}
-
-				toplevel = TRUE;
-
-				/* assign a new uid for this message */
-				if (newuid == ~0) {
-					newuid = s->nextuid++;
-					docopy = TRUE;
-				} else {
-					/* make sure we account for this uid when assigning uid's */
-					/* this really needs a pre-scan pass ... *sigh* */
-					camel_mbox_summary_set_uid(s, newuid);
-				}
-
-				/* setup index name for this uid */
-				sprintf(name, "%u", newuid);
-				camel_mime_filter_index_set_name(mfi, name);
-				/* remove all references to this name from the index */
-				if (s->index)
-					ibex_unindex(s->index, name);
-
-				d(printf("Message content starts at %d\n", camel_mime_parser_tell(mp)));
-				
-				if (docopy) {
-					/* now, copy over bits of mbox from last write, and insert the X-Evolution header (at the top of headers) */
-					/* if we already have a valid x-evolution header, use that, dont need to copy */
-					camel_mbox_summary_copy_block(fd, fdout, last_write, from-last_write);
-					last_write = from;
-
-					headerlen = header_write(fdout, camel_mime_parser_headers_raw(mp), newuid, 0);
-					sprintf(name, "X-Evolution: %08x-%04x\n\n", newuid, 0);
-					safe_write(fdout, name, strlen(name));
-					d(printf("new X-Evolution at %d\n", headerlen + from + write_offset));
-					xevoffset = headerlen + from + write_offset;
-					old_offset = write_offset;
-
-					write_offset += (headerlen - (camel_mime_parser_tell(mp)-from)) + strlen(name);
-					last_write = camel_mime_parser_tell(mp);
-				}
-			} else {
-				old_offset = write_offset;
-			}
-
-			/* we only care about the rest for actual content parts */
-			/* TODO: Cleanup, this is a huge mess */
-			if (state != HSCAN_HEADER) {
-				if (message == NULL) {
-					message = message_struct_new(mp, parent, camel_mime_parser_tell_start_headers(mp)+old_offset, camel_mime_parser_tell(mp)+write_offset, xevoffset);
-					parent = (CamelMboxMessageContentInfo *)message->info.content;
-					if (newuid != ~0) {
-						message->info.uid = g_strdup_printf("%u", newuid);
-						message->info.flags = newflags;
-					} else {
-						g_warning("This shouldn't happen?");
-					}
-				} else {
-					parent = body_part_new(mp, parent, camel_mime_parser_tell_start_headers(mp)+old_offset, camel_mime_parser_tell(mp)+write_offset);
-				}
-				break;
-			}
-
-			if (message == NULL) {
-				message = message_struct_new(mp, parent, camel_mime_parser_tell_start_headers(mp)+old_offset, camel_mime_parser_tell(mp)+write_offset, xevoffset);
-				body = (CamelMboxMessageContentInfo *)message->info.content;
-				if (newuid != ~0) {
-					message->info.uid = g_strdup_printf("%u", newuid);
-					message->info.flags = newflags;
-				} else {
-					g_warning("This shouldn't happen?");
-				}
-			} else {
-				body = body_part_new(mp, parent, camel_mime_parser_tell_start_headers(mp)+old_offset, camel_mime_parser_tell(mp)+write_offset);
-			}
-
-			/* check headers for types that we can index */
-			ct = camel_mime_parser_content_type(mp);
-			if (header_content_type_is(ct, "text", "*")) {
-				char *encoding;
-				const char *charset;
-				
-				/* TODO: The filters should all be cached, so they aren't recreated between
-				   messages/message parts */
-				encoding = header_content_encoding_decode(camel_mime_parser_header(mp, "content-transfer-encoding", NULL));
-				if (encoding) {
-					if (!strcasecmp(encoding, "base64")) {
-						d(printf("Adding decoding filter for base64\n"));
-						if (mf64 == NULL)
-							mf64 = camel_mime_filter_basic_new_type(CAMEL_MIME_FILTER_BASIC_BASE64_DEC);
-						enc_id = camel_mime_parser_filter_add(mp, (CamelMimeFilter *)mf64);
-					} else if (!strcasecmp(encoding, "quoted-printable")) {
-						d(printf("Adding decoding filter for quoted-printable\n"));
-						if (mfqp == NULL)
-							mfqp = camel_mime_filter_basic_new_type(CAMEL_MIME_FILTER_BASIC_QP_DEC);
-						enc_id = camel_mime_parser_filter_add(mp, (CamelMimeFilter *)mfqp);
-					}
-					g_free(encoding);
-				}
-				
-				charset = header_content_type_param(ct, "charset");
-				if (charset!=NULL
-				    && !(strcasecmp(charset, "us-ascii")==0
-					|| strcasecmp(charset, "utf-8")==0)) {
-					d(printf("Adding conversion filter from %s to utf-8\n", charset));
-					if (mfc == NULL)
-						mfc = camel_mime_filter_charset_new_convert(charset, "utf-8");
-					if (mfc) {
-						chr_id = camel_mime_parser_filter_add(mp, (CamelMimeFilter *)mfc);
-					} else {
-						g_warning("Cannot convert '%s' to 'utf-8', message display may be corrupt", charset);
-					}
-				}
-
-				doindex = TRUE;
-
-				/* and this filter actually does the indexing */
-				idx_id = camel_mime_parser_filter_add(mp, (CamelMimeFilter *)mfi);
+	camel_mime_parser_seek(mp, offset, SEEK_SET);
+
+	if (offset > 0) {
+		if (camel_mime_parser_step(mp, NULL, NULL) == HSCAN_FROM) {
+			if (camel_mime_parser_tell_start_from(mp) != offset) {
+				g_warning("The next message didn't start where I expected\nbuilding summary from start");
+				camel_mime_parser_drop_step(mp);
+				offset = 0;
+				camel_mime_parser_seek(mp, offset, SEEK_SET);
+				camel_folder_summary_clear((CamelFolderSummary *)mbs);
 			} else {
-				doindex = FALSE;
-			}
-			break;
-
-			/* fixme, this needs thought *sigh* */
-		case HSCAN_MESSAGE_END:
-		case HSCAN_MULTIPART_END:
-			if (parent) {
-				parent->endpos = camel_mime_parser_tell(mp)+write_offset;
-				if (parent->info.parent == NULL) {
-					camel_mbox_summary_add(s, message);
-					message = NULL;
-					parent = NULL;
-				} else {
-					parent = (CamelMboxMessageContentInfo *)parent->info.parent;
-				}
-			}
-			break;
-
-		case HSCAN_BODY:
-			if (doindex) {
-				d(printf("Got content to index:\n%.*s", datalen, data));
-			}
-			break;
-
-		case HSCAN_BODY_END:
-			if (body) {
-				body->endpos = camel_mime_parser_tell(mp)+write_offset;
-				if (body->info.parent == NULL) {
-					camel_mbox_summary_add(s, message);
-					message = NULL;
-				}
+				camel_mime_parser_unstep(mp);
 			}
-
-			d(printf("end of content, removing decoders\n"));
-			if (enc_id != -1) {
-				camel_mime_parser_filter_remove(mp, enc_id);
-				enc_id = -1;
-			}
-			if (chr_id != -1) {
-				camel_mime_parser_filter_remove(mp, chr_id);
-				chr_id = -1;
-			}
-			if (idx_id != -1) {
-				camel_mime_parser_filter_remove(mp, idx_id);
-				idx_id = -1;
-			}
-			break;
-		}
-	}
-
-	/* did we actually write anything out?   Then rename and be done with it. */
-	if (last_write>0) {
-		eof = camel_mime_parser_tell(mp);
-		camel_mbox_summary_copy_block(fd, fdout, last_write, eof-last_write);
-
-		if (close(fdout) == -1) {
-			perror("Could not close output file");
-			unlink(tmpname);
 		} else {
-			printf("renaming %s to %s\n", tmpname, s->folder_path);
-			if (rename(tmpname, s->folder_path) == -1) {
-				perror("Error renaming file");
-				unlink(tmpname);
-			}
+			gtk_object_unref((GtkObject *)mp);
+			/* end of file - no content? */
+			return 0;
 		}
-	} else {
-		/* no, then dont bother touching the inbox */
-		printf("No written changes to mbox, removing tmp file\n");
-		close(fdout);
-		unlink(tmpname);
 	}
 
-	close(fd);
-
-	if (mf64) gtk_object_unref((GtkObject *)mf64);
-	if (mfqp) gtk_object_unref((GtkObject *)mfqp);
-	if (mfc) gtk_object_unref((GtkObject *)mfc);
-	if (mfi) gtk_object_unref((GtkObject *)mfi);
-
-	/* force an index sync? */
-	if (s->index) {
-		ibex_write(s->index);
-	}
-
-        gtk_object_unref((GtkObject *)mp);
-
-	/* and finally ... update the summary sync info */
-	{
-		struct stat st;
+	while (camel_mime_parser_step(mp, NULL, NULL) == HSCAN_FROM) {
+		CamelMessageInfo *info;
 
-		if (stat(s->folder_path, &st) == 0) {
-			s->time = st.st_mtime;
-			s->size = st.st_size;
+		info = camel_folder_summary_add_from_parser((CamelFolderSummary *)mbs, mp);
+		if (info == NULL) {
+			ok = -1;
+			break;
 		}
-	}
-
-	g_free(tmpname);
-
-	return 0;
-}
 
-CamelMboxSummary *camel_mbox_summary_new(const char *summary, const char *folder, ibex *index)
-{
-	CamelMboxSummary *s;
-
-	s = g_malloc0(sizeof(*s));
-
-	s->dirty = TRUE;
-	s->folder_path = g_strdup(folder);
-	s->summary_path = g_strdup(summary);
-	/* FIXME: refcount index? */
-	s->index = index;
-
-	s->messages = g_ptr_array_new();
-	s->message_uid = g_hash_table_new(g_str_hash, g_str_equal);
-
-	/* always force an update */
-	s->time = 0;
-	s->size = 0;
-
-	s->nextuid = 1;
+		g_assert(camel_mime_parser_step(mp, NULL, NULL) == HSCAN_FROM_END);
+	}
 
-	/* TODO: force an initial load right now? */
+	gtk_object_unref((GtkObject *)mp);
 
-	return s;
+	return ok;
 }
 
-void camel_mbox_summary_unref(CamelMboxSummary *s)
+int
+camel_mbox_summary_update(CamelMboxSummary *mbs, off_t offset)
 {
-	g_warning("Unimplemented function, mbox_summary_unref");
+	mbs->index_force = FALSE;
+	return summary_rebuild(mbs, offset);
 }
 
-/* check that the summary is uptodate, TRUE means it is uptodate */
-int camel_mbox_summary_check(CamelMboxSummary *s)
+int
+camel_mbox_summary_load(CamelMboxSummary *mbs, int forceindex)
 {
+	CamelFolderSummary *s = (CamelFolderSummary *)mbs;
 	struct stat st;
+	int ret = 0;
+	off_t minstart;
 
-	/* no folder at all? */
-	if (stat(s->folder_path, &st) != 0)
-		return FALSE;
-
-	return (st.st_size == s->size) && (st.st_mtime == s->time);
-}
-
-static void camel_mbox_summary_add(CamelMboxSummary *s, CamelMboxMessageInfo *info)
-{
-	CamelMboxMessageInfo *old;
+	mbs->index_force = forceindex;
 
-retry:
-	if (info->info.uid == NULL) {
-		info->info.uid = g_strdup_printf("%u", s->nextuid++);
-	}
-	if (( old = g_hash_table_lookup(s->message_uid, info->info.uid) )) {
-#warning do something fatal with a fatal error.
-		/* err, once i work out why it keeps getting called so often */
-		d(g_warning("Trying to insert message with clashing uid's new %s exist %s", info->info.uid, old->info.uid));
-		g_free(info->info.uid);
-		info->info.uid = NULL;
-		goto retry;
-	}
-	d(printf("adding %s\n", info->info.uid));
-	g_ptr_array_add(s->messages, info);
-	g_hash_table_insert(s->message_uid, info->info.uid, info);
-	s->dirty = TRUE;
-}
-
-static int summary_header_read(FILE *fp, guint32 *version, time_t *time, size_t *size, guint32 *nextuid)
-{
-	fseek(fp, 0, SEEK_SET);
-	*version = decode_fixed_int(fp);
-	*time = decode_fixed_int(fp);
-	*size = decode_fixed_int(fp);
-	*nextuid = decode_fixed_int(fp);
-	return ferror(fp);
-}
-
-static void
-summary_clear(CamelMboxSummary *s)
-{
-	int i;
-
-	for (i=0;i<s->messages->len;i++) {
-		message_struct_free(g_ptr_array_index(s->messages, i));
+	/* is the summary out of date? */
+	if (stat(mbs->folder_path, &st) == -1) {
+		camel_folder_summary_clear(s);
+		printf("Cannot summarise folder: '%s': %s\n", mbs->folder_path, strerror(errno));
+		return -1;
 	}
-	g_ptr_array_free(s->messages, TRUE);
-	g_hash_table_destroy(s->message_uid);
 
-	s->messages = g_ptr_array_new();
-	s->message_uid = g_hash_table_new(g_str_hash, g_str_equal);
-}
-
-int camel_mbox_summary_load(CamelMboxSummary *s)
-{
-	struct stat st;
-	FILE *fp;
-	int i, total;
-	CamelMboxMessageInfo *info;
-
-	summary_clear(s);
-
-	if ((fp = fopen(s->summary_path, "r")) == NULL) {
-		g_warning("Loading non-existant summary, generating summary for folder: %s: %s", s->summary_path, strerror(errno));
-		index_folder(s, 0);
-		camel_mbox_summary_save(s);
+	if (forceindex || camel_folder_summary_load(s) == -1) {
+		ret = summary_rebuild(mbs, 0);
 	} else {
-		guint32 version, nextuid;
-		time_t time;
-		size_t size;
-
-		if (stat(s->folder_path, &st) != 0) {
-			g_warning("Uh, no folder anyway, aborting");
-			fclose(fp);
-			return -1;
-		}
-
-		if (summary_header_read(fp, &version, &time, &size, &nextuid) != 0
-		    || version != CAMEL_MBOX_SUMMARY_VERSION) {
-			g_warning("Summary missing or version mismatch, reloading summary");
-			fclose(fp);
-			index_folder(s, 0);
-			camel_mbox_summary_save(s);
-			return 0;
+		minstart = st.st_size;
+#if 0
+		/* find out the first unindexed message ... */
+		/* TODO: For this to work, it has to check that the message is
+		   indexable, and contains content ... maybe it cannot be done 
+		   properly? */
+		for (i=0;i<camel_folder_summary_count(s);i++) {
+			CamelMessageInfo *mi = camel_folder_summary_index(s, i);
+			if (!ibex_contains_name(mbs->index, mi->uid)) {
+				minstart = ((CamelMboxMessageInfo *)mi)->frompos;
+				printf("Found unindexed message: %s\n", mi->uid);
+				break;
+			}
 		}
-
-		s->nextuid = MAX(s->nextuid, nextuid);
-		s->time = time;
-		s->size = size;
-		total = decode_fixed_int(fp);
-			
-		if (time != st.st_mtime || size != st.st_size) {
-			/* if its grown, then just index the new stuff, and load the rest from the summary */
-			if (size < st.st_size) {
-				g_warning("Indexing/summarizing from start position: %d", size);
-
-				d(printf("loading %d items from summary file\n", total));
-				for (i=0;i<total;i++) {
-					info = message_struct_load(fp);
-					if (info) {
-						camel_mbox_summary_add(s, info);
-					} else {
-						break;
-					}
+#endif
+		/* is the summary uptodate? */
+		if (st.st_size == mbs->folder_size && st.st_mtime == s->time) {
+			printf("Summary time and date match mbox\n");
+			if (minstart < st.st_size) {
+				/* FIXME: Only clear the messages and reindex from this point forward */
+				camel_folder_summary_clear(s);
+				ret = summary_rebuild(mbs, 0);
+			}
+		} else {
+			if (mbs->folder_size < st.st_size) {
+				printf("Index is for a smaller mbox\n");
+				if (minstart < mbs->folder_size) {
+					/* FIXME: only make it rebuild as necessary */
+					camel_folder_summary_clear(s);
+					ret = summary_rebuild(mbs, 0);
+				} else {
+					ret = summary_rebuild(mbs, mbs->folder_size);
 				}
-				fclose(fp);
-				s->dirty = FALSE;
-				index_folder(s, size); /* if it adds any, it'll dirtify it */
-				camel_mbox_summary_save(s);
 			} else {
-				g_warning("Folder changed/smaller, reindexing everything");
-				index_folder(s, 0);
-				camel_mbox_summary_save(s);
-				fclose(fp);
+				printf("index is for a bigger mbox\n");
+				camel_folder_summary_clear(s);
+				ret = summary_rebuild(mbs, 0);
 			}
-			return 0;
 		}
+	}
 
-		printf("loading %d items from summary file\n", total);
-		for (i=0;i<total;i++) {
-			info = message_struct_load(fp);
-			if (info) {
-				camel_mbox_summary_add(s, info);
-			} else {
-				break;
-			}
-		}
-		fclose(fp);
-		s->dirty = FALSE;
+	if (ret != -1) {
+		mbs->folder_size = st.st_size;
+		s->time = st.st_mtime;
+		printf("saving summary\n");
+		if (camel_folder_summary_save(s) == -1)
+			g_warning("Could not save summary: %s", strerror(errno));
+		printf("summary saved\n");
+		if (mbs->index)
+			ibex_save(mbs->index);
+		printf("ibex saved\n");
 	}
-	return 0;
-}
 
-static int summary_header_write(FILE *fp, CamelMboxSummary *s)
-{
-	fseek(fp, 0, SEEK_SET);
-	encode_fixed_int(fp, CAMEL_MBOX_SUMMARY_VERSION);
-	encode_fixed_int(fp, s->time);
-	/* if we're dirty, then dont *really* save it ... */
-	if (s->dirty)
-		encode_fixed_int(fp, 0);
-	else
-		encode_fixed_int(fp, s->size);
-	encode_fixed_int(fp, s->nextuid);
-	fflush(fp);
-	return ferror(fp);
+	return ret;
 }
 
-static int summary_header_save(CamelMboxSummary *s)
+static int
+header_write(int fd, struct _header_raw *header, char *xevline)
 {
-	int fd;
-	FILE *fp;
+        struct iovec iv[4];
+        int outlen = 0, len;
 
-	fd = open(s->summary_path, O_WRONLY|O_CREAT, 0600);
-	if (fd == -1)
-		return -1;
-	fp = fdopen(fd, "w");
-	if (fp == NULL)
-		return -1;
+        iv[1].iov_base = ":";
+        iv[1].iov_len = 1;
+        iv[3].iov_base = "\n";
+        iv[3].iov_len = 1;
 
-	summary_header_write(fp, s);
-	return fclose(fp);
-}
-
-int camel_mbox_summary_save(CamelMboxSummary *s)
-{
-	int i, fd;
-	FILE *fp;
-
-	printf("saving summary? %s\n", s->summary_path);
+        while (header) {
+		if (strcasecmp(header->name, "X-Evolution")) {
+			iv[0].iov_base = header->name;
+			iv[0].iov_len = strlen(header->name);
+			iv[2].iov_base = header->value;
+			iv[2].iov_len = strlen(header->value);
+		
+			do {
+				len = writev(fd, iv, 4);
+			} while (len == -1 && errno == EINTR);
+			
+			if (len == -1)
+				return -1;
+			outlen += len;
+		}
+                header = header->next;
+        }
 
-	/* FIXME: error checking */
-	if (s->dirty) {
-		printf("yes\n");
-		fd = open(s->summary_path, O_WRONLY|O_CREAT|O_TRUNC, 0600);
-		if (fd == -1)
-			return -1;
-		fp = fdopen(fd, "w");
-		if (fp == NULL)
-			return -1;
+        iv[0].iov_base = "X-Evolution: ";
+        iv[0].iov_len = strlen(iv[0].iov_base);
+        iv[1].iov_base = xevline;
+        iv[1].iov_len = strlen(xevline);
+        iv[2].iov_base = "\n\n";
+        iv[2].iov_len = 2;
 
-		s->dirty = FALSE;
+	do {
+		len = writev(fd, iv, 3);
+	} while (len == -1 && errno == EINTR);
 
-		summary_header_write(fp, s);
-		encode_fixed_int(fp, s->messages->len);
+	if (len == -1)
+		return -1;
 
-		printf("message count = %d\n", s->messages->len);
+	outlen += 1;
 
-		for (i=0;i<s->messages->len;i++) {
-			message_struct_save(fp, g_ptr_array_index(s->messages, i));
-		}
-		fclose(fp);
-	} else {
-		printf("no\n");
-	}
-	return 0;
-}
+	d(printf("Wrote %d bytes of headers\n", outlen));
 
-CamelMboxMessageInfo *camel_mbox_summary_uid(CamelMboxSummary *s, const char *uid)
-{
-	return g_hash_table_lookup(s->message_uid, uid);
+        return outlen;
 }
 
-CamelMboxMessageInfo *camel_mbox_summary_index(CamelMboxSummary *s, int index)
+static int
+copy_block(int fromfd, int tofd, off_t start, size_t bytes)
 {
-	return g_ptr_array_index(s->messages, index);
-}
+        char buffer[4096];
+        int written = 0;
 
-int camel_mbox_summary_message_count(CamelMboxSummary *s)
-{
-	return s->messages->len;
-}
+	d(printf("writing %d bytes ... ", bytes));
 
-guint32 camel_mbox_summary_next_uid(CamelMboxSummary *s)
-{
-	guint32 uid = s->nextuid++;
+	if (lseek(fromfd, start, SEEK_SET) != start)
+		return -1;
 
-	summary_header_save(s);
-	return uid;
-}
+        while (bytes>0) {
+                int toread, towrite;
 
-guint32 camel_mbox_summary_set_uid(CamelMboxSummary *s, guint32 uid)
-{
-	if (s->nextuid <= uid) {
-		s->nextuid = uid+1;
-		summary_header_save(s);
-	}
-	return s->nextuid;
-}
+                toread = bytes;
+                if (bytes>4096)
+                        toread = 4096;
+                else
+                        toread = bytes;
+		do {
+			towrite = read(fromfd, buffer, toread);
+		} while (towrite == -1 && errno == EINTR);
 
-void camel_mbox_summary_set_flags_by_uid(CamelMboxSummary *s, const char *uid, guint32 flags)
-{
-	CamelMessageInfo *info;
+		if (towrite == -1)
+			return -1;
 
-	info = (CamelMessageInfo *)camel_mbox_summary_uid(s, uid);
-	if (info != NULL) {
-		if (info->flags != flags) {
-			info->flags = flags | CAMEL_MESSAGE_FOLDER_FLAGGED;
-			s->dirty = TRUE;
+                /* check for 'end of file' */
+                if (towrite == 0) {
+			d(printf("end of file?\n"));
+                        break;
 		}
-	} else {
-		g_warning("Message has dissapeared from summary?  uid %s", uid);
-	}
-}
 
-/* update offsets in the summary to take account deleted parts */
-static void
-offset_content(CamelMboxMessageContentInfo *content, off_t offset)
-{
-	content->pos -= offset;
-	content->bodypos -= offset;
-	content->endpos -= offset;
-	content = (CamelMboxMessageContentInfo *)content->info.childs;
-	while (content) {
-		offset_content(content, offset);
-		content = (CamelMboxMessageContentInfo *)content->info.next;
-	}
-}
+		do {
+			toread = write(tofd, buffer, towrite);
+		} while (toread == -1 && errno == EINTR);
 
-void camel_mbox_summary_remove_uid(CamelMboxSummary *s, const char *uid)
-{
-	CamelMboxMessageInfo *oldinfo;
-	char *olduid;
+		if (toread == -1)
+			return -1;
 
-	if (g_hash_table_lookup_extended(s->message_uid, uid, (void *)&olduid, (void *)&oldinfo)) {
-#if 0
-		/* FIXME: this code should be executed to update the summary info,
-		   however, only if we're not depending on the info not changing yet */
-		off_t offset = info->endpos - info->pos;
-		int i, count, got = FALSE;
-
-		count = s->messages->len;
-		for (i=0;i<count;i++) {
-			CamelMboxMessageInfo *minfo = g_ptr_array_index(s->messages, i);
-			if (got) {
-				offset_content(minfo, offset);
-			} else if (minfo == info) {
-				got = TRUE;
-			}
-		}
-#endif
-		g_hash_table_remove(s->message_uid, uid);
-		g_ptr_array_remove(s->messages, oldinfo);
-		message_struct_free(oldinfo);
-		g_free(olduid);
-	}
+                written += toread;
+                bytes -= toread;
+        }
+
+        d(printf("written %d bytes\n", written));
+
+        return written;
 }
 
-/* perform expunge/update xev headers, etc */
-/* TODO: merge this with the indexing code, so that it can index new parts
-   without having to reread everything again? */
-/* TODO: sync with the mbox, if it has changed */
 int
-camel_mbox_summary_expunge(CamelMboxSummary *s)
+camel_mbox_summary_expunge(CamelMboxSummary *mbs)
 {
-	int quick = TRUE, work=FALSE;
+	CamelMimeParser *mp=NULL;
 	int i, count;
 	CamelMboxMessageInfo *info;
+	CamelFolderSummary *s = (CamelFolderSummary *)mbs;
 
-	printf("Executing expunge ...\n");
+	int fd=-1, fdout=-1;
+	off_t offset = 0;
+	char *tmpname=0;
+	char *buffer, *xev = NULL;
+	int len;
+	guint32 uid, flags;
+	int quick = TRUE, work = FALSE;
+
+	/* make sure we're in sync */
+	/*camel_mbox_summary_load(mbs, FALSE); ? */
 
-	count = camel_mbox_summary_message_count(s);
+	count = camel_folder_summary_count(s);
+	d(printf("Performing expunge, %d messages in inbox\n", count));
 	for (i=0;quick && i<count;i++) {
-		info = camel_mbox_summary_index(s, i);
-		if (info->xev_offset == -1 || info->info.flags & CAMEL_MESSAGE_DELETED)
+		info = (CamelMboxMessageInfo *)camel_folder_summary_index(s, i);
+		if (info->info.flags & (CAMEL_MESSAGE_DELETED|CAMEL_MESSAGE_FOLDER_NOXEV))
 			quick = FALSE;
 		else
 			work |= (info->info.flags & CAMEL_MESSAGE_FOLDER_FLAGGED) != 0;
 	}
-	if (quick) {
-		int fd;
-		char name[32];
 
-		if (!work)
-			return 0;
+	d(printf("Options: %s %s\n", quick?"quick":"", work?"Work":""));
 
-		camel_mbox_summary_save(s);
+	if (quick && !work)
+		return 0;
 
-		fd = open(s->folder_path, O_WRONLY);
+	fd = open(mbs->folder_path, O_RDWR);
+	if (fd == -1)
+		return -1;
 
-		if (fd == -1) {
-			g_error("Cannot open folder for update: %s", strerror(errno));
-			return -1;
-		}
+	mp = camel_mime_parser_new();
+	camel_mime_parser_scan_from(mp, TRUE);
+	camel_mime_parser_init_with_fd(mp, fd);
 
-		/* poke in the new xev info */
-		for (i=0;i<count;i++) {
-			info = camel_mbox_summary_index(s, i);
-			g_assert(info);
-			g_assert(info->xev_offset != -1);
-			if (info->info.flags & CAMEL_MESSAGE_FOLDER_FLAGGED) {
-				printf("Setting new flags to message %s = %04x\n", info->info.uid, info->info.flags & 0xffff);
-				lseek(fd, info->xev_offset, SEEK_SET);
-				sprintf(name, "X-Evolution: %08x-%04x\n\n", (unsigned int)strtoul(info->info.uid, NULL, 10), info->info.flags & 0xffff);
-				write(fd, name, strlen(name));
-				info->info.flags &= ~CAMEL_MESSAGE_FOLDER_FLAGGED;
-			}
+	if (!quick) {
+		tmpname = alloca(strlen(mbs->folder_path)+5);
+		sprintf(tmpname, "%s.tmp", mbs->folder_path);
+		d(printf("Writing tmp file to %s\n", tmpname));
+	retry_out:
+		fdout = open(tmpname, O_WRONLY|O_CREAT|O_EXCL, 0600);
+		if (fdout == -1) {
+			if (errno == EEXIST)
+				if (unlink(tmpname) != -1)
+					goto retry_out;
+			tmpname = 0;
+			goto error;
 		}
-		close(fd);
-	} else {
-		char *tmpname;
-		int fd, fdout;
-		int last_write = 0, offset = 0, summary_end = 0, last_start = 0;
-		CamelMboxMessageContentInfo *content;
-		struct stat st;
+	}
 
-		printf("We must expunge messages and/or write headers\n");
+	for (i=0;i<count;i++) {
+		off_t frompos, bodypos;
+		off_t xevoffset;
 
-		/* FIXME: This should check the current summary is correct */
+		info = (CamelMboxMessageInfo *)camel_folder_summary_index(s, i);
 
-		/* we have to write out new headers or delete messages, starting from message i */
+		g_assert(info);
 
-		fd = open(s->folder_path, O_RDONLY);
-		if (fd == -1) {
-			g_error("Cannot open folder for read: %s", strerror(errno));
-			return -1;
-		}
+		d(printf("Looking at message %s\n", info->info.uid));
 
-		tmpname = g_strdup_printf("%s.tmp", s->folder_path);
-		fdout = open(tmpname, O_WRONLY|O_CREAT|O_TRUNC, 0600);
-		if (fdout == -1) {
-			g_error("Cannot open tmp file for write: %s", strerror(errno));
-			close(fd);
-			g_free(tmpname);
-			return -1;
-		}
+		if (info->info.flags & CAMEL_MESSAGE_DELETED) {
+			d(printf("Deleting %s\n", info->info.uid));
 
-		for (i=0;i<count;i++) {
-			info = camel_mbox_summary_index(s, i);
-			g_assert(info);
-			g_assert(info->info.content);
+			g_assert(!quick);
+			offset -= (info->info.content->endpos - info->frompos);
+			camel_folder_summary_remove(s, (CamelMessageInfo *)info);
+			count--;
+			i--;
+			info = NULL;
+		} else if (info->info.flags & (CAMEL_MESSAGE_FOLDER_NOXEV|CAMEL_MESSAGE_FOLDER_FLAGGED)) {
+			int xevok = FALSE;
 
-			content = (CamelMboxMessageContentInfo *)info->info.content;
+			d(printf("Updating header for %s flags = %08x\n", info->info.uid, info->info.flags));
 
-			/* FIXME: Must also write out xev headers etc, as appropriate? */
+			/* find the next message, header parts */
+			camel_mime_parser_seek(mp, info->frompos, SEEK_SET);
+			if (camel_mime_parser_step(mp, &buffer, &len) != HSCAN_FROM)
+				goto error;
 
-			/* we need to use the end of the previous message, becuase the beginning of
-			   this message doesn't include the "^From " line. */
+			if (camel_mime_parser_tell_start_from(mp) != info->frompos) {
+				g_error("Summary/mbox mismatch, aborting expunge");
+				goto error;
+			}
+			
+			if (camel_mime_parser_step(mp, &buffer, &len) == HSCAN_FROM_END)
+				goto error;
 
-			/* do we remove this message? */
-			if (info->info.flags & CAMEL_MESSAGE_DELETED) {
-				printf("Deleting message: %s\n", info->info.uid);
-				camel_mbox_summary_copy_block(fd, fdout, last_write, last_start-last_write);
-				offset += (content->endpos - last_start);
-				last_write = content->endpos;
-				last_start = last_write;
+			xev = (char *)camel_mime_parser_header(mp, "X-Evolution", (int *)&xevoffset);
+			if (xev && header_evolution_decode(xev, &uid, &flags) != -1) {
+				char name[64];
 
-				/* remove this message from the index */
-				if (s->index) {
-					ibex_unindex(s->index, info->info.uid);
+				sprintf(name, "%u", uid);
+				if (strcmp(name, info->info.uid)) {
+					d(printf("Summary mismatch, aborting leaving mailbox intact\n"));
+					goto error;
 				}
-
-				camel_mbox_summary_remove_uid(s, info->info.uid);
-				i--; /* redo this index */
-				count--;
-			} else {
-				printf("Keeping message: %s\n", info->info.uid);
-				last_start = content->endpos;
-				offset_content(content, offset);
-				summary_end = content->endpos;
+				xevok = TRUE;
 			}
+			xev = header_evolution_encode(strtoul(info->info.uid, NULL, 10), info->info.flags);
+			if (quick) {
+				if (!xevok) {
+					g_error("The summary told me I had an X-Evolution header, but i dont!");
+					goto error;
+				}
+				buffer = g_strdup_printf("X-Evolution: %s", xev);
+				do {
+					len = write(fd, buffer, strlen(buffer));
+				} while (len == -1 && errno == EINTR);
+				g_free(buffer);
+				if (len == -1) {
+					goto error;
+				}
+			} else {
+				frompos = lseek(fdout, 0, SEEK_CUR);
+				write(fdout, "From -\n", strlen("From -\n"));
+				if (header_write(fdout, camel_mime_parser_headers_raw(mp), xev) == -1) {
+					d(printf("Error writing to tmp mailbox\n"));
+					goto error;
+				}
+				bodypos = lseek(fdout, 0, SEEK_CUR);
+				d(printf("pos = %d, endpos = %d, bodypos = %d\n",
+					 info->info.content->pos,
+					 info->info.content->endpos,
+					 info->info.content->bodypos));
+				if (copy_block(fd, fdout, info->info.content->bodypos, info->info.content->endpos - info->info.content->bodypos) == -1) {
+					g_warning("Cannot copy data to output fd");
+					goto error;
+				}
+				info->frompos = frompos;
+				offset = bodypos - info->info.content->bodypos;
+			}			
+			g_free(xev); xev = NULL;
+			camel_mime_parser_drop_step(mp);
+			camel_mime_parser_drop_step(mp);
+		} else {
+			d(printf("Nothing to do for this message\n"));
 		}
-		/* copy the rest of the file ... */
-		camel_mbox_summary_copy_block(fd, fdout, last_write, INT_MAX);
-
-		close(fd);
-		if (close(fdout) == -1) {
-			g_error("Cannot close tmp folder: %s", strerror(errno));
-			unlink(tmpname);
-			g_free(tmpname);
-			return -1;
+		if (!quick && info!=NULL && offset!=0) {
+			camel_folder_summary_offset_content(info->info.content, offset);
+			d(printf("pos = %d, endpos = %d, bodypos = %d\n",
+				 info->info.content->pos,
+				 info->info.content->endpos,
+				 info->info.content->bodypos));
 		}
+	}
 
-		if (rename(tmpname, s->folder_path) == -1) {
-			g_error("Cannot rename folder: %s", strerror(errno));
-			unlink(tmpname);
-			g_free(tmpname);
-			return -1;
-		}
+	d(printf("Closing folders\n"));
+
+	if (close(fd) == -1) {
+		g_warning("Cannot close source folder: %s", strerror(errno));
+		goto error;
+	}
+
+	if (!quick) {
+		struct stat st;
 
-		/* force an index sync */
-		if (s->index) {
-			ibex_write(s->index);
+		if (close(fdout) == -1) {
+			g_warning("Cannot close tmp folder: %s", strerror(errno));
+			goto error;
 		}
 
-		/* update summary size to match the actual (written) folder size ... */
-		s->size = summary_end;
-		/* and the time to match the newly written time, so we dont update needlessly */
-		if (stat(s->folder_path, &st) == 0) {
-			s->time = st.st_mtime;
+		if (rename(tmpname, mbs->folder_path) == -1) {
+			g_warning("Cannot rename folder: %s", strerror(errno));
+			goto error;
 		}
+		tmpname = 0;
 
-		camel_mbox_summary_save(s);
+		if (stat(mbs->folder_path, &st) == -1)
+			goto error;
+
+		s->flags |= CAMEL_SUMMARY_DIRTY;
+		s->time = st.st_mtime;
+		mbs->folder_size = st.st_size;
+		camel_folder_summary_save(s);
 	}
 
-	return 0;
-}
+	gtk_object_unref((GtkObject *)mp);
 
+	return 0;
+error:
+	d(printf("Error occured: %s\n", strerror(errno)));
+	count = errno;
 
-#if 0
-int main(int argc, char **argv)
-{
-	if (argc<2) {
-		printf("usage: %s mbox\n", argv[0]);
-		return 1;
-	}
+	close(fd);
+	close(fdout);
 
-	gtk_init(&argc, &argv);
+	g_free(xev);
 
-	index_folder(argv[1]);
+	if (tmpname)
+		unlink(tmpname);
+	if (mp)
+		gtk_object_unref((GtkObject *)mp);
 
-	return 0;
+	errno = count;
+	return -1;
 }
-#endif
diff --git a/camel/providers/mbox/camel-mbox-summary.h b/camel/providers/mbox/camel-mbox-summary.h
index fda574c6f8..6d98fd051d 100644
--- a/camel/providers/mbox/camel-mbox-summary.h
+++ b/camel/providers/mbox/camel-mbox-summary.h
@@ -3,82 +3,76 @@
  *
  *  Authors: Michael Zucchi <notzed@helixcode.com>
  *
- *  This program is free software; you can redistribute it and/or
- *  modify it under the terms of the GNU Library General Public License
- *  as published by the Free Software Foundation; either version 2 of
- *  the License, or (at your option) any later version.
+ *  This program is free software; you can redistribute it and/or 
+ *  modify it under the terms of the GNU General Public License as 
+ *  published by the Free Software Foundation; either version 2 of the
+ *  License, or (at your option) any later version.
  *
  *  This program is distributed in the hope that it will be useful,
  *  but WITHOUT ANY WARRANTY; without even the implied warranty of
  *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- *  GNU Library General Public License for more details.
+ *  GNU General Public License for more details.
  *
- *  You should have received a copy of the GNU Library General Public
- *  License along with this program; if not, write to the Free Software
- *  Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
+ *  USA
  */
 
 #ifndef _CAMEL_MBOX_SUMMARY_H
 #define _CAMEL_MBOX_SUMMARY_H
 
-#include <glib.h>
-#include <camel/camel-folder.h>
+#include <gtk/gtk.h>
+#include <camel/camel-folder-summary.h>
 #include <libibex/ibex.h>
 
-typedef struct {
-	CamelMessageContentInfo info;
+#define CAMEL_MBOX_SUMMARY(obj)         GTK_CHECK_CAST (obj, camel_mbox_summary_get_type (), CamelMboxSummary)
+#define CAMEL_MBOX_SUMMARY_CLASS(klass) GTK_CHECK_CLASS_CAST (klass, camel_mbox_summary_get_type (), CamelMboxSummaryClass)
+#define IS_CAMEL_MBOX_SUMMARY(obj)      GTK_CHECK_TYPE (obj, camel_mbox_summary_get_type ())
+
+typedef struct _CamelMboxSummary      CamelMboxSummary;
+typedef struct _CamelMboxSummaryClass CamelMboxSummaryClass;
 
-	/* position in stream of this part */
-	off_t pos;
-	off_t bodypos;
-	off_t endpos;
+/* extra summary flags */
+enum {
+	CAMEL_MESSAGE_FOLDER_NOXEV = 1<<16,
+/*	CAMEL_MESSAGE_FOLDER_FLAGGED = 1<<17,*/
+};
+
+typedef struct _CamelMboxMessageContentInfo {
+	CamelMessageContentInfo info;
 } CamelMboxMessageContentInfo;
 
-typedef struct {
+typedef struct _CamelMboxMessageInfo {
 	CamelMessageInfo info;
 
-	/* position of the xev header, if one exists */
-	off_t xev_offset;
+	off_t frompos;
 } CamelMboxMessageInfo;
 
-typedef struct {
-	int dirty;		/* if anything has changed */
-
-	char *folder_path;
-	char *summary_path;
-	ibex *index;
-
-	GPtrArray *messages;	/* array of messages matching mbox order */
-	GHashTable *message_uid; /* index to messages by uid */
+struct _CamelMboxSummary {
+	CamelFolderSummary parent;
 
-	int nextuid;
+	struct _CamelMboxSummaryPrivate *priv;
 
-	time_t time;		/* time/size of folder's last update */
-	size_t size;
-} CamelMboxSummary;
+	char *folder_path;	/* name of matching folder */
+	size_t folder_size;	/* size of the mbox file, last sync */
 
-CamelMboxSummary *camel_mbox_summary_new(const char *summary, const char *folder, ibex *index);
-void camel_mbox_summary_unref(CamelMboxSummary *);
-
-int camel_mbox_summary_load(CamelMboxSummary *);
-int camel_mbox_summary_save(CamelMboxSummary *);
-int camel_mbox_summary_check(CamelMboxSummary *);
-int camel_mbox_summary_expunge(CamelMboxSummary *);
-
-guint32 camel_mbox_summary_next_uid(CamelMboxSummary *);
-/* set the minimum uid */
-guint32 camel_mbox_summary_set_uid(CamelMboxSummary *s, guint32 uid);
-
-CamelMboxMessageInfo *camel_mbox_summary_uid(CamelMboxSummary *s, const char *uid);
-/* dont use this function yet ... */
-void camel_mbox_summary_remove_uid(CamelMboxSummary *s, const char *uid);
-CamelMboxMessageInfo *camel_mbox_summary_index(CamelMboxSummary *, int index);
-int camel_mbox_summary_message_count(CamelMboxSummary *);
-
-/* set flags within a summary item */
-void camel_mbox_summary_set_flags_by_uid(CamelMboxSummary *s, const char *uid, guint32 flags);
-
-/* TODO: should be in a utility library */
-int camel_mbox_summary_copy_block(int fromfd, int tofd, off_t readpos, size_t bytes);
+	ibex *index;
+	int index_force;	/* do we force index during creation? */
+};
+
+struct _CamelMboxSummaryClass {
+	CamelFolderSummaryClass parent_class;
+};
+
+guint		camel_mbox_summary_get_type	(void);
+CamelMboxSummary      *camel_mbox_summary_new	(const char *filename, const char *mbox_name, ibex *index);
+
+/* load/check the summary */
+int camel_mbox_summary_load(CamelMboxSummary *mbs, int forceindex);
+/* incremental update */
+int camel_mbox_summary_update(CamelMboxSummary *mbs, off_t offset);
+/* perform a folder expunge */
+int camel_mbox_summary_expunge(CamelMboxSummary *mbs);
 
 #endif /* ! _CAMEL_MBOX_SUMMARY_H */