Shite, -1 on error, >=0 on success. So i've just been truncating all the

2000-11-21  Not Zed  <NotZed@HelixCode.com>

	* providers/local/camel-mh-summary.c (mh_summary_sync_message):
	Shite, -1 on error, >=0 on success.  So i've just been truncating
	all the messages I touched, good one zed.
	(mh_summary_sync_message): Sigh, and write to the right damn fd as
	well.
	(mh_summary_sync_message): Argh, and we need to compare the length
	of the old xev -1 to the new xev, to check if we can optimise it.

	* camel-folder.c (camel_folder_change_info_new): Init the pool.
	(camel_folder_change_info_add_source): Allocate string in the
	pool.
	(camel_folder_change_info_add_source_list):
	(camel_folder_change_info_add_update): No longer free the key, as
	it cannot be yet.
	(change_info_add_uid): Add a new arg, copy, telling it whether to
	copy the uid argument or not, and copy using mempool_strdup.
	(change_info_cat): Tell add_uid to copy the string.
	(camel_folder_change_info_add_update): Call add_uid directly.
	(change_info_remove): Call add_uid directly, with no copy, and
	dont free the key.
	(change_info_free_update): No longer required since we dont malloc
	the keys.
	(camel_folder_change_info_add_uid): Fix for add_uid change.
	(camel_folder_change_info_remove_uid):
	(camel_folder_change_info_change_uid):
	(change_info_clear): No longer needed, just set the size to 0 on
	the array directly.
	(camel_folder_change_info_clear): Empty the arrays directly, and
	flush the mempool too, and also clear uid_source, incase anyone
	was silly enough to call us in the wrong order.
	(camel_folder_change_info_free): Dont bother clearing the array's
	contents, just free the pool and throw away all the indexes.

	* camel-folder.h: Added a mempool to CamelFolderChangeInfo to
	store the uid's we get.

	* camel-folder-search.c (search_match_all): If we are only
	matching a single info, just use that/do the search.
	(camel_folder_search_match_expression): New function.  Matches a
	single message info against an expression.
	(camel_folder_search_init): Init a hash table used to map the
	returned gptrarrays' to mempools.
	(camel_folder_search_execute_expression): Store all of the string
	data in a mempool, slightly faster, less wasted space (usually),.
	(camel_folder_search_free_result): Check for the mempool that
	stores the data for the list, and free that if we have it,
	otherwise assume we need to use g_free() (which should only happen
	if the list is empty at the moment).
	: commented out the debugging prints.  Got sick of 'executing
	header search' crap.

	* providers/vee/camel-vee-folder.c (camel_vee_folder_init): Init
	changes.
	(camel_vee_folder_finalise): Free changes.
	(vfolder_add_match): Simple helper to add a new matching info
	record.
	(camel_vee_folder_add_folder): Only trigger a changed event if we
	have changes.
	(vfolder_change_match): New function, changes our local vfolder
	info to match the source.
	(vfolder_add_match): Add a new info to the vfolder list.
	(vfolder_remove_match): Remove a no-longer matching info from the
	vfolder summary.
	(message_changed): check if the message still matches, and
	remove/etc as required.
	(camel_vee_folder_finalise, init): init/free search object.
	(vee_folder_build_folder): Build the changes to the folder into
	the changes data, as we go.
	(folder_changed): If the folder gave us an explicit list of
	changes, then process each one separately (unless there's a lot
	added/changed).

	* providers/vee/camel-vee-folder.h: Added a changes field to the
	folder.

svn path=/trunk/; revision=6628

11 files changed, 462 insertions, 95 deletions

diff --git a/camel/ChangeLog b/camel/ChangeLog
index d4cf8d6fa3..1bea526cde 100644
--- a/camel/ChangeLog
+++ b/camel/ChangeLog
@@ -1,5 +1,80 @@
 2000-11-21  Not Zed  <NotZed@HelixCode.com>
 
+	* providers/local/camel-mh-summary.c (mh_summary_sync_message):
+	Shite, -1 on error, >=0 on success.  So i've just been truncating
+	all the messages I touched, good one zed.
+	(mh_summary_sync_message): Sigh, and write to the right damn fd as
+	well.
+	(mh_summary_sync_message): Argh, and we need to compare the length
+	of the old xev -1 to the new xev, to check if we can optimise it.
+
+	* camel-folder.c (camel_folder_change_info_new): Init the pool.
+	(camel_folder_change_info_add_source): Allocate string in the
+	pool.
+	(camel_folder_change_info_add_source_list): 
+	(camel_folder_change_info_add_update): No longer free the key, as
+	it cannot be yet.
+	(change_info_add_uid): Add a new arg, copy, telling it whether to
+	copy the uid argument or not, and copy using mempool_strdup.
+	(change_info_cat): Tell add_uid to copy the string.
+	(camel_folder_change_info_add_update): Call add_uid directly.
+	(change_info_remove): Call add_uid directly, with no copy, and
+	dont free the key.
+	(change_info_free_update): No longer required since we dont malloc
+	the keys.
+	(camel_folder_change_info_add_uid): Fix for add_uid change.
+	(camel_folder_change_info_remove_uid): 
+	(camel_folder_change_info_change_uid): 
+	(change_info_clear): No longer needed, just set the size to 0 on
+	the array directly.
+	(camel_folder_change_info_clear): Empty the arrays directly, and
+	flush the mempool too, and also clear uid_source, incase anyone
+	was silly enough to call us in the wrong order.
+	(camel_folder_change_info_free): Dont bother clearing the array's
+	contents, just free the pool and throw away all the indexes.
+
+	* camel-folder.h: Added a mempool to CamelFolderChangeInfo to
+	store the uid's we get.
+
+	* camel-folder-search.c (search_match_all): If we are only
+	matching a single info, just use that/do the search.
+	(camel_folder_search_match_expression): New function.  Matches a
+	single message info against an expression.
+	(camel_folder_search_init): Init a hash table used to map the
+	returned gptrarrays' to mempools.
+	(camel_folder_search_execute_expression): Store all of the string
+	data in a mempool, slightly faster, less wasted space (usually),.
+	(camel_folder_search_free_result): Check for the mempool that
+	stores the data for the list, and free that if we have it,
+	otherwise assume we need to use g_free() (which should only happen
+	if the list is empty at the moment).
+	: commented out the debugging prints.  Got sick of 'executing
+	header search' crap.
+
+	* providers/vee/camel-vee-folder.c (camel_vee_folder_init): Init
+	changes.
+	(camel_vee_folder_finalise): Free changes.
+	(vfolder_add_match): Simple helper to add a new matching info
+	record.
+	(camel_vee_folder_add_folder): Only trigger a changed event if we
+	have changes.
+	(vfolder_change_match): New function, changes our local vfolder
+	info to match the source.
+	(vfolder_add_match): Add a new info to the vfolder list.
+	(vfolder_remove_match): Remove a no-longer matching info from the
+	vfolder summary.
+	(message_changed): check if the message still matches, and
+	remove/etc as required.
+	(camel_vee_folder_finalise, init): init/free search object.
+	(vee_folder_build_folder): Build the changes to the folder into
+	the changes data, as we go.
+	(folder_changed): If the folder gave us an explicit list of
+	changes, then process each one separately (unless there's a lot
+	added/changed).
+
+	* providers/vee/camel-vee-folder.h: Added a changes field to the
+	folder.
+
 	* Makefile.am (libcamel_la_SOURCES): Added
 	camel-folder-thread.[ch].
 
diff --git a/camel/camel-folder-search.c b/camel/camel-folder-search.c
index ff9539c9ec..4d61b5cfa1 100644
--- a/camel/camel-folder-search.c
+++ b/camel/camel-folder-search.c
@@ -39,11 +39,13 @@
 #include "camel-mime-message.h"
 #include "gmime-content-field.h"
 #include "camel-stream-mem.h"
+#include "e-util/e-memory.h"
 
-#define d(x) x
-#define r(x) x
+#define d(x)
+#define r(x)
 
 struct _CamelFolderSearchPrivate {
+	GHashTable *mempool_hash;
 };
 
 #define _PRIVATE(o) (((CamelFolderSearch *)(o))->priv)
@@ -88,16 +90,39 @@ camel_folder_search_init (CamelFolderSearch *obj)
 	p = _PRIVATE(obj) = g_malloc0(sizeof(*p));
 
 	obj->sexp = e_sexp_new();
+
+	/* use a hash of mempools to associate the returned uid lists with
+	   the backing mempool.  yes pretty weird, but i didn't want to change
+	   the api just yet */
+
+	p->mempool_hash = g_hash_table_new(0, 0);
+}
+
+static void
+free_mempool(void *key, void *value, void *data)
+{
+	GPtrArray *uids = key;
+	EMemPool *pool = value;
+
+	g_warning("Search closed with outstanding result unfreed: %p", uids);
+
+	g_ptr_array_free(uids, TRUE);
+	e_mempool_destroy(pool);
 }
 
 static void
 camel_folder_search_finalize (CamelObject *obj)
 {
 	CamelFolderSearch *search = (CamelFolderSearch *)obj;
+	struct _CamelFolderSearchPrivate *p = _PRIVATE(obj);
+
 	if (search->sexp)
 		camel_object_unref((CamelObject *)search->sexp);
 
 	g_free(search->last_search);
+	g_hash_table_foreach(p->mempool_hash, free_mempool, obj);
+	g_hash_table_destroy(p->mempool_hash);
+	g_free(p);
 }
 
 CamelType
@@ -266,6 +291,8 @@ camel_folder_search_execute_expression(CamelFolderSearch *search, const char *ex
 	GPtrArray *matches = g_ptr_array_new ();
 	int i;
 	GHashTable *results;
+	EMemPool *pool;
+	struct _CamelFolderSearchPrivate *p = _PRIVATE(search);
 
 	/* only re-parse if the search has changed */
 	if (search->last_search == NULL
@@ -281,6 +308,11 @@ camel_folder_search_execute_expression(CamelFolderSearch *search, const char *ex
 	if (r
 	    && r->type == ESEXP_RES_ARRAY_PTR) {
 		d(printf("got result ...\n"));
+		/* we use a mempool to store the strings, packed in tight as possible, and freed together */
+		/* because the strings are often short (like <8 bytes long), we would be wasting appx 50%
+		   of memory just storing the size tag that malloc assigns us and alignment padding, so this
+		   gets around that (and is faster to allocate and free as a bonus) */
+		pool = e_mempool_new(512, 256, E_MEMPOOL_ALIGN_BYTE);
 		if (search->summary) {
 			/* reorder result in summary order */
 			results = g_hash_table_new(g_str_hash, g_str_equal);
@@ -291,17 +323,25 @@ camel_folder_search_execute_expression(CamelFolderSearch *search, const char *ex
 			for (i=0;i<search->summary->len;i++) {
 				CamelMessageInfo *info = g_ptr_array_index(search->summary, i);
 				if (g_hash_table_lookup(results, info->uid)) {
-					g_ptr_array_add(matches, g_strdup(info->uid));
+					char *s = e_mempool_alloc(pool, strlen(info->uid) + 1);
+					strcpy(s, info->uid);
+					g_ptr_array_add(matches, s);
 				}
 			}
 			g_hash_table_destroy(results);
 		} else {
 			for (i=0;i<r->value.ptrarray->len;i++) {
+				char *s = e_mempool_alloc(pool, strlen(g_ptr_array_index(r->value.ptrarray, i)) + 1);
 				d(printf("adding match: %s\n", (char *)g_ptr_array_index(r->value.ptrarray, i)));
-				g_ptr_array_add(matches, g_strdup(g_ptr_array_index(r->value.ptrarray, i)));
+				strcpy(s, g_ptr_array_index(r->value.ptrarray, i));
+				g_ptr_array_add(matches, s);
 			}
 		}
 		e_sexp_result_free(r);
+		/* instead of putting the mempool_hash in the structure, we keep the api clean by
+		   putting a reference to it in a hashtable.  Lets us do some debugging and catch
+		   unfree'd results as well. */
+		g_hash_table_insert(p->mempool_hash, matches, pool);
 	} else {
 		printf("no result!\n");
 	}
@@ -314,12 +354,52 @@ camel_folder_search_execute_expression(CamelFolderSearch *search, const char *ex
 	return matches;
 }
 
+/**
+ * camel_folder_search_match_expression:
+ * @search: 
+ * @expr: 
+ * @info: 
+ * @ex: 
+ * 
+ * Returns #TRUE if the expression matches the specific message info @info.
+ * Note that the folder and index may need to be set for body searches to
+ * operate as well.
+ * 
+ * Return value: 
+ **/
+gboolean
+camel_folder_search_match_expression(CamelFolderSearch *search, const char *expr, const CamelMessageInfo *info, CamelException *ex)
+{
+	GPtrArray *uids;
+	int ret = FALSE;
+
+	search->match1 = (CamelMessageInfo *)info;
+
+	uids = camel_folder_search_execute_expression(search, expr, ex);
+	if (uids) {
+		if (uids->len == 1)
+			ret = TRUE;
+		camel_folder_search_free_result(search, uids);
+	}
+	search->match1 = NULL;
+
+	return ret;
+}
+
 void camel_folder_search_free_result(CamelFolderSearch *search, GPtrArray *result)
 {
 	int i;
+	struct _CamelFolderSearchPrivate *p = _PRIVATE(search);
+	EMemPool *pool;
 
-	for (i=0;i<result->len;i++)
-		g_free(g_ptr_array_index(result, i));
+	pool = g_hash_table_lookup(p->mempool_hash, result);
+	if (pool) {
+		e_mempool_destroy(pool);
+		g_hash_table_remove(p->mempool_hash, result);
+	} else {
+		for (i=0;i<result->len;i++)
+			g_free(g_ptr_array_index(result, i));
+	}
 	g_ptr_array_free(result, TRUE);
 }
 
@@ -352,6 +432,27 @@ search_match_all(struct _ESExp *f, int argc, struct _ESExpTerm **argv, CamelFold
 	r = e_sexp_result_new(ESEXP_RES_ARRAY_PTR);
 	r->value.ptrarray = g_ptr_array_new();
 
+	/* we are only matching a single message? */
+	if (search->match1) {
+		search->current = search->match1;
+
+		if (argc>0) {
+			r1 = e_sexp_term_eval(f, argv[0]);
+			if (r1->type == ESEXP_RES_BOOL) {
+				if (r1->value.bool)
+					g_ptr_array_add(r->value.ptrarray, search->current->uid);
+			} else {
+				g_warning("invalid syntax, matches require a single bool result");
+			}
+			e_sexp_result_free(r1);
+		} else {
+			g_ptr_array_add(r->value.ptrarray, search->current->uid);
+		}
+		search->current = NULL;
+
+		return r;
+	}
+
 	if (search->summary == NULL) {
 		/* TODO: make it work - e.g. use the folder and so forth for a slower search */
 		g_warning("No summary supplied, match-all doesn't work with no summary");
diff --git a/camel/camel-folder-search.h b/camel/camel-folder-search.h
index f74813e6f7..3f04881fde 100644
--- a/camel/camel-folder-search.h
+++ b/camel/camel-folder-search.h
@@ -46,6 +46,7 @@ struct _CamelFolderSearch {
 	CamelFolder *folder;	/* folder for current search */
 	GPtrArray *summary;	/* summary array for current search */
 	CamelMessageInfo *current; /* current message info, when searching one by one */
+	CamelMessageInfo *match1; /* message info, when searching a single message only */
 	CamelMimeMessage *current_message; /* cache of current message, if required */
 	ibex *body_index;
 };
@@ -95,6 +96,8 @@ void camel_folder_search_set_folder(CamelFolderSearch *search, CamelFolder *fold
 void camel_folder_search_set_summary(CamelFolderSearch *search, GPtrArray *summary);
 void camel_folder_search_set_body_index(CamelFolderSearch *search, ibex *index);
 GPtrArray *camel_folder_search_execute_expression(CamelFolderSearch *search, const char *expr, CamelException *ex);
+gboolean camel_folder_search_match_expression(CamelFolderSearch *search, const char *expr,
+					      const CamelMessageInfo *info, CamelException *ex);
 void camel_folder_search_free_result(CamelFolderSearch *search, GPtrArray *);
 
 #endif /* ! _CAMEL_FOLDER_SEARCH_H */
diff --git a/camel/camel-folder.c b/camel/camel-folder.c
index f2fd6a4cec..1453f91f3f 100644
--- a/camel/camel-folder.c
+++ b/camel/camel-folder.c
@@ -30,6 +30,7 @@
 #include "camel-store.h"
 #include "camel-mime-message.h"
 #include "string-utils.h"
+#include "e-util/e-memory.h"
 
 static CamelObjectClass *parent_class = NULL;
 
@@ -1114,10 +1115,37 @@ camel_folder_change_info_new(void)
 	info->uid_removed = g_ptr_array_new();
 	info->uid_changed = g_ptr_array_new();
 	info->uid_source = NULL;
+	info->uid_pool = e_mempool_new(512, 256, E_MEMPOOL_ALIGN_BYTE);
 
 	return info;
 }
 
+static void
+change_info_add_uid(CamelFolderChangeInfo *info, GPtrArray *uids, const char *uid, int copy)
+{
+	int i;
+
+	/* TODO: Check that it is in the other arrays and remove it from them/etc? */
+	for (i=0;i<uids->len;i++) {
+		if (!strcmp(uids->pdata[i], uid))
+			return;
+	}
+	if (copy)
+		g_ptr_array_add(uids, e_mempool_strdup(info->uid_pool, uid));
+	else
+		g_ptr_array_add(uids, uid);
+}
+
+static void
+change_info_cat(CamelFolderChangeInfo *info, GPtrArray *uids, GPtrArray *source)
+{
+	int i;
+
+	for (i=0;i<source->len;i++) {
+		change_info_add_uid(info, uids, source->pdata[i], TRUE);
+	}
+}
+
 /**
  * camel_folder_change_info_add_source:
  * @info: 
@@ -1132,7 +1160,7 @@ camel_folder_change_info_add_source(CamelFolderChangeInfo *info, const char *uid
 		info->uid_source = g_hash_table_new(g_str_hash, g_str_equal);
 
 	if (g_hash_table_lookup(info->uid_source, uid) == NULL)
-		g_hash_table_insert(info->uid_source, g_strdup(uid), (void *)1);
+		g_hash_table_insert(info->uid_source, e_mempool_strdup(info->uid_pool, uid), (void *)1);
 }
 
 /**
@@ -1154,7 +1182,7 @@ camel_folder_change_info_add_source_list(CamelFolderChangeInfo *info, const GPtr
 		char *uid = list->pdata[i];
 
 		if (g_hash_table_lookup(info->uid_source, uid) == NULL)
-			g_hash_table_insert(info->uid_source, g_strdup(uid), (void *)1);
+			g_hash_table_insert(info->uid_source, e_mempool_strdup(info->uid_pool, uid), (void *)1);
 	}
 }
 
@@ -1172,15 +1200,14 @@ camel_folder_change_info_add_update(CamelFolderChangeInfo *info, const char *uid
 	int value;
 
 	if (info->uid_source == NULL) {
-		camel_folder_change_info_add_uid(info, uid);
+		change_info_add_uid(info, info->uid_added, uid, TRUE);
 		return;
 	}
 
 	if (g_hash_table_lookup_extended(info->uid_source, uid, (void **)&key, (void **)&value)) {
 		g_hash_table_remove(info->uid_source, key);
-		g_free(key);
 	} else {
-		camel_folder_change_info_add_uid(info, uid);
+		change_info_add_uid(info, info->uid_added, uid, TRUE);
 	}
 }
 
@@ -1204,14 +1231,8 @@ camel_folder_change_info_add_update_list(CamelFolderChangeInfo *info, const GPtr
 static void
 change_info_remove(char *key, void *value, CamelFolderChangeInfo *info)
 {
-	camel_folder_change_info_remove_uid(info, key);
-	g_free(key);
-}
-
-static void
-change_info_free_update(char *key, void *value, CamelFolderChangeInfo *info)
-{
-	g_free(key);
+	/* we dont need to copy this, as they've already been copied into our pool */
+	change_info_add_uid(info, info->uid_removed, key, FALSE);
 }
 
 /**
@@ -1231,29 +1252,6 @@ camel_folder_change_info_build_diff(CamelFolderChangeInfo *info)
 	}
 }
 
-static void
-change_info_add_uid(CamelFolderChangeInfo *info, GPtrArray *uids, const char *uid)
-{
-	int i;
-
-	/* TODO: Check that it is in the other arrays and remove it from them/etc? */
-	for (i=0;i<uids->len;i++) {
-		if (!strcmp(uids->pdata[i], uid))
-			return;
-	}
-	g_ptr_array_add(uids, g_strdup(uid));
-}
-
-static void
-change_info_cat(CamelFolderChangeInfo *info, GPtrArray *uids, GPtrArray *source)
-{
-	int i;
-
-	for (i=0;i<source->len;i++) {
-		change_info_add_uid(info, uids, source->pdata[i]);
-	}
-}
-
 /**
  * camel_folder_change_info_cat:
  * @info: 
@@ -1280,7 +1278,7 @@ camel_folder_change_info_cat(CamelFolderChangeInfo *info, CamelFolderChangeInfo
 void
 camel_folder_change_info_add_uid(CamelFolderChangeInfo *info, const char *uid)
 {
-	change_info_add_uid(info, info->uid_added, uid);
+	change_info_add_uid(info, info->uid_added, uid, TRUE);
 }
 
 /**
@@ -1293,7 +1291,7 @@ camel_folder_change_info_add_uid(CamelFolderChangeInfo *info, const char *uid)
 void
 camel_folder_change_info_remove_uid(CamelFolderChangeInfo *info, const char *uid)
 {
-	change_info_add_uid(info, info->uid_removed, uid);
+	change_info_add_uid(info, info->uid_removed, uid, TRUE);
 }
 
 /**
@@ -1306,18 +1304,7 @@ camel_folder_change_info_remove_uid(CamelFolderChangeInfo *info, const char *uid
 void
 camel_folder_change_info_change_uid(CamelFolderChangeInfo *info, const char *uid)
 {
-	change_info_add_uid(info, info->uid_changed, uid);
-}
-
-static void
-change_info_clear(GPtrArray *uids)
-{
-	int i;
-
-	for (i=0;i<uids->len;i++) {
-		g_free(uids->pdata[i]);
-	}
-	g_ptr_array_set_size(uids, 0);
+	change_info_add_uid(info, info->uid_changed, uid, TRUE);
 }
 
 /**
@@ -1343,9 +1330,14 @@ camel_folder_change_info_changed(CamelFolderChangeInfo *info)
 void
 camel_folder_change_info_clear(CamelFolderChangeInfo *info)
 {
-	change_info_clear(info->uid_added);
-	change_info_clear(info->uid_removed);
-	change_info_clear(info->uid_changed);
+	g_ptr_array_set_size(info->uid_added, 0);
+	g_ptr_array_set_size(info->uid_removed, 0);
+	g_ptr_array_set_size(info->uid_changed, 0);
+	if (info->uid_source) {
+		g_hash_table_destroy(info->uid_source);
+		info->uid_source = NULL;
+	}
+	e_mempool_flush(info->uid_pool, TRUE);
 }
 
 /**
@@ -1357,12 +1349,10 @@ camel_folder_change_info_clear(CamelFolderChangeInfo *info)
 void
 camel_folder_change_info_free(CamelFolderChangeInfo *info)
 {
-	if (info->uid_source) {
-		g_hash_table_foreach(info->uid_source, (GHFunc)change_info_free_update, info);
+	if (info->uid_source)
 		g_hash_table_destroy(info->uid_source);
-	}
 
-	camel_folder_change_info_clear(info);
+	e_mempool_destroy(info->uid_pool);
 
 	g_ptr_array_free(info->uid_added, TRUE);
 	g_ptr_array_free(info->uid_removed, TRUE);
diff --git a/camel/camel-folder.h b/camel/camel-folder.h
index f7a3659f31..4060d36731 100644
--- a/camel/camel-folder.h
+++ b/camel/camel-folder.h
@@ -49,6 +49,7 @@ struct _CamelFolderChangeInfo {
 	GPtrArray *uid_changed;
 
 	GHashTable *uid_source;	/* used to create unique lists */
+	struct _EMemPool *uid_pool;	/* pool used to store copies of uid strings */
 };
 
 struct _CamelFolder
diff --git a/camel/camel-mime-parser.c b/camel/camel-mime-parser.c
index b019fa5ee0..0d151b7e9b 100644
--- a/camel/camel-mime-parser.c
+++ b/camel/camel-mime-parser.c
@@ -784,7 +784,10 @@ camel_mime_parser_read(CamelMimeParser *m, const char **databuffer, int len)
 	if (len == 0)
 		return 0;
 
+	d(printf("parser::read() reading %d bytes\n", len));
+
 	there = MIN(s->inend - s->inptr, len);
+	d(printf("parser::read() there = %d bytes\n", there));
 	if (there > 0) {
 		*databuffer = s->inptr;
 		s->inptr += there;
@@ -795,6 +798,8 @@ camel_mime_parser_read(CamelMimeParser *m, const char **databuffer, int len)
 		return -1;
 
 	there = MIN(s->inend - s->inptr, len);
+	d(printf("parser::read() had to re-read, now there = %d bytes\n", there));
+
 	*databuffer = s->inptr;
 	s->inptr += there;
 
diff --git a/camel/camel-mime-part-utils.c b/camel/camel-mime-part-utils.c
index 2809346a00..778255393e 100644
--- a/camel/camel-mime-part-utils.c
+++ b/camel/camel-mime-part-utils.c
@@ -35,7 +35,7 @@
 #include "camel-mime-filter-charset.h"
 #include "camel-mime-filter-crlf.h"
 
-#define d(x)
+#define d(x) /*(printf("%s(%d): ", __FILE__, __LINE__),(x))*/
 
 /* simple data wrapper */
 static void
@@ -114,6 +114,7 @@ simple_data_wrapper_construct_from_parser(CamelDataWrapper *dw, CamelMimeParser
 		start = camel_mime_parser_tell(mp) + seekable_source->bound_start;
 	}
 	while ( camel_mime_parser_step(mp, &buf, &len) != HSCAN_BODY_END ) {
+		d(printf("appending o/p data: %.*s\n", len, buf));
 		if (buffer) {
 			if (buffer->len > 20480 && seekable_source) {
 				/* is this a 'big' message?  Yes?  We dont want to convert it all then.*/
diff --git a/camel/camel-mime-part.c b/camel/camel-mime-part.c
index 61c0fb7f1f..5fb2775ca1 100644
--- a/camel/camel-mime-part.c
+++ b/camel/camel-mime-part.c
@@ -40,7 +40,7 @@
 #include "camel-mime-filter-charset.h"
 #include "camel-exception.h"
 
-#define d(x)
+#define d(x) /*(printf("%s(%d): ", __FILE__, __LINE__),(x))*/
 
 typedef enum {
 	HEADER_UNKNOWN,
diff --git a/camel/providers/local/camel-mh-summary.c b/camel/providers/local/camel-mh-summary.c
index 2f3c829e8d..c28ee1f495 100644
--- a/camel/providers/local/camel-mh-summary.c
+++ b/camel/providers/local/camel-mh-summary.c
@@ -297,23 +297,32 @@ mh_summary_sync_message(CamelLocalSummary *cls, CamelMessageInfo *info, CamelExc
 	camel_mime_parser_init_with_fd(mp, fd);
 	if (camel_mime_parser_step(mp, 0, 0) != HSCAN_EOF) {
 		xev = camel_mime_parser_header(mp, "X-Evolution", &xevoffset);
+		d(printf("xev = '%s'\n", xev));
 		xevnew = camel_local_summary_encode_x_evolution(cls, info);
 		if (xev == NULL
 		    || camel_local_summary_decode_x_evolution(cls, xev, NULL) == -1
-		    || strlen(xev)+1 != strlen(xevnew)) {
+		    || strlen(xev)-1 != strlen(xevnew)) {
+
+			d(printf("camel local summary_decode_xev = %d\n", camel_local_summary_decode_x_evolution(cls, xev, NULL)));
 
 			/* need to write a new copy/unlink old */
 			tmpname = g_strdup_printf("%s/.tmp.%d.%s", cls->folder_path, getpid(), info->uid);
+			d(printf("old xev was %d %s new xev is %d %s\n", strlen(xev), xev, strlen(xevnew), xevnew));
+			d(printf("creating new message %s\n", tmpname));
 			outfd = open(tmpname, O_CREAT|O_WRONLY|O_TRUNC, 0600);
 			if (outfd != -1) {
 				outlen = 0;
-				if ( (len = camel_local_summary_write_headers(outfd, camel_mime_parser_headers_raw(mp), xevnew)) == 0) {
+				len = camel_local_summary_write_headers(outfd, camel_mime_parser_headers_raw(mp), xevnew);
+				if (len != -1) {
 					while (outlen != -1 && (len = camel_mime_parser_read(mp, &buffer, 10240)) > 0) {
+						d(printf("camel mime parser read, read %d bytes: %.*s\n", len, len, buffer));
 						do {
-							outlen = write(fd, buffer, len);
+							outlen = write(outfd, buffer, len);
 						} while (outlen == -1 && errno == EINTR);
 					}
 				}
+
+				d(printf("len = %d outlen = %d, renaming/finishing\n", len, outlen));
 				if (close(outfd) == -1
 				    || len == -1
 				    || outlen == -1
@@ -326,6 +335,7 @@ mh_summary_sync_message(CamelLocalSummary *cls, CamelMessageInfo *info, CamelExc
 			}
 			g_free(tmpname);
 		} else {
+			d(printf("stamping in updated X-EV at %d\n", (int)xevoffset));
 			/* else, we can just update the flags field */
 			lseek(fd, xevoffset+strlen("X-Evolution: "), SEEK_SET);
 			do {
diff --git a/camel/providers/vee/camel-vee-folder.c b/camel/providers/vee/camel-vee-folder.c
index da47761754..77c3017179 100644
--- a/camel/providers/vee/camel-vee-folder.c
+++ b/camel/providers/vee/camel-vee-folder.c
@@ -24,6 +24,9 @@
 #include "camel-vee-folder.h"
 #include "camel-folder-summary.h"
 #include "camel-mime-message.h"
+#ifdef DYNAMIC
+#include "camel-folder-search.h"
+#endif
 
 #include <string.h>
 
@@ -128,6 +131,11 @@ camel_vee_folder_init (CamelVeeFolder *obj)
 		CAMEL_MESSAGE_DRAFT |
 		CAMEL_MESSAGE_FLAGGED |
 		CAMEL_MESSAGE_SEEN;
+
+	obj->changes = camel_folder_change_info_new();
+#ifdef DYNAMIC
+	obj->search = camel_folder_search_new();
+#endif
 }
 
 static void
@@ -137,12 +145,19 @@ camel_vee_folder_finalise (CamelObject *obj)
 	struct _CamelVeeFolderPrivate *p = _PRIVATE(vf);
 	GList *node;
 
+	/* FIXME: some leaks here, summary etc */
+
 	node = p->folders;
 	while (node) {
 		CamelFolder *f = node->data;
 		camel_object_unref((CamelObject *)f);
 		node = g_list_next(node);
 	}
+
+	camel_folder_change_info_free(vf->changes);
+#ifdef DYNAMIC
+	camel_object_unref((CamelObject *)vf->search);
+#endif
 }
 
 /**
@@ -156,8 +171,7 @@ camel_vee_folder_finalise (CamelObject *obj)
  * Return value: A new CamelVeeFolder widget.
  **/
 CamelFolder *
-camel_vee_folder_new (CamelStore *parent_store, const char *name,
-		      CamelException *ex)
+camel_vee_folder_new (CamelStore *parent_store, const char *name, CamelException *ex)
 {
 	CamelFolder *folder;
 	CamelVeeFolder *vf;
@@ -192,16 +206,136 @@ camel_vee_folder_new (CamelStore *parent_store, const char *name,
 	return folder;
 }
 
+#ifdef DYNAMIC
+static void
+vfolder_remove_match(CamelVeeFolder *vf, CamelVeeMessageInfo *vinfo)
+{
+	printf("removing match %s\n", vinfo->info.uid);
+
+	g_hash_table_remove(vf->messages_uid, vinfo->info.uid);
+	g_ptr_array_remove_fast(vf->messages, vinfo);
+	camel_folder_change_info_remove_uid(vf->changes, vinfo->info.uid);
+	camel_message_info_free((CamelMessageInfo *)vinfo);
+}
+
+static CamelVeeMessageInfo *
+vfolder_add_match(CamelVeeFolder *vf, CamelFolder *f, const CamelMessageInfo *info)
+{
+	CamelVeeMessageInfo *mi;
+
+	mi = g_malloc0(sizeof(*mi));
+	camel_message_info_dup_to(info, (CamelMessageInfo*)mi);
+	g_free (mi->info.uid);
+	mi->info.uid = g_strdup_printf("%p:%s", f, info->uid);
+	mi->folder = f;
+	g_ptr_array_add(vf->messages, mi);
+	g_hash_table_insert(vf->messages_uid, mi->info.uid, mi);
+
+	printf("adding match %s\n", mi->info.uid);
+
+	camel_folder_change_info_add_uid(vf->changes, mi->info.uid);
+	return mi;
+}
+#endif
+
+static void
+vfolder_change_match(CamelVeeFolder *vf, CamelVeeMessageInfo *vinfo, const CamelMessageInfo *info)
+{
+	CamelFlag *flag;
+	CamelTag *tag;
+
+	printf("changing match %s\n", vinfo->info.uid);
+
+	vinfo->info.flags = info->flags;
+	camel_flag_list_free(&vinfo->info.user_flags);
+	flag = info->user_flags;
+	while (flag) {
+		camel_flag_set(&vinfo->info.user_flags, flag->name, TRUE);
+		flag = flag->next;
+	}
+	camel_tag_list_free(&vinfo->info.user_tags);
+	tag = info->user_tags;
+	while (tag) {
+		camel_tag_set(&vinfo->info.user_tags, tag->name, tag->value);
+		tag = tag->next;
+	}
+	camel_folder_change_info_change_uid(vf->changes, vinfo->info.uid);
+}
+
 static void
 folder_changed(CamelFolder *sub, gpointer type, CamelVeeFolder *vf)
 {
 	CamelException *ex;
 
-	ex = camel_exception_new();
-	vee_folder_build_folder(vf, sub, ex);
-	camel_exception_free(ex);
-	/* FIXME: should only raise follow-on event if the result changed */
-	camel_object_trigger_event( CAMEL_OBJECT(vf), "folder_changed", GINT_TO_POINTER(0));
+#ifdef DYNAMIC
+	CamelFolderChangeInfo *changes = type;
+
+	/* assume its faster to search a long list in whole, than by part */
+	if (changes && (changes->uid_added->len + changes->uid_changed->len) < 500) {
+		int i;
+		char *vuid;
+		CamelVeeMessageInfo *vinfo;
+		gboolean match;
+		const CamelMessageInfo *info;
+
+		ex = camel_exception_new();
+
+		/* see if added ones now match us */
+		for (i=0;i<changes->uid_added->len;i++) {
+			info = camel_folder_get_message_info(sub, changes->uid_added->pdata[i]);
+			if (info) {
+				camel_folder_search_set_folder(vf->search, sub);
+				match = camel_folder_search_match_expression(vf->search, vf->expression, info, ex);
+				if (match)
+					vinfo = vfolder_add_match(vf, sub, info);
+			}
+		}
+
+		/* check if changed ones still match */
+		for (i=0;i<changes->uid_changed->len;i++) {
+			info = camel_folder_get_message_info(sub, changes->uid_changed->pdata[i]);
+			vuid = g_strdup_printf("%p:%s", sub, (char *)changes->uid_changed->pdata[i]);
+			vinfo = (CamelVeeMessageInfo *)vee_get_message_info((CamelFolder *)vf, vuid);
+			if (info) {
+				camel_folder_search_set_folder(vf->search, sub);
+				match = camel_folder_search_match_expression(vf->search, vf->expression, info, ex);
+				if (vinfo) {
+					if (match)
+						vfolder_change_match(vf, vinfo, info);
+					else
+						vfolder_remove_match(vf, vinfo);
+				} else if (match)
+					vfolder_add_match(vf, sub, info);
+			} else if (vinfo)
+				vfolder_remove_match(vf, vinfo);
+
+			g_free(vuid);
+		}
+
+		camel_exception_free(ex);
+
+		/* mirror removes directly, if they used to match */
+		for (i=0;i<changes->uid_removed->len;i++) {
+			vuid = g_strdup_printf("%p:%s", sub, (char *)changes->uid_removed->pdata[i]);
+			vinfo = (CamelVeeMessageInfo *)vee_get_message_info((CamelFolder *)vf, vuid);
+			if (vinfo)
+				vfolder_remove_match(vf, vinfo);
+			g_free(vuid);
+		}
+	} else {
+#endif
+		ex = camel_exception_new();
+		vee_folder_build_folder(vf, sub, ex);
+		camel_exception_free(ex);
+#ifdef DYNAMIC
+	}
+#endif
+
+	/* cascade up, if we need to */
+	if (camel_folder_change_info_changed(vf->changes)) {
+		camel_object_trigger_event( CAMEL_OBJECT(vf), "folder_changed", vf->changes);
+		camel_folder_change_info_clear(vf->changes);
+	}
 }
 
 /* track flag changes in the summary */
@@ -209,24 +343,49 @@ static void
 message_changed(CamelFolder *f, const char *uid, CamelVeeFolder *mf)
 {
 	const CamelMessageInfo *info;
-	CamelMessageInfo *vinfo;
-	CamelFlag *flag;
+	CamelVeeMessageInfo *vinfo;
 	char *vuid;
+#ifdef DYNAMIC
+	gboolean match;
+	CamelException *ex;
+#endif
 
 	info = camel_folder_get_message_info(f, uid);
-
 	vuid = g_strdup_printf("%p:%s", f, uid);
-	vinfo = (CamelMessageInfo *)vee_get_message_info((CamelFolder *)mf, vuid);
-	if (info && vinfo) {
-		vinfo->flags = info->flags;
-		camel_flag_list_free(&vinfo->user_flags);
-		flag = info->user_flags;
-		while (flag) {
-			camel_flag_set(&vinfo->user_flags, flag->name, TRUE);
-			flag = flag->next;
-		}
-		camel_object_trigger_event( CAMEL_OBJECT(mf), "message_changed", vinfo->uid);
+	vinfo = (CamelVeeMessageInfo *)vee_get_message_info((CamelFolder *)mf, vuid);
+
+	/* see if this message now matches/doesn't match anymore */
+
+	/* Hmm, this might not work if the folder uses some weird search thing,
+	   and/or can be slow since it wont use any index index, hmmm. */
+
+#ifdef DYNAMIC
+	camel_folder_search_set_folder(mf->search, f);
+	ex = camel_exception_new();
+	match = camel_folder_search_match_expression(mf->search, mf->expression, info, ex);
+	camel_exception_free(ex);
+	if (info) {
+		if (vinfo) {
+			if (match)
+				vfolder_change_match(mf, vinfo, info);
+			else
+				vfolder_remove_match(mf, vinfo);
+		} else if (match)
+			vfolder_add_match(mf, f, info);
+	} else if (vinfo)
+		vfolder_remove_match(mf, vinfo);
+#else
+	if (info && vinfo)
+		vfolder_change_match(mf, vinfo, info);
+#endif
+
+	/* cascade up, if required.  This could probably be delayed,
+	   but doesn't matter really, that is what freeze is for. */
+	if (camel_folder_change_info_changed(mf->changes)) {
+		camel_object_trigger_event( CAMEL_OBJECT(mf), "folder_changed", mf->changes);
+		camel_folder_change_info_clear(mf->changes);
 	}
+
 	g_free(vuid);
 }
 
@@ -245,8 +404,18 @@ camel_vee_folder_add_folder(CamelVeeFolder *vf, CamelFolder *sub)
 	ex = camel_exception_new();
 	vee_folder_build_folder(vf, sub, ex);
 	camel_exception_free(ex);
-	/* FIXME: should only raise follow-on event if the result changed */
-	camel_object_trigger_event( CAMEL_OBJECT(vf), "folder_changed", GINT_TO_POINTER(0));
+
+	/* we'll assume the caller is going to update the whole list after they do this
+	   this may or may not be the right thing to do, but it should be close enough */
+#if 0	
+	if (camel_folder_change_info_changed(vf->changes)) {
+		camel_object_trigger_event( CAMEL_OBJECT(vf), "folder_changed", vf->changes);
+		camel_folder_change_info_clear(vf->changes);
+	}
+#else
+	camel_folder_change_info_clear(vf->changes);
+#endif
+	
 }
 
 
@@ -441,6 +610,11 @@ vee_folder_build(CamelVeeFolder *vf, CamelException *ex)
 	messages = g_ptr_array_new();
 	messages_uid = g_hash_table_new(g_str_hash, g_str_equal);
 
+	g_ptr_array_free(vf->messages, TRUE);
+	vf->messages = messages;
+	g_hash_table_destroy(vf->messages_uid);
+	vf->messages_uid = messages_uid;
+
 	node = p->folders;
 	while (node) {
 		GPtrArray *matches;
@@ -465,11 +639,6 @@ vee_folder_build(CamelVeeFolder *vf, CamelException *ex)
 		camel_folder_search_free(f, matches);
 		node = g_list_next(node);
 	}
-
-	g_ptr_array_free(vf->messages, TRUE);
-	vf->messages = messages;
-	g_hash_table_destroy(vf->messages_uid);
-	vf->messages_uid = messages_uid;
 }
 
 
@@ -489,6 +658,7 @@ vee_folder_build_folder(CamelVeeFolder *vf, CamelFolder *source, CamelException
 	for (i=0;i<vf->messages->len;i++) {
 		CamelVeeMessageInfo *mi = g_ptr_array_index(vf->messages, i);
 		if (mi->folder == source) {
+			camel_folder_change_info_add_source(vf->changes, mi->info.uid);
 			g_hash_table_remove(vf->messages_uid, mi->info.uid);
 			g_ptr_array_remove_index_fast(vf->messages, i);
 
@@ -511,9 +681,13 @@ vee_folder_build_folder(CamelVeeFolder *vf, CamelFolder *source, CamelException
 			mi->folder = f;
 			g_ptr_array_add(messages, mi);
 			g_hash_table_insert(messages_uid, mi->info.uid, mi);
+
+			camel_folder_change_info_add_update(vf->changes, mi->info.uid);
 		}
 	}
 	camel_folder_search_free(f, matches);
+
+	camel_folder_change_info_build_diff(vf->changes);
 }
 
 
diff --git a/camel/providers/vee/camel-vee-folder.h b/camel/providers/vee/camel-vee-folder.h
index 2263854912..8ea76a6227 100644
--- a/camel/providers/vee/camel-vee-folder.h
+++ b/camel/providers/vee/camel-vee-folder.h
@@ -24,6 +24,9 @@
 
 #include <camel/camel-folder.h>
 
+/* try the dynamic update version */
+#define DYNAMIC
+
 #define CAMEL_VEE_FOLDER(obj)         CAMEL_CHECK_CAST (obj, camel_vee_folder_get_type (), CamelVeeFolder)
 #define CAMEL_VEE_FOLDER_CLASS(klass) CAMEL_CHECK_CLASS_CAST (klass, camel_vee_folder_get_type (), CamelVeeFolderClass)
 #define IS_CAMEL_VEE_FOLDER(obj)      CAMEL_CHECK_TYPE (obj, camel_vee_folder_get_type ())
@@ -43,6 +46,10 @@ struct _CamelVeeFolder {
 	/* FIXME: Move this to a summary object??? */
 	GPtrArray *messages; /* message info's */
 	GHashTable *messages_uid;
+	CamelFolderChangeInfo *changes;
+#ifdef DYNAMIC
+	CamelFolderSearch *search;
+#endif
 };
 
 struct _CamelVeeFolderClass {