Fixes for the summary messageid changes. Hash the messageid and store it.

2000-11-20  Not Zed  <NotZed@HelixCode.com>

	* providers/nntp/camel-nntp-utils.c (get_XOVER_headers): Fixes for
	the summary messageid changes.  Hash the messageid and store it.
	(get_XOVER_headers): Use camel_folder_summary_info_new() to create
	the summary item before adding it.

	* camel-folder-summary.h (CamelMessageInfo): Changed the
	messgae-id to be an 8 byte md5 hash, and the references list to be
	an array of these.

	* providers/local/camel-mh-summary.c (mh_summary_sync_message):
	New function, sync out the message info stuff.  Only updates the
	X-Ev header if it can get away with it, otherwise writes out a
	whole new message.
	(mh_summary_sync): Added more functionality.  All summary info is
	now written to the X-Ev header, etc, and new messages re-written
	if required during the sync process.

	* providers/local/camel-local-folder.c
	(local_set_message_user_flag): Set the XEVCHANGE flag.
	(local_set_message_user_tag): And here too.

	* providers/local/camel-local-summary.h: New flag
	CAMEL_MESSAGE_FOLDER_XEVCHANGE to indicate the XEV header has
	probably changed size and needs to be rewritten in whole.

	* camel-folder-summary.c (next_uid_string): Want this static, not
	const.
	(message_info_new): Store the references and message-id values as
	64 bit, binary hashes.
	(message_info_load): fix for message-id/references changes.
	(message_info_save): Likewise.
	(camel_message_info_dup_to): And here.
	(camel_message_info_free): And here too.  No longer free
	message_id, and simple free for references array.
	(CAMEL_FOLDER_SUMMARY_VERSION): Bumped file revision.
	(camel_folder_summary_init): Init memchunk allocators to empty.
	(camel_folder_summary_finalize): Free memchunk allocators if
	there.
	(message_info_new): Use the chunk allocator to allocate message
	info's.
	(camel_folder_summary_info_new): New helper to allocate the
	message info, and setup the memchunk if required.
	(content_info_alloc): Likewise for content info's.
	(message_info_load): Use summary_info_new_empty.
	(content_info_new): Use content_info_alloc.
	(content_info_load): "
	(content_info_free): Free the content info as a memchunk.
	(message_info_free): Free everything directly and the base as a
	memchunk, rather than calling camel_message_info_free(), which
	assumes a malloc'd array.

	* providers/local/camel-local-summary.c: Include ctype.h, kill a
	warning.
	(local_summary_decode_x_evolution): If we get a NULL message info,
	then dont try and set anything, just check for validity.
	(camel_local_summary_write_headers): New function to write a set
	of headers to an fd.
	(camel_local_summary_check): Added some statistic generation
	stuff for memory profiling.

	* providers/local/camel-mbox-summary.c (header_write): Changed to
	use stdoi functions to write out the header to a buffered stream,
	instead of using writev, which is apparently slow (and writing
	each line separately is slow anyway).
	(mbox_summary_sync_full): New implementation.  Does things
	differently, doesn't use or require the content info stuff.
	(summary_rebuild): Dont return an error if we start scanning at
	the end of file.
	(mbox_summary_sync_full): If we are not writing out new headers,
	make sure we copy the From line as we go, and update frompos
	appropriately.
	(mbox_summary_sync_full): Always copy the From line from the
	existing one, rather than trying to make one up ourselves.
	(mbox_summary_sync): If we can get by with a quick-sync, then try
	it, if that fails, then try a full sync anyway.
	(mbox_summary_sync_quick): Quick sync.  Only update system flags,
	etc.
	(mbox_summary_sync_full): Use the proper local summary encode_xev
	function.
	(header_evolution_decode): Removed, no longer needed.
	(header_evolution_encode): Same.
	(copy_block): No longer needed, removed.
	(header_write): Removed, replaced with
	camel_local_summary_write_headers.
	(mbox_summary_sync_full): Fixed for header_write change.

	* camel-mime-parser.c (folder_scan_step): Implement the new
	optional parser state HSCAN_PRE_FROM, that returns the (currently
	unfiltered) input data.
	(folder_scan_drop_step): Do the right thing for the PRE_FROM
	state.
	(camel_mime_parser_scan_from): Update the doco.
	(camel_mime_parser_scan_pre_from): Ok, make this behaviour
	optional, it simplifies a lot of loops that dont otherwise need to
	know about it.
	(folder_scan_step): Made the PRE_FROM state optional.
	(struct _header_scan_state): Made the bool vars 1 bit.
	(folder_pull_part): Free the from_line buffer if it is there.
	(folder_scan_skip_line): Added a new arg, can save the skpped data
	to a byte_array, as we go.
	(folder_scan_step): Fixed calls to skip_line approrpiately.  Now
	we save the from line as we parse it.
	(camel_mime_parser_read): New function to read from the mime
	parser buffer directly.  Useful if you use the parser to read the
	first/some headers, then need to scan the rest of the data,
	without needing to use a seek(), or allocate your own buffers.

	* camel-mime-parser.h (struct _header_state): Added a new parser state,
	pre-from which returns any data found before a from line during
	parsing (all other data can be retrieved by the caller except
	this).

svn path=/trunk/; revision=6618

1 files changed, 138 insertions, 33 deletions

diff --git a/camel/camel-folder-summary.c b/camel/camel-folder-summary.c
index efb8c1e195..12e276245d 100644
--- a/camel/camel-folder-summary.c
+++ b/camel/camel-folder-summary.c
@@ -40,6 +40,8 @@
 #include <camel/camel-stream-mem.h>
 
 #include "hash-table-utils.h"
+#include "e-util/md5-utils.h"
+#include "e-util/e-memory.h"
 
 /* this should probably be conditional on it existing */
 #define USE_BSEARCH
@@ -51,7 +53,7 @@
 extern int strdup_count, malloc_count, free_count;
 #endif
 
-#define CAMEL_FOLDER_SUMMARY_VERSION (9)
+#define CAMEL_FOLDER_SUMMARY_VERSION (10)
 
 struct _CamelFolderSummaryPrivate {
 	GHashTable *filter_charset;	/* CamelMimeFilterCharset's indexed by source charset */
@@ -91,7 +93,7 @@ static CamelMessageContentInfo * content_info_load(CamelFolderSummary *, FILE *)
 static int		         content_info_save(CamelFolderSummary *, FILE *, CamelMessageContentInfo *);
 static void		         content_info_free(CamelFolderSummary *, CamelMessageContentInfo *);
 
-const char *next_uid_string(CamelFolderSummary *s);
+static char *next_uid_string(CamelFolderSummary *s);
 
 static CamelMessageContentInfo * summary_build_content_info(CamelFolderSummary *s, CamelMimeParser *mp);
 static CamelMessageContentInfo * summary_build_content_info_message(CamelFolderSummary *s, CamelMessageInfo *msginfo, CamelMimePart *object);
@@ -139,6 +141,9 @@ camel_folder_summary_init (CamelFolderSummary *s)
 	s->message_info_size = sizeof(CamelMessageInfo);
 	s->content_info_size = sizeof(CamelMessageContentInfo);
 
+	s->message_info_chunks = NULL;
+	s->content_info_chunks = NULL;
+
 	s->version = CAMEL_FOLDER_SUMMARY_VERSION;
 	s->flags = 0;
 	s->time = 0;
@@ -171,6 +176,11 @@ camel_folder_summary_finalize (CamelObject *obj)
 
 	g_free(s->summary_path);
 
+	if (s->message_info_chunks)
+		e_memchunk_destroy(s->message_info_chunks);
+	if (s->content_info_chunks)
+		e_memchunk_destroy(s->content_info_chunks);
+
 	if (p->filter_index)
 		camel_object_unref ((CamelObject *)p->filter_index);
 	if (p->filter_64)
@@ -356,8 +366,6 @@ void camel_folder_summary_set_uid(CamelFolderSummary *s, guint32 uid)
 char *
 camel_folder_summary_next_uid_string(CamelFolderSummary *s)
 {
-	char *(*next_uid_string)(CamelFolderSummary *);
-
 	return ((CamelFolderSummaryClass *)(CAMEL_OBJECT_GET_CLASS(s)))->next_uid_string(s);
 }
 
@@ -505,7 +513,7 @@ camel_folder_summary_save(CamelFolderSummary *s)
  *
  * The @info record should have been generated by calling one of the
  * info_new_*() functions, as it will be free'd based on the summary
- * class.
+ * class.  And MUST NOT be allocated directly using malloc.
  **/
 void camel_folder_summary_add(CamelFolderSummary *s, CamelMessageInfo *info)
 {
@@ -1428,13 +1436,50 @@ camel_folder_summary_format_string(struct _header_raw *h, const char *name)
 	}
 }
 
+/**
+ * camel_folder_summary_info_new:
+ * @s: 
+ * 
+ * Allocate a new camel message info, suitable for adding
+ * to this summary.
+ * 
+ * Return value: 
+ **/
+CamelMessageInfo *
+camel_folder_summary_info_new(CamelFolderSummary *s)
+{
+	CamelMessageInfo *mi;
+
+	if (s->message_info_chunks == NULL)
+		s->message_info_chunks = e_memchunk_new(32, s->message_info_size);
+	mi = e_memchunk_alloc(s->message_info_chunks);
+	memset(mi, 0, s->message_info_size);
+	return mi;
+}
+
+static CamelMessageContentInfo *
+content_info_alloc(CamelFolderSummary *s)
+{
+	CamelMessageContentInfo *ci;
+
+	if (s->content_info_chunks == NULL)
+		s->content_info_chunks = e_memchunk_new(32, s->content_info_size);
+	ci = e_memchunk_alloc(s->content_info_chunks);
+	memset(ci, 0, s->content_info_size);
+	return ci;
+}
+
 static CamelMessageInfo *
 message_info_new(CamelFolderSummary *s, struct _header_raw *h)
 {
 	CamelMessageInfo *mi;
 	const char *received;
+	guchar digest[16];
+	struct _header_references *refs, *scan;
+	char *msgid;
+	int count;
 
-	mi = g_malloc0(s->message_info_size);
+	mi = camel_folder_summary_info_new(s);
 
 	mi->subject = camel_folder_summary_format_string(h, "subject");
 	mi->from = camel_folder_summary_format_address(h, "from");
@@ -1450,12 +1495,34 @@ message_info_new(CamelFolderSummary *s, struct _header_raw *h)
 		mi->date_received = header_decode_date(received + 1, NULL);
 	else
 		mi->date_received = 0;
-	mi->message_id = header_msgid_decode(header_raw_find(&h, "message-id", NULL));
+
+	msgid = header_msgid_decode(header_raw_find(&h, "message-id", NULL));
+	if (msgid) {
+		md5_get_digest(msgid, strlen(msgid), digest);
+		memcpy(mi->message_id.id.hash, digest, sizeof(mi->message_id.id.hash));
+		g_free(msgid);
+	}
 	/* if we have a references, use that, otherwise, see if we have an in-reply-to
 	   header, with parsable content, otherwise *shrug* */
-	mi->references = header_references_decode(header_raw_find(&h, "references", NULL));
-	if (mi->references == NULL)
-		mi->references = header_references_decode(header_raw_find(&h, "in-reply-to", NULL));
+	if ((refs = header_references_decode(header_raw_find(&h, "references", NULL))) != NULL
+	    || (refs = header_references_decode(header_raw_find(&h, "in-reply-to", NULL))) != NULL) {
+		count = header_references_list_size(&refs);
+		mi->references = g_malloc(sizeof(*mi->references) + ((count-1) * sizeof(mi->references->references[0])));
+		count = 0;
+		scan = refs;
+		while (scan) {
+			/* FIXME: the id might be NULL because of a small bug in camel-mime-utils */
+			if (scan->id) {
+				md5_get_digest(scan->id, strlen(scan->id), digest);
+				memcpy(mi->references->references[count].id.hash, digest, sizeof(mi->message_id.id.hash));
+				count++;
+			}
+			scan = scan->next;
+		}
+		mi->references->size = count;
+		header_references_list_clear(&refs);
+	}
+
 	return mi;
 }
 
@@ -1467,7 +1534,7 @@ message_info_load(CamelFolderSummary *s, FILE *in)
 	guint count;
 	int i;
 
-	mi = g_malloc0(s->message_info_size);
+	mi = camel_folder_summary_info_new(s);
 
 	io(printf("Loading message info\n"));
 
@@ -1482,13 +1549,17 @@ message_info_load(CamelFolderSummary *s, FILE *in)
 	camel_folder_summary_decode_string(in, &mi->cc);
 	mi->content = NULL;
 
-	camel_folder_summary_decode_string(in, &mi->message_id);
+	camel_folder_summary_decode_fixed_int32(in, &mi->message_id.id.part.hi);
+	camel_folder_summary_decode_fixed_int32(in, &mi->message_id.id.part.lo);
 
 	camel_folder_summary_decode_uint32(in, &count);
-	for (i=0;i<count;i++) {
-		char *id;
-		camel_folder_summary_decode_string(in, &id);
-		header_references_list_append_asis(&mi->references, id);
+	if (count > 0) {
+		mi->references = g_malloc(sizeof(*mi->references) + ((count-1) * sizeof(mi->references->references[0])));
+		mi->references->size = count;
+		for (i=0;i<count;i++) {
+			camel_folder_summary_decode_fixed_int32(in, &mi->references->references[i].id.part.hi);
+			camel_folder_summary_decode_fixed_int32(in, &mi->references->references[i].id.part.lo);
+		}
 	}
 
 	camel_folder_summary_decode_uint32(in, &count);
@@ -1518,7 +1589,7 @@ message_info_save(CamelFolderSummary *s, FILE *out, CamelMessageInfo *mi)
 	guint32 count;
 	CamelFlag *flag;
 	CamelTag *tag;
-	struct _header_references *refs;
+	int i;
 
 	io(printf("Saving message info\n"));
 
@@ -1532,14 +1603,17 @@ message_info_save(CamelFolderSummary *s, FILE *out, CamelMessageInfo *mi)
 	camel_folder_summary_encode_string(out, mi->to);
 	camel_folder_summary_encode_string(out, mi->cc);
 
-	camel_folder_summary_encode_string(out, mi->message_id);
+	camel_folder_summary_encode_fixed_int32(out, mi->message_id.id.part.hi);
+	camel_folder_summary_encode_fixed_int32(out, mi->message_id.id.part.lo);
 
-	count = header_references_list_size(&mi->references);
-	camel_folder_summary_encode_uint32(out, count);
-	refs = mi->references;
-	while (refs) {
-		camel_folder_summary_encode_string(out, refs->id);
-		refs = refs->next;
+	if (mi->references) {
+		camel_folder_summary_encode_uint32(out, mi->references->size);
+		for (i=0;i<mi->references->size;i++) {
+			camel_folder_summary_encode_fixed_int32(out, mi->references->references[i].id.part.hi);
+			camel_folder_summary_encode_fixed_int32(out, mi->references->references[i].id.part.lo);
+		}
+	} else {
+		camel_folder_summary_encode_uint32(out, 0);
 	}
 
 	count = camel_flag_list_size(&mi->user_flags);
@@ -1565,7 +1639,15 @@ message_info_save(CamelFolderSummary *s, FILE *out, CamelMessageInfo *mi)
 static void
 message_info_free(CamelFolderSummary *s, CamelMessageInfo *mi)
 {
-	camel_message_info_free(mi);
+	g_free(mi->uid);
+	g_free(mi->subject);
+	g_free(mi->from);
+	g_free(mi->to);
+	g_free(mi->cc);
+	g_free(mi->references);
+	camel_flag_list_free(&mi->user_flags);
+	camel_tag_list_free(&mi->user_tags);
+	e_memchunk_free(s->message_info_chunks, mi);
 }
 
 static CamelMessageContentInfo *
@@ -1573,7 +1655,7 @@ content_info_new(CamelFolderSummary *s, struct _header_raw *h)
 {
 	CamelMessageContentInfo *ci;
 
-	ci = g_malloc0(s->content_info_size);
+	ci = content_info_alloc(s);
 
 	ci->id = header_msgid_decode(header_raw_find(&h, "content-id", NULL));
 	ci->description = header_decode_string(header_raw_find(&h, "content-description", NULL));
@@ -1595,7 +1677,7 @@ content_info_load(CamelFolderSummary *s, FILE *in)
 
 	io(printf("Loading content info\n"));
 
-	ci = g_malloc0(s->content_info_size);
+	ci = content_info_alloc(s);
 
 	camel_folder_summary_decode_off_t(in, &ci->pos);
 	camel_folder_summary_decode_off_t(in, &ci->bodypos);
@@ -1666,10 +1748,10 @@ content_info_free(CamelFolderSummary *s, CamelMessageContentInfo *ci)
 	g_free(ci->id);
 	g_free(ci->description);
 	g_free(ci->encoding);
-	g_free(ci);
+	e_memchunk_free(s->content_info_chunks, ci);
 }
 
-const char *
+static char *
 next_uid_string(CamelFolderSummary *s)
 {
 	return g_strdup_printf("%u", camel_folder_summary_next_uid(s));
@@ -2068,10 +2150,18 @@ camel_message_info_dup_to(const CamelMessageInfo *from, CamelMessageInfo *to)
 	to->to = g_strdup(from->to);
 	to->cc = g_strdup(from->cc);
 	to->uid = g_strdup(from->uid);
-	to->message_id = g_strdup(from->message_id);
+	memcpy(&to->message_id, &from->message_id, sizeof(from->message_id));
 
 	/* Copy structures */
-	to->references = header_references_dup(from->references);
+	if (from->references) {
+		int len = sizeof(*from->references) + ((from->references->size-1) * sizeof(from->references->references[0]));
+
+		to->references = g_malloc(len);
+		memcpy(to->references, from->references, len);
+	} else {
+		to->references = NULL;
+	}
+
 	flag = from->user_flags;
 	while (flag) {
 		camel_flag_set(&to->user_flags, flag->name, TRUE);
@@ -2107,14 +2197,29 @@ camel_message_info_free(CamelMessageInfo *mi)
 	g_free(mi->from);
 	g_free(mi->to);
 	g_free(mi->cc);
-	g_free(mi->message_id);
-	header_references_list_clear(&mi->references);
+	g_free(mi->references);
 	camel_flag_list_free(&mi->user_flags);
 	camel_tag_list_free(&mi->user_tags);
 	/* FIXME: content info? */
 	g_free(mi);
 }
 
+#ifdef DOESTRV
+const char *camel_message_info_string(CamelMessageInfo *mi, int type)
+{
+	if (mi->strings == NULL || type >= CAMEL_MESSAGE_INFO_STRING_COUNT)
+		return "";
+	return e_strv_get(mi->strings, type);
+}
+
+void camel_message_info_set_string(CamelMessageInfo *mi, int type, const char *str)
+{
+	g_assert(mi->strings != NULL);
+		
+	mi->strings = e_strv_set(mi->strings, type, str);
+}
+#endif
+
 #if 0
 static void
 content_info_dump(CamelMessageContentInfo *ci, int depth)