[Feature Request] Multipart Message (Draft: V 1.3)

[AyrA]

Why is there a text/html content type? Wouldn't this be exactly the same thing as application/octet-stream if the data is text/html data?

The text/html type shows the client, that the content folowing is html and can be displayed as such.
application/octet-stream is a generic 8-bit type and should always be treated as attachment.

How do we send a basic message that will show up in the UI as a normal message (possibly with attachments)?

I propose the first part to have no headers and to be text-only. This would make reading in old/unsupported clients easier for its users.
If you want to have attachments, just add another part after the message.
If you do not want any attachments it is even easier, just put your raw text in the message. A client will find, that there is no part splitter at the first line and will treat the message as text/plain

What is the point of the integer inside the part declarations? That integer makes it more difficult to scan the text looking for the next part declaration marker.

It tells the number of headers that are folowing. Parsing with regular expressions is quite easy. If you want to get rid of the number, we can define, that between the headers and the content is an empty line. E-Mail clients do this.

[AyrA]

I have added this Thread to reddit. Please support: http://redd.it/1c9loe

[ISibboI]

Right now, there's one bitmessage client, and I'm just writing another one. Wouldn't it make more sense to use the Message Encodings of the protocol for that?
It would be simpler to add a Multi-Part message encoding and a var_int which contains the amount of parts that follow. The parts are then two var_int, one for the message encoding, which can be anything except for another multi-part and one for the length of the following byte[]. The byte[] itself contains the data.

[AyrA]

I have made here a system, that does not relies on integers at all and is fully compatible with the current clients and protocol spec. If you create a standard you have the full possibility to make it without any silly integers, that can cause problems between big endian and little endian systems. technically an integer limmits you to transport a 4 GB File maximum. Without any sort of integer at all you have no limits. Also your idea forces us to invent a part for each content type, that can be transferred.

[SkyN]

http://mailman2.u.washington.edu/pipermail/imap-protocol/2007-November/000729.html
You want to do it from scratch?

or you can use ready   http://docs.python.org/2/library/email.html

[AyrA]

As told, I do not want the full MIME spec and your first link describes IMAP<->MIME mapping which is unusable here because bitmessage does not uses MIME, I often work with E-Mail systems and servers and virtually all IMAP servers respond to a message query with either plaintext or multipart ID, all others are unused because mail clients send either plaintext or multipart and not raw audio for example. Many E-Mail clients also declare all attachments as application/octet-stream regardless of the actual file type. MIME was developed over 20 years ago and much of it is no longer best practice to use today because all systems have 8 bit ascii. It is just kept for compatibility.
We have the opportunity to implement a standard here that fits exactly the requirements we have and we should do it instead of taking an obsolete standard. MIME is bad because every time something new came it was expanded instead of rewritten and this is why it has so many stuff, that is not used since 10 or more years.
people may want to write their own bitmessage client and probably do not want to include a third party component so the standard should be as easy to implement as possible.

[AyrA]

EncodingTypes 0-2 are defined in the wiki. I suggest defining the Type 3 for BitMIME

[AyrA]

I have changed all Declarations. The Values of Attributes now can be up to 1024 Chars long (excluding the Line Break) to support long File Names on some File Systems and containing more Information, especially if it is hash based or a PGP Key

[boondoggle]

As far as I know there wouldn't be any advantage to multi-part messages in terms of the proof of work.  Maybe there could be a security/anonymity case to be made, but if so I'm not sure what that would be.

[AyrA]

Multipart Messages allow you to add attachments to your Message or use Headers to include your PGP or S/MIME key. it does not changes how the POW is done, it will (if it is accepted by Atheros) standardize the way messages are transmitted and interpreted, so people can implement a Bitmessage client without the need to implement countless pseudo standards. It's basically the first step towards using the Bitmessage Network like E-Mail, without the Need for Servers.


The existing MIME was avoided on purpose because it is 7 Bit only. Converting 8 Bits to 7 increases the number of bytes required, which in turn increases POW Time for no reason.
This standard is similar to MIME, because it allows you to implement parts of the existing standard, making it a lot easier to implement.

[nimda]

OK, after a review, I support this in principle. However, I have a few things:

Range: 0-99

This seems rather arbitrary. Maybe you want to be able to RegEx it like \d\d? but that seems a little silly. At minimum, why not fill up a UCHAR and support 0-255?
"Any MIME or MIME-like encoding format, should they be used, should make use of Bitmessage's 8-bit bytes."
https://bitmessage.org/wiki/Protocol_specification#Message_Encodings

This name can also be used, when referencing to this part inside a message (for example a text/html part may reference to a png image inside the same message).

Should probably require the part to have already been declared if this is happening. For example, you must have the png data before the html data. Otherwise we run the risk of infinite loops that will be difficult to prevent. Similarly, if at some point this spec supports compression containers, they should be separate from the multipart spec, lest we run the risk of zip files all the way down.

An Atrribute contains only Letters, Numbers and dashes, it is followed by a colon, a space and its Value. Maximum Length of an Attribute (without colon and space) is 32 Chars

Change "dashes" to "underscores" and you get the \w POSIX character class.

A Value can be made of any printable character of the UTF subset. It ends with a Line break. Maximum Length (without Line break) is 1024 Chars.

Values should be allowed to be blank, with the colon immediately followed by a line break. Given this possibility, it would make the most sense for maxLength to be 2**n - 1, so that Value can be stored as a character array whose index is n bits.
Therefore I suggest maxLength be either 1023 (least change from current proposal) 2047 (least upwards change from current) or 65535 (USHORT).

A Line break must be the same as used withhin message content to avoid confusion of the decoder.

We should either standardize \n or \r\n as line break now, and then handle conversion, or the specification should have an explicit way to define a line break, like "the sequence of \r and \n codes following the first boundary declaration"

Or, we could stop trying to reinvent the wheel, and just use prebuilt MIME libraries like Python's email module. I will admit, however, that MIME is like XML: really complex, and it might be worth it to create a simpler version.
http://docs.python.org/2/library/email.html#module-email

[AyrA]

OK, after a review, I support this in principle. However, I have a few things:

Range: 0-99

This seems rather arbitrary. Maybe you want to be able to RegEx it like \d\d? but that seems a little silly. At minimum, why not fill up a UCHAR and support 0-255?
"Any MIME or MIME-like encoding format, should they be used, should make use of Bitmessage's 8-bit bytes."
https://bitmessage.org/wiki/Protocol_specification#Message_Encodings

Since the number is written in ASCII anyway and not as a single byte, the range 0-99 has no real impact on this. The complete MIME spec is written in ASCII so people without MIME support can at least read the message a little. We could increase it to 0-999 but who needs more than 100 parts in a message anyways?

This name can also be used, when referencing to this part inside a message (for example a text/html part may reference to a png image inside the same message).

Should probably require the part to have already been declared if this is happening. For example, you must have the png data before the html data. Otherwise we run the risk of infinite loops that will be difficult to prevent. Similarly, if at some point this spec supports compression containers, they should be separate from the multipart spec, lest we run the risk of zip files all the way down.

The current MIME implementation for E-Mail does not requires you to predefine parts before you want to use them so why limit us to this here? MIME messages should be parsed 2 times, the first time it is taken apart and split into its pieces, and with the second part, the parsing of HTML content is done. How do you want to make loops? You can include Images and audio in HTML, but not other HTML pages, also you should not be able to include itself so we are pretty safe from it.

An Atrribute contains only Letters, Numbers and dashes, it is followed by a colon, a space and its Value. Maximum Length of an Attribute (without colon and space) is 32 Chars

Change "dashes" to "underscores" and you get the \w POSIX character class.

The E-Mail MIME implementation also uses dashes and I want this implementation to be as identical as possible, so existing MIME parsers can be rewritten with little effort.

A Value can be made of any printable character of the UTF subset. It ends with a Line break. Maximum Length (without Line break) is 1024 Chars.

Values should be allowed to be blank, with the colon immediately followed by a line break. Given this possibility, it would make the most sense for maxLength to be 2**n - 1, so that Value can be stored as a character array whose index is n bits.
Therefore I suggest maxLength be either 1023 (least change from current proposal) 2047 (least upwards change from current) or 65535 (USHORT).

There is no reason for a value to be empty, instead of empty values, the specific header is completely omitted, however, you can just instantly make a line break after the colon and the space if you really want. You probably want to change the length due to the \0 terminator in char arrays, but this is not needed, because you must not include the \0 char in the value, also 1024 refers to "printable chars" so, if you write in japanese, a char is actually 2 byte wide. Characters can be (in the existing codepages) maximum 3 bytes long, so if you program this in ANSI C or C++ you should allocate at least 3 times more RAM to store the header.

A Line break must be the same as used withhin message content to avoid confusion of the decoder.

We should either standardize \n or \r\n as line break now, and then handle conversion, or the specification should have an explicit way to define a line break, like "the sequence of \r and \n codes following the first boundary declaration"

I rather standardize it to the Windows linebreak (\r\n), so it will work with Windows, Linux and Mac without any impacts, also It is compatible with the E-Mail standard.

Or, we could stop trying to reinvent the wheel, and just use prebuilt MIME libraries like Python's email module. I will admit, however, that MIME is like XML: really complex, and it might be worth it to create a simpler version.
http://docs.python.org/2/library/email.html#module-email

No, because MIME natively does not supports 8bit attachments (the 8BITMIME is not even supported by all E-mail servers), instead you need to fall back to Base64 or UUencode. I also do not want to include all the junk in MIME that we do not need in bitmessage. If you send a Mail using and E-Mail clients it will set about 10 Headers, they can even be distributed over multiple lines and we do not need this at all.

[nimda]

OK, after a review, I support this in principle. However, I have a few things:

Range: 0-99

This seems rather arbitrary. Maybe you want to be able to RegEx it like \d\d? but that seems a little silly. At minimum, why not fill up a UCHAR and support 0-255?
"Any MIME or MIME-like encoding format, should they be used, should make use of Bitmessage's 8-bit bytes."
https://bitmessage.org/wiki/Protocol_specification#Message_Encodings

Since the number is written in ASCII anyway and not as a single byte, the range 0-99 has no real impact on this. The complete MIME spec is written in ASCII so people without MIME support can at least read the message a little. We could increase it to 0-999 but who needs more than 100 parts in a message anyways?

Who needs more than 73 parts in a message anyways? 99 is very arbitrary, less so than 255, which is a UCHAR.

This name can also be used, when referencing to this part inside a message (for example a text/html part may reference to a png image inside the same message).

Should probably require the part to have already been declared if this is happening. For example, you must have the png data before the html data. Otherwise we run the risk of infinite loops that will be difficult to prevent. Similarly, if at some point this spec supports compression containers, they should be separate from the multipart spec, lest we run the risk of zip files all the way down.

The current MIME implementation for E-Mail does not requires you to predefine parts before you want to use them so why limit us to this here? MIME messages should be parsed 2 times, the first time it is taken apart and split into its pieces, and with the second part, the parsing of HTML content is done. How do you want to make loops? You can include Images and audio in HTML, but not other HTML pages, also you should not be able to include itself so we are pretty safe from it.

There are iFrames and other such things. Do you know the entire HTML spec? How about the entire MIME spec? The goal should be to prevent whole classes of attacks, not pick them off individually.

An Atrribute contains only Letters, Numbers and dashes, it is followed by a colon, a space and its Value. Maximum Length of an Attribute (without colon and space) is 32 Chars

Change "dashes" to "underscores" and you get the \w POSIX character class.

The E-Mail MIME implementation also uses dashes and I want this implementation to be as identical as possible, so existing MIME parsers can be rewritten with little effort.

I can't really get what you're saying. First, rewriting MIME parsers is going to be worse than making a new one from scratch. Have you seen the MIME RFC's?

MIME is specified in six linked RFC memoranda: RFC 2045, RFC 2046, RFC 2047, RFC 4288, RFC 4289 and RFC 2049, which together define the specifications.

Second, if you want it to be simpler than MIME, make it simpler than MIME.

A Value can be made of any printable character of the UTF subset. It ends with a Line break. Maximum Length (without Line break) is 1024 Chars.

Values should be allowed to be blank, with the colon immediately followed by a line break. Given this possibility, it would make the most sense for maxLength to be 2**n - 1, so that Value can be stored as a character array whose index is n bits.
Therefore I suggest maxLength be either 1023 (least change from current proposal) 2047 (least upwards change from current) or 65535 (USHORT).

There is no reason for a value to be empty, instead of empty values, the specific header is completely omitted, however, you can just instantly make a line break after the colon and the space if you really want. You probably want to change the length due to the \0 terminator in char arrays, but this is not needed, because you must not include the \0 char in the value, also 1024 refers to "printable chars" so, if you write in japanese, a char is actually 2 byte wide. Characters can be (in the existing codepages) maximum 3 bytes long, so if you program this in ANSI C or C++ you should allocate at least 3 times more RAM to store the header.

There are certainly reasons for values to be empty, as there is a difference between "I know the <attribute> and it's an empty string" and "I don't know the <attribute>."
NTFS specification allows paths to be 32,000 (2^15 or so) characters, so a USHORT (2^16) would make the most sense.

The Unicode versions of several functions permit a maximum path length of approximately 32,000 characters composed of components up to 255 characters in length. To specify that kind of path, use the "\\?\" prefix. The maximum path of 32,000 characters is approximate, because the "\\?\" prefix can be expanded to a longer string, and the expansion applies to the total length.

A Line break must be the same as used withhin message content to avoid confusion of the decoder.

We should either standardize \n or \r\n as line break now, and then handle conversion, or the specification should have an explicit way to define a line break, like "the sequence of \r and \n codes following the first boundary declaration"

I rather standardize it to the Windows linebreak (\r\n), so it will work with Windows, Linux and Mac without any impacts, also It is compatible with the E-Mail standard.

That's fine.

Or, we could stop trying to reinvent the wheel, and just use prebuilt MIME libraries like Python's email module. I will admit, however, that MIME is like XML: really complex, and it might be worth it to create a simpler version.
http://docs.python.org/2/library/email.html#module-email

No, because MIME natively does not supports 8bit attachments (the 8BITMIME is not even supported by all E-mail servers), instead you need to fall back to Base64 or UUencode. I also do not want to include all the junk in MIME that we do not need in bitmessage. If you send a Mail using and E-Mail clients it will set about 10 Headers, they can even be distributed over multiple lines and we do not need this at all.

It just seems to me that you're trying to make it as hard as possible to implement:
- Different from MIME, so no Python email lib
- As similar to MIME as possible, so no easy POSIX matching
- Different from MIME, so a new attachment type has to be implemented
- As similar to MIME as possible, opening up security holes

[AyrA]

Who needs more than 73 parts in a message anyways? 99 is very arbitrary, less so than 255, which is a UCHAR.

you can also fit the number 99 in a uchar. (or byte in higher level languages). We somewhere have to set an upper limit and this should be a limit, that regular users understand, if they are looking at a spec. Specs are not only for developers, they are for users too who want to learn how stuff is built. Instead of putting up the number 255 (technically 256 because you cant have a multipart message with 0 parts) I use the number 99, because people understand the two digit limit better than the 2ⁿ-1 logic. It technically does not matter at all because you will still be parsing ASCII text with your regular expression.

There are iFrames and other such things. Do you know the entire HTML spec? How about the entire MIME spec? The goal should be to prevent whole classes of attacks, not pick them off individually.

Are you from the past? When have you seen an E-mail client displaying iFrames? Even Internet Explorer 4 had protection against iFrame looping and this thing was a browser (or at least looked like one). This has nothing to do with MIME, this is a general application issue, if your client displays iFrame and probably also has a javascript engine.

I can't really get what you're saying. First, rewriting MIME parsers is going to be worse than making a new one from scratch. Have you seen the MIME RFC's?
Second, if you want it to be simpler than MIME, make it simpler than MIME.

Rewriting MIME parsers is easy, instead of checking for the mime border at the content-type tag you look at the first line. Much easier. E-mails have headers, bitmessage has none, a user beginning his message with the Text with the line From: Me would already trigger header parsing. This thing does not only needs to be somewhat similar to MIME, it must also be compatible with the existing bitmessage standards. And the existing bitmessage standard has no real headers.

There are certainly reasons for values to be empty, as there is a difference between "I know the <attribute> and it's an empty string" and "I don't know the <attribute>."

Tell me the reason please? if the Subject header is missing, then there is no subject to display. If it is there but empty, there is no subject to display but the message is bigger for no reason, causing more POW.

NTFS specification allows paths to be 32,000 (2^15 or so) characters, so a USHORT (2^16) would make the most sense.

The Unicode versions of several functions permit a maximum path length of approximately 32,000 characters composed of components up to 255 characters in length. To specify that kind of path, use the "\\?\" prefix. The maximum path of 32,000 characters is approximate, because the "\\?\" prefix can be expanded to a longer string, and the expansion applies to the total length.

And now you want to include NTFS paths in the headers? What the hell should a linux user do with your NTFS Path? If it is about File names being 255 chars long: this is not supported by all file systems, someone probably runs this from a FAT formatted USB drive, somebody has an old linux partition, where this is not supported. Filenames longer than 50 chars are suspicious and complicated to handle anyway, oh and on some linux filesystems you can have the colon ( : ) in file names, I want your NTFS see storing that. You seem concentrating too much on windows.

- Different from MIME, so no Python email lib

and no unneeded obsolete features, also there are other languages out there.

- As similar to MIME as possible, so no easy POSIX matching

This was the issue about dashes in the spec. I love that argument. just entered "regex" and "dash" into google and the first result was a regular expression from which you can easily derive the correct expression to parse it.
http://stackoverflow.com/questions/5887678/alphanumeric-dash-and-underscore-but-no-spaces-regular-expression-check-javascr

- Different from MIME, so a new attachment type has to be implemented

It's not that much different from MIME if you have ever looked at a mime encoded mail

- As similar to MIME as possible, opening up security holes

So your failure in implementing a correct HTML rendering engine is now a MIME issue?
***ATTENTION! I start to be sarcastic below and you probably want to ignore it. Thanks***
We probably should also include the autoexec attribute that can be true or false (or undefined). It defaults to "true", if it is true, the attachment is automatically stored on the harddrive and executed (in the reverse order they appear in the message just for fun).
***END***
Just kidding, but seriously, this is not a MIME issue, this is a GUI issue because MIME just splits parts into pieces, nothing more. The only thing you need to do is prevent inclusion by building a tree with references. If a reference appears multiple times on different levels (1.html --> 2.html --> 3.html --> 1.html) you got a recursion. Browsers are doing the same thing.

[nimda]

Who needs more than 73 parts in a message anyways? 99 is very arbitrary, less so than 255, which is a UCHAR.

you can also fit the number 99 in a uchar. (or byte in higher level languages). We somewhere have to set an upper limit and this should be a limit, that regular users understand, if they are looking at a spec. Specs are not only for developers, they are for users too who want to learn how stuff is built. Instead of putting up the number 255 (technically 256 because you cant have a multipart message with 0 parts) I use the number 99, because people understand the two digit limit better than the 2ⁿ-1 logic. It technically does not matter at all because you will still be parsing ASCII text with your regular expression.

Find me one person, who does not program, who knows the value of MAX_PATH. Users who want to learn how stuff is built will do fine with 2**n.

There are iFrames and other such things. Do you know the entire HTML spec? How about the entire MIME spec? The goal should be to prevent whole classes of attacks, not pick them off individually.

Are you from the past? When have you seen an E-mail client displaying iFrames? Even Internet Explorer 4 had protection against iFrame looping and this thing was a browser (or at least looked like one). This has nothing to do with MIME, this is a general application issue, if your client displays iFrame and probably also has a javascript engine.

Like I said, though, the goal should be to prevent classes of attacks entirely, not leave them open in principle and then code up a shield. Remember zip bombs?

I can't really get what you're saying. First, rewriting MIME parsers is going to be worse than making a new one from scratch. Have you seen the MIME RFC's?
Second, if you want it to be simpler than MIME, make it simpler than MIME.

Rewriting MIME parsers is easy, instead of checking for the mime border at the content-type tag you look at the first line. Much easier. E-mails have headers, bitmessage has none, a user beginning his message with the Text with the line From: Me would already trigger header parsing. This thing does not only needs to be somewhat similar to MIME, it must also be compatible with the existing bitmessage standards. And the existing bitmessage standard has no real headers.

You rewrite a MIME parser, I'll do one up from scratch, and we'll see who finishes first. (Hint: you're going to be knee-deep in abstractions before you finish the second RFC)
Multipart MIME is a different thing than email headers.

There are certainly reasons for values to be empty, as there is a difference between "I know the <attribute> and it's an empty string" and "I don't know the <attribute>."

Tell me the reason please? if the Subject header is missing, then there is no subject to display. If it is there but empty, there is no subject to display but the message is bigger for no reason, causing more POW.

There are nearly infinite number of attributes, so there are nearly an infinite number of reasons.
Content-Type: (If null, it's not specified what it is, but it's NOT text/plain.)
FileName: (If null, don't suggest one automatically)
It can mean "known, but not telling" as opposed to "unknown." It can be used to override defaults with a blank string. Or, it can be used to, you know, specify an empty string! Don't tell me you have no 0 byte files on your computer.

NTFS specification allows paths to be 32,000 (2^15 or so) characters, so a USHORT (2^16) would make the most sense.

The Unicode versions of several functions permit a maximum path length of approximately 32,000 characters composed of components up to 255 characters in length. To specify that kind of path, use the "\\?\" prefix. The maximum path of 32,000 characters is approximate, because the "\\?\" prefix can be expanded to a longer string, and the expansion applies to the total length.

And now you want to include NTFS paths in the headers? What the hell should a linux user do with your NTFS Path? If it is about File names being 255 chars long: this is not supported by all file systems, someone probably runs this from a FAT formatted USB drive, somebody has an old linux partition, where this is not supported. Filenames longer than 50 chars are suspicious and complicated to handle anyway, oh and on some linux filesystems you can have the colon ( : ) in file names, I want your NTFS see storing that. You seem concentrating too much on windows.

Now you're just contradicting yourself.

The Values of Attributes now can be up to 1024 Chars long (excluding the Line Break) to support long File Names on some File Systems

"What the hell should a FAT32 user do with your 1024 character path?"
"Filenames longer than 50 chars are complicated to handle"
"EncFS encrypts on-the-fly, I want to see your 'some File Systems' storing that."
"You seem concentrating too much on 'some File Systems'"

- Different from MIME, so no Python email lib

and no unneeded obsolete features, also there are other languages out there.

Do you want to rewrite implementations or not?
Are there other languages out there with a Bitmessage client?

- As similar to MIME as possible, so no easy POSIX matching

This was the issue about dashes in the spec. I love that argument. just entered "regex" and "dash" into google and the first result was a regular expression from which you can easily derive the correct expression to parse it.
http://stackoverflow.com/questions/5887678/alphanumeric-dash-and-underscore-but-no-spaces-regular-expression-check-javascr

You said you wanted simplicity. \w, or [[:word:]], is as simple as it gets.
just entered "regex" and "URL" into google and the first result was this huge thing. It exists. Yes. So does MIME. MIME is complicated.

- Different from MIME, so a new attachment type has to be implemented

It's not that much different from MIME if you have ever looked at a mime encoded mail

That's like saying C# and Java aren't much different.

- As similar to MIME as possible, opening up security holes

So your failure in implementing a correct HTML rendering engine is now a MIME issue?

Do you have any idea how big HTML rendering engines are? How hard they are to make? You might as well say, "it's your fault for making a 10 story building that wasn't earthquake-proof."