Welcome, Guest. Please login or register.

Author Topic: Bitmessage client for Android - A rough design  (Read 25348 times)

JonathanCoe

  • Full Member
  • ***
  • Posts: 178
  • Karma: +26/-0
    • View Profile
Bitmessage client for Android - A rough design
« on: May 19, 2013, 11:21:26 AM »
Hi everyone,

I'm a computer science student in the UK, and my summer university project is developing an Android client for Bitmessage. The project was approved a few weeks ago, and now I'm beginning work on it in earnest. The project specification can be found on my supervisor's homepage, here: http://www-users.cs.york.ac.uk/~dkolovos/research/student-projects/2013/9

I've spent the last couple of weeks sketching out a rough design for how I think a smartphone client for Bitmessage might work, and now I'm looking for some feedback from the community. My main concern is to find if there are any major flaws in the proposed design before I move on to prototyping.

The attached document gives a broad overview of the design I'm proposing.  It's still very rough at this stage, so there are bound to be lots of mistakes, omissions, and other problems. One part of the design requires a (small) change the protocol, so I expect a lot of debate about that in particular. I'm very happy to discuss all aspects of the design.

I've asked my supervisor and he has no problem with me posting this design to the community forum for feedback. We've agreed that it is the fastest way to ensure that the overall design is suitable. I will record and acknowledge any feedback I receive in my final report. Furthermore I won't accept any assistance beyond feedback for the proposed design (i.e. no volunteered code or other more detailed assistance). Therefore there should be no concern that taking feedback will damage the academic integrity of the project.

I'd be extremely grateful for all feedback, even if it's something that renders this design completely unworkable. I'd rather find out now than after spending a month on development. Thanks to everyone in advance.

Jonathan (yes, another one  :) )

EDIT: Version 10 of the rough design document is now obsolete, so I've replaced it with version 19, which is the current version.
« Last Edit: May 23, 2013, 06:46:36 PM by kv802 »
My address: BM-NBdhY8vpWJVL2YocA2Gfjf7eVoZAgbEs

dokument

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 488
  • Karma: +37/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #1 on: May 20, 2013, 03:09:32 AM »
Awesome! Very cool to see someone tackling this head on. Just a few questions comments.

Why have stream dedicated servers? Why can't all of the servers use any of the streams?

Quote
"The server has a database of all the clients that have registered with it, including:
◦ A unique ID number of the client
◦ The network information necessary to connect with the client.
◦ The hashes and public keys of the addresses of that client in the stream(s) which
the server operates in.
◦ The list of broadcast addresses to which the client is subscribed"
[\quote]

Will there be a warning to the user of the app that this data is being tracked?

Adding identifiers to the messages would kill a lot of the security and privacy that bitmessage offers. Yes it makes it much harder to implement a mobile version (without some thought) but I hate to say that development would stop if it were forced to do that. At least the majority of users would stop using it once they understand. You point this out in the pdf, I am just stating it for the glancing readers.

Since the proposed method reduces privacy, why not move it around a bit. Instead of changing the network as a whole, why not only make it effect mobile users. Use the servers as mobile gateways into the Bitmessage network.

Also, even though it wouldn't be everyone running their own mobile server at home, giving it out as an option could reduce load on the dedicated servers.

The biggest problem with Lowering PoW is that it makes it vastly easier for people to maliciously flood the network. Even if it was only from android phones, they could be emulated on a pc. Flooding is the main purpose of PoW, not spam.  Again, you touch on this I just wanted to make sure you understand the reason for the PoW.

Overall, the main problem is PoW is it not? Assuming that there is an option to only connect on wifi, 4g, etc then the network traffic really isn't too bad. Yes the messages.dat could become quite big, but this can be alleviated by lowering the storage time of messages. Also it could periodically check in with servers instead of keeping the connection open. I (naively) think that a receive-only mobile client would be quite straight forward. You wouldn't even have to generate an address if you were receiving broadcasts.

Offloading PoW would have to be done with an account or paid subscription or other verification technique. Otherwise it really would leave the door open to malicious attacks.

Maintaining security and anonymity as much as possible. What if there was a transaction fee paid by bitcoin from the mobile user? It could be a small fee, but it would go towards someone letting you use their machine to generate the PoW. Your servers could essentially act as a proxy and staging area so that the only information that the PoW miner would know, is that it is a mobile phone (although it could be a pc user just as easily so that is not certain)

Idk, just throwing out my initial thoughts. Good luck with this, I will definitely keep an eye on it.

.dok



BM-2cTtoitr47Q7weyKr9pFX363YBRMQfBWzt

JonathanCoe

  • Full Member
  • ***
  • Posts: 178
  • Karma: +26/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #2 on: May 20, 2013, 10:13:03 AM »
Thanks! It's great to be working on something so interesting.

Thanks a lot for your comments, I'll attempt to answer them point-by-point.

Quote
1) Why have stream dedicated servers? Why can't all of the servers use any of the streams?
Servers can use as many streams as they have the capacity for. Obviously if the use of Bitmessage grows significantly then it would be impractical for any one server to operate permanently in all streams (the same as any other full node).

Quote
2) Will there be a warning to the user of the app that this data is being tracked?
Definitely.

Quote
3) Adding identifiers to the messages would kill a lot of the security and privacy that bitmessage offers. Yes it makes it much harder to implement a mobile version (without some thought) but I hate to say that development would stop if it were forced to do that. At least the majority of users would stop using it once they understand. You point this out in the pdf, I am just stating it for the glancing readers.
I think this is the most difficult problem. I'm glad you've picked up on this, because I really want to find a way of making this work without degrading the privacy and security of the network.

Thankfully I think I may have come up with a solution, although I really haven't had time to think it through in any depth. I've just started trying to work out if it's at all feasible, so there's lots still to be worked out, but I wanted to record it here in case someone else beats me to the punch.

The basic problem is that in order for Bitmessage's  flooding technique for message distribution to work, you have to receive all messages which may be addressed to you and be able to decrypt any that are actually for you. Clearly we can't allow the server to decrypt all mobile client messages and forward them on, as that would mean a total loss of security. Therefore at the moment the server has no way of knowing which messages to forward to the mobile clients it serves.

To overcome this, you could have a system in which messages to be sent to Android clients are encrypted twice, using two different addresses. The message is encrypted once for the server, and once for the Android client.

When an Android client generates a new address (A) in a given stream, it will send that address and the corresponding public keys to the server it is registered with in that stream. The server will then also generate a new address (B), and record the two together as a pair (A and B). The server then publishes these two addresses and 4 public keys to the rest of the stream, as if it had generated both addresses itself. The server will then send B to the client, so that it also has the pair of addresses.

Now when the user of the Android client (Alice) wants to give someone (Bob) their address, they give them the address pair A and B. When Bob wants to send a message to Alice, he first creates the real message for Alice and encrypts it using the public encryption key for address A. Then he takes the encrypted message for A (EMA) and encrypts it with the public keys for address B, creating encrypted message for B (EMB).

Bob then sends EMB through the Bitmessage stream in question. Eventually it arrives at the server, which acts as a full node in that stream. The server decrypts EMB, producing EMA, and recognises that the address it has used to decrypt EMB corresponds to the address pair A and B. Now when the Android client which generated address A polls the server to check for messages, the server can forward EMA to the Android client, which can then decrypt and read it.

This is a totally half-baked idea, but I think it might work, and if it does then it would free us from having to add an identifier to messages. I'll spend some time trying to think it through now. 

EDIT: One of the most obvious problems with this idea would be that the user of the Android client would have to give out two addresses (A and B) to anyone that wanted to send them a message. Perhaps this could be solved as follows:

When an Android client creates a new address (A), it sends the address and the two public keys for that address (A1, A2) to one of the servers that operates in that address's stream.

The server then generates a second address (B) and creates a new record in its database of A and B as a pair.

The server then calculates a third address (C). The hash part of this address has four public keys as inputs - the public encryption and signing keys for addresses A and B (A1, A2, B1, B2).

The server then publishes C to the rest of the nodes in the stream, along with the public keys A1, A2, B1, B2. These keys could be distributed either as two standard pubkey objects or as a one double-size pubkey object.

The server then sends address C to the Android client. The user of the Android client can then give out address C to their contacts.

The two stage encryption and decryption process can then proceed as above, with the change that only address C is needed to request and verify the 4 public keys needed.

Separate point: When sending a message to an Android client proof of work could be done only once, for the outer layer of encryption, to avoid creating extra work.

EDIT 2: Having given some thought to the double-encryption idea, I think that it would work, but it would only be worthwhile if we were able to place some degree of trust in the servers. Using the same terminology as above, address C will have to be passed around to many different servers, along with the corresponding 6 keys that the server has access to. This would mean someone could easily set up a rogue server or to grab them. They could then decrypt the outer layer of any messages sent to that address, which defeats the purpose of the double encryption.

Therefore perhaps the best approach would be to return to the idea of adding a hash of the destination address into the message header, but have this only apply to messages for Android clients, so that it does not degrade the security of the rest of the network. Android clients would generate 'mobile addresses' of a separate version from those generated by PC clients. Then when any user wants to send a message to a 'mobile address', they will include a hash of that address in the message header.

In the worst case this allows a network eavesdropper to detect when you are sent a message (and possibly what size it is). To do this, the attacker has to first find out your address, in order to match it with the hash seen over the network. This would only represent a small possible loss of privacy for users of the Android client, and would not effect users of the PC client at all.

Given the extent to which mobile users' activities are tracked anyway, this would not be a particularly great concern. Indeed, someone who found it objectionable could simply use the PC client instead. In addition, this solution would be far simpler to code and maintain.


Quote
4) The biggest problem with Lowering PoW is that it makes it vastly easier for people to maliciously flood the network. Even if it was only from android phones, they could be emulated on a pc. Flooding is the main purpose of PoW, not spam.  Again, you touch on this I just wanted to make sure you understand the reason for the PoW.

Overall, the main problem is PoW is it not? Assuming that there is an option to only connect on wifi, 4g, etc then the network traffic really isn't too bad. Yes the messages.dat could become quite big, but this can be alleviated by lowering the storage time of messages. Also it could periodically check in with servers instead of keeping the connection open. I (naively) think that a receive-only mobile client would be quite straight forward. You wouldn't even have to generate an address if you were receiving broadcasts.

Offloading PoW would have to be done with an account or paid subscription or other verification technique. Otherwise it really would leave the door open to malicious attacks.
You're right, I have been conflating flooding and spam, even through they're not the same thing. Thanks for pointing that out.

The proof of work issue is a significant concern, but at the moment I think it will be ok to just have the proof of work done by the Android client locally. It will be a problem for useability, but there are things we can do (e.g. working in the background) to make the problem less serious. I think that having the server do the proof of work is probably out of the question, so it will probably have to be done on the phone itself, one way or another. Someone could build a mechanism which links the phone to their PC client and has the PC do the proof of work, but I think that's outside the reasonable scope of the project at the moment.

Quote
5) Maintaining security and anonymity as much as possible. What if there was a transaction fee paid by bitcoin from the mobile user? It could be a small fee, but it would go towards someone letting you use their machine to generate the PoW. Your servers could essentially act as a proxy and staging area so that the only information that the PoW miner would know, is that it is a mobile phone (although it could be a pc user just as easily so that is not certain)
I agree that this could work, but at the moment I don't think it's a better solution than just having the mobile client do the proof of work locally and finding ways to mitigate the useability problem which that produces. Obviously adding Bitcoin integration would make the development of the mobile client significantly more complicated, and would tie its success to that of Bitcoin. If in a few years Bitcoin is extremely popular then this might be a better way of doing it, but at the moment I think it would create a set of problems greater than the one it would solve.
« Last Edit: May 22, 2013, 12:29:43 PM by kv802 »
My address: BM-NBdhY8vpWJVL2YocA2Gfjf7eVoZAgbEs

JonathanCoe

  • Full Member
  • ***
  • Posts: 178
  • Karma: +26/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #3 on: May 23, 2013, 03:39:27 PM »
I've produced an updated version of the rough design document.

The main change is that there is now a different address type for Bitmessage addresses generated by the Android client. For the moment I'm calling these 'lite client addresses'. When any client (PC or Android) sends a message to one of these addresses, it will calculate a hash of that address and attach it to the message header. This will allow the servers which serve the Android clients to route the message to the correct client. This means that users of the PC client will suffer absolutely no ill effects from the small protocol changes required to make the proposed design work. Hopefully that is a solution that can gain the community's approval.

Other than that, the document has been generally improved. The previous version can now be considered obsolete.

As before, I'd be extremely grateful for any and all feedback, no matter how critical. I hope to start prototyping next week, so if anyone has any significant criticisms of the proposed design, now would be a really good time to hear them  :)
My address: BM-NBdhY8vpWJVL2YocA2Gfjf7eVoZAgbEs

srmojuze

  • Full Member
  • ***
  • Posts: 155
  • Karma: +6/-0
    • View Profile
    • BitChirp.org
Re: Bitmessage client for Android - A rough design
« Reply #4 on: May 23, 2013, 03:46:19 PM »
I've produced an updated version of the rough design document.

The main change is that there is now a different address type for Bitmessage addresses generated by the Android client. For the moment I'm calling these 'lite client addresses'. When any client (PC or Android) sends a message to one of these addresses, it will calculate a hash of that address and attach it to the message header. This will allow the servers which serve the Android clients to route the message to the correct client. This means that users of the PC client will suffer absolutely no ill effects from the small protocol changes required to make the proposed design work. Hopefully that is a solution that can gain the community's approval.

Other than that, the document has been generally improved. The previous version can now be considered obsolete.

As before, I'd be extremely grateful for any and all feedback, no matter how critical. I hope to start prototyping next week, so if anyone has any significant criticisms of the proposed design, now would be a really good time to hear them  :)

Very cool... Given the improvements in ARM, how long will it take if the Android smartphone (say Tegra 3) were to do everything in-device as per PC implementation? Minutes to send? Tens of minutes? How much battery power might it use?

Good luck...!

AyrA

  • BM-Bc7Rspa4zxAPy9PK26vmcyoovftipStp
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1240
  • Karma: +74/-7
  • bitmessage.ch and timeservice operator
    • View Profile
    • AyrAs Homepage
Re: Bitmessage client for Android - A rough design
« Reply #5 on: May 23, 2013, 05:36:14 PM »
Somebody made a C++ test implementation and the language is so much more efficient than Python, it did POW for Messages within a few Seconds. If this can be replicated on ARM Devices, using bitmessage would not even use half as much battery than mobile games, regarding the fact, that you are not doing POW 24/7
My Address: BM-Bc7Rspa4zxAPy9PK26vmcyoovftipStp
Bitmessage Time Service (Subscribe): BM-BcbRqcFFSQUUmXFKsPJgVQPSiFA3Xash
Support the Multipart Message Declaration Draft for Bitmessage: https://bitmessage.org/forum/index.php/topic,1553.0.html
Free Bitmessage to E-Mail Gateway: https://bitmessage.ch

srmojuze

  • Full Member
  • ***
  • Posts: 155
  • Karma: +6/-0
    • View Profile
    • BitChirp.org
Re: Bitmessage client for Android - A rough design
« Reply #6 on: May 23, 2013, 05:40:07 PM »
Somebody made a C++ test implementation and the language is so much more efficient than Python, it did POW for Messages within a few Seconds. If this can be replicated on ARM Devices, using bitmessage would not even use half as much battery than mobile games, regarding the fact, that you are not doing POW 24/7

Nice.

JonathanCoe

  • Full Member
  • ***
  • Posts: 178
  • Karma: +26/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #7 on: May 23, 2013, 06:44:22 PM »
Very cool... Given the improvements in ARM, how long will it take if the Android smartphone (say Tegra 3) were to do everything in-device as per PC implementation? Minutes to send? Tens of minutes? How much battery power might it use?

Good luck...!

Somebody made a C++ test implementation and the language is so much more efficient than Python, it did POW for Messages within a few Seconds. If this can be replicated on ARM Devices, using bitmessage would not even use half as much battery than mobile games, regarding the fact, that you are not doing POW 24/7

Frankly, at the moment I don't know how long an average proof of work for a Bitmessage might take on an average Android phone. It's something I still need to tackle properly. It seems reasonable to think that it might take many minutes, which is ok for the academic purpose of the project, but obviously not great for real-world use. The client will be written in Java, which is the default choice for Android applications, but it looks like it might work a lot better if the proof of work component is written in C++ (thanks a lot for that tip AryA). It's still early days yet, so I have quite a lot of flexibility with how I go about it.
My address: BM-NBdhY8vpWJVL2YocA2Gfjf7eVoZAgbEs

dokument

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 488
  • Karma: +37/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #8 on: May 23, 2013, 08:48:24 PM »
It depends on the length and size of the message along with the PoW settings that the recipient has set.
BM-2cTtoitr47Q7weyKr9pFX363YBRMQfBWzt

srmojuze

  • Full Member
  • ***
  • Posts: 155
  • Karma: +6/-0
    • View Profile
    • BitChirp.org
Re: Bitmessage client for Android - A rough design
« Reply #9 on: May 24, 2013, 07:06:50 AM »
Frankly, at the moment I don't know how long an average proof of work for a Bitmessage might take on an average Android phone. It's something I still need to tackle properly. It seems reasonable to think that it might take many minutes, which is ok for the academic purpose of the project, but obviously not great for real-world use. The client will be written in Java, which is the default choice for Android applications, but it looks like it might work a lot better if the proof of work component is written in C++ (thanks a lot for that tip AryA). It's still early days yet, so I have quite a lot of flexibility with how I go about it.

For Android, if it just takes a few minutes and can run in the background then that is not too bad, because you can notify the user when it is sent/etc. Since Bitmessage is, at this stage, not an instant messaging system per se?

JonathanCoe

  • Full Member
  • ***
  • Posts: 178
  • Karma: +26/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #10 on: May 24, 2013, 08:25:56 AM »
For Android, if it just takes a few minutes and can run in the background then that is not too bad, because you can notify the user when it is sent/etc. Since Bitmessage is, at this stage, not an instant messaging system per se?

Yes that's the basic plan at the moment, messages will be processed in the background and the user notified when they are finally sent.

From what I understand even Java should be considerably more efficient than Python, although not quite as fast as C++, so even if the whole thing is written in Java we should see some considerable efficiency gains over the PC client.

AryA, do you know if the C++ implementation was made publicly available, or who made it? I've tried searching for some trace of it but I can't seem to find anything specific. If someone has already written the proof of work component in C++ then I could simply re-use it.
My address: BM-NBdhY8vpWJVL2YocA2Gfjf7eVoZAgbEs

AyrA

  • BM-Bc7Rspa4zxAPy9PK26vmcyoovftipStp
  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1240
  • Karma: +74/-7
  • bitmessage.ch and timeservice operator
    • View Profile
    • AyrAs Homepage
Re: Bitmessage client for Android - A rough design
« Reply #11 on: May 24, 2013, 10:50:30 AM »
luckily I almost never delete anything (yes I have 200 TB worth of Data on external Disks). Below is the Bitmessage text sent to the old general Mailing List:


The oldest Message is on the Bottom


Quote
Message ostensibly from BM-ooknYdggPUSo8NUsSgDWTnFNRrcU9kpPT:


Parallelising using openmp I can get the 3 seconds down to 1 second on a dual core processor.


------------------------------------------------------
Message ostensibly from BM-ooknYdggPUSo8NUsSgDWTnFNRrcU9kpPT:


There are parameters to adjust the proof of work difficulty, and I don't know whether those are hard-coded or adjust automatically.  I havn't tried it yet but with parallelisation the 3 second timing probably can be erduced further.  I'm also running quite old hardware, so this is by no means a state of the art machine.


------------------------------------------------------
Message ostensibly from BM-BbiaTWWMxNKBUHC3gnug3LaLomXxCczY:


Aplying this 3 seconds to my calculation (and assuming the address generation is as fast as the message sending) on a 24 Core HP Proliant ML350 my Company is selling to customers I can send 12 message every three seconds and generate 12 addresses every 3 seconds. (or 4 message/addresses a second), wouldn't this sort of making spam possible again? assuming you can use the graphics card (which may be up to 100 times faster than the CPU) this would make possible what we try to avoid, SPAM.
Another question is if the Proof of Work gets more complicated like bitcoin mining gets more complicated over time or will it stay the same?


-Random


------------------------------------------------------
Message ostensibly from BM-ooknYdggPUSo8NUsSgDWTnFNRrcU9kpPT:


In my initial tests on a short "this is a test" message the proof of work takes 3 seconds.


ANSI C is certainly a possibility.  I though I'd use C++ because of its better string/stream handling functions.


------------------------------------------------------
Message ostensibly from BM-BbiaTWWMxNKBUHC3gnug3LaLomXxCczY:


How much faster is it?
I also thought about using the GPU, like you do in bitcoin mining.


-Random


------------------------------------------------------
Message ostensibly from BM-opkeq2vnbUNwhnrUhyd4bxhYoCms7BJix:


If you write it in the ANSI C subset of C++ you can increase the range of places where your library can be used. Good luck!


------------------------------------------------------
Message ostensibly from BM-opa6xy5V4H25rmpNCPzMk9g2aboQLYbmw:


To whom it may concern, I've started an attempt to implement the BitMessage protocol in C++.  Certainly the proof of work calculation is faster than in Python.


https://github.com/fuzzgun/cppbitmessage
My Address: BM-Bc7Rspa4zxAPy9PK26vmcyoovftipStp
Bitmessage Time Service (Subscribe): BM-BcbRqcFFSQUUmXFKsPJgVQPSiFA3Xash
Support the Multipart Message Declaration Draft for Bitmessage: https://bitmessage.org/forum/index.php/topic,1553.0.html
Free Bitmessage to E-Mail Gateway: https://bitmessage.ch

JonathanCoe

  • Full Member
  • ***
  • Posts: 178
  • Karma: +26/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #12 on: May 24, 2013, 11:53:56 AM »
That's brilliant AryA, thanks so much! Whether I use the code directly or not, it will be very helpful to have it as a reference.
My address: BM-NBdhY8vpWJVL2YocA2Gfjf7eVoZAgbEs

dokument

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 488
  • Karma: +37/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #13 on: May 24, 2013, 02:21:21 PM »
If nothing else you could code the PoW part in C++ and interface to it with java/python/etc. This might make development easier.
BM-2cTtoitr47Q7weyKr9pFX363YBRMQfBWzt

JonathanCoe

  • Full Member
  • ***
  • Posts: 178
  • Karma: +26/-0
    • View Profile
Re: Bitmessage client for Android - A rough design
« Reply #14 on: May 24, 2013, 03:32:09 PM »
If nothing else you could code the PoW part in C++ and interface to it with java/python/etc. This might make development easier.

Yes that would be the plan. I definitely wouldn't write the whole app in C++. It will either be all in Java, or mostly Java with the POW component in C++.

The server-side code will probably be written in Java as well, to make development easier, although I'll probably just use PyBitmessage for the Bitmessage part of it, rather than trying to re-implement the whole thing in another language.
My address: BM-NBdhY8vpWJVL2YocA2Gfjf7eVoZAgbEs