swarm/storage/mru: Client-side MRU signatures (#784)

* swarm/storage/mru: Add embedded publickey and remove ENS dep

This commit breaks swarm, swarm/api...
but tests in swarm/storage/mru pass

* swarm: Refactor swarm, swarm/api to mru changes, make tests pass

* swarm/storage/mru: Remove self from recv, remove test ens vldtr

* swarm/storage/mru: Remove redundant test, expose ResourceHash mthd

* swarm/storage/mru: Make HeaderGetter mandatory + godoc fixes

* swarm/storage: Remove validator prefix for metadata chunk

* swarm/storage/mru: Use Address instead of PublicKey

* swarm/storage/mru: Change index from name to metadata chunk addr

* swarm/storage/mru: Refactor swarm/api/... to MRU index changes

* swarm/storage/mru: Refactor cleanup

* swarm/storage/mru: Rebase cleanup

* swarm: Use constructor for GenericSigner MRU in swarm.go

* swarm/storage: Change to BMTHash for MRU hashing

* swarm/storage: Reduce loglevel on chunk validator logs

* swarm/storage/mru: Delint

* swarm: MRU Rebase cleanup

* swarm/storage/mru: client-side mru signatures

Rebase to PR #668 and fix all conflicts

* swarm/storage/mru:  refactor and documentation

* swarm/resource/mru: error-checking  tests for parseUpdate/newUpdateChunk

* swarm/storage/mru: Added resourcemetadata tests

* swarm/storage/mru: Added tests  for UpdateRequest

* swarm/storage/mru: more test coverage for UpdateRequest and comments

* swarm/storage/mru: Avoid fake chunks in parseUpdate()

* swarm/storage/mru: Documented resource.go extensively

moved some functions where they make most sense

* swarm/storage/mru: increase test coverage for UpdateRequest and

variable name changes throughout to increase consistency

* swarm/storage/mru: moved default timestamp to NewCreateRequest-

* swarm/storage/mru: lookup refactor

* swarm/storage/mru: added comments and renamed raw flag to rawmru

* swarm/storage/mru: fix receiver typo

* swarm/storage/mru: refactored update chunk new/create

* swarm/storage/mru:  refactored signature digest to avoid malleability

* swarm/storage/mru: optimize update data serialization

* swarm/storage/mru: refactor and cleanup

* swarm/storage/mru: add timestamp struct and serialization

* swarm/storage/mru: fix lint error and mark some old code for deletion

* swarm/storage/mru: remove unnecessary variable

* swarm/storage/mru: Added more comments throughout

* swarm/storage/mru: Refactored metadata chunk layout + extensive error...

* swarm/storage/mru: refactor cli parser
Changed resource info output to JSON

* swarm/storage/mru: refactor serialization for extensibility

refactored error messages to NewErrorf

* swarm/storage/mru: Moved Signature to resource_sign.
Check Sign errors in server tests

* swarm/storage/mru: Remove isSafeName() checks

* swarm/storage/mru: scrubbed off all references to "block" for time

* swarm/storage/mru: removed superfluous isSynced() call.

* swarm/storage/mru: remove isMultihash() and ToSafeName functions

* swarm/storage/mru: various fixes and comments

* swarm/storage/mru: decoupled cli for independent create/update
* Made resource name optional
* Removed unused LookupPrevious

* swarm/storage/mru: Decoupled resource create / update & refactor

* swarm/storage/mru: Fixed some comments as per issues raised in PR #743

* swarm/storage/mru: Cosmetic changes as per #743 comments

* swarm/storage/mru: refct request encoder/decoder > marshal/unmarshal

* swarm/storage/mru: Cosmetic changes as per review in #748

* swarm/storage/mru: removed timestamp proof placeholder

* swarm/storage/mru: cosmetic/doc/fixes changes as per comments in #704

* swarm/storage/mru: removed unnecessary check in Handler.update

* swarm/storage/mru: Implemented Marshaler/Unmarshaler iface in Request

* swarm/storage/mru: Fixed linter error

* swarm/storage/mru: removed redundant address in signature digest

* swarm/storage/mru: fixed bug: LookupLatestVersionInPeriod not working

* swarm/storage/mru: Unfold Request creation API for create or update+create
set common time source for mru package

* swarm/api/http: fix HandleGetResource error variable shadowed
when requesting a resource that does not exist

* swarm/storage/mru: Add simple check to detect duplicate updates

* swarm/storage/mru: moved Multihash() to the right place.

* cmd/swarm: remove unneeded clientaccountmanager.go

* swarm/storage/mru: Changed some comments as per reviews in #784

* swarm/storage/mru: Made SignedResourceUpdate.GetDigest() public

* swarm/storage/mru: cosmetic changes as per comments in #784

* cmd/swarm: Inverted --multihash flag default

* swarm/storage/mru: removed Verify from SignedResourceUpdate.fromChunk

* swarm/storage/mru: Moved validation code out of serializer
Cosmetic / comment changes

* swarm/storage/mru: Added unit tests for UpdateLookup

* swarm/storage/mru: Increased coverage of metadata serialization

* swarm/storage/mru: Increased test coverage of updateHeader serializers

* swarm/storage/mru: Add resourceUpdate serializer test

1 files changed, 175 insertions, 0 deletions

diff --git a/swarm/storage/mru/request_test.go b/swarm/storage/mru/request_test.go
new file mode 100644
index 000000000..dba55b27e
--- /dev/null
+++ b/swarm/storage/mru/request_test.go
@@ -0,0 +1,175 @@
+package mru
+
+import (
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"reflect"
+	"testing"
+)
+
+func areEqualJSON(s1, s2 string) (bool, error) {
+	//credit for the trick: turtlemonvh https://gist.github.com/turtlemonvh/e4f7404e28387fadb8ad275a99596f67
+	var o1 interface{}
+	var o2 interface{}
+
+	err := json.Unmarshal([]byte(s1), &o1)
+	if err != nil {
+		return false, fmt.Errorf("Error mashalling string 1 :: %s", err.Error())
+	}
+	err = json.Unmarshal([]byte(s2), &o2)
+	if err != nil {
+		return false, fmt.Errorf("Error mashalling string 2 :: %s", err.Error())
+	}
+
+	return reflect.DeepEqual(o1, o2), nil
+}
+
+// TestEncodingDecodingUpdateRequests ensures that requests are serialized properly
+// while also checking cryptographically that only the owner of a resource can update it.
+func TestEncodingDecodingUpdateRequests(t *testing.T) {
+
+	signer := newCharlieSigner()  //Charlie, our good guy
+	falseSigner := newBobSigner() //Bob will play the bad guy again
+
+	// Create a resource to our good guy Charlie's name
+	createRequest, err := NewCreateRequest(&ResourceMetadata{
+		Name:      "a good resource name",
+		Frequency: 300,
+		StartTime: Timestamp{Time: 1528900000},
+		Owner:     signer.Address()})
+
+	if err != nil {
+		t.Fatalf("Error creating resource name: %s", err)
+	}
+
+	// We now encode the create message to simulate we send it over the wire
+	messageRawData, err := createRequest.MarshalJSON()
+	if err != nil {
+		t.Fatalf("Error encoding create resource request: %s", err)
+	}
+
+	// ... the message arrives and is decoded...
+	var recoveredCreateRequest Request
+	if err := recoveredCreateRequest.UnmarshalJSON(messageRawData); err != nil {
+		t.Fatalf("Error decoding create resource request: %s", err)
+	}
+
+	// ... but verification should fail because it is not signed!
+	if err := recoveredCreateRequest.Verify(); err == nil {
+		t.Fatal("Expected Verify to fail since the message is not signed")
+	}
+
+	// We now assume that the resource was created and propagated. With rootAddr we can retrieve the resource metadata
+	// and recover the information above. To sign an update, we need the rootAddr and the metaHash to construct
+	// proof of ownership
+
+	metaHash := createRequest.metaHash
+	rootAddr := createRequest.rootAddr
+	const expectedSignature = "0x1c2bab66dc4ed63783d62934e3a628e517888d6949aef0349f3bd677121db9aa09bbfb865904e6c50360e209e0fe6fe757f8a2474cf1b34169c99b95e3fd5a5101"
+	const expectedJSON = `{"rootAddr":"0x6e744a730f7ea0881528576f0354b6268b98e35a6981ef703153ff1b8d32bbef","metaHash":"0x0c0d5c18b89da503af92302a1a64fab6acb60f78e288eb9c3d541655cd359b60","version":1,"period":7,"data":"0x5468697320686f75722773207570646174653a20537761726d2039392e3020686173206265656e2072656c656173656421","multiHash":false}`
+
+	//Put together an unsigned update request that we will serialize to send it to the signer.
+	data := []byte("This hour's update: Swarm 99.0 has been released!")
+	request := &Request{
+		SignedResourceUpdate: SignedResourceUpdate{
+			resourceUpdate: resourceUpdate{
+				updateHeader: updateHeader{
+					UpdateLookup: UpdateLookup{
+						period:   7,
+						version:  1,
+						rootAddr: rootAddr,
+					},
+					multihash: false,
+					metaHash:  metaHash,
+				},
+				data: data,
+			},
+		},
+	}
+
+	messageRawData, err = request.MarshalJSON()
+	if err != nil {
+		t.Fatalf("Error encoding update request: %s", err)
+	}
+
+	equalJSON, err := areEqualJSON(string(messageRawData), expectedJSON)
+	if err != nil {
+		t.Fatalf("Error decoding update request JSON: %s", err)
+	}
+	if !equalJSON {
+		t.Fatalf("Received a different JSON message. Expected %s, got %s", expectedJSON, string(messageRawData))
+	}
+
+	// now the encoded message messageRawData is sent over the wire and arrives to the signer
+
+	//Attempt to extract an UpdateRequest out of the encoded message
+	var recoveredRequest Request
+	if err := recoveredRequest.UnmarshalJSON(messageRawData); err != nil {
+		t.Fatalf("Error decoding update request: %s", err)
+	}
+
+	//sign the request and see if it matches our predefined signature above.
+	if err := recoveredRequest.Sign(signer); err != nil {
+		t.Fatalf("Error signing request: %s", err)
+	}
+
+	compareByteSliceToExpectedHex(t, "signature", recoveredRequest.signature[:], expectedSignature)
+
+	// mess with the signature and see what happens. To alter the signature, we briefly decode it as JSON
+	// to alter the signature field.
+	var j updateRequestJSON
+	if err := json.Unmarshal([]byte(expectedJSON), &j); err != nil {
+		t.Fatal("Error unmarshalling test json, check expectedJSON constant")
+	}
+	j.Signature = "Certainly not a signature"
+	corruptMessage, _ := json.Marshal(j) // encode the message with the bad signature
+	var corruptRequest Request
+	if err = corruptRequest.UnmarshalJSON(corruptMessage); err == nil {
+		t.Fatal("Expected DecodeUpdateRequest to fail when trying to interpret a corrupt message with an invalid signature")
+	}
+
+	// Now imagine Evil Bob (why always Bob, poor Bob) attempts to update Charlie's resource,
+	// signing a message with his private key
+	if err := request.Sign(falseSigner); err != nil {
+		t.Fatalf("Error signing: %s", err)
+	}
+
+	// Now Bob encodes the message to send it over the wire...
+	messageRawData, err = request.MarshalJSON()
+	if err != nil {
+		t.Fatalf("Error encoding message:%s", err)
+	}
+
+	// ... the message arrives to our Swarm node and it is decoded.
+	recoveredRequest = Request{}
+	if err := recoveredRequest.UnmarshalJSON(messageRawData); err != nil {
+		t.Fatalf("Error decoding message:%s", err)
+	}
+
+	// Before discovering Bob's misdemeanor, let's see what would happen if we mess
+	// with the signature big time to see if Verify catches it
+	savedSignature := *recoveredRequest.signature                               // save the signature for later
+	binary.LittleEndian.PutUint64(recoveredRequest.signature[5:], 556845463424) // write some random data to break the signature
+	if err = recoveredRequest.Verify(); err == nil {
+		t.Fatal("Expected Verify to fail on corrupt signature")
+	}
+
+	// restore the Evil Bob's signature from corruption
+	*recoveredRequest.signature = savedSignature
+
+	// Now the signature is not corrupt, however Verify should now fail because Bob doesn't own the resource
+	if err = recoveredRequest.Verify(); err == nil {
+		t.Fatalf("Expected Verify to fail because this resource belongs to Charlie, not Bob the attacker:%s", err)
+	}
+
+	// Sign with our friend Charlie's private key
+	if err := recoveredRequest.Sign(signer); err != nil {
+		t.Fatalf("Error signing with the correct private key: %s", err)
+	}
+
+	// And now, Verify should work since this resource belongs to Charlie
+	if err = recoveredRequest.Verify(); err != nil {
+		t.Fatalf("Error verifying that Charlie, the good guy, can sign his resource:%s", err)
+	}
+}