swarm/storage/mru: Client-side MRU signatures (#784)

* swarm/storage/mru: Add embedded publickey and remove ENS dep

This commit breaks swarm, swarm/api...
but tests in swarm/storage/mru pass

* swarm: Refactor swarm, swarm/api to mru changes, make tests pass

* swarm/storage/mru: Remove self from recv, remove test ens vldtr

* swarm/storage/mru: Remove redundant test, expose ResourceHash mthd

* swarm/storage/mru: Make HeaderGetter mandatory + godoc fixes

* swarm/storage: Remove validator prefix for metadata chunk

* swarm/storage/mru: Use Address instead of PublicKey

* swarm/storage/mru: Change index from name to metadata chunk addr

* swarm/storage/mru: Refactor swarm/api/... to MRU index changes

* swarm/storage/mru: Refactor cleanup

* swarm/storage/mru: Rebase cleanup

* swarm: Use constructor for GenericSigner MRU in swarm.go

* swarm/storage: Change to BMTHash for MRU hashing

* swarm/storage: Reduce loglevel on chunk validator logs

* swarm/storage/mru: Delint

* swarm: MRU Rebase cleanup

* swarm/storage/mru: client-side mru signatures

Rebase to PR #668 and fix all conflicts

* swarm/storage/mru:  refactor and documentation

* swarm/resource/mru: error-checking  tests for parseUpdate/newUpdateChunk

* swarm/storage/mru: Added resourcemetadata tests

* swarm/storage/mru: Added tests  for UpdateRequest

* swarm/storage/mru: more test coverage for UpdateRequest and comments

* swarm/storage/mru: Avoid fake chunks in parseUpdate()

* swarm/storage/mru: Documented resource.go extensively

moved some functions where they make most sense

* swarm/storage/mru: increase test coverage for UpdateRequest and

variable name changes throughout to increase consistency

* swarm/storage/mru: moved default timestamp to NewCreateRequest-

* swarm/storage/mru: lookup refactor

* swarm/storage/mru: added comments and renamed raw flag to rawmru

* swarm/storage/mru: fix receiver typo

* swarm/storage/mru: refactored update chunk new/create

* swarm/storage/mru:  refactored signature digest to avoid malleability

* swarm/storage/mru: optimize update data serialization

* swarm/storage/mru: refactor and cleanup

* swarm/storage/mru: add timestamp struct and serialization

* swarm/storage/mru: fix lint error and mark some old code for deletion

* swarm/storage/mru: remove unnecessary variable

* swarm/storage/mru: Added more comments throughout

* swarm/storage/mru: Refactored metadata chunk layout + extensive error...

* swarm/storage/mru: refactor cli parser
Changed resource info output to JSON

* swarm/storage/mru: refactor serialization for extensibility

refactored error messages to NewErrorf

* swarm/storage/mru: Moved Signature to resource_sign.
Check Sign errors in server tests

* swarm/storage/mru: Remove isSafeName() checks

* swarm/storage/mru: scrubbed off all references to "block" for time

* swarm/storage/mru: removed superfluous isSynced() call.

* swarm/storage/mru: remove isMultihash() and ToSafeName functions

* swarm/storage/mru: various fixes and comments

* swarm/storage/mru: decoupled cli for independent create/update
* Made resource name optional
* Removed unused LookupPrevious

* swarm/storage/mru: Decoupled resource create / update & refactor

* swarm/storage/mru: Fixed some comments as per issues raised in PR #743

* swarm/storage/mru: Cosmetic changes as per #743 comments

* swarm/storage/mru: refct request encoder/decoder > marshal/unmarshal

* swarm/storage/mru: Cosmetic changes as per review in #748

* swarm/storage/mru: removed timestamp proof placeholder

* swarm/storage/mru: cosmetic/doc/fixes changes as per comments in #704

* swarm/storage/mru: removed unnecessary check in Handler.update

* swarm/storage/mru: Implemented Marshaler/Unmarshaler iface in Request

* swarm/storage/mru: Fixed linter error

* swarm/storage/mru: removed redundant address in signature digest

* swarm/storage/mru: fixed bug: LookupLatestVersionInPeriod not working

* swarm/storage/mru: Unfold Request creation API for create or update+create
set common time source for mru package

* swarm/api/http: fix HandleGetResource error variable shadowed
when requesting a resource that does not exist

* swarm/storage/mru: Add simple check to detect duplicate updates

* swarm/storage/mru: moved Multihash() to the right place.

* cmd/swarm: remove unneeded clientaccountmanager.go

* swarm/storage/mru: Changed some comments as per reviews in #784

* swarm/storage/mru: Made SignedResourceUpdate.GetDigest() public

* swarm/storage/mru: cosmetic changes as per comments in #784

* cmd/swarm: Inverted --multihash flag default

* swarm/storage/mru: removed Verify from SignedResourceUpdate.fromChunk

* swarm/storage/mru: Moved validation code out of serializer
Cosmetic / comment changes

* swarm/storage/mru: Added unit tests for UpdateLookup

* swarm/storage/mru: Increased coverage of metadata serialization

* swarm/storage/mru: Increased test coverage of updateHeader serializers

* swarm/storage/mru: Add resourceUpdate serializer test

1 files changed, 189 insertions, 0 deletions

diff --git a/swarm/storage/mru/metadata.go b/swarm/storage/mru/metadata.go
new file mode 100644
index 000000000..0ab0ed1d9
--- /dev/null
+++ b/swarm/storage/mru/metadata.go
@@ -0,0 +1,189 @@
+// Copyright 2018 The go-ethereum Authors
+// This file is part of the go-ethereum library.
+//
+// The go-ethereum library is free software: you can redistribute it and/or modify
+// it under the terms of the GNU Lesser General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// The go-ethereum library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+// GNU Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+package mru
+
+import (
+	"encoding/binary"
+	"hash"
+
+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/swarm/storage"
+)
+
+// ResourceMetadata encapsulates the immutable information about a mutable resource :)
+// once serialized into a chunk, the resource can be retrieved by knowing its content-addressed rootAddr
+type ResourceMetadata struct {
+	StartTime Timestamp      // time at which the resource starts to be valid
+	Frequency uint64         // expected update frequency for the resource
+	Name      string         // name of the resource, for the reference of the user or to disambiguate resources with same starttime, frequency, owneraddr
+	Owner     common.Address // public address of the resource owner
+}
+
+const frequencyLength = 8 // sizeof(uint64)
+const nameLengthLength = 1
+
+// Resource metadata chunk layout:
+// 4 prefix bytes (chunkPrefixLength). The first two set to zero. The second two indicate the length
+// Timestamp: timestampLength bytes
+// frequency: frequencyLength bytes
+// name length: nameLengthLength bytes
+// name (variable length, can be empty, up to 255 bytes)
+// ownerAddr: common.AddressLength
+const minimumMetadataLength = chunkPrefixLength + timestampLength + frequencyLength + nameLengthLength + common.AddressLength
+
+// binaryGet populates the resource metadata from a byte array
+func (r *ResourceMetadata) binaryGet(serializedData []byte) error {
+	if len(serializedData) < minimumMetadataLength {
+		return NewErrorf(ErrInvalidValue, "Metadata chunk to deserialize is too short. Expected at least %d. Got %d.", minimumMetadataLength, len(serializedData))
+	}
+
+	// first two bytes must be set to zero to indicate metadata chunks, so enforce this.
+	if serializedData[0] != 0 || serializedData[1] != 0 {
+		return NewError(ErrCorruptData, "Invalid metadata chunk")
+	}
+
+	cursor := 2
+	metadataLength := int(binary.LittleEndian.Uint16(serializedData[cursor : cursor+2])) // metadataLength does not include the 4 prefix bytes
+	if metadataLength+chunkPrefixLength != len(serializedData) {
+		return NewErrorf(ErrCorruptData, "Incorrect declared metadata length. Expected %d, got %d.", metadataLength+chunkPrefixLength, len(serializedData))
+	}
+
+	cursor += 2
+
+	if err := r.StartTime.binaryGet(serializedData[cursor : cursor+timestampLength]); err != nil {
+		return err
+	}
+	cursor += timestampLength
+
+	r.Frequency = binary.LittleEndian.Uint64(serializedData[cursor : cursor+frequencyLength])
+	cursor += frequencyLength
+
+	nameLength := int(serializedData[cursor])
+	if nameLength+minimumMetadataLength > len(serializedData) {
+		return NewErrorf(ErrInvalidValue, "Metadata chunk to deserialize is too short when decoding resource name. Expected at least %d. Got %d.", nameLength+minimumMetadataLength, len(serializedData))
+	}
+	cursor++
+	r.Name = string(serializedData[cursor : cursor+nameLength])
+	cursor += nameLength
+
+	copy(r.Owner[:], serializedData[cursor:])
+	cursor += common.AddressLength
+	if cursor != len(serializedData) {
+		return NewErrorf(ErrInvalidValue, "Metadata chunk has leftover data after deserialization. %d left to read", len(serializedData)-cursor)
+	}
+	return nil
+}
+
+// binaryPut encodes the metadata into a byte array
+func (r *ResourceMetadata) binaryPut(serializedData []byte) error {
+	metadataChunkLength := r.binaryLength()
+	if len(serializedData) != metadataChunkLength {
+		return NewErrorf(ErrInvalidValue, "Need a slice of exactly %d bytes to serialize this metadata, but got a slice of size %d.", metadataChunkLength, len(serializedData))
+	}
+
+	// root chunk has first two bytes both set to 0, which distinguishes from update bytes
+	// therefore, skip the first two bytes of a zero-initialized array.
+	cursor := 2
+	binary.LittleEndian.PutUint16(serializedData[cursor:cursor+2], uint16(metadataChunkLength-chunkPrefixLength)) // metadataLength does not include the 4 prefix bytes
+	cursor += 2
+
+	r.StartTime.binaryPut(serializedData[cursor : cursor+timestampLength])
+	cursor += timestampLength
+
+	binary.LittleEndian.PutUint64(serializedData[cursor:cursor+frequencyLength], r.Frequency)
+	cursor += frequencyLength
+
+	// Encode the name string as a 1 byte length followed by the encoded string.
+	// Longer strings will be truncated.
+	nameLength := len(r.Name)
+	if nameLength > 255 {
+		nameLength = 255
+	}
+	serializedData[cursor] = uint8(nameLength)
+	cursor++
+	copy(serializedData[cursor:cursor+nameLength], []byte(r.Name[:nameLength]))
+	cursor += nameLength
+
+	copy(serializedData[cursor:cursor+common.AddressLength], r.Owner[:])
+	cursor += common.AddressLength
+
+	return nil
+}
+
+func (r *ResourceMetadata) binaryLength() int {
+	return minimumMetadataLength + len(r.Name)
+}
+
+// serializeAndHash returns the root chunk addr and metadata hash that help identify and ascertain ownership of this resource
+// returns the serialized metadata as a byproduct of having to hash it.
+func (r *ResourceMetadata) serializeAndHash() (rootAddr, metaHash []byte, chunkData []byte, err error) {
+
+	chunkData = make([]byte, r.binaryLength())
+	if err := r.binaryPut(chunkData); err != nil {
+		return nil, nil, nil, err
+	}
+	rootAddr, metaHash = metadataHash(chunkData)
+	return rootAddr, metaHash, chunkData, nil
+
+}
+
+// creates a metadata chunk out of a resourceMetadata structure
+func (metadata *ResourceMetadata) newChunk() (chunk *storage.Chunk, metaHash []byte, err error) {
+	// the metadata chunk contains a timestamp of when the resource starts to be valid
+	// and also how frequently it is expected to be updated
+	// from this we know at what time we should look for updates, and how often
+	// it also contains the name of the resource, so we know what resource we are working with
+
+	// the key (rootAddr) of the metadata chunk is content-addressed
+	// if it wasn't we couldn't replace it later
+	// resolving this relationship is left up to external agents (for example ENS)
+	rootAddr, metaHash, chunkData, err := metadata.serializeAndHash()
+	if err != nil {
+		return nil, nil, err
+	}
+
+	// make the chunk and send it to swarm
+	chunk = storage.NewChunk(rootAddr, nil)
+	chunk.SData = chunkData
+	chunk.Size = int64(len(chunkData))
+
+	return chunk, metaHash, nil
+}
+
+// metadataHash returns the metadata chunk root address and metadata hash
+// that help identify and ascertain ownership of this resource
+// We compute it as rootAddr = H(ownerAddr, H(metadata))
+// Where H() is SHA3
+// metadata are all the metadata fields, except ownerAddr
+// ownerAddr is the public address of the resource owner
+// Update chunks must carry a rootAddr reference and metaHash in order to be verified
+// This way, a node that receives an update can check the signature, recover the public address
+// and check the ownership by computing H(ownerAddr, metaHash) and comparing it to the rootAddr
+// the resource is claiming to update without having to lookup the metadata chunk.
+// see verifyResourceOwnerhsip in signedupdate.go
+func metadataHash(chunkData []byte) (rootAddr, metaHash []byte) {
+	hasher := hashPool.Get().(hash.Hash)
+	defer hashPool.Put(hasher)
+	hasher.Reset()
+	hasher.Write(chunkData[:len(chunkData)-common.AddressLength])
+	metaHash = hasher.Sum(nil)
+	hasher.Reset()
+	hasher.Write(metaHash)
+	hasher.Write(chunkData[len(chunkData)-common.AddressLength:])
+	rootAddr = hasher.Sum(nil)
+	return
+}