The language of the network

Another quick post, I hope this makes sense, it is perhaps the most exiting aspect of MaidSafe I have found so far and the consequences of this are far reaching indeed. It will not be obvious, but please ask questions on maidsafe.org and I will try and explain more of why this is very important, not only to launch with but to move forward and also model system components very quickly.

The notes we never heard

MaidSafe, that strangely named bunch of folks who are looking to change the world with something that sounds:

  • Amazing
  • Impossible
  • Frightening

Yes the vision that is rock solid and unmoving against all odds and all comers no matter what. The vision to give to all the people of the world Privacy Security and Freedom, but how do they do it? how do they explain it?

Well until now, not very well. There are tests, experimental data, research, 18 months of network simulations modelling, papers, interviews, videos, blog posts, comments and answers, email trails, presentations and all in all it still sounds … too good to be true! For many it sound’s like an impossibility. Well it works, it produces results and we still cannot explain it. Is this about to change ?

The network critical parts

Routing is a critical and central part of the codebase, it provides routing information and security of the network, handles churn and ensures the xor relationships are intact and accurate. It is the workhorse of the network relying heavily on a very efficient reliable UDP implementation underneath. Crux, our new rudp is that implementation. I wanted to check this code in detail. I was particularly interested in the consensus checks and ensuring they were solid and as efficient as possible (they will always be more efficient as time goes by, but for now they need to be rock solid, so worth the effort), Consensus groups and checks are a very important part of the network for group based actions to ensure the correct authority is network agreed. This along with client digital signatures makes all the various authorities work in unison and provide harmony out of apparent chaos when you look at the network.

In routing we have a GroupSize and QuorumSize, the Quorum is the number we require to validate a groups intentions as an acceptable level of comfort the intent is agreed with the group.

Software development

Many who know me or know of me will know I am an Engineer, as happy with a hammer or soldering iron as I am with a compiler. I like Engineering and I think I act like an Engineer, what cannot be fixed? give it here, I accept the challenge!, Impossible great that will be fun, lets work out how to do it!.

Software development to me is Engineering, I care not if I am trying to loosen a nut with a damp dish-towel or making code do something for me. I see a problem, look at what is available and attack! Then though you have to build amazing things and it is different, you plan, build, test, build, plan, build test, build and so on, getting to the place where it makes sense. Then start all over again and implement the answer, neat repeatable and stunning.

With MaidSafe the problem we took on is, the Internet (web etc.) is fundamentally broken and taking the wrong route, answer, well fix it! OK yer on.

So we did, with many years of build, test, plan, build etc. we got a system that could fix this issue and went though testnets. On testnet2 I decided with a lot of coaxing to get back to code and re-factor routing for stability and security analysis in particular. In doing so I went though all sorts of hell, no sleep and fighting logic every day asking why like this? why like that? (not the code the sheer complexity) Ok rewrite part of routing, I can do this fast, I always can when I focus. So no xmas for me, there’s work to be done (and the lines of “the band played waltzing Matilda” resonates, especially “never knew there were worse things than dyin”).

Tough days, harder nights and I was in a familiar place, deep deep thought, what are these types in the code (c++ has types, they are called classes), why do they exist, what fundamental facets make them unique enough to exist as a type? Yes questioning God, gravity and C being a limiting factor of the universe, this was all familiar. I was in innovation mode again, that dark place where nothing can get in the way of your thoughts, not even a million blog posts and comments, questions etc. nothing. You are in cruise control, focussed on a burning question, why so complex? why? What am I missing?

The MaidSafe issue was again prevalent, I Cannot explain this to anyone, it is stunning and beautiful, but so so complex.

Back to school

Then it was back to school thinking, I remembered the huge equations of experimental data on a blackboard and what we did to it. We factored it. What starts out as a massive multi variable mess of junk, eventually factors to x = 2y + 6.

This is what MaidSafe is, un-factored (well not factored enough). This is why we cannot explain it, it’s why changes to the code are so hard. It is doing something nothing else has done and doing it well, but with such huge complexity. I am not happy, we never factored the equation. How though, how?

Back to types, what are these and why?

I deviate from some software developers here, I see types not as useful code helpers to stop you typing the wrong name or something into a functions. I see types as a thing and that thing has a particular Genus, you are making something with a type, something unique and with purpose, different from any other thing you have made. So it needs thought. Really think hard, what is it and why.

So back to routing, I am looking at the network as an electron or message would, the vast array of connections all held together in xor space, I know I can do certain things, but what can the network tell of me. Can it recognise where I have been, or where I am and if so what am I when I am here at this place in space and time?

I know we have consensus groups, I know folk do not find it easy,  but they are there all over the place inter-lapping, constantly changing and deterministic (well if you can stop time and just analyse the network fully, you may be able to identify the state, but like Zeno’s arrow it will give more questions than answers). I know these groups exist, I know what they do and I know there are lots of them, each interwoven with others in a dance that makes your head spin in this cable.

I then notice something, something that makes everything stop. I see a familiar thing, a pattern, a repeatable, explainable pattern and it is describable. In the mist there is a view coming to me and it’s taking a form, what is it?

It’s a consensus group type, not only that but it stamps itself on a message until the next group gets to it. There is something I can see and more importantly name. These patterns are everywhere. I look at the vault code. Over 260 c++ files, beautiful code, amazing skill, but it’s doing something I can see. Most of the time this code is trying to work out all the functions and features available at any stage in a messages travel. It’s looking for the answer, but not always the same way. It’s not hearing the music and it makes up for it with more code and more algorithms.

Now I am in shock, I always knew we would uncover the secrets of the network, I knew it was our path to follow and find this, but was this it? I had to take time out, Dude needs a walk on the beach, off we go. I end up running back to the code, I was right this is it. I need to think.

Ok what is it

So the revelation is this and it is quite simple. There are several persona types, all personas are actually one of these regardless of what code we put in it. A persona group is a certain and guaranteed thing that the network in the real world can see, validate and secure. This all happens very dynamically and the network knows these things. Even though we have calculated this, the network knows and it knows better that any amount of advanced code we can throw at it. This is one of those, “of course it is“! moments. The network can tell us who we are with respect to a message, we do not need to calculate it in upper layers, routing knows already it just needs to tell us and we must listen. This is such an under-researched area that this is a truly amazing find, in the group consensus model we have this really focusses thinking and allows significant improvements in explaining also everything about the network. Of course self authentication, encryption and data types etc. are all still very new, but this is a huge aspect (several man years of effort to find this) and it will transform not only us, but hopefully this industry of decentralised approaches to software and services based on it.

People will (and should) look at this now and say, well that’s obvious. I can assure you, it only is after you see it (like a wheel or boat).  So what are these types and why do they exist?

Here we go.

We have (4 type, only 4, no more no less)

Client Manager – This is a type of persona this is the group of routing nodes closest to a client node. They can tell as they have a connection that is not a routing table node, so it must be a client! Examples of these types are MaidManager (the group that looks after a Maid account), MpidManager (the group that looks after public name and public shares/drive for public clients)

Nae Manager – Network Addressable Element manager groups. They know they are this type of group as they are close to the address that equals the name of the network addressable element (not a Network Addressable Node, but data or function elements). Examples of these are DataManagers (look after data pointers) and VersionManagers (looks after directory versions and any other mutating directly addressable node)

Node Manager – This is the group surrounding a node and they know as the node appears in their routing table. An example is a PmidManager (the group looking after nodes holding data).

Managed Node – This is a routing node in a group of Node Managers such as a PmidNode (i.e. a node actually supposed to be holding a data element). Future, ComputeNode to handle computations (using zk-snark etc.)

That’s is it. Now we call these authorities as they represent a particular authority. So lets look at a client putting data on the network.

  • Client sends message to his own address on the network
  • ClientManagers receive this and check it is a client he has enough storage available (paid enough safecoin)
  • They send to the DataManagers
  • DataManagers check the from Authority is a ClientManager group and look to see if data is already stored. If so then Finish, else
  • DataManagers send to a PmidManager group.
  • PmidManagers check from authority was DataManagers and they store in a node close to the address of the group.
  • PmidNode receives this Put and checks the from authority was his PmidManager group.
  • Pmid Node stores data. Finish.

“Hold on! I spot a problem”. It is OK these nodes can confirm who they are, but here we mention from authority. It seems the from authority is required to confirm the chain of deterministic actions is correct, otherwise people can claim to be anyone.

This is the other neat part actually. We know and have proven experimentally many times the quorum of a group is secure. So you know what group you are (remember the network can absolutely tell you with incredible accuracy what persona you are). So you send your persona type in the message to the next group. The next group then accumulate results, check your signature and validate all your group said they were persona type X. So this seals the chain of events into a secure deterministic chain. This provides great security, but more importantly the language now becomes simple.

Put 

Client -> ClientManager<->NeaManager->NodeManager->ManagedNode

Get

Client <->NaeManager<->ManagedNode

We can add in symbols such as ->>>> to represent going to a group and some more small symbols (such as <-> meaning may return result from here) and we have a complete language to describe the vault code (to be published soon). Importantly given such strict rules and types this can be coded very quickly with a strict specification. We have tried to do some things that would break security and guess what, this language stops it dead in it’s tracks. So we can develop very quickly new persona types for these categories for many things, safecoin, compute, AI engine, Search and many more. This speeds up development to a point we should be able to test different ideas in days, not weeks and months as it is with the huge un-factored code base. Please do not misunderstand this does not mean code in days (although a massive reduction in time for code is obvious), I mean we can write down data flows and analyse them in a manner people understand easily. This allows  many people to be involved in such thought experiments prior to implementation. This is a huge step, no longer does one or perhaps two people understand what is happening, but everyone can (within reason).

This is huge and the most significant find I have had in 9 years working hard on these problems. It is still very exciting to me, even now. For instance our vault code will now go from over 260 template heavy and complex source files to a handful of very simple header files.  Testnet3 will include this, Crux and routing_V2 which adds in a sentinel, this is a crucial accumulation and security check. So we can confirm cryptographic authority (a client has signed and action on something he owns) and group / consensus based authority. A description of the sentinel is probably required now, so sorry for the inordinately long post.

Sentinel overview

Quick intro to network consensus, authority and crypto usage.

In a decentralised autonomous network there are many challenges to face. One such challenge is the range of attacks that consist of Sybil / Spartacus and forgery attacks (did the message really come from who you think). One of the simplest attack to foil is the forgery attack, thanks to asymmetric cryptography. This allows for a public key to be known by everyone and when anything is encrypted with this key it can only (supposedly) be decrypted by the private key of that keypair. Assuming a reasonable algorithm, keysize and implementation this holds true.

This also removes the Spartacus type attacks (claim to be another identity)., but not necessarily sybil attacks, where an attack on a bit of a network is enough to persuade the rest of the network that any request is valid or indeed anything asked of that part of the network is done as expected. To overcome this MaidSafe have several techniques used in parallel. These boil down to

  1. Have nodes create key chains (a chain of keys each signing the next until one is selected). We call these Fobs. A Publicfob consists of a public_key & signature as well as a name field. The name is the SHA512HASH(public_key+signature) making forging a key crypto hard (we can confirm the signature is also signed by a valid key pair by checking the signature there, where this ‘pure key’ is self signed). The Fob type is this PublicFob + the private key.
  2. Ask network to store the PublicFob for a node. The network will accept this if the node has certain characteristics (based on rank – later discussion) and the key is less than 3 leading bits different from the current group of nodes. This makes key placement distribute equally across the address range (as for rank consider only a single non ranked node allowed per group, and failure to increase rank means the key is deleted from the network and has to be re-stored if possible).
  3. This now resembles a PKI network where to speak to node ABC you go get the PublicFob at ABC and either encrypt a message to the node or check a message from the node is signed by using that PublicFob.public_key. The difference being no central authority exists and the network distributes and collects keys as any DHT would (except in this case the DHT is secured by the very PKI it manages). So this is very secure and does not require any human intervention (unlike a certificate authority)
  4. Assemble nodes into groups that will act in unison on any request/response. So these groups are selected to be large enough to ensure a Sybil attack would require at least 3X network size of attackers to be able to join (a single attacker with no other node types joining). The magic number here is 28, realistically this number is closer to 17.
  5. Allow a failure rate as failures will definitely happen. This is done by having a GroupSize of say 32 and set the QuorumSize to 28. This means for any action we require 28 nodes close to a target address to agree and carry out an action.

This Quorum creates a mechanism where another group or node can believe the action is correct and valid. This is called group consensus.

The group consensus provides the network a way to request or carry out actions and ensure such requests are valid and actions actually done. This is required as the network looks after itself (autonomous).

A client has a close group and requires to persuade this group to request the network take an action which Puts something on the network (a data element/message etc.) Clients create data and messages, the network handles these. As the client cannot just connect to an arbitrary group and demand something be done, they connect to their close group and register themselves (with their Fob) an account. The close group can then be persuaded by the client to request another part of the network create something (a Put). In the case of Maidsafe the close group request the client pay via safecoin (it used to be they paid with storage that another group managed an agreed). So the client persuades the close group to put data. (ignore how payment is made, it actually requires client send safecoin to a provable recycle part of the network (another group confirms this)).

So a client can sign request to the group (crypto secure) and the group can check validity of the request and then ask the appropriate group close to the address of the data or message to accept this request.

After anything is put the client can mutate this thing directly (if they have signed it). This is the case for directory entries, where a client can add versions to a list (StructuredDataVersion) as it was Put signed by the client. So the owner of the signature can sign a request to alter this. This is crypto secured authority and does not require the close group for consensus.

In the case of groups requesting actions then we have group based consensus and the network grants authority based on a group that can be measured as a valid close group, where each member has signed a request as being from that close group. This authority is what the sentinel confirms prior to the routing object processing an action request. Almost all messages are Sentinel checked, with the exception of get_group as this fetches Fob’s which are self validating and it fetches a copy of all Fobs from all group members and confirms they agree and validate. Get_group is only used for making sure we connect to our close group as a client or node.

Sentinel components

The sentinel consists of few components and requires no network connection. This is to allow for such a crucial element to be fully tested. The elements are limited to two (2)accumulator pairs. There are two pairs for two different authority types,

  1. Node level direct authority (i.e. could be a client)
  2. Group base consensus

In 1 we just accumulate a single message and get the Fob to check a signature. In 2 We require to get at least QuorumSize messages and this is for group based consensus and then we get the Fobs and again check signature to confirm the group. We also check the nodes are as close to each other in xor space as our own group is (with varying error rate)

To achieve this the process is

  1. Message Arrives
  2. Check Accumulator has seen it, if not Send a GetKey request (for a group or single node)
  3. Add to accumulator. If return is true then check the key accumulator of that pair -> if true then confirm the signature with the Fob (asymm::CheckSignature(Fob.public_key, message)

If the key accumulator did not have the key(s) accumulated (i.e. accumulator.CheckQuorumReached) then we do nothing and continue with other work. Then though

  1. Key arrives (from GetKey response)
  2. Check value accumulators have(address) the address is the source_id+message_id of the request, may be a group ID or nodeId
  3. If not found then ignore message
  4. Otherwise accumulator.Add(key) to the proper key accumulator of the pair
  5. If this returns true, then we get the keys and values (via accumulator.GetAll() calls from both accumulators and confirm signatures. group and return a valid message to the object holding the sentinel (the sentinel add call will be async)

The accumulators are LRU cache based and will empty old messages that do not confirm in this manner.

Conclusion

This language which describes the vaults and consensus logic will help us get closer to formal proofs, explaining the network and moving much much faster with a significant reduction in codebase. It also allows with a crtp template pattern a complete separation of vault logic and the network, so at last we can test fully logic and then attach it to the network. The last few months have been a special hell for me as we are in launch mode and then this. The opportunity is way to large to bypass so we are implementing this as we speak ( at breakneck speed ) and it’s looking fantastic. I will be posting to the community forum with updates, but next week I am taking a few days off and heading North for a break. I will do so in a huge sigh of relief and inner peace after this discovery. It is still something that has shook me to the bones in its elaborate beauty and simplicity. Truly amazing.

Posted in complex systems, MaidSafe, strategy

Experts and the corruption of truth!

This is another observation post and not a member of MaidSafe series. I need to get around to writing more on that. This is related though for sure. It’s also a rant which may not be very useful. This should perhaps be entitled Why David Irvine is absolutely rubbish at answering questions. As usual, like presenting I do not research, read up or prepare this stuff so expect mistakes.

Show me the maths proof!

I hear this a lot with respect to MaidSafe and generally from ‘experts’. I get very frustrated by it which seems to show a lack of ability to some. I argue this is not the case. We have recently developed some maths equations for the security of close groups in MaidSafe, they are far from simple, but agree with results. I must add they are probably not understandable by many, not really. (we will publish these in papers, but believe me it’s a tiny part of a large story, a headline, not full research). I am not putting it in this story for the reason I am writing this story, it’s a badge, like standing in front of a whiteboard with prepared equations, or people who speak in unknown languages of maths to people, in attempt to meet an ‘expert’ status. I will place these equations in papers where they fit for maths people, to portray part of a story, but only part.

You see, to me it seems a complex equation, preferably with some calculus and well placed greek letters can look amazing. It also can be a complete bluff, smoke and mirrors. A good example of this is I was recently chatting to a lecturer/researcher in a local Uni, he is a leading expert in DHT technology and well read and published (even in the peer-to-peer handbook – a bible of DHT tech). This person who for years, along with the community of industry ‘experts’ figured D1HT was a truly amazing, scalable, fast and very efficient DHT. This was published in many places and well-regarded, the maths was impressive and ‘proved’ the case.

For years we had maths that proved bees could not fly. We had maths to prove aeroplanes flew by splitting molecules of air and the longer areas of the top of the wing made these go faster lifting the wing up (well I am reducing this a little). Prior to this we had maths that showed how everything worked (Newton provided much physics to prove this),  we also had a ton of maths to prove zero did not exist (in fact was the devils number) and the world was flat.

Prove it by measurement!

Another field of physics/maths to prove things by experimentation. I was watching a Stephen Hawkins series there and heard a few times, “this measurement proved my thesis”, I also read that a lot in a book called why does e=mc^2 by the professor Brian Cox (the pop band chap, very entertaining and great). These tests do not prove any theory, they agree with a thesis within the confines of the variables involved in this test, which necessarily is limited. So it’s a limited agreement with a thesis, at best.

Make it repeatable!

OK, repeatable experiments are really good, but !

We repeatedly seen a bee fly, we repeated measured planes flying, we repeatedly built things using Newtonian maths/physics, we repeatedly worked for 600 years after Aristotle without the number zero (lets not even discuss complex numbers of negative, and infinity, well …), well maybe not all of us, the Chinese had gaps in numbers to represent zero, but not a name for it (as the mafia would say).

So the question is repeat what ?

Make it intelligent!

Aha, lets prove stuff by giving them, either a complex name or better a complex name with a muckle big equation with loads of components and if it does not fit, shove in a constant or two (it was good enough for Einstein). Now there you go we are experts, nearly nobody knows what we are talking about and we can stand beside a blackboard full of chalk, or even publish a ton of scientific peer-reviewed papers. We are set, experts, proof and above all a satisfied (for now) ego.

Hold on

  • The bee flew (we now believe it flaps its wings really hard)
  • D1HT failed (yes somebody built it and it does not work)
  • Aircraft wings now apparently are believed to behave like a sail on a boat (are we surprised molecules are not splitting in some weird dance with matter ?)
  • Einstein ‘proves’ Newton was wrong
  • Hawkings ‘proves’ Einstein was wrong
  • Recent ‘proof’ shows Newton was right

A single test or maths equations that agrees with a theory does not prove a theory and make a law.

So does e=MC^2, well we don’t know! but we all believe it to be true as though it were a law or proven theory, we teach it as such. Well all this is a problem, there seems to be no agreement. We also need to be careful of peer review and field testing as the NSA / NIST have shown that what is reported and agreed is not necessarily the truth, so this is a huge problem. A huge talk on its own, suffice to say, you need to believe the people who measure and give results (Pharma companies are already known to misrepresent results and actually throw away bad results)

Make it simple

This is the crux of the matter, it just is not simple.

We all need to be able to understand our first principles are probably wrong. We need to realise field testing only tells us a bit of what we need to know within limits. It is just not simple, if its simple it need to be a belief like religion and that leads to people holding onto something that is not proven. Those people will stop innovation and improvement on our global knowledge base.

What do we need to know

Maths is hugely important and is a must, but do not use it as a badge, it’s a tool like a spanner, it may be beautiful to you, but don’t shove it in folks faces.

Measurements are also hugely important to home in on improving the validity of a theory, but should never be stated as proof of a theory in their own right, unless they cover 100% of variables. We live in a pretty variable universe so it would be a brave person …

Repeated measurements really help, but repeat with what variables, do you use the same seed values or make them random (for many tests)?  This is why experimental physics is hard (very hard), there is a huge amount of things to consider.

Another important thing to consider is change, not only of your field but all around you. For instance RSA was very secure and beyond feasible for a computer to break. Now there are at least three things I know of that influence this a lot

  • Computers are exponentially more capable of solving factors via brute force.
  • Discreet maths is moving very fast, so the amount of work to solve these primes is way less than was thought possible when the RSA algorithm was created.
  • Cloud / grid computing allows huge resources to be applied and it can be costed (so we know its circa $5 of amazon time to solve a 256 RSA key)

Never mind far off quantum computing and all that clever sounding chat about qubits and shor’s algorithms etc. (oh nearly became an expert there!), it’s all part of a large pot of stuff we need to know.

Conclusion

There is no short-cut, there is no final proof, we are merely allowed to see only part of what makes the universe work, we do not know gravity, anti matter, black holes or even if any of these things exist! We cannot even tell why prime numbers happen in the order they do, what we do not know is very simple things that we really should know. So how on earth could we know the big stuff? Bottom line, this game we are playing only shows us a few of the rules at any one time, things will change as we discover more rules, but they force us to reconsider all our previous moves continually and as nature shows us more rules it will force us to be humble and start again.

The number of unknowns is enormous and trying to ignore them by simply an equation, fancy word for something, a measurement or even a series of experiments is simply not enough. All together they offer an ability to start to ask, none of them offer a final answer (and never will). So don’t be an expert, be an explorer and if you are nice to your fellow explorers they may even show you ways you have not yet considered. If they talk in maths riddles and hide behind fancy papers and equations then they are safe to ignore. There is no easy answer, only more information and potentially all you know may be wrong, certainly the majority is certainly wrong, so don’t be a believer, be ready to infer new conclusions as you find out more info, which may not even look related. So look at everything and prepare for massive surprises, they will happen!

This is why I struggle to give simple answers to folk, I hate lies and part of a truth is closer to a lie that saying nothing. Many understand this position, but many don’t yet. It is interesting to see though that the experts seem to be the very people who are the believers and not explorers, when folk also realise what they know is trivial and likely wrong then perhaps things will move along faster.

In saying that I also agree that the inability to easily explain something is an indication of a lack of understanding. A quandary, well yes … Just another thing I don’t know, I wish I did.

Posted in Uncategorized

MaidSafe part III – Joining & anonymity

This is a large area of the network, partially covered  in a small post. I hope it makes sense. My intent is lots of posts and all small, I think it works better.

A lot of chat about, is the system really anonymous is based on tracking IP addresses etc. I was surprised recently when I heard that Tor/i2p exposed users ip address at compromised web sites. Not because there was a flaw, that happens, but because the source IP address was available at all.  I thought further and it made sense. There is a significant difference and this issue will always exist in those systems as they are currently designed.  This is not an anti Tor or i2p post in any way, but a foray into a different approach. I would love the TOR/i2p folks to be part of this journey, so poking a stick in their eyes is not an intent of mine.

A different approach

As we covered in earlier parts of this series, MaidSafe uses a secured DHT implementation, based on XOR networking.  This is hugely important, the DHT has the ability to hold information. Some of that is encrypted and secured (in our case all of it bar, public data and public keys).  So lets not dive too deeply in that for now. Keep in mind though this is not an encryption mechanism to traverse todays Internet (web services), its very different.

This is peoples data we are talking about, it is very important!

It may be a debate to some, but it should be restricted to pure logic where possible.

 Types of nodes on MaidSafe

There are two main node types for now (to keep it simple), a client and a vault. So lets recap

Client -> Unique and private ID that has no link to any public ID or name. This ID is used to GET and PUT data. It is not linked to a person or a name, it is a unique 512 bit id. Even owners will probably not know what it is.  A client does not perform routing tasks, that means it is like a passive producer/consumer of information.

Vault-> Unique and private ID that has no link to any public ID or name. This ID is used to store data. These ID’s are the routing infrastructure and what we all connect to, essentially.

Connection to network

Both node types join the network in fundamentally the same way. They read from either a locally cached list of previously known nodes, or they fall back to hard coded nodes in the source code. The nodes they list have IP:PORT and Public keys. The node will encrypt a message to one of these nodes requesting login (or connect).  The bootstrap node gathers this info and returns it to the joining node (encrypted).

The joining node then connects (encrypted) to the closest nodes as returned by the bootstrap node.

So there you have it, 100% encrypted communications from the very first message.  [note the node connects to multiple bootstrap nodes to confirm the answers match and a user can be emailed a connection list from a friend if they wish. This gets away from arguments about centralised root nodes etc, which is pretty silly, but folk do argue that point].

Operate on the network

As a client you will join / login (explained in a later post) as above and be connected. Many people ask, can a client store data (for the network) etc. or be part of routing and we say no. This seems harsh, but important.  I will explain.

A client connects to its close nodes (with an anonymous ID as explained) and then requests data etc. The request goes to the close nodes (who do know the IP:PORT)  and is then relayed across the network (with the IP:PORT stripped).  At the end or location where the data exists, it’s sent back to the anonymous ID. Each node sends this closer to the ID and the close nodes eventually deliver it.

So on the network there is no notion of IP:PORT for messages unless you are close to the node (XOR close).

So then people say, I will get close and then get the IP address. Well this is not simple, close to where? You cannot connect or ask for connection info to a client, it cannot answer, it is not part of routing.  Where is it located geographically for you to snoop on? Can you become a close node vault? That’s going to take a huge amount of work now. It will mean getting a vault ID close to the node. You do not choose your vault ID the network does and this is random across the address range. Even then you are not part of routing either until you have enough rank. This is not an easy feat at all, very likely millions of computers running correctly for a while with a random ID  and to potentially get close to a single node, which may change its ID anyway. A very hard task indeed. At the end of such an attack, who are you monitoring, you will have zero clue as any public name info is further encrypted inside the messages encrypted by the anonymous ID. We could get into reams and reams of what ifs here, I intentionally don’t as it makes the post too long and boring for most.  The system docs go into great detail and are continually updated for those answers.  TL;DR a targeted attack like this is not easy. Non targeted attacks are covered later in the consensus chain mechanism, this is a good precursor to that.

We could go further and create a random client ID every time, or even easier create a data-getter/putter id every time. I do not think it necessary, but very simple if required.  So you cannot see an IP address or tie a data ID to a person, so already it feels a bit secure. As every single message is encrypted, who cares if you compromise the routers etc. We don’t.

Vaults are persistent on the network and connect to more than just their close nodes, but each node decides to respond to a request for connection or not. It will only connect if the request looks good, i.e. improves the routing table. To put this in perspective after a node starts the first such accepted connection will be 1 in 1 million, after that its circa 1 in several million and increases quickly. What this means is that to improve a nodes routing table you will need a huge amount (100’s of millions) of connection requests and these have to be from valid well ranked nodes.

I hope this small post helps people put the network into perspective. The fact the IP address does not traverse the network is important. This is all down to the fact everything lives on a decentralised addressable network where all communications and exchanges of messages are encrypted by mechanism that ensure in transit encryption and end to end identification of every network hop.  If people ‘get’ that part then it makes this journey so much simpler, but also shows this path has not been trodden, so there is a huge amount to consider. A simplification is to say, this is a fully encrypted system that includes the data within it, as data is included there is no leak out to any third parties at any point (no server or machines that know or choose what they store).

So there you go another 6am post, I am off for a sleep now.

[edit fixed typo i2c/i2p too much arduino recently :-) Thanks  @anonymouscoin (Kristov Atlas)

 

 

Posted in complex systems, MaidSafe

Proof Of Storage (maidsafe part II), follow up video

This is a mini post to show how MaidSafe uses hashes to identify data after processing it in a way that’s invisible to end users. I hope it is at least a little bit useful. (note the use of encryption to obfuscate and later I may get into a debate about why we consider it only for obfuscation and creation of non repeating data). This is relevant to the previous post, where we describe hashes and their use for proof of retrievability and corruption free data.

How to make data safe?

Ok, to make this a shorter entry, I have embedded a video Shona in the office did for us to explain self encryption. It confuses many people, but it quite straightforward.

It should be noted, this process is a very small part of the “securing data” story. This will most likely secure data beyond any known decryption ability known today, but, and a big but!

 This is logical security, what I mean is where do you store the actual data? On Dropbox, Google, some other provider, on your hard drive etc. Well no!

None of these give what you want, physical security! Physical security (to me) means that nobody can know where your chunks of data are, so this means that this data has to be scattered to the 4 corners of the planet, on many machines, owned by many people, non of whom know what they hold or can disable or corrupt it in a way that affects the network copies of the data. It also means nobody can ban access, monitor or affect your relationship with your data in any form. This data is yours, it belongs to nobody else and there is no logical reason that anyone gets between you and your data, none whatsoever. This fundamental issue lies at the heart of MaidSafe and it’s design. It is not something people ask for, most likely because they do not realise it is possible.  It is possible, it works and its coming to everyone.

Why is it needed?

Well now we can start seeing the need for not only logically secure data, but also physical security. This about this for a minute, when in history of computing could we have private, physically secured data, accessible across the world? A mechanism where you can log in or access privately and without any third party get your private data securely. It is amazing, but never is the answer. You could argue, use truecrypt, backup tapes, copy to off-line and on-line backups etc.. All of which are hard, leak information about your actions at the very least and most importantly depend on great technical knowledge.

This is where you need an autonomous network geographically distributed and on peers that are monitored, not by humans, but by the network. This allows people to physically damage chunks, but the network spots that very quickly and creates a new copy and at the same time de-ranks your machine. Nothing to gain, except loss of safecoin earning now.  So vandals can happily vandalise, they can use their vandalism energy on that task, that’s fine.

Without an autonomous network this system of protection is only 50% of the job, only logical encryption, worse than that the upload and downloads are monitored in on-line cloud providers, so leak of security is certain. On disks and tapes, then a quick search will find info, if it ever was really secure to begin with.

The point it we need to secure data physically and logically, logical security on its own is at best a hassle.

In later parts of this series, when I get some time. I will explain again self authentication, the vital third part of the network after self encryption and an autonomous secure network. This is where your data maps are made available to you in a pretty fascinating structure. Next week though, I will continue the series on the network itself, that is a very large body of work with a lot of innovation (I believe).

Further details on self encryption can be found here

http://maidsafe.net/SystemDocs/system_components/self_encryption.html

and of course the code is all on github

https://github.com/maidsafe/MaidSafe-Encrypt

 

Posted in complex systems, MaidSafe

Proof Of Storage (maidsafe part II)

Sorry folks, another too fast an furious post, forgive mistakes and I will again update with any comments and improvements.

Proof of Storage != Proof Of Resource 

Proof of storage is an important part of proof of resource, but only a part of it. I will explain the whole proof of resource as this series continues. This part of the MaidSafe network should in fact be rather simple and make people wonder, why did we not see this before? So lets get to the problem quickly. 

“I think you hold some data, but I need to be sure you have it” (are you telling lies?). 

There are many grand schemes about for trying to achieve this very thing. It is considered a very hard problem and it is, well until you can see a solution then its a nothing problem.First we need to understand what a hash is, it sounds hard, but it is pretty straightforward. 

What is a hash?

In simple terms a hash is a fingerprint of some information. It is (like a human fingerprint) a certain size, or fixed length. so a SHA256 hash is 256bits long, a SHA512 hash is 512 bits long. Thats all that fixed length means. No matter how large or small the data we want the hash of is, the answer is always 256 or 512 in the above example. We use 512 bit hashes, so lets say no matter what we have, the answer is a 512 bit long fingerprint. That’s it really. Very simple.  N,B. As a human fingerprint does not contain the human, a digital fingerprint does not contain the data (weird maths arguments can start here for data less then hash length etc. but lets ignore that). 

So a hash is the digital version of a human fingerprint, but for data! It does not contain data, you cannot reverse a hash to get data, you cannot reverse a fingerprint to get a human. They are always the same length for any given hash type. 

An irreversible data fingerprint.

What is a secure hash?

This is a statement that throws people, they think, oh secure, it encrypts my data. It does not (read previous paragraph).  All that the security means is:

The more secure the algorithm, the less chance two pieces of data will share the same fingerprint (hash). [This is called a hash collision] 

It also means (and we use this) that for given random inputs the hash result will be evenly distributed across the whole address range. This means even a tiny change to data, will produce a wildly different hash result

“I think you hold some data, but I need to be sure you have it”

Right OK, back to the question.

Now we see, hash is a pretty simple to understand thing (its actually pretty simple code, which shows its beauty). In MaidSafe we already know that data is stored in an unusual manner (again simplicity). Each data element uses the hash (fingerprint) of its contents as the name. This means 

When you construct that piece of data in code, you hash the content and check the name is the same as that result.

This helps a little, but is not proof you have the data (we can prove that many ways, the simplest is to ask you for it :-) ). What MaidSafe does for many reasons is NOT to ask for the data, but instead we want you to prove you have it, more than that we want to prove you have the data and not a virus (a good reason not to ask for it). To achieve this we use a very simple process, called Integrity Check (code is here).

Integrity Check

feh_008831_000001_UnversionedData

Do not panic, this is just a reminder that there are nodes on the network (Data Managers) who knows some nodes that should hold data (PmidNodes) and these PmidNodes are managed by the nodes closest to them. You can sum this all up, by just stating, the network knows what data you should be holding

This proof in MaidSafe uses a mechanism similar to a zero knowledge proof. In this case the check should not require to know the content of any data to be checked, but must know the data is in fact held and held in a manner that is accurate. This means zero corruption or viruses…etc… can have affected the data. This is achieved with the following steps:

  1. A checking group (Data Managers) creates a random string
  2. This random string is sent, encrypted to all holders of the data
  3. The data holder takes this string and appends to the original data and hashes the result
  4. The result is collected at the checking group and compared
  5. If any node returns a different result then it is believed compromised and de-ranked

This mechanism triggers on Get requests and during account transfers etc. It is non deterministic and randomised by use by users. It is considered to be secure and uses zero knowledge, not to conceal content (as anyone can ask for any data), but to ensure any data with a contamination is not required to be transferred.

It is really this simple. 

An observation

There are two really important points to make here, and I cannot emphasise them enough

1: Hard things are only hard till you solve them, difficult is not always smart maths or amazing crypto babble, difficult means we cannot imagine the solution, we are trained not to and defeating all those years of conditioning is very hard. 

2: If a solution is complex then its very likely to not be an optimal solution in itself. If there are lots of conditions and checks in the solution, then I tend to not like it, at best it is a sticking plaster solution and will not last. The optimum solution will be simple in its own right, anyone should be able to understand it without too much effort (or its not optimal enough, arguably). 

This is actually what makes MaidSafe hard and complex looking.  Most of the solutions are very simple and start by requiring us to go down the sticking plaster approach, this is the part where folks say, thats amazing and complex, you must be smart, actually the opposite is likely the case. Then we remove that for a more simple algorithm that improves the solution, this is very hard, but the result should look as though it was easy all along (quandary? you bet).

This boils down to a situation where, you can explain a little part (as I am supposed to be doing here) and people say, oh yea thats rubbish cause look at this other problem, bet you have not thought of that (bet highly we have, not 100% but bet highly).  So explain the little parts and making them simple is OK, the complexity in MaidSafe is that it is a network of many of these little parts and they compliment each other. So to understand it you need to understand the little parts in isolation, then stitch them together. This is a difficult thing for people who want the answer now!

Further considerations 

This short text has touched on the issues that require solving and hopefully presented a simple, easy to understand solution to proof of storage. It is the part folk talk and pontificate about with elaborate schemes to solve. This shows the solution is in fact simple. It masks the actual issue though (as do all the expert opinions on this). What about mutating data, i.e. directories change when a file changes, thats not held as a hash, safecoin and transactions need a key that cannot be a hash of the data, so … there is a taster of the actual issue, when you solve this one. This is all for later in the series though, I hope you will see it’s just as simple, but all this is required info to get to that position.

Part 1 of this series is here.

Posted in complex systems, MaidSafe, Uncategorized

Shortest distance between two points is not always a straight line

This post is probably very brief and disingenuous for any mathematician out there, it is aimed at as many people as possible and I hope it comes across in an easy to understand way. We will see I suppose :-)

I find myself explaining MaidSafe many times to many people and try to do so in  a language and counting system they will understand. I ‘get it’ that people want this, so do I, but to understand MaidSafe it’s really not possible to think in our comfortable world of linear number lines and counting. The term used to describe what I am talking about is ‘non euclidean maths’. I dislike that a lot as its like calculus or similar, a term to scare people off. This is not necessary as it is way simpler than a confusing term.  It is also way more simple if you immerse yourself in this type of maths, many people will not and never understand you, but it leads to many hours of quiet solitude and relaxation as a whole new world of possibilities opens up in front of you. I find that you can lose yourself in such counting systems, a bit like the Gauss clock counting base number systems, it’s very unusual and interesting. Not at all clever, but very different.

Down to business 

MaidSafe is based on an XOR network at is root. This is not really the crux of what makes it different, but it seems to be what people focus on (“what if my computer goes off?”, “I can Sybil attack this easily!” etc.). I figured it was time to try and explain this as understanding issues like mutating data such as safecoin etc. are impossible without understanding this way of thinking.

So here is how we think.

0--1--2--3--4--5--6--7 (real number line)

And here is XOR

13706-20c

So what?

Immediately we will look and run our eyes along the bottom and think it’s our comfortable world again. It is far from it, we will see why soon.

XOR opens up huge possibilities and many unusual attributes (and I hope I do not sound like I am using a weird maths unintelligible language that makes people think this is clever, it is very easy, honestly).  Many smarty pants people will tell you this is all irrelevant, but just think for a minute and realise you have moved on a level, understood a little more than the real number system. So lets looks at some different properties of this system:

1: Distance is not based on the normal number system, i.e. the distance from 4->3 is 7, but the distance from 2->3 is 1 (get a scientific calculator with xor in it and try).

2: Each node has a unique distance from every other number bar 1. (i.e. the distance from you to another is symmetric, but you share no distance from you to any other specific node with that partner, plus every other node is such a partner).

3: There is no straight line distance from any node to any other node (like points on a sphere, but no calculus, don’t worry)

4: Every number on this line sees a different network from any other number, so to see the network from a numbers point of view you need to become that number (after all its distances are unique to it, remember!).

These additional comments can also be made, feel free to ignore these though as they are a bit more confusing (sorry, but some may find them useful)

a: Distance from any node to any other is almost the same as the height you need to climb the tree, you can verify by actually XORing the numbers if you wish (I do to check). Be aware though the last bit of the address is important, you are closer to the address ending in your last bit than the opposite of the last bit. I check the side of the tree in the last position, so if you are on the left you are closer to the node also on the left (if you look closely at the diagram you will see this) than the one on the right.

b: The other (maths) way to measure distance is to evaluate the most common leading bits of any number. (again you need to consider the last bit as if your address ends in a 0 you are closer to it than the address that ends in a 1)

 A bit more interesting

Another thing we do as humans is revert immediately to straight line thinking, but additionally we also assume a fully populated network or tree as we picture above. If you stay in dream world a little longer and think what a sparse tree would look like then its eye opening.

1 - - - - 6 - - -- -- 9-10-11-12-13------21----

So in the above tree we can see that its sparse, i.e. full of holes where nodes could be. This makes it more clear that if we know each node has a unique distance (to it) from every other node then it gets interesting. To make this clear, from any node all distances to every other node are unique, but the distance to any node is symmetrical, i.e. dist(A->B) == dist(B->A) but this distance will be unique to A and B, neither of them will share that distance with any other node on the network. There is a strange (sometimes called triangle property)  property as well so [N:B ⊕ means XOR]  ((AB)C)(A(BC)) which basically means that if you XOR something with something you will get an answer. If you XOR the answer with any of the two inputs you get the other input as a result.

so  4⊕5 == 1 so 1 is the answer. If we then do 5⊕1 we get 4, or if we do 4⊕1 we would get 5. So this is a handy thing we use to obfuscate information at times.  If the 5 is your info and 4 is a completely random number only you know and use once then you have a very powerful secrecy mechanism (if the 4 is the same length or larger than the 5 you have ‘perfect secrecy‘). On that note readers should take a look at self encryption for fun. Anyway I digress. 

So XOR has some pretty strange peculiarities when you first come across it. We have listed only a few here, to whet your appetite to know more, but its this tricky and somewhat hidden set of attributes that has people looking at an XOR network and saying, “oh that’s obvious” etc. I thought so too and especially when I first came across Kademlia. That paper is very tricky, I read it and thought, simple enough let’s do this, then it became clear there was much more to this than met the eye. After digging it was also clear there were deficiencies in kadmelia networks that would be very hard to overcome.  So on we go with our excursion into non linear counting methods and some really cool side effects.  There is more info on some of these kademlia deficiencies (well from our perspective, for public only data its way good enough)  here.

OK What about closeness

This concept takes the XOR thinking and expands on that a bit more, again with surprising side effects that do not seem to be noticed easily. In MaidSafe the notion of closeness is used to evaluate a nodes capability to make a decision on an action depending on the action type, data ID and some other parts (too much for this post, you can read about it here though). In other words closeness is important. So if we can assume a secured mechanism in place for address creation and maintenance (there is and its called PKI) then we can look further, normally you need to argue for days to get to this point if you are discussing MaidSafe so this is great progress.

If we imagine a close group is size 4 for now then look at this network

1 2 - 4 - - - 8 9 10 11

We can calculate the close groups, these are

for node 1 – 2, 4 & 8

for node 8 – 9,10,11

Hold on!

If we ask node 1 then node 8 is close to it, but if we ask node 8 then 1 is nowhere to be seen! We know that dist(1->8) == 9 and conversely the dist(8->1) is 9.

Wait! now look at what happens when a node appears.

1 2 - 4 - - 7 8 9 10 11
            ^

Now look again at the groups (and remember to stay away from linear thinking)

node 1 -2.4,7

node 8 -9,10,11 (not 7?)

now look at node 7 (4,2,1) its miles away from 8! (15 away actually), stay away from the dark side of linear counting at all times with this stuff, there be dragons :-)

[Edit from our own Mark Hughes, a clean example (thanks Mark @markhughes)

 Here are binary representations of the digits 7 to 11:
     7        8       9      10     11
0111 1000 1001 1010 1011

To get the distance between 8 and each of the others, we XOR the number with 1000 (binary 8), remember we have to become 8 to see the distances form it.

So…
0111 (binary 7) XORed with 1000 (binary 8) is 1111 (binary 15, quite a large distance).
1011 (binary 11) XORed with 1000 is 0011 (binary 3, which is smaller than 15, and so eleven is closer to eight in XOR space than seven.

]

So there is a simple example and description of closeness, it is uni-directional or asymmetric and distance is bi-directional or symmetric.

A network of networks

First thing in MaidSafe to realise is that the address space is huge (2^512) in fact its way larger than all the atoms in the visible universe.  The distances between nodes will be numbers so large they have no names. So these distances are vast, it makes little difference though. This ensures we do not run out of addresses and data locations to store (although in MaidSafe these are typed, i.e. we can have a node at position 001 and a data element of type 1 say at position 001, so not happy with this huge number we have gone a bit mad (or so it seems)) This basically means that we have multiple types in the same space, so message type1 has a 2^512 network, message type 2 has another 2^512 network etc. This can be thought of as each message / data type has its own network.

At this point it is easy to spot that every node sees a different distance from other nodes. We have kinda proved that above. Now this starts to hurt a wee bit, its like everyone sees their own rainbow, no two people ever see that same one (think, think). So from any node the network looks different. We also know that to measure distance from a node to another (especially closeness) we need to become that node. It’s not possible to do otherwise, in MaidSafe this means getting as many nodes around that point as we can (16) and then sort that list of nodes from the address we are interested in. Then we can tell which nodes are the closest. So we do not become that node per se, but we do look at the network from that nodes perspective, told you it was gonna hurt.

Right, so now we see we have a network of networks. Some still at this point will say “so what”, ignore them and continue.

 The large interconnected Venn diagram appears

Venn_diagram_ABC_BW_Explanation

In the above diagram imagine A B and C are nodes. They are looking after an address in common. C shares a small part of the responsibility for this common data, as does A and B. If we imagine another bit of data A is responsible for you can see from the diagram that C or B may not even see that data. This shows that you cannot look at a group and work out what they are responsible for.  You must become each data element and look at what group is responsible for that element. Again the take away is close groups are different close groups depending on the data element in consideration. The network of networks is not restricted to nodes, bit to every single data element within that network. It is very unlikely two data elements will share the same close group. This makes attacks in groups of data more difficult. If you also consider the network alters continually with nodes going on and off you will see why this is not a problem for MaidSafe, its a huge security feature. The target is moving now, so a piece of data has different protectors with each churn event.  So it is not how we handle machines going on an off, we welcome it as it provides a layer of security that is not obvious. We will make great use of this as this series moves on to show where the security of consensus chains becomes inordinately capable in such an environment.

I am not good at maths, or in a traditional sense anyway. I see shapes and sort of dream myself into the problem, in this case the network and poke around it in my head, even while sleeping. It’s hard to do (for me anyway) and you almost get into a trance to really focus your mind on this.  So I think Kademlia is like a semi organised collection of random nodes, this is indicated by the fact emule used to have 30% duplicate addresses and you cannot be sure you are close to data or addresses, you just search until you find stuff. It’s not good enough, we need more accuracy.

So we take our network of networks, our large Venn diagram and make a big change. We connect all nodes to the 16 closest nodes they know off, add in some more for uni directional interest and make the rest random across the address range (should be equally spaced in a perfect system, but it will not be). Now these nodes have great knowledge of each other and can spot when one changes, very quickly. The random part actually takes the shape of a kademlia routing table and each node monitors addresses to try to improve this table. This allows logarithmic search or in layman’s terms massively reduces the number of hops to a target.

When a node goes off or comes on the network it will affect its circle of closeness to it. It will also appear at a point in the network that means it should be doing something and looking after some small state (state is distributed evenly across all nodes). As no node has the same distances from any other node, then the state each node holds will be different from its peers. It will share some info, but its list will be unique. Each item will be on at least the number we have chosen as close groups (4 in our case).

So a node going off will affect the close part to it. The whole network is reconfigured without any other node far away knowing it is until they look. No node can look everywhere so the equilibrium of control establishes extremely fast. No need to transmit messages all over the place to let others know what’s happening, they do not need to know and can do nothing about it anyway.

I hope this very brief foray will be helpful, it’s a bit of a head dump, so may contain some info that could be better, it will be too simple for some. My intent is that anyone can read this and begin to understand the journey I hope to take this blog series down over the next few weeks. This will explain consensus chains and mechanisms and then on to how to think about decentralised control and applications in a way that I cannot see has been done before. I do hope it helps. I hope at least people can look at this fundamental part of project SAFE and realise there is nothing out there like it as far as I can see, it’s certainly wildly different from Tor/i2c/Freenet/Bittorrent etc., or at least I hope that’s becoming more obvious now. This will be a long journey, but at the end I hope readers will be pleased at the differences this approach offers.

Tagged with: , ,
Posted in complex systems, MaidSafe

What is a vision?

This is more of a quick note than a full post. I hope it helps others who are trying to change this pale blue dot for the better! [watch this, it explains a lot and is very enlightening]

Recently as MaidSafe has come out of the shadows it has been very interesting to watch from the sidelines. People seem to split into many camps

  • Don’t care
  • Not reading this
  • Not listening
  • Rubbish
  • Scammers
  • Canny happen
  • These folk are nuts
  • This is amazing
  • Wow cannot wait
  • Freedom at last

And so on. It’s an interesting thing to watch.

What vision

MaidSafe have had the same vision since day 1. Privacy Security and Freedom for all the worlds people. This is what is interesting. Practically nobody disputes this vision, many actually actively support it and see it as significant. Lets assume for the rest of this article this is true of most people. I cannot see anyone not wanting these foundations of a society anyway.

I believe I have noted three basic groups of people in the journey so far and more in number in the last few weeks. the ratio seems solid though. I shall try to describe what I have seen, not that I am trying to say that this is always the case or even a repeatable pattern. I do find it intriguing though. So I am noting it down.

The care later (maybe) group

Now lets look back at the list. It is actually very representative of what I have witnessed. So about 1/3 care about the vision, but cannot be bothered to hear about it or invest time looking at how it may be achieved. These people have different immediate goals and do not want bothered with more work to do.

I think these people have many reasons for this behaviour:

  • Too busy
  • Too tired
  • Too stressed
  • have very urgent issues to deal with
  • Do not see the problem

Who can blame these people for apparently not caring in your vision. Sounds like they have their own issues to work on and these issues at this time are more important.

You have a desire to change the world, give up everything and they do not care! Well they do care, but it’s not their time, that time is later on. These people are not your enemy, they are not opposed to your dream, they frankly do not care, but in the nicest way, they really do not care if you win or lose. Not yet anyway!

The no way team

The next 1/3 immediately shun the idea and to do that effectively. They simply state it has some fundamental flaw. This 1/3 are interesting. Some invest zero time and just throw out statements, often very hurtful and cutting. Some read headlines and presume total knowledge and some simply refuse to believe it. These people will inflict any pain they can to stop a change dead in its tracks. It becomes their duty and they will try very hard to do that.

These people also fall into various categories I think.

Lied to before

This group is understandably stand offish. Like a police officer who deals with crooks continually, they see every person as a threat. Every opportunity as a con. This group is actually OK and a few messages back and forth and they start to wonder, what if? As long as you are honest, true and really believe in your vision, these people can become supporters. It is not an easy thing to speak with these people, sort of like cuddling a porcupine really, but many times it will be worth it. They actually want this to be true, but have been bitten, possibly more than once.

They need to be persuaded you are honest and capable, that is all really and in all honesty it is not a big ask.

Protectors of the status quo

This is the group that will see a plane fly overhead and still deny flight is possible. This is a tough group. Many here may even revert to statements like, well even if the rest is true you have a silly name, how can we take that seriously. Occasionally they just shout scam or fraud and then put their fingers in their ears. This is the horse you take to water and it refuses to drink, it would prefer to die than admit to needing a drink.

All is not lost, I think this group just need to see they will not drown drinking from a pool. They need reassurance this will not be sore or somehow damage them. You need to guide them off the path, they will not go far so you need to show them quickly there are no dragons living in the bushes.

These people will take several hours to persuade that they should look further. It is a porcupine thing though, you will continually be jagged as they tense up and stick you repeatedly. So you need your Kevlar skin suite to deal with them.

Cannot leave the ego at the door

Well these people, I believe have given up on education, they have stopped learning and will take way too much time to persuade. They will not speak to you anyway, instead shout at you for their supporters to hear and applaud their genius (an immediate giveaway). These people will do everything to kill your project and idea off. In fact these people will show immense strength and tenacity to make you just go away and die. Yes they would like to see you disappear and never come back, no matter the price.

There is nothing you can do, these people will suck any energy from you and will give everything they have to fight change. The only way to win here is to be polite, fix obvious errors in any statement and move on, do not look back and never read the reply to your polite statements. It will honestly just suffocate you.

The kindred spirits

The last 1/3 are interesting though, many will not really know much detail, but want the change you have noticed so badly they jump right in. However, then something magical happens, they ask and they ask a lot. The more they find out the deeper they are involved and more convinced this is worthwhile. Many of these people, will just know this is special and become great friends. It is a humbling experience.

This is where your strength is actually challenged, not fighting the second group, but supporting this third group. You now not only have shared a vision, you have support and this support becomes the bedrock of your vision. Break this and it is all over.  This group though will amaze! After very polite hello’s and introductions this group will defend, not you, but the vision. They came here for this vision, not for you and that is great. The vision will be more important than you.

This third group now do something rather amazing, after a small amount of time, you are them and they are you. A transformation or metamorphosis occurs that changes everyone involved. The vision is still solid and arguably even more cemented in place. The legions of people who are willing it to happen, by moral support, encouragement, new ideas, questions and occasionally helping with the manufacturing process is now doing something you could not.  They are growing in all directions, new sites appear, people across the Internet start supporting and defending (dealing with the 2nd group) this vision. Everyone starts working out ways to me more involved  and turn a vision into a reality.

Who owns a vision?

This is a great question, it is like asking who owns a particular word, or who owned the wheel or even who owns your telephone or tv. The people who get involved believe in a vision, there may be people who pushed this vision and worked to make it happen, but they cannot own it. It is such a strange thing to ask when you think of it. So why then during the 90’s and on did companies all rush to come up with a vision statement, put it in a glass box and shove it on a wall? More importantly why did these companies not know what they were in business to achieve, or was their goal purely profit?

Herein lies the big difference. If you look at some of the successful large companies you will notice something particular about them. No matter what happens after they IPO as that’s another matter, they are then obliged to increase profit at all costs. Before that though, these large successful companies had a clear vision, to connect everyone, to index all the worlds data, to make computing available to all in a stunning package etc. The successful companies seemed to have a clear vision and the community wanted it to happen.

What is a vision?

This is pretty clear I think. A vision is noticing that we can advance a little as a society. People can imagine that if things were to be changed a little then we all would live better. The vision that is inclusive and fair gains momentum fastest. This is evolution, so a vision is merely a step in an evolutionary chain. Just as species will physically evolve to better work in its surroundings. A vision is important to allow us to evolve and make better use of our surroundings. That is all.

Why the 2nd group then?

This group is actually important in many ways. Any change that is dramatic, is likely not fully thought out and tested. This second group forces the vision to be tested and their opposition to it actually makes it stronger and more likely to succeed, if it is indeed an added value to society. Whether these people can see that or not, they are vital to big change, the tenacity and grim determination to kill something off, often makes it stronger. This is like a comprehensive set of tests on the network in our case.

The wild accusations etc. are just fluff and easily seen by many as just noise. The deeper probing of this community can be a very effective tool, the key here is to not take it personally, even when it’s delivered in an extremely personal way. Inwardly thank these people, but never to their face, that would drive them mental.

So this second group are vicious, dangerous and absolutely required. An interesting point.

Nature allows evolution, but does so very slowly. Many evolutionary steps in organisms is once every 20,000 generations, so not fast. The reason is that the surroundings need to also adapt to these change in a species. The surrounding species need to also evolve to become more efficient. Otherwise imbalance happens. This second group merely act as the brake and slow down change in many ways. This is perhaps why nature made us in this way, so we can all evolve together. Look at schools, they still teach primarily Newtonian physics and almost ignore quantum physics, even after 40 years. Change takes time, not matter how obvious it is.

What is the lesson?

The lesson is not one for the masses, it is one for you, the reader. Change is something we have inbuilt in our species, it drives many and we all take part in many ways. You need to embrace all the parties involved and realise that nobody owns a vision, nobody can. The more you share your vision and the quicker it becomes a vision of a movement who may not (if you are lucky) even know the name of the originator. If you have such a vision, then believe in that vision above all else, above profit, above ego and most of all above your own self. If anyone can help you achieve this then take that help, even if it is a competitor, after all did you not go after a vision!

An important point is that you can only hope to create the seed of change, it will take many iterations to get right. Look at anything, the first phone, bitcoin or the first car etc. many people will try to stop you by pointing out something that is not required now (like a fancy braking system for a Ford model T). Do not try to put ABS in your Model T, just make it drive first, remember evolution is a continual thing and you do not need to aim for perfection, although detractors will use lack of perfection as a reason your vision is flawed.

Who cares who cured smallpox, polio or who created the first wheel, we just care these got cured or discovered. All of your effort is to help the vision come true, if you believe it is yours or is personal you will lose immediately. If you truly believe  in the vision and what it will mean for everyone and remove yourself from ‘ownership’ then that vision will take on a strength that is just not possible to stop. You become part of a team to make this happen, you may even be thrown out the team, who knows. If you see the vision come to life and fulfill the promise, should you really care who did it? (I don’t)

Tagged with:
Posted in Personal Opinion, strategy

Member of The Internet Defense League

Follow Metaquestions on WordPress.com

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 1,080 other followers

BTC address
13YSCv8SLrBw27AdyRQY8adfsGa56viQcJ
Follow

Get every new post delivered to your Inbox.

Join 1,080 other followers