Meshcentral.com
Meshcentral
High Level Architecture
Version 0.0.3
Monday, June 25, 2012
Ylian Saint-Hilaire
© 2011 Intel Corporation. All Rights Reserved.
High Level Architecture
MeshCentral.com
Legal Notices and Disclaimers
Disclaimers
INTEL CORPORATION MAKES NO WARRANTY OF ANY KIND WITH REGARD TO THIS MATERIAL,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE. INTEL CORPORATION ASSUMES NO RESPONSIBILITY FOR ANY
ERRORS THAT MAY APPEAR IN THIS DOCUMENT. INTEL CORPORATION MAKES NO COMMITMENT
TO UPDATE NOR TO KEEP CURRENT THE INFORMATION CONTAINED IN THIS DOCUMENT.
THIS SPECIFICATION IS COPYRIGHTED BY AND SHALL REMAIN THE PROPERTY OF INTEL
CORPORATION. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE TO ANY
INTELLECTUAL PROPERTY RIGHTS IS GRANTED HEREIN.
INTEL DISCLAIMS ALL LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT OF ANY
PROPRIETARY RIGHTS, RELATING TO IMPLEMENTATION OF INFORMATION IN THIS
SPECIFICATION. INTEL DOES NOT WARRANT OR REPRESENT THAT SUCH IMPLEMENTATIONS WILL
NOT INFRINGE SUCH RIGHTS.
NO PART OF THIS DOCUMENT MAY BE COPIED OR REPRODUCED IN ANY FORM OR BY ANY MEANS
WITHOUT PRIOR WRITTEN CONSENT OF INTEL CORPORATION.
INTEL CORPORATION RETAINS THE RIGHT TO MAKE CHANGES TO THESE SPECIFICATIONS AT ANY
TIME, WITHOUT NOTICE.
Legal Notices
Intel software products are copyrighted by and shall remain the property of Intel Corporation. Use,
duplication or disclosure is subject to restrictions stated in Intel's Software License Agreement, or
in the case of software delivered to the government, in accordance with the software license
agreement as defined in FAR 52.227-7013.
The Intel logo is a registered trademark of Intel Corporation.
Other brands and names are the property of their respective owners.
i
High Level Architecture
MeshCentral.com
Table of Contents
Legal Notices and Disclaimers .......................................................................................................... i
Disclaimers....................................................................................................................................... i
Legal Notices ................................................................................................................................... i
1.
Abstract ................................................................................................................................... 1
2.
Introduction ............................................................................................................................. 1
3.
Mesh Agent ............................................................................................................................. 3
3.1
Management port (TCP 16991) ....................................................................................... 4
3.2
Peer-to-peer ports (TCP & UDP 16990) .......................................................................... 4
3.3
Mesh Server Connection ................................................................................................. 5
4.
Swarm Server ......................................................................................................................... 6
5.
AJAX Server ........................................................................................................................... 7
6.
Social Server ........................................................................................................................... 9
7.
Conclusion .............................................................................................................................. 9
Annex 1: Routing Cookie Encryption ............................................................................................. 11
ii
High Level Architecture
MeshCentral.com
Document Changes
Sept 22. 2011 – Version 0.0.1
Initial version.
Oct 17. 2011 – Version 0.0.2
Added WebSocket port 8085 on the AJAX server.
June 12. 2012 – 0.0.3
Grammatical corrections.
iii
High Level Architecture
MeshCentral.com
1. Abstract
This document is intended to be a high level overview of how Meshcentral works: From the mesh
agents to the basic functions of the online service.
This document is intended for anyone that wants to understand how this technology works,
review its design or for developers who will be using mesh as a way to write more complicated
applications and route traffic across networks. Basic knowledge of what Meshcentral is, its
features and usages must be understood by the reader before taking on this document.
2. Introduction
The mesh project allows users the ability to remotely manage and control PC’s and devices from
any network using a single web site with no changes to the local network infrastructure. Just
install the mesh agent and the computer will show up on the Meshcentral web site and is
immediately manageable.
To make this solution work, let’s start by splitting the design into its components and see the high
level network data flow between them. First, we have the mesh agent. The agent runs on each
managed computer and has data flows to other agents, the mesh central server, and possibly
applications that make use of its services. It also stores local data into a local database. As a
result, the high level diagram of the mesh agent looks something like this:
1
High Level Architecture
MeshCentral.com
Note that the agent can logically be split into two components, the peer-to-peer portion and the
management portion. In this diagram, the arrows point in the direction of connection initiation, but
traffic in all cases is two way. Both portions of the agent are tightly bound, but provide different
features. We will come back to the agent in a later section.
The server can also be split into different components and again, we have network protocols
used for communication between these components. The light level design of the server looks
something like this:
2
High Level Architecture
MeshCentral.com
Here, all of the server’s components may run on the same computer or different computers. Since
network protocols and not internal API’s link the components. How the server is deployed
depends on scalability tradeoffs. Most of the data between server components is transferred thru
the database or Microsoft Message Queuing, with the exception of the AJAX server that
occasionally needs to route connection traffic directly.
In the following sections, we will look at both the mesh agent and meshcentral server in more
detail. The goal will be overall understanding of the mesh components and their interaction and
how security is maintained.
3. Mesh Agent
The mesh agent is truly the core of the entire solution, its unique design allows for many
interesting usages and benefits. The mesh agent is entirely build with raw C code with the
exception of a few C++ modules that are specific to Windows. So far, the mesh agent has been
ported to Windows, Mac OS X, Linux, MeeGo, Android operating systems. When applicable, the
agent runs both 32 and 64 bits and on x86, ARM and MIPS processors.
The mesh agent links against two major libraries: OpenSSL and SQLite. Both are widely
supported on many platforms. OpenSSL offers all of the cryptographic methods used by the
mesh agent, and SQLite the disk storage primitives. Sometimes other libraries are also linked to
support specific OS specific features like remote desktop.
In general, we opt to statically link all required libraries to the mesh agent. This makes the agent
significantly larger, in fact over 10 times larger (200k to 2 megabytes), but we gain in having each
agent able to run on a wider array of computers and possibly some security gains.
When compiled, each agent is tagged with an architecture ID, a version number and signed using
a developer certificate. All this information is used to perform remote updates of the agent (both
peer-to-peer and server-to-node).
3
High Level Architecture
MeshCentral.com
If we look at the network, the Mesh agent opens up 4 sockets for incoming local traffic. These are
port TCP 16991, TCP 16990, UDP 16990 and UDP 1900. Upon installation of the mesh agent,
the proper firewall rules will be added to the Microsoft Windows firewall automatically. Let’s look
at what each of these inbound ports is used for:
3.1 Management port (TCP 16991)
The management port is simple to understand, it’s a port that accepts mutually authenticated TLS
connections and once authenticated using a client certificate, allows the caller to perform
administrative and management tasks on this computer. The features offered by the
manageability agent include: remote power control, remote file management, remote desktop and
remote command prompt. Any computer on the network can attempt to connect to this port. If
authenticated, it can perform the commands. Often times, we use this port along with mesh traffic
relay over the internet, but it can also be used within the local network for quick management.
Once the TLS connection is established, the management port uses binary commands to
operate. Commands are always 4 byte headers optionally followed by more data.
3.2 Peer-to-peer ports (TCP & UDP 16990)
Mesh agents will initially discovery each other using a local multicast on UDP port 16990. So the
mesh agent not only opens this port, it also joins it to a multicast channel. Agents will multicast
periodically until they find another agent, at which point the multicast stops and all future
discovery is done using peer-to-peer unicast instead.
The mesh agents have a unilateral, one way relation with other peers. The peers that are friends
with a node are not identical to the friends on this node. Also, the mesh agent has a maximum 20
peers it can communicate with. The process of selecting these peers is outside the scope of this
document, but the result is a “balanced graph” with each node having 20 friends that are correctly
selected.
Once a candidate peer is selected for connection, the mesh agent starts by attempting a
mutually-authenticated HTTPS request to the remote peer. If successful, the remote peer will
return a session key that will be used for all future encrypted UDP communication between these
two nodes.
4
High Level Architecture
MeshCentral.com
As a result, at a high level peer communicate to each other following this timeline:
When the HTTPS session is established, each node uses a TLS certificate that is derived from its
node certificate in such a way that each node can compute the other node’s node identifier. The
node identifier is the SHA256 hash of the node’s public key. In this design, the nodes never
transmit what nodeid they have, rather the nodeid is inferred using the certificate using in the
HTTPS session.
In order for Node B to maintain no state about Node A, the session key returned by Node B is
actually a hash of the Node A identifier with a node B secret session random. This way, Node B
does not have to keep any state yet, can decrypt all UDP messages sent by Node A.
UDP messages are encrypted using AES-CBC and authenticated using HMAC-SHA256.
The periodic UDP Sync serves to indicate that the peer is still present and serves to trade any
state that may have changed. All peer-to-peer state in this design is blocks that are signed using
RSA2048 certificates, so state that is moved from node to node can’t be changed while in transit.
3.3 Mesh Server Connection
Now that we have addressed the inbound management and peer to peer ports, let’s turn our
attention to the connection made by mesh agents to the central server. The connection is always
initiated from the mesh agent to the server, this allows for router, firewall and proxy traversal.
The server connection is authenticated and secured using a mutually authenticated TLS
connection. Just like when peer-to-peer connections occur, each side of the connection infers the
others parties identifier by performing a hash of the public key of the certificate. In the case of a
server connection, the agent checks the mesh server’s certificate against its policy and the server
infers the nodes identifier.
5
High Level Architecture
MeshCentral.com
The mesh agent connects to a component called the “swarm server”. The swarm server
communicates to all mesh agents using binary commands that are secured using TLS. The mesh
agent starts by sending the server its signed information block, then, other peer information is
sent. The connection is kept open and often idle except for occasional 4 byte ping commands.
Using this connection, the administrator can initiate operations such as power commands or TCP
relay from the server to the local private network. All operations are authenticated and secured
within the TLS connection. The mesh agent will also verify that the server is authorized to perform
a remote operation by checking its mesh policy.
4. Swarm Server
The swarm server is a C# application that runs on one or more server and is built to speak the
binary language that mesh agents and mesh consoles (tools) use to do complex operations.
Much like ASP.net speaks HTTP, the swarm server has its own port and speaks its own protocol.
We opted to use a binary protocol to facilitate transferring bulk data for socket relay and to keep
the protocol very efficient. Here is a more complete overview of the Swarm Server.
When first starting up, the swarm service loads many settings from a text configuration file. If
many swarm servers are running, each instance must have its own configuration file. The file
indicates the certificates to use, ports to bind to, MSMQ settings and more. The logs generated
by the swarm service into log files are indented to help debug the software; they include all
exceptions and unexpected failures.
6
High Level Architecture
MeshCentral.com
The only exposed public port is port 8080, it’s used by various actors but the most frequent by far
is mesh agents who connects to the server and authenticate based on their mesh policy. Port
8080 is secured using Microsoft .NET TLS. TLS connections that use a TLS client certificate are
mesh agents, other TLS connections that don’t use a client certificate are mesh tools or TCP
relay requests.
Port 8088 is identical to port 8080 except that it is not authenticated and must only be exposed to
“localhost” or to the swarm server’s private network. When a connection arrived on this port, it’s
assumed to already be authenticated and coming from a trusted entity. Currently the AJAX server
makes use of this port.
Two other API’s are used to access the database (Microsoft SQL Server) and the message
queue (Microsoft Queuing Server). The queuing server is setup using parameters found in the
configuration file, there is one inbound queue and one or more outbound queue can be setup. In
general, commands sent to the queue must be received by all other components. The commands
content is binary.
5. AJAX Server
The AJAX server was built to provide web applications on Meshcentral live two way
communication. An interactive web page is loaded from IIS and the Javascript then connects
back to the AJAX server with long standing requests that wait until data is available.
When a web application first contacts the AJAX server, the server will keep state about this
browser along with a data queue. When the data queue fills up with information for the browser, it
is kept until Javascript makes the next request for data. If no data is in the queue, the browser
7
High Level Architecture
MeshCentral.com
session is kept waiting until data is put into the queue. This system allows Meshcentral to offer
usages such a remote desktop, remote command prompt, list chat and remote file management.
The AJAX port 8084 is TLS secured using the same certificate used by IIS. In fact, IIS is the
underlining HTTP session handler for the AJAX server. To setup the AJAX server, we must run
the following commands on the server:
// Setup the certificate
netsh http add sslcert ipport=0.0.0.0:8084 certhash=xxxxxxxxxxxxxxxxxx
appid={8bf83834-1594-4051-b4a7-5693561b257a} clientcertnegotiation=enable
// Setup both AJAX server mappings
netsh http add urlacl url="https://+:8084/" user="Local Service"
netsh http add urlacl url="https://+:443/ajax/" user="Local Service"
// View the resulting settings
netsh http show sslcert
netsh http show urlacl
These commands setup the HTTPS certificate and HTTP handlers. Note that the AJAX server
can be accessed on its own port 8084, or on port 443 with the suffix “/ajax”. Because some
network environments restrict communication to ports other than 443, most of the AJAX web
applications will use the 443 port as much as possible. When relaying a full web page, we need
control of the root path and so, port 8084 must be used.
Both IIS and AJAX server should make use of a proper web certificate that is signed by an
authorized certificate authority.
In order to access services provided by the AJAX server, the web application must has previously
obtained an authentication token. Two authentication systems exist:


This token is created and stored in the database once ASP.NET has authenticated the
user. ASP.net embeds the token in the web application’s page. This token is passed to
the AJAX server by the web browser.
An AES encrypted routing cookie is first obtained by the web application, the browser
then makes requests to the AJAX server using this cookie.
The second authentication system was added later and has the added benefit that the cookie can
be created (encrypted using AES) by a completely different server. As long as the other server
has the AES encrypted key & their clocks are in sync, it can generate routing cookies as needed.
Both authentication systems have an embedded timeout, generally 30 minutes.
The AJAX server also hosts a WebSocket port (8085) which is a new HTML5 technology for live
two-way data transfer over an HTTP channel. Port 8085 is also secured using TLS using the
same trusted certificate that secures port 443 and 8084. More information on WebSocket can be
found on Wikipedia at: http://en.wikipedia.org/wiki/WebSocket. Our WebSocket implementation
on port 8085 supports both “draft-ietf-hybi-thewebsocketprotocol-00” and “draft-ietf-hybithewebsocketprotocol-06”.
8
High Level Architecture
MeshCentral.com
6. Social Server
The social server is a separate C# application that facilitates the interaction between
meshcentral.com and social media sites. Currently, this application only supports Twitter. The
application listens to the “MeshCentral” Twitter feed for user commands, and posts events
obtained from the MSMQ software bus over to Twitter when needed. The social server uses
OAuth to gain permission on post on user’s Twitter feed, this is the industry norm.
This server has no exposed public ports. It mostly uses the database to check permissions, get
OAuth information and other information to complete text messages. If uses MSMQ to get actions
that may be interesting to broadcast and to send command that users have performed (A user
may power on or off a computer using a Twitter command).
7. Conclusion
The combination of the Mesh Agent, ASP.net application, AJAX server, Swarm server & Social
Server allows for a rich set of online web based interactive usages. These components are also
split into this architecture to make it easier to debug, and make the server more scalable. By
building on top of known robust database and message queuing components, we insure that this
solution may scale well going forward.
9
High Level Architecture
MeshCentral.com
Security is a prime concern of this entire solution. Most of it is solved using industry standards
such as TLS and PKI infrastructure to provide privacy, authentication and access control.
10
High Level Architecture
MeshCentral.com
Annex 1: Routing Cookie Authentication & Encryption
The routing cookie is used by Meshcentral to both provide routing information and authenticate
that the routing is allowed. The cookie is authenticated using HMAC-SHA256 and encrypted
using AES128-CBC.
The encoding method takes in a 48 byte long key, the first 32 bytes used for HMAC and the
remaining 16 bytes for AES. The initialization vector is
The following C# code is used to encrypt the routing cookie using AES. The key and the routing
instruction and provided as input and the output is a Base64 secured message.
private static string EncodeHttpRoutingCookieEx(byte[] key, byte[] msg)
{
if (key == null || key.Length != 48 || msg == null) return null;
MemoryStream mem2 = MeshUtils.GetMemoryStream();
// Split the key
byte[] key1 = new byte[32]; // HMAC
byte[] key2 = new byte[16]; // AES
Array.Copy(key, 0, key1, 0, 32);
Array.Copy(key, 32, key2, 0, 16);
// Select an IV
byte[] iv = new byte[16];
for (int i = 0; i < 16; i++) iv[i] = (byte)Random.Next(256);
// Setup and perform HMAC-SHA256
using (HMACSHA256 hmac = new HMACSHA256(key1))
{
11
High Level Architecture
MeshCentral.com
byte[] hmacresult = hmac.ComputeHash(msg);
mem2.Write(hmacresult, 0, hmacresult.Length);
mem2.Write(msg, 0, msg.Length);
msg = mem2.ToArray();
mem2.SetLength(0);
}
// Setup AES128-CBC
byte[] buf;
using (Rijndael rij = new RijndaelManaged())
{
rij.KeySize = 128;
rij.Mode = CipherMode.CBC;
rij.Padding = PaddingMode.PKCS7;
rij.Key = key2;
rij.IV = iv;
// Perform AES128 encrypt
using (MemoryStream mem = new MemoryStream())
{
using (CryptoStream cs =
new CryptoStream(mem, rij.CreateEncryptor(), CryptoStreamMode.Write))
{
cs.Write(msg, 0, msg.Length);
cs.FlushFinalBlock();
buf = mem.ToArray();
}
}
}
// Perform formatting
mem2.Write(iv, 0, 16);
mem2.Write(buf, 0, buf.Length);
string r = "MRC" + UrlEscapeBase64(Convert.ToBase64String(mem2.ToArray()));
MeshUtils.RecycleMemoryStream(mem2);
return r;
}
To be compatible with .NET 2.0, we use “Rijndael” instead of AES, but they are identical for our
purposes.
12