Meshcentral.com Meshcentral High Level Architecture Version 0.0.3 Monday, June 25, 2012 Ylian Saint-Hilaire © 2011 Intel Corporation. All Rights Reserved. High Level Architecture MeshCentral.com Legal Notices and Disclaimers Disclaimers INTEL CORPORATION MAKES NO WARRANTY OF ANY KIND WITH REGARD TO THIS MATERIAL, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. INTEL CORPORATION ASSUMES NO RESPONSIBILITY FOR ANY ERRORS THAT MAY APPEAR IN THIS DOCUMENT. INTEL CORPORATION MAKES NO COMMITMENT TO UPDATE NOR TO KEEP CURRENT THE INFORMATION CONTAINED IN THIS DOCUMENT. THIS SPECIFICATION IS COPYRIGHTED BY AND SHALL REMAIN THE PROPERTY OF INTEL CORPORATION. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED HEREIN. INTEL DISCLAIMS ALL LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT OF ANY PROPRIETARY RIGHTS, RELATING TO IMPLEMENTATION OF INFORMATION IN THIS SPECIFICATION. INTEL DOES NOT WARRANT OR REPRESENT THAT SUCH IMPLEMENTATIONS WILL NOT INFRINGE SUCH RIGHTS. NO PART OF THIS DOCUMENT MAY BE COPIED OR REPRODUCED IN ANY FORM OR BY ANY MEANS WITHOUT PRIOR WRITTEN CONSENT OF INTEL CORPORATION. INTEL CORPORATION RETAINS THE RIGHT TO MAKE CHANGES TO THESE SPECIFICATIONS AT ANY TIME, WITHOUT NOTICE. Legal Notices Intel software products are copyrighted by and shall remain the property of Intel Corporation. Use, duplication or disclosure is subject to restrictions stated in Intel's Software License Agreement, or in the case of software delivered to the government, in accordance with the software license agreement as defined in FAR 52.227-7013. The Intel logo is a registered trademark of Intel Corporation. Other brands and names are the property of their respective owners. i High Level Architecture MeshCentral.com Table of Contents Legal Notices and Disclaimers .......................................................................................................... i Disclaimers....................................................................................................................................... i Legal Notices ................................................................................................................................... i 1. Abstract ................................................................................................................................... 1 2. Introduction ............................................................................................................................. 1 3. Mesh Agent ............................................................................................................................. 3 3.1 Management port (TCP 16991) ....................................................................................... 4 3.2 Peer-to-peer ports (TCP & UDP 16990) .......................................................................... 4 3.3 Mesh Server Connection ................................................................................................. 5 4. Swarm Server ......................................................................................................................... 6 5. AJAX Server ........................................................................................................................... 7 6. Social Server ........................................................................................................................... 9 7. Conclusion .............................................................................................................................. 9 Annex 1: Routing Cookie Encryption ............................................................................................. 11 ii High Level Architecture MeshCentral.com Document Changes Sept 22. 2011 – Version 0.0.1 Initial version. Oct 17. 2011 – Version 0.0.2 Added WebSocket port 8085 on the AJAX server. June 12. 2012 – 0.0.3 Grammatical corrections. iii High Level Architecture MeshCentral.com 1. Abstract This document is intended to be a high level overview of how Meshcentral works: From the mesh agents to the basic functions of the online service. This document is intended for anyone that wants to understand how this technology works, review its design or for developers who will be using mesh as a way to write more complicated applications and route traffic across networks. Basic knowledge of what Meshcentral is, its features and usages must be understood by the reader before taking on this document. 2. Introduction The mesh project allows users the ability to remotely manage and control PC’s and devices from any network using a single web site with no changes to the local network infrastructure. Just install the mesh agent and the computer will show up on the Meshcentral web site and is immediately manageable. To make this solution work, let’s start by splitting the design into its components and see the high level network data flow between them. First, we have the mesh agent. The agent runs on each managed computer and has data flows to other agents, the mesh central server, and possibly applications that make use of its services. It also stores local data into a local database. As a result, the high level diagram of the mesh agent looks something like this: 1 High Level Architecture MeshCentral.com Note that the agent can logically be split into two components, the peer-to-peer portion and the management portion. In this diagram, the arrows point in the direction of connection initiation, but traffic in all cases is two way. Both portions of the agent are tightly bound, but provide different features. We will come back to the agent in a later section. The server can also be split into different components and again, we have network protocols used for communication between these components. The light level design of the server looks something like this: 2 High Level Architecture MeshCentral.com Here, all of the server’s components may run on the same computer or different computers. Since network protocols and not internal API’s link the components. How the server is deployed depends on scalability tradeoffs. Most of the data between server components is transferred thru the database or Microsoft Message Queuing, with the exception of the AJAX server that occasionally needs to route connection traffic directly. In the following sections, we will look at both the mesh agent and meshcentral server in more detail. The goal will be overall understanding of the mesh components and their interaction and how security is maintained. 3. Mesh Agent The mesh agent is truly the core of the entire solution, its unique design allows for many interesting usages and benefits. The mesh agent is entirely build with raw C code with the exception of a few C++ modules that are specific to Windows. So far, the mesh agent has been ported to Windows, Mac OS X, Linux, MeeGo, Android operating systems. When applicable, the agent runs both 32 and 64 bits and on x86, ARM and MIPS processors. The mesh agent links against two major libraries: OpenSSL and SQLite. Both are widely supported on many platforms. OpenSSL offers all of the cryptographic methods used by the mesh agent, and SQLite the disk storage primitives. Sometimes other libraries are also linked to support specific OS specific features like remote desktop. In general, we opt to statically link all required libraries to the mesh agent. This makes the agent significantly larger, in fact over 10 times larger (200k to 2 megabytes), but we gain in having each agent able to run on a wider array of computers and possibly some security gains. When compiled, each agent is tagged with an architecture ID, a version number and signed using a developer certificate. All this information is used to perform remote updates of the agent (both peer-to-peer and server-to-node). 3 High Level Architecture MeshCentral.com If we look at the network, the Mesh agent opens up 4 sockets for incoming local traffic. These are port TCP 16991, TCP 16990, UDP 16990 and UDP 1900. Upon installation of the mesh agent, the proper firewall rules will be added to the Microsoft Windows firewall automatically. Let’s look at what each of these inbound ports is used for: 3.1 Management port (TCP 16991) The management port is simple to understand, it’s a port that accepts mutually authenticated TLS connections and once authenticated using a client certificate, allows the caller to perform administrative and management tasks on this computer. The features offered by the manageability agent include: remote power control, remote file management, remote desktop and remote command prompt. Any computer on the network can attempt to connect to this port. If authenticated, it can perform the commands. Often times, we use this port along with mesh traffic relay over the internet, but it can also be used within the local network for quick management. Once the TLS connection is established, the management port uses binary commands to operate. Commands are always 4 byte headers optionally followed by more data. 3.2 Peer-to-peer ports (TCP & UDP 16990) Mesh agents will initially discovery each other using a local multicast on UDP port 16990. So the mesh agent not only opens this port, it also joins it to a multicast channel. Agents will multicast periodically until they find another agent, at which point the multicast stops and all future discovery is done using peer-to-peer unicast instead. The mesh agents have a unilateral, one way relation with other peers. The peers that are friends with a node are not identical to the friends on this node. Also, the mesh agent has a maximum 20 peers it can communicate with. The process of selecting these peers is outside the scope of this document, but the result is a “balanced graph” with each node having 20 friends that are correctly selected. Once a candidate peer is selected for connection, the mesh agent starts by attempting a mutually-authenticated HTTPS request to the remote peer. If successful, the remote peer will return a session key that will be used for all future encrypted UDP communication between these two nodes. 4 High Level Architecture MeshCentral.com As a result, at a high level peer communicate to each other following this timeline: When the HTTPS session is established, each node uses a TLS certificate that is derived from its node certificate in such a way that each node can compute the other node’s node identifier. The node identifier is the SHA256 hash of the node’s public key. In this design, the nodes never transmit what nodeid they have, rather the nodeid is inferred using the certificate using in the HTTPS session. In order for Node B to maintain no state about Node A, the session key returned by Node B is actually a hash of the Node A identifier with a node B secret session random. This way, Node B does not have to keep any state yet, can decrypt all UDP messages sent by Node A. UDP messages are encrypted using AES-CBC and authenticated using HMAC-SHA256. The periodic UDP Sync serves to indicate that the peer is still present and serves to trade any state that may have changed. All peer-to-peer state in this design is blocks that are signed using RSA2048 certificates, so state that is moved from node to node can’t be changed while in transit. 3.3 Mesh Server Connection Now that we have addressed the inbound management and peer to peer ports, let’s turn our attention to the connection made by mesh agents to the central server. The connection is always initiated from the mesh agent to the server, this allows for router, firewall and proxy traversal. The server connection is authenticated and secured using a mutually authenticated TLS connection. Just like when peer-to-peer connections occur, each side of the connection infers the others parties identifier by performing a hash of the public key of the certificate. In the case of a server connection, the agent checks the mesh server’s certificate against its policy and the server infers the nodes identifier. 5 High Level Architecture MeshCentral.com The mesh agent connects to a component called the “swarm server”. The swarm server communicates to all mesh agents using binary commands that are secured using TLS. The mesh agent starts by sending the server its signed information block, then, other peer information is sent. The connection is kept open and often idle except for occasional 4 byte ping commands. Using this connection, the administrator can initiate operations such as power commands or TCP relay from the server to the local private network. All operations are authenticated and secured within the TLS connection. The mesh agent will also verify that the server is authorized to perform a remote operation by checking its mesh policy. 4. Swarm Server The swarm server is a C# application that runs on one or more server and is built to speak the binary language that mesh agents and mesh consoles (tools) use to do complex operations. Much like ASP.net speaks HTTP, the swarm server has its own port and speaks its own protocol. We opted to use a binary protocol to facilitate transferring bulk data for socket relay and to keep the protocol very efficient. Here is a more complete overview of the Swarm Server. When first starting up, the swarm service loads many settings from a text configuration file. If many swarm servers are running, each instance must have its own configuration file. The file indicates the certificates to use, ports to bind to, MSMQ settings and more. The logs generated by the swarm service into log files are indented to help debug the software; they include all exceptions and unexpected failures. 6 High Level Architecture MeshCentral.com The only exposed public port is port 8080, it’s used by various actors but the most frequent by far is mesh agents who connects to the server and authenticate based on their mesh policy. Port 8080 is secured using Microsoft .NET TLS. TLS connections that use a TLS client certificate are mesh agents, other TLS connections that don’t use a client certificate are mesh tools or TCP relay requests. Port 8088 is identical to port 8080 except that it is not authenticated and must only be exposed to “localhost” or to the swarm server’s private network. When a connection arrived on this port, it’s assumed to already be authenticated and coming from a trusted entity. Currently the AJAX server makes use of this port. Two other API’s are used to access the database (Microsoft SQL Server) and the message queue (Microsoft Queuing Server). The queuing server is setup using parameters found in the configuration file, there is one inbound queue and one or more outbound queue can be setup. In general, commands sent to the queue must be received by all other components. The commands content is binary. 5. AJAX Server The AJAX server was built to provide web applications on Meshcentral live two way communication. An interactive web page is loaded from IIS and the Javascript then connects back to the AJAX server with long standing requests that wait until data is available. When a web application first contacts the AJAX server, the server will keep state about this browser along with a data queue. When the data queue fills up with information for the browser, it is kept until Javascript makes the next request for data. If no data is in the queue, the browser 7 High Level Architecture MeshCentral.com session is kept waiting until data is put into the queue. This system allows Meshcentral to offer usages such a remote desktop, remote command prompt, list chat and remote file management. The AJAX port 8084 is TLS secured using the same certificate used by IIS. In fact, IIS is the underlining HTTP session handler for the AJAX server. To setup the AJAX server, we must run the following commands on the server: // Setup the certificate netsh http add sslcert ipport=0.0.0.0:8084 certhash=xxxxxxxxxxxxxxxxxx appid={8bf83834-1594-4051-b4a7-5693561b257a} clientcertnegotiation=enable // Setup both AJAX server mappings netsh http add urlacl url="https://+:8084/" user="Local Service" netsh http add urlacl url="https://+:443/ajax/" user="Local Service" // View the resulting settings netsh http show sslcert netsh http show urlacl These commands setup the HTTPS certificate and HTTP handlers. Note that the AJAX server can be accessed on its own port 8084, or on port 443 with the suffix “/ajax”. Because some network environments restrict communication to ports other than 443, most of the AJAX web applications will use the 443 port as much as possible. When relaying a full web page, we need control of the root path and so, port 8084 must be used. Both IIS and AJAX server should make use of a proper web certificate that is signed by an authorized certificate authority. In order to access services provided by the AJAX server, the web application must has previously obtained an authentication token. Two authentication systems exist: This token is created and stored in the database once ASP.NET has authenticated the user. ASP.net embeds the token in the web application’s page. This token is passed to the AJAX server by the web browser. An AES encrypted routing cookie is first obtained by the web application, the browser then makes requests to the AJAX server using this cookie. The second authentication system was added later and has the added benefit that the cookie can be created (encrypted using AES) by a completely different server. As long as the other server has the AES encrypted key & their clocks are in sync, it can generate routing cookies as needed. Both authentication systems have an embedded timeout, generally 30 minutes. The AJAX server also hosts a WebSocket port (8085) which is a new HTML5 technology for live two-way data transfer over an HTTP channel. Port 8085 is also secured using TLS using the same trusted certificate that secures port 443 and 8084. More information on WebSocket can be found on Wikipedia at: http://en.wikipedia.org/wiki/WebSocket. Our WebSocket implementation on port 8085 supports both “draft-ietf-hybi-thewebsocketprotocol-00” and “draft-ietf-hybithewebsocketprotocol-06”. 8 High Level Architecture MeshCentral.com 6. Social Server The social server is a separate C# application that facilitates the interaction between meshcentral.com and social media sites. Currently, this application only supports Twitter. The application listens to the “MeshCentral” Twitter feed for user commands, and posts events obtained from the MSMQ software bus over to Twitter when needed. The social server uses OAuth to gain permission on post on user’s Twitter feed, this is the industry norm. This server has no exposed public ports. It mostly uses the database to check permissions, get OAuth information and other information to complete text messages. If uses MSMQ to get actions that may be interesting to broadcast and to send command that users have performed (A user may power on or off a computer using a Twitter command). 7. Conclusion The combination of the Mesh Agent, ASP.net application, AJAX server, Swarm server & Social Server allows for a rich set of online web based interactive usages. These components are also split into this architecture to make it easier to debug, and make the server more scalable. By building on top of known robust database and message queuing components, we insure that this solution may scale well going forward. 9 High Level Architecture MeshCentral.com Security is a prime concern of this entire solution. Most of it is solved using industry standards such as TLS and PKI infrastructure to provide privacy, authentication and access control. 10 High Level Architecture MeshCentral.com Annex 1: Routing Cookie Authentication & Encryption The routing cookie is used by Meshcentral to both provide routing information and authenticate that the routing is allowed. The cookie is authenticated using HMAC-SHA256 and encrypted using AES128-CBC. The encoding method takes in a 48 byte long key, the first 32 bytes used for HMAC and the remaining 16 bytes for AES. The initialization vector is The following C# code is used to encrypt the routing cookie using AES. The key and the routing instruction and provided as input and the output is a Base64 secured message. private static string EncodeHttpRoutingCookieEx(byte[] key, byte[] msg) { if (key == null || key.Length != 48 || msg == null) return null; MemoryStream mem2 = MeshUtils.GetMemoryStream(); // Split the key byte[] key1 = new byte[32]; // HMAC byte[] key2 = new byte[16]; // AES Array.Copy(key, 0, key1, 0, 32); Array.Copy(key, 32, key2, 0, 16); // Select an IV byte[] iv = new byte[16]; for (int i = 0; i < 16; i++) iv[i] = (byte)Random.Next(256); // Setup and perform HMAC-SHA256 using (HMACSHA256 hmac = new HMACSHA256(key1)) { 11 High Level Architecture MeshCentral.com byte[] hmacresult = hmac.ComputeHash(msg); mem2.Write(hmacresult, 0, hmacresult.Length); mem2.Write(msg, 0, msg.Length); msg = mem2.ToArray(); mem2.SetLength(0); } // Setup AES128-CBC byte[] buf; using (Rijndael rij = new RijndaelManaged()) { rij.KeySize = 128; rij.Mode = CipherMode.CBC; rij.Padding = PaddingMode.PKCS7; rij.Key = key2; rij.IV = iv; // Perform AES128 encrypt using (MemoryStream mem = new MemoryStream()) { using (CryptoStream cs = new CryptoStream(mem, rij.CreateEncryptor(), CryptoStreamMode.Write)) { cs.Write(msg, 0, msg.Length); cs.FlushFinalBlock(); buf = mem.ToArray(); } } } // Perform formatting mem2.Write(iv, 0, 16); mem2.Write(buf, 0, buf.Length); string r = "MRC" + UrlEscapeBase64(Convert.ToBase64String(mem2.ToArray())); MeshUtils.RecycleMemoryStream(mem2); return r; } To be compatible with .NET 2.0, we use “Rijndael” instead of AES, but they are identical for our purposes. 12