Difference between revisions of "SHIPHarbour"

Latest revision as of 10:28, 3 December 2018

Glossary of Terms

• Socket - A communications pipe that uses TCP/IP to send or receive data (not both)
• protobuf API - The Google protocol buffer library of functions and objects used to encode and decode data across different programming languages
• SHIPHarbour API - The custom protocol defined by Serious Integrated and written using protobuf syntax
• SHIPHarbour Language Binding - A language-specific library of functions intended to make the SHIPHarbour API more convenient for use in the target language
• SHIPHarbour Master - An application developed by Serious Integrated which translates SHIPHarbour API messages to SHIPBridge API commands
• SHIPBridge API - The proprietary protocol and library of functions defined by Serious Integrated and written in embedded C
• Message - A series of bytes representing either a Google protocol buffer or a SHIPBridge command/response
• Routine - A Google protocol buffer message containing instructions to execute one or more SHIPBridge commands

Introduction

SHIPHarbour API Overview

All Serious Integrated modules are capable of using the SHIPBridge protocol to be remote controlled over common communication interfaces (e.g., USB, UART, SPI, etc.). The SHIPBridge API is freely available to be ported to customer-defined embedded platforms. However, customers may also wish to control modules using a PC-based application. Most PC application developers are not familiar with embedded C and are not prepared to port the SHIPBridge API to their target language. The SHIPHarbour API allows the developer to use familiar and expected design concepts of modern high-level languages.

SHIPHarbour is more than just a high-level API to execute individual SHIPBridge commands. Due to the real-time embedded nature of the platforms, we cannot expect the module to service a SHIPBridge command without either blocking or being blocked by other important processes. Therefore, most SHIPBridge commands are simplified to ensure shorter execution and response times overall. This also means multiple SHIPBridge commands may need to be issued to complete what a user might think should be a single operation. SHIPHarbour messages are designed to reduce this complexity for the high-level developer such that an operation that takes dozens of SHIPBridge commands will instead only require a single SHIPHarbour request. SHIPHarbour also facilitates multiple applications communicating with a module over singular or multiple physical connections all simultaneously. Furthermore, no changes to the API or application code are explicitly required to enable communications over new interfaces. It is not necessary here to go into detail how the SHIPHarbour messages are translated into SHIPBridge commands or how the SHIPBridge commands and responses are transmitted and received. This is because all the necessary processing is standardized and abstracted by the SHIPHarbour Master PC application. Customer applications using SHIPHarbour are expected to be implemented as clients of this program.

Sockets Used in SHIPHarbour Master

While it is not ultimately required to use SHIPHarbour or SHIPHarbour Master to control modules, it is certainly the fastest method for most developers looking to control modules directly from the PC environment. Part of the reason for this is the choice of sockets as the method for communicating with SHIPHarbour Master. Every modern high-level programming language will be able to use sockets and will typically have a built-in API for configuring and using sockets. Most of these APIs will also allow the developer to take advantage of multi-threading principles. This is important as the developer typically cannot predict when SHIPHarbour Master will need to send messages about the changes in the status of a module. This also avoids having to enforce strict synchronization between the customer application and SHIPHarbour Master about when requests are allowed.

The term socket, used here, refers to one half of a bidirectional TCP/IP interface which is capable of exchanging arbitrary data. Using the correct addresses, the PC can transmit the data to itself without having to send the data over an actual network connection. This allows SHIPHarbour Master to exchange data with the reliability and universality provided by using the TCP/IP standard. Incidentally, this also means that it is possible for the developer to configure SHIPHarbour Master to be accessible from an actual network connection. The receive socket is typically referred to as a server since it will listen for incoming requests and pass the request data to an application to service the request. The transmit socket is referred to as a client since it merely sends requests to a server.

The server sockets the developer must implement are categorized as Application Control or Module Control sockets. An Application Control server socket must be active throughout the entire time the custom application is intended to communicate with SHIPHarbour Master. These sockets are used to receive spontaneous module status updates as well as other data required for the custom application to create Routine requests. There is typically only one Application Control server socket per application. In contrast, multiple Module Control server sockets may be created. These sockets are used to receive any Routine status updates to a previously issued Routine request.

The client sockets the developer must implement are categorized as Announce, Application Control, or Module Control sockets. The Announce client socket is always the starting point for an application to begin connecting to SHIPHarbour Master and will only be active during the sending of a single, predetermined SHIPHarbour request. This socket is used to send information about how SHIPHarbour Master should connect to the custom application’s Application Control server socket. An Application Control client socket is expected to be active after SHIPHarbour Master sends information to the Application Control server socket about how the application should connect the client socket and for the entire time the custom application intends to communicate. An Application Control client socket is used to send requests for managing the socket connections to SHIPHarbour Master. Module Control client sockets are used to send any Routine requests to be executed on a target module. A single Module Control socket may be used to communicate to any and all connected modules simultaneously. However, using this approach requires the developer to analyze certain fields of the response in order to relate it back to the original request properly.

Implementation Using protobuf

All messages defined in the SHIPHarbour API are defined using Google’s protobuf proto3 syntax. Using protobuf allows the message definitions to be language independent while also taking advantage of the provided protoc compiler to generate optimal language-specific code for creating, sending, and receiving each message. The protobuf language has many conveniences that allow message definitions to be inherently extensible, backwards compatible, and result in a highly flexible encoding format. The protobuf library is open and free to use without the typical open source project limitations and supports most common high-level languages such as C++, Java, C#, Python, Go, etc. More information can be found at the following website: https://developers.google.com/protocol-buffers/

SHIPHarbour API

File structure

The following files define the SHIPHarbour API:

• SHIPHarbour_DataTypes.proto - Defines the basic types and message structures used as fields in other messages
• SHIPHarbour_Generic.proto - Defines the wrapper message that is used to send all other messages
• SHIPHarbour_Master.proto - Defines the message structures required to execute SHIPHarbour Master control Routines
• SHIPHarbour_Platform.proto - Defines the message structures required to execute platform-related Routines
• SHIPHarbour_Firmware.proto - Defines the message structures required to execute firmware-related Routines
• SHIPHarbour_Files.proto - Defines the message structures required to execute file-related Routines
• SHIPHarbour_XVars.proto - Defines the message structures required to execute external-variable-related Routines

Keywords

• import - used to allow the current file to access the definitions of another .proto file
• message - indicates the start of a message structure definition
• oneof - indicates only one of the scoped fields can be populated at any given time
• enum - indicates the start of a definition of integer values with special meanings
• repeated - indicates the field should be accessed as a list, may be empty

A simple example of proto3 syntax:

message RoutineInfo1 {
    Endpoint2    ep3   = 14; 
    UInt32Value5 id    = 2;
    bool6        start = 3;
}

1 The name that will be used to refer to this type of message
2 A previously defined message type contained within this message definition
3 The name that will be used to refer to this field of the message
4 An internally used number that uniquely identifies a field
5 A wrapper for a primitive data type used to facilitate detecting if the field is present in a received message
6 A primitive data type, default values are always 0 (no distinction can be made between the assigned value being equivalent to the default and the value not having been assigned)

For more details on the proto3 syntax, go to https://developers.google.com/protocol-buffers/docs/proto3.

In order to avoid name collisions with other libraries, the typical naming scheme for SHIPHarbour API structures will start with something similar to "SHIPHarbour.Message." followed by a shortened version of the filename of the .proto file in which the structure is defined, after which, the specific definition will appear. For example, the full namespace for creating a message to cancel a previously requested Routine may look something like: "SHIPHarbour.Message.Generic.Routine.Cancel". The exact syntax will depend on the target language being used. For additional details, go to https://developers.google.com/protocol-buffers/docs/reference/overview.

SHIPHarbour Messages

There are two logical categories of messages, the control message and the routine message. All message definitions simultaneously define the command and response structure for that particular request. However, this does not mean all requests require a response. Control messages are typically meant to be used along with the Application Control sockets. These messages open and close socket connections and carry important data required for the application to build routine messages correctly. Routine messages must be sent using Module Control sockets and are then sorted, delivered, and queued for execution by SHIPHarbour Master with respect to a specified module. Executing one of these messages means reading the fields to drive internal logic which sends one or more SHIPBridge commands directly to the module.

• Examples of control messages
  • SHIPHarbour.Message.Master.Open
  • SHIPHarbour.Message.Master.ModuleInfo
• Examples of routine messages
  • SHIPHarbour.Message.Platform.SetState
  • SHIPHarbour.Message.Firmware.Installed
  • SHIPHarbour.Message.XVars.Exchange

Using Protobufs

Building messages

While the exact code for constructing a SHIPHarbour message using the protobuf API is language dependent, there are a few things to note:

• All messages sent to or received from SHIPHarbour Master must use the Generic message as a wrapper.

• All messages must have the version field set.

• All messages must have exactly one of the fields of the payload field set.

• A message which is not a response to a previous request should not have the error field set.

C++ Example:

SHIPHarbour::Message::Generic generic;
generic.set_version(/* version */);

SHIPHarbour::Message::Master::Open * pOpen = generic.mutable_open();
pOpen->mutable_host()->set_value(/* host address string */);
pOpen->mutable_port()->set_value(/* port number */);

Java example:

StringValue host = StringValue.newBuilder()
	.setValue(/* host address string */)
	.build();

UInt32Value port = UInt32Value.newBuilder()
	.setValue(/* port number */)
	.build();

Open open = Open.newBuilder()
	.setHost(host)
	.setPort(port)
	.build();

Generic generic = Generic.newBuilder()
	.setVersion(/* version */)
	.setOpen(open)
	.build();

For more information on using protobufs in the target language, visit https://developers.google.com/protocol-buffers/docs/reference/overview

Sending/Receiving Messages

By default, the serialized protobuf does not include a description of the full size of the message. In order to allow sending messages as fast as possible, the SHIPHarbour API requires that the byte size of the message to be parsed appear as a varint-encoded 32-bit integer immediately before the message bytes. For some languages, the protobuf API implements a set of functions named similarly to writeDelimitedTo and parseDelimitedFrom. All messages must either be encoded/decoded using these functions, or the developer must implement the varint encoding/decoding of the message size manually.

C++ example:

/* 
  Least-significant byte first, most-significant bit is 1 when more bytes. 
  There can be up to 5 bytes of size information.
 */
void parseVarInt32(char * pSrc, int * pVarInt, int * pIndex) {
	char data;
	*pVarInt = 0;
	do {
		if (pSrc->size() <= *pIndex) {
			break;
		}
		data		= pSrc[*pIndex];
		*pVarInt	= (data & 0x7F) << ((*pIndex)++ * 7);
	} while ((data & 0x80) && (*pIndex < 6));
}

void serializeVarInt32(char * pDst, int * pIndex, int varint) {
	char data;
	do {
		data 	= varint & 0x7F; 
		varint	= varint >> 7;
		if (varint) {
			data 	= data | 0x80; 
		}
		pDst[(*pIndex)++]	= data;
	} while (varint);
}

ByteArray msg = new ByteArray(5 + generic.ByteSize());

int index;
serializeVarInt32(msg.data(), &index, generic.ByteSize());

generic.SerializeToArray(&(msg.data()[index]), generic.ByteSize());
pSocket->write(msg);

Java example:

 ByteArrayOutputStream bos = new ByteArrayOutputStream();
 generic.writeDelimitedTo(bos);
 socket.getOutputStream().write(bos.toByteArray());

Available Language Bindings

For some languages, Serious has already developed a full-featured language binding. These language bindings can be useful as they allow the developer to immediately start connecting and sending messages to SHIPHarbour Master by calling abstracted functions.

If a language binding for the target language is not available or the standard language binding is not suitable for the target environment, Serious may be able to create or modify the official language binding by request. Otherwise, refer to the section "Using the SHIPHarbour API Directly" .

Usage examples, source code, and binaries for each language binding are available by request. Refer to the following pages for documentation on how to use each language binding: - Java

Using the SHIPHarbour API Directly

Getting Started with protobuf

This section of the documentation is only necessary if a language binding made by Serious does not exist or cannot be used. To create a custom language binding the following steps will be required:

1. Compile the protobuf source into a library for the target language (at the time of writing: v3.0.0-beta3)
2. Download a pre-built version of the protobuf library
3. Compile the proto files for the target language
   • All of the .proto files for the SHIPHarbour API should be compiled with the protobuf compiler "protoc" using the following command structure:
     

     protoc --<language_switch>=<destination_dir> --proto_path=<source_dir>

   • Examples of language switch:
     

     --cpp_out
     --csharp_out
     --java_out

   • Use the following command to find the appropriate switch for the target language:
     

     protoc --help

   • The destination directory can be any location and is where the generated files will appear
     
   • The source directory must be the location of all of the SHIPHarbour .proto files.

Connecting to SHIPHarbour Master

The below steps should be followed to establish connection between SHIPHarbour Master and another application:

• Open a receive socket on any port (referred to as Socket_AppControl_Rx).
• Open a transmit socket to SHIPHarbour Master, the default port is 10400 (referred to as Socket_Announce_Tx).
• Send an Open message into Socket_Announce_Tx with your host address and the port number used by Socket_AppControl_Rx.
• Socket_Announce_Tx will be forcibly closed when the message is received as it is no longer needed.
• An Open message will be sent into Socket_AppControl_Rx describing how to create Socket_AppControl_Tx.
• A PluginInfo message will be sent into Socket_AppControl_Rx describing what plugins SHIPHarbour Master has loaded.
• A ModuleInfo message will be sent into Socket_AppControl_Rx describing which modules are currently connected to SHIPHarbour Master.
• It is now possible to issue application-level control messages using Socket_AppControl_Tx.
• The above sequence corresponds to the first half of the flow chart.

Getting information on connected modules

While it is possible to poll this information by sending a ModuleInfo message into Socket_AppControl_Tx, it is not recommended or necessary to do so. SHIPHarbour Master will always send a ModuleInfo message into Socket_AppControl_Rx whenever the connection status of a module or one of the endpoints changes.

Connecting To a Module

The below steps should be followed to establish connection between a custom PC application and a module:

• Open a receive socket on any port (referred to as Socket_ModuleControl_Rx).
• Send an Open message into Socket_AppControl_Tx with your host address and the port number used by Socket_ModuleControl_Rx.
• An Open message will be sent into Socket_ModuleControl_Rx describing how to create Socket_ModuleControl_Tx.
• It is now possible to issue routine messages to a specified module using Socket_ModuleControl_Tx.
• The above sequence corresponds to the second half of the flow chart.

Using ModuleInfo Messages

A ModuleInfo message has only one field which is marked as a repeatable of type ModuleDetails. This means that a single ModuleInfo message may contain the information for multiple modules so the field must be accessed as a list, array, or some other collection of ModuleDetails depending on the target language.

Detection of whether a module has become connected or disconnected must be done per application. SHIPHarbour Master does not maintain a list of what modules each connected PC application has seen. When any one module (or endpoint within a module) changes state, the current information for all modules is broadcasted to all connected PC applications. The PC application must keep a record of known modules and endpoints and then compare and manage this record accordingly as new ModuleInfo messages are received from SHIPHarbour Master.

SHIPHarbour_Master.proto:

 message ModuleInfo {
     repeated DataTypes.ModuleDetails info = 1;
 }

The ModuleDetails that are contained within a ModuleInfo message contain some fields which may be considered superfluous by the PC application. The field that all PC applications must pay attention to is the module field of type Module. The Module datatype contains the unique serial number of the SIM or SCM as a 64-bit value and, more importantly, a list of Endpoint data describing how the module is currently connected to SHIPHarbour Master.

SHIPHarbour_DataTypes.proto:

 message ModuleDetails {
     Module      module          = 1;
     UInt32Value family          = 2;
     UInt32Value version         = 3;
     UInt32Value variant         = 4;
     UInt32Value lcdCode         = 5;
     UInt32Value id              = 6;
     Firmware    currentProgram  = 7;
 }
 
 message Module {
                 Fixed64Value   sn       = 1;
                 BytesValue     product  = 2;
     repeated    Endpoint       version  = 3;
 }

The Endpoint datatype has an id field which is required to send all routine messages. These directly correspond to some physical communications interface over which the module is using SHIPBridge to communicate with SHIPHarbour Master. The PC application must always supply a valid Endpoint within a routine message in order to direct SHIPHarbour Master to begin executing a SHIPBridge operation over a specific physical interface. As the module may reset or otherwise lose the SHIPBridge connection context with SHIPHarbour Master, the id of the Endpoint will change over time. The PC application is responsible for recognizing when the id of a previously known Endpoint has changed and transitioning future operations or in-progress multi-request operations to use the new id.

SHIPHarbour_DataTypes.proto:

 message Endpoint {
     UInt32Value id      = 1;
     UInt32Value plugin  = 2;
     UInt64Value speed   = 3;
 }

Controlling a Module

Any operation that can be performed on a module is always done by sending what are classified as routine messages. Routine messages are defined in files according to what feature of the module will be accessed. Due to the way Generic messages are used, all routine messages that can possibly be sent can be seen in the SHIPHarbour_Generic.proto file. The following is an example of sending a routine message to retrieve information about what firmware files are installed on a module:

C++ example:

SHIPHarbour::Message::Generic generic;
generic.set_version(/* version */);

/* 
  This gets us a new instance of a Routine and sets the payload field of the
  Generic to this new instance.
*/
SHIPHarbour::Message::Generic::Routine * pRoutine = generic.mutable_routine();
SHIPHarbour::Message::DataTypes::RoutineInfo * pInfo = pRoutine->mutable_info();
pInfo->mutable_ep()->mutable_id()->set_value(/* known endpoint id */);
pInfo->mutable_id()->set_value(/* caller-generated unique id */); 
pInfo->set_start(true);

/* 
  This sets the routine field of the Routine to a new instance of an Installed
  message. Since there are no outgoing parameters required, we don’t 
  need to save the return value.
*/
pRoutine->mutable_installed();

ByteArray msg = new ByteArray(5 + generic.ByteSize());

int index;
serializeVarInt32(msg.data(), &index, generic.ByteSize());

generic.SerializeToArray(&(msg.data()[index]), generic.ByteSize());
pSocket->write(msg);

Java example:

Endpoint ep = Endpoint.newBuilder()
	.setId(/* known endpoint id */)
	.build();

RoutineInfo info = RoutineInfo.newBuilder()
	.setEp(ep)
	.setId(/* caller-generated unique id */)
	.start(true)
	.build();

/* 
  There are no outgoing parameters required, so we build the Installed message
  only as a parameter to the Routine builder.
*/
Routine routine = Routine.newBuilder()
	.setInfo(info)
	.setInstalled(Installed.newBuilder().build())
	.build();

Generic generic = Generic.newBuilder()
	.setVersion(/* version */)
	.setRoutine(routine)
	.build();

ByteArrayOutputStream bos = new ByteArrayOutputStream();
generic.writeDelimitedTo(bos) ;
socket.getOutputStream().write(bos.toByteArray());