X-Git-Url: https://gerrit.automotivelinux.org/gerrit/gitweb?a=blobdiff_plain;f=docs%2F4_APIs_and_Services%2F4.5_Message_Signaling%2F1_Message_Signaling%2Farchitecture.md;fp=docs%2F4_APIs_and_Services%2F4.5_Message_Signaling%2F1_Message_Signaling%2Farchitecture.md;h=0000000000000000000000000000000000000000;hb=eefc3ab6cbb8a5901632f46d99e13c8d90b2415d;hp=aaf7e0827f9caf78b2bf72e4f7bb38ed89fcb648;hpb=4aad369c9728061c97b3de792286e743ee884b09;p=AGL%2Fdocumentation.git diff --git a/docs/4_APIs_and_Services/4.5_Message_Signaling/1_Message_Signaling/architecture.md b/docs/4_APIs_and_Services/4.5_Message_Signaling/1_Message_Signaling/architecture.md deleted file mode 100644 index aaf7e08..0000000 --- a/docs/4_APIs_and_Services/4.5_Message_Signaling/1_Message_Signaling/architecture.md +++ /dev/null @@ -1,476 +0,0 @@ ---- -edit_link: '' -title: Message Signaling -origin_url: >- - https://raw.githubusercontent.com/automotive-grade-linux/docs-sources/master/docs/signaling/architecture.md ---- - - - ---- -title: AGL - Message Signaling Architecture -author: Fulup Ar Foll (IoT.bzh) -date: 2016-06-30 - -categories: architecture, appfw -tags: architecture, signal, message -layout: techdoc - ---- - -**Table of Content** - -1. TOC -{:toc} - -## Context - -Automotive applications need to understand in real time the context in which -vehicles operate. In order to do so, it is critical for automotive application -to rely on a simple, fast and secure method to access data generated by the -multiple sensors/ECU embedded in modern cars. - -This signaling problem is neither new, neither unique to the automotive and -multiple solutions often described as Message Broker or Signaling Gateway have -been around for a while. - -The present document is the now implemented since AGL Daring Dab version, to -handle existing signaling/message in a car. It relies on [[APbinder]] -binder/bindings model to minimize complexity while keeping the system fast -around secure. We propose a model with multiple transport options and a full set -of security feature to protect the service generating the signal as well as -consuming them. - -## Objectives - -Our objectives are solving following 3 key issues: - -1. reduce as much as possible the amount of exchanged data to the meaningful - subset really used by applications -1. offer a high level API that obfuscates low level and proprietary interface to - improve stability in time of the code -1. hide specificities of low level implementation as well as the chosen - deployment distribution model. - -To reach first objective, events emission frequency should be controlled at the -lowest level it possibly can. Aggregation, composition, treatment, filtering of -signals should be supported at software level when not supported by the hardware. - -Second objectives of offering long term stable hight level API while allowing -flexibility in changing low level implementation may look somehow conflicting. -Nevertheless by isolating low level interface from high level and allowing -dynamic composition it is possible to mitigate both objectives. - -## Architecture - -Good practice is often based on modularity with clearly separated components -assembled within a common framework. Such modularity ensures separation of -duties, robustness, resilience and achievable long term maintenance. - -This document uses the term "**Service**" to define a specific instance of this -proposed common framework used to host a group of dedicated separated components -that handle targeted signals/events. Each service exposes to services/applications -the signals/events it is responsible for. - -As an example, a CAN service may want to mix non-public proprietary API with -CANopen compatible devices while hiding this complexity to applications. The -goal is on one hand to isolate proprietary piece of code in such a way that it -is as transparent as possible for the remaining part of the architecture. On a -second hand isolation of code related to a specific device provides a better -separation of responsibilities, keeping all specificity related to a given -component clearly isolated and much easier to test or maintain. Last but not -least if needed this model may also help to provide some proprietary code -directly as binary and not as source code. - -Communicating between the car and regular apps should be done using a 2 levels -AGL services which have two distincts roles: - -- low level should handle communication with CAN bus device (read, decoding, - basic and efficient filtering, caching, ...) -- high level should handle more complex tasks (signals compositions, complex - algorythms like Kalman filter, business logic...) - -![image](images/signal-service-arch.svg "Signal Agent Architecture") - -To do so, the choice has been to use a similar architecture than [[OpenXC]], a -Ford project. Principle is simple, from a JSON file that describes all CAN -signals wanted to be handled, in general a conversion from a **dbc** file, AGL -generator convert it to a C++ source code file. This file which in turn is used -as part of the low level CAN service which can now be compiled. This service -reads, decodes and serves this CAN signals to a high level CAN service that -holds business logic and high level features like described is the above -chapter. - -![image](images/can-generator.svg "AGL CAN generator") - -While in some cases it may be chosen to implement a single service responsible -for everything, other scenarii may chose to split responsibility between -multiple services. Those multiple services may run on a single ECU or on -multiple ECUs. Chosen deployment distribution strategy should not impact the -development of components responsible for signals/events capture. As well as it -should have a loose impact on applications/services consuming those events. - -A distributed capable architecture may provide multiple advantages: - -- it avoids to concentrate complexity in a single big/fat component. -- it leverages naturally multiple ECUs and existing network architecture -- it simplifies security by enabling isolation and sandboxing -- it clearly separates responsibilities and simplifies resolution of conflicts - -Distributed architecture has to be discussed and about now is not fully -implemented. Low level CAN service isn't fully functional nor tested to assume -this feature but its architecture let the possibility open and will be -implemented later. - -![image](images/distributed-arch.png "Distributed Architecture") - -Performance matters. There is a trade-off between modularity and efficiency. -This is specially critical for signals where propagation time from one module to -the other should remain as short as possible and furthermore should consume as -little computing resources as possible. - -A flexible solution should provide enough versatility to either compose modules -in separate processes; either chose a model where everything is hosted within a -single process. Chosen deployment model should have minor or no impact on -development/integration processes. Deployment model should be something easy to -change, it should remain a tactical decision and never become a structuring -decision. - -Nevertheless while grouping modules may improve performance and reduce resource consumption, on the other hand, -it has a clear impact on security. No one should forget that some signals have very different level of security from other ones. -Mixing everything within a single process makes all signal's handling within a single security context. -Such a decision may have a significant impact on the level on confidence one may have in the global system. - -Providing such flexibility constrains the communication model used by modules: - -- The API of integration of the modules (the API of the framework) that enables - the connection of modules must be independent of the implementation of - the communication layer -- The communication layer must be as transparent as possible, its - implementation shouldn't impact how it is used -- The cost of the abstraction for modules grouped in a same process - must be as little as possible -- The cost of separating modules with the maximum of security must remain as - minimal as possible - -Another point impacting performance relates to a smart limitation on the number -of emitted signals. Improving the cost of sending a signal is one thing, -reducing the number of signals is an other one. No one should forget that the -faster you ignore a useless signal the better it is. The best way to achieve -this is by doing the filtering of useless signal as close as possible of the -component generating the signal and when possible directly at the hardware level. - -To enable the right component to filter useless signals, consumer clients must -describe precisely the data they need. A filter on frequency is provided since -Daring Dab version, as well as minimum and maximum limits. These filters can be -specified at subscription time. Also, any data not required by any client should -at the minimum never be transmitted. So only changed data is transmitted and if -another service needs to receive at a regular time, it has to assume that if no -events are received then it is that the value hasn't change. Furthermore when -possible then should even not be computed at all, a CAN signal received on -socket is purely ignored if no one asks for it. - -Describing expected data in a precise but nevertheless simple manner remains a -challenge. It implies to manage: - -- requested frequency of expected data -- accuracy of data to avoid detection of inaccurate changes -- when signaling is required (raising edge, falling edge, - on maintained state, ...), -- filtering of data to avoid glitches and noise, -- composition of signals both numerically and logically (adding, - subtracting, running logical operators like AND/OR/XOR, getting the mean, ...) -- etc... - -It is critical to enable multiple features in signal queries to enable modules -to implement the best computing method. The best computing method may have an -impact on which device to query as well as on which filters should be applied. -Furthermore filtering should happen as soon as possible and obviously when -possible directly at hardware level. - -### Transport Solutions - -D-Bus is the standard choice for Linux, nevertheless it has some serious -performance limitation due to internal verbosity. Nevertheless because it is -available and pre-integrated with almost every Linux component, D-Bus may still -remains an acceptable choice for signal with low rate of emission (i.e. HMI). - -For a faster communication, Jaguar-Land-Rover proposes a memory shared signal -infrastructure. Unfortunately this solution is far from solving all issues and -has some drawbacks. Let check the open issues it has: - -- there is no management of what requested data are. This - translate in computing data even when not needed. -- on top of shared memory, an extra side channel is required for processes - to communicate with the daemon. -- a single shared memory implies a lot of concurrency handling. This might - introduce drawbacks that otherwise would be solved through communication - buffering. - -ZeroMQ, NanoMSG and equivalent libraries focused on fast communication. Some -(e.g. ZeroMQ) come with a commercial licensing model when others (e.g. NanoMSG) -use an open source licensing. Those solutions are well suited for both -communicating inside a unique ECU or across several ECUs. However, most of them -are using Unix domain sockets and TCP sockets and typically do not use shared -memory for inter-process communication. - -Last but not least Android binder, Kdbus and other leverage shared memory, zero -copy and sit directly within Linux kernel. While this may boost information -passing between local processes, it also has some limitations. The first one is -the non support of a multi-ECU or vehicle to cloud distribution. The second one -is that none of them is approved upstream in kernel tree. This last point may -create some extra burden each time a new version of Linux kernel is needed or -when porting toward a new hardware is required. - -### Query and Filtering Language - -Description language for filtering of expected data remains an almost green -field where nothing really fit signal service requirements. Languages like -Simulink or signal processing graphical languages are valuable modelling tools. -Unfortunately they cannot be inserted in the car. Furthermore those languages -have many features that are not useful in proposed signal service context and -cost of integrating such complex languages might not be justified for something -as simple as a signal service. The same remarks apply for automation languages. - -Further investigations leads to some specifications already presents like the -one from Jaguar Land Rover [[VISS]], for **Vehicule Information Service -Specification** and another from Volkwagen AG named [[ViWi]], stand for -**Volkwagen Infotainment Web Interface**. Each ones has their differences and -provides different approach serving the same goal: - -| VISS | ViWi | -|---------------------------------------------------------------|-----------------------------------------------------------------| -| Filtering on node (not possible on several nodes or branches) | Describe a protocol | -| Access restrictions to signals | Ability to specify custom signals | -| Use high level development languages | RESTful HTTP calls | -| One big Server that handle requests | Stateless | -| Filtering | Filtering, sorting | -| Static signals tree not extensible [[VSS]] | Use JSON objects to communicate | -| Use of AMB ? | Identification of resources may be a bit heavy going using UUID | -| Use of Websocket | | - -About **[[VISS]]** specification, the major problem comes from the fact that -signals are specified under the [[VSS]], **Vehicle Signal Specification**. So, -problem is that it is difficult, if not impossible, to make a full inventory -of all signals existing for each car. More important, each evolution in signals -must be reported in the specification and it is without seeing the fact that -car makers have their names and set of signals that would mostly don't -comply with the [[VSS]]. VISS doesn't seems to be an valuable way to handle -car's signals, a big component that responds requests, use of **Automotive -Message Broker** that use DBus is a performance problem. Fujitsu Ten recent -study[[1]] highlights that processor can't handle an heavy load on CAN bus and -that Low level binding adopted for AGL is about 10 times[[2]] less impact on -performance. - -## Describing Signal Subscriptions using JSON - -JSON is a rich structured representation of data. For requested data, it allows -the expression of multiple features and constraints. JSON is both very flexible -and efficient. There are significant advantages in describing requested data at -subscription time using a language like JSON. Another advantage of JSON is that -no parser is required to analyse the request. - -Existing works exists to describe a signals that comes first from Vector with -its proprietary database (`DBC`) which widely used in industry. Make a -description based on this format appears to be a good solution and Open Source -community already has existing tools that let you convert proprietary file -format to an open one. So, a JSON description based on work from [[OpenXC]] is -specified [here](https://github.com/openxc/vi-firmware/blob/master/docs/config/reference.rst) -which in turn is used in Low level CAN service in AGL: - -```json -{ "name": "example", - "extra_sources": [], - "initializers": [], - "loopers": [], - "buses": {}, - "commands": [], - "0x3D9": { - "bus": "hs", - "signals": { - "PT_FuelLevelPct": { - "generic_name": "fuel.level", - "bit_position": 8, - "bit_size": 8, - "factor": 0.392157, - "offset": 0 - }, - "PT_EngineSpeed": { - "generic_name": "engine.speed", - "bit_position": 16, - "bit_size": 16, - "factor": 0.25, - "offset": 0 - }, - "PT_FuelLevelLow": { - "generic_name": "fuel.level.low", - "bit_position": 55, - "bit_size": 1, - "factor": 1, - "offset": 0, - "decoder": "decoder_t::booleanDecoder" - } - } - } -} -``` - -From a description like the above one, low level CAN generator will output -a C++ source file which let low level CAN service that uses it to handle such -signal definition. - -## Naming Signal - -Naming and defining signals is something very complex. For example just -***speed***, as a signal, is difficult to define. -What unit is used (km/h, M/h, m/s, ...)? -From which source (wheels, GPS, AccelMeter)? -How was it captured (period of measure, instantaneous, mean, filtered)? - -In order to simplify application development we should nevertheless agree on -some naming convention for key signals. Those names might be relatively complex -and featured. They may include a unit, a rate, a precision, etc. - -How these names should be registered, documented and managed is out of scope of -this document but extremely important and at some point in time should be -addressed. Nevertheless this issue should not prevent from moving forward -developing a modern architecture. Developers should be warned that naming is a -complex task, and that in the future naming scheme should be redefined, and -potential adjustments would be required. - -About Low level CAN signals naming a doted notation, like the one used by Jaguar -Landrover, is a good compromise as it describe a path to an car element. It -separates and organize names into hierarchy. From the left to right, you -describe your names using the more common ancestor at the left then more you go -to the right the more it will be accurate. Using this notation let you subscribe -or unsubscribe several signals at once using a globbing expression. - -Example using OBD2 standard PID: - -```path -engine.load -engine.coolant.temperature -fuel.pressure -intake.manifold.pressure -engine.speed -vehicle.speed -intake.air.temperature -mass.airflow -throttle.position -running.time -EGR.error -fuel.level -barometric.pressure -commanded.throttle.position -ethanol.fuel.percentage -accelerator.pedal.position -hybrid.battery-pack.remaining.life -engine.oil.temperature -engine.torque -``` - -Here you can chose to subscribe to all engine component using an expression -like : `engine.*` - -## Reusing existing/legacy code - -About now provided services use: - -- **Low Level** [[OpenXC]] project provides logic and some useful libraries to - access a CAN bus. It is the choice for AGL. - -- **High Level** In many cases accessing to low level signal is not enough. - Low level information might need to be composed (i.e. GPS+Gyro+Accel). - Writing this composition logic might be quite complex and reusing existing - libraries like: LibEkNav for Kalman filtering [[9]] or Vrgimbal for 3 axes - control[[10]] may help saving a lot of time. AGL apps should access CAN - signals through High Level service. High level can lean on as many low level - service as needed to compute its **Virtual signals** coming from differents - sources. Viwi protocol seems to be a good solution. - -## Leveraging AGL binder - -Such a model is loosely coupled with AGL binder. Low level CAN service as well -as virtual signal components may potentially run within any hosting environment -that would provide the right API with corresponding required facilities. -Nevertheless leveraging [[APbinder]] has multiple advantages. It already -implements event notification to support a messaging/signaling model for -distributed services. It enables a subscribe model responding to the -requirement and finally it uses JSON natively. - -This messaging/signalling model already enforces the notion of subscription for -receiving data. It implies that unexpected data are not sent and merely not -computed. When expected data is available, it is pushed to all waiting -subscriber only one time. - -The [[APbinder]] provides transparency of communication. -It currently implements the transparency over D-Bus/Kdbus and WebSocket. -Its transparency mechanism of communication is easy to extend to other -technologies: pools of shared memory or any proprietary transport model. - -When bindings/services are loaded by the same binder, it provides transparently -`in-memory` communication. This in-memory communication is really efficient: on -one hand, the exchanged JSON objects are not serialized (because not streamed), -on the other hand, those JSON objects provide a high level of abstraction able -to transfer any data. - -Technically a service is a standard [[APbinder]] binding which is also handled -by the system and launched as a daemon by systemD. -Therefore Signal/Agent inherits of security protection through SMACK, access -control through Cynara, transparency of API to transport layer, life cycle -management, ... Like any other [[APbinder]] process is composed of a set of -bindings. In signal service specific case, those bindings are in fact the -`signal modules`. - -The proposed model allows to implement low level dependencies as independent -signal modules. Those modules when developed are somehow like "Lego Bricks". -They can be spread or grouped within one or multiple services depending on -deployment constraints (performance, multi-ECU, security & isolation -constraints,...). - -On top of that low level signal modules, you should use a high level service. -A first implementation of [[ViWi]] is available [here](https://github.com/iotbzh/high-level-viwi-service) -and can be use to integrate business logic and high level features. - -The model naturally uses JSON to represent data. - -## Multi-ECU and Vehicule to Cloud interactions - -While this might not be a show stopper for current projects, it is obvious that -in the near future Signal/Agent should support a fully distributed -architectures. Some event may come from the cloud (i.e. request to start -monitoring a given feature), some may come from SmartCity and nearby vehicles, -and last but not least some may come from another ECU within the same vehicle or -from a virtualized OS within the same ECU (e.g. cluster & IVI). In order to do -so, Signal modules should enable composition within one or more [[APbinder]] -inside the same ECU. Furthermore they should also support chaining with the -outside world. - -![image](images/cloud-arch.svg "Cloud & Multi-ECU Architecture") - -1. Application requests Virtual Signal exactly like if it was a low level signal -1. Agent Signal has direct relation to low level signal -1. Agent needs to proxy to an other service inside the same ECU to access the signal -1. Signal is not present on current ECU. Request has to be proxied to the outside world - -[AppFw]: http://iot.bzh/download/public/2016/appfw/01_Introduction-to-AppFW-for-AGL-1.0.pdf "Application Framework" -[APcore]: http://iot.bzh/download/public/2016/appfw/03_Documentation-AppFW-Core-1.0.pdf "AppFw Core" -[APmain]: https://gerrit.automotivelinux.org/gerrit/#/q/project:src/app-framework-main "AppFw Main" -[APbinder]: https://gerrit.automotivelinux.org/gerrit/#/q/project:src/app-framework-binder "AppFw Binder" -[APsamples]: https://gerrit.automotivelinux.org/gerrit/gitweb?p=src/app-framework-binder.git;a=tree;f=bindings/samples "AppFw Samples" -[Signal-K]: http://signalk.org/overview.html -[1]: http://schd.ws/hosted_files/aglmmwinter2017/37/20170201_AGL-AMM_F10_kusakabe.pdf -[2]: https://wiki.automotivelinux.org/_media/agl-distro/20170402_ften_can_kusakabe_v2.pdf -[6]: https://github.com/otcshare/automotive-message-broker -[7]: http://ardupilot.org/rover/index.html -[8]: https://github.com/ArduPilot/ardupilot/tree/master/libraries -[9]: https://bitbucket.org/jbrandmeyer/libeknav/wiki/Home -[10]: http://ardupilot.org/rover/docs/common-vrgimbal.html -[11]: http://elinux.org/R-Car/Boards/Porter:PEXT01 -[12]: https://github.com/gpsnavi/gpsnavi -[VISS]: http://rawgit.com/w3c/automotive/gh-pages/vehicle_data/vehicle_information_service.html -[VSS]: https://github.com/GENIVI/vehicle_signal_specification -[ViWi]: https://www.w3.org/Submission/2016/SUBM-viwi-protocol-20161213/ -[OpenXC]: http://openxcplatform.com/ -[low level CAN service]: https://gerrit.automotivelinux.org/gerrit/#/admin/projects/src/low-level-can-generator -[high level ViWi]: https://github.com/iotbzh/high-level-viwi-service