Using D-ISP as an Event Sourcing Data Store
First Some Background
D-ISP (Dataparency® Information Sharing Platform) is a patented, Data Handling platform, offered (in pre-alpha) by Dataparency, LLC, conforming to the Data Custodian Model. This model prescribes the methods of access control of an Entity’s (user, device, app, etc.) data. It accomplishes this through use of a RDID (Relationship Distributed Identifier) assigned to an authorized relationship between two parties (Entities), the Target and the Requestor. This authorization can come from the Target entity itself or from a proxy Entity assigned to control Entity data.
D-ISP is implemented as a Key-Value Data Store of data bound to an individual Entity by RDID. Data is stored in a Path (typically Domain/Entity-type/RDID/collection) as a CHF (Common Hierarchical Format) document. A Data Storage Request (POST only) is parsed by format specific parsers; currently JSON, in the future YAML, CSV, XML, and others, which convert input data into an ordered array of nodes, each defining a ‘tag’ and optionally a ‘value’ and Role and Group access restrictions. These tags and values are entered into a dictionary which is used for fast query lookups and stored in the document’s CHF. A unique Document Identifier (DocID) is created and stored with the document CHF and returned as the response of a storage request.
Documents under a specific path are stored in a DocList Document, a LIFO array of Document Identifiers and their Meta Data under a Path, and further entered into an Entity Document Path document under the Path specified in the storage request. As new CHF documents are created for each storage request and entered in the DocList, this DocList becomes the history (event) source. Document query requests will traverse and return the documents in this DocList matching the query request. Optionally, the documents can be retrieved as ‘latest’, ‘first’, or from a specific entry time (with 100 nanosecond precision). In addition, the query can specify key (tag), value, substring, key/value, key/substring from the resultant nodes. This adds fast query functions not available in other event storage databases.
As the DocList is bound to an individual entity by Path, it is possible to have an event source for each Collection, in the Path, of Entity data. This is part of standard processing and requires no further definition. This is also something not found in most event storage databases.
Using D-ISP Event Sourcing – Why?
OK, now that you have some background into how D-ISP provides an Event Sourcing Model, let’s delve deeper into how this functionality provides exceptional flexibility in Event Sourcing.
Typically, Event Storage/Sourcing (EvS from hereon) systems store state changes as Events in an immutable, append-only log. Current state can be built up from traversal of this log. State at a specific time can be built by reading the log up to a time. Queries can only be made by Event ID assigned when entered. Queries for other values must traverse the entire log, searching for matches, first entry to last. This can be an exhausting and resource demanding process.
“Edge cloud technology has the potential to bring smart technology to nearly every sector. For the model to reach its full potential, new networking requirements at the edge data center must be addressed.”
D-ISP EvS overcomes many of the restrictions of other EvS in the following ways:
- EvS is inherent in normal D-ISP processing. It requires no configuration, definitions, or apriori setup.
- EvS queries operate as normal D-ISP queries. Again with no apriori (in advance) setup. This provides complete query functionality to your applications.
- Events easily retrieved within a time period. Last and First entries are immediate and direct.
- Events can be stored with an expiration time after which they are deleted (DELETE operation not implemented or allowed). This allows deletion of events after their value has passed.
- EvS(s) bound to a specific Path automatically. Again, no setup required.
- Queries between static data and event data are handled in the same way.
- D-ISP can be enabled in a D-DDN (Dataparency Distributed Data Network) providing distributed and auto-synchronized and backed up data. D-DDN provides a Network of data services distributed to multiple D-ISP servers for scalability.
- Data servers are bound to queue ids in D-DDN. Each queue id relates to a particular Geo/Enterprise, domain/Entity, and binds the D-ISP server and its data to it. This offers GDPR and Enterprise compliance not typically found in other EvS. Entity registration process binds an Entity to the queue id of the servers relative to the jurisdiction to which it belongs.
Using D-ISP Event Sourcing – Examples
Sensor Network Capture and Aggregating Analysis
Here we describe an example of using D-ISP/D-DDN as a Sensor Network Capture and Analysis Platform. We show three stages in the capture and aggregation of sensor data.
Stage One
Figure 1 below shows the infrastructure required to capture and produce the event sourced data. It requires a sensor receiving gateway. This gateway is a program receiving sensor data, building JSON data documents, and storing the documents in D-ISP as events under each sensor Entity. These events are stored, with a specified expiration explained below, in the order-preserving documents in paths relating to the sensor Entity’s RDID with the gateway. These events can be persisted in sequences for as little as one second time periods.
Local analysis can be performed by accessing the gateway’s retrieval functions and accessed through the gateway’s network interface. This data can be retrieved from the D-ISP server by HTTPS REST requests or by an application using D-ISP’s NATS Client library.
Figure 1
Documents created by the Sensor Receiving App are POSTed to the local D-ISP server by making the appropriate request to that server’s request QID #1, say “Gateway.Farmington”. Requests for retrieval of documents can be made from anywhere by using the QID. The NATS server built into the controlling D-ISP server will handle the routing of the request by the QID. Additional ‘worker’ D-ISP servers can be started watching the QID for requests. NATS dispatches requests to any available server. Data changes are automatically updated by posting the updates to a side channel update QID. As posted documents are singular in instance, there is no need for the overhead of a ‘raft’ type of coordination.
Documents posted to D-ISP can have an expiration time after which they are deleted. This can be used to control the lifetime of event documents and provide cleanup of events after they have been aggregated in an update cycle. The update cycle can be used to broadcast the aggregated update to one or more Enterprise QIDs for further analysis.