NTV.documentation.user_guide

@author: philippe@loco-labs.io

JSON-NTV (named and typed value) : a semantic format for interoperability

JSON-NTV is a universal representation format. It allows the sharing and conversion of any type of data.

The NTV format is part of the Environmental Sensing Project

For more information, see the user guide or the github repository.

What is NTV ?

The semantic level of shared JSON (or CSV) data (e.g. Open Data) remains low, which makes automated reuse difficult.

JSON-NTV proposes to enrich it to obtain a real interoperable exchange format.

NTV format

The NTV format consists of representing data by three attributes: a name, a type and a value. This representation is common in programming languages (for example a variable with Python typing is defined by age: int = 25), however the JSON format represents data with only a value or a key:value pair.

The JSON-NTV extension consists of including the type in the name to associate it with a value (for example {'age:int': 25} is the JSON representation of the NTV triplet ('age', 'int' , 25 ) ).

This approach makes it possible to reversibly represent any simple or complex data by a JSON structure (high interoperability).

Examples

In [1]: from shapely.geometry import Point
        from datetime import date
        from pprint import pprint

In [2]: pprint(Ntv.obj(21).expand())
Out[2]: {'name': '', 'type': 'json', 'value': 21}

In [3]: pprint(Ntv.obj({"paris:point": [2.3, 48.9] }).expand())
Out[3]: {'name': 'paris', 'type': 'point', 'value': [2.3, 48.9]}

In [4]: pprint(Ntv.obj({"cities::point": [[2.3, 48.9], [4.8, 45.8] }).expand())
Out[4]: {'name': 'cities',
         'type': 'point',
         'value': [{'name': '', 'type': 'point', 'value': [2.3, 48.9]},
                   {'name': '', 'type': 'point', 'value': [4.8, 45.8]}]}

In [5]: pprint(Ntv.obj({"paris:point": [2.3, 48.9], "start:date": "2023-08-03", "measurement": 45.8}).expand())
Out[5]: {'name': '',
         'type': '',
         'value': [{'name': 'paris', 'type': 'point', 'value': [2.3, 48.9]},
                   {'name': 'start', 'type': 'date', 'value': '2023-08-03'},
                   {'name': 'measurement', 'type': 'json', 'value': 45.8}]}

Note: This typing syntax can also be used for CSV file headers

NTV structure

With this approach, two NTV entities are defined:

a primitive entity which is not composed of any other entity (NTV-single),
a structured entity which is an ordered sequence of NTV entities (NTV-list).

as well as two JSON formats:

simple format when the name and the type are not present (e.g. 25),
named format when the name or type is present ((e.g. {'age': 25} or {':int': 25})).

The type incorporates a notion of namespace that can be nested.

For example, the type: ns1.ns2.type_a means that:

ns1. is a namespace defined in the global namespace,

ns2. is a namespace defined in the ns1. namespace.,

type_a is defined in the ns2. namespace.

The type can be extended with additional data.

For example, the type: float[kg] means that:

float is a main type of the data,

kg is an additional type (e.g. an unit),

float[kg] may represent a Quantity.

This structuring of type makes it possible to reference any type of data that has a JSON representation and to consolidate all the shared data structures within the same tree of types.

NTV uses

Several variations and use cases of the NTV format are defined:

Tabular and multidimensional data exchange format (e.g. open-data)
Compact, reversible and semantic pandas-JSON interface
Comment and change management of JSON data
visualization of JSON or NTV tree
JSON data editor
data validation (value conformity to the Datatype)

NTV and JSON

The flowchart below explain how to convert and exchange native entities through NTV and JSON format.

flowchart LR
    text["#10240;#10240;JSON#10240;#10240;\ntext"]
    val["#10240;JSON-NTV#10240;\nvalue"]
    ntv["#10240;#10240;#10240;NTV#10240;#10240;#10240;\nentity"]
    nat["#10240;native#10240;\nentity"]
    text--->|JSON load|val
    val--->|JSON dump|text
    val--->|NTV from JSON|ntv
    ntv--->|from NTV|nat
    ntv--->|NTV to JSON|val
    nat--->|to NTV|ntv

The conversion between native entity and JSON-text is reversible (lossless round trip).

In [6]: loc_and_date = {'newyear': date(2023, 1, 2), 'Paris': Point(2.3, 48.9)}
        json_loc_date = Ntv.obj(loc_and_date).to_obj(encoded=True)
        print(json_loc_date, type(json_loc_date))
Out[6]: {"newyear:date": "2023-01-02", "Paris:point": [2.3, 48.9]} <class 'str'>

In [7]: Ntv.obj(json_loc_date).to_obj(format='obj') == loc_and_date
Out[7]: True

Properties :

each NTV object has a unique JSON representation
each JSON data corresponds to a unique NTV entity
an NTV entity is a tree where each node is an NTV entity and each leaf an NTV-Single entity
an NTV entity is a neutral representation (independent of a software or hardware platform)

package installation

json_ntv package

The json_ntv package includes

ntv module
- Ntv abstract class
- NtvSingle and NtvList child classes
namespace module
- TypeBase, Datatype, Namespace, DatatypeError classes
ntv_util module
- NtvConnector, NtvUtil, NtvTree, NtvJsonEncoder, NtvError classes
ntv_connectormodule (NtvConnector child classes)
- SfieldConnec, SdatasetConnec, NfieldConnec, NdatasetConnec, MermaidConnec, ShapelyConnec, CborConnec classes
ntv_patch module
- NtvOp, NtvPatch, NtvPointer, NtvOpError classes
ntv_validatemodule
- Validator, ValidateErrorclasses

The 'Namespaces' used are defined in the configuration files

Installation

json_ntv itself is a pure Python package. It can be installed with pip

pip install json_ntv

dependency:

only packages associated to used connectors (e.g. Mermaid if we use MermaidConnec)
Examples and Uses

Several variations and use cases of the NTV format are defined:

Overview

This overview presents different facets and covers all the features of this package.

JSON and NTV data editor

This notebook presents with an example the functionality available to access, represent, update and transform JSON (or NTV) data.

Tabular and multidimensional NTV connector

This compact, reversible and semantic converter is defined in a separate package ntv-pandas and ntv-numpy.

Comment and change

TBD

documentation

The documentation presents :

documents

a general presentation
the JSON-NTV specification (Json for NTV data)
an example of namespace
the NTV-TAB specification (Json for tabular data)

Python Connectors documentation

API
- dev
- v1.4.0
- v1.3.0
- v1.2.0
- v1.1.0
- v1.0.1
- v1.0.0
Release
- v0
- v1

Roadmap

NTV schema : use data schema compatible with Json-schema
multidimensional data : specific Internet-Draft for this format
indicators : extract structural data
types : tools to control (NTV_values, NTV_type)
CBOR interface : using CBOR tags
types catalog : compatibility with others catalogs
visualization : add visualization tools (Mermaid extension and other tools)
NTV editor : HTML/JavaScript to build and understand NTV structure and JSON-NTV format
JavaScript : add NTV javascript tool
NTVpath : extension of JSONpath to NTV data

View Source

 1# -*- coding: utf-8 -*-
 2"""
 3@author: philippe@loco-labs.io
 4
 5.. include:: ../README.md
 6.. include:: ../json_ntv/README.md
 7.. include:: ../example/README.md
 8.. include:: ./README.md    
 9
10"""