python.observation

Environmental Sensing Package

Created on Fri Dec 24 15:21:14 2021

@author: philippe@loco-labs.io

Why a project for Environmental Data ?

The project was born from the following observations:

  • there is no standard format (apart from the Environmental Sensing Service Bluetooth) used by the sensors to transmit the information (binary and textual),
  • there is no data exchange format presenting at the same time a temporal, spatial and physical component (apart from file formats),
  • the main standards used to exchange data are CSV file or JSON object. These two standards are not suitable and not optimized for complex data,
  • we spend a lot of energy converting this type of data to make it usable.

The Environmental Sensing project

The ES project is made of :

  • A data model that makes it possible to represent elementary observations (a simple one-off measurement), complex observations (multi-dimensions), detailed levels of representation (for example, the evolution of a plume of smoke).
  • Data formats adapted to interfaces (binary payload for networks, json for requests or for NoSQL API, files)
  • structured tools to structure, analyse and optimize data (e.g. control conceptual data model cardinality in a dataset)
  • A library of connectors for different uses (sensors, database, storage, networks, etc.) in different languages (python, C++)
  • Bidirectional interfaces to data processing tools (eg Numpy, Xarray, GIS).

It allows to :

  • accelerate standards convergence
  • Facilitate the use and sharing of environmental data
  • Standardize both data acquisition equipment (sensors) and processing applications,
  • Implement a software architecture replacing all coding / decoding operations (interfaces) by the use of standard connectors,
  • Respect and rely on the main existing standards
  • Collectively share and develop a set of open-source connectors responding to all situations (platform)

Examples of achievements

  • Bluetooth extension for Air Pollutants (available in sept-21)
  • Add 'relationship' property in TableSchema (proposal)
  • Development of 'indexed list' theory to deal with complex datasets (available)
  • standard data exchange format suitable for complex data sets (available, data size divided by 5 to 10)
  • data interoperability connectors (available in python since july-22)

Documentation

Documentation is available in other pages :

Modules contain the following classes:

  1# -*- coding: utf-8 -*-
  2"""
  3***Environmental Sensing Package***
  4
  5Created on Fri Dec 24 15:21:14 2021
  6
  7@author: philippe@loco-labs.io
  8
  9
 10# Why a project for Environmental Data ?
 11
 12The project was born from the following observations:
 13
 14- there is no standard format (apart from the Environmental Sensing Service Bluetooth)
 15used by the sensors to transmit the information (binary and textual),
 16- there is no data exchange format presenting at the same time a temporal,
 17spatial and physical component (apart from file formats),
 18- the main standards used to exchange data are CSV file or JSON object. These two
 19 standards are not suitable and not optimized for complex data,
 20- we spend a lot of energy converting this type of data to make it usable.
 21
 22# The Environmental Sensing project
 23
 24The [ES project](https://github.com/loco-philippe/Environmental-Sensing#readme) is made of :
 25
 26- A data model that makes it possible to represent elementary observations
 27(a simple one-off measurement), complex observations (multi-dimensions),
 28detailed levels of representation (for example, the evolution of a plume of smoke).
 29- Data formats adapted to interfaces (binary payload for networks, json for requests
 30or for NoSQL API, files)
 31- structured tools to structure, analyse and optimize data (e.g. control conceptual data
 32model cardinality in a dataset)
 33- A library of connectors for different uses (sensors, database, storage, networks, etc.)
 34in different languages (python, C++)
 35- Bidirectional interfaces to data processing tools (eg Numpy, Xarray, GIS).
 36
 37It allows to :
 38
 39- accelerate standards convergence
 40- Facilitate the use and sharing of environmental data
 41- Standardize both data acquisition equipment (sensors) and processing applications,
 42- Implement a software architecture replacing all coding / decoding operations
 43(interfaces) by the use of standard connectors,
 44- Respect and rely on the main existing standards
 45- Collectively share and develop a set of open-source connectors responding to
 46all situations (platform)
 47
 48# Examples of achievements
 49
 50- Bluetooth extension for Air Pollutants (available in sept-21)
 51- Add 'relationship' property in TableSchema (proposal)
 52- Development of 'indexed list' theory to deal with complex datasets (available)
 53- standard data exchange format suitable for complex data sets
 54(available, data size divided by 5 to 10)
 55- data interoperability connectors (available in python since july-22)
 56
 57# Documentation
 58
 59Documentation is available in other pages :
 60
 61- The concepts of 'observation', 'indexed list' and 'ES value' are describe in
 62[the wiki](https://github.com/loco-philippe/Environmental-Sensing/wiki) and in
 63[the presentation](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/Ilist_principles.pdf).
 64- The non-regression tests are at
 65[this page](https://github.com/loco-philippe/Environmental-Sensing/tree/main/python/Tests)
 66- Examples are
 67[here](https://github.com/loco-philippe/Environmental-Sensing/tree/main/python/Examples)
 68- data exchange standard for [observation](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/ObsJSON-Standard.pdf),
 69[indexed list](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/IlistJSON-Standard.pdf) and
 70[values](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/ESJSON-Standard.pdf)
 71
 72Modules contain the following classes:
 73
 74- Observation :
 75
 76    - `python.observation.esobservation`
 77
 78- ESValue :
 79
 80    - `python.observation.esvalue`(`python.observation.esvalue.DatationValue`,
 81    - `python.observation.esvalue.LocationValue`,
 82    - `python.observation.esvalue.PropertyValue`, `python.observation.esvalue.NamedValue`,
 83    - `python.observation.esvalue.ExternValue`, `python.observation.esvalue_base.ESValue`)
 84
 85- Datasets :
 86
 87    - `python.observation.datasets.Sdataset`, 
 88    - `python.observation.datasets.Ndataset`, 
 89
 90- Dataset (abstract classes):
 91
 92    - `python.observation.dataset` , 
 93    - `python.observation.dataset_structure`, 
 94    - `python.observation.dataset_interface`
 95    - `python.observation.dataset_analysis`
 96
 97- Field :
 98
 99    - `python.observation.field`, `python.observation.ntvfieldructure`, 
100    `python.observation.field_interface`
101
102- TimeSlot :
103
104    - `python.observation.timeslot`
105
106- ES :
107
108    - `python.observation.esconstante`.
109"""
110from observation.esobservation import Observation
111from observation.esvalue import NamedValue, DatationValue, LocationValue, PropertyValue, ExternValue
112from observation.esvalue_base import ESValue
113from observation.dataset import Dataset
114from observation.dataset_interface import DatasetInterface, DatasetError
115from observation.dataset_structure import DatasetStructure
116from observation.field import Field
117from observation.field_structure import FieldStructure
118from observation.field_interface import CborDecoder, FieldError, FieldEncoder, FieldInterface
119from observation.fields import Nfield, Sfield
120from observation.datasets import Ndataset, Sdataset
121#from observation.essearch import ESSearch
122from observation.esconstante import ES, Es, _classval
123from observation.util import util
124from observation.timeslot import TimeSlot
125from observation.dataset_analysis import Analysis
126#print('package :', __package__)