python.observation
Environmental Sensing Package
Created on Fri Dec 24 15:21:14 2021
@author: philippe@loco-labs.io
Why a project for Environmental Data ?
The project was born from the following observations:
- there is no standard format (apart from the Environmental Sensing Service Bluetooth) used by the sensors to transmit the information (binary and textual),
- there is no data exchange format presenting at the same time a temporal, spatial and physical component (apart from file formats),
- the main standards used to exchange data are CSV file or JSON object. These two standards are not suitable and not optimized for complex data,
- we spend a lot of energy converting this type of data to make it usable.
The Environmental Sensing project
The ES project is made of :
- A data model that makes it possible to represent elementary observations (a simple one-off measurement), complex observations (multi-dimensions), detailed levels of representation (for example, the evolution of a plume of smoke).
- Data formats adapted to interfaces (binary payload for networks, json for requests or for NoSQL API, files)
- structured tools to structure, analyse and optimize data (e.g. control conceptual data model cardinality in a dataset)
- A library of connectors for different uses (sensors, database, storage, networks, etc.) in different languages (python, C++)
- Bidirectional interfaces to data processing tools (eg Numpy, Xarray, GIS).
It allows to :
- accelerate standards convergence
- Facilitate the use and sharing of environmental data
- Standardize both data acquisition equipment (sensors) and processing applications,
- Implement a software architecture replacing all coding / decoding operations (interfaces) by the use of standard connectors,
- Respect and rely on the main existing standards
- Collectively share and develop a set of open-source connectors responding to all situations (platform)
Examples of achievements
- Bluetooth extension for Air Pollutants (available in sept-21)
- Add 'relationship' property in TableSchema (proposal)
- Development of 'indexed list' theory to deal with complex datasets (available)
- standard data exchange format suitable for complex data sets (available, data size divided by 5 to 10)
- data interoperability connectors (available in python since july-22)
Documentation
Documentation is available in other pages :
- The concepts of 'observation', 'indexed list' and 'ES value' are describe in the wiki and in the presentation.
- The non-regression tests are at this page
- Examples are here
- data exchange standard for observation, indexed list and values
Modules contain the following classes:
Observation :
ESValue :
Datasets :
Dataset (abstract classes):
Field :
python.observation.field
,python.observation.ntvfieldructure
,python.observation.field_interface
TimeSlot :
ES :
1# -*- coding: utf-8 -*- 2""" 3***Environmental Sensing Package*** 4 5Created on Fri Dec 24 15:21:14 2021 6 7@author: philippe@loco-labs.io 8 9 10# Why a project for Environmental Data ? 11 12The project was born from the following observations: 13 14- there is no standard format (apart from the Environmental Sensing Service Bluetooth) 15used by the sensors to transmit the information (binary and textual), 16- there is no data exchange format presenting at the same time a temporal, 17spatial and physical component (apart from file formats), 18- the main standards used to exchange data are CSV file or JSON object. These two 19 standards are not suitable and not optimized for complex data, 20- we spend a lot of energy converting this type of data to make it usable. 21 22# The Environmental Sensing project 23 24The [ES project](https://github.com/loco-philippe/Environmental-Sensing#readme) is made of : 25 26- A data model that makes it possible to represent elementary observations 27(a simple one-off measurement), complex observations (multi-dimensions), 28detailed levels of representation (for example, the evolution of a plume of smoke). 29- Data formats adapted to interfaces (binary payload for networks, json for requests 30or for NoSQL API, files) 31- structured tools to structure, analyse and optimize data (e.g. control conceptual data 32model cardinality in a dataset) 33- A library of connectors for different uses (sensors, database, storage, networks, etc.) 34in different languages (python, C++) 35- Bidirectional interfaces to data processing tools (eg Numpy, Xarray, GIS). 36 37It allows to : 38 39- accelerate standards convergence 40- Facilitate the use and sharing of environmental data 41- Standardize both data acquisition equipment (sensors) and processing applications, 42- Implement a software architecture replacing all coding / decoding operations 43(interfaces) by the use of standard connectors, 44- Respect and rely on the main existing standards 45- Collectively share and develop a set of open-source connectors responding to 46all situations (platform) 47 48# Examples of achievements 49 50- Bluetooth extension for Air Pollutants (available in sept-21) 51- Add 'relationship' property in TableSchema (proposal) 52- Development of 'indexed list' theory to deal with complex datasets (available) 53- standard data exchange format suitable for complex data sets 54(available, data size divided by 5 to 10) 55- data interoperability connectors (available in python since july-22) 56 57# Documentation 58 59Documentation is available in other pages : 60 61- The concepts of 'observation', 'indexed list' and 'ES value' are describe in 62[the wiki](https://github.com/loco-philippe/Environmental-Sensing/wiki) and in 63[the presentation](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/Ilist_principles.pdf). 64- The non-regression tests are at 65[this page](https://github.com/loco-philippe/Environmental-Sensing/tree/main/python/Tests) 66- Examples are 67[here](https://github.com/loco-philippe/Environmental-Sensing/tree/main/python/Examples) 68- data exchange standard for [observation](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/ObsJSON-Standard.pdf), 69[indexed list](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/IlistJSON-Standard.pdf) and 70[values](https://github.com/loco-philippe/Environmental-Sensing/tree/main/documentation/ESJSON-Standard.pdf) 71 72Modules contain the following classes: 73 74- Observation : 75 76 - `python.observation.esobservation` 77 78- ESValue : 79 80 - `python.observation.esvalue`(`python.observation.esvalue.DatationValue`, 81 - `python.observation.esvalue.LocationValue`, 82 - `python.observation.esvalue.PropertyValue`, `python.observation.esvalue.NamedValue`, 83 - `python.observation.esvalue.ExternValue`, `python.observation.esvalue_base.ESValue`) 84 85- Datasets : 86 87 - `python.observation.datasets.Sdataset`, 88 - `python.observation.datasets.Ndataset`, 89 90- Dataset (abstract classes): 91 92 - `python.observation.dataset` , 93 - `python.observation.dataset_structure`, 94 - `python.observation.dataset_interface` 95 - `python.observation.dataset_analysis` 96 97- Field : 98 99 - `python.observation.field`, `python.observation.ntvfieldructure`, 100 `python.observation.field_interface` 101 102- TimeSlot : 103 104 - `python.observation.timeslot` 105 106- ES : 107 108 - `python.observation.esconstante`. 109""" 110from observation.esobservation import Observation 111from observation.esvalue import NamedValue, DatationValue, LocationValue, PropertyValue, ExternValue 112from observation.esvalue_base import ESValue 113from observation.dataset import Dataset 114from observation.dataset_interface import DatasetInterface, DatasetError 115from observation.dataset_structure import DatasetStructure 116from observation.field import Field 117from observation.field_structure import FieldStructure 118from observation.field_interface import CborDecoder, FieldError, FieldEncoder, FieldInterface 119from observation.fields import Nfield, Sfield 120from observation.datasets import Ndataset, Sdataset 121#from observation.essearch import ESSearch 122from observation.esconstante import ES, Es, _classval 123from observation.util import util 124from observation.timeslot import TimeSlot 125from observation.dataset_analysis import Analysis 126#print('package :', __package__)