Deleting sensors and observations

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Deleting sensors and observations

danji_ma90
Hello,
I have some questions on how to delete sensors with their corresponding observations.

I have been using the 52°North SOS Test Client function GetCapabilities and the database in postgresql in order to monitor anything I change, and still have difficulties understanding how the test client interacts with the database.

I tried deleting sensors using the DeleteSensor function in the client:

REQUEST:
<?xml version="1.0" encoding="UTF-8"?>
<swes:DeleteSensor
    xmlns:swes="http://www.opengis.net/swes/2.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" service="SOS" version="2.0.0" xsi:schemaLocation="http://www.opengis.net/swes/2.0 http://schemas.opengis.net/swes/2.0/swes.xsd">
    <swes:procedure>My_Sensor</swes:procedure>
</swes:DeleteSensor>

RESPONSE:
<?xml version="1.0" encoding="UTF-8"?>
<swes:DeleteSensorResponse xmlns:swes="http://www.opengis.net/swes/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/swes/2.0 http://schemas.opengis.net/swes/2.0/swesDeleteSensor.xsd">
  <swes:deletedProcedure>My_Sensor</swes:deletedProcedure>
</swes:DeleteSensorResponse>

The result was that the procedure My_Sensor was deleted on the "procedures" section when getting the capabilities (GetCapabilities) in the test client, in the postgres database the character in the "deleted" column of the "procedures" table was set to "T". I then used the "Delete deleted observations" in the datasource maintenance test client, hoping that would wipe the db clean of all the observations with deleted sensors. Instead, the test client says "Error! Request failed: 0 error" after 30 seconds and the server crashes. The database remains untouched and once rebooting the server, My_Sensor reappears in the procedures when getting the capabilities (GetCapabilities).

Why can't SOS handle deleting deleted records? Am I using the wrong functions?
Is there a more effective way of clearing data from the SOS db?

Thanks for the help
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

danji_ma90
I've now managed do delete deleted observations by increasing the RAM dedicated to the Apache Tomcat to 500Mb. This deleted the observations in the DB linked to the procedure My_Sensor.

Is there a way to completely delete previously inserted sensors? All attempts tried (DeleteSensor in the test client, deleting the procedure description in the web client/admin/settings/procedure descriptions) just set the "deleted" table to true "T". On the client the deleted sensors/procedures do get deleted at first (don't show up), but reappear when the server is rebooted or reloading the capabilities cache.

For our SOS we need a function to COMPLETELY delete specific sensors and all the related observations. Is there such a function?

Thanks for the help
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

Carsten Hollmann
Hi,

> On the client the deleted
> sensors/procedures do get deleted at first (don't show up), but reappear
> when the server is rebooted or reloading the capabilities cache.

There was a bug [0] in the SOS which will be fixed in the next release.

> For our SOS we need a function to COMPLETELY delete specific sensors and all
> the related observations. Is there such a function?

No, only the observations which are marked as deleted would be deleted
in the database if you execute the "Delete deleted observations" function.

Best,
Carsten

[0] https://github.com/52North/SOS/issues/446
_______________________________________________
SWE mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/swe
http://sensorweb.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

danji_ma90
Hi Carsten,
thanks for your response.
1)Are you saying there is NO way to delete a sensor from the database (apart from clearing the datasource) once you inserted it with InsertSensor?

2)As an alternative to deleting the sensor, is there maybe a way to modify certain sections of a registered sensor, such as naming conventions, position (coordinates), observable properties...? I have tried this with the UpdateSensorDescription function in the test client. I used the following XML script to change the name of the sensor from "Meteo_Beratungsring_GS_L0_Nals2OberauCarli" to "Meteo_Nals2OberauCarli". If I understood correctly the script calls the existing sensor in the  <swes:procedure> tag. I pasted the <sml:PhysicalSystem tag from the original InsertSensor script, but modified gml:id="Meteo_Nals2OberauCarli" and the identifier.

REQUEST:
<?xml version="1.0" encoding="UTF-8"?>
<swes:UpdateSensorDescription
    xmlns:swes="http://www.opengis.net/swes/2.0"
    xmlns:sos="http://www.opengis.net/sos/2.0"
    xmlns:swe="http://www.opengis.net/swe/2.0"
    xmlns:sml="http://www.opengis.net/sensorml/2.0"
    xmlns:gml="http://www.opengis.net/gml/3.2"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:gco="http://www.isotc211.org/2005/gco"
    xmlns:gmd="http://www.isotc211.org/2005/gmd" service="SOS" version="2.0.0" xsi:schemaLocation="http://www.opengis.net/swes/2.0 http://schemas.opengis.net/swes/2.0/swes.xsd">
    <swes:procedure>Meteo_Beratungsring_GS_L0_Nals2OberauCarli</swes:procedure>
    <swes:procedureDescriptionFormat>http://www.opengis.net/sensorml/2.0</swes:procedureDescriptionFormat>
    <swes:description>
        <swes:SensorDescription>
            <swes:data>
                <sml:PhysicalSystem gml:id="Meteo_Nals2OberauCarli">
                  <gml:identifier codeSpace="uniqueID">Meteo_Nals2OberauCarli</gml:identifier>
                  <sml:identification>
                    <sml:IdentifierList>
                      <sml:identifier>
                        <sml:Term definition="urn:ogc:def:identifier:OGC:1.0:longName">
                          <sml:label>longName</sml:label>
                          <sml:value>Meteo station of the Beratungsring Association - Nals2OberauCarli</sml:value>
                        </sml:Term>
                      </sml:identifier>
                      <sml:identifier>
                        <sml:Term definition="urn:ogc:def:identifier:OGC:1.0:shortName">
                          <sml:label>shortName</sml:label>
                          <sml:value>Meteo Nals2OberauCarli</sml:value>
                        </sml:Term>
                      </sml:identifier>
                    </sml:IdentifierList>
                  </sml:identification>
                  <sml:capabilities name="offerings">
                    <sml:CapabilityList>
                      <sml:capability name="offeringID">
                        <swe:Text definition="urn:ogc:def:identifier:OGC:offeringID">
                          <swe:label>Meteo_Nals2OberauCarli</swe:label>
                          <swe:value>Meteo_Nals2OberauCarli</swe:value>
                        </swe:Text>
                      </sml:capability>
                    </sml:CapabilityList>
                  </sml:capabilities>
                  <sml:featuresOfInterest>
                    <sml:FeatureList
                      definition="http://www.opengis.net/def/featureOfInterest/identifier">
                      <swe:label>featuresOfInterest</swe:label>
                      <sml:feature
                        xlink:href="http://sdcgbe01.eurac.edu:8083/52n-sos-webapp/service/rest/sensors/Nals2OberauCarli" />
                    </sml:FeatureList>
                  </sml:featuresOfInterest>
                  <sml:inputs>
                    <sml:InputList>
                      <sml:input name="Air_Humidity">
                        <sml:ObservableProperty definition="Air_Humidity" />
                      </sml:input>
                      <sml:input name="Air_Temperature">
                        <sml:ObservableProperty definition="Air_Temperature" />
                      </sml:input>
                      <sml:input name="Precipitation">
                        <sml:ObservableProperty definition="Precipitation" />
                      </sml:input>
                      <sml:input name="Wind_Speed">
                        <sml:ObservableProperty definition="Wind_Speed" />
                      </sml:input>
                      <sml:input name="Irrigation">
                        <sml:ObservableProperty definition="Irrigation" />
                      </sml:input>
                    </sml:InputList>
                  </sml:inputs>
                  <sml:outputs>
                    <sml:OutputList>
                      <sml:output name="AH_5min_avg">
                        <swe:Category definition="Air_Humidity_5min_Average">
                          <swe:codeSpace xlink:href="NOT_DEFINED" />
                        </swe:Category>
                      </sml:output>
                      <sml:output name="AT_5min_avg">
                        <swe:Category definition="Air_Temperature_5min_Average">
                          <swe:codeSpace xlink:href="NOT_DEFINED" />
                        </swe:Category>
                      </sml:output>
                      <sml:output name="P_5min_sum">
                        <swe:Category definition="Precipitation_5min_Sum">
                          <swe:codeSpace xlink:href="NOT_DEFINED" />
                        </swe:Category>
                      </sml:output>
                      <sml:output name="I_5min_sum">
                        <swe:Category definition="Irrigation_5min_Sum">
                          <swe:codeSpace xlink:href="NOT_DEFINED" />
                        </swe:Category>
                      </sml:output>
                      <sml:output name="WS_5min_avg">
                        <swe:Category definition="Wind_Speed_5min_Average">
                          <swe:codeSpace xlink:href="NOT_DEFINED" />
                        </swe:Category>
                      </sml:output>                                       
                      <sml:output name="I_5min_bol">
                        <swe:Category definition="Irrigation_5min_Bol">
                          <swe:codeSpace xlink:href="NOT_DEFINED" />
                        </swe:Category>
                      </sml:output>
                    </sml:OutputList>
                  </sml:outputs>
                  <sml:position>
                    <swe:Vector referenceFrame="urn:ogc:def:crs:EPSG::4326">
                      <swe:coordinate name="easting">
                        <swe:Quantity axisID="x">
                          <swe:uom code="degree" />
                          <swe:value>11.2057040</swe:value>
                        </swe:Quantity>
                      </swe:coordinate>
                      <swe:coordinate name="northing">
                        <swe:Quantity axisID="y">
                          <swe:uom code="degree" />
                          <swe:value>46.5601320</swe:value>
                        </swe:Quantity>
                      </swe:coordinate>
                      <swe:coordinate name="altitude">
                        <swe:Quantity axisID="z">
                          <swe:uom code="m" />
                          <swe:value>257</swe:value>
                        </swe:Quantity>
                      </swe:coordinate>
                    </swe:Vector>
                  </sml:position>                       
                </sml:PhysicalSystem>
            </swes:data>
        </swes:SensorDescription>
    </swes:description>
</swes:UpdateSensorDescription>


RESPONSE:

<?xml version="1.0" encoding="UTF-8"?>
<swes:UpdateSensorDescriptionResponse xmlns:swes="http://www.opengis.net/swes/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/swes/2.0 http://schemas.opengis.net/swes/2.0/swesUpdateSensorDescription.xsd">
  <swes:updatedProcedure>Meteo_Beratungsring_GS_L0_Nals2OberauCarli</swes:updatedProcedure>
</swes:UpdateSensorDescriptionResponse>


The client seems to accept the request, but does not change anything.
3)Do you have any clue on what I am doing wrong?

Being able to modify and delete sensors is fundamental to our project, as sensors are constantly updated, replaced or removed.

Thanks for the help

Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

Carsten Hollmann
Hi Daniel,

1) Yes, that is right. There is no way to delete a sensor physically
from the database via the SOS. The DeleteSensor operation only sets a
deleted flag and the sensor is no longer shown in the capabilities and
you can not insert observations for the sensor.

2) You can modify parts of the sensor description and update the sensor
description in the SOS with the UpdateSensorDescription operation. For
example you can change the position (coordinates) but you can not change
the identifier of the sensor or add observable properties so you can
then insert data for the added observable property. As the name of the
UpdateSensorDescription operation says, it updates the sensor
description not the inserted sensor.

Yes, the operation calls the sensor in the <swes:procedure> tag and
updates its sensor description. The gml:identifier is the that would be
used in the SOS queries and the same as you have set in the
<swes:procedure> tag in your UpdateSensorDescription. This identifier is
unique in the SOS and you can not change it.

3) You're not doing anything wrong but the OGC SOS 2.0 standard does not
define an operation to update a sensor/procedure, for example to add or
remove the observed properties the sensor creates observations for.

You use case/project sounds interesting and it would be great to get
some more information about this. If you do not want to send the details
of your project via the mailing list you can also contact me directly.

Perhaps there are other ways to map your data to the SOS which fit
better to your requirements. For example the position/location can be
modelled as the featureOfInterest you can use the samplingGeometry (OGC
SOS 2.0 Spatial Filtering Profile).

Best,
Carsten
_______________________________________________
SWE mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/swe
http://sensorweb.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

and.vianello
Hi Carsten,
I'm working with Daniel and other colleagues in SOS for the MONALISA project: http://www.monalisa-project.eu/en/home/Pages/default.aspx


At the moment we have about 70 sensors in about 30 different locations and we also have data arriving from a chemical laboratory. The number of sensor is going to increase a little bit in few time.
At the moment we have 30 millions records and we need to maintain and update this dataset for a long time.
We have environmental and meteo station plus this laboratory results that we modeled as a normal sensor at the moment, may be you have a good suggestions.
I read about the possibility to use the Specimen model but I didn't find good example on it, we will continue to test it in future.

I attached a get_capabilities xml file if you want to have a look, it's just a temporary version, we are going to modify sensors soon to better describe it.

The dataset will be accessible to the project partners and just a small dataset will be accessible to everybody (probably a month dataset ) until partners publish research results.

We used the sos-feeder to insert data and now we are using some python scripts that are faster.

Finally we are testing an xslt file to create a metadata in Geonetwork starting from SOS describe-sensor xml file. In this way we can have sensor metadata in our metadata catalogue.

I have other questions but i will use  a new threads.
Thank you and if you have questions just ask to me or Daniel.

Andrea.MONALISA_capabilities.xml
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

danji_ma90
In reply to this post by Carsten Hollmann
Hi Carsten,
thank you very much for your detailed response, this was already very helpful. Since you are interested in our project, I could send you some PDF files regarding the extent of our sensor network and the SOS implementation we are using. Andrea Vianello, my colleague working on our SOS, will also enter this thread and will give you some details.

Regarding deleting or updating sensors I wanted to present an example issue in our SOS database, so maybe you can direct us to the best solution:

We have inserted a sensor using InsertSensor in the test client and have imported n observations of the year 2015 with the SOS feeder. In 2016, that same sensor was adjusted on site and n new parameters (observable properties) added, therefore n columns added to the csv files it generates. You say OGC SOS standards have no operations to modify or add observable properties, so the obvious and "cleanest" choice would be deleting the sensor and reinserting it with the updated parameters. But using DeleteSensor won't delete the sensor from the postgres db, it only sets the "deleted" column to "T". This means the sensor can't be reinserted with the same name and same offering, as this violates the SOS db constraints. We would have to create a new sensor with a different name and offering, which is not a sustainable way to maintain our database (it would fill our procedure and offering tables with loads of pointless rows and end up very confusing if done multiple times). Clearing the datasource completely is not an option either, as this means wiping our db clean and inserting over 70 sensors and 30 million records all over again, and this every time we change a single sensor. There must be a more effective way of handling this problem. 1) Do you maybe have any ideas?

2) You say UpdateSensorDescription only updates the sensor description. I used the DescribeSensor operator in the test client and found that the <swes:SensorDescription> tag includes most of the xml script, including the <gml:identifier>, <sml:outputs> (observable properties) and <sml:capabilities name="offerings"> tag. You will find test.xml attached to this thread to see for yourself. Why can't I update these details?

test.xml

Thanks again for your help,

kind regards,

Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

Carsten Hollmann
In reply to this post by and.vianello
Hi Andrea,

thanks a lot for the information about your project.

> We have environmental and meteo station plus this laboratory results that we
> modeled as a normal sensor at the moment, may be you have a good
> suggestions.

Instead of a System you can model the laboratory results as ProcessModel
(SensorML 1.0.1) or SimpleProcess (SensorML 2.0).

> I read about the possibility to use the Specimen model but I didn't find
> good example on it, we will continue to test it in future.

The SF Specimen is an option to describe the laboratory process but it
can also be used for the definition of the featureOfInterest.
In the current development branch [0] we have implemented the
SF_Specimen support as FeatureOfInterest.

> The dataset will be accessible to the project partners and just a small
> dataset will be accessible to everybody (probably a month dataset ) until
> partners publish research results.

I would be great if you could provide the link if the publicly
accessible datasets are available.

Best,
Carsten
_______________________________________________
SWE mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/swe
http://sensorweb.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines
Reply | Threaded
Open this post in threaded view
|

Re: Deleting sensors and observations

Carsten Hollmann
In reply to this post by danji_ma90
Hi Daniel,

> Since you are interested in our project, I could send you some PDF
> files regarding the extent of our sensor network and the SOS implementation
> we are using. Andrea Vianello, my colleague working on our SOS, will also
> enter this thread and will give you some details.

That would be great if you could send me some PDF files about your
network and SOS implementation.

> Regarding deleting or updating sensors I wanted to present an example issue
> in our SOS database, so maybe you can direct us to the best solution:
>
> We have inserted a sensor using InsertSensor in the test client and have
> imported n observations of the year 2015 with the SOS feeder. In 2016, that
> same sensor was adjusted on site and n new parameters (observable
> properties) added, therefore n columns added to the csv files it generates.
> You say OGC SOS standards have no operations to modify or add observable
> properties, so the obvious and "cleanest" choice would be deleting the
> sensor and reinserting it with the updated parameters. But using
> DeleteSensor won't delete the sensor from the postgres db, it only sets the
> "deleted" column to "T". This means the sensor can't be reinserted with the
> same name and same offering, as this violates the SOS db constraints.

No, the 52N SOS supports the reinsertion of the sensor with the same
identifier and offering. But in that case the observations which were
inserted before the deletion of the sensor are not accessible because
they are marked as deleted.

> We would have to create a new sensor with a different name and offering, which
> is not a sustainable way to maintain our database (it would fill our
> procedure and offering tables with loads of pointless rows and end up very
> confusing if done multiple times). Clearing the datasource completely is not
> an option either, as this means wiping our db clean and inserting over 70
> sensors and 30 million records all over again, and this every time we change
> a single sensor. There must be a more effective way of handling this
> problem. 1) Do you maybe have any ideas?

A possibility would be to use the hierarchical procedure feature of the
52N SOS. You insert the station as a PhysicalSystem (parent) and then
insert the parameters as PhysicalComponent (childs) into the SOS with
linkage to the station (parent) by using the "attachedTo" element in the
PhysicalComponent.

Here you find some more information about the hierarchical procedure and
how to use it:

https://wiki.52north.org/SensorWeb/HierarchicalProcedure4x

> 2) You say UpdateSensorDescription only updates the sensor description. I
> used the DescribeSensor operator in the test client and found that the
> <swes:SensorDescription> tag includes most of the xml script, including the
> <gml:identifier>, <sml:outputs> (observable properties) and
> <sml:capabilities name="offerings"> tag. You will find test.xml attached to
> this thread to see for yourself. Why can't I update these details?

As you said, the UpdateSensorDescription only updates the sensor
description and no values in the database. The elements you mentioned
contains information which can be used as parameter in the SOS queries.

If the values in these elements are not the same as than those provided
by the SOS, his might be lead to problems with clients that uses the
sensor description to create SOS queries.

For example, Andrea mentioned that you test to harvest metadata from the
sensor descriptions in your metadata catalog. If now a client uses the
metadata from the catalog to query the SOS and the parameter are not the
parameter provided by the SOS, the requests would throw exceptions.

Therefore the SOS sets the values with those he provides.

Best,
Carsten
_______________________________________________
SWE mailing list
[hidden email]
http://list.52north.org/mailman/listinfo/swe
http://sensorweb.forum.52north.org
Please respect our mailing list guidelines:
http://52north.org/resources/mailing-lists-and-forums/guidelines