The perfsonar data store, esmond, is accessible via a HTTP REST interface. This makes it straightforward to interface to the archive from other tools.
Each combination of test parameters (called the "test metadata") creates a new archive. So for example, if you change your throughput testing interval or the duration of the test, the subsequent results will be stored in a different archive.
The hardest part is scanning through the available archives and picking the one which has the data you want. After that it is straightforward to retrieve the data.
In your laptop's web browser, go to
http://x.x.x.x/esmond/perfsonar/
Replace x.x.x.x with your perfsonar host.
It will probably say "sorry, not implemented". In that case, change the URL to:
http://x.x.x.x/esmond/perfsonar/?format=json
You should now see a short snippet of JSON giving the top-level resources. It's all on one line, although when reformatted it would look like this:
{
"archive": {
"list_endpoint": "/esmond/perfsonar/archive/",
"schema": "/esmond/perfsonar/archive/schema/"
},
"event-type": {
"list_endpoint": "/esmond/perfsonar/event-type/",
"schema": "/esmond/perfsonar/event-type/schema/"
}
}Now browse to the archive top level:
http://x.x.x.x/esmond/perfsonar/archive/?format=json
On a system with a lot of tests large amount of unformatted JSON will be shown.
Using the CLI you can get the json reformatted to make it easier to read. However there is still a large amount to scroll through.
curl http://localhost/esmond/perfsonar/archive/ | python -mjson.tool | less(You can replace 'localhost' with your perfsonar node's full hostname, or the full hostname of another node)
Create a file called (for example) list-archives.py in the current directory, with the following contents:
#!/usr/bin/python
from __future__ import print_function
import requests
import json
import sys
resp = requests.get('http://' + sys.argv[1] + '/esmond/perfsonar/archive/')
data = json.loads(resp.text)
print('uri', 'source', 'destination', 'tool-name', 'time-interval', 'time-duration', 'time-probe-interval')
for row in data:
print(row['uri'],
row.get('source','-'),
row.get('destination','-'),
row.get('tool-name','-'),
row.get('time-interval','-'),
row.get('time-duration','-'),
row.get('time-probe-interval','-')
)Make the script executable:
chmod +x list-archives.py
./list-archives.py localhost(replace "localhost" with the hostname of a remote measurement host if you wish)
Note: you may need to install the python libraries required. For example, under CentOS:
yum install python-requestsFrom your list of archives, pick the URI of the one you wish to look at - for example one of the iperf3 ones. You will get more options showing the different components of the archives.
curl http://localhost/perfsonar/archive/fa79c5837621460886c0f60fde06072b/ | python -mjson
...
{
"bw-parallel-streams": "1",
"destination": "XXX.XXX.31.1",
"event-types": [
{
"base-uri": "/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/throughput-subintervals/base",
"event-type": "throughput-subintervals",
"summaries": [],
"time-updated": 1441692905
},
{
"base-uri": "/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/failures/base",
"event-type": "failures",
"summaries": [],
"time-updated": null
},
{
"base-uri": "/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/packet-retransmits/base",
"event-type": "packet-retransmits",
"summaries": [],
"time-updated": 1441692905
},
{
"base-uri": "/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/throughput/base",
"event-type": "throughput",
"summaries": [
{
"summary-type": "average",
"summary-window": "86400",
"time-updated": 1441692905,
"uri": "/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/throughput/averages/86400"
}
],
"time-updated": 1441692905
},
{
"base-uri": "/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/packet-retransmits-subintervals/base",
"event-type": "packet-retransmits-subintervals",
"summaries": [],
"time-updated": 1441692905
}
],
"input-destination": "pfsnr.maseno-town-pop.k.kenet.or.ke",
"input-source": "XXX.XXX.25.1",
"ip-transport-protocol": "tcp",
"measurement-agent": "XXX.XXX.25.1",
"metadata-key": "fa79c5837621460886c0f60fde06072b",
"source": "XXX.XXX.25.1",
"subject-type": "point-to-point",
"time-duration": "20",
"time-interval": "14400",
"tool-name": "bwctl/iperf3",
"uri": "/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/"
}Pick one of the sections, e.g. throughput/base and query again
curl http://pfsnr.uon-pop.n.kenet.or.ke/esmond/perfsonar/archive/fa79c5837621460886c0f60fde06072b/throughput/base | python -mjson.tool
...
[
{
"ts": 1441635104,
"val": 72042700.0
},
{
"ts": 1441650420,
"val": 299625000.0
},
{
"ts": 1441664360,
"val": 395284000.0
},
{
"ts": 1441677771,
"val": 418651000.0
},
{
"ts": 1441692883,
"val": 134926000.0
}
]Now you can see the raw timeseries data with timestamp (ts) and value (val).
You can convert the timestamps like this:
# python -c 'import datetime; print datetime.datetime.fromtimestamp(1441692883)'
2015-09-08 09:14:43Or if you have Ruby installed:
ruby -e 'puts Time.at 1441692883'There are additional parameters you can add to limit the results returned. For example, if you are doing a throughput test every hour, the following script pulls out the JSON giving the tests over the last 2.5 hours (i.e. the last 2-3 results)
curl -sS "http://localhost/esmond/perfsonar/archive/...<archiveID>.../throughput/base?time-start=$(date -d "150 minutes ago" +%s)"This in turn is easily consumed by Javascript in a web page if you are building your own dashboard.