A Configurable Benchmark to Evaluate RSP Engines using Smart City Datasets

CityBench, a comprehensive benchmarking suite to evaluate RSP engines within smart city applications and with smart city data. CityBench includes real-time IoT data streams generated from various sensors deployed within the city of Aarhus, Denmark. We provide a configurable testing infrastructure and a set of continuous queries covering a variety of data- and application- dependent characteristics and performance metrics, to be executed over RSP engines using CityBench datasets. We evaluate two state of the art RSP engines using our testbed and discuss our experimental results. This work can be used as a baseline to identify capabilities and limitations of existing RSP engines for smart city applications.

Configurable Testbed Infrastructure (CTI)

CityBench provides a Configurable Testbed Infrastructure (CTI) containing a set of Application Programming Interface (API’s) to set up the testbed environment. CTI allows its users to configure a variety of metrics for the evaluation of RSP engine. Figure shown above, provides an overview of the CTI, there are three main modules,

  1. Dataset Configuration Module: allows configuration of stream related metrics.
  2. Query Configuration Module: allows configuration of query related metrics.
  3. Performance Evaluator: is responsible for recording and storing the measurements of the performance metrics.

CityBench Datasets

CityBench used datasets collected from the city of Aarhus, Denmark. These datasets are collected as part of work conducted in CityPulse project. A prepared version of the datasets ready for CityBench test can be downloaded from CityBench github repository which includes both static and stream datasets for a period between Aug.2014-Sep. 2014. However, a complete list of datasets including future uploads, can be accessed from CityPulse Dataset Collection *

We used following dataset for Citybench

  • Vehicle Traffic Dataset (Real Data streams from traffic sensors)
  • Parking Dataset (Parking availability streams from parking lot sensors)
  • Weather Dataset (weather sensor stream)
  • Pollution Dataset (Synthesized)
  • Cultural Event Dataset (Quasi-static dataset with update stream)
  • Library Event Data (Quasi-static dataset with update stream)
  • User Location Stream (synthesized as fictional users roaming around in the city)

CityBench Queries

We devised 13 quereis for CityBench, these queries are generated while keeping 3 smart city applications in view

  1. Multi-modal Context-aware Travel Planner
  2. Parking Space Finder Application
  3. Smart City Administration Console
A complete list of queries, their description and syntax ( both for CQELS and C-SPARQL) can be found here.

CityBench Benchmarking Suite (Source Code)

Source code for all tools accompanying CityBench together with details to set up testbed infrastructure can be download from CityBench Source Code at github. In future, we plan to provide a web interface which will allow CityBench users to choose any supported RSP engine and Queries online and setup test infrastructure by selecting various values for the configuration parameters.

Download and Installation Instructions

CityBench download and installation instructions can be accessed here.

Issue Tracker

If you notice any issue or having any problems with setting up testbed please report the issue at CityBench github issue tracker.

Contribution to CityBench (Datasets and Queries)

If you have any smart city dataset or queries related to smart city application which you can contribute, please contact the primary contact. In future, we are planning to provide an easy to use interface for the contribution of datasets and queries.

CityBench Team:

* We acknowledge the CityPulse consortium team for the provision of Datasets . CityBench datasets are made publicly available by EU Project CityPulse, for use of any part of these datasets, the source must be properly acknowledged. The accuracy or reliability of the data is not guaranteed or warranted in any way.