CityBench, a comprehensive benchmarking suite to evaluate RSP engines within smart city applications and with smart city data. CityBench includes real-time IoT data streams generated from various sensors deployed within the city of Aarhus, Denmark. We provide a configurable testing infrastructure and a set of continuous queries covering a variety of data- and application- dependent characteristics and performance metrics, to be executed over RSP engines using CityBench datasets. We evaluate two state of the art RSP engines using our testbed and discuss our experimental results. This work can be used as a baseline to identify capabilities and limitations of existing RSP engines for smart city applications.
Configurable Testbed Infrastructure (CTI)
CityBench provides a Configurable Testbed Infrastructure (CTI) containing a set of Application Programming Interface (API’s) to set up the testbed environment. CTI allows its users to configure a variety of metrics for the evaluation of RSP engine. Figure shown above, provides an overview of the CTI, there are three main modules,
- Dataset Configuration Module: allows configuration of stream related metrics.
- Query Configuration Module: allows configuration of query related metrics.
- Performance Evaluator: is responsible for recording and storing the measurements of the performance metrics.
CityBench used datasets collected from the city of Aarhus, Denmark. These datasets are collected as part of work conducted in CityPulse project. A prepared version of the datasets ready for CityBench test can be downloaded from CityBench github repository which includes both static and stream datasets for a period between Aug.2014-Sep. 2014. However, a complete list of datasets including future uploads, can be accessed from CityPulse Dataset Collection *
We used following dataset for Citybench
- Vehicle Traffic Dataset (Real Data streams from traffic sensors)
- Parking Dataset (Parking availability streams from parking lot sensors)
- Weather Dataset (weather sensor stream)
- Pollution Dataset (Synthesized)
- Cultural Event Dataset (Quasi-static dataset with update stream)
- Library Event Data (Quasi-static dataset with update stream)
- User Location Stream (synthesized as fictional users roaming around in the city)
We devised 13 quereis for CityBench, these queries are generated while keeping 3 smart city applications in view
- Multi-modal Context-aware Travel Planner
- Parking Space Finder Application
- Smart City Administration Console
CityBench Benchmarking Suite (Source Code)
Source code for all tools accompanying CityBench together with details to set up testbed infrastructure can be download from CityBench Source Code at github. In future, we plan to provide a web interface which will allow CityBench users to choose any supported RSP engine and Queries online and setup test infrastructure by selecting various values for the configuration parameters.
Download and Installation InstructionsCityBench download and installation instructions can be accessed here.
If you notice any issue or having any problems with setting up testbed please report the issue at CityBench github issue tracker.
Contribution to CityBench (Datasets and Queries)
If you have any smart city dataset or queries related to smart city application which you can contribute, please contact the primary contact. In future, we are planning to provide an easy to use interface for the contribution of datasets and queries.
- Muhammad Intizar Ali (primary contact). Email: firstname.lastname@example.org
- Feng Gao. Email: email@example.com
- Alessandra Mileo. Email: firstname.lastname@example.org