p2pkit 2.0 (beta) benchmarking

Justina Kostinaite By Justina Kostinaite

We often get asked what p2pkit is all about and why we think it is a smart alternative to existing “proximity” technologies. To help clear out those questions we decided to run internal tests and evaluate p2pkit on several dimensions Vs. alternative solutions. There was one clear conclusion: p2pkit is the next evolution of proximity technologies.

p2pkit – the next evolution of proximity technologies

Before we plunge in, let’s define the notion of proximity: Proximity occurs when two or more smart devices enter a physical range and their onboard sensors can “sense” each other. For example: when two devices come into Bluetooth range and are able to scan each other we can consider them as being in proximity.

There are various technologies that could be used to derive proximity. The most notable ones are Beacons, GPS, Wi-Fi and Proximity SDKs. p2pkit falls under the latter SDK category. Other technologies such as NFC could also be used to derive proximity, but their range is very limited (by design).

For the sake of simplicity, we decided to only compare proximity technologies that could operate in the range from 40 meters down to a few centimeters. Therefore, we benchmarked p2pkit by Uepaa with other proximity SDKs which rely on onboard radios such as Bluetooth, WiFi, and even voice, namely:

  • Nearby SDK by Google;
  • Proximity Platform SDK by NewAer;
  • The Hype SDK by Hype;
  • Mobile peer-to-peer mesh networking library by Underdark.

We have performed tests with a 20-device group to measure speed, stability, and consistency of the discovery engine over the course of 10 minutes.

We would also like to add that the tests were not performed in a scientific manner and do not represent a “real world” scenario, especially not in terms of server load, background behavior, and radio interference. We included more information in the notes below.

Speed – p2pkit was x2 faster in discovering 80% of the cluster

During our first test, we measured the duration until 80% of devices had been discovered. For example, for 20 devices to reach 80% discovery we calculate ((20*19)*0.8) = 304 discovery events.

The test was performed as follows:

  • 20 nearby devices spread across our office and connected to a custom testbed, running a custom app (in foreground) which receives a signal to “start discovering” nearby devices using the “currently benchmarked” SDK.
  • When a device discovered/lost a nearby device, a log was sent to the testbed.
  • After 10 minutes the test was finished and the results processed.
  • 3 test runs were executed for each SDK, those that have reached 80% discovery had an additional 5 runs (8 in total).

Two out of the five tested SDKs actually never reached 80% discovery. Out of the three SDKs that did reach 80%, p2pkit proved to be the fastest (avg. 5.6s, sd. 1.4, *) – 2x faster than Google Nearby (avg. 11.8s, sd. 4.2, *) and approximately 13x faster than NewAer (avg. 77.1s, sd. 53.7, *, only reached 80% in 3/8 runs).
* 8 test runs

Graph 2: p2pkit ~2x faster than Google Nearby, with the lowest standard deviation at 1.4 seconds. [* 8 test runs; ** 3 runs, never reached 80% discovery]

It’s fair to mention that Hype and Underdark are not necessarily designed as a “proximity” solution, rather as a “mesh network” solution which may be the cause for their decreased performance.

Accuracy & Stability – p2pkit maintains ~99.8% “ground truth”

In addition to speed, we have also measured the SDKs’ abilities to maintain a notion of discovered devices over the course of 10 minutes. We refer to it as “stability”, but you should think of it as “ground truth”. Basically, it means how many devices are still visible after 10 minutes.  p2pkit discovered 100% of nearby devices and proved to be the only SDK to maintain 99.8% stability over the course of 10 minutes with almost no “lost events”. Google Nearby also reached 100% but was only able to maintain 72.8% stability after 10 min, NewAer max. 78.1% with 72.0% stability.

Graph 3: p2pkit discovered 100% of devices at 0% standard deviation [* 8 test runs; ** 3 runs]

Graph 4: p2pkit retained 99.8% “ground truth” at 0.2% standard deviation [* 8 test runs; ** 3 runs]

Consistency – p2pkit performs consistently well!

In terms of consistency, we have observed the standard deviation (SD) of tests results. High SD means the SDK is not performing consistently, low SD means the SDK is very consistent in its performance. This is important since it defines how well an SDK could perform for a certain use case.

Observing the above graphs, it is very clear that p2pkit shines with the lowest SD across all the tests. While other SDKs showcased performance inconsistencies, p2pkit had a 1.4sec SD in the 80% speed test, 0.0% SD in max discovery and 0.2% SD in terms of stability for the selected 8 runs.

Beacons & GPS:

Proximity SDKs such as p2pkit mainly overlap with two other technologies: Beacons and GPS. Below we’ve highlighted some of the advantages over the two of them.

p2pkit vs. beacons

  • The implementation, deployment, maintenance and scalability of p2pkit is more cost effective – p2pkit doesn’t require any hardware deployment or maintenance (i.e: beacons) since most mobile devices already carry onboard all that is needed.
  • p2pkit is truly mobile – unlike beacons which are mostly tailored for indoor use, p2pkit is not bound to any location. p2pkit works equally well stationary and on the move. This is extremely important when considering the cost of constantly deploying beacons at new locations.
  • p2pkit is much easier to prototype with as you don’t need to actually buy any beacons.
  • p2pkit works directly device to device – it does not require any 3rd medium such as a beacon. This enables a variety of features based on signal strength processing such as our own “proximity ranging”.

p2pkit vs. GPS

  • In order to derive accurate and timely proximity, GPS needs to operate at an accurate mode, thus heavily draining the smartphone’s battery – p2pkit is more battery efficient.
  • p2pkit works best where GPS ends: from 40m down to a few centimeters. It is more accurate for use cases that involve discovering devices in a hyperlocal setting where good quality GPS signal is unavailable.
  • p2pkit derives relative proximity directly between devices, while GPS accesses absolute location information over the satellite and then sends it to a server to be processed and matched. As there is no need to process the user’s absolute location, we consider p2pkit to have privacy advantages over GPS.
  • GPS is mainly functional outdoors (requires satellite signal) – p2pkit works indoors, outdoors and underground.
  • p2pkit is more suitable in dynamic settings such as on the train. This is due to the fact that devices are stationary in relation to each other. Whereas with GPS, the devices are in motion in relation to the satellite, hence requiring frequent and constant location updates.

Build great apps with p2pkit 2.0

We understand what it means to build great products and the meticulous task of choosing the right technologies to power them. We are very proud of the test results and are convinced that p2pkit 2.0 is the right technology choice for your proximity project.

We would love to hear from you! And as always, if you need any assistance while developing your apps with p2pkit – we’d be glad to help!

 

Get Started Now!

 


Note:

The above tests are in no way scientific. All tests were executed in our office using a testbed which we have developed internally to support our engineering team during development. We chose a mix of 10 iOS and 10 Android devices. All SDKs were at the latest version as of March 2017. All devices were connected to the internet via our office WiFi, apps running in the Foreground. All devices were restarted before switching to a new SDK. Occasionally some tests demonstrated bad results, we, therefore, repeated the test and excluded the “bad” test from the final results.