DIY Project: Create a Tracking App Part 1

The discussion about mobile phone location tracking of people and tracing back to potential transmissions is one of the hot topics at moment. In Germany we could expect an app officially being launched towards end of April. I attempt to go through the technical considerations by myself. A hands-on coding excursion with Android to use Bluetooth to scan nearby devices and exchange data with them.

The most basic requirements for a tracking app to be successful:

  • A person need to posses and carry a switched-on mobile (smart) phone.
  • The phone must have GPS and Bluetooth feature and both being enabled.
  • The location need to be recorded as fine-grain as possible. Use of GPS is mandatory, the celldata is way too coarse (see previous post). Though we might consider to skip location completely and rely on the paring of fingerprints solely, depending on the approach.
  • Approach 1: We record the location and time of a device (aka person) and transmit the data to a server immediately and try to match data with other devices on the server. Hard to implement in a GDPR compliant way and users most likely wont buy in.
  • Approach 2: We record the location and time on the device and any digital fingerprint of devices nearby. This anonymous pairings we transmit to the server. Once one device is flagged as infected, the server can flag any other device “paired” previously and push (or pull) a notification to the impacted devices. This way most data remains on the device. A more GDPR compliant way of solving this. Some details need to be worked out though in regards of matching and informing the respective user.
  • Approah 3: Even better if we could rely solely on the fingerprint of nearby devices and the timestamp.
  • The more user we have in the system, the bigger the impact and the chance to trace and inform and potentially stop spreading further.
  • We must have a mean to report an infection and inform affected other users (and still stay within the boundaries of GDPR).

Before walking into the Bluetooth space, some facts:

  • The not-for-profit organisation Bluetooth Special Interest Group (SIG) is responsible for thedevelopment of Bluetooth standards since 1998. (Wikipedia)
  • There is a regular update to the Bluetooth standard, by January this year SIG released version 5.2. It takes time for the hardware manufacturers to adopt the newer standards.
  • We need to distinguish between Bluetooth Classic and Bluetooth Low Energy (BLE). BLE was introduced with version 4 and supported by Android 4.3.
  • Bluetooth Classic is designed for continous short distance two-way data transfer at a speed of up to 5 Mbps (2.1 Mbps with Bluetooth 4). BLE was made to work with other devices at a lower speed and greater distance.
  • Android 8.0 onwards support Bluetooth 5 which is a significant milestone for Bluetooth technology in terms of range, speed and power consumption.
  • It is not possible to programmatically check the supported Bluetooth version in Android, though you can check if BLE is available on the phone.
  • The MAC address of the Bluetooth adapter is fixed and can’t be changed (except for rooted phones). This way it becomes the digital fingerprint.

Are we running out of MAC addresses ?

MAC addresses (used by ethernet, wifi and bluetooth adapters), as per IEEE 802 definition, have 48 bits (6×8 bytes).
Sample AC:07:5F:F8:2F:44
This would result in some 281 trillion (2^48) possible combinations, but the first 3 bytes are reserved to identify the hardware manufacturer. For above sample AC:07:5F it is Huawei. The remaining 3 bytes are used as unique identifier, resulting in only 16 million (2^24) unique devices. Quite likely this number would be used up more or less quickly by a big manufacturer. In reality we also could have 16 million unique manufacturer ID’s, Huawei owns about 600 of these, giving a total of currently 10 billion devices. We need to consider this numbers when we talk about unique fingerprints (MAC), though it is unlikely at a country level to have duplicates. In Germany we have ~83 million citizens and about 142 million mobile phones from different manufacturers, small chance that two persons (actually using the tracking app) will have the same MAC address.
You can check/download the identifiers here.

Lets get started with some coding..

Basic: Android to list paired devices

Before we jump into the more complex discovering, pairing and communication between devices (using threads,) we start with the basics. Lets enumerate the paired devices.

Required Permission

At minimum access to coarse location (since Android 6) is needed since Bluetooth can be used to derive the users location. I skip the code to request the permission, only location access being a critical permission. (complete code will be pusblished at the end).

<uses-permission android:name="android.permission.BLUETOOTH"/>
<uses-permission android:name="android.permission.BLUETOOTH_ADMIN"/>
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />

Check and Activate Bluetooth Adapter

public class MainActivity extends AppCompatActivity {

    private static final String TAG = "bt.MainActivity";
    private BluetoothAdapter bAdapter = BluetoothAdapter.getDefaultAdapter();

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        checkAndRequestPermissions();

        if(bAdapter==null){
            Log.i(TAG,"Bluetooth not supported.");
        } else {
            Log.i(TAG,"Bluetooth supported.");
			
	        if(bAdapter.isEnabled()){
				Log.i(TAG,"Bluetooth enabled.");
				if (!getPackageManager().hasSystemFeature(PackageManager.FEATURE_BLUETOOTH_LE))
					Log.i(TAG, "BLE not supported.");
				else
					Log.i(TAG, "BLE supported.");
			} else {
				Log.i(TAG,"Bluetooth not enabled.");
				startActivityForResult(new Intent(BluetoothAdapter.ACTION_REQUEST_ENABLE),1);
			}
        }
    }
..

List existing pairings

Quite simple to iterate through the existing pairings and list their name and MAC Address

private void showPairedDevices(){
	Set<BluetoothDevice> pairedDevices = bAdapter.getBondedDevices();
	if (pairedDevices.size() > 0) {
		for (BluetoothDevice device : pairedDevices) {
			String deviceName = device.getName();
			String deviceMAC = device.getAddress();
			Log.i(TAG,"Device: " + deviceName + "," + deviceMAC);
		}
	}
}
I/bt.MainActivity: Device: HUAWEI P20,AC:07:5F:XX:XX:XX
I/bt.MainActivity: Device: moto x4,0C:CB:85:XX:XX:XX

In the next post we will discover nearby Bluetooth devices and setting up a communication channel between two devices.

Stay tuned for more tracking..

References

Image by Brian Merrill from Pixabay

Daily Tech Observations 3

Google releases Comunity Reports

Google released a report for each country they are collecting users movement data in an aggregated form. It reveals the popular places, compared to an earlier time frame in February 2020. The data of the current report was released on March 29th summarizing the movement of the 2..3 days before the report. Very drastic the differences between Italy and US, you can clearly see how one country after another only starts to shut down public life. A excellent use of the data with full respect to privacy.

Italy (March 29th report)
US (March 29th report)

Path Inaccuracy – Mobile Phone in the pocket

I noticed the accuracy of the path (recorded by Android/Google) depends a lot on the phone’s exposure to the GPS signal (no surprise). The below sample path (carrying the phone in the pocket of my jacket) demonstrates this. The red/orange dots reflect the correct path, while the blue line is the Google interpretation, obviously interpolating a straight line between location recordings. It works better for places where I have been stationary for a while. Take-away: With Googles location tracking alone we don’t achieve enough accuracy to stop the infection by identifying meeting points or overlaps of data.

Sample Path

Note: This is not a comment against Google or the tracking feature, it highlights potential weakness in the data acquisition process. The feature is an optional setting and can be switched on and off in the configuration of any Android phone.

Daily Tech Observations

As the pandemic crisis continues, more discussion, data exchange and research is happening and progressing in the digital space. I wont mention the massive increase of security threads here (reference info at Trendmicro), but rather look at the non-malicious activities.

PEPP-PT

The PEPP-PT (Pan-European Privacy-Preserving Proximity Tracing) project around a number of prominent research institutes across Europe is working on a proximity-based solution utilising BLE technology embedded in mobile phones. It will be in line with GDPR regulations and to be used on a voluntary base. It is supposed to track and report your whereabouts adn nearby other app users to a server anonymously only, and inform you when you have been close to an infected person, all that without using personal information, which is the key concern of many parties. A key element for the success of such a solution is the penetration factor. It need to build up a database with a significant number of users and traces. Instead of releasing yet another app, they try to piggyback into existing apps, such as NINA (an app to publish and warn about local dangerous incidents in Germany). It has not been published yet, I assume the technical field test was successful is reported, still they have to sort out the communication channels in the case of an infected user.

COVID-19 Apps

There are no new apps in the Google Playstore since my last post, though I have to correct the app I mentioned previously, TraceTogether, only appears for Singapore based accounts. In the German Playstore we see two apps, the app “COVID-19” transmits the status of a COVID test to the respective user, only reducing the need to physically visit a place to retrieve the results. The other app, Coronika, tries to assist individuals to trace their locations and contacts.

Google to hand over anonymous location data

Google and the other big players are in active talks in various countries with the respective authorities about releasing data, either aggregated or anonymous or both. Depends very much on the local regulations. In the context of stopping the pandemy this would provide valuable insights. Aggregated data can help to identify streams of persons or hotspots of too many people in the same area or similar. If anonymization alone is good enough to protect personal data, I would question, the trace that everyone leaves with an Android phone (location services enabled) would easily allow to identify an individual or a small group, you just look at regularly visited places to identify someone’s home or office etc.

You know you can not only see your traces in Google Maps but also export the data (as well delete it permanently if you want) with the Take-Out feature?

Your Timeline – Google Maps
Take Out – Personal Google Maps Data

You are looking for some well formatted data to play with ? Download your own location data and have some hands-on datascience exercise. Easy to request and download, all nicely packaged in self-explaining JSON formatted monthly files.

Stay tuned and safe !

Daily Tech Observations

Some technology related observations from the last few days in these exceptional times we are facing at the moment. A lot of positive actions and activities going on.

COVID-19 Apps

A number of organizations try to build apps that help to control the spread or inform affected people. Most ideas take personal data protection into consideration and allow transfer of data to authorities or servers only upon consent and approval. Though nothing has been releases yet at a larger scale.

  • CoEpi: Community Epidemiology In Action (link)
  • geoHealthApp: Spread the app not the virus (link)
  • ebolaApp: Its team tries to reuse an existing app (link)

Google Playstore and COVID-19 Apps

With hackers, malware and virus producers jumping on the train to misuse the situation (just to mention the ransomware that people behind coronavirusapp[.]site trying to spread an app that locks mobile phones), Google decided not to release any apps in this context until they find appropiate ones. Until a few days ago you did not find any app when searching for the term. By now you find only 1 app in the Germany Play Store with a app released by Singapore government funded app.

Hackathons

Put smart people together and let them work in short time on ideas, this time solely virtually.

  • #WirVsVirus Hackathon by the German government last weekend (link)
    Not less than 42.000 people attended in one way or another.
  • Hack the crisis – Accelerate Estonia (link)
  • AI for mankind hackathon (link)
  • Not to forget to mention the Kaggle Competition (link)

Transparent Prices for panic-buying products

While supermarkets run out of toiletpaper and a few other more important products, mainly due to distribution and logistic problems, the net revelead a whole universe of profiteers via the usual selling and market platforms or through self-created shop websites. Interesting to observe the price development, two weeks ago it was very much wild west with skyrocketing prices, by now the market platforms try to remove and ban any ridicolously priced offers and shops. The below screenshots from a local price tracking shows only samples of “reasonable” increased prices.
Good to see other companies like car manufacturers jumping in and trying to produce equipment in high demand (which hopefully can be produced b y non-medical companies). (Link)

Toiletpaper prices up 40%
Desinfection (hand sanitizer) price tripled

Still it need someone to buy these items and willing to pay the price or being very desperate. Open market physics..

Apps in Trend

Now communication apps like Zoom, Skype, Teams, etc are seeing more downloads. Also streaming apps like Netflix and Amazon Prime Video. But some trending apps are rather weird, like the game Plague Inc. in the current top 10 of games. The game was released in 2012 (not to blame the company) and was not updated since Feb 2019 but getting downloaded more often now. Not sure who wants to play this while we face it for real without a “restart game” button..

Quote description: “Can you infect the world? Plague Inc. is a unique mix of high strategy and terrifyingly realistic simulation. Your pathogen has just infected ‘Patient Zero’. Now you must bring about the end of human history by evolving a deadly, global Plague whilst adapting against everything humanity can do to defend itself.

DIY Tracking and Tracing

In the current situation to trace people with an infectious disease is key and quite some manual Sherlock Holmes style work to find the traces of a patient in a certain region and who her/she/it met and potentially infected.

Technology could be at help here. GDPR is protecting personal data and the wherabout’s of a person falls into this category, but GDPR describes the current situation.

Reference (eur-lex.europa.eu)

  • Article 46
    The processing of personal data should also be regarded to be lawful where it is necessary to protect an interest which is essential for the life of the data subject or that of another natural person. Processing of personal data based on the vital interest of another natural person should in principle take place only where the processing cannot be manifestly based on another legal basis. Some types of processing may serve both important grounds of public interest and the vital interests of the data subject as for instance when processing is necessary for humanitarian purposes, including for monitoring epidemics and their spread or in situations of humanitarian emergencies, in particular in situations of natural and man-made disasters.
  • Article 6 (d) – Lawfulness of processing
    processing is necessary in order to protect the vital interests of the data subject or of another natural person;

Some data (geolocation data) has been provided by German Telekom of its mobile customers to the RKI for people movement research, no individual identifiable data though. Please note it is a small subset and anonymized, there are quite a number of social meida posts and comments informing wrongly.

This is the right thing to do in this situation, the data exist and can speed up the containment. Though it need to be ensured that the date is not used for other purposes or beyond the crisis (as long personal data is stored). Unfortunately this could serve as reference for more (meta-)data harvesting by authorities in future without immediate purpose.

Google and Apple would be in the best position to track down individuals and find the cross roads of tracks and potential infection clusters etc. Most of the people have Android or iOS phones and even not all of them have GPS enabled, the devices still log into the cell towers.

In an earlier project I created an app that is recording the cell tower information while you are on the move. (The app is not in the playstore due to GDPR). Using the recorded cell tower info and matching with the opencellid celltower geolocations I created a map of my own movements in Python using the Folium library.

The geolocation data of the individual towers coming from opencellid is not always 100% accurate on the spot but certainly good enough to to estimate the track. Individual points are too coarse, in this sample dataset, created when I was driving along the highway (red line) the phone connected to various celltowers along the way, even to towers further away from my route. Conclusion: Only the complete dataset can help datascientists to estimate my track and potential contact points with other people.

It will be much harder to track down an individual in a urban environment with 100’s of cell towers, see OpenCellid sample for Frankfurt. I would guess that is the reason RKI only tries to identify streams of people between places.

In China or Israel the technology is already used to pin down individuals. I leave it to you to comment. In Europe, in the interest of personal data protection, an innovative approach would be to (continue to) trace (yourself) but allow the individual to be alerted or verify against an open dataset to be informed about clusters etc. Though, as mentioned before, at some stage the authoroities should make use of the data as long it matches laws and is purposeful.

Further Reading: Wired Magazine

Crisis-Driven-Innovation

I am not attempting to write yet another blog entry about the virus and comment on actions, non-actions, wrong or late actions by countries, governments, organizations or individuals in the current situation that we face as human mankind. This massive challenge will bring out the best and the worst in people. I believe that we will overcome this with many bruises, but hopefully we are smart enough to learn from it. Only collaboration and solitarity across nations and organizations will help us.

Technology is absolutely part of this, both to taggle the current situation (eg. by finding vaccine or trace infections) and support us in future to handle, maybe predict and avoid similar scenarios. Now is time to support existing efforts and start thinking about innovative and smart solution for future re-occurences. We shall not focus on the monetary aspects in the first place, but emphasize on the social and environmental impact. Not to say there are no commcercial ways to fund and run a solution, at a reasonable price that can propell the innovation forward and not targeting a big pile of cash at the end.

One way to support is the participation in the Folding@Home project, a distributed research computing project that run simulations with proteins to develop therapeutics. Volunteers can provide computer resources at home to run small pieces of the massive simulations. Drop by the Folding@Home website and read more. I recommend to use personal computers, not your companies equipment, except it is officially supported.

Folding@Home Control Panel

I let it run while I am in my office. Be aware, depending on the settings, it consumes generously your ressources. My NVIDIA GTX 1060 runs at 4.4 TFLOPS but it certainly consumes some energy. I will try to run it on the Jetson Nano, which usually wont surpass 10W.

It reminds me of the old Seti@Home (Berkeley Research) project started in 1999, when your screensaver could help to analyze radio signals from space. It was hibernated recently.

Another way to do something is to join the Kaggle Platform and do some hands-on research with 2GB worth of data in the COVID-19 Challenge.

More options coming your way, the EU is funding start-ups with the EIC Accelerator (European Innovation Council) program.

My closing comment for this post: No, setting up an e-commerce website and taking 40 Euros from people for package of toilet paper is NOT innovative, is pure avarice.

A related Forbes article for further reading.

Are you ready to innovate ? Lets get started !

Stay tuned and good luck !

Android and Speech Recognition (2)

In part 2 about speech recognition we do the reverse, instead od openly calling Google Search or integrating the speech recognition intent (prominently showing the Google logo/splash screen) we call the same recognition method in a headless way, making the experience more seamless. To make it menaingful in the aviation context, we will request BA flight information (through IAG Webservices) through audio.

(Please note, the below code snippets are incomplete and only highlighting the key methods, the complete sourcecode you find at Github, see link at the end of the post.)

Calling the speech recognizer

sr = SpeechRecognizer.createSpeechRecognizer(this);
sr.setRecognitionListener(new listener());

Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);

intent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS,10);
sr.startListening(intent);

Implementation of the speech recognition class

class listener implements RecognitionListener
{
	public void onResults(Bundle results)
	{
		ArrayList data = results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION);
		StringBuffer result = new StringBuffer();
		for (int i = 0; i < data.size(); i++)
		{
			Log.d(TAG, "result " + data.get(i));
			result.append(i + ":" + data.get(i) + "\n");
		}
		textViewResult.setText(result);
	}
}

With this approach we get the usual 2 acoustic sounds signalling the begin of the speech recognition phase and the end (after a short time out).

If we need to create a hands-free user experience, avoiding the user to touch the screen, we will make use of the play or call button that you usually find on headsets. We can capture the event that gets fired when pressing these hardware buttons too.

Capture hardware button events

@Override
public boolean onKeyDown(int keyCode, KeyEvent event) {
	Log.v(TAG, event.toString());
	if(keyCode == KeyEvent.KEYCODE_HEADSETHOOK || keyCode==KeyEvent.KEYCODE_MEDIA_PLAY){
		Log.i(TAG, event.toString());
		listenHeadless();
		return true;
	}
	return super.onKeyDown(keyCode, event);
}

Text-To-Speech

The missing link to the hands-in-the-pocket experience is the audio response by the app through our headset. We will add the standard Android Text-To-Speech (TTS) implementation.

textToSpeech = new TextToSpeech(getApplicationContext(),null);
...
textToSpeech.setPitch(1.0f);
textToSpeech.setSpeechRate(0.80f);
int speechStatus = textToSpeech.speak(textToSpeak, TextToSpeech.QUEUE_FLUSH, null, null);

if (speechStatus == TextToSpeech.ERROR) {
	Log.e("TTS", "Error in converting Text to Speech!");
}

A remark about text to speech engines, the default speech engines are not very pleasant to listen to, for an application speaking to users repeatedly or over a long time I recommend to look for commercial TTS engines (eg. Nuance, Cereproc,..). Check out the sound samples at the end of the post. TTS engines are usually produced per language. For a multi-lingual application you need to cater for every language.

Adding Webservice

To make this sample app more business relevant (airport related), we use the spoken text to request for flight data. For this purpose I will reuse the BA webservice call that was used in an earlier post. Do not forget to add permission for internet access to our app.

We straight away hit the first challenge here, we receive a text version of the spoken request. As we wont implement a semantic or contextual speech recognition we have to apply regular expression in order to attempt to find the key elements such as fligh number and date. The human requests the information in dozens of possible variations plus the varying interpretations by the speech-to-text engine, some listed below in the regex test screen (link).
To keep it simple we allow the user to request flight info for one particular British Airways flight number between yesterday and tomorrow. We will look for a 2 character 1..4 number combination in the text, plus using the keyword yesterday/tomorrow/today (or no statement representing today).

regular expression to identify flight number

To push the scenario further we can let the TTS engine read the response to the user, after selecting relevant information from the JSON formatted WS response.

Soundsamples

Sourcecode

References