Android and Speech Recognition (2)

In part 2 about speech recognition we do the reverse, instead od openly calling Google Search or integrating the speech recognition intent (prominently showing the Google logo/splash screen) we call the same recognition method in a headless way, making the experience more seamless. To make it menaingful in the aviation context, we will request BA flight information (through IAG Webservices) through audio.

(Please note, the below code snippets are incomplete and only highlighting the key methods, the complete sourcecode you find at Github, see link at the end of the post.)

Calling the speech recognizer

sr = SpeechRecognizer.createSpeechRecognizer(this);
sr.setRecognitionListener(new listener());

Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);

intent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS,10);
sr.startListening(intent);

Implementation of the speech recognition class

class listener implements RecognitionListener
{
	public void onResults(Bundle results)
	{
		ArrayList data = results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION);
		StringBuffer result = new StringBuffer();
		for (int i = 0; i < data.size(); i++)
		{
			Log.d(TAG, "result " + data.get(i));
			result.append(i + ":" + data.get(i) + "\n");
		}
		textViewResult.setText(result);
	}
}

With this approach we get the usual 2 acoustic sounds signalling the begin of the speech recognition phase and the end (after a short time out).

If we need to create a hands-free user experience, avoiding the user to touch the screen, we will make use of the play or call button that you usually find on headsets. We can capture the event that gets fired when pressing these hardware buttons too.

Capture hardware button events

@Override
public boolean onKeyDown(int keyCode, KeyEvent event) {
	Log.v(TAG, event.toString());
	if(keyCode == KeyEvent.KEYCODE_HEADSETHOOK || keyCode==KeyEvent.KEYCODE_MEDIA_PLAY){
		Log.i(TAG, event.toString());
		listenHeadless();
		return true;
	}
	return super.onKeyDown(keyCode, event);
}

Text-To-Speech

The missing link to the hands-in-the-pocket experience is the audio response by the app through our headset. We will add the standard Android Text-To-Speech (TTS) implementation.

textToSpeech = new TextToSpeech(getApplicationContext(),null);
...
textToSpeech.setPitch(1.0f);
textToSpeech.setSpeechRate(0.80f);
int speechStatus = textToSpeech.speak(textToSpeak, TextToSpeech.QUEUE_FLUSH, null, null);

if (speechStatus == TextToSpeech.ERROR) {
	Log.e("TTS", "Error in converting Text to Speech!");
}

A remark about text to speech engines, the default speech engines are not very pleasant to listen to, for an application speaking to users repeatedly or over a long time I recommend to look for commercial TTS engines (eg. Nuance, Cereproc,..). Check out the sound samples at the end of the post. TTS engines are usually produced per language. For a multi-lingual application you need to cater for every language.

Adding Webservice

To make this sample app more business relevant (airport related), we use the spoken text to request for flight data. For this purpose I will reuse the BA webservice call that was used in an earlier post. Do not forget to add permission for internet access to our app.

We straight away hit the first challenge here, we receive a text version of the spoken request. As we wont implement a semantic or contextual speech recognition we have to apply regular expression in order to attempt to find the key elements such as fligh number and date. The human requests the information in dozens of possible variations plus the varying interpretations by the speech-to-text engine, some listed below in the regex test screen (link).
To keep it simple we allow the user to request flight info for one particular British Airways flight number between yesterday and tomorrow. We will look for a 2 character 1..4 number combination in the text, plus using the keyword yesterday/tomorrow/today (or no statement representing today).

regular expression to identify flight number

To push the scenario further we can let the TTS engine read the response to the user, after selecting relevant information from the JSON formatted WS response.

Soundsamples

Sourcecode

References

Android and Speech Recognition (1)

Speech recognition, the translation of spoken language to text, as an interdisciplinary subfield of computational linguistics finds its roots and first steps as early as 1952 with Audrey, a device that could recognize numerical digits, created in the Bell Labs, and the 1960’s when IBM developed the Shoebox, a machine that could recognize and arithmetic and sent the result to an attache printer (I highly recommend the video in the IBM archive). A lot of research happened in the field by IBM, DARPA, CMU but it would take more than 2 decades before products hit the shelf to be used by a wider audience. In 1981 it took up to 100 minutes to decode 30 seconds of spoken text (see the Sarasota Business Journal article).

The first time I started working with commerical products in this field was in the mid-end 1990’s with Dragon Dictate and IBM Via Voice, the engine had to be trained for a specific speaker in a 30min+ training session. Once you had passed the training you could achieve decent results when talking to Word using a plugin, the experience was not quite real-time, as you saw the text magically being typed few seconds after you said it. The product also allowed saying commands to control Windows to open or close windows, starting applications and similar simple tasks.

Fast forward to today, and you find speech recognition in quite some consumer applications, most prominently in the assistance area with products like Amazon Alexa, Apple Siri, Google Assistant dominating the market. If we look at the Gartner Hype Cycle 2019, reaching the Plateau of Productivity “Speech Recognition is less than two years to mainstream adoption and is predicted to deliver the most significant transformational benefits of all technologies on the Hype Cycle.” [Quote].

For many simple use-cases or applications a conversational model nor a semantic interpretation is required, we can focus on recognizing keywords. I will discuss pro and cons later in this series.

In this first part I am demonstrating how simple it is to integrate speech recognition in Android Apps. To put this into an aviation context, lets pretend we search for a flight by calling the flightnumber.

We can start with an empty activity skeleton application in Android Studio.
Required is the permission to record audio.

String requiredPermission = Manifest.permission.RECORD_AUDIO;

We need to call the Android Speech by calling the respective intent.

import android.speech.RecognizerIntent;
..
Intent i = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());
i.putExtra(RecognizerIntent.EXTRA_PROMPT, "Say something");
..
startActivityForResult(i, 100);

Handle the result of the activity response

@Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
	super.onActivityResult(requestCode, resultCode, data);
	if (requestCode == 100) {
		if (resultCode == RESULT_OK && null != data) {
			ArrayList res = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
			Log.d("TTS", res.toString());
		}
	}
}
The default Google style audio input.
Result for “Flight LH 778”

The longer the statement we speak the more results we get back in the text array.

Result for “Departure Information for flight LH 778”

We could defer the search directly into the Google Search by changing the code to another action. Here I have to ask a complete question and cannot just say the flight number. The request is diverted to the web search and wont return to the application. So this is more completeness but not helpful for our use-case.

Intent i = new Intent(RecognizerIntent.ACTION_WEB_SEARCH);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_WEB_SEARCH);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());

In the next part we will omit the Google standard screen and read the audio input directly. We will also look at the further processing challenges, as well add speech synthesis.

Stay tuned !

Airport Simulator on Android

Dont rise your expectation to high on this topic for now. Before jumping into the serious use cases for airport process and environment simulations, as well the gamification of enterprise environments, I will review some simulator apps available on Android and PC in a rather humorous way.

I just love simulators, it is amazing to see how current 3D and physic engines running on today’s powerful and affordable hardware can execute real-time simulations that were only possible in well equipped research labs 2 decades ago. I am experimenting with various 3D engines like Unity and CryEngine, both freely available as personal or educational version or for indie games.

OK, lets have a look at simulators available for the Android mobile platform. If you search for the term simulator you will get thousands of apps, and there is a simulator for almost everything and anything you never heard of before, though the majority is plain crap and only exists because Unity makes it so easy to create a game by clicking-together some assets and adjusting some properties, almost without the need to code. Most of them are just badly made and often only serve the purpose to bombard you with ads once installed or running.

We find 100’s of car, truck, train simulation apps, plus dozens of somewhat strange apps to simulate dogs, cats, dinosaurs, Fishing, Fork Lifts, shark attacks, Miami Crime, swimming trains, flying boats and tuk tuk’s and endless more objects.

Finetuning our search towards the term airport (vehicle) simulator we still reveal dozens of results. Here we have a selection of flying, airport construction and all kind of driving around the airport tasks. The majority is made with Unity engine and the free assets, one reason why most of these apps display the same assets like cars, trucks and planes.

AirportSimulator_13

Selection of simulator apps

Most of them have in common: Well rendered icons and preview images in the play store to catch your interest, generous with requested permissions and truckloads of ads.
Surprisingly all of them have downloads of well beyond 100.000 ! Guess we don’t know the uninstall-rate though, but if you look at the user comments you know.

AirportSimulator_30

Permissions galore inclusive to run as service during startup, snooping accounts and location, downloading files.

Here some actual screenshots of the often weird gameplay or scenery I found.

AirportSimulator_9

Factory Airport ? Not much room to maneuver. Where are taxiways, positions, gates ? But 3 Towers !

AirportSimulator_28

If your are short of pushbacks, go for a regular truck as fallback !

AirportSimulator_29

Personal pickup service for MIB passengers in Area 51 ?

AirportSimulator_3

Taking off from a construction site with speed limits and stop signs. Here the tower also a bit too close to the runway maybe.

AirportSimulator_19

Disembarking Zombie passengers walking down from the service staircase to the waiting Cobus(?)

AirportSimulator_11

Driving a heavy duty tow truck between the planes parked on the grass. Nothing else to do. Maybe one of these defunc airline desert airplane parking grounds ?

AirportSimulator_25

This time we drive an airport security car in a totally static airport. Someone forgot to add tarmac lines here ?

AirportSimulator_7

This plane looks like the airport fire brigade drill setup. But in this app all planes look like this model (front part is mocking a war train from the 1930’s). Not so standard cargo handling either.

AirportSimulator_32

Interesting, floating mobile stairs supported by a RC wreck. Walls at the tarmac ?

AirportSimulator_33

Winter wonderland. Random assets like palms, lamp posts and others stuff thrown at the scene. Big Christmas trees at the end of the runway… Walls again.

AirportSimulator_34

Gives the term Greenfield Airport a more genuine meaning ! Randomly appearing zombie passengers again. Leave the boarding gate through the window ?

AirportSimulator_35

Oh yes, that is a smart way to avoid a potential IP conflict with a famous airline ! Bonus: Floating bridge and you fly with a 1:5 scale plane ?

AirportSimulator_36

Smart parking, what to say ? Smart as in stacking cars on the back of a truck or smart as in placing this thing at the end of the runway ?

Conclusion: Dont expect anything when downloading these simulators, other than uninstalling it again and potentially being spied or bombarded with ads. None of the app I tried even remotely comes close to any real operations. These are games – confirmed – nothing else.

In the next blog entry we will look at Simulator Software available for Windows PC’s.

Stay tuned.

 

Budget Android Phone Security Considerations

The market for Android phones and tablets (plus other devices) is huge, Android is dominating the market at almost 90%, followed by iOS around 10% and rest together less than 1%. Google claimed in May 2017 there are 2 billion active devices worldwide.
IT security for mobile devices is a huge topic and we face similar threats as in the desktop OS landscape with virus, trojan horses, backdoor apps and other malware. Some remarks about this at the end of the article.

The mobile phone manufacturer segment is still dominated by Samsung, followed by Huawei, LG, Motorola, to name a few. A medium range device with decent specs costs around $200 to $400.

Apart from the known brands there is wide range of china produced phones at a partially much lower price. Remark: The famous brands might produce as well in China but you pay a higher price for the brand in some sense.

Wouldn’t it be great to get a phone with good specs for less than $100,- ? Some of the brands are HOMTOM, Cubot, OUKITEL, Jesy, DOOGEE and other never-heard-of names.

The temptation is great to buy one of these phones, but it comes at a price. How do you value privacy and security ? Already end of 2016 these devices made it into the news because of sending user data over to servers in China.

A7_Connections_1

Android Phone connections

Lets get some hands-on and investigate with a current phone. I bought a phone for the sole purpose of development testing, not to use for calls or storing sensitive data. I decided to get a BLACKVIEW A7 PRO, available at U$ 70,- from various online stores.

Blackview A7 Pro Android Phone

The phone casing and screen makes a good first impression, the specs do not need to hide behind other medium range phones:

  • Android 7.0
  • MTK6737 1.3GHz, Quad Core
  • 2GB RAM
  • 16GB ROM
  • 4G/LTE with 2 sim card slots
  • 5″ HD screen

Out-of-the-box it comes with the default Android apps, access to the Google Playstore and without any visible bloatware.

Before getting too much excited lets take a look at the communication this device tries to establish after connecting to the internet. I installed a connection and communication inspection tool, these tools are available on the Playstore too. I do not recommend any specific tools here as some of these tools have questionable permission requests.

It does not take too much time before the device starts communicating with a server based in China.

2017-11-07 17_07_05-Screenshot_20171105-132835.png - Windows Photo Viewer2017-11-07 17_07_26-Screenshot_20171105-122804.png - Windows Photo Viewer

Both domains are registered with a hosting company, but you find plenty of news and forum entries if you google for them.

Most interesting is the pre-installed Beau _t_ ySnap application that is not only invisible as app to the user but permanently running in the background. You can find it with a package inspector. Looking at the permissions the application is not only hidden but also very generous with system access.

2017-11-07 17_12_34-Screenshot_20171106-082101.png - Windows Photo Viewer

Bea_u_tySnap permissions

Needless to say you cant restrict the permissions because it is not visible as app but part of the firmware. It also seem to request for hardware specific permissions.

2017-11-07 17_15_12-Screenshot_20171106-082133.png - Windows Photo Viewer

Lets look at the traffic and the data exchanged, seems the app tries to call home a few time per hour.

traffic2

 

Quite obviously they transmit some basic data un-encrypted, like version, current internet connection (wifi), the phone (A7Pro), timezone, screensize and they created a user id to identify the device (aka its user). IMSI and IMEI are not used, though the fields exist. There are some extra fields with encrypted info, I suspect the soc, scc, noc,.. fields might store the celltower info, which serves the coarse geoloaction even if you have GPS disabled.

The least I could do with this phone without rooting it, is to disable the Be_a_utySnap application. Do this by going to Setttings – Apps, enable the display of system apps, select Be_a_utySnap and disable it.

Besides this app, there is still the A_dU_ps FO_TA app and the M_T_K_Logger which are worth looking at. They cant be disabled. I have not caught them yet transferring data out.

Lessons learned

  • Cheap comes at a price. If you are concerned about privacy and security do not buy or use such a phone as your main device to access sensitive personal data or sites like online banking, etc. If you dont want to be tracked and data sold or given to other unknown parties this phone is not for you either. Though there is no guarantee expensive branded devices are protected or cannot be affected too.
    You can use the phone when you know how to disable the spyware elements (and still be careful) or root the phone and install another OS like LineageOS (which comes with separate risks).
  • Do not download applications (apk files) from outside the Google Playstore as the minimum protection and be careful even you download apps from the Playstore by looking at the permissions of the app. You dont need to get a malware infection if you allow a regular app do retrieve your contacts, phone identity and location without knowing what the company behind the app does with the data. GDPR might be irrelevant and impossible to enforce if the app is provided by a company in some non-EU country and has no proper business entity.

Comments on the Android threats landscape

Unlike in the Windows world where we learned the hard way to deal with virus and malware, pretty much securing the OS after treats materialized. Though the first viruses made their first public appearance in the 1980’s, only in 1990/91 the first serious antivirus products were established.
When Android was born some safe concepts were integrated, and with the newer releases more and more security measures were taken (please refer to the below links). Some key elements:

  • Apps run in a sandbox and cant access the OS.
  • Apps are deployed through a central repository (Playstore)
  • User can can control system access via permissions
  • Controlled inter-app communication

Certainly some user still break this by downloading apk files from outside the store, rooting the device, allowing all permissions, etc.

Recommended Reading

Build an Async Restful Webservice Client in Android in 5min and read BA flight data

In an earlier blog post we build a Restful WS running on Wildfly, now lets build the client part for Android. Thanks to the okHttp, an Apache 2.0 licensed java and Android library, this becomes a very easy challenge. In a mobile application we definitely want to implement a async call as we cannot rely on the response time being fast and a blocked application is not a pleasant user experience.

To make it a bit more interesting for the aviation context of my blog, we will take a real airline webservice and show operational flight data  in the second step.

Step 1: The Basic Webservice Client

Lets get started, I skip the project creation steps, you can create any basic empty Android application.

Add the dependency

..
dependencies {
    compile fileTree(include: ['*.jar'], dir: 'libs')
    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    })
    compile 'com.android.support:appcompat-v7:24.1.1'
    testCompile 'junit:junit:4.12'
compile 'com.squareup.okhttp3:okhttp:3.9.0'
}

Implement the WS call with a call back

(we use the serverside we implemented previously with 2 parameters)

    private void testOKHttp() {

        OkHttpClient httpClient = new OkHttpClient();
        String callURL = "http://www.yourservername.com/DemoRandomizer/randomservice/random/numberrange";

        HttpUrl.Builder urlBuilder = HttpUrl.parse(callURL).newBuilder();
        urlBuilder.addQueryParameter("min", "10");
        urlBuilder.addQueryParameter("max", "100");
        callURL = urlBuilder.build().toString();

        Request request = new Request.Builder().url(callURL).build();
        httpClient.newCall(request).enqueue(new Callback() {

            @Override
            public void onFailure(Call call, IOException e) {
                System.out.println(e.getMessage());
            }
            @Override
            public void onResponse(Call call, Response response) throws IOException {
                ResponseBody responseBody = response.body();
                if (!response.isSuccessful()) {
                    throw new IOException("Error response " + response);
                } else {
                    String str = new String(responseBody.bytes());
                    System.out.println(str);
                }
            }
        });

    }

It is good practice to block UI features (buttons, etc) that rely on the WS response or use a progress dialogm which is in some way blocking but the user is aware of the ongoing request.

 

// Set progress dialog befor the call
        prgDialog = new ProgressDialog(this);
        prgDialog.setMessage("Please wait...");
        prgDialog.setCancelable(false);
        prgDialog.show();

// remove the call after successful response or error
        prgDialog.dismiss();

Step 2: Client to read British Airways data

device-2017-10-14-092750

Lets gets our hand on some real data and tap into one of the open API’s offered by some airlines and airports, though I have to say that only very few players offer their data for public consumption. I will list some API’s in another blog post. Today we will make use of the British Airways API running on the Tibco Mashery platform. BA offers a variety of flight related data and we look at the flight status webservice. We can use the service for evaluation purpose up to 1 call a second and 500 calls a day, good enough to play with their data. We will collect the status of outbound flights from LHR (London Heathrow) for +/- 2hours. Before jumping into sourcode I recommend using a tool to test the webservice manually, eg. the Chrome Browser Rest Web Service Client extension.

2017-10-14 09_24_58-Rest Web Service Client

Using Chrome extension to get data from BA Webservice

One of the challenges, every API offering might support JSON and XML as WS GET, but the attributes and response format is different, we have to implement for every airline or airport we want to use the data, or add some kind of mapping layer.

For the BA service we have to change to call from step 1 above because we need to add the API key that you apply for under your account, also the parameter are not handled as query parameters but as parameter matrix.

The first code block only lists the relevant parts and the complete code block below adds some extras to handle GUI access, exception handling and the parameter creation.

 private void testWSCall() {

        OkHttpClient httpClient = new OkHttpClient();
        String callURL = "https://api.ba.com/rest-v1/v1/flights;departureLocation=LHR;startTime=12:00;endTime=18:00;";

        Request request = new Request.Builder().url(callURL)
                .addHeader("Content-Type", "application/json")
                .addHeader("client-key", "YOUR_KEY_HERE")
                .build();

        httpClient.newCall(request).enqueue(new Callback() {

            @Override
            public void onFailure(Call call, IOException e) {
                System.out.println("WS Call failed (1):" + e.getMessage());
            }

            @Override
            public void onResponse(Call call, Response response) {
                ResponseBody responseBody = response.body();
                if (!response.isSuccessful()) {
                    final String errRep = "WS Call failed (2):" + response.toString();
		    System.out.println(errRep);
                } else {
                    String str = new String(responseBody.bytes());
		    System.out.println(str);
                }
            }
        });

}
 private void testOKHttp() {

        OkHttpClient httpClient = new OkHttpClient();

        DateFormat dateFormatShort = new SimpleDateFormat("HH:mm");
        Date date = new Date();
        long timeNow = date.getTime();
        long timePlus = timeNow + 120 * 60 * 1000;
        long timeMinus = timeNow - 120 * 60 * 1000;
        Date datePlus = new Date(timePlus);
        Date dateMinus = new Date(timeMinus);

        String callURL = "https://api.ba.com/rest-v1/v1/flights;departureLocation=LHR;startTime=" + dateFormatShort.format(dateMinus) +";endTime=" + dateFormatShort.format(datePlus) +";";
        System.out.println(callURL);

        Request request = new Request.Builder().url(callURL)
                .addHeader("Content-Type", "application/json")
                .addHeader("client-key", "YOUR_KEY_HERE<span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>")
                .build();

        httpClient.newCall(request).enqueue(new Callback() {

            @Override
            public void onFailure(Call call, IOException e) {
                System.out.println("WS Call failed (1):" + e.getMessage());
                prgDialog.dismiss();
            }

            @Override
            public void onResponse(Call call, Response response) {
                ResponseBody responseBody = response.body();
                if (!response.isSuccessful()) {
                    final String errRep = "WS Call failed (2):" + response.toString();
                    mHandler.post(new Runnable() {
                        @Override
                        public void run() {
                            editTextResult.setText(errRep);
                        }
                    });
                    prgDialog.dismiss();
                } else {
                    try {
                        System.out.println("Response " + response.toString());
                        String str = new String(responseBody.bytes());
                        prgDialog.dismiss();

                        final JSONObject svcresponse = new JSONObject(str);
                        int spacesToIndentEachLevel = 2;
                        final String prettyPrintString = svcresponse.toString(spacesToIndentEachLevel);
                        mHandler.post(new Runnable() {
                            @Override
                            public void run() {
                                editTextResult.setText(prettyPrintString);
                            }
                        });

                    } catch (Exception e) {
                        e.printStackTrace();
                        final String errStr = e.getMessage();
                        mHandler.post(new Runnable() {
                            @Override
                            public void run() {
                                editTextResult.setText(errStr);
                            }
                        });
                        prgDialog.dismiss();
                    }
                }
            }
        });

    }

Read My Boarding Pass App

In the previous blog post I discussed the underlying standards of the BCBP (Bar Coded Boarding Pass) following IATA Reso 792. Today I will built an Android mobile app that scans the PDF417 barcode and shows the raw content as well the decoded fields.

The are 3 main challenges for building the app, scanning/reading the barcode and decoding the text to individual attributes and as last, not to use any internet connection (to assure the user the users privacy and avoid any potential identity theft discussions)

As we build a native Android app we can rely on third party libraries to scan and decode barcodes. There is a number of commercial libraries in the market, but as I build a free app I will use the zxing-android-embedded library, which is a port of the ZXing (“Zebra Crossing”) barcode scanning library for Android with added embedding features, ZXing only provides the decoding logic. Both are licensed under Apache 2.0, ZXing can decode all the common types, such as EAN-8, EAN-13, UPC, ITF, PDF417, QRCode, Aztec, Data Matrix and a few more.

Integration Barcode Library ZXing

With the library the integration becomes as simple as adding a few lines of code only.

Add the dependency to the build gradle file

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    })
    compile 'com.android.support:appcompat-v7:25.3.1'
    compile 'com.android.support.constraint:constraint-layout:1.0.2'
    testCompile 'junit:junit:4.12'

    compile 'com.journeyapps:zxing-android-embedded:3.5.0'
}

Trigger the scan and read the result

public void scanCode(View view){
        new IntentIntegrator(this).initiateScan();
    }

    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        IntentResult result = IntentIntegrator.parseActivityResult(requestCode, resultCode, data);
        if(result != null) {
            if(result.getContents() == null) {
                System.out.println("Scan failed or cancelled.");
            } else {
                System.out.println(result.getContents());
            }
        } else {
            super.onActivityResult(requestCode, resultCode, data);
        }
    }

For now, the app (“ReadMyBoP – Read My Boarding Pass”) does nothing but scan the barcode, identify if it is a valid boarding pass barcode, display the raw content and makes the content more human-readable. You can download from Google Playstore. It works with Android 4.1 and above.
There is one extra feature for now, it decodes the IATA fare codes (First, business, economy classes and the various discounted codes, it follows IATA Reso 728 if you want to look for the complete codeset.

Decoding the Raw Text

Given the fact, this is a fixed-length field text, it is no big deal to split the relevant info by substring’ing it using the decoding table that we started in the previous post. As basic validation we can use the mandatory “M” on position 1 and a length of not less than 59 characters (mandatory fields).

# Element Mandatory Size Sample Remark
1 Format Code M 1 M Always “M”
2 Number of legs encoded M 1
3 Passenger Name M 20
4 Electronic Ticket Indicator M 2 E
5 Operating carrier PNR Code M 7
6 Origin IATA Code M 3 FRA Airport Code
7 Destination IATA Code M 3 SIN Airport Code
8 Operating carrier IATA Code M 3 LH Airline
9 Flight Number M 5 3456
10 Date of Flight M 3 280 Julian Date
11 Compartment Code M 1 B First, Business, Economy
12 Seat Number M 4 25A
13 Check-in Sequence Number M 5 0012
14 Passenger Status M 1 00
15 Size of optional Block M 2 5D hexadecimal
16 Start Version Number 1 > Always “>”
17 Version Number 1 5
18 Field Size of follow ing structured message 2
19 Passenger Description 1
20 Source of check-in 1
21 Source of Boarding Pass Issuance 1
22 Date of Issue of Boarding Pass (Julian Date) 4
23 Document Type 1
24 Airline Designator of boarding pass issuer 3
25 Baggage Tag Licence Plate Number 1 13
26 Baggage Tag Licence Plate Number 2 13
27 Baggage Tag Licence Plate Number 3 13
28 Field Size of follow ing structured message 2
29 Airline Numeric Code 3
30 Document Form/Serial Number 10

Is there a roadmap ? For sure, if I find the time I will add the optional fields, an airline and airport code dictionary (must check the size of a local sqllite db if we want to stay offline). Maybe add baggage tag reader feature and local barcode image storage for boarding. Stay tuned !

readmybop1.2

Application Disclaimer: The application is for educational and research purpose. It is provided as-is, no warranty included. It does not transmit data over the internet and does not store any data upon exiting the app.

What’s in my boarding pass barcode ?

Since more than 10 years passengers are used to the barcode imprinted on the traditional ticket boarding pass slip, the home printed boarding pass or the boarding pass displayed in the mobile app provided by the airline. To be more precise the 2004 IATA Passenger Service Conference approved Resolution 792 setting the BCBP (Bar Coded Boarding Pass) standard as part of the STB (Simplify The Business) program.

The barcode simplifies passenger handling as the barcode can be read automatically by barcode readers along the passenger journey for bag-drop, security check area access and boarding. It significantly reduces the error rate to the time before barcode and eventually saved millions of Euros/Dollars due to mishandling and delays. Today we see self-boarding gates that remove the need for an agent, though business and first class passenger are still welcomed by a human gate agent. Btw, the magnetic stripe on the back of the old tickets expired in 2010.

Old Passenger Ticket without barcode (Image Public Domain)

The BCBP standard defines PDF417 (public domain standard, no fees or licenses) as the barcode symbol format as well defines the encoded content. The content in the barcode contains the same information as printed on the standard boarding pass, though some airlines omit information on the self-printed version in favor of simplicity, some extra info is optionally encoded.

2017-10-07 06_47_41-Boarding-Pass Barcode Aztec QR Generator

Sample PDF 417 barcode

M1SMITH/JOHN          EHJK345 FRASINLH 3456 280C015A0001 100

The standard covers 3 additional barcodes that are not used for printing, but used for mobile apps, these are Aztec and QR Code.

The encoding is straight foward using fix-length fields and the code can carry up to 4 legs of a journey.

Element Size Sample Remark
Format Code 1 M Always “M”
Number of legs encoded 1 1
Passenger Name 20
Electronic Ticket Indicator 2 E
Operating carrier PNR Code 7
Origin IATA Code 3 FRA Airport Code
Destination IATA Code 3 SIN Airport Code
Operating carrier IATA Code 3 LH Airline
Flight Number 5 3456
Date of Flight 3 280 Julian Date
Compartment Code 1 B First, Business, Economy
Seat Number 4 25A
Check-in Sequence Number 5 0012
Passenger Status 1 00

These are the mandatory fields, there are additional optional fields and blocks for baggage info, document info, frequent flyer number or security data.

To be noted, the IATA PADIS XML message standard shall be used for the exchange of BCBP data between systems, defined in Resolution 783 – Passenger and Airport Data Interchange Standards.

I like to add also, the printed barcode is the current common nominator for international travel, but there are initiatives on the way to simplify the passenger journey even further with newer technology such as biometric ID’s and identity management, eg. IATA OneIdentity Initiative.

In the next post we will assemble a simple Android application to read the boarding pass barcode. Stay tuned.

Disclaimer: The information provided here might not be correct or complete. It is for educational purpose only. For reliable information please refer to the IATA manuals.