Visualization Use Case Part 1: Airline Arrival Delays in the US (Tableau)

Going beyond sample datasets and basic visualizations I was looking for open data in my professional domain, the aviation and airport industry. Potential candidates for visualizations are connections, routes, flight plans, airport and airline performance. Performance is usually the comparison of scheduled operations vs. actual milestones. The delay of arriving or departure flights is not only affecting passengers and many parties inside and outside the airport community, but it is driving sentiments, perception and reputation and eventually costs money. This kind of data is not something operators like to release but thanks to the Freedom of Information Act (FOIA), a US Federal law, public gets access to all kind of statistics. From the US DOT (Department of Transportation) you can access and download a variety of datasets, one of them is the On-Time Arrival Performance of US airlines in the US and their delay causes since the year 2003 (link). You can filter by airline, airport and timeframe, review the summary on the DOT website or download the set as CSV for your own analysis. I downloaded the complete dataset for 2015, a 2,25 MB file with roughly 13.500 records.

Arrival Delays in Tableau

 

Airline Delays in the US in 2015 by DOT

 

It provides total arriving flights, cancelled and diverted flights, the delay count and total time by reason (weather, carrier, NAS, security, late aircraft) for each month-airport-airline combination for 14 carriers at 322 airports.

Airline Delays in the US in 2015 by DOT

Airports – Ready for the Cloud ?

Unlike airlines which are used to distributed operations and having systems like a reservation system hosted centrally at their hub (originating in the times of mainframe servers with access to this crucial part of their operations only via a remote connection), airports still tend to follow a much more traditional approach. Airport operations are local and and not geographically distributed like airlines, over decades they established local data-centres on-premise and created a mindset of full control only available with the server and IT services right in their basement. Along come big IT departments with teams of server-, network-, db-admins and support.

St. Albert at Dublin Airport, circa 1950 (CC by National Library of Ireland)

This paradigm is slowly changing, due to the fact airports need to cut cost and operate more efficiently. In parallel we can observe an attitude change at management levels, becoming more open to solutions which are outside of their physical control, they buy in the concept of SAAS, consuming a service on a subscription base with a well defined SLA and availability. This shift started with less crucial back-office systems, like Email-Server and document repositories, and now moving on towards more operation critical systems. Slow adopters or companies restricted by policies or governance issues start moving towards a private cloud, eventually cutting down on operations costs. Airports start to understand internet availability in the year 2015 reached a commodity level like water and electricity, they start to adopt even public cloud hosted services.
Zero tolerance systems like ATC or something less life critical like a FIDS system will remain certainly a local solution, but AODB’s are moving into the cloud. All the vendors jumped on the bandwagon and offer some kind of cloud solution, be in a private cloud offering (with the vendor) or even deployed to a public cloud. The potential in this approach is the opportunity to offer an AODB solution at a fraction of a price of traditional AODB projects. Deploying to a public cloud, without any local requirements other than an internet connection and a browser, a small airport can start using an AODB without any investment, maybe at a price as cheap as 3.000,- Euro monthly subscription. Assuming a smaller airport (less than 1 million PAX/year or something like 25..50 commercial flights a day plus GA) is operating with simple requirements (flight plan import and management, operational flight tracking,  billing, Type B and AFTN message interface).

To answer the questions: Yes, they are ready.
But it depends on the IT strategy of medium to big airports or the restricted budget and need of smaller airports.

Let’s see who is serving the long tail in the airport market !

Airport AODB and Big Data

There are 4 technology terms that almost every IT company in any vertical business picked up during the last few years: Cloud, Big Data, Mobile and Internet of Things. In this series I review some of these buzz words in the Airport IT business context.

Today I will review the question: Does AODB data qualify for Big Data ?

What is Big Data ? A term that everyone uses, from consumer, consultants, user to people at CxO level. All have it, want it, need it, do it. Everyone joins the crowd, though few really know what it is and how to apply it in one’s own context or if one actually has a reasonable use-case for Big Data. I would not attempt to explain Big Data in this article, rather pick key elements and try to apply them. Big Data is too big to swallow and it comes in so many use cases, technologies, brands, products, flavours and attributes from Amazon, Google, Facebook to SAP Hana, No SQL, Mongo DB, Map Reduce, Hadoop and NSA. You can google for the term Big Data, read the Wikipedia Page about it or read one of the hundreds of books about, I can recommend a few here that I read.

Terra Ceia Island Farms gladiolus being loaded onto a U.S. Airlines plane at the Sarasota Airport (by State Library and Archives of Florida)

Do not expect an explicit yes or no answer in the end, though we will do a Big Data compliance check for every paradigm, but still I like to be open to find use cases.

The 3 (+3*) key words for that usually used to define Big Data:

• Volume
  An enormous amount of data created by human interaction, machines and sensors
• Velocity
  A massive and continuous flow of data from various sources, usually in real-time, but not limited to.
• Variety
  Data from many sources in structured and unstructured form
• Veracity*
  Noise and abnormality in data.
  
• Validity*
  Correctness and accuracy of the data
  
• Volatility*
  Data relevance. Lifespan of the data in which data shall be considered for any analysis.

(* Only the first 3 V’s – Volume, Velocity and Variety – are the canonical Big Data attributes, the additional V’s show up sometimes in discussion or papers, but they basically apply to all data.)

Some basic facts about an AODB product. An Aiport Operational Database is an enterprise application with a closed and rather predictable user-base. Depending on the size of the airport and the community serving it we might have up to 200 concurrent users and 1000 active accounts. It is not consumer facing, it is not social media, no click-streams and no input of sensors.

Volume

We need to build a data scenario in a typical or average AODB setup to discuss the term volume. I will try to create a typical scenario in order to create a total number of records or attributes over a typical time-span. Please consider this is as a simplified sample, the figures might vary a lot, depending on various factors, like country and region, international or regional, hub, primary airport, etc.
Usually the airport size in publications or comparisons is derived from the number of PAX and or movements of aircrafts per year.
For some references please refer to the ACI website.

A midsized airport around 20 to 30 million PAX a year might have around 500 turnarounds, this will be 1000 movements (arrival and departure).

Lets assume every movement have ..
20 milestones (scheduled, estimated, actual timings, A-CDM milestones or proprietary timings,..). Each of these milestones gets 3 updates (in average!)
10 load attributes (number of pax, total, by class, infants, cargo and weights,..). Each of these attributes gets 2 updates (in average!)
20 network messages (IATA Type B, AFTN, CFMU,..) This can vary extremely depending on the system landscape.
25 various attributes (flight number, registration, tailnumber, flightplan ID, aircraft type, callsign, resources, connecting flights,..) Each of these attributes gets 2 updates (in average!)
This results in 150 attributes (inclusive of updates) per movement. Applied to 1.000 movements day, we will have
150.000 attributes per day,
4.500.000 attributes per month
27.000.000 attributes per season (6 months, one seasonal schedule)

This approach is conservative, it does not cover audit or system logging. It does not consider a situation where the AODB serves as central data repository (warehouse?), with data-feeds from other systems for permanent storage. In more complex environments I saw requirements to process and store 10.000.000 ground radar updates or 1.000.000 updates from the Building Management system a day.

Do 27 million attributes in 6 months qualify for big data volume ?
In this case I would say no, but taking into the account the option to store more than one season of data and maybe to cover more than one location in a multi-airport situation, maybe yes !

Velocity

Do 150.000 attributes a day qualify for big data velocity ?
Braking down to an average of 1,7 updates a second, rather not. It does not require an Big Data architecture to process this.
Compare with Twitter (not a fair comparison though): ~10.000 tweets a second.

Variety

First, we have almost no unstructured data. Once the AODB has been put in place and production there are hardly changes in the structure of the data. Unstructured data might come with free format messages or partial free format content.
The variety also depends on the complexity of the IT landscape and the number of interfaces, AODB’s often play the role of central system integration and we face lot of inbound data streams, but they usually come in an agreed format.

 

Thoughts on Big Data Analysis

One of the selling points of Big Data is the analytic you can apply to the vast amount of data to identify patterns, extract useful knowledge and business value from the data collected. This might help to improve your business strategies or processes or focus on certain areas of value, even predict future scenarios given certain repeating conditions. We can see definitely value added to the AODB context here, though lot of the data is given and can be adjusted only with limitations or not at all, eg. flight schedules provided by airlines (usually result from slot coordination procedures) and the airport physical resources (stands, gates, belts,..). The potential lies in the analytic of actual data, even the airport can’t necessary change schedules, but with patterns emerging from actual data vs. scheduled data (eg. delays in dependency of certain weather, season, etc.), we can optimize the resources. Analysing connecting flight info can help to improve turnaround ground times and avoid delays, detecting frequent aircraft changes can help to improve gate allocation an other scenarios.
And looking at the big picture, if we would be able to collect from a network or countrywide or on a level like Eurocontrol, Big Data analysis certainly will create more valuable insights and improve on-time performance.

Big Data Bookshelf

Big Data: Principles and Best Practices of Scalable Realtime Data Systems

Big Data For Dummies

Ethics of Big Data: Balancing Risk and Innovation

Data Science for Business: What you need to know about data mining and data-analytic thinking

Some assorted links

• A Very Short History Of Big Data  http://onforb.es/16jHac8
• http://whatsthebigdata.com/2013/07/25/big-data-3-vs-volume-variety-velocity-infographic
• http://www.computerworld.com/article/2690856/8-big-trends-in-big-data-analytics.html

Airport AODB in the Cloud and Big Data

Working for more than 15 years with IT systems in airport operations I pretty much experienced most areas from development, testing, system integration, documentation, training and project management. I spend many years on and with a classic client-server based system, partially with legacy technology, and designed and built an AODB from the scratch in new technology and deployed it to the cloud. Two topics drew my attention particularly in the last few years, the “AODB in the Cloud” and “AODB and Big Data”, I like to review them in some short articles here. This talk is politics-free and does not refer to specific products or companies.

For the reader without airport operational background my elevator speech description for AODB. (Did you notice ? there is no entry in Wikipedia for AODB.)

AODB – Aiport Operational Database
An AODB system is usually the core IT system to support the airport ground operations, it integrates with various systems from the heterogeneous IT landscape found at the airport, compiling data from airlines flights schedules, flight plan management, communication between airlines and airports to building management systems, live ground movements and many more systems.
It serves as platform for CDM (Collaborative Decision Making) for the various parties forming the airport community, from airport operators, airlines, groundhandling agents to ATC (Air Traffic Control).
It handles seasonal and operational flights by providing both real-time and historical data, supports resource management for facilities and equipment, as well for staff, it is an information portal.

The AODB itself can be reduced to a simple 3-tier piece architecture, a database, business logic processing layer and a frontend.
It can be complemented by ESB (Enterprise Service Bus), BI (Business Intelligence) and other components.

Unlike consumer (software) products, this is rather a niche market, and AODB systems are offered only by a couple of companies. To name a few (in alphabetical order):
Amadeus, Arinc, T-Systems, IBS, Inform, Intersystems, ISO, Ultra,.. I leave it to the reader to find out more about these companies and their products.

I like to discuss in the next few blog entries the following topics triggered by the technical and operational evolution AODB systems are going through.

• The AODB in the Cloud – Are airports ready for it ?
  
• Is AODB data “Big Data” ?
• Airport AODB goes NoSQL (Part 1)

Image: At Portland International Airport 05/1973 by The U.S. National Archives (cc)

Airtilus presentation @ prezi

How old is MS Powerpoint ? It was introduced in May 1990, a staggering 22 years in place and the main concept did not change – slides (the digital version of the old slide projector). Companies still depend on it and it is always a struggle to get it running at conferences, talks or customer sites (must bring your own notebook, cannot connect to projector,..).
..You still see people using the typewriter effects in presentations.
..You always see presenters fiddling around with powerpoint design mode when setting up or getting started.

Yes, at least we have free alternative, OpenOffice or LibreOffice (coming with the same features)

If you want to be a bit more presentation 2.0’ish you should try prezi which allows you to create very pleasing and fresh online (and offline!!!) presentations. You can edit and show them online, without relying on notebooks, installed powerpoint viewers etc. It also comes with an iPad app for mobile offline presentation. The only thing I dislike: it relies on Flash, guess just a matter of time until a html5 version is available.

I just created a product presentation with it and get a lot of good feedback, even it is not fully utilizing the prezi features.

prezi presentation

prezi design screen

 

What is Airtilus ?

A cloudbased (private and public) web solution for the airport community (authorities, groundhandler, airlines,..) that I am working on with my team. It covers Flightplan and Resource Management, Service Contract Management and more features. It is considered a AODB like (Airport Operational Database) application but stepping away from the traditional monolithic application silo towards a more open and flexible platform.

You want to know more, drop me a note (sven (at) bighugesystems (dot) com) and I will send you the link to above presentation.

Online IATA Telex Processor

I launched a first version of a Telex processor with a web frontend. It is a beta version and currently only processes MVT standard messages.

Some words about the requirements for a flexible interface processor

• Though IATA Telexes are defined by a standard, variations are common because some are produced automatically by other systems and some are created manually, which causes more errors. The processing of telexes, the pattern recognition, must be flexible enough to be able to handle extra inline whitespaces and dots, as well extra lines with free text or extra headers and trailer, eg. now it is more common to receive telexes via email and often some extra email information is added as header before it reaches your system. Customers also might create their own telex standards, meaning the whole message is transported as free text message, but inside the message the customer uses his own syntax for data transmission.
  This requires a message interpreter that can be configured for new or non-standard formats on the fly, without the need to change any sourcecode and to redeploy a system.
  (I saw a project at one airport where the change of LDM format interpretation would have cost the customer around 10.000 Euro because one of the cargo airlines send messages with an extra header line)
• Other standard messages, such as AFTN, NOTAM or CFMU should be processed by the same engine using the same approach. One interface engine with the flexibility of the scripts covers the various aspects of the different types.

A few words about concept and architecture

• ESB
  Certainly the word ESB sometimes might appear bloated like other IT buzzwords, but it hardly makes sense today to implement distinct own interface systems for every protocol or subsystem type you come across. In a heterogeneous IT landscape like an airport an ESB allows you to easily connect inbound and outbound to a number of other systems via TCPIP, Email, FTP,.. or even talk to other standard systems like SAP, Salesforce.com and so on. We use one connector to talk to the ESB, the rest we orchestrate in the ESB itself. With MULE ESB we have the freedom of an opensource product as well the power of enterprise support. The learning curve for MULE is not too steep.
  For the sample of telexes: Sometimes you ‘receive’ telexes by using the auto export function of the Sitatex application and retrieve the files with the messages via FTP, or you receive the messages as email or via a queuing server from a central corporate entrypoint. We can swing over to another source or run in parallel without touching the main system.
• Script Engine
  Instead of hardcoding the various formats, we use a Java Script engine executing Groovy Scripts. These scripts, one for each message type, are stored in the DB and can be adjusted or customized easily. The scripts produce an internal XML formatted standard output which easily can be un-marshalled during the downstream processing using proper XSD.
• Data Processing
  Whatever requirements you have how to handle the received data. In our sample system here, receive from the web frontend and make it human readable.

Please feel free to drop by http://xxxxx (currently not available, apologies) and try by yourself. Please note: Do not process confidential as the data is transmitted unsecured and might be stored (to improve the quality). This is NOT a commercial offering but a technology showcase. There is no warranty that the server is available or the processor correct. You can use the example message and modify it, otherwise copy and paste your own message.

The service is currently running on a Amazon EC2 micro instance, performance might decrease with a lot of traffic.

Online Telex Processor

Outlook

• Summary for errors and rejected messages.
• For the next versions I will add some of the other available telex types will follow such as LDM and CPM.
• Add AFTN message interpretation.
• Email Reply (send an email to the service and the human readable version is emailed back to the user).