d3.js with Dynamic Data II

A slightly adopted version using SVG. A form to control radius of randomly drawn circles.

<!DOCTYPE html>
<html>
    <head>
        <title>Hello, data!</title>
        <script type="text/javascript" src="lib/d3.js"></script>
    </head>
    <body>

        <form name="myform" onSubmit="return handleClick()">
            <input name="Submit"  type="submit" value="Radius" />
            <input type="text" id="myRadius"/>
        </form>

        <div id="viz"></div>

        <script type="text/javascript">

            var mySVG = d3.select("#viz")
            .append("svg:svg")
            .attr("width", 600)
            .attr("height", 400);

            function handleClick(event){
                console.log(document.getElementById("myRadius").value)
                draw(document.getElementById("myRadius").value)
                return false;
            }

            function draw(rad){
                mySVG.append("svg:circle")
                .style("stroke", "black")
                .style("fill", "grey")
                .attr("r", rad)
                .attr("cx", Math.round(Math.random()*600))
                .attr("cy", Math.round(Math.random()*400))
            }

        </script>
    </body>
</html>

SVG circles

d3.js with Dynamic Data

On the journey to learn d3.js it is most important to understand the basic principles first, but once you got your hands dirty, thanks to lots of samples, you want make data ‘speak’. The available samples refer to static data sitting in variables or coming from a json or csv files, sooner or later you want to feed data dynamic into your visualization. In this very simple sample we want values from a form sitting on the same html page to be added to a list, as simple as no svg involved.

<!DOCTYPE html>
<html>
    <head>
        <title>Hello, data!</title>
        <script type="text/javascript" src="lib/d3.js"></script>
    </head>
    <body>
        <form name="myform" onSubmit="return handleClick()">
            <input name="Submit"  type="submit" value="Add Value" />
            <input type="text" id="myVal"/>
        </form>

        <p></p>
        <p></p>
        <p></p>
        <p></p>
        <p></p>

        <script type="text/javascript">

            var dataset = [];
            for(i=0; i<5; i++){
                dataset.push(Math.round(Math.random()*50));
            }

            var p = d3.select("body").selectAll("p")
            .data(dataset)
            .text(function(d,i){return i + " : " + d;});

            function handleClick(event){
                console.log(document.getElementById("myVal").value)
                draw(document.getElementById("myVal").value)
                return false;
            }

            function draw(val){
                d3.select("body").append("p");
                dataset.push(val);
                var p = d3.select("body").selectAll("p")
                .data(dataset)
                .text(function(d,i){return i + " : " + d;})
            }

        </script>

    </body>
</html>

Dynamic Data

In the next step we will play with some svg, from there I will explore ZK feeding a visualization from a DB or any other feed.

Online IATA Telex Processor

I launched a first version of a Telex processor with a web frontend. It is a beta version and currently only processes MVT standard messages.

Some words about the requirements for a flexible interface processor

  • Though IATA Telexes are defined by a standard, variations are common because some are produced automatically by other systems and some are created manually, which causes more errors. The processing of telexes, the pattern recognition, must be flexible enough to be able to handle extra inline whitespaces and dots, as well extra lines with free text or extra headers and trailer, eg. now it is more common to receive telexes via email and often some extra email information is added as header before it reaches your system. Customers also might create their own telex standards, meaning the whole message is transported as free text message, but inside the message the customer uses his own syntax for data transmission.
    This requires a message interpreter that can be configured for new or non-standard formats on the fly, without the need to change any sourcecode and to redeploy a system.
    (I saw a project at one airport where the change of LDM format interpretation would have cost the customer around 10.000 Euro because one of the cargo airlines send messages with an extra header line)
  • Other standard messages, such as AFTN, NOTAM or CFMU should be processed by the same engine using the same approach. One interface engine with the flexibility of the scripts covers the various aspects of the different types.

A few words about concept and architecture

  • ESB
    Certainly the word ESB sometimes might appear bloated like other IT buzzwords, but it hardly makes sense today to implement distinct own interface systems for every protocol or subsystem type you come across. In a heterogeneous IT landscape like an airport an ESB allows you to easily connect inbound and outbound to a number of other systems via TCPIP, Email, FTP,.. or even talk to other standard systems like SAP, Salesforce.com and so on. We use one connector to talk to the ESB, the rest we orchestrate in the ESB itself. With MULE ESB we have the freedom of an opensource product as well the power of enterprise support. The learning curve for MULE is not too steep.
    For the sample of telexes: Sometimes you ‘receive’ telexes by using the auto export function of the Sitatex application and retrieve the files with the messages via FTP, or you receive the messages as email or via a queuing server from a central corporate entrypoint. We can swing over to another source or run in parallel without touching the main system.
  • Script Engine
    Instead of hardcoding the various formats, we use a Java Script engine executing Groovy Scripts. These scripts, one for each message type, are stored in the DB and can be adjusted or customized easily. The scripts produce an internal XML formatted standard output which easily can be un-marshalled during the downstream processing using proper XSD.
  • Data Processing
    Whatever requirements you have how to handle the received data. In our sample system here, receive from the web frontend and make it human readable.

Please feel free to drop by http://tools.airtilus.net:8080/AirtilusTelexR and try by yourself. Please note: Do not process confidential as the data is transmitted unsecured and might be stored (to improve the quality). This is NOT a commercial offering but a technology showcase. There is no warranty that the server is available or the processor correct. You can use the example message and modify it, otherwise copy and paste your own message.

The service is currently running on a Amazon EC2 micro instance, performance might decrease with a lot of traffic.

Online Telex Processor

Outlook

  • Summary for errors and rejected messages.
  • For the next versions I will add some of the other available telex types will follow such as LDM and CPM.
  • Add AFTN message interpretation.
  • Email Reply (send an email to the service and the human readable version is emailed back to the user).

IATA Telex Types Definition

Updates about the online telex processor here

Working in the airport IT industry you are always challenged with integration tasks at each airport. Usually you face a heterogeneous landscape of home-brew or taylored solutions and standard software running on anything from mainframe to virtual instances in a private clouds. Using an ESB we can tackle a lot of interfacing work and focus on the data integration part. One interface that you will find on all airports that operate commercial flights, is a link to the SITA network to send and receive IATA Telexes.

It is hard to find any information online, so I summarised the available message types here. Btw, these telex types are often referred to as SITA Telex types, which is not correct, IATA (Air Transport Association) defines the available telex types and SITA is operating the network to distribute the messages between airlines, airports, ATC, groundhandling agents and other relevant members of the airport community.

This list should be almost complete, giving you the type, the description and the AHM (Airport Handling Manual) or RP (Recommended Practice) reference. The AHM that you can purchase from IATA gives you all the syntax and details for most of the available types.

Find more info about baggage messaging here: IATA Type B Bag Messages and Baggage Messaging Refresher

 MVT  AIRCRAFT MOVEMENT MESSAGE IATA AHM 780
 LDM  LOAD MESSAGE IATA AHM 583
 PTM  PASSENGER TRANSFER MESSAGE IATA RP 1718
 PSM  PASSENGER SERVICE MESSAGE IATA RP 1715
 DIV  AIRCRAFT DIVERSION MESSAGE IATA AHM 781
 BSM  BAGGAGE SERVICE MESSAGES IATA RP 1745
 CPM  CONTAINER/PALLET DISTRIBUTION MESSAGE IATA AHM 587
 UCM  ULD CONTROL MESSAGE IATA AHM 388
 SCM  ULD STOCK CHECK MESSAGE IATA AHM 385
 SLS  STATISTICAL LOAD SUMMARY IATA AHM 588
 ASM  ADHOC SCHEDULED MESSAGE IATA AHM 785
 PAL  PASSENGER ASSISTANCE LIST IATA RP1707b, 1708
 LPM  LOAD PLANNING MESSAGE IATA AHM 580
 ALI  ABBREVIATED LOAD INFORMATION MESSAGE IATA AHM 584
 SOM  SEATS OCCUPIED MESSAGE IATA RP 1712
 TPM  TELETYPE PASSENGER MANIFEST IATA RP 1717
 RQL  REQUEST LIST MESSAGE IATA RP 1709
 PNL  PASSENGER NAME LIST IATA RP 1708
 PFS  PASSENGER FINAL SALES MESSAGE IATA RP 1719
 IDM  INDUSTRY DISCOUNT MESSAGE IATA RP 1714
 ASL  ADDITIONS AND DELETIONS LIST IATA RP 1708
 SAL  SEATS AVAILABLE LIST IATA RP 1713
 RQM  REQUEST INFORMATION MESSAGE IATA AHM 783
 UWS  ULD/BULK LOAD WEIGHT SIGNAL IATA AHM 581
 FMM  FUEL MONITORING MESSAGE IATA AHM 782

Disclaimer: The list might not be correct or complete. It is for educational purpose only. For reliable information please refer to the IATA manuals.

MVT, LDM, PTM, TPM, CPM, PSM, UCM are among the most common telexes from my experience. If you implement a telex interpreter you definitely need to implement these types first.

I am working on an online telex processor, a demo version I will put online soon.

Drools Fusion Samples

I want to share some more tips and more realistic data and rules scenarios.

  • Netbeans: If you run the java file (Run File)with the sample code instead of running the whole application, the resources and NOT rebuild, means any changes made to the rules would not be reflected. So rither Cleand and Build or run the application then resources are rebuild automatically.
  • Threads: In the first sample we did not use a separate thread to fire the rules. If you update or insert facts after the initial ksession.fireAllRules() you will not see any output from the engine, only if you call it again. I recommend the thread approach with ksession.fireUntilHalt()as shown in the last tutorial.
  • Container: A simple java application might be good enough for the first steps, but more realistic is the execution in a web application running in a container. I will post a separate tutorial how to run the rule engine inside a singleton EJB on Glassfish.

Rule samples and code snippets:  Managing Flights and Times
The previous samples with message instances are not close to anything realistic, so I create a little airport scenario. It wont be able to run an airport’s operations but gives an idea. I skip the basic initialization etc. We will use the pseudo clock.

  • We operate an airport with arriving and departing flights.
  • Our simplified data model
    Flights with attributes: Flightnumber, airlinecode, aircrafttype, origin, destination, number of pax.  + internal primary key
    Flight times with attributes:   Timestamp, type. + foreign key to Flights
  • Arriving flights usually have this or similar sequence of events: Scheduled Time (STOA) – Estimated (ETOA) – Landed/Touchdown (LAND) – OnBlock Time (ONBL)
    (in real operations you will have more events, also from different sources)
  • Rule 1: Alert me if the ETOA (often coming from automated interfaces like ATC or IATA Telex messages from the origin airport of the flight) is more than 30min after scheduled arrival time  STOA (maybe to inform public at the airport)
  • Rule 2: Alert me if the time between LAND and ONBL is more than 5min.
  • The rule does not make much operational sense, because you want to warn someone that the aircraft is still not at the position after minutes on the ground, means it is still taxiing and you want to inform the ground crew about it
    Rule 2b: Alert me if there is no ONBL 5min after the LAND event.
  • The Flight class
    public class Flight {
    
        private String flightKey;
        private String flightNumber;
        private String airline2lc;
        private String origin;
        private String destination;
        ... more attributes, getter-setter etc.
    }
    
  • The Flight Time class
    public class FlightTime {
    
        private Date flighttime;
        private long flighttimestamp;
        private String type;
        private String flightkey;
        ... constructor,getter-setter etc.
    }
    
  • For the simple comparison of times (Rule 1 and 2) we can ignore the session clock and use this simple rule
    rule "RULE 5"
    when
        time1:FlightTime(key:flightkey,type=='STOA') from entry-point entryone
        time2:FlightTime(flightkey == key, type=='ETOA', this after[ 30m ] time1) from entry-point entryone
    then
        System.out.println("More than 30min between STOA and ETOA: " + time1.getFlightkey() );
    end
    

    Insert facts (flight times), rules get fired by the separate thread (see previous sample)

    ..
    entrypoint1.insert(new FlightTime(new SimpleDateFormat("MM/dd/yy HH:mm").parse("03/08/12 12:00"), "STOA", "1"));
    entrypoint1.insert(new FlightTime(new SimpleDateFormat("MM/dd/yy HH:mm").parse("03/08/12 12:40"), "ETOA", "1"));
    ..
    
  • For Rule 2b we need to use the session clock
    rule "RULE 6"
    when
        time1:FlightTime(key:flightkey,type=='LAND') from entry-point entryone
        not (
            FlightTime(flightkey == key, type=='ONBL', this after[ 0,5m ] time1) from entry-point entryone
            )
    then
        System.out.println("Missing ONBL after 5min LAND: " + time1.getFlightkey() );
        System.out.println("Session Time at rule trigger: " + (drools.getWorkingMemory().getSessionClock().getCurrentTime())/60000);
    end
    

    Note: We cant use the separate thread like previously, but we must fire ksession.fireAllRules(); everytime we advance the clock !

         SessionPseudoClock clock = ksession.getSessionClock();
    
         clock.advanceTime(2, TimeUnit.MINUTES);
         entryPoint1.insert(new FlightTime((clock.getCurrentTime()), "LAND", "1"));
    
         clock.advanceTime(10, TimeUnit.MINUTES);
    
         ksession.fireAllRules();
         ..
    
  • Run with Rule 2b

    Missing ONBL after 5min

  • Challenge for now: The engine looks at the timestamp of fact insertion, so I cant backdate my timestamps, which would be necessary if you get delayed updates from an interface.
    Eg. at 02:00 you get LAND with timestamp 01:00 you would expect the rule to be triggered at 06:00 (01:00 + 5min) but it gets triggered at 07:00
    Will explore Fusion further to cover this situation.

Gettings started with JBoss jBPM5

In my journey through all the modules of Drools (Expert, Fusion, Planner), I also visit jBPM (fomerly known as Drools Flow). In this tutorial we just get started with Netbeans and a simple hello world project and focusing on the basics, such as required libraries and running within the Netbeans IDE.
Netbeans is not as close to the BPM community as Eclipse, there is no plugin available, the SOA project is abandoned (I guess). But using Netbeans as our IDE, I will not move over to Eclipse, but see how I can run and debug with Netbeans but still use some Eclipse features.

About BPMN:

The JBoss engine execute BPMN files, we can create this files manually or using one of the visual tools. For the basic understanding you should create a simple BPMN by hand to learn a bit about the notation.

Lets create the most simple BPM flow possible:

Pre-Requisites:

Continue reading

Getting started with Drools Fusion

JBoss Drools is more than Expert and Guvnor only, the suite of products also offers Fusion, Planner and Flow (now called jBPM). After getting my hands with dirty with Expert, I want to look at the Fusion part, which covers the event handling. Based on CEP (Complex Event Processing) as an event-driven architecture which is an own science by itself, whole books written about nothing else than events (link).

There is not much information on the web about Fusion other than the JBoss documentation, some chapters in Books (Packt Publishing: Drools 5 Developer Guide, Drools Cookbook) and a handful of blog entries (link), practically no working samples or end-to-end tutorials.

In this tutorial we will apply the necessary changes to our HelloDrools project from the previous tutorial:

  • The good thing: We dont need any other libraries than the ones already used.
  • Copy the previous project and give it a new name ‘HelloDroolsFusion’
  • In the last tutorial we had a simple message class and inserted a message as fact. A rule was triggered when the message type is equal to ‘Hello’
  • As refresher: The first rule
    import hellodrools.Message
    rule "Hello World"
    when
        message:Message (type=="Hello")
    then
        System.out.println("Hello World, Drools!");
    end
    

    Continue reading