d3.js – From tree to cluster and radial projection

Some visualizations seem to be more sophisticated (to implement) than they actual are, specifically the radial projections. Starting from a tree representation with nodes and links it is quite easy to get to the radial version.
Remark: Of course there are more challenging radial diagrams like the bundle, but lets get started with something simple first !

“Standard” Tree
(I added few more nodes to make the visual difference more obvious)

Tree Visualization

We change 1 line of our sourcecode (see previous tutorial for complete code):

var tree = d3.layout.tree().size([300,300]);
to
var tree = d3.layout.cluster().size([300,300]);

Continue reading

d3.js – Most simple stack layout with bars

In order to understand the physics of various visualizations I walk through the samples and gists I find, dissect them and strip them down to the bare minimum. From there I explore my own variations. I took Mike’s sample for a stacked bar chart and changed it to chart reading from a matrix instead of the csv file.

Stacked Bar Chart

The matrix object

            var matrix = [
                [ 1,  5871, 8916, 2868],
                [ 2, 10048, 2060, 6171],
                [ 3, 16145, 8090, 8045],
                [ 4,   990,  940, 6907],
                [ 5,   450,  430, 5000]
            ];

The stack layout expects an object with an array of objects, each member of the array represents the values of one value group (eg. “all dark blue values”), and each object contains x and y value. It will process theis array amnd a y0 value (the offset to the baseline).
You can see this if you run the below code with the console.log output.

Console

<!DOCTYPE html>
<html>
    <head>
        <title>Simple Stack</title>
        <script type="text/javascript" src="libnew/d3.v2.min.js"></script>
        <style>
            svg {
                border: solid 1px #ccc;
                font: 10px sans-serif;
                shape-rendering: crispEdges;
            }
        </style>
    </head>
    <body>

        <div id="viz"></div>

        <script type="text/javascript">

            var w = 960,
            h = 500

            // create canvas
            var svg = d3.select("#viz").append("svg:svg")
            .attr("class", "chart")
            .attr("width", w)
            .attr("height", h )
            .append("svg:g")
            .attr("transform", "translate(10,470)");

            x = d3.scale.ordinal().rangeRoundBands([0, w-50])
            y = d3.scale.linear().range([0, h-50])
            z = d3.scale.ordinal().range(["darkblue", "blue", "lightblue"])

            console.log("RAW MATRIX---------------------------");
	    // 4 columns: ID,c1,c2,c3
            var matrix = [
                [ 1,  5871, 8916, 2868],
                [ 2, 10048, 2060, 6171],
                [ 3, 16145, 8090, 8045],
                [ 4,   990,  940, 6907],
                [ 5,   450,  430, 5000]
            ];
            console.log(matrix)

            console.log("REMAP---------------------------");
            var remapped =["c1","c2","c3"].map(function(dat,i){
                return matrix.map(function(d,ii){
                    return {x: ii, y: d[i+1] };
                })
            });
            console.log(remapped)

            console.log("LAYOUT---------------------------");
            var stacked = d3.layout.stack()(remapped)
            console.log(stacked)

            x.domain(stacked[0].map(function(d) { return d.x; }));
            y.domain([0, d3.max(stacked[stacked.length - 1], function(d) { return d.y0 + d.y; })]);

            // show the domains of the scales
            console.log("x.domain(): " + x.domain())
            console.log("y.domain(): " + y.domain())
            console.log("------------------------------------------------------------------");

            // Add a group for each column.
            var valgroup = svg.selectAll("g.valgroup")
            .data(stacked)
            .enter().append("svg:g")
            .attr("class", "valgroup")
            .style("fill", function(d, i) { return z(i); })
            .style("stroke", function(d, i) { return d3.rgb(z(i)).darker(); });

            // Add a rect for each date.
            var rect = valgroup.selectAll("rect")
            .data(function(d){return d;})
            .enter().append("svg:rect")
            .attr("x", function(d) { return x(d.x); })
            .attr("y", function(d) { return -y(d.y0) - y(d.y); })
            .attr("height", function(d) { return y(d.y); })
            .attr("width", x.rangeBand());

        </script>
    </body>
</html>

My gist: bl.ocks.org/2940908

d3.js Playing with Selections

One should have a good understanding of selections, it is worth playing with a sandbox and explore the various options you have. I adopted the tutorial from Mike here.

Lets assume we have a simple HTML table:

<!DOCTYPE html>
<html>
    <head>
        <title>Selections</title>
        <script type="text/javascript" src="lib/d3.v2.min.js"></script>
    </head>
    <body>
        <table>
            <thead>
                <tr><td>  A</td><td>  B</td><td>  C</td><td>  D</td></tr>
            </thead>
            <tbody>
                <tr><td>  0</td><td>  1</td><td>  2</td><td>  3</td></tr>
                <tr><td>  4</td><td>  5</td><td>  6</td><td>  7</td></tr>
                <tr><td>  8</td><td>  9</td><td> 10</td><td> 11</td></tr>
                <tr><td> 12</td><td> 13</td><td> 14</td><td> 15</td></tr>
            </tbody>
        </table>
        <script type="text/javascript">
           //nothing yet
        </script>
    </body>
</html>

HTML table

Continue reading

d3.js with Dynamic Data II

A slightly adopted version using SVG. A form to control radius of randomly drawn circles.

<!DOCTYPE html>
<html>
    <head>
        <title>Hello, data!</title>
        <script type="text/javascript" src="lib/d3.js"></script>
    </head>
    <body>

        <form name="myform" onSubmit="return handleClick()">
            <input name="Submit"  type="submit" value="Radius" />
            <input type="text" id="myRadius"/>
        </form>

        <div id="viz"></div>

        <script type="text/javascript">

            var mySVG = d3.select("#viz")
            .append("svg:svg")
            .attr("width", 600)
            .attr("height", 400);

            function handleClick(event){
                console.log(document.getElementById("myRadius").value)
                draw(document.getElementById("myRadius").value)
                return false;
            }

            function draw(rad){
                mySVG.append("svg:circle")
                .style("stroke", "black")
                .style("fill", "grey")
                .attr("r", rad)
                .attr("cx", Math.round(Math.random()*600))
                .attr("cy", Math.round(Math.random()*400))
            }

        </script>
    </body>
</html>

SVG circles

d3.js with Dynamic Data

On the journey to learn d3.js it is most important to understand the basic principles first, but once you got your hands dirty, thanks to lots of samples, you want make data ‘speak’. The available samples refer to static data sitting in variables or coming from a json or csv files, sooner or later you want to feed data dynamic into your visualization. In this very simple sample we want values from a form sitting on the same html page to be added to a list, as simple as no svg involved.

<!DOCTYPE html>
<html>
    <head>
        <title>Hello, data!</title>
        <script type="text/javascript" src="lib/d3.js"></script>
    </head>
    <body>
        <form name="myform" onSubmit="return handleClick()">
            <input name="Submit"  type="submit" value="Add Value" />
            <input type="text" id="myVal"/>
        </form>

        <p></p>
        <p></p>
        <p></p>
        <p></p>
        <p></p>

        <script type="text/javascript">

            var dataset = [];
            for(i=0; i<5; i++){
                dataset.push(Math.round(Math.random()*50));
            }

            var p = d3.select("body").selectAll("p")
            .data(dataset)
            .text(function(d,i){return i + " : " + d;});

            function handleClick(event){
                console.log(document.getElementById("myVal").value)
                draw(document.getElementById("myVal").value)
                return false;
            }

            function draw(val){
                d3.select("body").append("p");
                dataset.push(val);
                var p = d3.select("body").selectAll("p")
                .data(dataset)
                .text(function(d,i){return i + " : " + d;})
            }

        </script>

    </body>
</html>

Dynamic Data

In the next step we will play with some svg, from there I will explore ZK feeding a visualization from a DB or any other feed.

Drools Fusion Samples

I want to share some more tips and more realistic data and rules scenarios.

  • Netbeans: If you run the java file (Run File)with the sample code instead of running the whole application, the resources and NOT rebuild, means any changes made to the rules would not be reflected. So rither Cleand and Build or run the application then resources are rebuild automatically.
  • Threads: In the first sample we did not use a separate thread to fire the rules. If you update or insert facts after the initial ksession.fireAllRules() you will not see any output from the engine, only if you call it again. I recommend the thread approach with ksession.fireUntilHalt()as shown in the last tutorial.
  • Container: A simple java application might be good enough for the first steps, but more realistic is the execution in a web application running in a container. I will post a separate tutorial how to run the rule engine inside a singleton EJB on Glassfish.

Rule samples and code snippets:  Managing Flights and Times
The previous samples with message instances are not close to anything realistic, so I create a little airport scenario. It wont be able to run an airport’s operations but gives an idea. I skip the basic initialization etc. We will use the pseudo clock.

  • We operate an airport with arriving and departing flights.
  • Our simplified data model
    Flights with attributes: Flightnumber, airlinecode, aircrafttype, origin, destination, number of pax.  + internal primary key
    Flight times with attributes:   Timestamp, type. + foreign key to Flights
  • Arriving flights usually have this or similar sequence of events: Scheduled Time (STOA) – Estimated (ETOA) – Landed/Touchdown (LAND) – OnBlock Time (ONBL)
    (in real operations you will have more events, also from different sources)
  • Rule 1: Alert me if the ETOA (often coming from automated interfaces like ATC or IATA Telex messages from the origin airport of the flight) is more than 30min after scheduled arrival time  STOA (maybe to inform public at the airport)
  • Rule 2: Alert me if the time between LAND and ONBL is more than 5min.
  • The rule does not make much operational sense, because you want to warn someone that the aircraft is still not at the position after minutes on the ground, means it is still taxiing and you want to inform the ground crew about it
    Rule 2b: Alert me if there is no ONBL 5min after the LAND event.
  • The Flight class
    public class Flight {
    
        private String flightKey;
        private String flightNumber;
        private String airline2lc;
        private String origin;
        private String destination;
        ... more attributes, getter-setter etc.
    }
    
  • The Flight Time class
    public class FlightTime {
    
        private Date flighttime;
        private long flighttimestamp;
        private String type;
        private String flightkey;
        ... constructor,getter-setter etc.
    }
    
  • For the simple comparison of times (Rule 1 and 2) we can ignore the session clock and use this simple rule
    rule "RULE 5"
    when
        time1:FlightTime(key:flightkey,type=='STOA') from entry-point entryone
        time2:FlightTime(flightkey == key, type=='ETOA', this after[ 30m ] time1) from entry-point entryone
    then
        System.out.println("More than 30min between STOA and ETOA: " + time1.getFlightkey() );
    end
    

    Insert facts (flight times), rules get fired by the separate thread (see previous sample)

    ..
    entrypoint1.insert(new FlightTime(new SimpleDateFormat("MM/dd/yy HH:mm").parse("03/08/12 12:00"), "STOA", "1"));
    entrypoint1.insert(new FlightTime(new SimpleDateFormat("MM/dd/yy HH:mm").parse("03/08/12 12:40"), "ETOA", "1"));
    ..
    
  • For Rule 2b we need to use the session clock
    rule "RULE 6"
    when
        time1:FlightTime(key:flightkey,type=='LAND') from entry-point entryone
        not (
            FlightTime(flightkey == key, type=='ONBL', this after[ 0,5m ] time1) from entry-point entryone
            )
    then
        System.out.println("Missing ONBL after 5min LAND: " + time1.getFlightkey() );
        System.out.println("Session Time at rule trigger: " + (drools.getWorkingMemory().getSessionClock().getCurrentTime())/60000);
    end
    

    Note: We cant use the separate thread like previously, but we must fire ksession.fireAllRules(); everytime we advance the clock !

         SessionPseudoClock clock = ksession.getSessionClock();
    
         clock.advanceTime(2, TimeUnit.MINUTES);
         entryPoint1.insert(new FlightTime((clock.getCurrentTime()), "LAND", "1"));
    
         clock.advanceTime(10, TimeUnit.MINUTES);
    
         ksession.fireAllRules();
         ..
    
  • Run with Rule 2b

    Missing ONBL after 5min

  • Challenge for now: The engine looks at the timestamp of fact insertion, so I cant backdate my timestamps, which would be necessary if you get delayed updates from an interface.
    Eg. at 02:00 you get LAND with timestamp 01:00 you would expect the rule to be triggered at 06:00 (01:00 + 5min) but it gets triggered at 07:00
    Will explore Fusion further to cover this situation.

Gettings started with JBoss jBPM5

In my journey through all the modules of Drools (Expert, Fusion, Planner), I also visit jBPM (fomerly known as Drools Flow). In this tutorial we just get started with Netbeans and a simple hello world project and focusing on the basics, such as required libraries and running within the Netbeans IDE.
Netbeans is not as close to the BPM community as Eclipse, there is no plugin available, the SOA project is abandoned (I guess). But using Netbeans as our IDE, I will not move over to Eclipse, but see how I can run and debug with Netbeans but still use some Eclipse features.

About BPMN:

The JBoss engine execute BPMN files, we can create this files manually or using one of the visual tools. For the basic understanding you should create a simple BPMN by hand to learn a bit about the notation.

Lets create the most simple BPM flow possible:

Pre-Requisites:

Continue reading

Getting started with Drools Fusion

JBoss Drools is more than Expert and Guvnor only, the suite of products also offers Fusion, Planner and Flow (now called jBPM). After getting my hands with dirty with Expert, I want to look at the Fusion part, which covers the event handling. Based on CEP (Complex Event Processing) as an event-driven architecture which is an own science by itself, whole books written about nothing else than events (link).

There is not much information on the web about Fusion other than the JBoss documentation, some chapters in Books (Packt Publishing: Drools 5 Developer Guide, Drools Cookbook) and a handful of blog entries (link), practically no working samples or end-to-end tutorials.

In this tutorial we will apply the necessary changes to our HelloDrools project from the previous tutorial:

  • The good thing: We dont need any other libraries than the ones already used.
  • Copy the previous project and give it a new name ‘HelloDroolsFusion’
  • In the last tutorial we had a simple message class and inserted a message as fact. A rule was triggered when the message type is equal to ‘Hello’
  • As refresher: The first rule
    import hellodrools.Message
    rule "Hello World"
    when
        message:Message (type=="Hello")
    then
        System.out.println("Hello World, Drools!");
    end
    

    Continue reading

Debugging Drools Rules

At the ‘hello world’ level we cant see much need to debug our rules, but with growing complexity we will be challenged quickly. We need to see which rules was fired, what parameter, what object, etc. Eclipse IDE users have the advantage of the plugin which even visualizes the RETE tree, for the rest-of-us, aka Netbeans user, we need to rely on the debugging output available. We have 4 options that give us access to almost all information of interest.

To try the below debugging options, use the previous HelloDrools Tutorial

1. Default Debug Listener

Out-of-the-box we can use 2 debug event listener:

        ...
        ksession = kbase.newStatefulKnowledgeSession();

        ksession.addEventListener( new DebugAgendaEventListener() );
        ksession.addEventListener( new DebugWorkingMemoryEventListener() );
        ...

Continue reading

Getting started with Drools and Netbeans

Rule engines are a rather complex topic with potentially a steep learning curve. I am looking at a few options, Drools being one of them. As usual one can read all the theoretical papers, but rather I have a something to look at and play with fast. Ultimately I need to integrate the rule engine into my application, so I dont want to play with eclipse editors for rules but seeing a rule engine ticking inside my Netbeans project. I found one tutorial from Vishal Akhouri which I updated to the latest version of Drools and some minor fixes. Most of the sample source code is from Vishal.

Pre-Requirements:

Tutorial: