D3.js Getting started (again..)

The same questions as some years back: How to get started with it ?

The learning curve for beginners is certainly a bit steep if you dont have a basic background in at least HTML, CSS and JavaScript, in this case you rather go for a visualization tool like Tableau if you look for fast results.
The good news: There is plenty of material out there about D3, books, websites, tutorials. I purchased a couple of books from Packt Books and O’Reilly as PDF version, I think they have the most books, though most of them based on earlier D3 versions.

Packt Books

Packt Books

O'Reilly Books

O’Reilly Books

I wont recommend a specific one, most follow the same principles, a short intro to the DOM, explain the D3 basics followed by more or less complex samples. As with any other technology or programming language you learn most hands-on, there so many examples and good tutorials available. I never attempted to understand everything, understand the concepts, dissect existing visualizations and get creative !

Continue reading

d3.js – From tree to cluster and radial projection

Some visualizations seem to be more sophisticated (to implement) than they actual are, specifically the radial projections. Starting from a tree representation with nodes and links it is quite easy to get to the radial version.
Remark: Of course there are more challenging radial diagrams like the bundle, but lets get started with something simple first !

“Standard” Tree
(I added few more nodes to make the visual difference more obvious)

Tree Visualization

We change 1 line of our sourcecode (see previous tutorial for complete code):

var tree = d3.layout.tree().size([300,300]);
to
var tree = d3.layout.cluster().size([300,300]);

Continue reading

d3.js – Most simple stack layout with bars

In order to understand the physics of various visualizations I walk through the samples and gists I find, dissect them and strip them down to the bare minimum. From there I explore my own variations. I took Mike’s sample for a stacked bar chart and changed it to chart reading from a matrix instead of the csv file.

Stacked Bar Chart

The matrix object

            var matrix = [
                [ 1,  5871, 8916, 2868],
                [ 2, 10048, 2060, 6171],
                [ 3, 16145, 8090, 8045],
                [ 4,   990,  940, 6907],
                [ 5,   450,  430, 5000]
            ];

The stack layout expects an object with an array of objects, each member of the array represents the values of one value group (eg. “all dark blue values”), and each object contains x and y value. It will process theis array amnd a y0 value (the offset to the baseline).
You can see this if you run the below code with the console.log output.

Console

<!DOCTYPE html>
<html>
    <head>
        <title>Simple Stack</title>
        <script type="text/javascript" src="libnew/d3.v2.min.js"></script>
        <style>
            svg {
                border: solid 1px #ccc;
                font: 10px sans-serif;
                shape-rendering: crispEdges;
            }
        </style>
    </head>
    <body>

        <div id="viz"></div>

        <script type="text/javascript">

            var w = 960,
            h = 500

            // create canvas
            var svg = d3.select("#viz").append("svg:svg")
            .attr("class", "chart")
            .attr("width", w)
            .attr("height", h )
            .append("svg:g")
            .attr("transform", "translate(10,470)");

            x = d3.scale.ordinal().rangeRoundBands([0, w-50])
            y = d3.scale.linear().range([0, h-50])
            z = d3.scale.ordinal().range(["darkblue", "blue", "lightblue"])

            console.log("RAW MATRIX---------------------------");
	    // 4 columns: ID,c1,c2,c3
            var matrix = [
                [ 1,  5871, 8916, 2868],
                [ 2, 10048, 2060, 6171],
                [ 3, 16145, 8090, 8045],
                [ 4,   990,  940, 6907],
                [ 5,   450,  430, 5000]
            ];
            console.log(matrix)

            console.log("REMAP---------------------------");
            var remapped =["c1","c2","c3"].map(function(dat,i){
                return matrix.map(function(d,ii){
                    return {x: ii, y: d[i+1] };
                })
            });
            console.log(remapped)

            console.log("LAYOUT---------------------------");
            var stacked = d3.layout.stack()(remapped)
            console.log(stacked)

            x.domain(stacked[0].map(function(d) { return d.x; }));
            y.domain([0, d3.max(stacked[stacked.length - 1], function(d) { return d.y0 + d.y; })]);

            // show the domains of the scales
            console.log("x.domain(): " + x.domain())
            console.log("y.domain(): " + y.domain())
            console.log("------------------------------------------------------------------");

            // Add a group for each column.
            var valgroup = svg.selectAll("g.valgroup")
            .data(stacked)
            .enter().append("svg:g")
            .attr("class", "valgroup")
            .style("fill", function(d, i) { return z(i); })
            .style("stroke", function(d, i) { return d3.rgb(z(i)).darker(); });

            // Add a rect for each date.
            var rect = valgroup.selectAll("rect")
            .data(function(d){return d;})
            .enter().append("svg:rect")
            .attr("x", function(d) { return x(d.x); })
            .attr("y", function(d) { return -y(d.y0) - y(d.y); })
            .attr("height", function(d) { return y(d.y); })
            .attr("width", x.rangeBand());

        </script>
    </body>
</html>

My gist: bl.ocks.org/2940908

d3.js Playing with Selections

One should have a good understanding of selections, it is worth playing with a sandbox and explore the various options you have. I adopted the tutorial from Mike here.

Lets assume we have a simple HTML table:

<!DOCTYPE html>
<html>
    <head>
        <title>Selections</title>
        <script type="text/javascript" src="lib/d3.v2.min.js"></script>
    </head>
    <body>
        <table>
            <thead>
                <tr><td>  A</td><td>  B</td><td>  C</td><td>  D</td></tr>
            </thead>
            <tbody>
                <tr><td>  0</td><td>  1</td><td>  2</td><td>  3</td></tr>
                <tr><td>  4</td><td>  5</td><td>  6</td><td>  7</td></tr>
                <tr><td>  8</td><td>  9</td><td> 10</td><td> 11</td></tr>
                <tr><td> 12</td><td> 13</td><td> 14</td><td> 15</td></tr>
            </tbody>
        </table>
        <script type="text/javascript">
           //nothing yet
        </script>
    </body>
</html>

HTML table

Continue reading

d3.js with Dynamic Data II

A slightly adopted version using SVG. A form to control radius of randomly drawn circles.

<!DOCTYPE html>
<html>
    <head>
        <title>Hello, data!</title>
        <script type="text/javascript" src="lib/d3.js"></script>
    </head>
    <body>

        <form name="myform" onSubmit="return handleClick()">
            <input name="Submit"  type="submit" value="Radius" />
            <input type="text" id="myRadius"/>
        </form>

        <div id="viz"></div>

        <script type="text/javascript">

            var mySVG = d3.select("#viz")
            .append("svg:svg")
            .attr("width", 600)
            .attr("height", 400);

            function handleClick(event){
                console.log(document.getElementById("myRadius").value)
                draw(document.getElementById("myRadius").value)
                return false;
            }

            function draw(rad){
                mySVG.append("svg:circle")
                .style("stroke", "black")
                .style("fill", "grey")
                .attr("r", rad)
                .attr("cx", Math.round(Math.random()*600))
                .attr("cy", Math.round(Math.random()*400))
            }

        </script>
    </body>
</html>

SVG circles

d3.js with Dynamic Data

On the journey to learn d3.js it is most important to understand the basic principles first, but once you got your hands dirty, thanks to lots of samples, you want make data ‘speak’. The available samples refer to static data sitting in variables or coming from a json or csv files, sooner or later you want to feed data dynamic into your visualization. In this very simple sample we want values from a form sitting on the same html page to be added to a list, as simple as no svg involved.

<!DOCTYPE html>
<html>
    <head>
        <title>Hello, data!</title>
        <script type="text/javascript" src="lib/d3.js"></script>
    </head>
    <body>
        <form name="myform" onSubmit="return handleClick()">
            <input name="Submit"  type="submit" value="Add Value" />
            <input type="text" id="myVal"/>
        </form>

        <p></p>
        <p></p>
        <p></p>
        <p></p>
        <p></p>

        <script type="text/javascript">

            var dataset = [];
            for(i=0; i<5; i++){
                dataset.push(Math.round(Math.random()*50));
            }

            var p = d3.select("body").selectAll("p")
            .data(dataset)
            .text(function(d,i){return i + " : " + d;});

            function handleClick(event){
                console.log(document.getElementById("myVal").value)
                draw(document.getElementById("myVal").value)
                return false;
            }

            function draw(val){
                d3.select("body").append("p");
                dataset.push(val);
                var p = d3.select("body").selectAll("p")
                .data(dataset)
                .text(function(d,i){return i + " : " + d;})
            }

        </script>

    </body>
</html>

Dynamic Data

In the next step we will play with some svg, from there I will explore ZK feeding a visualization from a DB or any other feed.

Drools Fusion Samples

I want to share some more tips and more realistic data and rules scenarios.

  • Netbeans: If you run the java file (Run File)with the sample code instead of running the whole application, the resources and NOT rebuild, means any changes made to the rules would not be reflected. So rither Cleand and Build or run the application then resources are rebuild automatically.
  • Threads: In the first sample we did not use a separate thread to fire the rules. If you update or insert facts after the initial ksession.fireAllRules() you will not see any output from the engine, only if you call it again. I recommend the thread approach with ksession.fireUntilHalt()as shown in the last tutorial.
  • Container: A simple java application might be good enough for the first steps, but more realistic is the execution in a web application running in a container. I will post a separate tutorial how to run the rule engine inside a singleton EJB on Glassfish.

Rule samples and code snippets:  Managing Flights and Times
The previous samples with message instances are not close to anything realistic, so I create a little airport scenario. It wont be able to run an airport’s operations but gives an idea. I skip the basic initialization etc. We will use the pseudo clock.

  • We operate an airport with arriving and departing flights.
  • Our simplified data model
    Flights with attributes: Flightnumber, airlinecode, aircrafttype, origin, destination, number of pax.  + internal primary key
    Flight times with attributes:   Timestamp, type. + foreign key to Flights
  • Arriving flights usually have this or similar sequence of events: Scheduled Time (STOA) – Estimated (ETOA) – Landed/Touchdown (LAND) – OnBlock Time (ONBL)
    (in real operations you will have more events, also from different sources)
  • Rule 1: Alert me if the ETOA (often coming from automated interfaces like ATC or IATA Telex messages from the origin airport of the flight) is more than 30min after scheduled arrival time  STOA (maybe to inform public at the airport)
  • Rule 2: Alert me if the time between LAND and ONBL is more than 5min.
  • The rule does not make much operational sense, because you want to warn someone that the aircraft is still not at the position after minutes on the ground, means it is still taxiing and you want to inform the ground crew about it
    Rule 2b: Alert me if there is no ONBL 5min after the LAND event.
  • The Flight class
    public class Flight {
    
        private String flightKey;
        private String flightNumber;
        private String airline2lc;
        private String origin;
        private String destination;
        ... more attributes, getter-setter etc.
    }
    
  • The Flight Time class
    public class FlightTime {
    
        private Date flighttime;
        private long flighttimestamp;
        private String type;
        private String flightkey;
        ... constructor,getter-setter etc.
    }
    
  • For the simple comparison of times (Rule 1 and 2) we can ignore the session clock and use this simple rule
    rule "RULE 5"
    when
        time1:FlightTime(key:flightkey,type=='STOA') from entry-point entryone
        time2:FlightTime(flightkey == key, type=='ETOA', this after[ 30m ] time1) from entry-point entryone
    then
        System.out.println("More than 30min between STOA and ETOA: " + time1.getFlightkey() );
    end
    

    Insert facts (flight times), rules get fired by the separate thread (see previous sample)

    ..
    entrypoint1.insert(new FlightTime(new SimpleDateFormat("MM/dd/yy HH:mm").parse("03/08/12 12:00"), "STOA", "1"));
    entrypoint1.insert(new FlightTime(new SimpleDateFormat("MM/dd/yy HH:mm").parse("03/08/12 12:40"), "ETOA", "1"));
    ..
    
  • For Rule 2b we need to use the session clock
    rule "RULE 6"
    when
        time1:FlightTime(key:flightkey,type=='LAND') from entry-point entryone
        not (
            FlightTime(flightkey == key, type=='ONBL', this after[ 0,5m ] time1) from entry-point entryone
            )
    then
        System.out.println("Missing ONBL after 5min LAND: " + time1.getFlightkey() );
        System.out.println("Session Time at rule trigger: " + (drools.getWorkingMemory().getSessionClock().getCurrentTime())/60000);
    end
    

    Note: We cant use the separate thread like previously, but we must fire ksession.fireAllRules(); everytime we advance the clock !

         SessionPseudoClock clock = ksession.getSessionClock();
    
         clock.advanceTime(2, TimeUnit.MINUTES);
         entryPoint1.insert(new FlightTime((clock.getCurrentTime()), "LAND", "1"));
    
         clock.advanceTime(10, TimeUnit.MINUTES);
    
         ksession.fireAllRules();
         ..
    
  • Run with Rule 2b

    Missing ONBL after 5min

  • Challenge for now: The engine looks at the timestamp of fact insertion, so I cant backdate my timestamps, which would be necessary if you get delayed updates from an interface.
    Eg. at 02:00 you get LAND with timestamp 01:00 you would expect the rule to be triggered at 06:00 (01:00 + 5min) but it gets triggered at 07:00
    Will explore Fusion further to cover this situation.