Taming the beast – Some GPU benchmarking

Resuming with the setup and benchmarking of the RTX 3080TI. After the initial basic 3D rendering FPS-tests, time to get the hands dirty with some ML tests. Before trying to benchmark the GPU, we need to get the required Tensorflow packages and NVDIA toolkits up and running under Windows.

For this setup we assume we have Windows 10 and we will use PyCharm as our Python IDE.

The required NVIDIA basic ingredients :

  1. Download and install the latest driver for the GPU from the NVDIA download. The CUDA toolkit requires a minimum driver version (more info).
  2. Download and install the CUDA toolkit (link) (at the time of this post, version 11.6)
  3. Download and install the cuDNN library (link). Beware, there is a dependency between the versions of cuDNN and CUDA. I was not able to make the latest version of both (cuDNN 8.3.1 and CUDA 11.6) to work for our Tensorflow setup.
    Download the latest 8.1.x version of cuDNN instead.

Following the official installation guide (adding the insghts from some blogs and forums), we still have to make some manual changes to our system.

  • Copy relevant library files from the cuDNN zipfile content to the respective CUDA path folders.
  • Ensure the relevant path are setup in the Windows system settings for environment variables.

With this we can start PyCharm, create a project and embedd the Tensorflow packages. If you choose a different method, make sure you use virtual environments, the packages sum up to up to 2GB coming with their potential dependency problems interfering with other projects in case you share packages.

Lets pip-install the Tensorflow-GPU package and check if the GPU is found.

import tensorflow as tf

tf.config.list_physical_devices("GPU")

if tf.test.gpu_device_name():
    print(tf.test.gpu_device_name())
else:
   print("No GPU.")

gpu_devices = tf.config.list_physical_devices('GPU')
if gpu_devices:
  details = tf.config.experimental.get_device_details(gpu_devices[0])
  print(details)

We also can install some of the basic ML packages and verify them.

import sys

import tensorflow.keras
import pandas as pd
import sklearn as sk
import tensorflow as tf

print(f"Tensor Flow Version: {tf.__version__}")
print(f"Keras Version: {tensorflow.keras.__version__}")
print()
print(f"Python {sys.version}")
print(f"Pandas {pd.__version__}")
print(f"Scikit-Learn {sk.__version__}")
gpu = len(tf.config.list_physical_devices('GPU'))>0
print("GPU is", "available" if gpu else "NOT AVAILABLE")

The final step would be to perfrom some kind of ML benchmark performed by the GPU. Doing a quick search I found only one easy applicable solution at the website ai-benchmark.com. Unfortunately, the page, created by some people from the CV Lab at the ETH Zurich, is no longer maintained (no current cards in the rating list). We still can use the package and do a basic scoring.

Comparing, by running the same test, the RTX 3080TI with the CPU AMD Ryzen 1700 (left side in the screenshot).

This provides some order-of-magnitude but hard to say if this a proper proper approach.

More real hands-on exercises will come up soon. Stay tuned..

Android and Speech Recognition (1)

Speech recognition, the translation of spoken language to text, as an interdisciplinary subfield of computational linguistics finds its roots and first steps as early as 1952 with Audrey, a device that could recognize numerical digits, created in the Bell Labs, and the 1960’s when IBM developed the Shoebox, a machine that could recognize and arithmetic and sent the result to an attache printer (I highly recommend the video in the IBM archive). A lot of research happened in the field by IBM, DARPA, CMU but it would take more than 2 decades before products hit the shelf to be used by a wider audience. In 1981 it took up to 100 minutes to decode 30 seconds of spoken text (see the Sarasota Business Journal article).

The first time I started working with commerical products in this field was in the mid-end 1990’s with Dragon Dictate and IBM Via Voice, the engine had to be trained for a specific speaker in a 30min+ training session. Once you had passed the training you could achieve decent results when talking to Word using a plugin, the experience was not quite real-time, as you saw the text magically being typed few seconds after you said it. The product also allowed saying commands to control Windows to open or close windows, starting applications and similar simple tasks.

Fast forward to today, and you find speech recognition in quite some consumer applications, most prominently in the assistance area with products like Amazon Alexa, Apple Siri, Google Assistant dominating the market. If we look at the Gartner Hype Cycle 2019, reaching the Plateau of Productivity “Speech Recognition is less than two years to mainstream adoption and is predicted to deliver the most significant transformational benefits of all technologies on the Hype Cycle.” [Quote].

For many simple use-cases or applications a conversational model nor a semantic interpretation is required, we can focus on recognizing keywords. I will discuss pro and cons later in this series.

In this first part I am demonstrating how simple it is to integrate speech recognition in Android Apps. To put this into an aviation context, lets pretend we search for a flight by calling the flightnumber.

We can start with an empty activity skeleton application in Android Studio.
Required is the permission to record audio.

String requiredPermission = Manifest.permission.RECORD_AUDIO;

We need to call the Android Speech by calling the respective intent.

import android.speech.RecognizerIntent;
..
Intent i = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());
i.putExtra(RecognizerIntent.EXTRA_PROMPT, "Say something");
..
startActivityForResult(i, 100);

Handle the result of the activity response

@Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
	super.onActivityResult(requestCode, resultCode, data);
	if (requestCode == 100) {
		if (resultCode == RESULT_OK && null != data) {
			ArrayList res = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
			Log.d("TTS", res.toString());
		}
	}
}
The default Google style audio input.
Result for “Flight LH 778”

The longer the statement we speak the more results we get back in the text array.

Result for “Departure Information for flight LH 778”

We could defer the search directly into the Google Search by changing the code to another action. Here I have to ask a complete question and cannot just say the flight number. The request is diverted to the web search and wont return to the application. So this is more completeness but not helpful for our use-case.

Intent i = new Intent(RecognizerIntent.ACTION_WEB_SEARCH);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_WEB_SEARCH);
i.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());

In the next part we will omit the Google standard screen and read the audio input directly. We will also look at the further processing challenges, as well add speech synthesis.

Stay tuned !

METAR and TAF Webservice

ozedf

I am regularly exploring the Web Service space in the aviation domain looking for new datasets and feeds. I just stumbled upon CheckWX.com collecting weather messages from various sources and redistribute them. Thanks to Henry who is operating the page (it is not run by a company) we have access to big pool of weather messages via an API. The WS use is free, as long you wont hit the service with more than 2000 calls a day.

METAR (Meteorological Terminal Aviation Routine Weather Report) is a weather forecast in a standard message format, usually used by pilots for the pre-flight briefing. The standard defined by ICAO. The data is collected by weather observation stations.
TAF (Terminal aerodrome forecast), similar to METAR focus on aerodromes and provides a forecast for the next ~24 hours, usually every 6 hours.

You find more information in the ICAO reference “Manual of Aeronautical Meteorological Practice” or in the Airforce “Aircrew Quick Reference to the METAR and TAF Codes”.

Lets build a 5min Android app, similar to the BA Webservice client in a previous post. Thanks to the OkHttp library we get data nicely JSON formatted with a few lines of code. We just need to pass the airportcode (ICAO) to get back the current METAR info (plus some extra info about the airport).

private void callWXweatherService(String airportCode) {

	OkHttpClient httpClient = new OkHttpClient();
	String callURL = "https://api.checkwx.com/metar/" + airportCode + "/decoded";

	Request request = new Request.Builder().url(callURL)
			.addHeader("Content-Type", "application/json")
			.addHeader("X-API-KEY", "{your own API key}")
			.build();

	httpClient.newCall(request).enqueue(new Callback() {
		@Override
		public void onFailure(Call call, IOException e) {
			System.out.println("WS Call failed (1):" + e.getMessage());
		}

		@Override
		public void onResponse(Call call, Response response) {
			ResponseBody responseBody = response.body();
			if (!response.isSuccessful()) {
				final String errRep = "WS Call failed (2):" + response.toString();
			} else {
				try {
					System.out.println("Response " + response.toString());
					String str = new String(responseBody.bytes());
					final JSONObject svcresponse = new JSONObject(str);
					int spacesToIndentEachLevel = 2;
					final String prettyPrintString = svcresponse.toString(spacesToIndentEachLevel);

				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		}
	});
}

The API documentation: api.checkwx.com

screenshotcheckwx

If I have time I will create a better readable gui and integrate with Google Maps.

Disclaimer: This discussion, datamodel and sourcecode or application is for study purpose solely. It does not reflect or replicate any existing commercial product. It is also not fit for or meant for production.

Please note, weather data from CheckWX is not a substitute for an official weather briefing.

Asymetric Encryption 5min walk-through

2018-09-14 13_05_12-343286-PAMSX1-119.jpg - Windows Photo Viewer

Cryptography is one of the key-elements in the Blockchain world. We were already looking into the one-way encryption with hash function in the previous post. Encryption of data to be decrypted again, is equally essential and covered by symmetric and asymmetric encryption. I would not attempt to explain the mathematical  and rather complex background of cryptography, but show with some sourcecode how easy it is to integrate such powerful tools in any solution.

Essential for the asymmetric encryption is the creation of a pair of keys, a public and a private key. The public key can be shared with anyone, it is “public”, it can be used by anyone to create an encrypted message (or data) and send it to a receiver which is the only one holding the private key. This private key is the only way to decrypt the message. This provides a very secure communication mechanism. On the contrary there is Symmetric encryption which uses the same key for encryption and decryption.

Anyone who want to deep-dive into cryptography and algorithm I can refer to books and courses. Otherwise we do an express walk-through here in Java, with the shortest possible sample for key generation (5 lines of code) and encryption/decryption (8 lines of code) of text.

Using the Java core packages java.security it is boiling down to a few methods to create a key pair. You have to choose the algorithm (RSA,DSA,DiffieHellman) and initialize with the keysize. It is obviously a crucial design decision where and how to create the keys and distribute it to the right party.

import java.security.Key;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.PrivateKey;
import java.security.PublicKey;
...

private KeyPairGenerator keyGen;
private KeyPair pair;
static PrivateKey privateKey;
static PublicKey publicKey;
...

keyGen = KeyPairGenerator.getInstance("RSA");
keyGen.initialize(512);
pair = keyGen.generateKeyPair();
privateKey = pair.getPrivate();
publicKey = pair.getPublic();

To encode and decode a text is similar simple. Take note of the binary format, we have to use Base64 to convert to String and back to binary.

	private String encryptText(String messageText, Key key) {

		Base64.Encoder encoder = Base64.getEncoder();
		try {
			cipher = Cipher.getInstance("RSA");
			cipher.init(Cipher.ENCRYPT_MODE, key);
			return encoder.encodeToString(cipher.doFinal(messageText.getBytes("UTF-8")));

		} catch (Exception e) {
			e.printStackTrace();
			return "nil";
		}
	}

	private String decryptText(String cipherText, Key key) {

		Base64.Decoder decoder = Base64.getDecoder();
		try {
			cipher = Cipher.getInstance("RSA");
			cipher.init(Cipher.DECRYPT_MODE, key);
			return new String(cipher.doFinal(decoder.decode(cipherText)), "UTF-8");

		} catch (Exception e) {
			System.out.println(">>>>>> " + e.getMessage());
			return "nil";
		}
	}

Lets run the sample application, create keys, encode and decode a text.
Take note, attempting to decode with the same key fails and the encoding/decoding works with both keys, you can encode with the public or private key and decode with the respective counterpart. Similar to hash we can see the change of 1 char (“My to MY”) changes the cipher completely.

---- KEY GENERATION ----
Algorithm: RSA
Format: PKCS#8
Private Key:
MIIBVQIB(...obfuscated....)Pk/Vp+iTx64pMSQRiXq7UbXX
Public Key:
MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALsDbKCfo55sevpoRL/GjxtlDBFdFLGedfaemCWzh7b/K5ybZbdDrjTVhoyionyGVZuClkZnN5aSRdR6nVXeVL8CAwEAAQ==
------------------------
---- ENCRYPT WITH PUBLIC KEY----
Plain Text: My 2018 Cryptography Sample !
Cipher Text: BkwSG5zczP//7N7AgiFdAyxTvJryPeCLye9zRD1dnKExKR5nABGG3kkiogl8ryujJ+MQB1ISh1EUKSDUm6KYuw==
----------------------------------------------------------
Plain Text: MY 2018 Cryptography Sample !
Cipher Text: Zs8LHPlShaNX/ch8LfrWZrguBJT+WMahMyPQcg06+Nk98i5yd2qyEvP3LfA1QxCmPdFQEPoS7nQxvnPVxE7s0g==
----------------------------------------------------------
---- DECRYPT WITH PRIVATE KEY----
Cipher Text: Zs8LHPlShaNX/ch8LfrWZrguBJT+WMahMyPQcg06+Nk98i5yd2qyEvP3LfA1QxCmPdFQEPoS7nQxvnPVxE7s0g==
Plain Text: MY 2018 Cryptography Sample !
------------------------
---- DECRYPT WITH PUBLIC KEY----
>>>>>> Decryption error
Cipher Text: Zs8LHPlShaNX/ch8LfrWZrguBJT+WMahMyPQcg06+Nk98i5yd2qyEvP3LfA1QxCmPdFQEPoS7nQxvnPVxE7s0g==
Plain Text: nil
----------------------------------------------------------
---- ENCRYPT WITH PRIVATE KEY----
Plain Text: My 2018 Cryptography Sample !
Cipher Text: sjm/MwtAy/MwThhRVKa8lJ0ro1PTHXSK5QfRNLHnYb7X/ezSCLTRPCU7z5TZg7S5ptR2Tvkj4P1J/fk8CNY6xA==
----------------------------------------------------------
---- DECRYPT WITH PUBLIC KEY----
Cipher Text: sjm/MwtAy/MwThhRVKa8lJ0ro1PTHXSK5QfRNLHnYb7X/ezSCLTRPCU7z5TZg7S5ptR2Tvkj4P1J/fk8CNY6xA==
Plain Text: My 2018 Cryptography Sample !

The complete sample sourcecode here on github.
Stay tuned.

Disclaimer: This discussion, datamodel and sourcecode or application is for study purpose solely. It does not reflect or replicate any existing commercial product. It is also not fit for or meant for production.

Blockchain – Big Topic broken down to pieces Part 2 (Bag Custody Sample)

bccloud3

I will continue the exploration of the basic components of the blockchain technology started in the previous post, where I implemented a basic java class to visualize hashing and the linking of blocks (chain). To provide a sample for the aviation industry walking along the use-case of tracking the baggage custody changes during a flight journey .

3. Validation

A blockchain need to be validated otherwise we would not know if data was changed or corrupted. We have to iterate through the sequence of blocks, for this we create an array of bag transactions to allow easy iteration and compare the stored hash of each block with the recalculated hash. If the values match the blockchain is valid, if not it is corrupted (and all subsequent blocks). Ideally this would be event-driven, any change to the blockchain triggers the revalidation. In a real-world implementation changes through the endpoints would not be possible, something we will look at later when working with Corda or Ethereum.

		String currentBagBlockHash = "0";

		String myBagTag = randomBagTagID();
		String myPNR = randomPNR();

		ArrayList allBagTransactions = new ArrayList();
		BagTransaction tempBagTransaction = null;

		// Print Bag Tag (Genesis Block)
		tempBagTransaction = new BagTransaction(myBagTag, myPNR, Entity.NIL.name(), Entity.PAX.name(), 0, "0");
		currentBagBlockHash = tempBagTransaction.getHash();
		allBagTransactions.add(tempBagTransaction);

...

public void checkBlockchainIntegrity(ArrayList allBagTransactions) {
	// Check Blockchain. Compare recalculated hash with hash attribute stored

	for (BagTransaction b : allBagTransactions){
		System.out.print("Block " + b.getBlockID() + " Stored Hash: " + b.getHash() + " -- Calculated Hash:" + b.createHash());
		if (b.getHash().equals(b.createHash()))
			System.out.println(" -- OK");
		else
			System.out.println(" -- FAIL. Blockchain broken.");
	}
	System.out.println(" -- ");
}
	}

For better reading we convert each block into a JSON object.

{
  "timeStamp": "2018-08-26T09:18:09.227Z",
  "blockID": 0,
  "blockHash": "ce41abbf21a162152e30fc511eaf594db43c84948691d12ef78fad6f31fa6043",
  "previousBlockHash": "0",
  "bagTag": "5347241966",
  "pnr": "EONT9T",
  "custodyTransfer": [
    {"transferFrom": "NIL"},
    {"transferTo": "PAX"}
  ]
}
{
  "timeStamp": "2018-08-26T09:18:09.287Z",
  "blockID": 1,
  "blockHash": "6b6073122ef996acc1e3c3e74c3963b5903112800781b4ba88e96baf0a8e2e04",
  "previousBlockHash": "ce41abbf21a162152e30fc511eaf594db43c84948691d12ef78fad6f31fa6043",
  "bagTag": "5347241966",
  "pnr": "EONT9T",
  "custodyTransfer": [
    {"transferFrom": "PAX"},
    {"transferTo": "AIRP"}
  ]
}

Running the validation for a clean and corrupted blockchain

Block 0 Stored Hash: ce41abbf21a162152e30fc511eaf594db43c84948691d12ef78fad6f31fa6043 -- Calculated Hash:ce41abbf21a162152e30fc511eaf594db43c84948691d12ef78fad6f31fa6043 -- OK
Block 1 Stored Hash: 6b6073122ef996acc1e3c3e74c3963b5903112800781b4ba88e96baf0a8e2e04 -- Calculated Hash:6b6073122ef996acc1e3c3e74c3963b5903112800781b4ba88e96baf0a8e2e04 -- OK
Block 2 Stored Hash: a33bbb5ebdec589e9f1b7c7952d6fef3fae2235b5a0600b8755b07040bba8227 -- Calculated Hash:a33bbb5ebdec589e9f1b7c7952d6fef3fae2235b5a0600b8755b07040bba8227 -- OK
Block 2 Stored Hash: 464319fed3a2c49df9a1fbeedae3f3e7c6692db0f6ab97a4c983f9fdf4269e6d -- Calculated Hash:464319fed3a2c49df9a1fbeedae3f3e7c6692db0f6ab97a4c983f9fdf4269e6d -- OK
-- Now corrupt Block 1 by changing the data
Block 0 Stored Hash: ce41abbf21a162152e30fc511eaf594db43c84948691d12ef78fad6f31fa6043 -- Calculated Hash:ce41abbf21a162152e30fc511eaf594db43c84948691d12ef78fad6f31fa6043 -- OK
Block 1 Stored Hash: 6b6073122ef996acc1e3c3e74c3963b5903112800781b4ba88e96baf0a8e2e04 -- Calculated Hash:09fd4223571f079258bd69d935f99cb552cd3ac0854410b81a2313d215929ebd -- FAIL. Blockchain broken.

 

4. Mining Blocks

Now we get to the first more complex concepts of the blockchain, the mining process. We are still operating at a very basic level though with a single node, but we can introduce the mining operation. Without going into too much detail, mining blocks is the step to close/hash a block and creating a new one as part of the consensus process. The incentive to the miner community is a transaction fee given to that miner that solves a hard cryptographic problem first. The more computing power you invested in, the higher chance you have as miner to solve the problem and get the fee. Based on scarcity this consensus approach is called Proof-of-Work (recommended reading). Unfortunately this lead to the current hardware race consuming vast amount of energy for literally no purpose (you let CPU/GPU’s guess numbers basically). This is seen as limitation, together with the long transaction times, and some blockchain start to move to other concepts, such as Proof-of-Stake.

Breaking down the Proof-of-Work to a simple algorithm, we build a hash function that need to create a certain pattern before being accepted. The hash of (data current block + hash previous block + a nonce value) need to have a number of leading “0” in front. The number of “0” is the difficulty and the nonce is an integer value that is changed/increased until the hash matches the required pattern.

public String mineHash(int difficulty) {
	String returnHash = "";
	String tempHash = "";

	String target = new String(new char[difficulty]).replace('\0', '0');
	tempHash = createHash();

	while (!tempHash.substring(0, difficulty).equals(target)) {
		nonce++;
		tempHash = createHash();
	}

	return tempHash;
}

With a growing difficulty (more leading “0”) it takes obviously longer to crack the challenge and with more CPU power you can run through the guessing cycle faster. Some samples below with increasing difficulty running on an ordinary notebook i7 CPU (java executing on a single core/thread in this case).

Difficulty: 1
Attempts: 11
Milliseconds: 10

Difficulty: 2
Attempts: 393
Milliseconds: 30

Difficulty: 3
Attempts: 1.794
Milliseconds: 40

Difficulty: 4
Attempts: 115.756
Milliseconds: 230

Difficulty: 5
Attempts: 3.366.041
Milliseconds: 3.210

Difficulty: 6
Attempts: 5.322.279
Milliseconds: 4.530

Difficulty: 7
Attempts: 76.339.743
Milliseconds: 60.850

Block
{
  "timeStamp": "2018-08-26T10:38:36.734Z",
  "blockID": 0,
  "blockHash": "0000000158d606c953a5df346d459aad949e9dbb4abe74f30ecf225c23112b14",
  "previousBlockHash": "0",
  "bagTag": "7821666095",
  "pnr": "ZHY4RT",
  "custodyTransfer": [
    {"transferFrom": "NIL"},
    {"transferTo": "PAX"}
  ]
}

Have a look at the below website for the current real life difficulty for bitcoin (hashing algorithm not implemented in the simple way we did it here for illustration purpose). Looking at the current difficulty (6,727,225,469,722) you can guess what kind of hardware setup you need to be fast. I gave up beyond difficulty 7 with the algorithm above.

Bitcoin Difficulty

Stay tuned for more blockchain.

Disclaimer: This discussion, datamodel and sourcecode or application is for study purpose solely. It does not reflect or replicate any existing commercial product.

Sourcecode at github

Blockchain – Big Topic broken down to pieces

bccloud2

Blockchain is certainly not the latest buzzword any longer, it moved well forward on the Gartner Hype Cycle, passed the peak of inflated expectations and I am sure we will find it in the trough of disillusionment in the soon to be updated 2018 version. It is picked up by various industries looking for use-cases and applications. Unfortunately we are looking at Blockchain fatigue already, as there is much hype but little visible implementations outside the cryptocurrency space. I prefer projects that implement blockchain as the right tool for a particular problem over the “let’s see which business case we can throw blockchain at” approach.

In the aviation (airport) space I believe Blockchain has its appliance, but as previously stated I wont attempt to build an AODB with Blockchain as “database” for milestones just for the sake of integrating this technology. For some scenarios you certainly need immutability of data, but we can implement this with the means of other immutable data storage. It is also no point implementing a blockchain into a corporate network infrastructure with few nodes under the control of one entity, this does not fulfill the promise of distributed ledger and trust. A few use-cases that I see, usually involving multiple business parties:

  • Baggage tracking from end-to-end (goes well with IATA 753 effective since June 1st)
  • Service and contract management, billing (eg. groundhandler-airline)
  • Aircraft spare parts management (here the track of provenance have a huge impact)

Trying to understand blockchain can be overwhelming, ranging from Satoshi Nakamoto’s original whitepaper to a endless number of books, talks, websites.
One approach to understand the technology is to break it down into smaller pieces that are implementing proven existing technology or algorithms and understand how they come together eventually forming the much more complex blockchain.

I wont attempt to explain blockchain here, this is redundant, plenty of knowledgeable people have written books and articles you can refer to, but split it into some basic easy digestible portions, some coding included. Before attempting to code against real blockchain implementations, like Ethereum or Hyperledger, I will implement the most basic and simple blockchain first.

1. Hash

The most essential element of blockchain is a hash, a digital signature. A hash is a one way encryption,  once something is hashed there is no way to reverse the process and reveal the original text (decrypt it). Using the SHA (Secure Hash Algorithm), the most popular algorithm with its variants of 256, 384 and 512 bit, defined by NIST, we can convert a text (data) of any length to a 256 bit representation (for SHA256) which is represented by a 64 byte hex string. There is a number of libraries implementing the algorithm, below is the Apache DigestUtils version.

	private void testSHA(){

		String sha256hex = DigestUtils.sha256Hex("Jim Smith");
		System.out.println("\n1 SHA256: " + sha256hex);

		String sha256hex2 = DigestUtils.sha256Hex("jim smith");
		System.out.println("\n2 SHA256: " + sha256hex2);

		String sha256hex3 = DigestUtils.sha256Hex("jim smith and the lazy brown fox");
		System.out.println("\n3 SHA256: " + sha256hex3);

		String sha384hex = DigestUtils.sha384Hex("Jim Smith");
		System.out.println("\n4 SHA384: " + sha384hex);

		String sha512hex = DigestUtils.sha512Hex("Jim Smith");
		System.out.println("\n5 SHA512: " + sha512hex);
	}

resulting in

1 SHA256: 65742910cc03889474f1ee2c8f321a105603d0ae2f91070ffd95b35f8da88261
2 SHA256: bfae13266154ec3c4de5c09cf14358305e44f48d2156953723ebbb184a724499
3 SHA256: e5a4a1b8bd88eb7cf8bff9ee5dd235f87ef996262d4d0213c1387f6141ab9574
4 SHA384: c6e76ad773905c1eedb6a0bd9c0b1602a56928d1ce95d70190cd908797466b948dd342aa69dd0343251afece2e48bfc2
5 SHA512: f813c3d9deb66d4999f6839acc60eb6e2fff6a84266c02e0d4b183f5e56d9674c70b0b136f9e1388673cefbc9278f583e3a4c9803ef0c49f9af28aca60dae5ac

Important to notice:
– Change of one character in the original text produces a complete new hash.
– Independent from the length of the original text the hash has the same length.

2. Chained blocks

As the wording implies, there are blocks of information that are linked together. Sounds like a linked list, where every list entry is pointing to the next information. The chained blocks are linked differently, every block points to the hashed previous block.

For illustration I choose a typical baggage journey (simplified).

blockchain

A bag passes different key touchpoints and changes its custody a few times between the various parties during the handling. Every time there is a new milestone event we record it, eg. bag scanned by groundhandler at the chute at what time, and include the hash of the previous milestone. This way the lifecycle from bag drop at departure to bag delivery at destination is recorded in an immutable way and cannot be changed afterwards.

Pitfall: The bag journey is recorded in an immutable way, but the blockchain cannot verify or confirm the milestone actually happened. This falls into the responsibility of the overall design and service orchestration.

Let’s build a very simple application implementing the above blockchain for baggage handling.

A java class BagTransaction representing the bag attributes inclusive timestamp and the custody transfer.

package blockchaindemo;

import java.time.Instant;

import org.apache.commons.codec.digest.DigestUtils;

public class BagTransaction {

	private String bagTag;
	private String timeStamp;
	private String pnr;
	private String transferFrom;
	private String transferTo;

	private long blockID;
	private String blockHash;
	private String previousBlockHash;

	public BagTransaction(String bagTag, String pnr, String transferFrom, String transferTo, long blockID,
			String previousBlockHash) {
		super();
		this.bagTag = bagTag;
		this.timeStamp = Instant.now().toString();
		this.pnr = pnr;
		this.transferFrom = transferFrom;
		this.transferTo = transferTo;
		this.blockID = blockID;
		this.previousBlockHash = previousBlockHash;

		this.blockHash = createCurrentHash();
	}

	public String getHash() {
		return this.blockHash;
	}

	@Override
	public String toString() {
		return "BagTransaction [bagTag=" + bagTag + ", timeStamp=" + timeStamp + ", pnr=" + pnr + ", transferFrom="
				+ transferFrom + ", transferTo=" + transferTo + ", blockID=" + blockID + ", blockHash=" + blockHash
				+ ", previousBlockHash=" + previousBlockHash + "]";
	}

	private String createCurrentHash() {
		String returnHash = "";

		returnHash = DigestUtils.sha256Hex(
				this.bagTag + this.timeStamp + this.pnr + this.transferFrom + this.transferTo + this.previousBlockHash);

		return returnHash;
	}

}

Take note of the hashing method that includes all fields inclusive of the previous hash.

A java class BagDemoApp using the transaction class.

package blockchaindemo;

import java.util.Random;

public class BagDemoApp {

	public static void main(String[] args) {

		BagDemoApp demoApp = new BagDemoApp();
		demoApp.demo1();

	}

	public void demo1() {

		String myBagTag = randomBagTagID();
		String myPNR = randomPNR();

		String currentBagBlockHash = "";

		// Print Bag Tag (Genesis Block)
		BagTransaction bagTransaction1 = new BagTransaction(myBagTag, myPNR, Entity.NIL.name(), Entity.PAX.name(), 1,
				"0");
		currentBagBlockHash = bagTransaction1.getHash();

		// Bag Drop
		BagTransaction bagTransaction2 = new BagTransaction(myBagTag, myPNR, Entity.PAX.name(), Entity.AIRP.name(), 2,
				currentBagBlockHash);
		currentBagBlockHash = bagTransaction2.getHash();

		// Bag SEC Scan
		BagTransaction bagTransaction3 = new BagTransaction(myBagTag, myPNR, Entity.AIRP.name(), Entity.SEC.name(), 3,
				currentBagBlockHash);
		currentBagBlockHash = bagTransaction3.getHash();

		// Display Transactions
		System.out.println(bagTransaction1);
		System.out.println(bagTransaction2);
		System.out.println(bagTransaction3);

	}

	// HELPER METHODS --------------------------------------------

	public String randomPNR() {
		final String alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ";
		final int N = alphabet.length();

		Random r = new Random();
		StringBuffer tempPNR = new StringBuffer();

		for (int i = 0; i < 6; i++) {
			char nxtChar = alphabet.charAt(r.nextInt(N));
			while ((i == 0) && (Character.isDigit(nxtChar)))
				nxtChar = alphabet.charAt(r.nextInt(N));
			tempPNR.append(nxtChar);
		}
		return tempPNR.toString();
	}

	public String randomBagTagID() {
		String tempBagTag = "";

		long range = 9999999999L;
		Random r = new Random();
		long number = (long) (r.nextDouble() * range);

		tempBagTag = String.format("%010d", number);
		return tempBagTag;
	}

	public enum Entity {

		PAX {
			@Override
			public String toString() {
				return "Passenger";
			}
		},
		GH {
			@Override
			public String toString() {
				return "Groundhandler";
			}
		},
		AIRL {
			@Override
			public String toString() {
				return "Airline";
			}
		},
		AIRP {
			@Override
			public String toString() {
				return "Airport";
			}
		},
		SEC {
			@Override
			public String toString() {
				return "Security";
			}
		},
		NIL {
			@Override
			public String toString() {
				return "nil";
			}
		}

	}

}

Executing the application

BagTransaction [bagTag=1691462171, timeStamp=2018-08-12T08:02:25.745Z, pnr=ICSEAH, transferFrom=NIL, transferTo=PAX, blockID=1, blockHash=3ff736f7158d224db6e2e8ba25f3d50321903cd911646576f442a60f8c5872ed, previousBlockHash=0]
BagTransaction [bagTag=1691462171, timeStamp=2018-08-12T08:02:25.808Z, pnr=ICSEAH, transferFrom=PAX, transferTo=AIRP, blockID=2, blockHash=88dd4a2be3bc90ebce71635bedd6bcb63b326044e4bd49634a859a86458de243, previousBlockHash=3ff736f7158d224db6e2e8ba25f3d50321903cd911646576f442a60f8c5872ed]
BagTransaction [bagTag=1691462171, timeStamp=2018-08-12T08:02:25.808Z, pnr=ICSEAH, transferFrom=AIRP, transferTo=SEC, blockID=3, blockHash=ea7767ddb2dd7c2bfed4d3a038b9249e43df15a0396ff71da717783db9fee3c4, previousBlockHash=88dd4a2be3bc90ebce71635bedd6bcb63b326044e4bd49634a859a86458de243]

Please note, this is the most simple implementation of a blockchain for illustration purpose, it still misses a lot of features to pass to production, eg. mining, proof-of-work, etc.

In a second part might spin this a bit further. Stay tuned.

 

Disclaimer: This discussion, datamodel and sourcecode or application is for study purpose solely. It does not reflect or replicate any existing commercial product.

Build an Async Restful Webservice Client in Android in 5min and read BA flight data

In an earlier blog post we build a Restful WS running on Wildfly, now lets build the client part for Android. Thanks to the okHttp, an Apache 2.0 licensed java and Android library, this becomes a very easy challenge. In a mobile application we definitely want to implement a async call as we cannot rely on the response time being fast and a blocked application is not a pleasant user experience.

To make it a bit more interesting for the aviation context of my blog, we will take a real airline webservice and show operational flight data  in the second step.

Step 1: The Basic Webservice Client

Lets get started, I skip the project creation steps, you can create any basic empty Android application.

Add the dependency

..
dependencies {
    compile fileTree(include: ['*.jar'], dir: 'libs')
    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    })
    compile 'com.android.support:appcompat-v7:24.1.1'
    testCompile 'junit:junit:4.12'
compile 'com.squareup.okhttp3:okhttp:3.9.0'
}

Implement the WS call with a call back

(we use the serverside we implemented previously with 2 parameters)

    private void testOKHttp() {

        OkHttpClient httpClient = new OkHttpClient();
        String callURL = "http://www.yourservername.com/DemoRandomizer/randomservice/random/numberrange";

        HttpUrl.Builder urlBuilder = HttpUrl.parse(callURL).newBuilder();
        urlBuilder.addQueryParameter("min", "10");
        urlBuilder.addQueryParameter("max", "100");
        callURL = urlBuilder.build().toString();

        Request request = new Request.Builder().url(callURL).build();
        httpClient.newCall(request).enqueue(new Callback() {

            @Override
            public void onFailure(Call call, IOException e) {
                System.out.println(e.getMessage());
            }
            @Override
            public void onResponse(Call call, Response response) throws IOException {
                ResponseBody responseBody = response.body();
                if (!response.isSuccessful()) {
                    throw new IOException("Error response " + response);
                } else {
                    String str = new String(responseBody.bytes());
                    System.out.println(str);
                }
            }
        });

    }

It is good practice to block UI features (buttons, etc) that rely on the WS response or use a progress dialogm which is in some way blocking but the user is aware of the ongoing request.

 

// Set progress dialog befor the call
        prgDialog = new ProgressDialog(this);
        prgDialog.setMessage("Please wait...");
        prgDialog.setCancelable(false);
        prgDialog.show();

// remove the call after successful response or error
        prgDialog.dismiss();

Step 2: Client to read British Airways data

device-2017-10-14-092750

Lets gets our hand on some real data and tap into one of the open API’s offered by some airlines and airports, though I have to say that only very few players offer their data for public consumption. I will list some API’s in another blog post. Today we will make use of the British Airways API running on the Tibco Mashery platform. BA offers a variety of flight related data and we look at the flight status webservice. We can use the service for evaluation purpose up to 1 call a second and 500 calls a day, good enough to play with their data. We will collect the status of outbound flights from LHR (London Heathrow) for +/- 2hours. Before jumping into sourcode I recommend using a tool to test the webservice manually, eg. the Chrome Browser Rest Web Service Client extension.

2017-10-14 09_24_58-Rest Web Service Client

Using Chrome extension to get data from BA Webservice

One of the challenges, every API offering might support JSON and XML as WS GET, but the attributes and response format is different, we have to implement for every airline or airport we want to use the data, or add some kind of mapping layer.

For the BA service we have to change to call from step 1 above because we need to add the API key that you apply for under your account, also the parameter are not handled as query parameters but as parameter matrix.

The first code block only lists the relevant parts and the complete code block below adds some extras to handle GUI access, exception handling and the parameter creation.

 private void testWSCall() {

        OkHttpClient httpClient = new OkHttpClient();
        String callURL = "https://api.ba.com/rest-v1/v1/flights;departureLocation=LHR;startTime=12:00;endTime=18:00;";

        Request request = new Request.Builder().url(callURL)
                .addHeader("Content-Type", "application/json")
                .addHeader("client-key", "YOUR_KEY_HERE")
                .build();

        httpClient.newCall(request).enqueue(new Callback() {

            @Override
            public void onFailure(Call call, IOException e) {
                System.out.println("WS Call failed (1):" + e.getMessage());
            }

            @Override
            public void onResponse(Call call, Response response) {
                ResponseBody responseBody = response.body();
                if (!response.isSuccessful()) {
                    final String errRep = "WS Call failed (2):" + response.toString();
		    System.out.println(errRep);
                } else {
                    String str = new String(responseBody.bytes());
		    System.out.println(str);
                }
            }
        });

}
 private void testOKHttp() {

        OkHttpClient httpClient = new OkHttpClient();

        DateFormat dateFormatShort = new SimpleDateFormat("HH:mm");
        Date date = new Date();
        long timeNow = date.getTime();
        long timePlus = timeNow + 120 * 60 * 1000;
        long timeMinus = timeNow - 120 * 60 * 1000;
        Date datePlus = new Date(timePlus);
        Date dateMinus = new Date(timeMinus);

        String callURL = "https://api.ba.com/rest-v1/v1/flights;departureLocation=LHR;startTime=" + dateFormatShort.format(dateMinus) +";endTime=" + dateFormatShort.format(datePlus) +";";
        System.out.println(callURL);

        Request request = new Request.Builder().url(callURL)
                .addHeader("Content-Type", "application/json")
                .addHeader("client-key", "YOUR_KEY_HERE<span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>")
                .build();

        httpClient.newCall(request).enqueue(new Callback() {

            @Override
            public void onFailure(Call call, IOException e) {
                System.out.println("WS Call failed (1):" + e.getMessage());
                prgDialog.dismiss();
            }

            @Override
            public void onResponse(Call call, Response response) {
                ResponseBody responseBody = response.body();
                if (!response.isSuccessful()) {
                    final String errRep = "WS Call failed (2):" + response.toString();
                    mHandler.post(new Runnable() {
                        @Override
                        public void run() {
                            editTextResult.setText(errRep);
                        }
                    });
                    prgDialog.dismiss();
                } else {
                    try {
                        System.out.println("Response " + response.toString());
                        String str = new String(responseBody.bytes());
                        prgDialog.dismiss();

                        final JSONObject svcresponse = new JSONObject(str);
                        int spacesToIndentEachLevel = 2;
                        final String prettyPrintString = svcresponse.toString(spacesToIndentEachLevel);
                        mHandler.post(new Runnable() {
                            @Override
                            public void run() {
                                editTextResult.setText(prettyPrintString);
                            }
                        });

                    } catch (Exception e) {
                        e.printStackTrace();
                        final String errStr = e.getMessage();
                        mHandler.post(new Runnable() {
                            @Override
                            public void run() {
                                editTextResult.setText(errStr);
                            }
                        });
                        prgDialog.dismiss();
                    }
                }
            }
        });

    }

MOOC – E-Learning on Steroids

Hardly any industry is moving as fast as the IT industry. While your operational experience and knowledge of the vertical domain you are working in, is growing naturally along your career, it is not the same for IT. For example the airport environment, the underlying basics and physics of handling aircrafts, planning flights, etc. are exposed to changes, innovations and challenges, look at the A-CDM program, it took quite some years to take off and become main-stream, that is a much slower than any new general IT technology or platform soaring. Though this industry is picking up speed too and the boundaries between the digital and physical world start to blur more and more, airports are running digital transformation programs, though passengers still flying in the physical world.

But on the IT side of things, the speed is way beyond breath-taking and it is hard to keep a minimal overview over many areas of IT concerns as well dive into specific topics. How to stay up-to-date and tune into relevant topics ? Books (ink and electronic versions) and forums are certainly the traditional approach, on top of that you join conferences and in-persons seminars and training (which comes at a cost and time spent).

Since the 2000’s online courses came into the picture, as the successor for e-learning, and allow a much bigger audience to learn new technologies, skills and more. The very positive part, there are lots and lots of free courses, most platforms offer free and commercial courses, sometimes free to participate and only charge a fee if you want to get an official certificate (one can argue about the value of such certs) but most important, you can learn and move forward and update your knowledge with the click of a button.

The big challenge though is to identify what you need or interested in, find the right courses and, most important, manage your time. Using your spare time you have to choose wisely, you can’t run for every course out there, even they are so many you are interested in and you are temped to sign up for a dozen of courses, only not to finish any of them.

Todays key-/buzzword for this is MOOC or Massive Open Online Courses. This is like e-learning on steroids, in the past you had to look at dull corporate slides pretty much by yourself, now we look at videos, reading material online and offline, interactions with the organizer, mentor, trainer or your virtual peers at various levels.

Not only the organizations that started online learning, like schools and universities, are into the game, as well companies operating specific online course platforms and now book publishing companies offering courses and finally professional social platforms like LinkedIn.

I attended online courses at Coursera and Udacity, which offer a broad range of topics, and now started with some specific courses on HCI and UX at Interaction Design Foundation which solely offers courses on UX, HCI, Visualization and related topics. Though the courses are unattended (except the rating of your text answers or comments) but repeating, you still have a motivation to participate and go though the lessons because you pay money and they help you pacing the whole course by releasing the lesson packages over time.

Stay tuned for the results.

Build a RESTful Webservice in less than 5 minutes

There are quite some tutorials around about building and exposing a RESTful Webservice, but some of them are outdated and make you wade through complex dependencies and tinkering with deployment descriptors and web.xml files. But using RESTeasy, the JBoss implementation that is fully compliant with the JAX-RS 2.0 JCP specification, and Eclipse you can build a simple webservice (“hello world”) with less than 10 lines of sourceode with annotations and no web.xml used in a few minutes and run it on Wildfly.

Lets build a webservice that creates random numbers.

Continue reading

PostgreSQL Replication Express Setup

The system I work on we deploy almost solely on the Amazon AWS platform. Even I try to design the architecture in a way not to be locked-in too much into Amazon, I make use of the Amazon tools and products as much as possible (EC2, VPC, S3, SNS). PostgreSQL is our reference DB and the only DB product in production environments, still we run dedicated instances with PostgreSQL. I am quite delighted about AWS recent offering staring RDS with PostgreSQL. While is is still in BETA and I did not started yet with a conclusive test and migration plan, I need to maintain our existing instances.

There are plenty of books and tutorials about setting up PostgreSQL replication with on-board tools, without going into the details I share the express setup in this tutorial based on Streaming Replication which is part of PostgreSQL since version 9.0. I highly recommend to review the parameters and settings from the below tutorial as your project might have different requirements.

References

Tutorial

Remarks

  • The tutorial is based on PostgreSQL 9.2 running on Ubuntu Server
  • Paths and settings are all the PostgreSQL defaults.
  • This is async setup, the master will not wait for feedback from the salve and continue to work even the slave is not available

Prerequisite

  • 2 Server running the same PostgreSQL version (9.0+)
  • Backup your data or use a sandbox environment.
  • In the tutorial I refer to
    MASTER (ip: 0.0.0.1) and
    SLAVE (ip: 0.0.0.2)

Configuration

Master

  • Create a replicator user
    sudo -u postgres psql -c "CREATE USER replicator REPLICATION LOGIN ENCRYPTED PASSWORD 'mypassword';"
    
  • Add the slave ip to /etc/postgresql/9.2/main/pg_hba.conf
    host    replication     all      0.0.0.2/32         trust
    
  • Modify parameters in /etc/postgresql/9.2/main/postgresql.conf
    wal_level = hot_standby
    max_wal_senders = 3
    checkpoint_segments = 3
    wal_keep_segments = 3
    

    Review these parameters and set them up according to your requirements

  • Start the PostgreSQL instance
    service postgresql start
    

Slave

  • Modify parameters in /etc/postgresql/9.2/main/postgresql.conf
    wal_level = hot_standby
    max_wal_senders = 3
    checkpoint_segments = 3
    wal_keep_segments = 3
    hot_standby = on
    
  • Stop the PostgreSQL instance
    service postgresql stop
    
  • Clean up the old data directory
    sudo -u postgres rm -rf /var/lib/postgresql/9.2/main
    
  • Copy the database from the master with pg_basebackup
    sudo -u postgres pg_basebackup -h 0.0.0.1 -D /var/lib/postgresql/9.2/main -U replicator -v -P -x
    

    You can see the backup progress and it should result in something likes

    root@:/var/lib/postgresql/9.2/main# transaction log start point: 41/7D000020
    31524952/31524952 kB (100%), 2/2 tablespaces (/var/lib/postgresql/9.2/main/PG_9.)
    transaction log end point: 41/7D0002A8
    pg_basebackup: base backup completed
    
  • Create a recovery configuration file /var/lib/postgresql/9.2/main/recovery.conf
    standby_mode = 'on'
    primary_conninfo = 'host=0.0.0.1 port=5432 user=replicator password=mypassword sslmode=require'
    trigger_file = '/tmp/postgresql.trigger'
    
  • Start the PostgreSQL instance
    service postgresql start
    

    Check the pglogs.

Test the replication

  • Open any table with pgadmin on the master and apply a change, it should be reflected in the slave within short time.
  • Try to change data on the slave, it will fail due to the hot-standby mode

Monitor the replication

  • The master instance will not alert you when the replication is down. You can check by yourself or create a little cronjob to do it for you with this sql statement.
    sudo -u postgres psql -x -c "select * from pg_stat_replication;"
    

    You get the status back (if it is running, otherwise the statement will return ‘no rows’).

    Check replication

    Check replication