Sharing Sparks

Generate realtime charts using Vert.x websocket and highchart.js

Tue, 15 May 2012 19:53:00 +0200

Objectif: Update chart data on the client side as events occurs on the business side.

Update: It’s the second time that DZone.com promote my links, thank you, it really encourage myself to write better articles.

Tools:

Vert.x, the java equivalent to node.js
Hightcharts.js a advanced javascript charting library.
WebSocket is a full duplex protocol used to connect the browser to the server. ws for brevity.

Result:

Structure: Highcharts directory contains

jquery-1.7.2.js
highcharts.js
charts.html
WebSocketsServer.groovy

Command: after having installed Vert.x on at the same level of the directory use the command vertx run highcharts/WebSocketsServer.groovy then point your browser to localhost:8080.

How it works: When the page is loaded the first time it establish a websocket connection with the server, the connection is not closed until you close the tab of the browser. When you click on the draw button, you’re actually sending a event to the server which in counterpart is sending a json response through the ws channel. Javascript parse the location of the new point and update the chart.

Charts.html

<html>
<head><title>Web Socket Test</title></head>
<script src="jquery.js" type="text/javascript"></script>
<script src="highcharts.js" type="text/javascript"></script>
<body>

<script>

    var dataseries;

    var socket;
    if (window.WebSocket) {
        socket = new WebSocket("ws://localhost:8080/myapp");
        socket.onmessage = function(event) {
            //alert('data'+event.data)
            var point = JSON.parse(event.data);
            //alert('json ok'+point)
            var npoint = [parseInt(point.x),parseInt(point.y)];
            dataseries.addPoint(npoint);
        }
    } else {
        alert("Your browser does not support Websockets. (Use Chrome)");
    }

    function send(message) {
        if (!window.WebSocket) {
            return;
        }
        if (socket.readyState == WebSocket.OPEN) {
            socket.send(message);
        } else {
            alert("The socket is not open.");
        }
    }

    $(function () {
        var chart;
        $(document).ready(function() {
            chart = new Highcharts.Chart({
                chart: {
                    backgroundColor: {
                       linearGradient: [0, 0, 0, 400],
                       stops: [
                          [0, 'rgb(250, 180, 180)'],
                          [1, 'rgb(255, 240, 240)']
                       ]
                    },
                    renderTo: 'container',
                    type: 'scatter',
                    margin: [70, 50, 60, 80],
                    events: {
                        load: function(e) {
                                dataseries = this.series[0];
                        }
                    }
                },
                title: {
                    text: 'User supplied Love'
                },
                subtitle: {
                    text: 'Because we all love our Moms'
                },
                xAxis: {
                    minPadding: 0.2,
                    maxPadding: 0.2,
                    maxZoom: 60
                },
                yAxis: {
                    title: {
                        text: 'Value'
                    },
                    minPadding: 0.2,
                    maxPadding: 0.2,
                    maxZoom: 60,
                    plotLines: [{
                        value: 0,
                        width: 1,
                        color: '#FF0000'
                    }]
                },
                legend: {
                    enabled: false
                },
                exporting: {
                    enabled: false
                },
                plotOptions: {
                  line: {
                       dataLabels: {
                          color: '#CCC'
                       },
                       marker: {
                          lineColor: '#333'
                       }
                    },
                    series: {
                        lineWidth: 2,
                        color: '#FF0000',
                        point: {
                            events: {
                                'click': function() {
                                    if (this.series.data.length > 1) this.remove();
                                }
                            }
                        }
                    }
                },
                series: [{
                    data: [[0, 0]]
                }]
            });
        });
        
    });
</script>

<form onsubmit="return false;">
    <input type="hidden" name="message" value='draw'/>
    <input type="button" value="Draw" onclick="send(this.form.message.value)"/>
</form>
<div id="container" style="min-width: 400px; height: 400px; margin: 0 auto"></div>
</body>
</html>

WebSocketServer.groovy

package websockets

def halfpart = []
halfpart.add([-2,25])
halfpart.add([-3,35])
halfpart.add([-4,45])
halfpart.add([-6,55])
halfpart.add([-7,65])
halfpart.add([-7,80])
halfpart.add([-6,90])
halfpart.add([-5,97])
halfpart.add([-4,100])
halfpart.add([-3,97])
halfpart.add([-2,90])
halfpart.add([-1,82])
halfpart.add([-0,75])

def drawing = []

halfpart.each{ drawing << it }
drawing.pop()

halfpart.reverse().each{ drawing << [-it[0],it[1]] }
drawing << [0,0]

println(drawing)

def counter = 0

vertx.createHttpServer().websocketHandler { ws ->
  ws.dataHandler { data ->
  		def location = drawing[counter%drawing.size()]
  		def template = """{"x":"${location[0]}","y":"${location[1]}"}""" 
  		ws.writeTextFrame(template)
  		counter++
}
}.requestHandler { req ->
  if (req.uri == "/") req.response.sendFile "highcharts/charts.html"
  if (req.uri == '/jquery.js') req.response.sendFile('highcharts/jquery-1.7.2.js')
  if (req.uri == '/highcharts.js') req.response.sendFile('highcharts/highcharts.js')
}.listen(8080)

Stomp, Bootstrap and ActiveMQ meet for realtime HTML5 directed by Camel

Sun, 13 May 2012 21:45:00 +0200

About the actors of this story :

Apache ActiveMQ 5.6.0 and the new stomp connector, “the master”
Apache Camel : “the great conductor” !
Stomp.js, connecting directly javascript to ActiveMQ, “the shadow worker”
Twitter Bootstrap.js, “the beautifull babe

update : after some magic HTML5 canvas function, you get this asynchronous, loosely tied, event-based rendering.

Steps to complete :

Look how to active the stomp component here
Start ActiveMQ in the bin directory, you can test your installation by pointing to http://localhost:8161/
Create the Html template, for brevity I use a static one but you can deploy it into your container ( Apache, Node.js, Vertx, Buzz word here )
Setup the Javascript that combine the listening of Stomp event and BootStrap
Faking stocks values using Camel, you can check my previous blog post on how to get real ones

1.html template

The template is empty and do nothing except the layout and loading the js and css ressources.

<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”>

<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.01 Transitional//EN” “http://www.w3.org/TR/html4/loose.dtd”>

<html>

<head>

<meta http-equiv=”Content-Type” content=”text/html; charset=iso-8859-1” />

<title>Chat Example Using Stomp Over WebSocket</title>

<link rel=”stylesheet” type=”text/css” href=”css/docs.css”></link>

<link rel=”stylesheet” type=”text/css” href=”css/bootstrap.css”></link>

<link rel=”stylesheet” type=”text/css” href=”css/bootstrap-responsive.css”>

<script type=”text/javascript” src=”js/jquery-1.7.2.js”></script>

<script type=”text/javascript” src=’js/stomp.js’></script>

<script type=”text/javascript” src=’js/quotes.js’></script>

<script type=”text/javascript” src=’js/bootstrap.js’></script>

<script>

$(document).ready(function() {

var supported = (“WebSocket” in window);

if(!supported) {

var msg = “Your browser does not support Web Sockets. This example will not work properly.<br>”;

msg += “Please use a Web Browser with Web Sockets support (WebKit or Google Chrome).”;

$(“#connect”).html(msg);

}

});

</script>

</head>

<body>

<div class=”container-fluid”>

<div class=”row show-grid” id=”quotes”>

<!— Quotes templates will append here—>

</div>

</div>

<div id=”disconnect”>

<form id=’disconnect_form’>

<input type=submit id=’disconnect_submit’ value=”Disconnect”>

</form>

</div>

</body>

</html>

2 Quotes.js

On event, check the json object that is received and look at the quote attribute then if this attribute exist as and id in the html update his children fields or create it.

$(document).ready(function(){

var client, destination;

var url = ‘ws://localhost:61614/stomp’;

var login = ‘guest’;

var passcode = ‘guest’;

destination = ‘/topic/messages’;

client = Stomp.client(url);

// this allows to display debug logs directly on the web page

client.debug = function(str) {

$(“#debug”).append(str + “\n”);

};

// the client is notified when it is connected to the server.

var onconnect = function(frame) {

client.debug(“connected to Stomp”);

$(‘#disconnect’).fadeIn();

$(‘#send_form_input’).removeAttr(‘disabled’);

client.subscribe(destination, function(message) {

var j = jQuery.parseJSON(message.body);

if(!$(‘#’+j.quote).length) {

alert(“not founded”);

$(‘#quotes’).append(‘<div id=”’+j.quote+’” class=”span2”><h5>’+j.quote+’</h5><div class=”quote_open”>-</div><div class=”quote_hi”>-</div><div class=”quote_low”>-</div><div class=”quote_close”>-</div></div>’);

}

$(‘#’+j.quote).find(‘.quote_open’).html(j.open)

$(‘#’+j.quote).find(‘.quote_hi’).html(j.hi)

$(‘#’+j.quote).find(‘.quote_low’).html(j.low)

$(‘#’+j.quote).find(‘.quote_close’).html(j.close)

});

};

client.connect(login, passcode, onconnect);

$(‘#disconnect_form’).submit(function() {

client.disconnect(function() {

$(‘#disconnect’).fadeOut({ duration: ‘fast’ });

});

return false;

});

});

3. Camel Code

Generate fake values and send them to the queue. You can write this part in any languages you want, ruby, javascript,.net. That’s why it’s called an integration tool. Adding a value to this topic (and not queue) share it with all the stomp.js instances that are listening. Now you only have to replace this part with your true data stream.

import java.util.Random;

import org.apache.camel.CamelContext;

import org.apache.camel.Exchange;

import org.apache.camel.Processor;

import org.apache.camel.builder.RouteBuilder;

import org.apache.camel.impl.DefaultCamelContext;

public class YahooFeed {

/**

* @param args

*/

public static void main(String[] args) {

CamelContext ctx = new DefaultCamelContext();

try {

ctx.addRoutes(new RouteBuilder() {

@Override

public void configure() throws Exception {

from(“quartz://sources/fake?cron=0/5+*+*+*+*+?”).process(new YahooFeed.FakeProcessor()).to(“activemq:topic:messages”);

}

});

ctx.start();

} catch (Exception e) {

e.printStackTrace();

}

}

static class FakeProcessor implements Processor {

private final Random rdm = new Random();

private final String[] stocks = new String[] { “OCZ”, “RHT”, “FIO”,

“AMZN”, “XIO” };

@Override

public void process(Exchange ex) throws Exception {

String share = stocks[rdm.nextInt(stocks.length)];

int low = rdm.nextInt(100) + 10;

int hi = low * 3;

int open = low * 2;

int close = open + rdm.nextInt(20);

String rs = “{"quote":"” + share + “","open":"” + open

+ “","hi":"” + hi + “","low":"” + low + “","close":"”

+ close + “"}”;

System.out.println(“generated “+rs);

ex.getIn().setBody(rs);

}

}

}

And that’s it. Meteor.js, Node.js, Vert.X, Play2 are all new tools promoting new coding style. It doesn’t matter your choice! Each language/framework teams provide the right tools that implements the right concept. Choose the one you are confortable with and share the knowledge.

Disclosure: ‘am Long OCZ ;)

Putting Jasig-Cas on Cloud [Cloudbees]

Thu, 02 Feb 2012 19:31:00 +0100

Jasig Cas stand for Central Authentication Service. It’s a webapp that allows Single Sign On/Out for all your applications. It can be coupled to ldap, jdbc, X.509, etc. You can find more on authentification modules here.

What’s the meaning ? in fact in the cloud you will deploy a lot of applications each one representing a service for your compagny. Should the user identify itself when he use the CRM and then login again when using the ERP ? CAS is the solution.

CAS provide a single login window which is displayed when you tried to access a service for the first time. It memorize the Url of the service you want to access, check your credentials, generate a ticket and send it back to the service. The next service ask Cas for a ticket when you tried to reach it and if found you are granted or send back to the login.

In this tutorial i’ll show you how to setup a basic Cas server on Cloudbees and how to fill credential in a simple file. You’ll be able to configure it on your own using this base.

Preparations

Download the Cas server implementation here
Install Maven if not already installed.

Getting Started

Unzip the Cas server distribution

Prepare the file identification module

Cas is very modulable, you can load only the part that is required to your project. The generic module can load identification from a XML declaration.

Enter the directory cas-server-3.4.XX\cas-server-support-generic
Type : mvn install to start the building of the module
Get the cas-server-support-generic-XXX.jar in the target directory just created

Prepare the web application

The webapps contains the core of the Cas server, it load authentification modules, display the login page which you can brand to your compagny and allow you to configure the services that can be CASified.

Enter the directory cas-server-3.4.XX\cas-server-webapp
Type : mvn install to start the building of the web archive
War are just zip file, extract the archive, i use 7zip
In the Web-Inf/lib, paste the cas-server-support-generic-XXX.jar

Configure the authentification

Define the admin

Open the file deployerConfigContext.xml and found the section

<sec:user-service id=”userDetailsService”>

<sec:user name=”@@THIS SHOULD BE REPLACED@@” password=”notused” authorities=”ROLE_ADMIN” />

</sec:user-service>

Replace @@THIS SHOULD BE REPLACED@@ with a new user : itsme

this user now have the role of Admin, once identified it can control the cas server.

Configure the authentification handler : source of credentials

In the same file replace the section

with this one

<map>

</map>

</property>

</bean>

Create the CloudBees environnement

Create a free account and quickly check http://wiki.cloudbees.com/bin/view/RUN/GettingStarted

Configure the environnement

Open the file WEB-INF/cas.properties
Configure server.prefix= http://cas.your_account.cloudbees.net

War

Now archive all the directory as a zip file and change the extension from .zip to .war. Take care to respect the deployment tree ! the zip contains WEB-INF not another sub directory !!!

UPLOAD !!!

Fast way : upload your file to cloudbees using the application manager
Production : open a dev@cloudbees.com and use git with ssh

And that’s it ! the server is ready. You can now point your browser to your application : http://cas. your_account .cloudbees.net

Configure the Cas services

Point your browser to http://cas. your_account .cloudbees.net/services and login using itsme/cantremember. Now you have access to the services control panel where you can configure all your CASified applications.

If you are starting with cas you can find CASified applications here:

https://wiki.jasig.org/display/CASUM/End-to-end+Windows+Example#End-to-endWindowsExample-DeploytheProtectedApplications

What’s really great for me is that this app is on the cloud, there’s is few possibilities of getting down and it extract the complexity of authenfication from my services.

Notes

Don’t know why, i got a lot of errors with local and file not found when trying to deploy on Jelastic which is also a great platform. If someone can provide indications, it’s more than welcome.

Integration of Trading Technologies XTrader with ActiveMQ

Thu, 02 Feb 2012 15:29:00 +0100

Trading Technologies XTrader is one of the leading platform in finance. It offers a lot of services, orders on Fx, Future, Options, etc. XTApi is the .ddl Api that connect directly to Gardian, the sublayer of XTrader and can be used to interface with your SI.

Microsoft languages are a long time legacy in this domain but your SI may be written using others techologies (Java, Scala, etc). There’s several ways to do the mix, you may choose databases, tools like Protobuffer or Thrift but for this one you better choose ApacheMQ for it’s combo with Camel.

Apache NMS is a sub project which allow you to send messages using C# but the main advantage is that with queues you can now setup advanced treatment pattern like pubsub or parallel distribution of the informations. When this is implemented all the project can be driven using Apache Camel rules.

Here’s a default implementation to help you start :

https://github.com/Solido/TTCamel

My free Zen Widget on Android

Sat, 14 Jan 2012 19:51:00 +0100

I’ve setup a zen sound widget on Android, get it for free here.

Azra le 6 octobre 2011

Perfect! I love this bell! Sounds just like an actual bell I own.

EVE le 14 mai 2011

Simple & calming widget, thanks.

julien le 9 mars 2011

Great app, very relaxing.

Paul le 9 mars 2011

Tip top la cloche!

joseph le 17 janvier 2011

Amazing sounding bell. Very long sustain.

Combine Yahoo Finance and HBase using Camel and Rest

Thu, 05 Jan 2012 12:16:00 +0100

Update : this article was published on the Articles on Camel section in the official Camel wiki. Thank you Claus !

Update : promoted on DZone : http://www.dzone.com/links/users/saved/964191.html !

Here’s our program :

How to get the quotes from Yahoo
Start HBase and Stargate (HBase using Rest)
Define HBase schema
Accessing Stargate
Camel
Conclusion

1: How to get the quotes from Yahoo

First we need to configure the yahoo url that return the quotes, there are many ways to get them. One is the configurable URL, see here.

Since i’m gonna works on OCZ and AMZN my url will look something like this:

http://download.finance.yahoo.com/d/quotes.csv?f=sl1d1t1ohgdr&s=OCZ+AMZN&throwExceptionOnFailure=false

If you put this URL in your browser a file quotes.csv will start to download and it shoud contains that :

“OCZ”,”OCZ Technology Gr”,6.85,”1/3/2012”,”4:00pm”,6.77,7.00,6.72,0.00,N/A

“AMZN”,”Amazon.com,Inc.”,179.03,”1/3/2012”,”4:00pm”,176.10,179.475,175.55,0.00,91.25

2: Start HBase and Stargate (HBase using Rest)

First you need a linux distribution, Hbase on windows is like hell ;)

Next it’s still hello because HBase will not be compatible with the last Mint 12 or Ubuntu. My setup use Ubuntu 10 LTS.

You can find how to install and start HBase in their getting started with HBase.

Start HBase: % ./bin/start-hbase.sh

The HBase Wiki team has wrote a complete article on how to start and test Stargate the bundle that brigde HBase and Rest. You can find how to setup it and test it using the curl command here.

Start Stargate : % ./bin/hbase rest start -p 8484

3: Define HBase schema

After starting the server you can access the shell using this command :

% ./bin/hbase shell

Designing schema in NoSql is a complete study and I cannot treat it here so i’m only suggesting this schema as a starter.

Create the table quotes and the first column family :

hbase(main):001:0> create ‘quotes’, ‘time’

And you can test a manual insert using this :

hbase(main):002:0> put ‘quotes’, ‘OCZ’, ‘time:20120103’, 6.80

And query all the quotes using :

hbase(main):003:0> scan ‘quotes’,

All this means that you can now query HBase using Http but also insert values using the POST method has we will see later.

4: Accessing Stargate

Now you can open you browser and check what’s going on !

pointing it to http://[hbase-server-host]:8484 will list you all available table

and http://[hbase-server-host]:8484/quotes/OCZ gonna list all the entries of quotes.

5: Camel

And now let enter the Maestro ! Camel !

Camel is a wonderful tool, it emerged from ActiveMQ and is now a top project of Apache. It’s main goal is the integration of services and that’s what we get. HBase, Http request, Yahoo, all will be put in a perfect dance.

Let’s start with the complete camel code and let’s dig into the details :

YahooFinanceRoute.java

import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;

import com.ncp.processors.YahooParserProcessor;
import com.ncp.processors.YahooFinanceProcessor;

public class YahooFinanceRoute extends RouteBuilder {

	@Override
	public void configure() throws Exception {
		from("quartz://sources/yahoo_finance?cron=0+0+8+?+*+MON-FRI")
		.process(new YahooFinanceProcessor())
		.to("ahc:activate_yahoo")
		.convertBodyTo(String.class) 
		.split(body(String.class).tokenize("\n"))
		.process(new YahooParserProcessor()) 
		.transform(simple("${in.headers.Last}"))
		.setHeader(Exchange.HTTP_URI, constant("http://hbase-server:8484/quotes/{in.headers.Instr}/time:{in.headers.Time}"))
		.setHeader(Exchange.HTTP_METHOD, constant("POST"))
		.setHeader(Exchange.CONTENT_TYPE, constant("application/octet-stream"))
		.to("ahc:activate_hbase");
	}

}

The event style programming :

Start a event every day from monday to friday at 8 AM

from("quartz://sources/yahoo_finance?cron=0/8+*+*+*+*+?")

Use a processor to configure the URL using required instruments code

.process(new YahooFinanceProcessor())

Use the async camel http component call the configured url

.to("ahc:activate_yahoo")

The component will return a ByteArrayInputStream that need to be converted to a String

.convertBodyTo(String.class)

Split the content to camel messages

.split(body(String.class).tokenize("\n"))

Parse each instrument line and store required informations in the exchange headers

.process(new YahooParserProcessor())

Transform the header of the message as the Body to be send in the Post request

.transform(simple("${in.headers.Last}"))

Camel offers us a subtil tool as Simple, a language which can query the Exchange and offers others services. See more here. For this part of the task I used simple to demonstrate how usefull it can be but you can also move all this part in a Processor if you want.

Configure the request to be send as a POST, don’t forget the content-type header.

.setHeader(Exchange.HTTP_URI, constant("http://hbase-server:8484/quotes/{in.headers.Instr}/time:{in.headers.Time}"))
.setHeader(Exchange.HTTP_METHOD, constant("POST"))
.setHeader(Exchange.CONTENT_TYPE, constant("application/octet-stream"))

Send the orders to the Stargate !

.to("ahc:activate_hbase");

YahooFinanceProcessor.java

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.commons.lang.StringUtils;

public class YahooFinanceProcessor implements Processor {
	
       // just add the Yahoo Instrument code here
	final String[] quotes = new String[]{"OCZ","AMZN"};
	final String url_quotes = StringUtils.join(quotes, "+");

	@Override
	public void process(Exchange exchange) throws Exception {
		exchange.getIn().setHeader(Exchange.HTTP_URI, "http://download.finance.yahoo.com/d/quotes.csv?f=sl1d1t1ohgdr&s="+url_quotes+"&throwExceptionOnFailure=false");
	}
}

YahooParserProcessor .java

import org.apache.camel.Exchange;
import org.apache.camel.Processor;

public class YahooParserProcessor implements Processor {

	@Override
	public void process(Exchange exchange) throws Exception {
		String[] datas = exchange.getIn().getBody(String.class).split(",");
		SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");
		exchange.getIn().setHeader("Instr", datas[0].replaceAll("\"", ""));
		exchange.getIn().setHeader("Last", datas[1]);
		exchange.getIn().setHeader("Time", sdf.format(new Date()));
	}

}

6: Conclusion

This conclude our integration, you can now setup your project and decide how-to start the route. For my own needs I use Spring. Don’t forget to setup all the required jars !

CloudExplorer

Sat, 31 Dec 2011 04:18:00 +0100

Update : this article was published in JavaFX links of the week, January 2 in the reference site fxexperience.com. Thank you Jonathan !

Since i wanna jauge JavaFX2, i’ve written a quick and dirty scala framework for charting. It simply parse a csv file and can draw it as a point cloud or a chart. It offers :

Interactive data selection
Multiple selection and deselection
Grouping ( select points and then Ctrl+num[0-9]) and recall of group
Pan
Zoom (disabled as Oracle removed support for wheel but active in the code)
AutoGrid

The two examples commands are : com.numbersam.cloud.ui.Cloud and com.numbersam.cloud.ui.Density

GitHub : https://github.com/Solido/CloudExplorer

JavaFX2 is a wonderfull tool ! it’s only opponent is Html5 Canvas but the framework is much more advanced. I guess than soon html will enjoy such powerfull tool.

HBase and the wrong base

Sat, 31 Dec 2011 02:54:00 +0100

HBase is one the implementation of the Google BigTable concept and one of the most interesting project in my eyes. It offers storage for bigdata and a more reconfigurable datastructure. The main problem came from the installation. Since it require ssh, on windows even for testing is quite a mess and now i’ve discovered that’s it’s not compatible with Ubuntu 11.xx. The only configuration I can get it works is Ubuntu 10.04 LTS and HBase 0.90.x.