About the Project

Project: Know before your customer knows: Monitoring trends to predict failures in the cloud

Here, we explain the project in detail. Please feel free to use the code and deploy in Bluemix using your own account.

First things first: Lets set up your Bluemix accounts! http://ace.ng.bluemix.net This will take care of your Sign Up and Log In into the Bluemix portal.

Get to know our App This app is to demonstrate Real time production Log analysis for proactively finding problems in your cloud operations. Your web server logs and user's activity logs have critical information on system and user's behavior and must be used wisely to analyse and possibly predict failures such as:

User visits and activities
Potential hacking attempts
Finding anomaly in the system usage metrics

Application architecture

Application Workflow

Producers:

We are Cloud Data Services team and provide a suite of data and analytic services such as dashDB, DataWorks, Cloudant, BigInsight and more in IBM Bluemix PaaS. Developers use our services to quickly compose enterprise grade web and mobile apps. In the flow above, producers represent some of our production servers pushing the logs and error events to Kafka via a log producer app in real time. We are using two topics called Logging and Activity in this case. We have used Node-red platform in Bluemix to create a programming workflow.

Producers API EndPoint:

This is an app that provides API end point for producers to publish logs and events to Kafka.

Kafka Event Hub:

This event hub is a hosted service. We have also created a Node for the Node-Red platform so developers can easily flow the event and consume them at the same time. The npm package for the node can be found here: https://www.npmjs.com/package/node-red-contrib-kafka-node You can just add this dependency in the package.json of your node-red folder and you will be all set.

Consumers:

We have used dashDB to consume events and R statistical model to analyze and predict outcomes. dashDB is an analytic service used in Bluemix, it provides integrated R and R-studio environment for developers to perform In-app and In-database analytics. A dashDB node also exists in node-red environment so that you can easily consume data in the same application workflow. Here are the results of some of the statistical analysis in R: Total # of visits per hour Potential Hack Attempts Anomaly in Error Types