Research & Development
Presented in the IEEE SoutheastCon 2015
Finding Interesting Outliers - A Belief Network based Approach
Abstract: Outliers are deviations from the usual trends of data; to discover interestingness among outliers i.e. finding anomalies which are of real-interest for subject matter experts is an active area of research in data mining and maching learning community. Due to its subjective nature, the definition of what amounts to ’interesting’ varies between domains and subject matter experts. In this research, we explore the quantification for measures of interestingness, using Bayesian Belief Networks as background knowledge. Mining outliers may help discover potential anomalies and fraudulent activities. Meaningful outliers can be retrieved and analyzed by using domain knowledge. Domain knowledge (or background knowledge) is represented using probabilistic graphical models such as Bayesian belief networks. Bayesian networks are graph-based representation used to model and encode mutual relationships between entities. Due to their probabilistic graphical nature, Belief Networks are an ideal way to capture the sensitivity, causal inference, uncertainty and background knowledge in real world data sets. Bayesian Networks effectively present the causal relationships between different entities (nodes) using conditional probability. This probabilistic relationship shows the degree of belief between entities. A quantitative measure which computes changes in this degree of belief acts as a sensitivity measure. In this research paper we provide an overview of interestingness measures, their use to measure sensitivity in belief networks and review the earlier work on so-called Interestingness Filtering Engine. Building upon these foundation, we introduce our algorithm IBOX - Interestingness based Bayesian Outlier eXplainer, which provides progressive improvement in the performance and sensitivity scoring of the earlier works. IBOX provides an iterative model to use multiple interestingness measures resulting in better performance and improved sensitivity analysis. The approach quantitatively validates probabilistic interestingness measures as an effective sensitivity analysis technique in rare class mining.
Topic Category: Data Mining and Machine Learning
Download Paper - NL-ESB - A Negative Latency Enterprise Service Bus
In a recent podcast by Scott Hanselman and Erica Stanley, an Internet of Things (IoT) primer, the guest mentioned how security is being treated as an afterthought for most things IoT. This is unfortunately true in various areas of software development; but especially with the unprecedented growth of IoT, this lax in providing security standards will fast become a safety and security dilemma.
To borrow the variety, velocity and volume analogy of Big Data, IoT is also subject to a very large variety of devices, supporting different velocities (performance capacities) and volumes (large number of devices, meshes etc). Protection of data in these devices and providing privacy of is definitely the key challenges in the IoT. It is also bad for business since lax security measures will cause decreased adoption impacting the success of the IoT and hinder overall development.
Following are some of the relevant links and papers which provide overview, analysis and taxonomy of security and privacy challenges in IoT.
References and Further Reading
- Erica's talk on The Internet of Things from the All Things Open Conference
- Proposed Security Model and Threat Taxonomy for the Internet of Things (IoT)
- Future Internet: The Internet of Things Architecture, Possible Applications and Key Challenges
- Internet of Things Demands Security by Design
- Internet of Things (IOT): Seven enterprise risks to consider
- SECURITY IN THE INTERNET OF THINGS Lessons from the Past for the Connected Future
- Cisco Security Products for the IoT
- IoT security: How to do it (mostly) right
- Developers Discuss IoT Security And Platforms Trends
- Understanding The Protocols Behind The Internet Of Things
- The Nest
- Intel Edison
- The Internet Of Things Is A Standards Thing
Penetration Testing techniques in web applications by Dimitris Mandilaras, Nikolaos Tsalis is an succinct info-graphic review of different security frameworks / methodologies including OWASP, PTES, ISSAF, NIST, OSSTM and PTF.
A short poster can be downloaded from here.
- Functional Programming For All! Scaling a MOOC for Students and Professionals Alike
- Reactive Web Applications with Dynamic Dataflow in F# A Tayanovskyy, S Fowler, L Denuzière, A Granicz - ifl2014.github.io
- The F# Computation Expression Zoo Tomas Petricek, Don Syme
- Thinking in LINQ: Harnessing the Power of Functional Programming in .NET ... By Sudipta Mukherjee
- Clash of the Lambdas by Aggelos Biboudis, Nick Palladinos, Yannis Smaragdakis
- Concurrency in Intrusion Detection Systems: A Study in F# by Deines, Jessica
- Functional Thinking: Paradigm Over Syntax By Neal Ford
- Experience in using a typed functional language for the development of a security application Damien Doligez (Inria), Christèle Faure (SafeRiver), Thérèse Hardin (UPMC), Manuel Maarek (SafeRiver)
Here is Prof. Hastie's recent talk from the H2O World conference. In this talk, professor Hastie takes us through Ensemble Learners like decision trees and random forests for classification problems.
Other excellent talks from the conference include the following.
- Michael Marks - Values and Art of Scale in Business
- Nachum Shacham of Paypal - R and ROI for Big Data
- Hassan Namarvar, ShareThis - Conversion Estimation in Display Advertising
- Ofer Mendelevitch, Hortonworks - Bayesian Networks with R and Hadoop,
- Sandy Ryza, Cloudera - MLlib and Apache Spark
- Josh Bloch, Lord of the APIs - A Brief, Opinionated History of the API
- Macro and Micro Trends in Big Data, Hadoop and Open Source
- Competitive Data Science Panel: Kaggle, KDD and data sports
- Practical Data Science Panel
The complete playlist can be found here.
Abstract. Functional programming is often taught at universities to first-year or second-year students and most of the teaching materials have been written for this audience. With the recent rise of functional programming in the industry, it becomes important to teach functional concepts to professional developers with deep knowledge of other paradigms, most importantly object-oriented. We present our experience with teaching functional programming and F# to experienced .NET developers through a book Real-World Functional Programming and commercially offered F# trainings. The most important novelty in our approach is the use of C# for relating functional F# with object-oriented C# and for introducing some of the functional concepts. By presenting principles such as immutability, higher-order functions and functional types from a different perspective, we are able to build on existing knowledge of professional developers. This contrasts with a common approach that asks students to forget everything they know about programming and think completely differently. We believe that our observations are relevant for trainings designed for practitioners, but perhaps also for students who explore functional relatively late in the curriculum.
Honorable mention to A Look at F# from C#’s corner
Following the great scholarly acceptance and outstanding academic success of "The Clairvoyant Load Balancing Algorithm for Highly Available Service Oriented Architectures, this year I present P Not Equal to NP - A Definitive Proof by Contradiction.
Click here to read the entire paper in PDF. P Not Equal to NP - A Definitive Proof by Contradiction.