Machine Learning

LA Machine Learning event on Mining Time Series Data w/ Sylvia Halasz

75 days
by in Algorithms, Events, Machine Learning

Last night's LA Machine Learning event on Mining Time Series Data w/ Sylvia Halasz of YP at OpenX Pasadena was quite interesting and well attended. Dr. Halasz spoke about Adaptive Ensemble Kalman Filter and her work on building n-gram correlation with the flu outbreaks. Some of the associated papers follow.

 

IMG00312-20130306-1935

Share

Causality, Probability, and Time - A Temporo-Philosophical Primer to Causal Inference with Case Studies

96 days
by in Algorithms, Bayesian, Books, Machine Learning, Research & Development

Causality, Probability and Time by Dr. Samantha Kelinberg is a whirlwind yet original journey of the interdisciplinary study of probabilistic temporal logic and causal inference. Probabilistic causation is a fairly demanding area of study which studies the relationship between cause and effect using the tools of probability theory. Judea Pearl, in his seminal text "Causality: Models, Reasoning, and Inference" refers to this quandary by stating that

(causality) connotes lawlike necessity, whereas probabilities connote exceptionality, doubt, and lack of regularity.

 

Dr. Kelinberg's work provides a balanced introduction to background work on this topic while breaking new grounds on a well-positioned approach of causality based on temporal logic. The envisioning problem is the problem of deducing the set of facts, possibly as the result of our actions leading to the decision problem. This is compounded with finding a timely and useful way to represent our knowledge about time, change, and chance.

CPT_cover

In this ~260 page book, Dr. Kelinberg begins with a brief history of causality leading to Probability, logic and probabilistic temporal logic. The author then defines causality from various different facets, proceeding to causality inference, token causality and then finally the case studies. With practical examples and algorithms, author devises simple mathematical tools for analyzing the relationships between causal connections, inference, causal significance, model complexity, statistical associations, actions and observations.

Exploiting the temporal nature of probabilistic events, Dr. Kelinberg's research is a thought provoking and valuable addition to the scientific community interested in learning causal effects and inference with respect to time. Built upon the works of the likes of Heckerman, Breese, Santos and Young, this book will pave the way probabilistic reasoning researchers think about temporal effects on causality for years to come.

David Hume believed that the causes are invariably followed by their effects: "We may define a cause to be an object, followed by another, and where all the objects similar to the first, are followed by objects similar to the second." So, would you like a well written margin-annotation-laden text which provides formal and practical case study based approach to this somewhat abstract concept of causality? Then look no further!

Share

Bayesian Network Repositories Collections

142 days
by in Bayesian, Machine Learning, Research & Development, Uncategorized

A #NoteToSelf style post regarding collection of bayesian network repositories including but not limited to bnet, net, bif, dsc and rda files.

Share

The Theory That Would Not Die - An Engaging History of Bayesian Philosophy

146 days
by in Bayesian, Books, Machine Learning, Research & Development

As statistician Dennis Lindley famously said, "Inside every nonBayesian there is a Bayesian struggling to get out"; it would be safe to interpolate that Sharon McGrayne's interesting tale of trials and triumph of the Bayes Rule, or more accurately Bayes-Laplace-Price rule, is an excellent historical journey, which may help get your Bayesian out of the closet.

The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy makes for an interesting and captivating read especially considering that writing about history of mathematics and statistics for general audience is a daunting task when compared with relatively popular topics like astronomy or physics. In this easy reading for popular-science audience, author covers over three hundred years of the history behind Bayes rule with its applications and engrossing stories of mathematical luminaries; some of which thought it was a brilliant way to model real-life scenarios while others considered it unscientific, an exercise in futility and vehemently fought against the idea of incorporating prior beliefs. Aside from providing thorough research on the subject matter, this text also delves into significant details about life and works of important scientists, mathematicians and statisticians including but not limited to Turing, Von Neumann, Price, Shannon, Bailey, Laplace, Fisher and Feynman. Regarding modern times, I was delighted to see Daphne Koller and Heckerman's work mentioned as well as the role Bayesian techniques played in contemporary discipline of Machine learning.

The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy

Starting with the compelling statement

When the facts change, I change my opinion. What do you do, sir?
—John Maynard Keynes

the ups and downs of adoption of Bayesian rule are listed as different eras and separated out as different parts of the book. The 17 chapters are divided into five parts namely Enlightenment and the anti-Bayesian reaction, Second World War era, the glorious revival, to prove it's worth and finally, victory. Did author do a good job explaining Bayes rule is the point of contention among earlier reviews. I agree that a few more concrete examples with algebraic expressions may have helped better explaining how Bayesian priors and it's mathematical formulation by early luminaries in the field makes it easy to work without complex integrals. However, it is to be noted that this book is not a course in antiquity of causality and inference but rather a study of Bayesian thought through centuries and it's profound impact on science and technology. The book very well covers the advances by 'Bayesian revolution' in variety of fields including medical diagnosis, ecology, geology, computer science, artificial intelligence, machine learning, genetics, astrophysics, archaeology, education performance, sports modeling, and more.

P(A|B)=\frac{P(B|A) P(A)}{P(B)}

Sharon McGrayne's has picked a very relevant topic for contemporary audience interested in mathematical and computational sciences; making this ~350 page book a very informative, absorbing and pleasurable reading. Although light on technical details, proofs, mathematical equations and problems, this book delivers what it sets to accomplish, to tell the story of Bayes theory. "The theory that would not die" tells the story of a robust idea which is simple, intuitive, unsettling to establishment and yet so resilient that despite of all the criticisms from mainstream frequentists, it stayed alive and well. To quote from the book

"Bayes is still young. Probability did not have any mathematics in it until 1700. Bayes grew up in data- poor and computationally poor circumstances. It hasn't settled down yet. We can give it time. We're just starting."

Share

A Deep Dive into Causality with Judea Pearl

163 days
by in Books, Machine Learning, Research & Development

For most researchers in the ever growing fields of probabilistic graphical models, belief networks, causal influence and probabilistic inference, ACM Turing award winner Dr. Judea Pearl and his seminary papers on causality are well-known and acknowledged. Representation and determination of Causality, the relationship between an event (the cause) and a second event (the effect), where the second event is understood as a consequence of the first, is a challenging problem. Over the years, Dr. pearl has written significantly on both Art and Science of Cause and Effect. In this book on “Causality: Models, Reasoning and Inference”, the inventor of Bayesian belief networks discusses and elaborates on his earlier workings including but not limited to Reasoning with Cause and Effect, Causal inference in statistics, Simpson's paradox, Causal Diagrams for Empirical Research, Robustness of Causal Claims, Causes and explanations, and Probabilities of causation Bounds and identification.

In these eleven chapters followed by an epilogue, Dr. Pearl’s manuscript postulates representational and computational foundation for the processing of information under uncertainty. It commences with introduction of simpler concepts in Bayesian inference, causality and corresponding proves. However, as text progresses into causal vs. statistical concepts along with theory of inferred causation, the theorems get arduous, somewhat counter-intuitive and the text becomes demanding to keep up. Chapter 3 is an interesting read where causality is discussed in context of philosophy and history. As Dr. Liu states, Judea Pearl’s thesis regarding statistics that it deals with quantitative constructs like mean, variance, correlation, regression, dependence, conditional independence, association, likelihood, collapsibility, risk ratio, odd ratio, marginalization, conditionalization, etc. Meanwhile the causal analysis deals with the topics of randomization, influence, effect, confounding, disturbance, correlation, intervention, explanation and attribution. One of the challenges while following Dr. Pearl’s work is that it abstracts causation discussing it in mathematical and philosophical manner without providing concrete mathematical and computational model for applied research. I believe the book provides great foundation for formal representation of causal analysis and its components, such as do(x) to represent intervention.

Automated Reasoning Group at UCLA has made some strides in this area however the applied research aspects of this formalism still needs to be ‘tightly bound’ by reason of scarcity of empirical evidence for the algorithms in practice.

Share

Caltech Entrepreneurs Forum Event – Big Data, Big Opportunities: Slides & Pictures

170 days
by in Events, Machine Learning, Research & Development

Recently attended Big Data Event @ Caltech. The topic was Big Data, Big Opportunities: Predicting the Future One Byte at a Time and the panel and speakers didn't disappoint. Following is the slidedeck and pictures from the event.

Share

A Truly Modern discourse in Bayesian Reasoning and Machine Learning

171 days
by in Bayesian, Books, Machine Learning, Research & Development

If you are scouring for an exploratory text in probabilistic reasoning, basic graph concepts, belief networks, graphical models, statistics for machine learning, learning inference, naïve Bayes, Markov models and machine learning concepts, look no further. Dr. Barber has done a praiseworthy job in describing key concepts in probabilistic modeling and probabilistic aspects of machine learning. Don’t let the size of this 700 page, 28 chapter long book intimidate you; it is surprisingly easy to follow and well formatted for the modern day reader.

 

With excellent follow ups in summary, code and exercises, Dr. David Barber a reader at University college London provides a thorough and contemporary primer in machine learning with Bayesian reasoning. Starting with probabilistic reasoning, author provides a refresher that the standard rules of probability are a consistent, logical way to reason with uncertainty. He proceeds to discuss the basic graph concepts and belief networks explaining how we can reason with certain or uncertain evidence using repeated application of Bayes' rule. Since belief network, a factorization of a distribution into conditional probabilities of variables dependent on parental variables, is a specific case of graphical models, the book leads us into the discipline of representing probability models graphically. Followed by efficient inference in trees and the junction tree, the text elucidates on key stages of moralization, triangularization, potential assignment, and message-passing.

I particularly enjoyed the follow up chapter called statistics for machine learning which uniquely discuss the classical univariate distributions including the exponential, Gamma, Beta, Gaussian and Poisson. It summarizes the measure of the difference between distributions, Kullback-Leibler divergence and states that Bayes' rule enables us to achieve parameter learning by translating a prior parameter belief into a posterior parameter belief based on observed data. Learning as inference, naïve bayes, Learning with Hidden Variables and Bayesian model selection is followed by machine learning concepts. I found the sequence of chapters to be a bit off (shouldn’t graphical models be discussed before a specific case?) but since the book is more rooted in practice than an exercise in theorem-proving’s, the order ultimately makes sense.

Since Nearest neighbor methods are general classification methods, the book continues with conditional mixture of guassian into Unsupervised Linear Dimension Reduction, supervised linear dimension reduction, kernel extensions, guassian processes, mixture models, latent linear models, latent ability models, discrete and continuous state markov models eventuating to distributed computation of models, sampling and the holy grail of Deterministic Approximate Inference.

One of the many great things about this book is the practical and code oriented approach; tips with applied insight like “Consistency methods such as loopy belief propagation can work extremely well when the structure of the distribution is close to a tree. These methods have been spectacularly successful in information theory and error correction.” makes this text distinguished and indispensable.

Share

Selected Papers on Interestingness Measures, Knowledge Discovery and Outlier Mining

188 days
by in Algorithms, Bayesian, Machine Learning, Research & Development
  • S.   Hettich    and   S.  D.   Bay.   Kdd   cup   1999  data.       UCI   KDD   Archive
    [http://kdd.ics.uci.edu/      /databases/kddcup99/kddcup99.html],         1999.
  • E.  Suzuki.     Lecture  Notes   in  Computer   Science,   volume  5579/2009,   chapter  Compression-Based    Measures  for Mining  Interesting   Rules,  pages  741-746. Springer  Berlin  /  Heidelberg,  2009.

 

 

Share

Hilary Mason - Machine Learning for Hackers

328 days
by in Algorithms, Machine Learning

An interesting beginners talk for machine learning enthusiasts.

Ever tried to use a regular expression to parse an unstructured street address? This talk is an introduction to a few machine learning algorithms and some tips for integrating them where they make the most sense and will save you the most headaches.

Hilary Mason - Machine Learning for Hackers from BACON: things developers love on Vimeo.

Share

On Bayesian Sensitivity Analysis in Digital Forensics

357 days
by in Algorithms, Bayesian, Machine Learning, Research & Development, Security

The idea of using of Bayesian Belief Networks in digital forensics to quantify the evidence has been around for a while now. To provide qualitative approaches to Bayesian evidential reasoning in the digital Meta-Forensics is however relatively new in the decision support systems research. For law enforcement, decision support and application of data mining techniques to “soft” forensic evidence is a large area in Bayesian forensic statistics which has depicted how Bayesian Networks can be used to infer the probability of defense and prosecution statements based on forensic evidence. Kevin B. Korb and Ann E. Nicholson's study on Sally Clark is Wrongly Convicted of Murdering Her Children and Linguistic Bayesian Networks for reasoning with subjective probabilities in forensic statistics gives an insight into an important development which helps to quantify the meaning of forensic expert testimony for "strong support".

The IEEE paper on Sensitivity Analysis of a Bayesian Network for Reasoning about Digital Forensic Evidence published in 3rd International Conference on Human-Centric Computing (HumanCom), 2010 is of particular interest since it has a comprehensive real-world list of evidence items and hypothesis.

Bayesian network representing an actual prosecuted case of illegal file sharing over a peer-to-peer network has been subjected to a systematic and rigorous sensitivity analysis. Our results demonstrate that such networks are usefully insensitive both to the occurrence of missing evidential traces and to the choice of conditionalevidential probabilities

one of the co-authors Dr. Overill has also covered grounds for A Complexity Based Forensic Analysis of the Trojan Horse Defence.

The evidence nodes are follows.

  • Modification time of the destination file equals that of the source file
  • Creation time of the destination file is after its own modification time
  • Hash value of the destination file matches that of the source file
  • BitTorrent client software is installed on the seized computer
  • File link for the shared file is created
  • Shared file exists on the hard disk
  • Torrent file creation record is found
  • Torrent file exists on the hard disk
  • Peer connection information is found
  • Tracker server login record is found
  • Torrent file activation time is corroborated by its MAC time and link file
  • Internet history record about the publishing website is found
  • Internet connection is available
  • Cookie of the publishing website is found
  • URL of the publishing website is stored in the web browser
  • Web browser software is available
  • Internet cache record about the publishing of the torrent file is found
  • Internet history record about the tracker server connection is found
  • The seized computer was used as the initial seeder to share the pirated file on a BitTorrent network

while the following hypothesis stand.

  • The pirated file was copied from the seized optical disk to the seized computer
  • A torrent file was created from the copied file
  • The torrent file was sent to newsgroups for publishing
  • The torrent file was activated, which caused the seized computer to connect to the tracker server
  • The connection between the seized computer and the tracker server was maintained

 

The authors conclude, exonerating the sparse evidence such that

The sensitivity analysis reported in this paper demonstrates that the BT BBN used in is insensitive to the occurrence of missing evidence and also to the choice of evidential likelihoods to an unexpected degree.

 

 

Our overall finding is gratifying because it implies that the exact choice of values for the inherently subjective evidential likelihoods is not as critical as might have been expected. Values falling within the consensus of experienced expert investigators are sufficiently reliable to be used in the BBN model. Furthermore, our results imply that the inability to recover one or more evidential traces in a digital forensic investigation is not generally critical for the probability of the investigatory hypothesis under consideration.

 

For some reason, this reminded me of a recent read SuperFreakonomics where authors devise a terrorist-algorithm with the following black-box variable.

“What finally made it work was one last metric that dramatically sharpened the aalgorithm.  In the interest of national security, was have been asked to not disclose the particulars; we’ll call it Variable X.

What makes Variable X so special?  

 

 

For one, it is a behavioral metric, not a demographic one.  The dream of anti-terrorist authorities everywhere is to somehow become a fly on the wall in a room full of terrorists.  In one small important way, Variable X accomplishes that.  Unlike most other metrics in the algorithm, which produce a yes or no answer, Variable X measures the intensity of a particular banking activity.  While not unusual in low intensities among the general population, this behavior occurs in high intensities much more frequently among those who have other terrorist markers.

 

 

 

 

This ultimately gave the algorithm great predictive power.  Starting with a database of millions of bank customers, Horsley was able to generate a list of about 30 highly suspicious individuals. According to his rather conservative estimate,  at least 5 of those 30 are almost certainly involved in actitvities.  Five out of 30 isn’t perfect—the algorithm misses many terrorists and still falsley identifies some innocents—but it sure beats 495 out of 500,495.”

Bayesian Belief Networks can definitely serve as a better probabilistic graphical model to achieve a improved visibility and prior/posterior probabilities for such network related algorithm.

 


Share
Next »
Go to Top