Understanding Hidden Variables with Python - Research Roadmap

/

Post Outline

  • Motivating the Journey
    • Where Do Edges Come From?
    • The Problem with Traditional Research
    • The Hidden Side
  • A Brief Description:
    • Part 1 - A Visual Introduction to Hidden Markov Models with Python
    • Part 2 - Exploring Mixture Models with Scikit-Learn and Python
    • Part 3 - Predicting Market Bottoms with Scikit-Learn and Python

Motivating the Journey

Where do Edges Come From?

Edges come from superior ability to identify and execute profitable strategies.

You can see this simply by imagining the first strategy able to identify pricing errors on identical items in different markets. This knowledge is valuable in two scenarios:  you can execute the transaction yourself or you know someone who can and will pay you for the "signal".

Abstractly, a signal can be thought of as a glitch in the matrix allowing us a view through a window into probabilistic future states. Signals can come from anywhere and are not always understood. 

Our job is to find these signals, vet them, and implement them. This is difficult in practice. The competitive environment we seek to understand is dynamic with positive and negative feedback loops operating at various scales. The system processes are very noisy making signal extraction confusing and difficult. Competitors are always seeking strategies that "work" until they don't.

Generally profitable edges stop working when both your identification and execution strategies are well known. Thus a profit motive for secrecy and obfuscation exists among participants. If you are familiar with poker this will sound very familiar.

This also means that using well known identification techniques puts you at a strategic disadvantage because your competitors have likely incorporated knowledge of your methods into their own strategies. 

Therefore we must continuously search for strategies that are not well understood, not well known, or otherwise difficult for our competitors to implement. 

The Problem with Traditional Research

Too much published "research" focuses on using well known statistical tools to draw conclusions that do not improve the odds of profitable investment. Worse still, many research papers' results are not reproducible. 

For periods of time, techniques involving technical analysis, regression, and simple correlations, were good enough to beat the market. This worked because the methodology was not well known or well understood. Times have changed. 

These methods have been taught and promoted to generations of practitioners. These techniques form the foundation of many market participants investment strategies.  Therefore the majority of well known strategies are already in use by the market.

This means sophisticated participants have had time and opportunity to develop counter strategies to take advantage of the limitations of publicly known methods.

Typical business finance teachings focus on the theory that stock values are directly tied to the expected value of net cash flows produced by the underlying operating business from now into some future period. Other research links stock prices to any number of other observable factors. My perception is that these well taught methods can bias our exploratory research when it comes to the art and science of prediction.  

Successful prediction does not require understanding or logic. Prediction does not require expertise in the industry or business which generated the data. These things can help solidify our belief in the power of the prediction, however successful prediction methods only require a stable, positive payoff function relative to prediction accuracy over an expanding time period. Nothing more, nothing less.  

Rigid knowledge structures can blind us to potential opportunities. Using statistics to explore observable factors only, ignores the entire spectrum of hidden, unobserved factors influencing asset returns. 

The Hidden Side

By definition a hidden factor is not directly observable. Its presence or influence is detected by its effect on observable factor(s) or on a delayed basis. 

Conceptualizing the influence of hidden factors is difficult for many decision makers to either understand or incorporate into already existing processes.

The combination of bias created by traditional finance and difficulty conceptualizing hidden factors, creates the barriers to entry we need for successful strategy development. We can reasonably assume this research pathway is still rich with profitable edges and worth pursuing.

A Brief Description:

Part 1 - A Visual Introduction to Hidden Markov Models with Python

In part 1, we will discuss Markov Models, Hidden Markov Models and a toy application for regime detection.

Part 2 - Exploring Mixture Models with scikit-learn and Python

In part 2, we will explore the motivation behind mixture models and how they improve on the weaknesses of K-means algorithms. We will also discuss the connection between Mixture Models and Hidden Markov Models. Finally we will extend our toy regime detector to use a mixture model instead.

Part 3 - Predicting market bottoms with scikit-learn and python

In part 3, we will implement a toy strategy using mixture models to predict market bottoms. The strategy assumes that we can calibrate a model to predict the market return distribution such that actual returns that fall below the confidence intervals are profitable long entries over short time periods. 

Post thumbnail picture taken from Bayesian Intelligence Slideshare presentation.