{"id":8318,"date":"2014-04-03T12:14:26","date_gmt":"2014-04-03T16:14:26","guid":{"rendered":"http:\/\/n2value.com\/blog\/?p=8318"},"modified":"2014-04-03T12:20:06","modified_gmt":"2014-04-03T16:20:06","slug":"what-medicine-can-learn-from-wall-street-part-i-history-of-analytics","status":"publish","type":"post","link":"https:\/\/n2value.com\/blog\/what-medicine-can-learn-from-wall-street-part-i-history-of-analytics\/","title":{"rendered":"What medicine can learn from Wall Street – Part I – History of analytics"},"content":{"rendered":"

\"floor\"<\/a>We, in healthcare, lag in computing technology and sophistication vs. other fields. \u00a0The standard excuses given are: healthcare is just too complicated, doctors and staff won\u2019t accept new ways of doing things, everything is fine as it is, etc\u2026 \u00a0But we are shifting to a new high-tech paradigm in healthcare, with ubiquitous computing supplanting or replacing traditional care delivery models. \u00a0Medicine has a \u2018deep moat\u2019 – both regulatory and through educational barriers to entry. \u00a0However, the same was said of the specialized skill sets of the financial industry. \u00a0Wall St. has pared its staffing down and has automated many jobs & continues to do so. \u00a0More product (money) is being handled by fewer people than before, an increase in real productivity<\/a>.<\/span><\/p>\n

Computing power in the 1960\u2019s-1970\u2019s on Wall street was large mainframe & mini-frame systems which were used for back-office operations. \u00a0Most traders operated by \u2018seat of your pants\u2019 hunches and guesses, longer term macro-economic plays, or using their privileged position as market-makers to make frequent small profits. \u00a0One of the first traders to use computing was Ed Seykota, who applied Richard Donchian\u2019s trend following techniques to the commodity markets. \u00a0Ed would run computer programs on an IBM 360 on weekends, and over six months tested four systems with variations (100 combinations), ultimately developing an exponential moving average trading system that would turn a $5000 account into $15,000,000.(1) \u00a0Ed would run his program and wait for the output. \u00a0He would then manually select the best system for his needs (usually most profitable). \u00a0He had access to delayed, descriptive data which required his analysis for a decision.<\/span><\/p>\n

In the 1980\u2019s – 1990\u2019s computing power increased with the PC, and text-only displays evolved to graphical displays. \u00a0Systems traders became some of the most profitable traders in large firms. \u00a0Future decisions were being made on historical data (early predictive analytics). \u00a0 On balance well-designed systems traded by experienced traders were successful more often than not. \u00a0Testing was faster, but still not fast (a single security run on a x386 IBM PC would take about 8 hours). \u00a0As more traders began to use the same systems, the systems worked less well. \u00a0This was due to an\u00a0<\/span>\u2018observer effect\u2019.<\/span><\/span><\/a>, with traders trying to exploit a particular advantage quickly causing the advantage to disappear!\u00a0 The system trader\u2019s \u2018edge\u2019 or profitability was constantly declining, and new markets or circumstances were sought. \u2018Program\u2019 trades were accused of being the cause of the 1987 stock market crash. \u00a0<\/span><\/p>\n

There were some notable failures in market analysis – Fast Fourier Transformations being one. \u00a0With enough computing power, you could fit a FFT to the market perfectly – but it would hardly ever work going forward. \u00a0The FFT fails because it presumes a cyclical formula, and the markets while cyclical, are not predictably so. \u00a0But an interesting phenomenon was that the better the fit in the FFT, the quicker and worse it would fall apart. \u00a0That was due to the phenomenon of curve-fitting.\u00a0 ‘Fractals’ were all the rage later & failed just as miserably – same problem.\u00a0 As an aside, it explains why simpler linear models in regression analysis are frequently \u2018better\u2019 than a high-n polynomial spline fit to the data<\/a>, particularly when considered for predictive analytics. \u00a0The closer you fit the data, the less robust the model becomes and more prone to real-world failure.<\/span><\/p>\n

Further advances in computing and computational statistics followed in the 1990\u2019s-2000\u2019s. \u00a0Accurate real-time market data became widely available and institutionally ubiquitous, and time frames became shorter and shorter. \u00a0 Programs running on daily data were switched to multi-hour, hour, and then in intervals of minutes.The trend-following programs of the past became failures as the market became more choppy, and anti-trend (mean reversion) systems were popular. \u00a0Enter the quants – \u00a0the statisticians.(2) \u00a0 With fast, cheap, near-ubiquitous computing, the scope of the systems expanded. \u00a0 Now many securities could be analyzed at once, and imbalances exploited. \u00a0Hence the popularity of \u2018pairs\u2019 trading. Real-time calculation of indices created index arbitrage, which were able to execute without human intervention.<\/span><\/p>\n

The index arbitrage (index-arb) programs relied on speed and proximity to the exchanges to have advantages in execution. \u00a0Statistical Arbitrage (Stat-arb) programs were the next development. These evolved into today\u2019s High-Frequency-Trading programs (HFT\u2019s) which dominate systems trading \u00a0These programs are tested extensively on existing data, and then are let loose on the markets to be run – with only high-level oversight. \u00a0They make thousands of trading decisions a second, incur real profits and losses, and compete against other HFT algorithms in a darwinian environment where the winners make money and are adapted further, and the losers dismissed with a digital death. \u00a0Master governing algorithms coordinate individual algorithms. (4)<\/span><\/p>\n

The floor traders, specialists, market-makers, and scores of support staff that once participated in the daily business have been replaced by glowing boxes sitting in a server rack next to the exchange. \u00a0<\/span><\/p>\n

Not to say that automated trading algorithms are perfect. \u00a0A rogue algorithm with insufficient oversight caused a forced sale of Knight Capital Group (KCG) in 2012. \u00a0(3) \u00a0The lesson here is significant – there ARE going to be errors once automated algorithms are in greater use – it is inevitable.<\/span><\/p>\n

So reviewing the history, what happened on wall st.?<\/span>
\n1. \u00a0First was descriptive analytics based upon historical data.<\/span>
\n2. \u00a0Graphical Interfaces were improved.<\/span>
\n3. \u00a0Improving technology led to more complicated algorithms which overfit the data. (WE ARE HERE)<\/span>
\n4. \u00a0Improving data accuracy led to real-time analytics.<\/span>
\n5. \u00a0Real time analytics led to shorter analysis timeframes<\/span>
\n6. \u00a0Shorter analysis timeframes led to dedicated trading algorithms operating with only human supervision<\/span>
\n7. \u00a0Master algorithms were created to coordinate the efforts of individual trading algorithms.<\/span><\/p>\n

Next post, I’ll show the corollaries in health care and use it to predict where we are going.<\/span><\/p>\n

\u00a0\u00a0<\/span><\/p>\n

(1) Jack Schwager, Market Wizards, Ed Seykota interview pp151-174.<\/span>
\n(2) David Aronson, Evidence-based Technical Analysis, Wiley 2007<\/span>
\n(3)\u00a0<\/span>Wall St. Journal, Trading Error cost firm $440 million, Marketbeat \u00a0<\/span><\/a><\/p>\n

(4)Personal communication, HFT trader (name withheld)<\/p>\n","protected":false},"excerpt":{"rendered":"

We, in healthcare, lag in computing technology and sophistication vs. other fields. \u00a0The standard excuses given are: healthcare is just too complicated, doctors and staff won\u2019t accept new ways of doing things, everything is fine as it is, etc\u2026 \u00a0But we are shifting to a new high-tech paradigm in healthcare, with ubiquitous computing supplanting or […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"N2Value - New Post : What medicine can learn from Wall Street - Part I.","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[4,8,2],"tags":[],"class_list":["post-8318","post","type-post","status-publish","format-standard","hentry","category-data-science","category-finance","category-healthcare"],"jetpack_publicize_connections":[],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4mtfP-2aa","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/8318","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/comments?post=8318"}],"version-history":[{"count":4,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/8318\/revisions"}],"predecessor-version":[{"id":8324,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/8318\/revisions\/8324"}],"wp:attachment":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/media?parent=8318"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/categories?post=8318"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/tags?post=8318"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}