The Future of Big Data & Analytics: Excerpts from IBM's Global Technology Outlook ( ) 2013 IBM Corporation

Transcription

1 The Future of Big Data & Analytics: Excerpts from IBM's Global Technology Outlook ( ) 2013 IBM Corporation

2 The fourth dimension of Big Data: Veracity handling data in doubt Volume Velocity Variety Veracity* Data at Rest Data in Motion Data in Many Forms Data in Doubt Terabytes to exabytes of exting data to process Streaming data, milleconds to seconds to respond Structured, unstructured, text, multimedia Uncertainty due to data inconstency & incompleteness, ambiguities, latency, deception, model approximations * Truthfulness, accuracy or precion, correctness 2

3 Uncertainty ares from many sources Process Uncertainty Data Uncertainty Model Uncertainty Processes contain randomness Data input uncertain All modeling approximate Actual Spelling Intended Spelling Text Entry??? Uncertain travel times GPS Uncertainty? Testimony? Fitting a curve to data? {Par Airport} Ambiguity Semiconductor yield Contaminated? Rumors {John Smith, Dallas} {John Smith, Kansas} Conflicting Data Forecasting a hurricane ( 3

4 Condensing data reduces uncertainty by constructing context Loyalty Michael San Jose, CA Buying Buying DSLR DSLR today today!! Influencers Intent Customer at Mall CONDENSE Credit Customer in Store #42 $999 $560 In-Store Pricing And Dcounts Required: tight integration to maximize context dcovery Data finds Data Mor Date Required: common practices followed by multiple standards for representing uncertain data and uncertainty of all types, provenance, and lineage and or metadata Son $560 Birthday Fact Dcovery OR $999 A & Spatial Reasoning NY Sense Making & Temporal Reasoning Correlation Corroboration (Evidence Combination) ETC. Buying a DSLR today! Maximum Context For Minimum Uncertainty Required: common APIs to enable sharing across uncertainty management pipeline No such common practices, standards or APIs ext today 4

5 An IBM Grand Challenge Build a system that rivals a human s ability to answer questions posed in natural language with accuracy, confidence, and speed. 5

6 Taking Watson beyond Jeopardy! Understanding Specific Questions Interacting Question-In/Answer-Out Explaining Prece Answers & Accurate Confidences Learning Batch Training Process The type of murmur associated with th condition harsh, systolic, and increases in intensity with Valsalva From specific questions to rich, incomplete problem scenarios (e.g. EHR) Evidence analys and look-ahead, drive interactive dialog to refine answers and evidence Move from quality answers to quality answers and evidence Answers, Corrections, Judgements Input, Responses Entire Medical Record Dialog Responses, Learning Questions Refined Answers, Follow-up Questions Rich Problem Scenarios Scale domain learning and adaptation rate and efficiency Interactive Dialog Teach Watson Comparative Evidence Profiles Continuous Training & Learning Process 6

7 Dialoguing to an answer Present Factors Red, painful eye Blurred vion Family htory of arthrit Medical Record Absent Factors Circular rash Fatigue Headache Q: What diagnos explains patient s condition? The The first first symptom symptom of of Lyme Lyme dease dease (also (also called called Lyme can affect different systems, Lyme s for 50% of aabody small, Lymedease dease can affect different systems, Lyme s dease) dease) for about about 50% of people people body small, such as nervous system, joints, skin, and red bull s-eye rash, called eryma migrans, at suchrash, as nervous system, joints, skin, red bull s-eye called eryma migrans, at and heart. Symptoms are often described site of an infected tick bite. Symptoms as site of an heart. infected tick bite. are often describedas happening in three stages Or acute are flu-like happening in three stages(although (although not Or early, early, acute Lyme Lyme are flu-likenot everyone experiences fatigue, muscles or fever, chills, everyone experiences allthree): three): fatigue, achy achy muscles or joints, joints, all fever, chills, stiff stiff 1.A within neck, glands, and aatypically headache. 1.Acircular circular rash, within weeks weeksofof neck, swollen swollen glands,rash, and typically headache. infection, often first sign of infection. infection, often first sign of infection. Lyme dease caused by bacterium Borrelia Lyme2.Along dease caused by bacterium Borrelia with rash, aaperson may have flu-like 2.Along with rash, person may have flu-like burgdorferi and transmitted to through burgdorferi and transmitted to humans humans through such as lymph nodes, such asswollen swollen lymph nodes,fatigue, fatigue, bite of infected blacklegged ticks. Typical bite of infected blacklegged ticks. Typical headache, and muscle aches. headache, and muscle aches. headache, include include high high temperature, temperature, headache, fatigue, fatigue, and and aa charactertic charactertic skin skin rash rash called called eryma migrans. eryma migrans. 7

8 Context at Scale brings technical challenges Fully contextualized information will require at least 10x storage of raw data. 10x Data Context New Storage Requirements Continual ingestion and curation will require continual deep analytics to dcover new insights. Dynamic schema requirements and temporality will drive new database requirements. Unique Data Requirements High dimensional data with dynamic content and relations lead to irregular graphs that are notoriously hard to partition and favor large in-memory systems. Context never rests but data grows continuously with streaming high velocity input. Highly dynamic graphs will require ultra scalable data-structures, that support local graph traversal and at same time representation/querying of global properties. Vertically Integrated Systems Context size, dynamics, and access patterns will require data-centric, scale-in, highly integrated systems. Hybrid Memory Cube Hybrid memory cubes and or druptive technologies will enable large-scale, real-time, contextual processing.

9