When Velocity wanted to build a machine learning system to analyze human motion, Concrete Interactive incubated the innovation.
We often provide complete engineering resources to start-ups and enterprise that want to launch new endeavors. Velocity’s scale requirements: over 20 million new records per day with peak transactions of 80,000 per second and a five millisecond latency made this a fast-paced project involving deep-learning, iOS app development and extensive knowledge of real-time bidding systems for digital advertising.
Velocity’s insight was to build the “Operating System for Human Motion,” a suite of machine learning training tools, and real-time prediction based on motion sensor data from millions of smartphones. Their first application determines real-time attention based on the way a person physically moves with their phone, a metric Velocity calls “Receptivity.” In simple terms, the way you move says a lot about whether you’ll respond to an ad. For instance if you are driving a car, that’s a bad time. But a passenger would be much more likely to click and engage.
“The Concrete Interactive team was instrumental from concept to production launch, bringing new ideas, new architecture, and a really vast set of skills to bear when building the Velocity platform.”
A sheep stands in his field. Munching. Baa-ing. Wait, is that—spaghetti on his head? A bib around his neck? Aww, that’s the cutest sheep I ever did see! But why is that darn sucker moving so slow?
In fact, our poor wooly friend is an animation from Zynga’s game FarmVille. When we worked on this game, 140 million of you were coming back daily to plant and sow. Well, not YOU, of course! But if no one admits to it, where did all those players come from?
When optimizing art assets for Zynga’s Farmville, we noticed that some graphics had huge performance problems. In fact, some of players’ most favorite animals on the farm slowed game play to a crawl. And since Zynga depends on solid game performance to retain its user base, this was serious business.
Our first instinct was to build an art optimizer to simplify the artwork and make it faster via a process called rasterization. And it worked! Great, done—that’s all folks, the end.
BUT we all wanted to know more—why were some artists better than others at making cute little animals that would also show up on your screen with lightning speed?
There were plenty of other sheep on the farm that looked great and didn’t slow game play, so why was our spaghetti guy causing all sorts of trouble? It turns out it wasn’t the sheep—it was the spaghetti.
Get out of the building moment
Suddenly, the problem the spaghetti bowl caused made so much sense—the artist had individually drawn each noodle in glorious pasta-riffic detail.
Displayed up close on a large-screen monitor, all that detail looked great. But in the game, the entire bowl of spaghetti was just a few pixels across, so the detail was superfluous.
At that moment, it occurred to us that the problem here wasn’t optimization, it was training.
I immediately scheduled a meeting with the FarmVille studio art director and a few of the artists on the team. They were really surprised at the impact a few extra pen strokes had on game performance.
To find performance bottlenecks in other artwork, we next built the Zynga Analyzer, a tool that integrated directly with the artist’s workflow. With each action in the authoring environment, an artist could see the impact on performance.
The Analyzer tool helped give everyone a sense of how much drawing “budget” they had for a given character. Some drawing techniques, such as transparency and shading, had more impact on performance than others.
Everyone wanted to install it to find out what they were doing right and which art assets could be improved. Given these constraints, I saw that Zynga artists were incredibly clever at making cute illustrations that also performed well.
What started as an engineering optimization problem had morphed into an artist training program. The tool that exposed total impact on game performance now instilled a bit of self-reinforcing competition within the art team. Artists were spurred on to create illustrations that looked great and performed well. By exposing character performance during the creation process artists felt they had skin in the game, so to speak.
And finally, for illustrations with no fat to cut, the Optimizer tool gave an animation a bit more pep in its step. Altogether the effort once again secured each special animal’s place on the farm, including our little sheep who can now munch spaghetti till the cows come home.
Lately, retailers (both large and small) have seemed to focus on their omni-channel strategy: leveraging social channels to drive traffic, traditional mediums to promote cross-channel awareness, and e-commerce to streamline the transaction process.
But what about the mobile touch point of the customer experience? These days, many retailers have a mobile app: but is this the right mobile app for you and your customer? In an age where nearly 58% of customers conduct online/social research prior to purchasing an item at a brick-and-mortar retail store, retailers should be thinking about how their app can (a) enhance the customer experience and (b) streamline the path to purchase. Sometimes these objectives are one in the same. Here are some questions to ask yourself when developing a mobile strategy for your retail environment.
What are your customer’s pain points? Every retailer is different: Different store layouts, different SKUs, different check-out process. As a retailer, you should ask your customers what their biggest pain points are when shopping in your brick-and-mortar retail store. By the same token, you should also ask yourself how you can solve this pain point with the customer’s mobile device. Some scenarios to think about:
Are your checkout lines too long?
Do they want to know what is on sale?
Do your customers need help with an item?
Do your customers need help navigating your store to find a category or SKU?
Would your customers prefer a ship-to-home option rather than hauling the item in their car?
Do your customers want to know what the price is of an item?
Why should your customer use your app? Once you’ve figured out your customer’s pain points, you should ask yourself why a customer should (a) download and (b) use your app. With thousands of apps on the market, and room for ~20 apps on the user’s Home screen, a better question may be: Why will the customer want to use your app more than once?
Do your app solve the problem (above) in a way that enhances the customer experience?
Do customers who use your app have a significant advantage over customers who don’t use it?
Does the customer receive value in the form of discounts or loyalty rewards?
Does the app enhance the offline and cross-channel customer experience?
Operationalizing the mobile experience to wow your customers
In some, or maybe all, of these scenarios the retailer may need to operationalize the experience around the mobile app. For example:
How do you handle loss prevention if you implement mobile check-out?
How do you greet loyal customers who enter your store?
How do you redeem loyalty rewards via the mobile app for a customer who is ready to check-out?
Do you offer flash-sales for customers who scan a SKU using their mobile phone, based on their purchasing history?
We believe the best mobile experiences are the ones that “start with the end” — and in the case of retail, we believe starting with the desired customer experiencein the context of mobile, will help bring brick-and-mortar retailing to the 21st century.
Look around you and you will see thousands of “things” all within your immediate vicinity. Your keychain. Your desk chair. Your favorite coffee mug filled with Italian Roast coffee. Your dying ficus plant. With today’s technology, there is no reason why these things cannot communicate with you, in-real time.
Your plant should tell you, not only that it needs water, but how much and what position to place it during the day
Your coffee mug should know what kind of roast you want to drink today
You should be able to find your keys at a moment’s notice because you have a bad habit of misplacing them the moment you are about to go somewhere
Your desk chair should automatically adjust itself when it detects you are sitting with poor posture (reminder: stop slouching)
As luck would have it, there are technologies for each one of these things, being built. Right. Now. (See for yourself: Plant | Coffee | Keys | Chair).
The Hardware (R)evolution.
While cliche, the world we are living in is becoming increasingly more connected, more now than ever before. While in research in development only a few years ago, technologies like RFID, NFC, and Zigbee are enabling the next generation of connected devices in a cost and energy efficient way. In fact, consumer goods that weren’t previously connected 18 months ago are now online. Recent examples, include:
Connected lightbulbs allow you to change its hue and control the lighting ambiance of your room via a mobile app
As enabling technologies become cheaper and smaller, companies will be forced to innovate and think about how their offline products can get online.
Getting offline products into the 21st century is only the tip of the iceberg. Enhancing these products with connected technologies has to transform the product experience, be personal, and have utility. The bar for product experiences is so high, not executing against these objectives will result in a gimmicky, failure of an experience.
For example: A shoe company may want to create a running shoe with a GPS Dot. These “online shoes” should not only track where (and how long) the user was running, but it should provide actionable insights based on what the shoe company already knows about you: recommend running trails based on your running style and preferences, alert you when your friends are close by, give you a discount if you walk by a their store, and let you know how hard to run based on your body fat and weight goals.
Utility vs. Privacy
Privacy generally is a topic of concern when more devices become online and “all knowing.” As we’ve seen from the internet and media today, and in light of recent NSA privacy concerns, users are willing to give up certain liberties to connect with friends (Facebook), share their thoughts (Twitter), utilize free email (Google), or make free international calls over the internet (Microsoft/Skype). We believe its important for companies who are contemplating an online product strategy understand these implications and balance the utility an online product with the user’s privacy and the company’s ethics/values.
When beginning a mobile, web or other app software project, keep in mind it’s more like adopting a pet than building a product. Software needs continual care, maintenance and feature development. Users expect updates — whether to take advantage of the latest mobile Operating System release, to fix a bug that somehow slipped through Quality Assurance, or simply to add features. Building an app is not a “one-time and you’re done” operation.
Product / Market Fit
Consult the experts, whether Marc Andressen (co-creator of the first web browser), Steve Blank of Stanford, Paul Graham of Y-Combinator or Sean Ellis of LogMeIn, they all agree: it’s about getting the right product, to the right people. That said, your app’s features should depend heavily on who that app is marketed to. To achieve this takes a lot of “get out of the building” type thinking promoted by Eric Ries in his book The Lean Startup. Interacting with your market as soon as possible is paramount.
Agile software development process, user-centric design, and Lean thinking can all help you discover what features to build, but all the theory in the world won’t help unless you learn from your market, measure feedback, and build the features that users desire: you must go through this “build, measure, learn” cycle a few times to get it right.
Accrue Technical Debt Wisely
You may be initially wowed by a software team that can deliver features fast and furiously — especially when you they look cool, and progress comes swiftly — but in just a few short months, a mobile app project can grind to a halt. Why? At the beginning of a project software developers often build features without building the surrounding infrastructure to support them. It’s like building a glorious bathroom complete with steam shower in a house with no plumbing.
The industry term for this is “technical debt.” While this form of indebtedness can get you a quick jolt of progress, it can also come back to bite you. For quick experiments, technical debt may be the right option, but for meaningful, high-quality app development, building it right means a robust software architecture, infrastructure to support scaling to a massive audience, and putting in place the security necessary to protect both your users and your investment.
There is a reason that, on average, large IT projects run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted. In a survey of 600 people closely related to a software project, 78% of respondents reported that the “Business is usually or always out of sync with project requirements.” It is extremely difficult to correctly estimate large software projects. So, smart teams have stopped trying.
But, without an estimate, how will your app hit deadlines and a budget? The secret is once again, in the agile process: you can correctly estimate software deliverables over the short term. The agile software development process promotes short “sprints” and we suggest a one-week time period. This way, your team releases a fully-functional and complete product every week! And since you are learning from your users, what you will do over the next few months, will be in direct response to their usage and feedback. Think of a product initiative as an experiment, where the goal is to learn what a market wants, and deliver it.
Asking your users directly what they want (or don’t want) is a pitfall to avoid. You are the innovator, and you understand the possibilities for future directions of your product better than anyone, including your users, so asking them is asking for trouble. Instead, simply observe them. Focus groups are notorious not only for their expense, but also their “false sense of science.” Studying users behind a one-way mirror may have worked well in that Mad Men episode, but for software, the “Starbucks Method” is about a million times cheaper, and much more insightful. Get into the cafe, hand out some 20’s, 0r buy people lunch in exchange for watching them use your product.
There is no harm in providing a few in-app survey questions. Take a look at Qualaroo for how to do this well. For building community, GetSatisfaction is a good bet. Bottom line: Target your questions to the specific user experience, not the overall product.
As the recent “Downpocalypse” of the Apple Developer Center demonstrated (no new iPhone apps could be created for over a week!) hitching your app to a single horse is a dangerous move. Though an initial iOS app release makes sense in many cases, building with cross-platform mobile technologies like HTML5, Adobe Phonegap, Ludei or Unity, allows your app to diversify its bets; placing expensive native features only where they’re needed. This way you can release on iOS, Android, the Web, even PC, Mac and gaming consoles.
Choose the Right Team
Select a development team that will maximize your budget and give the best value. Sometimes the cheapest guy on Craig’s List, Upwork, or RentACoder is the right way to go — such as with one-off experiments or when a quick and dirty initial draft of an app on a shoestring budget is expected to be tossed and re-written from scratch — but for most projects, it makes sense to proceed with a team that can provide end-to-end services, engage with your users, and help guide you strategically to that nirvana of mass adoption.
There are so many things Machine Learning applies to. When Terry Gross spends an hour talking about artificial intelligence on Fresh Air, you know robots and machine learning are top of mind in the public psyche. In her interview with John Markoff, author of, “Machines of Loving Grace,” he cites a broad definition of AI: “A robot can be … a machine that can walk around, or it can be software that is a personal assistant, something like Siri or Cortana or Google Now.”
Modern artificial intelligence and machine learning techniques apply so broadly, they will touch every aspect of our everyday lives (and some already are). At Concrete Interactive, we have chosen to focus on human motion learning—to capture, characterize, and make recommendations on how to improve any movement a human can perform. And just think of how many movements a person can do!
Owing to over a decade of experience with sensors, data acquisition, and control systems, our machine learning techniques are specialized to be able to detect nuanced motion, to see through the noisy signals that sensors produce, and to identify, count and judge accuracy on thousands of different motions. Weightlifting form, the beautifully aligned yoga pose, the perfect golf swing, medical grade fall prediction are all examples of how machine learning can help people perform better, improve faster and reduce injury.
Beyond Steps & Sleep
What sensors can we use? The accelerometers that are already in the smartphone in your pocket or purse to start! And the wearable on your wrist, the one clipped to your shoe, and the one coming that none of us even know about.
Our platform is designed to integrate with many wearables.
Think of Photoshop, where you can edit images captured with many different cameras, at many different resolutions, in many formats. Similarly, wearables may have different numbers of sensors, they may be worn on different parts of the body, but they are all measuring the same motions we wish to track.
“Even rocks compute,” a friend once remarked. I won’t comment on either of our sobriety, but at the time I just shot him a knowing look and nodded contemplatively. But what did he mean?
Maybe he meant that the reflections and refractions off rocks, or even better, crystals, perform a sort of mapping. The light waves come in from one direction, then reflect, refract, scatter, and project on to surrounding surfaces. Sometimes these light projections are beautiful, regular, even useful.
Light, being a vibration of electromagnetic radiation in the optical part of the spectrum, has a higher frequency, than sound waves, but like light, sound is a vibrational energy that imprints change in the resonance of objects around it. Taken this way, the quandary of the tree falling in the forest resolves in the affirmative, because someone is always there to witness it, namely the tree and forest itself.
The field of archeoacoustics explores the innate acoustical properties of artifacts via audio analysis—a study of the essential vibrational qualities of artifacts and environments. In examination of a 1969 claim by Richard G. Woodbridge III, the Mythbusters report it is in fact not possible to resurrect ancient roman voices from the grooves carved by their tools as they spun around a potter’s wheel 6,500 years ago. Maybe not yet anyway.
The vibrations from our voices to our footsteps, are like tiny tremors impacting the matter around us. And whether this matter is capable of recording them may have more to do with our playback technology than the indisputable fact that what we do influences the things around us.
As early as circa 1902, mathematician Charles Sanders Peirce wrote, “Give science only a hundred more centuries of increase in geometrical progression, and she may be expected to find that the sound waves of Aristotle’s voice have somehow recorded themselves.”
A rock, a tree, the earth—animals have used these as computational devices by echolocation long before humans evolved. Zoologists Roger Payne and Douglas Webb calculated that before ship traffic noise permeated the oceans, tones emitted by fin whales could have traveled as far as four thousand miles and still be heard against the normal background noise of the sea. Whales, bats, dolphins, and recently even some blind humans use echolocation to “see” objects by listening to reflections of sounds they themselves emit. The computation is performed by the reflecting objects, transforming the sound energy as it was emitted, by shifting frequency and waveform to imprint this energy with the characteristics of the objects that have influenced it such as distance, size, hardness, maybe even “tastiness”?
Is there an essential vibrational signature to all things that can be elucidated through computation? Yes, there is. And the next part of this series will introduce the modern machine learning techniques and sensor technologies being employed to further illuminate the useful (and perhaps mystical) properties in everything around us.
Brett Bond is President of Concrete Interactive, a software development and machine learning firm based in San Francisco and Santa Monica. When not writing software, Brett enjoys practicing yoga, preparing the nursery for his soon-to-arrive daughter, and building large-scale fire displays.
The challenge is to compare two pictures containing faces and decide whether they are pictures of the same person or two different people. In the experiment, 6000 pairs of images were examined, both by humans and by algorithms from various teams around the world. Three teams, including Baidu, Google and the Chinese University of Hong Kong achieved better-than-human recognition performance. The Baidu team made just 9 errors out of the 6000 examples.
Here is a slide from Professor Ng’s talk with images of the ones they got wrong:
You may notice that the top-left image pair is of movie actress Nicole Kidman, which the Baidu system incorrectly classified as two different people. What may be less obvious, is that the second image of Ms. Kidman was taken from her film, “The Hours” in which she is wearing a prosthetic nose.
Here are the overall results from implementations done by teams throughout the world. The dashed line indicates human performance at the same task. People typically think of recognizing faces as a very innate human task, but once again, we can be surprised that machine learning algorithms are now capable of equalling or outperforming humans.
In his talk, Professor Ng credits two major factors as contributing to the vast improvements in machine learning technology over the past five years. As an analogy, think of launching a rocket into orbit: we need 1) both giant rocket engines and 2) lots of rocket fuel.
The rocket engine in his scenario is the incredible computational performance improvements brought to us by GPU technology. The fuel then, is access to huge amounts of data, coming to us from the prevalence of internet connected sensors, online services, and our society’s march toward digitization.
To perform this astounding face comparison judgment, algorithms are trained on facial data. The flow chart in the image above shows that if the algorithm is not performing well on the training data, we need a more powerful rocket engine.
The second part of the flowchart above illustrates that if the algorithm is learning the training data quite well, but performing poorly when presented with new examples, then perhaps more “rocket fuel” is required, so gathering more training data would be the logical approach to improve the system.
Professor Ng compares the benefits of high performance computing with cloud computing and states that better training performance may be achievable say with 32 GPUs in a co-located rack than by a larger number of servers running in the cloud. His reasoning is that communication latency, server downtime and other failures are more prevalent in cloud-based systems because they are spread out across more machines with more network connections.
Machine learning has been demonstrated to be good at many things. The recent improvements are not limited to face comparisons, in fact in this same talk improvements to Baidu’s speech recognition system were shown to perform well in noisy environments.
Now that the 2015 HIMSS conference in Chicago has wrapped up, I will try to summarize the trends I observed and how Concrete Interactive fits in. It is clear that secure text messaging is a much-needed feature in healthcare. There are at least 2 established companies vehemently pursuing it: TigerTextImprivata (via their Coretext feature), and several startups presenting at HIMSS: Diagnotes, MyCareText, Cotap.
As we know TigerText just closed a $21M VC round. They claim to have 300 enterprise customers mostly in healthcare, including 4 of the largest for-profit hospital chains.
What isn’t clear to me is whether secure messaging is a separate app, or really a feature to be used with EHR (Electronic Health Record) apps that health companies already have. So for example, Imprivata’s CoreText is really positioned more as a feature of their larger system.
However, as a separate app, secure texting following the BYOD (bring your own device) (yes, this is literally the way they talk about it), is a very attractive feature that many people want, and could provide a solid scenario for deeper involvement or integration at a custom development level.
Another clear area of expansion is in medical device connectivity. For example, Qualcomm Life bought Healthy Circles, a deal supposedly in the $375M range. It is an iPhone app for continuous care. The patient goes home and plugs in a local bluetooth/3g router into a wall outlet. All the continuous care devices (Class 2 FDA approved medical devices) such as blood pressure monitoring, step counting, pulse, temp, glucose monitoring, even home ventilators and other Class 3 (high risk) devices.
The physician gets a portal. The patient can view and augment the data on the iPhone, though the app isn’t even required. This pattern is repeated over and over by other companies: device, connectivity, app, cloud-based portal.
The industry is only just awakening to the fact that data science will play a big role. Channels of information such as medical devices and apps are beginning to provide the big data they will use. I did make a nice connection at Wolters Kluwer. They are already doing rule-based processing to de-dupe health data. So if a doctor writes COD, they expand that to codeine. But they want to improve their systems via natural language processing (NLP).
I also met with Piers Nash from the University of Chicago at a Genomics SIG. He’s working with NCI and already has 6 Petabytes (PB) of genomic data from >10,000 patients online and available to the public (after a straightforward application process). He’s looking to host algorithms next and run compute cycles from virtual machines (PAAS type like AWS). One basic problem they are trying to improve is referred to as Single Nucleotide Variation calling (SNV calling). The problem is that each person’s DNA is slightly different, because we are different people. The trick is to identify which nucleotide (DNA letters) are different because of normal genetic variation between people, vs. mutations that cause cancer. One interesting aspect of this problem is that as algorithms improve, past recommendations may become invalid. And there may be a liability aspect at work. Samsung genomics was also in attendance at this meeting. They are launching an initiative to sequence tumors and make recommendations, but it’s similar to others already out there, such as Paradigm.
Also at the genomics meeting was Michael Hultner, the Chief Scientist for Lockheed Martin’s health and life sciences division. They are bidding as are many others for the UK’s 100,000 genomes project. He says their expertise lies in the integration of many technologies and thinks they are well positioned in the health space (not just outer space). So it’s fascinating to see the kinds of companies entering or expanding in this market.
Big Picture Strategy
The healthcare IT space is rapidly expanding as healthcare laws such as Meaningful Use Stage II come into effect increase incentives to leverage advances in the technology. There are many land grabs playing out. Any space worth entering will have competition, but based on my assessment of the overall quality in the space, I believe Concrete Interactive is well positioned to innovate, and stand up great apps against much larger players than ourselves.
This year’s Healthcare Information and Management Systems Society conference in Chicago is a veritable candy store of high-tech healthcare. Yes the smart hospital beds and baby monitoring bracelets are fascinating. But perhaps the highest impact, most impressive technology on offer is what you can’t see—the software. Though it has about as much shazam as a bed pan, the coming health communication infrastructure known as HL7 FHIR (pronounced like “fire”) will allow access to the coveted Electronic Health Record (EHR) via many new applications and devices.
Also very impressive and a bit more visible were the beautiful mobile workflow apps like Nextgen’s “Go for iPad.” What I like about this electronic health record and dictation recording tool is that it does not do everything. The heavy lifting of setting up records is done on the desktop (templating in healthcare parlance), and on-the-go actions such as dictation and prescription refills, can be executed in short order on the iPad.
I also learned that Greenway, a software provider of Practice Management (PM) and EHR tools, has an app marketplace (think iTunes). Topping their offering is Phreesia, a check-in app for iPad can replace all that form filling in the doctor’s office with a few taps of a touchscreen.
The Internet of Things (IoT) was also present, from Tyco’s tracking bracelets, for babies and elders, to decibel logging sensors that monitor noise levels. Quietyme, a HealthBox and Gener8tor accelerator graduate, establishes a mesh network of small volume monitors in each hospital room, the corridor, nurses station, etc. They perform some fancy data analytics (in partnership with Miosoft and Zero Locus). CEO John Bialk says that by comparing noise levels in patient rooms with patient surveys, they can document and predict which noisy areas are having a negative impact on healing. And from Ascom, voice over internet protocol (VoIP) portable devices are like little cordless phones that nurses can use on the local area network (LAN). Their Android device even supports internet instant messaging.
Thank you to all those who visited with Concrete Interactive, and those who described their wonderful products, software, services and innovation.