High Frequency Data Collection
THE QUICK LOOK
The client had developed a mobile heart rate monitor that was Bluetooth enabled and could be tethered to the user’s cellphone to provide data to their servers. The Software Practice was tasked with the design and development of a system that could consume that data for archival and notification purposes.
By its nature bio data is by highly volatile. It was therefore imperative for us to find the most robust methods to store and consume it while keeping costs to a minimum. The Software Practice achieved this by employing existing web services to create an integrated, fully-automated system.
We employed a number of Amazon services to achieve this. The first one was Kinesis – a highly scalable, cost-effective option that would serve as a pipeline to channel all the data. The processing was handled by EC2 virtual machines which work on a pay-per-use basis, thus saving the client recurring infrastructure costs. Meanwhile, RDS servers were used in conjunction with Posgres for data archiving. The former possess solid redundancy capabilities while the latter allowed us to set up triggers that would flag any interesting data. Once flagged, that information could instantly be sent via email, sms or push notifications with the help of Simple Queuing Service (SQS).
We also created numerous reporting tools that allowed the client to monitor the data and discern any patterns which could be used to form medical opinions. This has proven to be of invaluable help to the client.
THE CHALLENGES
We were working with significant time and resource constraints. Thus, the solution involved rapidly bringing together disparate technologies within one framework while keeping costs low.
With this in mind, we created an automated management system which provided us with an overview of the entire ecosystem we were constructing. This was a critical breakthrough. We could now see the entire data flow – from gathering and processing to storage and notification. This allowed us to make the system intelligent enough to spin up new servers when it anticipated new usage and spin them down when usage dipped. This kept the system lean while passing on significant cost savings to the client.
THE SUCCESS
We have demonstrated that the system is capable of handling thousands of users simultaneously. Having undergone exhaustive stress testing, it is now being evaluated by the client for final deployment.