On February 23, 2017, Google Project Zero and Cloudflare revealed the existence of the Cloudbleed bug. Fitbit uses Cloudflare as our content distribution network and the majority of our web and API traffic routes through the Cloudflare platform.
We learned of the issue the same way as everyone else – when security-minded team members at Fitbit alerted us to the Project Zero ticket and the associated Cloudflare Blog post.
For a complete description of the the bug details, we’d recommend reading Cloudflare’s blog post. In short though, a software error meant that in some situations memory containing parts of in-flight HTTP requests were leaking from Cloudflare’s systems.
Although the likelihood of these issues affecting our users is incredibly low, we have taken a number of preventative actions and have provided detailed guidelines to our consumers via our Help site. We hope through this post to share the steps that we took to investigate and respond to this incident.
After learning of the issue, we immediately started working to understand the potential impact of the issue for Fitbit users. We focused on three main questions:
- Which data elements may have been leaked by Cloudflare’s servers
- What would be the impact of those leaks
- How likely are those leaks to have occurred
We broke the data elements out into different groups:
- Elements that could be used to harm large numbers of users (e.g. privileged API tokens or credentials)
- Elements that affect a single user at a time (e.g. a user’s OAuth token or a single API response containing user data)
Using log data from our own systems, we investigated how many requests that contained these data elements had occurred during the affected period. As an example, we looked at how many users had logged in using a username and password during the affected period.
With this data in hand, we created a response plan. That plan focused on resolving issues that could affect a large number of users first. To that end we:
- Rotated the credentials of all privileged users on our platform
- Rotated our administrator credentials and API keys within Cloudflare’s management platform
- Rotated API keys with any of our other service providers that rely on Cloudflare
We then started to focus on issues that could affect individual users. Our concerns were leaked:
- Usernames and passwords
- Long-lived access tokens (as used by our mobile clients and third-party integrations)
- Other account data (e.g. step count)
While determining the best solution from an Engineering perspective, we wanted to balance the potential risk of account compromise with the stress, inconvenience and risk associated with forcing users to have to re-authenticate en-masse.
We considered our options, including forcing a password reset for all users, and invalidating all issued long-lived access tokens. The impact of this would have been that all users would have been forced to do a password reset and to re-authenticate on the web and on their mobile devices.
It is important to note that people of all ages and technical abilities rely on Fitbit to help them track their health and activity levels and many people use Fitbit to monitor the wellbeing of their loved ones. Experience has shown that many of our users had a friend or loved one set their device up for them and that even logging in without some support is a stressful and time consuming experience.
As a result, we took the following approach:
- Forced a password rotation and revoked the access tokens of the handful of users whose data we were able to see in internet caches. Our Customer Support team have contacted these users individually to ensure they are back online safely.
- Created a specific help page detailing how concerned customers can reset passwords and revoke application tokens
- Prepared tooling that would enable us to do a more widespread credential rotation and access token revocation in the event we started to see incidents relating to this
Like Cloudflare, we scoured the available internet caches for Fitbit data. We didn’t consider this to be the only attack vector, but wanted to get an idea of the likely scale of the issue. We found a handful of records of interest and have addressed them as outlined above, but ultimately we felt that Cloudflare’s assessment that a very small percentage of the requests processed by their platform were likely to have been affected was accurate.
We encourage any users that are concerned about this issue to reach out to our support teams. We’re working to ensure that we have the correct customer support resources on hand to quickly help anyone that feels they need help.
Contact details for support can be found at http://help.fitbit.com/?fs=ContactUs&cu=1