Engineering Fitness

Fitbit Response to Cloudflare Security Issue

On February 23, 2017, Google Project Zero and Cloudflare revealed the existence of the Cloudbleed bug. Fitbit uses Cloudflare as our content distribution network and the majority of our web and API traffic routes through the Cloudflare platform.

We learned of the issue the same way as everyone else – when security-minded team members at Fitbit alerted us to the Project Zero ticket and the associated Cloudflare Blog post.

For a complete description of the the bug details, we’d recommend reading Cloudflare’s blog post. In short though, a software error meant that in some situations memory containing parts of in-flight HTTP requests were leaking from Cloudflare’s systems.

Although the likelihood of these issues affecting our users is incredibly low, we have taken a number of preventative actions and have provided detailed guidelines to our consumers via our Help site. We hope through this post to share the steps that we took to investigate and respond to this incident.

After learning of the issue, we immediately started working to understand the potential impact of the issue for Fitbit users. We focused on three main questions:

  1. Which data elements may have been leaked by Cloudflare’s servers
  2. What would be the impact of those leaks
  3. How likely are those leaks to have occurred

We broke the data elements out into different groups:

  1. Elements that could be used to harm large numbers of users (e.g. privileged API tokens or credentials)
  2. Elements that affect a single user at a time (e.g. a user’s OAuth token or a single API response containing user data)

Using log data from our own systems, we investigated how many requests that contained these data elements had occurred during the affected period. As an example, we looked at how many users had logged in using a username and password during the affected period.

With this data in hand, we created a response plan. That plan focused on resolving issues that could affect a large number of users first. To that end we:

  1. Rotated the credentials of all privileged users on our platform
  2. Rotated our administrator credentials and API keys within Cloudflare’s management platform
  3. Rotated API keys with any of our other service providers that rely on Cloudflare

We then started to focus on issues that could affect individual users. Our concerns were leaked:

  1. Usernames and passwords
  2. Long-lived access tokens (as used by our mobile clients and third-party integrations)
  3. Other account data (e.g. step count)

While determining the best solution from an Engineering perspective, we wanted to balance the potential risk of account compromise with the stress, inconvenience and risk associated with forcing users to have to re-authenticate en-masse.

We considered our options, including forcing a password reset for all users, and invalidating all issued long-lived access tokens. The impact of this would have been that all users would have been forced to do a password reset and to re-authenticate on the web and on their mobile devices.

It is important to note that people of all ages and technical abilities rely on Fitbit to help them track their health and activity levels and many people use Fitbit to monitor the wellbeing of their loved ones. Experience has shown that many of our users had a friend or loved one set their device up for them and that even logging in without some support is a stressful and time consuming experience.

As a result, we took the following approach:

  1. Forced a password rotation and revoked the access tokens of the handful of users whose data we were able to see in internet caches. Our Customer Support team have contacted these users individually to ensure they are back online safely.
  2. Created a specific help page detailing how concerned customers can reset passwords and revoke application tokens
  3. Prepared tooling that would enable us to do a more widespread credential rotation and access token revocation in the event we started to see incidents relating to this

Like Cloudflare, we scoured the available internet caches for Fitbit data. We didn’t consider this to be the only attack vector, but wanted to get an idea of the likely scale of the issue. We found a handful of records of interest and have addressed them as outlined above, but ultimately we felt that Cloudflare’s assessment that a very small percentage of the requests processed by their platform were likely to have been affected was accurate.

We encourage any users that are concerned about this issue to reach out to our support teams. We’re working to ensure that we have the correct customer support resources on hand to quickly help anyone that feels they need help.

Contact details for support can be found at http://help.fitbit.com/?fs=ContactUs&cu=1

And as always, members of the security community can get in touch with us at security@fitbit.com (GPG key at https://www.fitbit.com/security-publickey.txt)

1 Comment   Join the Conversation

1 CommentLeave a comment

  • I posted this in a different topic (about Fitbit / Cloudflare blocking legitimate VPN users), but Cloudflare seems to be causing a lot of problems. Fitbit at least need to move the URLs on the app away from Cloudflare. I could live without the browsing to the website, but without syncing my Aria scales is just a large paperweight.

    Similar issue here but with direct browsing – i.e. not via VPN. I get a maximum of 48 hours access to fitbit.com and the app and then neither work for around 5 days – this has gone on for nearly two months.

    Like previous posters my Charge HR, Surge and Aria are useless if I can’t sync and Fitbit’s suggestion to use another network is impossible at home as I live in a very rural area with no phone data service. Although I was a loyal Fitbit customer, that isn’t enough to make me take my Aria scales for a ride in the car until we get a signal every time I weigh myself.

    After 4 years with Fitbit, I now have no choice but to move to probably Garmin.

    I provided large amounts of accurate data and diagnostics initially to Fitbit and then also to Cloudflare (who I had never heard of before) over many weeks, but got no sensible responses. Because I didn’t get any answers – other than being assured for weeks by both companies that my IP was not blacklisted (despite the blocked access), I even signed up for a Cloudflare account and migrated an old website to them, so that I could see for myself the reason my IP was being blocked. This also meant that I could contact Cloudflare support.

    My house network IP was indeed security checked (with captchas) when accessing my own website – but the Cloudflare dashboard was useless for diagnosing the reason – just showed zero threats and no firewall records at all when I entered the Cloudflare Ray IDs or my IP (both are shown at the bottom of every captcha screen). That’s probably all Fitbit network techs see as well – hence the reason they couldn’t give sensible answers to why I was blocked – they wouldn’t know why – only Cloudflare seem to know and they keep it secret.

    So, it is quite likely that despite Cloudflare implying that security level is controlled by their customers, Cloudflare’s own systems seem to be increasingly “black-box” controlling most of the protection for their customers, but I doubt that many Fitbit and Uber paying customers know that their internet traffic is all going via Cloudflare and certainly would not have an account to get support. Also, Cloudflare clearly can’t (and generally don’t) provide support to all the users of their millions of customer’s websites.

    In my own experience, my excellent ISP (Zen Internet) after 2 weeks of investigations correctly stated that it was a Cloudflare problem. Cloudflare stated that the protection level was up to each of their customers. But Fitbit initially told me that I should mention the problem with my IP to my ISP and then, when I said that I already had, told me that the “core” problem was outside of Fitbit i.e. Cloudflare. A perfect pass-the-buck circle and perfect recipe for user frustration!

    It looks like companies are now choosing to buy cheap or free DDOS protection from CDNs like Cloudflare rather than provide it themselves. But that is putting huge power into the hands of these CDNs over the internet access of the hundreds of millions of users of their customer sites. My ISP and BT’s behaviour are regulated by OFCOM in the UK, but who is regulating these usually foreign based CDNs who are getting increasing power? Cloudflare appears to act as a prosecutor, Judge and Jury in cases of alleged “internet bad behaviour”. However, this is effectively being done in secret (even their customers like Fitbit don’t know the Cloudflare IP reputation algorithm) and with the accused usually having no knowledge and certainly no access to evidence or being able to respond to the accusation. Sounds slightly worse than justice in the worst dictatorships.

    An attractive feature of Cloudflare’s product is that in true “big-brother” style they see the traffic of individual internet user access over millions of their customers sites. I was amazed how many of the sites that I visit all went through Cloudflare. This means that falsely identified bad traffic to just a few sites or perhaps even one, could damage the user’s IP reputation and hence block legitimate access to a very large part of the internet – again with no way of finding out what is happening. There are plenty of not very well run (or disreputable) sites routed via Cloudflare and in my experience Cloudflare does not check any of them. When I signed my test website up Cloudflare, I could have set up all sorts of weird behaviour or irrational security levels which presumably could have adversely impacted the global IP reputation of legitimate visitors.

    Cloudflare claim to be “one of the world’s largest networks, powering more than 10 trillion requests per month”. This amount of centralisation of vital internet functions means that if things go wrong it can potentially impact hundreds of millions of users across millions of vendors or websites. The classic example was “Cloudbleed” a few months ago where Google’s engineers’ discovered that a serious bug in Cloudflare had been leaking raw customer information for months including actual passwords, dating messages etc – i.e. when you accessed you Fitbit account, in addition to your information your browser might also get sent whole chunks of unencrypted information from other Cloudflare users on the same server e.g. from Uber, OkCupid, RTE.ie etc. Hence leak was far worse than the usual leak or hack of just one company. However, even worse, in Cloudflare’s case, the leaked data was being cached by Search engines across the globe – including those in unfriendly countries – who may or may not have responded to the subsequent major “purge” exercise that followed and hence easily available to all the bad guys on the internet with no hacking required.

    Back to my problems – eventually when I pushed hard for responses to specific questions about the diagnostics and their illogical replies, both support teams “closed the tickets” within a day of each other. I felt this was very disrespectful to me as my customer (especially with my investment in Fitbit devices) and the time that I had wasted to help the support teams.

    So, I escalated to Board level in both companies. To his great credit, Mathew Prince, CEO and founder of Cloudflare, very shortly after receiving my evening email, did pass it to his Cloudflare WAF Product Manager who responded at 21:30 with some data from their systems which I could easily map to my own network traffic logs.

    However, over the past week the problem has still persisted and despite Cloudflare’s data showing occasions of moderately high hits to Cloudflare sites – none of the traffic from my house is malicious in any way (I have now analysed every single TCP hit and DNS query from my house). Incidentally, my NetFlow analysis actually shows that my IP has more hits (than Cloudflare) to other CDN sites e.g., Akamai etc – obviously none of whom are blocking or restricting my completely legitimate traffic. This only started 2 months ago, and I haven’t changed by browsing habits at all – so I suspect this is an overzealous change at Cloudflare that is now targeting innocent internet users or being very generous, possibly users who are inadvertently falling foul of some website peculiarity.

    I again followed up (directly with support to respect their executives’ time) and got a very strange response from their previously friendly Product Manager along the lines that “we have already told you numerous times that you are running bots to target our customers sites” and when you stop this your IP reputation will recover. I wouldn’t even know how to “run a bot” and Cloudflare had absolutely never told me this before let alone numerous times.

    I sent an email back saying that I was going to not give up chasing them or Fitbit, but later I did decide to just give up – despite the major disruption to my internet use – life is too short. Hopefully eventually enough of Cloudflare’s customers like Fitbit will get complaints from genuine users and report this to Cloudflare, who will then modify their systems to better focus on the real bad guys and not block legitimate users. In the meantime, I have no choice except to join your club – but for the opposite reasons – i.e. with VPN’s and proxy servers – none of which are convenient for what I use the internet for. Previously I didn’t see the need for VPN for home use, but now that I know about CDNs etc, it is very scary that these fairly small companies can see so much traffic for so many users that they don’t even have a contractual relationship with.

    Fitbit didn’t tell me when I signed up that they were going to route all my traffic and personal data to a company that I had never heard of (and allow the same company huge power over my everyday life – literally – by interfering with my internet access which I regard as an essential utility the same as water and electricity). I wouldn’t be happy if the nice lady who runs my village post-office was opening and reading all of my mail – with the internet it is obviously far worse as the volume, scope and potential risk is much higher.

    Also, my premium ISP only issues static IPs which suits me, but it looks certain that if I had the norm (in the UK at least) i.e. a dynamic IP address I wouldn’t be getting any false IP reputation issues – at least not for 2 months anyway. So, I could change to a cheaper and less reliable ISP – which I really don’t want to do, any more than you all want to reduce your internet privacy by ditching your VPNs.

If you have questions about a Fitbit tracker, product availability, or the status of your order, contact our Support Team or search the Fitbit Community for answers.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.