Junade Ali
Author Archives: Junade Ali
- Home → Author's archive:
Junade Ali
Using data science and machine learning for improved customer support
In this blog post we’ll explore three tricks that can be used for data science that helped us solve real problems for our customer support group and our customers. Two for natural language processing in a customer support context and one for identifying attack Internet attack traffic.
Through these examples, we hope to demonstrate how invaluable data processing tricks, visualisations and tools can be before putting data into a machine learning algorithm. By refining data prior to processing, we are able to achieve dramatically improved results without needing to change the underlying machine learning strategies which are used.
Know the Limits (Language Classification)
When browsing a social media site, you may find the site prompts you to translate a post even though it is in your language.
We recently came across a similar problem at Cloudflare when we were looking into language classification for chat support messages. Using an off-the-shelf classification algorithm, users with short messages often had their chats classified incorrectly and our analysis found there’s a correlation between the length of a message and the accuracy of the classification (based on the browser Accept-Language header and the languages of the country where the request was submitted):
On a Continue reading
Time-Based One-Time Passwords for Phone Support
As part of Cloudflare’s support offering, we provide phone support to Enterprise customers who are experiencing critical business issues.
For account security, specific account settings and sensitive details are not discussed via phone. From today, we are providing Enterprise customers with the ability to configure phone authentication to allow for greater support to be offered over the phone without need to perform validation through support tickets.
After providing your email address to a Cloudflare Support representative, you can now provide a token generated from the Cloudflare dashboard or via a 2FA app like Google Authenticator. So, a customer is able to prove over the phone that they are who they say they are.
Configuring Phone Authentication
If you are an existing Enterprise customer interested in phone support, please contact your Customer Success Manager for eligibility information and set-up. If you are interested in our Enterprise offering, please get in contact via our Enterprise plan page.
If you already have phone support eligibility, you can generate single-use tokens from the Cloudflare dashboard or configure an authenticator app to do the same remotely.
On the support page, you will see a card called “Emergency Phone Support Hotline – Authentication”. From here you Continue reading
Project Crossbow: Lessons from Refactoring a Large-Scale Internal Tool
Cloudflare’s global network currently spans 200 cities in more than 90 countries. Engineers working in product, technical support and operations often need to be able to debug network issues from particular locations or individual servers.
Crossbow is the internal tool for doing just this; allowing Cloudflare’s Technical Support Engineers to perform diagnostic activities from running commands (like traceroutes, cURL requests and DNS queries) to debugging product features and performance using bespoke tools.
In September last year, an Engineering Manager at Cloudflare asked to transition Crossbow from a Product Engineering team to the Support Operations team. The tool had been a secondary focus and had been transitioned through multiple engineering teams without developing subject matter knowledge.
The Support Operations team at Cloudflare is closely aligned with Cloudflare’s Technical Support Engineers; developing diagnostic tooling and Natural Language Processing technology to drive efficiency. Based on this alignment, it was decided that Support Operations was the best team to own this tool.
Learning from Sisyphus
Whilst seeking advice on the transition process, an SRE Engineering Manager in Cloudflare suggested reading: “A Case Study in Community-Driven Software Adoption”. This book proved a truly invaluable read for anyone thinking of doing internal tool development Continue reading
Pwned Passwords Padding (ft. Lava Lamps and Workers)
The Pwned Passwords API (part of Troy Hunt’s Have I Been Pwned service) is used tens of millions of times each day, to alert users if their credentials are breached in a variety of online services, browser extensions and applications. Using Cloudflare, the API cached around 99% of requests, making it very efficient to run.
From today, we are offering a new security advancement in the Pwned Passwords API - API clients can receive responses padded with random data. This exists to effectively protect from any potential attack vectors which seek to use passive analysis of the size of API responses to identify which anonymised bucket a user is querying. I am hugely grateful to security researcher Matt Weir who I met at PasswordsCon in Stockholm and has explored proof-of-concept analysis of unpadded API responses in Pwned Passwords and has driven some of the work to consider the addition of padded responses.
Now, by passing a header of “Add-Padding” with a value of “true”, Pwned Passwords API users are able to request padded API responses (to a minimum of 800 entries with additional padding of a further 0-200 entries). The padding consists of randomly generated hash suffixes with the usage Continue reading
Concise Christmas Cryptography Challenges 2019
Last year we published some crypto challenges to keep you momentarily occupied from the festivities. This year, we're doing the same. Whether you're bored or just want to learn a bit more about the technologies that encrypt the internet, feel free to give these short cryptography quizzes a go.
We're withholding answers until the start of the new year, to give you a chance to solve them without spoilers. Before we reveal the answers; if you manage to solve them, we'll be giving the first 5 people to get the answers right some Cloudflare swag. Fill out your answers and details using this form so we know where to send it.
Have fun!
NOTE: Hints are below the questions, avoid scrolling too far if you want to avoid any spoilers.
Challenges
Client says Hello
Client says hello, as follows:
00000c07ac01784f437dbfc70800450000f2560140004006db58ac1020c843c
7f80cd1f701bbc8b2af3449b598758018102a72a700000101080a675bce16787
abd8716030100b9010000b503035c1ea569d5f64df3d8630de8bdddd1152e75f
528ae577d2436949ce8deb7108600004400ffc02cc02bc024c023c00ac009c0
08c030c02fc028c027c014c013c012009f009e006b0067003900330016009d
009c003d003c0035002f000a00af00ae008d008c008b010000480000000b
000900000663666c2e7265000a00080006001700180019000b0002010000
0d0012001004010201050106010403020305030603000500050100000000
001200000017000052305655494338795157524c656d6443436c5246574651
675430346754456c4f52564d674e434242546b51674e513d3d
[Raw puzzle without text wrap]
Time-Based One-Time Password
A user has an authenticator device to generate one time passwords for logins to their banking website. The implementation contains a fatal flaw.
At the following times, the following codes are generated (all in GMT/UTC):
- Friday, 21 December 2018 16:29:28 - Continue reading
Banking-Grade Credential Stuffing: The Futility of Partial Password Validation
Recently when logging into one of my credit card providers, I was greeted by a familiar screen. After entering in my username, the service asked me to supply 3 random characters from my password to validate ownership of my account.
It is increasingly common knowledge in the InfoSec community that this practice is the antithesis of, what we now understand to be, secure password management.
For starters; sites prompting you for Partial Password Validation cannot store your passwords securely using algorithms like BCrypt or Argon2. If the service provider is ever breached, such plain-text passwords can be used to login to other sites where the account holder uses the same password (known as a Credential Stuffing attack).
Increased difficulty using long, randomly-generated passwords from Password Managers, leads to users favouring their memory over securely generated unique passwords. Those using Password Managers must extract their password from their vault, paste it somewhere else and then calculate the correct characters to put in. With this increased complexity, it further incentivises users to (re-)use simple passwords they can remember and count off on their fingers (and likely repeatedly use on other sites).
This is not to distinct thinking that originally bought us complex Continue reading
Optimising Caching on Pwned Passwords (with Workers)
In February, Troy Hunt unveiled Pwned Passwords v2. Containing over half a billion real world leaked passwords, this database provides a vital tool for correcting the course of how the industry combats modern threats against password security.
In supporting this project; I built a k-Anonymity model to add a layer of security to performed queries. This model allows for enhanced caching by mapping multiple leaked password hashes to a single hash prefix and additionally being performed in a deterministic HTTP-friendly way (which allows caching whereas other implementations of Private Set Intersection require a degree of randomness).
Since launch, PwnedPasswords, using this anonymity model and delivered by Cloudflare, has been implemented in a widespread way across a wide variety of platforms - from site like EVE Online and Kogan to tools like 1Password and Okta's PassProtect. The anonymity model is also used by Firefox Monitor when checking if an email is in a data breach.
Since it has been adopted, Troy has tweeted out about the high cache hit ratio; and people have been asking me about my "secret ways" of gaining such a high cache hit ratio. Over time I touched various pieces of Cloudflare's caching systems; in late 2016 Continue reading
Going Proactive on Security: Driving Encryption Adoption Intelligently
It's no secret that Cloudflare operates at a huge scale. Cloudflare provides security and performance to over 9 million websites all around the world, from small businesses and WordPress blogs to Fortune 500 companies. That means one in every 10 web requests goes through our network.
However, hidden behind the scenes, we offer support in using our platform to all our customers - whether they're on our free plan or on our Enterprise offering. This blog post dives into some of the technology that helps make this possible and how we're using it to drive encryption and build a better web.
Why Now?
Recently web browser vendors have been working on extending encryption on the internet. Traditionally they would use positive indicators to mark encrypted traffic as secure; when traffic was served securely over HTTPS, a green padlock would indicate in your browser that this was the case. In moving to standardise encryption online, Google Chrome have been leading the charge in marking insecure page loads as "Not Secure". Today, this UI change has been pushed out to all Google Chrome users globally for all websites: any website loaded over HTTP will be marked as insecure.
That's not all though; Continue reading
DNS-Over-TLS Built-In & Enforced – 1.1.1.1 and the GL.iNet GL-AR750S
GL.iNet GL-AR750S in black, same form-factor as the prior white GL.iNet GL-AR750. Credit card for comparison.
Back in April, I wrote about how it was possible to modify a router to encrypt DNS queries over TLS using Cloudflare's 1.1.1.1 DNS Resolver. For this, I used the GL.iNet GL-AR750 because it was pre-installed with OpenWRT (LEDE). The folks at GL.iNet read that blog post and decided to bake DNS-Over-TLS support into their new router using the 1.1.1.1 resolver, they sent me one to take a look at before it's available for pre-release. Their new router can also be configured to force DNS traffic to be encrypted before leaving your local network, which is particularly useful for any IoT or mobile device with hard-coded DNS settings that would ordinarily ignore your routers DNS settings and send DNS queries in plain-text.
In my previous blog post I discussed how DNS was often the weakest link in the chain when it came to browsing privacy; whilst HTTP traffic is increasingly encrypted, this is seldom the case for DNS traffic. This makes it relatively trivial for an intermediary to work out what site you're sending Continue reading
Privacy-Protecting Portable Router: Adding DNS-Over-TLS support to OpenWRT (LEDE) with Unbound
If you want to skip ahead to instructions, scroll to the next section. But I, like a TLS handshake, am very verbose so please enjoy this opener.
Imagine this scenario - I'm at a restaurant and need to have a private phone conversation but unfortunately my phone's battery is drained. To get around this problem, I borrow my friend's phone and dial the number - to protect my privacy I walk outside. When I'm done with the call, I come back inside and return the phone.
Whilst the phone itself doesn't store the coversation I've had, it does have a log of the recently dialed number, if the friend from whom I borrowed the phone wanted to, they could easily see who I actually called - even if they don't specifically know the topic of conversation.
Sometimes, the data about who you've spoken to can tell an aweful lot about the conversation - if someone was to call an emotional support hotline or a debt collector, you could probably infer a lot about the conversation from the caller ID.
When we browse the internet, we use encryption to try and protect the conversations we have. When you connect to a Continue reading
Validating Leaked Passwords with k-Anonymity
Today, v2 of Pwned Passwords was released as part of the Have I Been Pwned service offered by Troy Hunt. Containing over half a billion real world leaked passwords, this database provides a vital tool for correcting the course of how the industry combats modern threats against password security.
I have written about how we need to rethink password security and Pwned Passwords v2 in the following post: How Developers Got Password Security So Wrong. Instead, in this post I want to discuss one of the technical contributions Cloudflare has made towards protecting user information when using this tool.
Cloudflare continues to support Pwned Passwords by providing CDN and security functionality such that the data can easily be made available for download in raw form to organisations to protect their customers. Further; as part of the second iteration of this project, I have also worked with Troy on designing and implementing API endpoints that support anonymised range queries to function as an additional layer of security for those consuming the API, that is visible to the client.
This contribution allows for Pwned Passwords clients to use range queries to search for breached passwords, without having to disclose a complete unsalted Continue reading
How Developers got Password Security so Wrong
Both in our real lives, and online, there are times where we need to authenticate ourselves - where we need to confirm we are who we say we are. This can be done using three things:
- Something you know
- Something you have
- Something you are
Passwords are an example of something you know; they were introduced in 1961 for computer authentication for a time-share computer in MIT. Shortly afterwards, a PhD researcher breached this system (by being able to simply download a list of unencrypted passwords) and used the time allocated to others on the computer.
As time has gone on; developers have continued to store passwords insecurely, and users have continued to set them weakly. Despite this, no viable alternative has been created for password security. To date, no system has been created that retains all the benefits that passwords offer as researchers have rarely considered real world constraints[1]. For example; when using fingerprints for authentication, engineers often forget that there is a sizable percentage of the population that do not have usable fingerprints or hardware upgrade costs.
Cracking Passwords
In the 1970s, people started thinking about how to better store passwords and cryptographic hashing started to Continue reading
Concise (Post-Christmas) Cryptography Challenges
It's the day after Christmas; or, depending on your geography, Boxing Day. With the festivities over, you may still find yourself stuck at home and somewhat bored.
Either way; here are three relatively short cryptography challenges, you can use to keep you momentarily occupied. Other than the hints (and some internet searching), you shouldn't require a particularly deep cryptography knowledge to start diving into these challenges. For hints and spoilers, scroll down below the challenges!
Challenges
Password Hashing
The first one is simple enough to explain; here are 5 hashes (from user passwords), crack them:
$2y$10$TYau45etgP4173/zx1usm.uO34TXAld/8e0/jKC5b0jHCqs/MZGBi
$2y$10$qQVWugep3jGmh4ZHuHqw8exczy4t8BZ/Jy6H4vnbRiXw.BGwQUrHu
$2y$10$DuZ0T/Qieif009SdR5HD5OOiFl/WJaDyCDB/ztWIM.1koiDJrN5eu
$2y$10$0ClJ1I7LQxMNva/NwRa5L.4ly3EHB8eFR5CckXpgRRKAQHXvEL5oS
$2y$10$LIWMJJgX.Ti9DYrYiaotHuqi34eZ2axl8/i1Cd68GYsYAG02Icwve
HTTP Strict Transport Security
A website works by redirecting its www. subdomain to a regional subdomain (i.e. uk.), the site uses HSTS to prevent SSLStrip attacks. You can see cURL requests of the headers from the redirects below, how would you practically go about stripping HTTPS in this example?
$ curl -i http://www.example.com
HTTP/1.1 302 Moved Temporarily
Server: nginx
Date: Tue, 26 Dec 2017 12:26:51 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
location: https://uk.example.com/
$ curl -i http://uk.example.com
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; Continue readingConcise (Post-Christmas) Cryptography Challenges
It's the day after Christmas; or, depending on your geography, Boxing Day. With the festivities over, you may still find yourself stuck at home and somewhat bored.
Either way; here are three relatively short cryptography challenges, you can use to keep you momentarily occupied. Other than the hints (and some internet searching), you shouldn't require a particularly deep cryptography knowledge to start diving into these challenges. For hints and spoilers, scroll down below the challenges!
Challenges
Password Hashing
The first one is simple enough to explain; here are 5 hashes (from user passwords), crack them:
$2y$10$TYau45etgP4173/zx1usm.uO34TXAld/8e0/jKC5b0jHCqs/MZGBi
$2y$10$qQVWugep3jGmh4ZHuHqw8exczy4t8BZ/Jy6H4vnbRiXw.BGwQUrHu
$2y$10$DuZ0T/Qieif009SdR5HD5OOiFl/WJaDyCDB/ztWIM.1koiDJrN5eu
$2y$10$0ClJ1I7LQxMNva/NwRa5L.4ly3EHB8eFR5CckXpgRRKAQHXvEL5oS
$2y$10$LIWMJJgX.Ti9DYrYiaotHuqi34eZ2axl8/i1Cd68GYsYAG02Icwve
HTTP Strict Transport Security
A website works by redirecting its www. subdomain to a regional subdomain (i.e. uk.), the site uses HSTS to prevent SSLStrip attacks. You can see cURL requests of the headers from the redirects below, how would you practically go about stripping HTTPS in this example?
$ curl -i http://www.example.com
HTTP/1.1 302 Moved Temporarily
Server: nginx
Date: Tue, 26 Dec 2017 12:26:51 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
location: https://uk.example.com/
$ curl -i http://uk.example.com
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; Continue readingSimple Cyber Security Tips (for your Parents)
Today, December 25th, Cloudflare offices around the world are taking a break. From San Francisco to London and Singapore; engineers have retreated home for the holidays (albeit with those engineers on-call closely monitoring their mobile phones).
Software engineering pro-tip:
— Chris Albon (@chrisalbon) December 20, 2017
Do not, I repeat, do not deploy this week. That is how you end up debugging a critical issue from your parent's wifi in your old bedroom while your spouse hates you for abandoning them with your racist uncle.
Whilst our Support and SRE teams operated on a schedule to ensure fingers were on keyboards; on Saturday, I headed out of the London bound for the Warwickshire countryside. Away from the barracks of the London tech scene, it didn't take long for the following conversation to happen:
- Family member: "So what do you do nowadays?"
- Me: "I work in Cyber Security."
- Family member: "There seems to be a new cyber attack every day on the news! What can I possibly do to keep myself safe?"
If you work in the tech industry, you may find a family member asking you for advice on cybersecurity. This blog post will hopefully save you Continue reading
Simple Cyber Security Tips (for your Parents)
Today, December 25th, Cloudflare offices around the world are taking a break. From San Francisco to London and Singapore; engineers have retreated home for the holidays (albeit with those engineers on-call closely monitoring their mobile phones).
Software engineering pro-tip:
— Chris Albon (@chrisalbon) December 20, 2017
Do not, I repeat, do not deploy this week. That is how you end up debugging a critical issue from your parent's wifi in your old bedroom while your spouse hates you for abandoning them with your racist uncle.
Whilst our Support and SRE teams operated on a schedule to ensure fingers were on keyboards; on Saturday, I headed out of the London bound for the Warwickshire countryside. Away from the barracks of the London tech scene, it didn't take long for the following conversation to happen:
- Family member: "So what do you do nowadays?"
- Me: "I work in Cyber Security."
- Family member: "There seems to be a new cyber attack every day on the news! What can I possibly do to keep myself safe?"
If you work in the tech industry, you may find a family member asking you for advice on cybersecurity. This blog post will hopefully save you Continue reading
The Curious Case of Caching CSRF Tokens
It is now commonly accepted as fact that web performance is critical for business. Slower sites can affect conversion rates on e-commerce stores, they can affect your sign-up rate on your SaaS service and lower the readership of your content.
In the run-up to Thanksgiving and Black Friday, e-commerce sites turned to services like Cloudflare to help optimise their performance and withstand the traffic spikes of the shopping season.
In preparation, an e-commerce customer joined Cloudflare on the 9th November, a few weeks before the shopping season. Instead of joining via our Enterprise plan, they were a self-serve customer who signed-up by subscribing to our Business plan online and switching their nameservers over to us.
Their site was running Magento, a notably slow e-commerce platform - filled with lots of interesting PHP, with a considerable amount of soft code in XML. Running version 1.9, the platform was somewhat outdated (Magento was totally rewritten in version 2.0 and subsequent releases).
Despite the somewhat dated technology, the e-commerce site was "good enough" for this customer and had done it's job for many years.
They were the first to notice an interesting technical issue surrounding how performance and security can often Continue reading
The Curious Case of Caching CSRF Tokens
It is now commonly accepted as fact that web performance is critical for business. Slower sites can affect conversion rates on e-commerce stores, they can affect your sign-up rate on your SaaS service and lower the readership of your content.
In the run-up to Thanksgiving and Black Friday, e-commerce sites turned to services like Cloudflare to help optimise their performance and withstand the traffic spikes of the shopping season.
In preparation, an e-commerce customer joined Cloudflare on the 9th November, a few weeks before the shopping season. Instead of joining via our Enterprise plan, they were a self-serve customer who signed-up by subscribing to our Business plan online and switching their nameservers over to us.
Their site was running Magento, a notably slow e-commerce platform - filled with lots of interesting PHP, with a considerable amount of soft code in XML. Running version 1.9, the platform was somewhat outdated (Magento was totally rewritten in version 2.0 and subsequent releases).
Despite the somewhat dated technology, the e-commerce site was "good enough" for this customer and had done it's job for many years.
They were the first to notice an interesting technical issue surrounding how performance and security can often Continue reading
The New DDoS Landscape
News outlets and blogs will frequently compare DDoS attacks by the volume of traffic that a victim receives. Surely this makes some sense, right? The greater the volume of traffic a victim receives, the harder to mitigate an attack - right?
At least, this is how things used to work. An attacker would gain capacity and then use that capacity to launch an attack. With enough capacity, an attack would overwhelm the victim's network hardware with junk traffic such that they can no longer serve legitimate requests. If your web traffic is served by a server with a 100 Gbps port and someone sends you 200 Gbps, your network will be saturated and the website will be unavailable.
Recently, this dynamic has shifted as attackers have gotten far more sophisticated. The practical realities of the modern Internet have increased the amount of effort required to clog up the network capacity of a DDoS victim - attackers have noticed this and are now choosing to perform attacks higher up the network stack.
In recent months, Cloudflare has seen a dramatic reduction in simple attempts to flood our network with junk traffic. Whilst we continue to see large network level attacks, in Continue reading
Performing & Preventing SSL Stripping: A Plain-English Primer
Over the past few days we learnt about a new attack that posed a serious weakness in the encryption protocol used to secure all modern Wi-Fi networks. The KRACK Attack effectively allows interception of traffic on wireless networks secured by the WPA2 protocol. Whilst it is possible to backward patch implementations to mitigate this vulnerability, security updates are rarely installed universally.
Prior to this vulnerability, there were no shortage of wireless networks that were vulnerable to interception attacks. Some wireless networks continue to use a dated security protocol (called WEP) that is demonstrably "totally insecure" 1; other wireless networks, such as those in coffee shops and airports, remain completely open and do not authenticate users. Once an attacker gains access to a network, they can act as a Man-in-the-Middle to intercept connections over the network (using tactics known as ARP Cache Poisoning and DNS Hijacking). And yes, these interception tactics can easily be deployed against wired networks where someone gains access to an ethernet port.
With all this known, it is beyond doubt that it is simply not secure to blindly trust the medium that connects your users to the internet. HTTPS was created to allow HTTP traffic to Continue reading