Introduction
I have been doing Bug bounties for over 8 years now as a researcher on different platforms (Hackerone, Bugcrowd, Synack etc) but recently I had recent experience in bug bounties from the other side of the table, from the program side. After conducting the feature assessment and POCs of major platforms (an activity spanning over three months) I would like to share my experience which would hopefully help you choose the “right” platform to achieve your bounty objectives. This Bug Bounty Platform Comparison post focuses on why you need a bug bounty platform, criteria, and considerations in choosing the bug bounty platform, Synack vs others comparison, high-level comparison between Hackerone, Bugcrowd, Intigriti, YesWeHack and finally some thoughts on the success and failure of your bug bounty program.
Why do we need a Bug Bounty platform?
It’s a good question that every organization must think about before opting for a bug bounty platform, why would you need a bug bounty platform when you can get reports directly from researchers? All you have to do is advertise on your website that you are starting out a bug bounty program, define and publish your scope, rules, exceptions, vulnerability disclosure guidelines, and establish a reporting process, an email where researchers can submit the vulnerability details or a web form and viola! you got yourself a bug bounty program, all you have to do now is sit back and let the vulnerability reports clog up your inbox. Well, not really, let’s reason this out, shall we? Certainly, there are some benefits the bug bounty platforms provide, and on the other hand, if you start a bug bounty program without a platform you don’t have to pay for the services costs of those platforms, which is a hefty number (sometimes its more than you have to pay for the bounties). Following are some of the reasons why you need a bug bounty platform:-
- Initial Triage – (mark duplicates, remove invalid reports, remove low severity reports, assign a severity, etc) is done by the bug bounty platform staff. if you choose to do this yourself you need to have a dedicated resource that would look after the triage.
- Detailed metrics reports – To track efficiency, bring transparency, and build/monitor KPIs. The level of metrics/reports depend on individual platforms but I have seen that all major bug bounty platforms do have a feature to get metrics/reports.
- Integrations – With issue tracking, notification systems and platforms (Jira, Slack etc). This is dependent on the specific bug bounty platform, not all platforms provide integrations out of the box.
- Role based access control – All major bug bounty platforms provide different levels of RBAC, like editor, administrator, read-only etc. Some platforms like Hackerone provides you the ability to create a customized access profile as well where you can choose what access a role should have. If you choose a self hosted bug bounty program you end up risking access of actions/information to employees who shouldn’t have access to those (remember the principle of leave privilege and need to know?)
- Access to researchers – Since thousands of researchers are already on the bug bounty platforms looking for programs to find vulnerabilities in, you instantly get access to these researchers, otherwise if you have to advertise that you have started a bug bounty platform and you are expecting researchers to find and report vulnerabilities.
- Bug bounty program success management – Most of the major platforms (hackerone, Synack, Bugcrowd, Intigriti, YesWeHack) assign you a customer success manager that will work with you to make sure your are getting the best out of the platform and your bug bounty program goals and KPI’s are being met. They will help you with scope adjustments, platform features, filtering of researchers, bounty pay-outs selection etc to increase the program effectiveness. If you don’t opt for a bug bounty platform and go for a self hosted program you have to dedicate additional time and resources to ensure program effectiveness.
- The Cost – This is the only positive one can think of for self hosted bug bounty program rather than paying for a bug bounty platform. This is an important one, I had the opportunity to review quotes from all major bug bounty platforms and I can tell you that they are not cheap! it is a real possibility that you pay more to the platform than you pay to security researchers (which is your primary audience). On the other hand if you are looking for all the features listed above and cost isn’t a major factor then your decision should be simple.
Considerations for platform evaluation
Now that you are SOLD you getting on a bug bounty platform, the question you must be asking which platform is the “best”. Well, you aren’t buying flavoured yogurt!, so its a bit complicated, it depends on what are your bug bounty goals, what compliance requirements do you have internally or externally (ill explain this point later), that are your business objectives, how deep are your pockets and which features are a must have while which ones are nice to have etc. Some of the considerations (These do not include specific features, but they are more on the high level) you should have for assessing the platforms are as follow :-
- Do you want you program to be public (any researcher can find it and submit vulnerabilities) or would you like to make it invite only and private?
- Do you want a fixed priced program which would include platform service charges + bounties or do you want to have them separate?
- Do you have compliance or regulatory requirements to keep logs of activities? which means all activities of researchers must pass through a VPN?
- Do you have a limit in mind of the number of vulnerabilities to expect per year? or you don’t want to cap that?
- How deep are your pockets? how much would you like to spend on your bug bounty program?
- Do you want Triage services, integrations, VDP?
Bug Bounty Platform Comparison
Before we get into comparing all the bug bounty platform, I would like to compare all of them with Synack first. Synack is different, their pricing model is different. Their onboarding of researchers is a lot different and the way they operate the program is different from all other bug bounty platforms. All other platforms (Hackerone, Bugcrowd, Intigriti, YesWeHack) are similar to one another in regards to their pricing model, their public and private programs, the bounty payment structure etc. So Lets firstly compare Synack with all others (Hackerone, Bugcrowd, Intigriti, YesWeHack).
Synack Hackerone, Bugcrowd, Intigriti, YesWeHack Fixed Priced contract, Synack pays the bounties itself to researchers and charges you with a fixed-priced contract which includes bounties + platform service charges Contract is divided into platform service charges and bounty amounts. You have to payout each valid submission based on your criteria/severity levels. Lack of transparency because as a program owner you don't know how much your funds have gone to Synack's platform services and how much has gone to researchers. Important to note that it could go either way, bounties could get more than what you expected to pay per year. Platform service charges and bounty amounts are separate, you can set the bounty amount yourself on how much you want to spend on bounties per year. There is more transparency on how much you would like to spend on bounties per year. Faster bounties to researchers (because of the fixed-priced contract Synack releases the bounties as soon as it confirms the vulnerability, without having to go back to the program owner. Typically Synack releases bounties in a few hours. Bounty payouts could take time, the program has to fix the vulnerability, estimate how much should be paid, and releases the payout, this could take from weeks to months, but this is the kind of difference that is more appealing to bounty researchers than program owners. Onboarding researchers for Synack Red Team is very detailed process in comparision to other platforms, you have to apply first with your CV (as you normally do for a job), you get interviewed (non-tech interview), then you have to pass a CTF styled exam for each category you want to hunt in, for example web application has its own exam, mobile and network has their own exams, and based on person experience, these CTF styled exams are tough!. As a program owner this means that your application will only be tested by the very best the bug bounty industry has to offer, this would result in quality high-value submissions. Onboarding researchers is very simple, researcher just need to sign up on the plaform and starts hunting for bounties! there is no exam, no filter or criteria, anyone can participate, as a program owner this means that you will get loads of low value, low severity bug bounty submissions. There will be more noise on your network/applications. There is no criteria on any researcher to be able to participate in your program. This also means lack of transparency and logging. VPN for researchers comes standard in the package and you don't have to pay for that, in fact, all researchers working on Synack must use their VPN to be able to do bug bounty hunting, it would not be possible otherwise. As a program owner, it means that you will have logs of all the activities performed by the researchers on your program, this comes in very handy in any incident response activity or compliance with regulations for some industries. VPN for researchers is not standard in any of the platforms (HackerOne, Bugcrowd, Integrity, YesWeHack). However, some platforms like HackerOne do provide the ability for a VPN but this is a paid service and is added on top of your contract. As a program owner it means that you will have no visibility of what the researcher is doing on your platform, it could also be an issue in incident response type situations, and depending on your sector of the industry you may require logs. Program on Synack is always a private program because the way Synack is set up requires researchers to go through a rigorous process to be selected once you get selected only then do you have the opportunity to see the programs. As a program owner, you may have to weigh the benefits and risks associated with public vs private programs Other platforms provide you the ability to either set your program to Public or private, in the private program only a limited number of researchers are allowed to participate, you can use some filter criteria to select which researchers should be allowed to participate in the program. Because synack is a fixed-priced program it doesn't allow you the ability to change the severity of the vulnerabilities, and normally this shouldn't matter to you because Synack is taking care of the payments but you do need to sometimes change the severity internally becasue severity contributes to prioritizing mitigations. You can easily change the severity of any vulnerability reported, change in severity must have a just because it contributes to how much the researcher is going to be paid. High severity means high reward.
As you can see Synack does things differently than other platforms, to say that its better or worse than other platform is entirely subjective to what you want out of a bug bounty platform, if you are looking at VPN to collect logs, high quality researchers and high quality vulnerability reports, a private-only fixed budget program then Synack is your choice, on the other hand if you are looking at more transparency around platform service charges and bounty amount to be paid to researchers then probably you should evaluate other platforms.
Hackerone, Bugcrowd, Intigriti, YesWeHack
Now that we have compared Synack vs all others, lets compare Hackerone, Bugcrowd, Intigriti, YesWeHack to each other. All of these platforms have a lot of common features, they also have divided costs between platform services charges and bounty amount. Before we dive into major differences between these platforms I would like to firstly mention the commonalities of features all of these platforms have to offer:-
- Triage services are standard, which means platform staff will discard duplicates, invalid reports, out of scope submissions etc.
- Program metric are provided by all of these platforms with some level of differences on the details.
- All of the platforms have the option to make your program publicly accessible or private (invite only).
- All the platforms have the option to change severity of the reported vulnerability, mark it as duplicate, change its state etc.
- All the programs have the option to change/update program scope, some of them have nice wizards that helps you in scope definitions etc.
Lets discuss on a high level some of the differences each platform has, note that the comparison is not feature by feature and a lot of those features are subjective, but this is more about the overall picture I got from doing POCs and assessments over the last few months.
Hackerone
- Biggest community of hackers, as a program owner it means you will get more vulnerability reports.
- Expensive than its competitors.
- In my assessment of its features it ticks all the boxes as far as features are concerned.
- Oldest and most mature platform.
- Unlimited number of vulnerability reports per year.
Intigriti
- They call themselves the Europe’s number 1 bug bounty platform, which I find to be an exaggeration.
- They keep their researchers engaged by CTFs/quizzes.
- They market themselves a European platform, as a program owner it means you don’t need to worry about any EU compliance, they got it covered.
- Flexibility in managing public and private programmes, in fact they have range of options for program management.
- Cheaper compared to Hackerone. (please note that price is subjective to scope)
- They offer limited number of vulnerability report per year. (What’s that about? I don’t get it)
YesWeHack
- The newest platform among others, less mature but aims to address pain points of other platforms.
- They offer limited number of vulnerability report per year. (What’s that about? It makes very-little-to-no-sense)
- Cheaper compared to hackerone (about the same as Intigriti)
Bugcrowd
- An old platform, their customer success management requires some work.
- Unlimited number of vulnerability reports per year.
- Scored the lowest in feature comparison.
Final thoughts
All of these platforms provide a some level of assurance for your bug bounty spending, all things considered they are similar to each other in-terms of what they offer, it also depends on you as a program owner to make your bug bounty program a success that achieves your objectives and justifies the spending. As a program owner you need to make sure you respond back to researchers in a reasonable amount to time, researchers talk to each other about which program is good, so you want to be in their good books, ultimately your objective is to get more vulnerability reports and it is only possible when you are prompt, pay on time, your scope has no ambiguity and you are paying well. The failure of your program means that you are not getting vulnerability reports, you don’t want that! Imagine spending $100K on platform fees per year and you get $10K worth of vulnerability reports! You cannot justify the platform or bug bounty spending my these statistics. You need to remain in close contact with the customer success manager assigned to you to keep optimizing the program for better results (more reports). Hola at me on linkedin or Twitter if you would like to discuss anything related to bug bounty platform comparison etc.
For Intigriti you state
“They call themselves the Europe’s number 1 bug bounty platform, which I find to be an exaggeration.”
Who would you consider the biggest in Europe? The other competitor I see her in Europe is YesWeHack, but they seem smaller…
Hi H4ck3rM4n,
I was pointing out that their sales pitch of being the biggest platform in Europe is actually not correct. I did ask them the same question during a POC session and they said, well you know its a marketing stunt but we are actually the third largest. If you browse their website it still says on the landing page that its the number 1 platform in Europe.