This is how I reduce the noise in Burp's HTTP history when testing thick clients. You can use the methods here to create your own Burp configuration file or build upon the one I have created. I am going to identify common noisy requests that appear in Windows and then ignore them in Burp.
TL;DR
- Proxy the applications directly.
- Application and tech stack specific proxy settings.
- Browsers: Use Firefox or pass
--proxy-server="http://localhost:8080"
to Chrome and Edge.
- Use a Virtual Machine.
- Use Burp's scope and filter to hide requests in
HTTP History
. - Add domains to Burp's
TLS Pass Through
.- Noisy domains. E.g., Windows update.
- Authentication/MFA domains. You do not want your credentials in Burp.
- Hide
OPTIONS
requests in Burp's history with extensions. - Use this Burp config file as base and add your domains.
Motivation
Noisy history == missing important requests.
Noise in Burp's History
Often times when proxying thick clients I have to use the WinINET proxy settings (also known as IE proxy settings). Burp's history will be full of noise.
Other Applications Stop Working
A lot of miscellaneous applications are also proxied when you enable the Windows proxy settings. They might stop working due to certificate pinning.
Secrets/Credentials in Saved in the Burp Project
Other proxied applications might contain secrets and other sensitive information. I keep my Burp projects forever so the secrets are now stored on the machine and I do not want that to happen.
Previous Work
Brian King talks about disabling some options in Firefox to reduce noise in Burp in
Towards a Quieter Firefox - Blackhills Infosec. He provides a
Firefox user.js
file to add those options. It does not block all Firefox
requests and does not work for other browsers.
I have also written about adding certain domains to Burp's TLS Pass through in another blog post named Quality of Life Tips and Tricks - Burp Suite Domains in the pass through section are not proxied and do not appear in history.
Reduce Incoming Requests
If we can prevent some requests from reaching Burp we have won.
Proxy The Application Directly
Proxy the application directly. If we proxy the application we ignore traffic for the rest of the system. Sometimes the app has proxy settings. These proxy settings are not always visible in the application. They might be in a config file. Search the internet and read the documentation/manual to see how it can be done.
Techstack Proxy Settings
The tech stack helps with proxying. Here are some quick tips:
- .NET: You can proxy these by using a config file. See Thick Client Proxying - Part 7 - Proxying .NET Applications via Config File
- Java: You can pass parameters that set a proxy. E.g.,
-Dhttps.proxyHost=127.0.0.1
and-Dhttps.proxyPort=8080
.
Browsers
Browsers are special thick clients. We can directly proxy them.
- Firefox: I usually use Firefox or its clones because it has its own proxy settings. Also FoxyProxy helps.
- Chromium based browsers such as Chrome and Edge use the Windows proxy settings
but it is possible to point to a proxy using command line parameters.
- E.g., Edge:
msedge.exe --proxy-server="http://localhost:8080"
- Create a shortcut that runs the browser with a specific proxy, this way you can run different instances of the browser for normal web browsing or proxying.
- E.g., Edge:
- Electron: Usually honor Windows proxy settings. Chromium based
but ignore
--proxy-server
.
Use a Virtual Machine
For obvious reasons, VMs are a must for testing thick clients. They are controlled environments. Your other applications do not break, the noise from their traffic does not show up in Burp, and you do not have to change the proxy settings because you need to attend a videoconference meeting.
Use Burp's Scope
It's possible add endpoints used by the application to the Burp's scope and hide everything not in scope. But this is not feasible in the early stages of recon because we do not know all the endpoints and we do not want to miss anything when the app contacts a new one. I usually do not use this setting at all.
Use Burp's TLS Pass Through
If it's not possible to reduce the incoming requests we can also tell Burp not to MitM some requests.
Harvesting Noisy Domains
First, we must make a list of any traffic that should not be proxied. Some examples are Windows specific traffic (e.g., telemetry), browsers calling home, and other application updates.
To create this list I did the following:
- Started Burp in a typical Windows 10 VM and set it in Windows proxy settings.
- Added the Burp's CA to the Windows certificate store
- Opened every application and browser that I normally use and interacted with them a bit (e.g., checked for updates).
- Left the VM idle for a few hours.
I switched to the Target > Site map
tab and looked for a way to copy every
domain. Burp does not have an option to just copy the domains but there is a
trick:
- Go to
Target > Scope
and checkUse advanced scope control
. - Go to
Target > Site map
and click onFilter
and click theShow all
button. - Use
ctrl+a
to select every target. - Right-click and select
Add to Scope
. - Switch back to the
Scope
tab. - Select one item in scope and use
ctrl+a
to select every domain scope. Then copy them withctrl+c
. - Paste the results in a text file and save it.
Now I had a list of regular expressions for the domains.
Adding Domains to Burp's TLS Pass Through
TLS Pass Through
has an option to paste URLs or load a list from a file. It
does not support the format we just created. The file should have one normal URL
per line (not regex). E.g., instead of ^accounts\.google\.com$
we
should have accounts.google.com
. *.google.com
does not work either.
A series of searches and replaces converted the previous domain list to the
correct format. I went to Proxy > Options
and scrolled down to TLS Pass Through
at the bottom of the tab. Clicked Load ...
and selected the file my domains.
Paste URL
only works if there is one URL in the clipboard or you have copied a
URL from another location in Burp.
Adding More Domains
To update the list we must add new domains manually in TLS Pass Through
.
Inside Burp:
- Right-click on the request and select
Copy URL
. - Go to
TLS Pass Through
and click thePaste URL
button.
Edit the config file directly:
- Export the config file (or edit the one from below).
- Add new items to
project_options > ssl_pass_through > rules
.- Don't forget to escape backslashes in the JSON string
- Reload the config file.
There were some domains that did not appear in the previous list. I added them manually:
.*mozilla\\.(com|net|org)
.*\.windows\.com
.*\\.live\\.com
.*\\.windowsupdate\\.com
.*\\.microsoft\\.com
.*visualstudio\\.com
Adding Authentication Domains
As I have explained in Burp Should Not Capture Corporate Credentials, you do not want to capture login credentials or other sensitive info in your Burp projects.
Discover these endpoints for your organization and then add them to the pass
through section. For example, auth.example.net
or Okta example.okta.com
.
What About HTTP?
It should have been obvious to me but I did not realize that HTTP requests still
get proxied. For example, http://ocsp.digicert.com
.
There's not much we can do here. Fortunately, there are only a few of them these days. A few suggestions:
- Redirect those domains to localhost using the
hosts
file. This will break things down the road especially if you are not in a VM. - With
FoxyProxy
you can designate filters for passing data to the proxy. - Live with the noise.
If you figure out a way to tell Burp not to capture HTTP requests (other than the nuclear option) please let me know.
Burp's HTTP History Filter
The filter for HTTP History is another good ally in our war against noise. I
usually remove CSS and some file extensions like woff/woff2 (fonts). You can add
these to the Filter by file extension > Hide
section and then turn it on/off
as needed. Bonus points for adding them to your default Burp config file. The
possibilities here are endless.
OPTIONS
Burp's history filter does not have a way to hide HTTP verbs like OPTIONS. The preflight requests will pollute your history. I described how I wrote an extension to hide them in Hiding OPTIONS - An Adventure in Dealing with Burp Proxy in an Extension. You can use this methodology to hide any HTTP verb from history.
Note: Capt. Meelo wrote an extension based on my blog post (seems like mine has stopped working), you can see their blog post at Filtering the OPTIONS Method in Burp.
The Nuclear Option
You can drop any request that is not in scope.
- Add all target domains to scope.
- Go to
Project Options > Out-of-Scope Requests (at the bottom)
and selectDrop all out-of-scope requests
.
I really do not suggest this even after you feel you have everything in scope. Thick client applications usually surprise me with new requests.
The Resulting Project Config
After doing this for a while, you will have a good list of noisy domains. Save
the project config at Project (menu) > Project options > Save project options
.
The domains will be in the config file (it's a JSON file) under
ssl_pass_through
.
Add them to your default config file with the rest of the things you like:
- Disable intercept at startup.
User options > Misc > Proxy Interception
- Disable pretty-print by default (added in v2020.4).
User options > HTTP Message Display > Pretty print by default
- Remove the default Burp Collaborator1.
Project options > Misc > Burp Collaborator Server > Don't use Burp Collaborator
- Remove
If-Modified-Since
andIf-None-Match
headers to always avoid 304s.Proxy > Options > Match and Replace > Enable built-in rules
.- Why? Disable Cached Responses
Some of these are User options
and do not appear in the JSON file that we
created above. User options can be exported at Burp (menu) > User options > Save user options
. This creates a second JSON file. Everything will be under a
key named user_options
.
{
"user_options":{
// removed
}
}
The project options file has a similar structure. Everything is under project_options
.
{
"project_options":{
// removed
}
}
Combining User and Project Options
You can combine these two files. Add user_options
and everything under it to
the other file. Note that project_options
and user_options
should be the two
top-level keys along with proxy
and a few move. The final config file will
look like:
{
"project_options":{
// removed
},
"user_options":{
// removed
},
"proxy": {
// removed
},
// removed
}
Now we can use this project file every time we create a new project. Save it somewhere (e.g., git repo) and update it regularly. It will save you a lot of time
Update June 2020: I reviewed the rules and decided that using wildcards to
ban entire domains is better. For example, to ban pretty much every request to
mozilla
we can use this regex ^.*mozilla\\.(com|net|org)$
. I updated the
burp config. It should be cleaner now.
Update June 2021: I have reduced the file size significantly. I compared the final config file with the default values and removed the JSON keys that were not modified. This does not have any performance effects.
See mine at:
- https://github.com/parsiya/Parsia-Clone/blob/master/configs/burp-default-config.json
- Raw download: https://raw.githubusercontent.com/parsiya/Parsia-Clone/master/configs/burp-default-config.json
What Did We Learn Here Today?
- How to reduce noise in Burp history using several techniques.
- How to combine project and user config files.
Thanks for reading and hit me up if you have any suggestions.
We honestly do not know what Portswigger collects from the collaborator. This is a must if you are doing consulting for 3rd parties. You don't want to leak your clients' internal info. You do not want your fancy payload to be harvested either. ↩︎