Skip to main content

What is a web page resources tracker?

The web page resources tracker is a utility that gives developers the ability to detect and track resources of any web page. It falls under the category of synthetic monitoring tools and helps ensure that the deployed application loads only the intended web resources (JavaScript and CSS) during its lifetime. If any unintended changes occur, which could result from a broken deployment or malicious activity, the tracker will promptly notify developers or IT personnel about the detected anomalies.

Additionally, security researchers focused on discovering potential security vulnerabilities in third-party web applications can use web page resources trackers. By being notified when the application's resources change, researchers can identify if the application has been upgraded, providing an opportunity to re-examine the application and potentially discover new vulnerabilities.

NOTE

Currently, Secutils.dev doesn't support tracking resources for web pages protected by application firewalls (WAF) or any form of CAPTCHA. If you require tracking resources for such pages, please comment on #secutils/34 to discuss your use case.

On this page, you can find guides on creating and using web page resources trackers.

Create a web page resources tracker​

In this guide, you'll create a simple resources tracker for the Hacker News:

  1. Navigate to Web Scraping → Resources trackers and click Track resources button
  2. Configure a new tracker with the following values:
Name
Hacker News
URL
https://news.ycombinator.com
  1. Click the Save button to save the tracker
  2. Once the tracker is set up, it will appear in the trackers grid
  3. Expand the tracker's row and click the Update button to make the first snapshot of the web page resources

It's hard to believe, but as of the time of writing, Hacker News continues to rely on just a single script and stylesheet!

Watch the video demo below to see all the steps mentioned earlier in action:

Detect changes with a web page resources tracker​

In this guide, you will create a web page resources tracker and test it using a custom HTML responder:

  1. First, navigate to Webhooks → Responders and click Create responder button
  2. Configure a few responders with the following values to emulate JavaScript files that we will track changes for across revisions:

This JavaScript will remain unchanged across revisions:

Name
no-changes.js
Path
/no-changes.js
Headers
Content-Type: application/javascript; charset=utf-8
Body
document.body.insertAdjacentHTML(
  'beforeend',
  'Source: no-changes.js<br>'
);

This JavaScript will change across revisions:

Name
changed.js
Path
/changed.js
Headers
Content-Type: application/javascript; charset=utf-8
Body
document.body.insertAdjacentHTML(
  'beforeend',
  'Source: changed.js, Changed: no<br>'
);

This JavaScript will be removed across revisions:

Name
removed.js
Path
/removed.js
Headers
Content-Type: application/javascript; charset=utf-8
Body
document.body.insertAdjacentHTML(
  'beforeend',
  'Source: removed.js<br>'
);

This JavaScript will be added in a new revision:

Name
added.js
Path
/added.js
Headers
Content-Type: application/javascript; charset=utf-8
Body
document.body.insertAdjacentHTML(
  'beforeend',
  'Source: added.js<br>'
);
  1. Now, configure a new responder with the following values to respond with a simple HTML page that references previously created JavaScript responders (except for added.js):
Name
track-me.html
Path
/track-me.html
Headers
Content-Type: text/html; charset=utf-8
Body
<!DOCTYPE html>
<html lang="en">
<head>
  <title>Evaluate resources tracker</title>
  <script type="text/javascript" src="./no-changes.js" defer></script>
  <script type="text/javascript" src="./changed.js" defer></script>
  <script type="text/javascript" src="./removed.js" defer></script>
</head>
<body></body>
</html>
  1. Click the Save button to save the responder
  2. Once the responder is set up, it will appear in the responders grid along with its unique URL
  3. Click on the responder's URL and make sure that it renders the following content:
Source: no-changes.js
Source: changed.js, Changed: no
Source: removed.js
  1. Now, navigate to Web Scraping → Resources trackers and click Track resources button
  2. Configure a new tracker for track-me.html responder with the following values:
Name
Demo
URL
https://[YOUR UNIQUE ID].webhooks.secutils.dev/track-me.html
Frequency
Daily
Notifications
☑
TIP

Configured tracker will fetch the resources of the track-me.html responder once a day and notify you if any changes are detected. You can change the frequency and notification settings to suit your needs.

  1. Click the Save button to save the tracker
  2. Once the tracker is set up, it will appear in the trackers grid
  3. Expand the tracker's row and click the Update button to make the first snapshot of the web page resources
  4. Once the tracker has fetched the resources, they will appear in the resources grid:
SourceDiffTypeSize
https://[YOUR UNIQUE ID].webhooks.secutils.dev/no-change.js-Script81
https://[YOUR UNIQUE ID].webhooks.secutils.dev/changed.js-Script91
https://[YOUR UNIQUE ID].webhooks.secutils.dev/removed.js-Script78
  1. Now, navigate to Webhooks → Responders and edit track-me.html responder to reference added.js responder, and remove reference to removed.js:
<!DOCTYPE html>
<html lang="en">
<head>
  <title>Evaluate resources tracker</title>
  <script type="text/javascript" src="./no-changes.js" defer></script>
  <script type="text/javascript" src="./changed.js" defer></script>
- <script type="text/javascript" src="./removed.js" defer></script>
+ <script type="text/javascript" src="./added.js" defer></script>
</head>
<body></body>
</html>
  1. Next, change the body of the changed.js responder to something like this:
document.body.insertAdjacentHTML(
  'beforeend',
- 'Source: changed.js, Changed: no<br>'
+ 'Source: changed.js, Changed: yes<br>'
);
  1. Finally, navigate to Web Scraping → Resources trackers and expand the Demo tracker's row
  2. Click Update button to fetch the next revision of the web page resources
NOTE

Normally, Secutils.dev caches web page resources for 10 minutes. This means that if you make changes to the web page resources and want to see them reflected in the tracker, you'll need to wait for 10 minutes before re-fetching resources. However, for this guide, I've disabled caching for the tracker so that you can see changes immediately.

  1. Once the tracker has fetched updated resources, they will appear in the resources grid together with the diff status:
SourceDiffTypeSize
https://[YOUR UNIQUE ID].webhooks.secutils.dev/no-change.js-Script81
https://[YOUR UNIQUE ID].webhooks.secutils.dev/changed.jsChangedScript91
https://[YOUR UNIQUE ID].webhooks.secutils.dev/added.jsAddedScript76
https://[YOUR UNIQUE ID].webhooks.secutils.dev/removed.jsRemovedScript78

Watch the video demo below to see all the steps mentioned earlier in action:

Filter resources with a web page resources tracker​

In this guide, you will create a web page resource tracker for the Reddit home page and learn how to track only specific resources:

  1. Navigate to Web Scraping → Resources trackers and click Track resources button
  2. Configure a new tracker with the following values:
Name
GitHub
URL
https://github.com/?rev=1
TIP

Normally, Secutils.dev caches web page resources for 10 minutes. This means that if you make changes to the web page resource tracker and want to see them take effect, you'll need to wait for 10 minutes before re-fetching resources. However, for this guide, I'm adding an arbitrary ?rev=X query string parameter to the URL to bypass caching and see the changes immediately. This trick can be quite handy when you are setting up a new tracker and need to fine-tune its configuration.

Note that every time you change the tracker's URL, all previously fetched resources will be removed.

  1. Click the Save button to save the tracker
  2. Once the tracker is set up, it will appear in the trackers grid
  3. Expand the tracker's row and click the Update button to make the first snapshot of the web page resources
  4. Once the tracker has fetched the resources, they will appear in the resources grid. You'll notice that there are nearly 80 resources used for the GitHub home page! In the case of large and complex pages like this one, it's recommended to have multiple separate trackers, e.g. one per logical functionality domain, to avoid overwhelming the developer with too many resources and consequently changes they might need to track. Let's say we're only interested in "vendored" resources.
  5. To filter out all resources that are not "vendored", we'll use the Resource filter/mapper feature. Click the pencil icon next to the tracker's name to edit the tracker and update the following properties:
URL
https://github.com/?rev=2
Resource filter/mapper
return resource.url?.includes('vendors')
  ? resource
  : null;
  1. The Resource filter/mapper property accepts a JavaScript function that is executed for each resource detected by the tracker. The function receives a single resource argument, which is the resource object. The function must return either the resource object or null. If the function returns null, the resource will be filtered out and will not be tracked. In our case, we're filtering out all resources that do not contain sso in their URL. You can learn more about resource filter/mapper scripts in the Annex: Resource filter/mapper script examples section.
  2. Now, click the Save button to save the tracker.
  3. Click the Update button to re-fetch web page resources. Once the tracker has re-fetched resources, only about half of the previously extracted resources will appear in the resources grid.

Watch the video demo below to see all the steps mentioned earlier in action:

Annex: Resource filter/mapper script examples​

In this section, you can find examples of resource filter and mapper scripts that you can use to filter out or map resources based on various criteria. The script essentially defines a function that is executed for each resource detected by the tracker and receives a single resource argument, which is the resource object. The function must return either the resource object or null. If the function returns null, the resource will be filtered out and will not be tracked.

The resource argument has the following interface:

interface Resource {
  // Resource full URL. This property is not defined for inline resources. 
  url?: string;
  // Resource content.
  data: string;
  // Resource type.
  type: 'script' | 'stylesheet';
}

Track only external resources​

return resource.url?.startsWith('http')
  ? resource
  : null;

Track only inline resources​

return !resource.url
  ? resource
  : null;

Track only JavaScript resources​

return resource.type === 'script'
  ? resource
  : null;

Track only CSS resources​

return resource.type === 'stylesheet'
  ? resource
  : null;

Strip query string parameters from resource URLs​

Sometimes, resources such as analytics and user tracking scripts load with unique query string parameters, even when the content of the resource remains constant. This can lead to confusion for the tracker and trigger unwanted change notifications. To address this, you can strip query string parameters from the URLs of such resources before calculating the resource fingerprint:

const isInlineResource = !resource.url;
if (isInlineResource) {
    return resource
}

const isGoogleAnalyticsResource = resource.url.includes('googletagmanager');
if (!isGoogleAnalyticsResource) {
    return resource
}

// Strip query string parameters from Google Analytics resource URLs.
const [urlWithoutQueryString] = resource.url.split('?');
return { ...resource, url: urlWithoutQueryString };