Scrapy: Arbitrary Module Import via Referrer-Policy Header in RefererMiddleware
Impact
Since version 1.4.0, Scrapy respects the Referrer-Policy response header to decide whether and how to set a Referer header on follow-up requests.
If the header value looked like a valid Python import path, Scrapy would import the referenced object and call it, assuming it referred to a referrer policy class (for example, scrapy.spidermiddlewares.referer.DefaultReferrerPolicy) and attempting to instantiate it to handle the Referer header.
A malicious site could exploit this by setting Referrer-Policy to a path such as sys.exit, causing Scrapy to import and execute it and potentially terminate the process.
Patches
Upgrade to Scrapy 2.14.2 (or later).
Workarounds
If you cannot upgrade to Scrapy 2.14.2, consider the following mitigations.
- Disable the middleware: If you don't need the
Refererheader on follow-up requests, setREFERER_ENABLEDtoFalse. - Set headers manually: If you do need a
Referer, disable the middleware and set the header explicitly on the requests that require it. - Set
referrer_policyin request metadata: If disabling the middleware is not viable, set thereferrer_policyrequest meta key on all requests to prevent evaluating preceding responses'Referrer-Policy. For example:
Request(
url,
meta={
"referrer_policy": "scrapy.spidermiddlewares.referer.DefaultReferrerPolicy",
},
)
Instead of editing requests individually, you can:
- implement a custom spider middleware that runs before the built-in referrer policy middleware and sets the
referrer_policymeta key; or - set the meta key in start requests and use the scrapy-sticky-meta-params plugin to propagate it to follow-up requests.
If you want to continue respecting legitimate Referrer-Policy headers while protecting against malicious ones, disable the built-in referrer policy middleware by setting it to None in SPIDER_MIDDLEWARES and replace it with the fixed implementation from Scrapy 2.14.2.
If the Scrapy 2.14.2 implementation is incompatible with your project (for example, because your Scrapy version is older), copy the corresponding middleware from your Scrapy version, apply the same patch, and use that as a replacement.
Deep Security Visibility Without the Complexity
SynScan provides clear, real-time security insights so you can monitor your attack surface, spot risks early, and act fast—without extra complexity.
- No setup fees
- 5-min deployment
- Cancel anytime
Frequently Asked Questions
A security vulnerability is a weakness in software, hardware, or configuration that can be exploited to compromise confidentiality, integrity, or availability. Many vulnerabilities are tracked as CVEs (Common Vulnerabilities and Exposures), which provide a standardized identifier so teams can coordinate patching, mitigation, and risk assessment across tools and vendors.
CVSS (Common Vulnerability Scoring System) estimates technical severity, but it doesn't automatically equal business risk. Prioritize using context like internet exposure, affected asset criticality, known exploitation (proof-of-concept or in-the-wild), and whether compensating controls exist. A "Medium" CVSS on an exposed, production system can be more urgent than a "Critical" on an isolated, non-production host.
A vulnerability is the underlying weakness. An exploit is the method or code used to take advantage of it. A zero-day is a vulnerability that is unknown to the vendor or has no publicly available fix when attackers begin using it. In practice, risk increases sharply when exploitation becomes reliable or widespread.
Recurring findings usually come from incomplete Asset Discovery, inconsistent patch management, inherited images, and configuration drift. In modern environments, you also need to watch the software supply chain: dependencies, containers, build pipelines, and third-party services can reintroduce the same weakness even after you patch a single host. Unknown or unmanaged assets (often called Shadow IT) are a common reason the same issues resurface.
Use a simple, repeatable triage model: focus first on externally exposed assets, high-value systems (identity, VPN, email, production), vulnerabilities with known exploits, and issues that enable remote code execution or privilege escalation. Then enforce patch SLAs and track progress using consistent metrics so remediation is steady, not reactive.
SynScan combines attack surface monitoring and continuous security auditing to keep your inventory current, flag high-impact vulnerabilities early, and help you turn raw findings into a practical remediation plan.