CVE-2021-41125
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
| Software | From | Fixed in |
|---|---|---|
| scrapy / scrapy | 2.0.0 | 2.5.1 |
| scrapy / scrapy | - | 1.8.1 |
| debian / debian_linux | 9.0 | 9.0.x |
scrapy
|
- | 1.8.1 |
|
scrapy
|
2.0.0 | 2.5.1 |
- http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpauth
- https://github.com/scrapy/scrapy/commit/b01d69a1bf48060daec8f751368622352d8b85a6
- https://github.com/scrapy/scrapy/security/advisories/GHSA-jwqp-28gf-p498
- https://lists.debian.org/debian-lts-announce/2022/03/msg00021.html
- https://w3lib.readthedocs.io/en/latest/w3lib.html#w3lib.http.basic_auth_header
Deep Security Visibility Without the Complexity
SynScan provides clear, real-time security insights so you can monitor your attack surface, spot risks early, and act fast—without extra complexity.
- No setup fees
- 5-min deployment
- Cancel anytime
Frequently Asked Questions
A security vulnerability is a weakness in software, hardware, or configuration that can be exploited to compromise confidentiality, integrity, or availability. Many vulnerabilities are tracked as CVEs (Common Vulnerabilities and Exposures), which provide a standardized identifier so teams can coordinate patching, mitigation, and risk assessment across tools and vendors.
CVSS (Common Vulnerability Scoring System) estimates technical severity, but it doesn't automatically equal business risk. Prioritize using context like internet exposure, affected asset criticality, known exploitation (proof-of-concept or in-the-wild), and whether compensating controls exist. A "Medium" CVSS on an exposed, production system can be more urgent than a "Critical" on an isolated, non-production host.
A vulnerability is the underlying weakness. An exploit is the method or code used to take advantage of it. A zero-day is a vulnerability that is unknown to the vendor or has no publicly available fix when attackers begin using it. In practice, risk increases sharply when exploitation becomes reliable or widespread.
Recurring findings usually come from incomplete Asset Discovery, inconsistent patch management, inherited images, and configuration drift. In modern environments, you also need to watch the software supply chain: dependencies, containers, build pipelines, and third-party services can reintroduce the same weakness even after you patch a single host. Unknown or unmanaged assets (often called Shadow IT) are a common reason the same issues resurface.
Use a simple, repeatable triage model: focus first on externally exposed assets, high-value systems (identity, VPN, email, production), vulnerabilities with known exploits, and issues that enable remote code execution or privilege escalation. Then enforce patch SLAs and track progress using consistent metrics so remediation is steady, not reactive.
SynScan combines attack surface monitoring and continuous security auditing to keep your inventory current, flag high-impact vulnerabilities early, and help you turn raw findings into a practical remediation plan.