projectdiscovery/httpx
1
1
Fork 0
mirror of https://github.com/projectdiscovery/httpx.git synced 2024-09-11 09:35:28 +03:00
Code Issues Packages Projects Releases Wiki Activity
2,062 Commits 7 Branches 55 Tags 40 MiB
f3af434678
Commit Graph

248 Commits

Author SHA1 Message Date
Ramana Reddy
9330887a58
Extract body_domains and body_fqdns to jsonl (#1750) 
* Extract body-domains and body-fqdns

* remvove port in domains

* Add test for domains extraction

* misc update

* improve domain regex

* fix test

* extract domain inside quotes

* sanitize urls

* fix test

* minor

* do not embed

* remove js variables fp + improve regex

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
 2024-06-23 01:01:43 +05:30
Ice3man
80d5532b15 misc fixing tests 2024-06-17 19:58:12 +05:30
Ice3man
ca3e696834 feat: use networkpolicy + wappalyzer singletons from library 2024-06-17 18:11:01 +05:30
Ramana Reddy
8d31a457b8 fix edge cases 2024-05-30 16:52:12 +05:30
Ramana Reddy
de9375e643 remove wildcard in domain 2024-05-30 15:03:15 +05:30
Ramana Reddy
c5ca655207 exclude root domains from fqdn 2024-05-30 14:12:01 +05:30
Ramana Reddy
ff262a084f Add extract-fqdn option 2024-05-30 09:16:23 +05:30
Ramana Reddy
91afa0cf76 Add fqdn to csp output 2024-05-30 08:34:58 +05:30
Mzack9999
7bb09b1fb1
File writing ops respect filters/matchers (#1720) 2024-05-29 20:45:19 +05:30
mzack
8b93f27dfe fixing example 2024-04-15 22:52:21 +02:00
mzack
ce8c01891c adding http11 mode 2024-04-15 21:45:25 +02:00
Aviv Keller
c72d98628e
Introducing CanHaveTitleTag Function for MIME Type Validation (#1608) 
* chore(deps): bump golang.org/x/crypto from 0.14.0 to 0.17.0 (#1493)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/cloudflare/circl from 1.3.3 to 1.3.7 (#1526)

Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.3.3 to 1.3.7.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.3.3...v1.3.7)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/quic-go/quic-go from 0.37.4 to 0.37.7 (#1531)

Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.4 to 0.37.7.
- [Release notes](https://github.com/quic-go/quic-go/releases)
- [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md)
- [Commits](https://github.com/quic-go/quic-go/compare/v0.37.4...v0.37.7)

---
updated-dependencies:
- dependency-name: github.com/quic-go/quic-go
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update Title Extraction

* Make Requsted Changes

* Import slices

* optional asn

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
 2024-03-23 21:47:40 +05:30
Dogan Can Bakir
a4b026111a
Fix content rel issues (#1630) 
* do not omit content length

* fix words and lines len
 2024-03-20 22:51:10 +05:30
Ramana Reddy
dd8b546b35
Make cdn and tech-detect as default in json output (#1614) 
* Make cdn and tech-detect as default in json output

* Reflect changes only in jsonl output
 2024-03-06 17:16:58 +05:30
Dogan Can Bakir
2e35ad15af
use networkpolicy (#1597) 
* use networkpolicy

* fix test
 2024-02-27 18:22:55 +05:30
Dogan Can Bakir
b9a444795d
Use useragent (#1562) 
* use projectdiscovery/useragent

* fix go.mod

* update useragent

* commit to last commit
 2024-01-31 20:53:17 +05:30
Mzack9999
7594c8bf5d
Merge branch 'dev' into orig-1469 2024-01-15 22:03:56 +01:00
mzack
0061bc3a20 small description change 2024-01-15 21:57:41 +01:00
Ramana Reddy
db1ea9fad3
Fix issue with host redirect on ports flag (#1529) 2024-01-15 15:27:24 +05:30
Ramana Reddy
64b6a0789c
Fix: Pipeline and Websocket not working for some targets (#1528) 
* Fix pipeline issue for port included targets

* Add more checks to websocket
 2024-01-15 15:06:28 +05:30
mzack
e557bf8662 using fastdialer deny 2024-01-07 21:10:56 +01:00
Karel
f3a6eadfce Do not use local resolvers when given resolver input 2023-11-27 16:37:06 +01:00
Doğan Can Bakır
d55d459f1d extend char set 2023-11-08 07:54:22 +00:00
Doğan Can Bakır
37591e1dd5 fix -title newline issue 2023-11-07 19:50:28 +00:00
Cody Cline
7dee9988ca
Feature: optional flag to skip a host if it has a private IP address. (#1408) 
* feat: Implement flag which when enabled, will skip any host with a private ip address instead of timing out.

* fix: Amend private ip check to include loopback and link-local addresses

* fix: Amend debug message to be more concise

* fix: Account for port numbers in skipPrivateHosts feature

* docs: Remove uncessary  debug statement

* readme update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-11-02 14:21:22 +05:30
Stefan Kraxberger
9685ffccba Updated storing the chain so that also the reponse body is stored 2023-09-20 11:51:57 +02:00
Mzack9999
4432f63634 fixing logic 2023-09-11 18:00:14 +02:00
Mzack9999
0f6edb50be refactor hsts implementation 2023-09-11 17:53:26 +02:00
Mzack9999
8b5dc7bc0f merge conflict 2023-09-11 16:58:31 +02:00
Ramana Reddy
76adadfeb4
Add flag to preview first N characters (#1316) 
* Add flag to preview first N characters

* use dynamic var for body-preview flag

* strip html tags from response

* add stripe option to remove html or xml tags

* typo

* resolve merge conflicts

* add test and update readme

* update readme

* typo

* typo no 2

* small refactor

* include body preview in jsonl output

* minor changes

---------

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2023-09-07 15:26:54 +05:30
JoMar
179678581d
Exclude WAF (#1341) 
* Exclude WAF

* Add missing 's'

* Revert "Add missing 's'"

This reverts commit 467ceca320.
 2023-09-05 19:53:06 +05:30
Stefan Kraxberger
964cc3bbd0 Implemented a redirect strategy which respects HSTS headers and modifies the URL accordingly if necessary 2023-08-31 10:58:51 +02:00
Stefan Kraxberger
0b702c5713 Revert "Added redirect policy which takes HSTS into account, so that even if the location would go from HTTPs to HTTP it would follow to HTTPs if an HTSTS header is present" 
This reverts commit c4810ffbb7.
 2023-08-31 10:44:09 +02:00
Stefan Kraxberger
c4810ffbb7 Added redirect policy which takes HSTS into account, so that even if the location would go from HTTPs to HTTP it would follow to HTTPs if an HTSTS header is present 2023-08-31 10:41:20 +02:00
Stefan Kraxberger
13f78bc302 Modified storing of response chains, because it was not correct 2023-08-30 16:06:45 +02:00
Doğan Can Bakır
2718a9aab7 use mlutils 2023-07-17 06:54:00 +00:00
mzack
e991ff1908 commenting out train logic 2023-06-28 11:08:09 +02:00
mzack
4ca3089d5b Merge branch 'dev' into pr/1245 2023-06-28 10:38:14 +02:00
Ramana Reddy
5f8e5d0dc5
fix tls-grab option with ztls mode (#1259) 
* fix tls-grab option with ztls mode

* using canonical concatenation

* mod tidy

---------

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
 2023-06-27 20:37:47 +05:30
mzack
ae1d9511df memory optimization 2023-06-22 07:27:16 +02:00
mzack
4c8ac6703e Merge branch 'dev' into pr/1245 2023-06-22 06:11:46 +02:00
Doğan Can Bakır
900f5623e8 add error page classifier 2023-06-20 14:13:48 +03:00
Mzack9999
a14af24496
Skip body read with status code 304 (#1231) 
* skip body read with status code 304

* dep update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-06-20 14:01:51 +05:30
Mzack9999
8ea85e901d
Adding prototype of tls impersonate (#1199) 
* Adding prototype of tls impersonate

* updating docs

* misc update

* dep update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-06-17 16:34:19 +05:30
dependabot[bot]
7d41db16f9
chore(deps): bump github.com/projectdiscovery/cdncheck from 0.0.4-0.20220413175814-b47bc2d578b1 to 1.0.0 (#1118) 
* chore(deps): bump github.com/projectdiscovery/cdncheck

Bumps [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) from 0.0.4-0.20220413175814-b47bc2d578b1 to 1.0.0.
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/commits/v1.0.0)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* cdncheck update fixes

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ramana Reddy <ramanaredy.manda@gmail.com>
 2023-04-27 12:46:28 +05:30
Mzack9999
00d0977963
Adding Screenshot support (#1097) 
* Adding Screenshot support

* adding headless body

* moving example into compilable file

* updating docs

* enabling store response with screenshot

* fixing output dir path

* fixing mod

* using native leakless

* splitting response+screenshot folders

* better handling json output

* readme update

* rel => abs path

* Added chromium into docker

* “Wipe them out. All of them.” – Darth Sidious

* utils bump

* go mod tidy

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-04-27 01:07:06 +05:30
Mzack9999
1defd0a9b7
Add support to skip decode (#1055) 
add support to skip decode
 2023-03-29 19:06:47 +05:30
Tarun Koyalwar
c7456929b4 fix integration tests and upgrade deps 2023-02-08 19:50:03 +05:30
Tarun Koyalwar
3a93d2d1e8 bump url utils to v0.0.7 2023-02-07 20:15:23 +05:30
Mzack9999
db1bc2ebba
Reworking favicon hash calculation (#906) 
* Reworking favicon hash calculation

* removing debug code

* bumping go to 1.19

* removing automatic follow redirect

* more edge cases

* added favicon path info
 2023-01-06 18:28:08 +05:30