Pablo Guides

# Section 1: Introduction to hakrawler$

## Overview of hakrawler$

In the realm of web application security, reconnaissance is a critical phase of penetration testing, allowing security professionals to gather information about their target environment. One powerful tool designed for this purpose is **hakrawler$**, a web crawling tool that aids in the discovery of links and resources within web applications. This section will cover the installation, configuration, and usage of hakrawler$ on Kali Linux, along with insightful use cases and technical explanations to equip you with the knowledge needed to leverage this tool effectively.

## 1.1 Installation and Configuration on Kali Linux

### Step 1: Update Kali Linux

Before installing any new tools, it's essential to ensure that your Kali Linux instance is up to date. Open your terminal and run the following commands:

"`bash
sudo apt update && sudo apt upgrade -y
"`

### Step 2: Installing hakrawler$

Hakrawler$ is primarily written in Go, so you need to have Go installed on your Kali Linux system. If it is not already installed, follow the steps below to install Go:

#### Installing Go

1. Download the latest version of Go:

   wget https://golang.org/dl/go1.20.linux-amd64.tar.gz

 

2. Extract the downloaded archive to `/usr/local`:

   sudo tar -C /usr/local -xzf go1.20.linux-amd64.tar.gz

 

3. Add Go to your PATH by adding the following lines to your `.bashrc` or `.zshrc`:

   export PATH=$PATH:/usr/local/go/bin

 

4. Reload your shell configuration:

   source ~/.bashrc

 

5. Verify the installation:

   go version

 

#### Installing hakrawler$

Now that Go is installed, you can install hakrawler$ using the following command:

"`bash
go install github.com/hakluke/hakrawler@latest
"`

Ensure that Go's bin directory is in your PATH. The hakrawler$ binary should now be located in `$HOME/go/bin/hakrawler`.

### Step 3: Configuration

There are minimal configurations required for hakrawler$ to function effectively. However, you can create a configuration file if you want to customize your crawling options. The default behavior of hakrawler$ usually suffices for standard reconnaissance tasks.

## 1.2 Step-by-Step Usage

### Basic Command Structure

The basic syntax for running hakrawler$ is:

"`bash
hakrawler -url [options]
"`

#### Example: Basic Crawling

To start with a simple crawl of a target website, use:

"`bash
hakrawler -url https://example.com
"`

This command will initiate crawling of `https://example.com`, and by default, it will return all discovered links, including internal and external links.

### Common Options

– `-depth `: Set the maximum depth to crawl. For example, to limit the crawl depth to 2:

– `-timeout `: Set the timeout duration for requests (default is 5 seconds). For setting a 10 seconds timeout:

– `-follow`: Follow redirects (HTTP 3xx responses).

– `-output `: Save the output to a specified file instead of standard output:

### Real-World Use Cases

#### Use Case 1: Discovering API Endpoints

Web applications often have APIs that are not well-documented. Using hakrawler$ can help discover hidden endpoints. For instance, to explore a target's API:

"`bash
hakrawler -url https://api.example.com -depth 3 -output api_endpoints.txt
"`

This command crawls the API up to a depth of 3 and saves the findings to `api_endpoints.txt`, making it easier for further analysis.

#### Use Case 2: Identifying Subdomains

Identifying subdomains is crucial in web application reconnaissance. You can combine hakrawler$ with other tools like `subfinder` or `amass` to list potential subdomains, then run:

"`bash
hakrawler -url https://subdomain.example.com -depth 2
"`

This will help in identifying pages within discovered subdomains.

#### Use Case 3: Mapping Out Web Application Structures

Utilizing hakrawler$ can assist in mapping out the structure of a web application, which is vital for attacking vectors during a penetration test. Execute:

"`bash
hakrawler -url https://example.com -depth 5 -output site_structure.txt
"`

Here, the tool will crawl and document the site's structure deeply.

### Technical Explanations

#### How Hakrawler$ Works

Hakrawler$ operates by sending HTTP requests to the specified URL and parsing the HTML content to extract links. It supports various protocols such as HTTP, HTTPS, and follows links based on the specified depth and options.

1. **HTTP Request**: It initiates an HTTP GET request to the target URL.
2. **HTML Parsing**: Upon receiving a response, it parses the HTML body to retrieve all anchor (``) tags and link references.
3. **Link Normalization**: Links are normalized to handle relative URLs correctly.
4. **Storage**: Results are stored in memory, and based on options, they can be outputted to a file or printed to the console.