URL encoding

When making an API request to Piloterr, you can include various parameters as query strings.

For example:

https://piloterr.com/api/v1/?url=https://example.com/&x_api_key=YOUR-API-KEY

Handling URLs with Query Parameters

If the URL you want to scrape already includes its own query parameters, such as param1 and param2, you might run into issues. For instance:

curl 'https://piloterr.com/api/v1/?url=https://example.com?param1=value1¶m2=value2&x_api_key=YOUR-API-KEY'

In this case, the API will return an error because it attempts to process param1 and param2 as Piloterr parameters, which are unrecognized by the system. You'll receive an error.

Solution: URL Encoding

To avoid this issue, you need to encode the URL along with its query parameters. Most HTTP clients in various programming languages handle this automatically. However, if your client does not (like cURL), you can manually encode the URL using a third-party tool or within your programming language.

Examples of URL Encoding in Different Languages

Curl

sudo apt-get install gridsite-clients
urlencode "YOUR URL"

Python

import urllib.parse
encoded_url = urllib.parse.quote("YOUR URL")

NodeJS

encoded_url = encodeURIComponent("YOUR URL")

Java

String encoded_url = URLEncoder.encode("YOUR URL", "UTF-8");

Ruby

require 'uri'
encoded_url = URI::encode("YOUR URL")

PHP

$url_encoded = urlencode("YOUR URL");

Go

package main

import (
    "net/url"
)

func main() {
    encoded_url := url.QueryEscape("YOUR URL")
}

Differences

Here’s the difference between the original URL and its encoded version:

Original URL:

https://es.indeed.com/jobs?q=sin+experiencia&l=Madrid%2C+Madrid+provincia&from=searchOnHP&vjk=27283b309df53de9

Encoded URL:

https%3A%2F%2Fes.indeed.com%2Fjobs%3Fq%3Dsin%2Bexperiencia%2526l%3DMadrid%252C%2BMadrid%2Bprovincia%2526from%3DsearchOnHP%2526vjk%3D27283b309df53de9