The curl command is a ubiquitous and indispensable utility for transferring data with URLs. It supports a myriad of protocols, including HTTP, HTTPS, FTP, FTPS, GOPHER, DICT, FILE, and LDAP. Mastery of curl is paramount for system administrators, developers, and anyone regularly interacting with network services or web APIs. This guide meticulously details its fundamental and practical applications, enabling you to effectively retrieve data, send requests, and debug network interactions directly from the command line.
Prerequisites
To follow this guide, you will need:
- A Linux-based operating system (e.g., Ubuntu, CentOS, Fedora).
- Access to a terminal or command-line interface.
- Basic familiarity with command-line operations.
curl is typically pre-installed on most Linux distributions. If not, install it using your system’s package manager (e.g., sudo apt install curl on Debian/Ubuntu, sudo yum install curl on CentOS/RHEL).
Understanding Basic `curl` Syntax
The fundamental syntax for curl is straightforward: curl [options] [URL]. By default, curl performs a GET request and outputs the retrieved content to standard output (your terminal).
Retrieve Content from a URL
To fetch the content of a webpage, simply specify the URL:
curl https://example.comThis command will display the HTML source of https://example.com directly in your terminal.
Practical Tip: When dealing with URLs containing special characters (e.g., ampersands, question marks), always enclose them in single or double quotes to prevent your shell from interpreting them prematurely. For instance: curl 'https://API.example.com/search?query=test&page=1'.
Downloading Files with `curl`
While curl‘s default behavior is to print content, it excels at downloading files, offering precise control over the output.
Saving Output to a Specific File
To save the content to a file with a name of your choosing, use the -o (lowercase ‘o’) option:
curl -o example_page.html https://example.comThis command downloads the HTML content of example.com and saves it as example_page.html in your current directory.
Saving Output with Original Filename
For convenience, use the -O (uppercase ‘O’) option to save the file with its remote filename:
curl -O https://example.com/images/logo.pngThis will download the image and save it as logo.png.
Pro-Tip: For downloads that might involve redirects, such as shortened URLs or moved resources, add the -L or --location option. This instructs curl to follow any HTTP 3xx redirects until it reaches the final destination. Example: curl -LO https://bit.ly/some-resource.
Making POST Requests
curl is invaluable for sending data to a server, often used for interacting with APIs or submitting form data.
Sending Form Data
Use the -d or --data option to send data in a POST request. By default, curl will set the Content-Type header to application/x-www-form-urlencoded.
curl -X POST -d "username=user&password=pass" https://api.example.com/loginThe -X POST explicitly sets the request method to POST, though -d implies it.
Warning: Directly embedding sensitive credentials in a command can expose them in your shell’s history file. For production environments, consider more secure methods like environment variables or dedicated configuration files.
Sending JSON Data
When interacting with modern APIs, you often need to send JSON payloads. This requires setting the Content-Type header explicitly using the -H or --header option:
curl -X POST -H "Content-Type: application/json" -d '{"name":"product","price":29.99}' https://api.example.com/productsPro-Tip: For larger JSON payloads or complex data, you can read the data from a file using @filename:
curl -X POST -H "Content-Type: application/json" -d @data.json https://api.example.com/productsWhere data.json contains your JSON payload.
Sending Custom Headers
Manipulating HTTP headers is crucial for various tasks, from authentication to custom user-agent strings. The -H option allows you to add or override headers.
Setting a Custom User-Agent
curl -H "User-Agent: MyCustomApp/1.0 (Linux)" https://example.com/api/statusThis can be useful for identifying your client to a server or bypassing simple bot detection.
Including an Authorization Header
For API authentication using tokens (e.g., Bearer tokens):
curl -H "Authorization: Bearer YOUR_ACCESS_TOKEN" https://api.example.com/protected-resourceReplace YOUR_ACCESS_TOKEN with your actual token.
Handling Authentication
curl provides explicit options for common authentication schemes.
Basic Authentication
For HTTP Basic Authentication, use the -u or --user option followed by username:password:
curl -u "admin:securepassword" https://secure.example.com/dashboardcurl will handle the Base64 encoding for you. Remember that Basic Auth sends credentials in a non-encrypted format, so always use HTTPS for secure transmission.
Working with Proxies
To route your curl requests through a proxy server, use the -x or --proxy option.
curl -x http://proxy.example.com:8080 https://example.comThis command directs your request for https://example.com through the specified HTTP proxy.
Pro-Tip: curl supports various proxy types. For SOCKS5 proxies, use --socks5 instead of -x.
Resuming Downloads
For large file downloads, especially over unreliable networks, the ability to resume an interrupted download is invaluable. Use the -C - or --continue-at - option.
curl -C - -O https://example.com/large_archive.zipThe -C - tells curl to automatically determine where to resume the transfer. The -O ensures the original filename is used. This is effective only if the server supports range requests.
Mastering curl extends far beyond these examples. Continue exploring its capabilities by consulting the official documentation via man curl. This will uncover advanced features such as cookie handling, SSL/TLS certificate management, FTP operations, and more intricate HTTP request customizations.
