Part 14. HTTP Requests to Web Resources in PowerShell

25 November 2023 12 minutes Author: Cyber Witcher

Using HTTP requests in PowerShell to interact with web resources

Managing automation processes, services, and servers with PowerShell is a fundamental capability for any IT and cybersecurity professional. PowerShell, developed by Microsoft, is a powerful scripting and automation tool that allows you to monitor and manage processes and services on computers and servers. In this article, we look at how to use PowerShell to monitor, start, stop, and configure processes and services, as well as to manage servers and their configuration. You will learn to use a variety of commands and scripts to effectively manage your IT infrastructure.

This article will also provide you with information on creating scripts to automate tasks such as regularly updating servers, monitoring system health, and fixing errors. There are a huge number of useful services on the Internet that you can work with by accessing their resources using the NTTR protocol. Web developers work with these HTTR requests all the time to access external API functions or to test their own applications. Traditionally, this is done using special applications (for example, Postman), console utilities such as cURL and wget, as well as NTTR clients integrated into development environments. PowerShell offers two standard cmdlets for accessing web resources using the NTTR protocol: Invoke-WebRequest and Invoke-RestMethod.

The Invoke-WebRequest cmdlet

You can use the Invoke-WebRequest cmdlet to send a request to a web server and get a response from it.

Analysis of NTMI pages

The Invoke-WebRequest cmdlet works well for Tayuke parsing: this cmdlet can save pages to local disk, it is similar to the wget console utility, and even has the following alias:

PS C:\Script> Get-Alias wget
CommandType Name
Alias wget -> Invoke-WebRequest

Use this cmdlet to navigate to a simple page, such as Domain Example (https://example.com/index.html) (see Figure 1). 14.1

By default, Invoke-WebRequest makes an HTTP GET request to a resource on the web server, with the resource address specified as the value of the -Uri parameter. As a result, an object of type Htm1WebResponseObject is returned, which stores information about the server’s response:

The home page of the Example Domain site in the browser
PS C:\Script> $web = Invoke-WebRequest -Uri https://exanple.com/index.ht3nl
PS C:\Script> $web | Get-Member
TypeName: Microsoft.PowerShell.Commands.HtmlWebResponseObject
PS C:\Script> $web
StatusCode : 200
StatusDescription : OK
Content : <!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8” />
<meta http-equiv="Content-type" content="text/html; char 
set=utf-8” />
<meta name="viewport" conten...
RawContent : HTTP/1.1 200 OK
Age: 497890
Vary: Accept-Encoding
X-Cache: HIT
Forms
Content-Length: 1256
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Mon, 12 Jul 2021 16:05:14 GMT
Exp...
: {}
Headers : {[Age, 497890], [Vary, Accept-Encoding], [X-Cache, HIT],
[Content-Length, 1256]... }
Images
InputFields
Links
: {}
: {}
: {@{innerHTML=More information...; innerText=More information
...; outerHTML=<A href="https://www.iana.org/domains/example
">More information...</A>; outerText=More information...;
tagName=A; href=https://www.iana.org/domains/example}}
ParsedHtml
RawContentLength
: mshtml.HTMLDocumentClass
: 1256

The statuscode field contains the response code from the server (200 for our example), the statusDescription field contains a text description of the response (OK).

The content of the response from the server and the HTTP header

The content of the response from the server is stored as a string in the content field. In our case, the HTML code will be written here:

PS C:\Script> $web.Content
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name-'viewport” content="width=device-width, initial-scale=l" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI",
"Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
}
div {
width: 600px;
margin: 5em auto;
padding: 2em;
background-color: #fdfdff;
border-radius: 0.5em;
box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
}
a:link, a:visited {
color: #38488f;
text-decoration: none;
@media (max-width: 700рх) {
div {
margin: 0 auto;
width: auto;
}
}
</style>
</head>
<body>
<div>
<hl>Example Domain</hl>
<p>This domain is for use in illustrative examples in documents. You may
use this domain in literature without prior coordination or asking
for permission.</p>
<p><a href=”https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

The Rawcontent field records the complete response from the server with HTTP headers at the beginning:

PS C:\Script> $web.RawContent
НТТР/1.1 200 OK
Age: 497890
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 1256
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Mon, 12 Jul 2021 16:05:14 GMT
Expires: Mon, 19 Jul 2021 16:05:14 GMT
ETag: "3147526947+ident"
Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
Server: ECS (dcb/7F83)
<!doctype html>
<html>
<head>
<title>Example Domain</title>

Also, the response headers are separated and stored in the form of a hash table as headers:

PS C:\Script> $web.headers
Key Value
Age 497890
Vary Accept-Encoding
X-Cache HIT
Content-Length 1256
Cache-Control max-,
Content-Type
Date
Expires
ETag
Last-Modified
Server
max-age=604800
text/html; charset=UTF-8
Mon, 12 Jul 2021 16:05:14 GMT
Mon, 19 Jul 2021 16:05:14 GMT
"3147526947+ident"
Thu, 17 Oct 2019 07:18:26 GMT 
ECS (dcb/7F83)

Saving web resources

To save the response from the server as a local file, use the -OutFile KEY when calling Invoke-WebRequest and specify the path to the desired file.

PS C:\Script> Invoke-WebRequest -Uri https://exanple.com/index.html
-OutFile page.html

Let’s check the contents of the page.html file and make sure that it contains the HTML markup of the saved page:

PS C:\Script> type .\page.html
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content=”text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=l" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI",
"Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
div {
width: 600px;
margin: 5em auto;
padding: 2em;
background-color: ttfdfdff;
border-radius: 0.5em;
box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
a:link, a:visited {
color: #38488f;
text-decoration: none;
}
@media (max-width: 700px) {
div {
margin: 0 auto;
width: auto;
}
}
</style>
</head>
<body>
<div>
<hl>Example Domain</hl>
<p>This domain is for use in illustrative examples in documents. You may
use this domain in literature without prior coordination or asking
for permission.</p>
<p><a href="https://www.iana.оrg/domains/example”>More information... </ax/p>
</div>
</body>
</html>

In a similar way, it is possible to store not only HTML files, but also other types of resources (text, graphic or multimedia files, etc.) that can be referred to the site.

Search for HTML elements on a page

The Forms (forms), Images (images), InputFields (INPUT FIELDS) and Links (links) properties of the HtmiwebResponseObject object contain arrays of objects that describe the corresponding elements of the HTML markup received from the server. By processing these collections, we can get information about the items we are interested in on the loaded page.

We will highlight all links to the habr.com site on this page. To do this, you need to filter the $web array. Links, leaving in it objects whose value of the href property corresponds to the mask *habr.com*:

PS C:\Script> $habr_links = $web.Links | Where-Object href -like '*habr.com*'

$habr_iinks contains two PSCustomObject objects containing different HTML attributes for the links:

PS C:\Script> $habr_links.count
2
PS C:\Script> $habr_links | Get-Member

For example, let’s output the value:

PS C:\Script> $habr_links | ForEach-Object {$_.href + " - " + $_.innerText }
https://habr.com/ru/company/ruvds/blog/487876/ -
Что такое Windows PowerShell и с чем его едят? / Хабр
https://habr.com/ru/company/ruvds/blog/487876/ -
habr.com >ru/company/ruvds/blog/487876/

The ParsedHtml property of the HtmlWebResponseObject object contains an object of type mshtmi.HTMLDocumentciass, which provides access to the DOM tree of the loaded HTML page.

PS С:\Users\andrv> $html = $web. ParsedHtml
PS C:\Users\andrv> Get-Member -Inputobject $html
TypeName: mshtmi.HTMLDocumentciass

Execution of POST requests

The Invoke-WebRequest cmdlet allows not only to perform GET requests, but also to call other methods defined in the HTTP protocol (delete, head, merge, patch, post, put, trace). And so the desired method is specified as the value of the -Method parameter. Consider an example of executing a request with the HTTP post method, which is often used to transfer data from web forms or upload files to a server. We will contact the resource http://httpbin.org/post, as a result, the server should inform us about the received data. We will pass two parameters with the names name and lastName, which we will place in the hash table $params:

PS С:\Users\andrv> $params = @{name='Andrey'; lastName='Popov'}

Let’s execute Invoke-WebRequest with the post method, placing the parameters passed in the request body (the -Body parameter):

PS С:\Users\andrv> Invoke-WebRequest -Uri http://httpbin.org/post -Method POST
-Body $params
StatusCode
StatusDescription
Content
: 200
: OK
: {
"args": {},
"data": "",
"files": {},
"form": {
"lastName": "Popov",
"name": "Andrey"
},
"headers": {
"Content-Length": "26",
"Content-Type": "application/x-www-form..
RawContent : HTTP/1.1 200 OK
Connection: keep-alive
Access-Control-Allow-Origin: ★
Access-Control-Allow-Credentials: true
Content-Length: 503
Content-Type: application/json
Date: Fri, 16 Jul 2021 03:03:13 GM...
Forms
Headers
: {}
: {[Connection, keep-alive], [Access-Control-Al
low-Origin, *], [Access-Control-Allow-Credent
ials, true], [Content-Length, 503]...}
Images
InputFields
Links
ParsedHtml
RawContentLength
: {}
: {}
: {}
: mshtml.HTMLDocumentClass
: 503

The parameters of the invoke-webRequest cmdlet sent to the server automatically result in the application/x-www-form-urlencoded format used when submitting data from web forms. The response in the content field indicates that the server accepted our parameters and determined that they were sent from the form. Sometimes you need to send data to the server in JSON format, and not in application/x-www-form-urlencoded. In this case, you need to specify the -contentType parameter with the value application/json. Example:

PS С:\Users\andrv> $json_params = "{ 'nameAndrey', ' lastNamePopov' }"
PS C:\Users\andrv> Invoke-WebRequest -Uri http://httpbin.org/post -ContentType
"application/json" -Method POST -Body $json_params
StatusCode : 200
StatusDescription : OK
Content : {
"args": {},
"data": "{ ’name':'Andrey’, 'lastNamePopov
’ Г,
"files": {},
"form": {},
"headers": {
"Content-Length": "39",
"Content-Type": "application/json",
"Host": "...
RawContent : HTTP/1.1 200 OK
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 475
Content-Type: application/json
Date: Fri, 16 Jul 2021 03:43:04 GM...
Forms
Headers
: {}
: {[Connection, keep-alive], [Access-Control-Allo
w-Origin, *], [Access-Control-Allow-Credentials
, true], [Content-Length, 475]...}
Images
InputFields
Links
ParsedHtml
RawContentLength
: И
: {}
: (}
: mshtml.HTMLDocumentClass
: 475

As you can see, with this method of sending the request, the server script that processes access to the http://httpbin.org/post resource extracted the received data and, when forming its JSON response, placed it in the data field, not in the form field. The Invoke-RestMethod cmdlet if we are contacting a web service that supports the REST API, the server’s response will most likely contain structured data in JSON or XML format. So, in the previous example, using the Invoke-WebRequest cmdlet, a POST request was sent to the http://httpbin.org/post resource and a JSON string was received in response:

post -Method POST -Body @ {name=' Andrey ’ ; lastName= ’ Popov' }
PS C:\Users\andrv> $web.Content
{
"args”: {},
’’data”:
"files": {},
"form": {
"lastName": "Popov",
"name": "Andrey"
},
"headers": {
"Content-Length": "26",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Windows NT; Windows NT 10.0; ru-RU)
WindowsPowerShell/5.1.19041.1023",
"X-Amzn-Trace-Id": "Root=l-60f2a42c-18f152db3b76b5d66dl73e31"
},
"json": null,
"origin": "85.95.179.209",
"url": "http://httpbin.org/post"

To work with response fields, we must convert the JSON to a PowerShell object using the convertFrom-Json cmdlet (we covered similar conversions in Chapter 12):

PS С:\Users\andrv> $response = $web.Content | ConvertFrom-Json
PS C:\Users\andrv> $response
args :
data :
files :
form : @{lastName=Popov;name=Andrey}
headers : @{Content-Tength=26; Content-Type=application/x-www-form-urlencoded;
Host-httpbin.org; User-Agent=Mozilla/5.0 (Windows NT; Windows NT 10.0;
ru-RU) Windows PowerShell/5.1.19041.1023;
X-Amzn-Trace-Id=Root=l-60f2a42c-18f152db3b76b5d66dl73e31}
j son :
origin : 85.95.179.209
url : http://httpbin.org/post
PS C:\Users\andrv> $response.form.name
Andrey

In such cases, when we receive structured data from the server, it is more convenient to use the invoke-RestMethod cmdlet, which acts similarly to Invoke-WebRequest and has the same parameters, but at the same time automatically converts the response from the server into a PowerShell object. Let’s execute our request using invoke-RestMethod, saving the result in the $result variable:

PS С:\Users\andrv> $result = Invoke-RestMethod -Uri http://httpbin.org/
post -Method POST -Body @ (name='Andrey' ; lastName=' Popov'}

We will check the type and content of the variable $ result:

PS С:\Users\andrv> Get-Member -Inputobject $result
TypeName: System.Management.Automation.PSCustomObject
uri NoteProperty string url=http://httpbin.org/post
PS C:\Users\andrv> $result
Name MemberType Definition
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
args NoteProperty System.Management.Automation.PSCusto
data NoteProperty string data=
files NoteProperty System.Management.Automation.PSCusto
form NoteProperty System.Management.Automation.PSCusto
headers NoteProperty System.Management.Automation.PSCusto
j son NoteProperty object json=null
origin NoteProperty string origin=85.95.179.209
args :
data :
files :
form : @(lastName=Popov;name=Andrey}
headers : @{Content-Length=26; Content-Type=application/x-www-form-urlencoded;
Host=httpbin.org; User-Agent=Mozilla/5.0 (Windows NT; Windows NT 10.0;
ru-RU) Windows PowerShell/5.1.19041.1023;
X-Amzn-Trace-Id=Root=l-60f2a897-6354ce272644b6d412560a9b}
j son :
origin : 85.95.179.209
uri : http://httpbin.org/post

As you can see, instead of a string in JSON format, we get a PowerShell object of type System.Management.Automation. PSCustomObject And we can immediately refer to the necessary ones properties:

PS С:\Users\andrv> $result.form.lastName
Popov

Therefore, it is more convenient to use the Invoke-webRequest cmdlet to make HTTP requests to web resources that return HTML pages, and the Invoke-RestMethod cmdlet is more suitable for working with external REST APIs that return structured data.

Results

  • PowerShell has two standard cmdlets for accessing web resources using the HTTP protocol: Invoke-webRequest and Invoke-RestMethod.

  • These cmdlets work similarly to each other, except that Invoke-RestMethod automatically converts the response from the server into a PowerShell object.

Other related articles
Found an error?
If you find an error, take a screenshot and send it to the bot.