You want to download a file from a web site on the Internet.

Use the DownloadFile() method from the .NET Framework's System.Net.WebClient class to download a file:

PS > $source = "http://www.leeholmes.com/favicon.ico"
PS > $destination = "c:\temp\favicon.ico"
PS >
PS > $wc = New-Object System.Net.WebClient
PS > $wc.DownloadFile($source, $destination)

The System.Net.WebClient class from the .NET Framework lets you easily upload and download data from remote web servers.

The WebClient class acts much like a web browser, in that you can specify a user agent, proxy (if your outgoing connection requires one), and even credentials.

All web browsers send a user agent identifier along with their web request. This identifier tells the web site what application is making the request—such as Internet Explorer, Firefox, or an automated crawler from a search engine. Many web sites check this user agent identifier to determine how to display the page. Unfortunately, many fail entirely if they can't determine the user agent for the incoming request. To make the System.Net.WebClient identify itself as Internet Explorer, use the following commands, instead:

$userAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2;)"
$wc = New-Object System.Net.WebClient
$wc.Headers.Add("user-agent", $userAgent)

Notice that the solution uses a fully qualified path for the destination file. This is an important step, as the DownloadFile() method saves its files to the directory in which PowerShell.exe started (the root of your user profile directory by default) otherwise.

You can use the DownloadFile() method to download web pages just as easily as you download files. Just supply only an URL as a source (such as http://blogs.msdn.com/powershell/rss.xml) instead of a filename. If you ultimately intend to parse or read through the downloaded page, the DownloadString() method may be more appropriate.

For more information on how to use download and parse web pages, see the section called “Download a Web Page from the Internet”.

You want to download a web page from the Internet and work with the content as a plain string.

Use the DownloadString() method from the .NET Framework's System.Net.WebClient class to download a web page or plain text file into a string.

PS > $source = "http://blogs.msdn.com/powershell/rss.xml"
PS >
PS > $wc = New-Object System.Net.WebClient
PS > $content = $wc.DownloadString($source)

The most common reason to download a web page from the Internet is to extract unstructured information from it. Although web services are becoming increasingly popular, they are still far less common than web pages that display useful data. Because of this, retrieving data from services on the Internet often comes by means of screen scraping: downloading the HTML of the web page and then carefully separating out the content you want from the vast majority of the content that you do not.

The technique of screen scraping has been around much longer than the Internet! As long as computer systems have generated output designed primarily for humans, screen scraping tools have risen to make this output available to other computer programs.

Unfortunately, screen scraping is an error-prone way to extract content.

If the web page authors change the underlying HTML, your code will usually stop working correctly. If the site's HTML is written as valid XHTML, you may be able to use PowerShell's built in XML support to more easily parse the content.

For more information about PowerShell's built-in XML support, see the section called “Access Information in an XML File”.

Despite its fragility, pure screen scraping is often the only alternative. Since screen scraping is just text manipulation, so you've got the same options you do with other text reports. For some fairly structured web pages, you can get away with a single regular expression replacement (plus cleanup), as shown in Example 12.1, “Search-Twitter.ps1”.

param($term = "PowerShell")

Add-Type -Assembly System.Web
$queryUrl = 'http://integratedsearch.twitter.com/search.html?q={0}'
$queryUrl = $queryUrl -f ([System.Web.HttpUtility]::UrlEncode($term))

$wc = New-Object System.Net.WebClient
$wc.Encoding = [System.Text.Encoding]::UTF8
$results = $wc.DownloadString($queryUrl)

$matches = $results | 
    Select-String -Pattern '(?s)<div[^>]*msg[^>]*>.*?</div>' -AllMatches

foreach($match in $matches.Matches)
{
    $tweet = $match.Value -replace '<[^>]*>', ''
    
    [System.Web.HttpUtility]::HtmlDecode($tweet.Trim()) + "`n"
}
	

Others, while possible to accomplish with complicated regular expressions, can often be made much simpler through more straight-forward text manipulation. Example 12.2, “Get-Answer.ps1” uses this second approach to fetch "Instant Answers" from Bing.

$question = $args -join " "

function Main
{
    Add-Type -Assembly System.Web

    $encoded = [System.Web.HttpUtility]::UrlEncode($question)
    $url = "http://www.bing.com/search?q=$encoded"
    $text = (new-object System.Net.WebClient).DownloadString($url)

    $startIndex = $text.IndexOf('<div class="ans">')
    
    $endIndex = $text.IndexOf('<div class="sn_att2">')
    if($endIndex -lt 0) { $endIndex = $text.IndexOf('<div id="results">') }

    if(($startIndex -ge 0) -and ($endIndex -ge 0))
    {
        $partialText = $text.Substring($startIndex, $endIndex - $startIndex)

        $partialText = $partialText -replace '<div[^>]*>',"`n"
        $partialText = $partialText -replace '<tr[^>]*>',"`n"
        $partialText = $partialText -replace '<li[^>]*>',"`n"
        $partialText = $partialText -replace '<br[^>]*>',"`n"
        $partialText = $partialText -replace '<span[^>]*>'," "
        $partialText = $partialText -replace '<td[^>]*>',"    "
           
        $partialText = CleanHtml $partialText
           
        $partialText = $partialText -split "`n" |
            Foreach-Object { $_.Trim() } | Where-Object { $_ }
        $partialText = $partialText -join "`n"

        [System.Web.HttpUtility]::HtmlDecode($partialText.Trim())
    }
    else
    {
        "`nNo answer found."
    }
}

function CleanHtml ($htmlInput)
{
    $tempString = [Regex]::Replace($htmlInput, "(?s)<[^>]*>", "")
    $tempString.Replace("&nbsp&nbsp", "")
}

. Main
      

For more information about running scripts, see the section called “Run Programs, Scripts, and Existing Tools”.

You want to connect to and interact with an Internet webservice.

Use the New-WebserviceProxy cmdlet to work with a webservice.

PS > $url = "http://terraservice.net/TerraService.asmx"
PS > $terraServer = New-WebserviceProxy $url -Namespace Cookbook
PS > $place = New-Object Cookbook.Place
PS > $place.City = "Redmond"
PS > $place.State = "WA"
PS > $place.Country = "USA"
PS > $facts = $terraserver.GetPlaceFacts($place)
PS > $facts.Center

                                  Lon                                  Lat
                                  ---                                  ---
                    -122.110000610352                     47.6699981689453

Although screen scraping (parsing the HTML of a web page) is the most common way to obtain data from the Internet, web services are becoming increasingly common. Web services provide a significant advantage over HTML parsing, as they are much less likely to break when the web designer changes minor features in a design.

The only benefit to web services isn't just their more stable interface, however. When working with web services, the .NET Framework lets you generate proxies that let you interact with the web service as easily as you would work with a regular .NET object. That is because to you, the web service user, these proxies act almost exactly the same as any other .NET object. To call a method on the web service, simply call a method on the proxy.

The New-WebserviceProxy cmdlet simplifies all of the work required to connect to a web service, making it just as easy as a call to the New-Object cmdlet.

The primary differences you will notice when working with a web service proxy (as opposed to a regular .NET object) are the speed and Internet connectivity requirements. Depending on conditions, a method call on a web service proxy could easily take several seconds to complete. If your computer (or the remote computer) experiences network difficulties, the call might even return a network error message (such as a timeout) instead of the information you had hoped for.

If the webservice requires authentication in a domain, specify the -UseDefaultCredential parameter. If it requires explicit credentials, use the -Credential parameter.

When you create a new webservice proxy, PowerShell creates a new .NET object on your behalf that connects to that webservice. All .NET types live within a namespace to prevent them from conflicting with other types that have the same name, so PowerShell automatically generates the namespace name for you. You normally won't need to pay attention to this namespace. However, some web services require input objects that the web service also defines, such as the Place object in the solution. For these web services, use the -Namespace parameter to place the web service (and its support objects) in a namespace of your choice.

The New-WebserviceProxy cmdlet was introduced in version two of PowerShell. If you need to connect to a webservice from version one of PowerShell, see the section called “Program: Connect-WebService”.

For more information about running scripts, see the section called “Run Programs, Scripts, and Existing Tools”.

You want to export the results of a command as a web page so that you can post it to a web server.

Use PowerShell's ConvertTo-Html cmdlet to convert command output into a web page. For example, to create a quick HTML summary of PowerShell's commands:

PS > $filename = "c:\temp\help.html"
PS >
PS > $commands = Get-Command | Where { $_.CommandType -ne "Alias" }
PS > $summary = $commands | Get-Help | Select Name,Synopsis
PS > $summary | ConvertTo-Html | Set-Content $filename

When you use the ConvertTo-Html cmdlet to export command output to a file, PowerShell generates an HTML table that represents the command output. In the table, it creates a row for each object that you provide. For each row, PowerShell creates columns to represent the values of your object's properties.

If the table format makes the output difficult to read, ConvertTo-Html offers the -As parameter that lets you set the output style to either Table or List.

While the default output is useful, you can customize the structure and style of the resulting HTML as much as you see fit. For example, the -PreContent and -PostContent parameters let you include additional text before and after the resulting table or list. The -Head parameter lets you define the content of the HEAD section of the HTML. Even if you want to generate most of the HTML from scratch, you can still use the -Fragment parameter to generate just the inner table or list.

For more information about the ConvertTo-Html cmdlet, type Get-Help ConvertTo-Html.

You want to send an email.

Use the Send-MailMessage cmdlet to send an email.

PS > Send-MailMessage -To guide@leeholmes.com `
>>     -From user@example.com `
>>     -Subject "Hello!" `
>>     -Body "Hello, from another satisfied Cookbook reader!" `
>>     -SmtpServer mail.example.com

The Send-MailMessage cmdlet supports everything you would expect an email-centric cmdlet to support: attachments, plain text messages, HTML messages, priority, receipt requests an more. The most difficult aspect is usually remembering the correct SMTP server to use.

The Send-MailMessage cmdlet works to help this problem, as well. If you don't specify the -SmtpServer parameter, it uses the server specified in the $PSEmailServer variable, if any.

The Send-MailMessage cmdlet was introduced in version two of PowerShell. If you need to send an email from version one of PowerShell, see the section called “Program: Send-MailMessage”.