Search Results

Keyword: ‘log monitor’

I have been usurped in the silly monitors department!

October 22nd, 2009 jpavleck 1 comment

Alas, it seems those Danes are just as goofy as I am. And I’m happy to hear it!

Jakob Svendsen over at CoreTech released a management pack today. That management pack will alert you when the coffee in your coffee pot gets low. I’ll go ahead and embed the video below, but you’ll have to go to their site to download it.

Now, I return to my dank and dark office, where I shall create a SUPERIOR coffee pot monitor!
I’m actually waiting on an order from SparkFun and Jameco to get here as we speak.

 

Released Now – the NEW Authoring Resource Kit!

October 21st, 2009 jpavleck No comments

Now here’s an awesome bit of good news! Just announced the OpsMgr R2 Authoring Resource Kit! It’s available at this URL.

I am REALLY excited about this! Why? Well let’s check out all the cool stuff we’re getting:

  • Management Pack Best Practices Analyzer
    MPBPA scans management packs for best practice compliance and provides automated resolution for numerous issues. This tool integrates with the Authoring Console.
  • Management Pack Spell Checker
    MP Spell Checker checks spelling in management packs to eliminate errors in display strings.
  • Management Pack Visio Generator
    MP Visio Generator allows you to generate a class inheritance and class relationship diagram using Microsoft Office Visio.
  • Management Pack Diff
    MP Diff shows the differences between two management packs.
  • Management Pack Cookdown Analyzer
    MP Cookdown Analyzer identifies workflows which may break cookdown.  Suggestions are provided for how to fix the performance problems.
  • All Reference Add-in
    All References Add-in helps find all management pack elements that reference the specific element chosen. For example, the ability to right click a class and find all rules, monitors, overrides, as well as anything else that targets that class is provided. This tool works on most management pack elements.
  • Workflow Analyzer
    The Workflow Analyzer provides the ability to statically analyze all types of workflows.  It also allows users to trace workflows running on any Health Service.
  • Workflow Simulator
    The Workflow Simulator provides the ability to test certain types of workflows such as discoveries, rules, and monitors without a Management Server and Management Group. Key functionality includes the ability to test workflows as well as view and validate output prior to signing and importing the MP into a Management Group for additional testing.
  • Management Packs
    Three management packs which are frequently used as dependencies are provided as part of the tools installation.  These MPs are necessary to allow the Authoring Console to open most MPs available online in the System Center Operations Manager MP Catalog.

This is great news! Finally we’ve all been given the power to generate powerful, extensive management packs. You will be able to build it, diagram it, and most importantly – test it for quality assurance. No longer will we be left guessing if a certain change is better or worse – we’ll be able to find out ourselves!

Try PowerShell and the SCOM cmdLet reference. (And a surprise)

October 20th, 2009 jpavleck 5 comments

If you’re still hesitant to dive into powershell, you’re not alone. To someone who is a light scripter or has limited or no development background it can seem like a daunting task. A lot of the examples you’ll find are easy and simple, but Write-Host –Fore Green “Hello World! I’m using PowerShell” won’t help you with automating all of those tasks out there.

There are a few places to go, and a few books to read, to help you get up to speed with it.
One of the most promising looking resources would have to be Doug Finke’s “Try PowerShell”.

TryPowerShell

Try PowerShell  is an interactive PowerShell tutorial created with PowerShell using WPF1 and PowerBoots.
With it you can easily try out PowerShell commands just by clicking on them, no typing needed. You can search for particular commands, run them, and even edit them to see what happens if you change something. This is an awesome tool I don’t think enough people know about.

Read more…

  1. Wikipedia: Windows Presentation Foundation []

New IPD available for OpsMgr 2007 R2

October 6th, 2009 jpavleck No comments

 

The new Infrastructure Planning and Design Guide for OpsMgr 2007 R2 has been released.

If you’ve never checked it out, I strongly encourage you to – especially if you have a new install coming up. The IPD guide walks you through all aspects of the planning and design of an OpsMgr network. After you’ve gone through it you’ll know how to:

  • Define the scope of the project
  • Identify necessary management packs and product connectors
  • Determine how monitoring will be implemented
  • Determine the number of management groups and agent security strategy
  • Design and place the OpsMgr server roles and databases
  • Design the notification system
  • And much much more!

These guides are pretty invaluable, especially if you’re not an expert in the technology.
So what are you waiting for? Download it now – and when you’re done, check out the rest of the design guides, too!

Note: The version number is still 1.0 and the published date remains June 2008 – I sent a note off to ask them to increment that.

IPD SCOM Decision flow

Categories: Guides, R2, SCOM Tags: , ,

How to monitor everything

October 4th, 2009 jpavleck 4 comments

Operations Manager has always been a fairly powerful tool. Even in it’s MOM2000 and MOM2005 variations it has a lot to offer. Sadly, in all the organizations I’ve been to, it’s never been utilized to it’s fullest.

I want to help change that.

Do you have a ‘batch’ server at your organization? You should. A batch server is a general all purpose scripting machine. It’s one you use to automate a lot of tasks that you don’t want to have a dedicated machine for, but are more important then you’d want to trust to run on your regular workstation. It doesn’t need to be anything particularly powerful – my batch server is a small Dell pizza box with now outdated hardware: Pentium 4 3.4Ghz CPU, maxed out at 2GB of ram, with an 80GB hard drive. Up until recently it ran Windows 2000 – it now runs Server 2008 standard.

A lot of the stuff I’ll be writing about will have a batch server in mind. I run these scripts there, and use OpsMgr to interact with the data. You’d be amazed at how much more you can accomplish and automate things with a simple ‘throw away’ desktop.

To start out, let’s get OpsMgr to record some stock prices. This will be a powershell script, that I’ll be running on our batch server. Read on to see how I do it.

Read more…

OpsMgr Command Shell Performance

May 5th, 2009 jpavleck 4 comments

Have you noticed that the command shell performance is lacking? Tab-completion can take over 15 minutes to complete, depending on how many servers you’re monitoring. I’m not 100% sure of the exact reasons behind it, but it seems that the auto-complete will first traverse the local path you’re in when it attempts to complete a command. And when you’re trying to do this from the monitoring:: provider, that means it will possibly look through tens of thousands of objects.

I thought that behavior was a little odd, because most other providers will attempt to auto-complete known cmdlets first THEN go through the local directory. So I went to MS Connect and logged a bug report, bug ID #436239.

And this morning I received a response from Cory Delamarter, the Senior Program Manager for Operations Manager. He wrote:

Jeremy – Thank you very much for reporting this. We’ve confirmed this behavior and we will defintely look into how we can improve this in future releases.

The feedback is much appreciated!

So hopefully we’ll get that fixed soon and it’ll be just as fast as all the other providers out there! There really isn’t a work-around for this, as most (If not all) of the OpsMgr commands need to be run from the monitoring:: provider, or else they will fail. So we’ll have to live with it for just awhile longer.

Categories: Command Shell, SCOM Tags: , ,

Using the SDK to manipulate alerts

November 11th, 2008 jpavleck 1 comment

A quick disclaimer, I’m not a programmer. I’m a scripter. My code reflects that – meaning it has plenty of room for improvement. Not to mention, this is a learning aid and as such doesn’t need to be absolutely amazing. Now, on with the show.

When you’re dealing with custom alerts, and using something like the ResolutionState field to manipulate who gets what alert, you need to create a battle plan. A method of attack if you will.

One way I do this is to create my alerts with a naming convention of <Base Criteria><Sub Criteria> – My Alert.

If I wanted to create a large number of alerts for a specified department, I’d create my alert names with a base criteria of the department, like “Windows Admin Group”. I then use the subcriteria to specify what subgroup the notification should be sent to, or what particular object this alert is for. So my alert names would look something like “Windows Admin Group – CPU Usage – High CPU Usage on Special Server” or “Windows Admin Group – Exchange Admin – Low Memory Reported”. I’m using examples which you can already easily limit right now, but the methodology remains the same for custom events as well, like “Widget Support – DBA Support – The widget app has a sql fault! Oh no!”.

So, now we have the base criteria ready. And I’ll assume you’ve gone and setup the alert resolution states like I’ve shown you before. Now we’ll work on building an application that works with the SDK service to change these alert states. This method is extremely fast and robust; in testing I’ve been able to process 10,000 alerts to 15 different alert resolution states in under a minute – a time impossible to achieve via the powershell methods.

I’m building this app in VB.Net, and it’s not using much error checking. It’s fairly simple, but it’s easy to build upon and alter to your needs.
I’ve also created a domain user to authenticate to the SDK service, and added it to the OpsMgr admin group.

First we need to build the framework. Be sure to reference the 2 DLL files “Microsoft.EnterpriseManagement.OperationsManager.Common.dll” and “Microsoft.EnterpriseManagement.OperationsManager.dll” located in your <Install Dir>\System Center Operations Manager 2007\SDK Binaries folder.

Imports System
Imports System.Collections.Generic
Imports System.Collections.ObjectModel
Imports System.Text
Imports System.Security
Imports Microsoft.EnterpriseManagement
Imports Microsoft.EnterpriseManagement.Administration
Imports Microsoft.EnterpriseManagement.Configuration
Imports Microsoft.EnterpriseManagement.Configuration.IO
Imports Microsoft.EnterpriseManagement.Monitoring

Module StateChanger

Sub Main()

First we’ll create a comment to use when we update the alert. This is optional, but it’s helpful to let others know how we’re doing this

Dim comment as String = "Resolution State Changed by my awesome StateChanger App!"

Then we’ll create another string to hold our search syntax for our base criteria. I’ll keep using “Windows Admin Group” here.

Dim baseCriteria as string = "Name LIKE ‘Windows Admin Group’ AND ResolutionState = 0"

Next, we’ll setup our connection the RMS. First we setup the user.

Dim mgSettings as New ManagementGroupConnectionSettings(RMS)

mgSettings.Domain = "YOURDOMAIN"
mgSettings.User = "UserName"

The password criteria is a tad annoying, as it uses SecureString. We can’t just blindly cast a string to SecureString, we have to do some work first.
So we first create a string with our user’s password in it.

Dim myPlainPassword as string = "GaurhothRox!"

Then we create a new SecureString object, and loop through our password Appending each character from our plain password to it. Voila, a securestring password. Woohoo!

Dim securePass as New SecureString()
For Each chr as Char In myPlainPassword
securePass.AppendChar(chr)
Next

NOW we can assign that to our password.

mgSettings.Password = securePass

Now we can actually connect to the RMS.

Dim mg As ManagementGroup = New ManagementGroup(mgSettings)

Now we search for all the alerts that match our base “Windows Admin Group” criteria with a res state of 0.
We first take the baseCriteria string and convert it to a MonitoringAlertCriteria object, then use that to fetch all the matching alerts.

Dim alertCriteria As MonitoringAlertCriteria = New MonitoringAlertCriteria(baseCriteria)
Dim alerts As ReadOnlyCollection(Of MonitoringAlert) = mg.GetMonitoringAlerts(alertCriteria)

Then we loop through all of those alerts, seeing if any of those match our sub-criteria. If any do, we’ll go ahead and change it’s Resolution state and update it, adding our comment to it.

If (alerts.Count > 0) Then ‘ Mae sure we actually have something to deal with!
For Each Alert As MonitoringAlert In alerts
If InStr(Alert.Name, "CPU Usage", CompareMethod.Text) = 0 Then
Console.WriteLine("Found a ‘CPU Usage’ alert – ID: {0}", Alert.Id)
Alert.ResolutionState = 10
Alert.Update(comment)
ElseIf InStr(Alert.Name, "Exchange Admin", CompareMethod.Text) = 0 Then
Console.WriteLine("Found an ‘Exchange Admin’ alert – ID: {0}", Alert.Id)
Alert.ResolutionState = 11
Alert.Update(comment)

End If
Next
End If
End Sub
End Module

There you go. You’ve just written your first SDK application. You can expand it as much as you want from there! Run it every 5 minutes or so, and there you go.

Categories: SCOM, SDK Tags: , ,

It’s bound to happen: How to handle alert storms

October 6th, 2008 jpavleck 2 comments

No matter how careful you are. No matter how many times you’ve checked your parameters and state conditions. No matter how many times you tested it out of production.

It happens.

Some misconfigured rule – or even an event that happens much differently in a production environment then in test – begins firing off alerts.  Maybe you don’t notice it right away – perhaps you haven’t setup notifications for this particular rule.

But then, one day you start getting emails from your RMS with scary event IDs like 2115 (Data source not receiving a response), 25017 (Backlogged event processing) and 29202 (Inconsistent database state).

So you decide to investigate, and open up the Operations Console.

Only… wow, it’s running a lot slower then it normally does. Insanely slow.
You click onton Monitoring > Active Alerts – and then wait. And wait some more. As our once friendly green progress bar seems to start taunting you. So you lock the desktop and go chat up that new girl they hired. Wow, she’s pretty amazing right? Funny and smart as a whip, too.

Feeling happy and content after working your suave IT skills on her, you literally float back to your desk and unlock your desktop. Wasn’t there something bothering you before? Oh well, must have not been all that important. You peek up from your cube and catch a glimpse at her, then move those eyes down and see your still open Operations Console. The evil green bar still chugging away. But then you also see why…

A rogue rule quickly causes the alert count to soar to over 140,000

A rogue rule quickly causes the alert count to soar to over 140,000

And your nemesis, the green progress bar, it still keeps going. That number is rising faster then your blood pressure right now.

Must be a bug, eh? Ok, well, we’ll just check it via SQL to be sure – so you open SQL Studio and run

SELECT Count(*) FROM Alert

And then your informed, without any gentleness of a WWII nurse as depicted in the movies, that you have a lot of open alerts.

Over 250,000 alerts according to SQL

Over 250,000 alerts according to SQL

Wow! You better fix this!
And you better do it on the RMS, because it’s taking forever from your desktop.
You already have a general idea of which rule did it – that active alerts panel should be filled with it. So your first stop is to get back to authoring panel and either disable that rule or setup some proper alert suppression. Then we just have to deal with cleanup.

You turn, as always, to our friend PowerShell to help us out. Surely the easiest and most obvious solution to this problem is to run

$alerts = Get-Alert | Where-Object {$_.Name -match "MyRule"}
ForEach($alert in $alerts) {
$alert.ResolutionState = 255 # Close the alert
$alert.Update("")
}

Then just wait for the nightly alert grooming to happen to nudge it along with a SQL exec p_AlertGrooming

Only, when you try to do it, you get an OutOfMemory exception.

Out of memory!

Out of memory!

Now what to do?! The console is crippling slow – if you had to close the alerts that way your company would have gone bankrupt during the Dot Com Re-Burst of 2799! And when you try with PowerShell, you’ve run out of memory!

That’s where I was, until I talked to an unnamed friend((If you want to be named, just let me know. Better to err on the side of caution and all that)) from MS that really helped me out. That, combined with hindsight, allows me help you out as well!

How To Clean Up an Alert Storm

  1. Try the console. We’re going to assume it’s running slower than <Insert joke about large celebrities in the 1980s doing something they’re known for>, so we’ll move on.
  2. Try the Command Shell. $alerts = Get-Alert |? {($_.ResolutionState -eq 255) -or ($_.Name -match “Rule name if you know the naughty one”)}  – Running out of memory still?
  3. Try the same command, only instead of piping it to Where-Object, use the builtin filter object.
    $alerts = Get-Alert -criteria ‘WHERE ResolutionState = 0 AND Name LIKE ”%Rule Name%”’
  4. Still OOM? Try running both of those commands on the RMS, or another management server. Pick one with the most amount of memory, and hope for the best.
  5. Still receiving Out of Memory exceptions? Let’s stop using the OS to manage our memory. Open RegEdit and navigate to HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Config Service – change the value of Should Manage Memory from False to True. Stop and restart the config service.
  6. Now try your command again. It will take some time, but it will complete. Now lets try running the whole script to fix this:
    $alerts = Get-Alert | Where-Object {($_.ResolutionState -eq 255) -and ($_.Name -match "MyRulename")}
    ForEach($alert in $alerts) {
    $alert.ResolutionState = 255
    $alert.Update("")
    }

    (Alternately, you can use the Resolve-Alert cmdLet, but from testing it’s not quite fast enough to keep up with the next step)

  7. Now when you ran step 6, it probably gave you a lot of errors when attempting to update the alert. That’s because there’s a small window of freshness to your alert object, and if you don’t update it within that window it becomes stale and unable to be used. To fix that, change the ForEach to look like this:
    ForEach($alert in $alerts) {
    $freshAlert = Get-Alert $alert.id
    $freshAlert.ResolutionState = 255
    $freshAlert.Update(“”)
    }
    That will grab a fresh version of that alert and update it.
  8. But what if you have thousands upon thousands of alerts? The above solutions could conceivably take days to run. Don’t worry, there’s a way around that, too.
    Before I show you, please be noted that this METHOD IS NOT SUPPORTED BY MICROSOFT and use of this method could possibly BLACKLIST YOUR OpsMgr INSTALL. It is the answer given out occasionally though, much to the dismay of the product group, so use that information how you’d like.
  9. Connect to your operations manager database and run the following update. This one updates every rule, but you could narrow it down with an additional AND WHERE RuleName = “My Rule Name”
    UPDATE Alert
    SET ResolutionState = 255
    WHERE ResolutionState = 0 AND TimeResolved IS NULL
  10. When that’s completed, you’ll need to update the TimeResolved via:
    UPDATE Alert
    SET TimeResolved = ‘20-06-20 00:00:00.000′
    WHERE ResolutionState = 255 AND TimeResolved IS NULL

    < Make TimeResolved be some day in the past so it will groom them out.

  11. Either wait overnight until the grooming jobs kick off or run
    Exec p_AlertGrooming
  12. You’re done. Now don’t do it again!

[print_link]

Reminder: System Center Virtual User Group Meeting 2, PowerScripting LIVE tonight

September 4th, 2008 jpavleck 2 comments

Two awesome events are happening tonight. I’ll be at both, come join me!

The first is the second meeting of the System Center Virtual User Group – Register to attend.
It begins 7pm CST (1am GMT) and the agenda is:
7:00PM-7:10PM: Introduction – Pete Zerger (Of SystemCenterForum.org)
7:10PM-7:40PM: Steve Rachui – Targeting in Operations Mananger 2007.
7:40PM-8:10PM: Cameron Fuller – Community Extensions for Operations Mananger 2007.
8:10PM-8:40PM: Pete Zerger – SNMP Device Monitoring in Operations Mananger 2007.
8.40PM-8:55PM: Closing – Pete Zerger

Logo for Google Calendar invite

The invitation has all the other information you need. This is a Live Meeting event, and dial in audio information is provided.

EDITED: Thanks for the additional info, Hal!

The second meeting happening today is episode 40 of the PowerScripting Podcast. This will also be the second (Or is it third?) time Hal and crew stream the show live!
If you can’t make it to the live UStream broadcast, be sure to subscribe to the podcast and get your weekly dose of PoShy goodness.
Along with the usual suspects of Hal, Joel and Jon, they have an extra-special guest this week; none other then the great Jeffrey Snover, one of the PowerShell architects. He’ll be taking your questions, so this would be a great time to ask about what’s new in v2, or why remote registry support sucks.

That all starts tonight at 9PM EST – so get your browsers ready to watch Hal and his beard. And if your place of work doesn’t block social networking sites, be sure to RSVP to the invite on FaceBook.

Logo for Google Calendar invite

Setting agent failover servers & Switching SNMP device proxies

July 7th, 2008 jpavleck 3 comments

By default, you can’t really specify a failover management server in OpsMgr. Why? Not really sure, though I think it’s a ploy to ensure you setup the OpsMgr Active Directory Integration, which will handle this for you.

No fret though, we can still do it – it’ll just take a little bit of actual effort.

First, we need to define our Primary and Failover management servers. This isn’t something you can just progmatically grab, so you’ll need to know the name yourself.

In my $PROFILE, I’ve set them to be defined to 2 variables with the following:

# Set Primary Management Server
$Primary_MS = Get-ManagementServer | ? {$_.Name -like "SERVERNAME*"}
# Set Failover Management Server
$Failover_MS = Get-ManagementServer | ? {$_.Name -like "BACKUPSERVER*"}

Now that we have that set, it’s simple to do the rest. First, lets grab all of the servers that don’t have a failover management server set.

$noFailoverSpecified = Get-Agent | ? {(!$_.GetFailoverManagementServers())}

What the above does is call the GetFailoverManagementServers() method on each agent. If they have a failover, it will return data and thus $True. If there aren’t any failovers, it will return nothing – which is the same as $False. So we look for all the ones that don’t return anything.

If you’re curious, you can see just how many servers are missing failovers with

$noFailoverSpecified.Count

- in my case it was 63.

Now, we just run a quick snippet that adds the failover server to the agent:

ForEach ($agent in $noFailoverSpecified) {
Set-ManagementServer -PrimaryManagementServer $Primary_MS -AgentManagedComputer $agent -FailoverServer $Failover_MS | Out-Null
}

That will crunch away as it’s doing it’s thing, we’re redirecting output to $null so we don’t have to see agents scrolling over and over. When it returns you to a prompt, you’re done. If you’d like to verify that you did indeed set all of the agents to have a failover, we can check real quick:

If ((Get-Agent |? {(!$_.GetFailoverManagementServers())}).Count -eq $null)
{
Write-Host "Every agent has a failover server, great job!" -ForeGroundColor Green
} else {
Write-Host "Looks like we missed some, try again!" -ForeGroundColor Magenta
}

And that’s that. All of your agents have a primary and failover server.

Screen shot of SCOM Command Shell showing steps to setup failover agents

But wait, you have a lot of remotely managed devices too? Monitoring SNMP on a bunch of different servers – what happens for that?

Well, we can’t setup a failover (From what I’ve seen, if I’m wrong please let me know) agent. But we can proactively write a script that will change the proxy agent on the devices, and run it as needed.

This was written in a response to this query on the newsgroups, and is only a cursory look into it. There may be other ways of doing this – and I’d love to hear it. As it stands, I’m not sure how to set them back to a management server as the monitor.

Firstly, we’ll have to pick an agent managed computer to use as the new proxy agent. You can’t use a management server for this, because they aren’t “Agent Managed” and you can’t use Set-ManagementServer because the devices aren’t “Remote Managed Computers”.  I have a seperate agent-managed server on my network I call “Timex” because it acts like a watcher node. So I’ll go ahead and use him.

$proxyAgent = Get-Agent |? {$_.Name -eq "timex.pavleck.net"}

Then gather a list of our current remotely managed devices

$remDevices = Get-RemotelyManagedDevice

Now just loop through it, setting the device to use the proxy agent we just instantiated:

ForEach($device in $remDevices) {
Set-ProxyAgent -ProxyAgent $proxyAgent -Device $device | Out-Null
}

That will loop through things changing the proxy server that it uses. When it’s done, we can verify it by running:

Get-RemotelyManagedDevice |? {$_.ProxyAgentPrincipalName -ne $proxyAgent.Name}

If it outputs nothing, then they’ve all been changed. Simple as that!

SCOM: Setting the proxy agent for a device via command shell