Pavleck.NET

Edit: 09/13/2008 – On the advice of Pete Zerger, updated script to include a throttling mechanism to prevent an overload if an alert storm occurs. Also changed things around to make it a more generic ‘run remote executable’ instead of run remote sound.

A question was recently asked on the MOM Mailing List over at myITforum.com.

That question was, quite to the point:
“How to create a audible alert? I like to create one for the critical alerts..”

I’ve been working earlier with a script that would go out and disable the run time tracing, stop it, then delete the log files. So I already had knew what would work – a simple PowerShell script that uses WMI’s process create method on a remote machine.

A caveat lector before I continue; while this solution will technically work, I haven’t tested it formally. Additionally, you’ll need to contend with permission issues that arise as well. If you’re running the OpsMgr services under a named account, you’ll need to give that same account local administrator access on whichever machine you plan to run this call against. If you’re using ‘Local System’ you’ll have to either add the RMS\Local System account to the remote machine’s admin group or embed credentials inside the WMI call((Be careful when doing this. I haven’t included directions for that because it’s just a nightmare waiting to happen. I can give you a jumping off point though.))

First, the script. It’s small and basic. It wants to know the machine you want to run the command on, the command, and because this is a a sound player, the path to the WAV file. It then creates the process via WMI, and decodes the return code. If it’s 0, everything is fine. If it’s anything else, the process creation failed and it writes an event to the Operations Manager event log, which you can create an additional rule to look for.

Download SCOM-RunRemoteExecutable.ps1

To implement this, open the Operations Console and go to Administration > Settings > Notification
Click on the Command tab, then click on add. Fill it out as you normally would:

Then click on OK, and you’ll see it with the rest of your commands:

Now to finish it up you’ll need to create a new notification recipient. Right-click on Notifications and select new recipient.

Make the display something to designate that it runs a command, I used “Sound Audible Alert”. And because the NOC isn’t manned 24/7, I limited the notification time to weekdays from 8am to 6pm. You can also adjust this from the devices tab, but I’m not going to include an emailing or other devices, so I prefer to set it in the general tab, this way it’s obvious even with a casual glance what the settings are.

After that, click on the “Notification Devices” tab, then click “Add”.
In the resulting popup, select our new notification command and enter anything for the delivery address – I used NA, because for this particular command we don’t require any additional information – but OpsMgr still needs something in there. Hit next, keep the schedule at always unless you’re adding additional channels, next again, name the device – I used “Send Audible Alert”

Click OK, and your set. Treat it like any other notification recipient – either create a new rule just for this, or edit an existing subscription and add our new recipient to it.

As you can see, using PowerShell inside of Operations Manager makes it very flexible and powerful. We can run all manner of things in response to alerts; From running a simple sound file all the way up to initiating disaster recovery scenarios and intense system diagnostics – both things which I’ll be showing you later on as we explore the Notification Command Channel together.

Edit 09/10/2008: Fixed the script, fixed the reference in point 14.

I haven’t seen this solution offered as a way to send more customized alerts, and am fairly excited about it. With some of the previous solutions, they involve using the command shell to create an alert notification. This is fine, except if you open the subscription in the GUI – once you’ve done that, you’ve essentially undid all that work and created a ‘catch all’ that sends an alert on any event. Why? Well, the GUI itself isn’t designed for the custom settings that can be done in Powershell. This makes it fairly difficult to add people or change the alert – not acceptable to me.

After messing around with authoring console and creating classes based on event viewer errors and other equally exotic methods I came upon something that works wonderfully. The catch? You can only create 254 rules this way.

What am I talking about? Some powershell scripts and the alert resolution states!

By default, there are only 2 states defined – 0 for New, and 255 for Closed. They are always there, and can not be deleted. This leaves 1 – 254 as user definable states. We can use these to make one-to-one events.

Let me start off that this isn’t an ideal solution, but it is the most readable and elegant solution for this particular problem. You probably shouldn’t do this on a single rule basis, but target it more at a wildcard match. You do have a naming convention for your rules and monitors, right? If not, this is the perfect reason to get one. I’ll typically use a convention of <Product Type>-<Product>-<Version (If multiple)-<Rule>. So if I had a rule targeted at exchange, I’d have a rule similar to “EMAIL – Exchange – Exchange 2007 – Search for ‘Jeremy is fired’ in execs mail”. Then when I’m using an exotic config to send an alert, such as this one, I can better fine tune alerts.

Remember, the more you move away from the  “Out of box” yfunctionality with OpsMgr, the more you should be documenting. Or even better, a wiki. Just make a reference to the wiki in the description, and people will know exactly what you’re trying to do – that’s for another post though.

Let’s get on with it, shall we?

I’m going to create a situation. I have a custom application which logs to the Application log. There’s one particular event that only one group in the organization cares about – all they want is a notification of this one single event and nothing else. How do we do it?

The Cliff’s Notes version of what we’ll be accomplishing today:

• Create a custom Resolution State
• Define a new rule
• Deploy a PowerShell script to the RMS to update the resolution state of matching alerts
• Create a notification subscription which responds to our particular state

Now, for the complete steps

1. First go to Authoring > Management Pack Objects > Rules – Right click and “Create new rule”
2. Under rule type, select Alert Generating, Event Based, NT Event Log (Alert) and select a management pack to use.
   
3. Enter a rule name that is distinctive enough that no other rules will have that same name.  Then enter a description, rule category and choose a target. You can go with the shotgun approach and pick “All Computers” here if you’d like.
   
   
4. Now walk through the rest of the wizard and configure your event log settings – for this test I’m using the Application log, Event ID of 926 and event source of “Pavleck.NET Test”. But you can put whatever you want here ace, it’s up to you.
5. Configure your alert. It should automatically copy over the rule name as the alert name. The alert name is what we’ll actually be alerting on, so it’s important that you remember what it is, and ensure it’s distinctive enough to not match something that already exists.
   
6. Now we can work on the other parts of this while our rule is propagating across the environment.
7. Go to Administration > Settings > Alerts – this is where we’ll define a new Alert Resolution State to use. Click on “New…” and name your state and choose an ID for it, I used 10 in this example.
   
8. Click Apply, then Ok and now we’re done with part 2.
9. Let’s go ahead and test and see if our rule works, just open a command window and use some EventCreate.exe magic.
   
10. Open up the alert console for whatever machine you ran that on and you should see our new alert in there – yay, we did something!
    
11. Now we’ll add the magic that changes the alert resolution state. It’s a fairly simple script, and it’s meant to be that way. For simplicity’s sake, we’ll be running this script as a timed response from the RMS. Depending on how your particular environment is setup, you could also run it inside of the rule itself, as an additional response to “Create Alert”. But that only works well if you only plan on doinbg this sparingly, otherwise it makes more sense to run this from the RMS and add onto the script as needed.
    First, download SCOM-UpdateResolution.ps1 here (Or view it after the jump) and edit the alert name, resolution state and RMS to what matches your environment.
12. Now we’ll need to go and create a new rule. Rule type is Timed Commands > Execute a command. Give it a name and description. I’ve set the rule category to “Maintenance” as that makes the most sense to me.
    
13. For the schedule, I’ve set mine to run every 2 minutes. This means there will be a delay of that much between alerts and notifications, but that’s acceptable to me. Then hit next.
14. Configure the command line execution settings as shown – remembering to use instead of “&”. I’ve set the timeout to 45 seconds.
    
15. Hit create and that’s almost all of it – all we need to do now is to create the alert subscription. Go to administration, right click on Subscriptions and choose “Create new notification subscription”
16. Step through it like normal, choosing all groups, all classes. When you get to the Alert Criteria page, uncheck “New” and “Closed” and check our new resolution state. If you keep ‘closed’ in there, it will pertain to all alerts that close. That’s one drawback to this method, you won’t get closed alerts.
    
17. Finish it up as you normally would, then lets test it! Create a few more test events, and lets see if it works.

That’s all there is to it. This works, reliably and 100% of the time. It’s extremely flexible and easy to follow for someone just walking into your environment.

By using a single PowerShell script, and targeting the RMS computer group you’ll be making sure that you have only a single simple script to edit and by mirroring the files and directory paths to any other management servers in your environment you maintain this method if you ever need to promote one to an RMS.

Read more…