A Green Report Has to Mean Something.

Three times a day, a separate operations support team logged into every storage array, SAN switch, and vCenter by hand. They read the metrics off each system, assembled them into dashboards, and emailed the result out. Then four hours later they did it again.

The emails arrived, but each was a snapshot. Nobody was comparing this morning’s error counters with last night’s. The previous reading was in an older email, so looking for a developing problem meant finding and comparing those messages.

My manager asked me to replace the manual process.

I started with PowerShell on Windows Task Scheduler. It connected to each system, collected the metrics the team had been reading, applied thresholds, and emailed a formatted report. I kept the three-times-a-day schedule because that was what people expected. The automated schedule came directly from the manual one.

It worked, but I had no record of which version ran, no archived output, and no safe way for anyone else to change it. Fixing that became a separate platform project. The health checks were its first workloads. I moved them to Jenkins pipelines using GitLab, purpose-built containers, controlled credentials, and archived results.

I wrote dozens of explicit thresholds for the different platforms and put them in configuration. I wanted the rules for identifying trouble to be visible without digging through the code. They covered link errors, optics health, replication lag, and capacity, and aggregate expansion was blocked outright if a change was too large or would leave the aggregate nearly full.

The SAN checks compared counters with their previous values. A port with 4,000 lifetime CRC errors might have had them for three years. A port gaining 100 since the last run needed investigation for an optic failing now. I retained state between runs so the rules could look at change and trend, and those rules waited for at least two saved readings. The old process hadn’t made those comparisons.

In the retained window, 209 of 213 runs succeeded.

The reports went to storage and virtualization engineers, the operations team, and my management chain. An engineer could use them to investigate an optic. A leader could use the same report to see whether conditions across the systems were getting worse.

What it catches

Capacity was what the checks caught most often. I wanted a capacity finding to lead to action instead of sitting in an email.

Each run checked volumes, aggregates, and datastores against their thresholds. I kept ninety days of capacity history to see how fast they were filling. A volume crossing its warning threshold was flagged. So was a volume projected to fill within a month, even if it hadn’t crossed a threshold yet.

Warning-level findings showed yellow and critical findings showed red. Each red finding added to a count in the email subject. I couldn’t assume someone would keep opening a report after ninety uneventful mornings. The subject needed to tell them there was something to look at.

Critical findings opened tickets automatically, without a person deciding to open one.

Expansion was a separate step with its own checks. The tooling refused a change that was too large or would leave the aggregate nearly full. Fixing a full volume shouldn’t leave the surrounding aggregate in worse trouble.

The checks ran three times a day, forecast capacity trouble, put it in front of people, and opened tickets at the critical threshold. The separate expansion checks blocked unsafe fixes. Together, those steps helped get capacity problems handled before they became incidents.

What they watch

SAN switches	More than a dozen, across multiple fabrics
vCenters	More than a dozen
Storage arrays	Several current, more at peak
NetApp clusters	A few, current and legacy generations
Virtual machines, one discovery run	Thousands found
Storage under watch	Hundreds of TB committed, over a petabyte provisioned

I estimated the manual work replaced at roughly two full-time positions of repeated manual work.

Nobody timed the old process before it went away. This is my estimate of the repeated work people stopped doing three times a day, not a measured baseline or a claim that two jobs were eliminated.

The saved history was another change. Dozens of thresholds were evaluated every run, error counters were compared with earlier readings, and the results were archived. Engineers had more to work from than a series of emails.

What worried me most was a check saying things were healthy when it couldn’t see them. A failed monitor is obvious. A green report can leave people believing systems are fine when they haven’t been checked.

Two examples from my own systems, both fixed:

The storage check treated missing data as healthy. A component with no reported health state defaulted to OK, and a parse failure displayed All OK. I fixed that in 2025 to report Unknown and write a log entry. I also documented the paths that hid failures and classified their risk.

The virtualization health score always reported 100 percent. After I rebuilt it, the first honest run showed dozens of active alarms, dozens of snapshots past their age limit, and about half of the vCenters unreachable from the monitoring host. Half the environment had been missing behind that green score.