Introduction
The Challenges of BMP Monitoring
Best management practices (BMPs) for stormwater control are structures designed to prevent, limit, or eliminate pollutant discharge. Design of an effective BMP for each specific location is often left to engineers, but implementation of post-installation monitoring programs often falls into the laps of local, county, or state level employees, many of whom have natural resources backgrounds with limited statistical and analytical experience.
The challenges of post-installation BMP monitoring are numerous. Urban stormwater runoff is notoriously flashy, often reaching high levels and discharge rates within minutes of the onset of precipitation. Commonly, storms happen at night or on the weekends, so it is difficult to collect manual samples. Many groups employ automated samplers to assist with water collection at all hours and throughout the entire hydrograph of a storm. Automated samplers are very effective, but keeping this expensive equipment dry and functioning well is challenging. Often, it can be a struggle to obtain any sample, much less samples from the same storm from both the inflow and outflow of a BMP.
Despite these inherent monitoring challenges, the inflow and outflow samples are not independent of one another (i.e., they are paired), and therefore should be treated as paired. Here I present examples of how assessing inflow and outflow samples from BMPs as paired samples will increase the statistical power of analyses for managers wishing to assess BMP efficacy.
Statistically Assessing BMP Efficacy
Removal Efficiency
The difference between the inflow and outflow loads can be measured on a storm-by-storm basis or using multiple storms to assess BMP removal of pollutants and water. Table 1 shows a hypothetical example of a BMP’s water removal from five storms of different sizes; the inflow and outflow volumes measured from each storm; and the removal, defined as the percent change in volume between the inflow and outflow. I will use these hypothetical data to present examples of calculating average removal efficiency, and also statistical tests for significant differences in volume between what the inflow and outflow of the BMP. You will see that depending upon how these analyses are approached, they can yield vastly different results.
Average removal efficiency can be calculated in two different ways: 1) using the overall average inflow and outflow data, or 2) by averaging the individual storm removals. Using the first approach, the averages of inflow and outflow data could be used to calculate removal efficiency as 32% (Table 1, [(34,000-23,600)/34,600 x 100]). In contrast, using the paired inflow and outflow data gives increased statistical power and information. With this approach, each storm is analyzed separately, and then an overall removal can be calculated as the average of each discrete event, or 47% (Table 1). Furthermore, it is apparent that the BMP volume removal shown in Table 1 is much higher in small events (>0.3 in) than in large. Without using paired samples, the removal efficiency would be underestimated (in this case by 15%), and the storm size removal effect (i.e., this BMP has higher removal in smaller than in larger storms) would go undetected.
Statistical Tests for Removal Between the Inflow and Outflow
Furthermore, a manager could use simple statistical tests to assess whether differences were due to random chance or the BMP efficacy. To test for significant removal efficiency, the simplest and most effective approach is to use a simple t-test to characterize statistically different means between the inflow and outflow samples. T-tests can be performed on either paired or unpaired data, meaning the inflow and outflow samples are either related or unrelated to each other (McHugh 2011). The inflow and outflow of a BMP are intrinsically linked statistically and non-independent–what comes into the BMP at the inflow point is directly related to what comes out at the outflow end for each storm. Therefore, it is most appropriate to use a paired t-test of non-independent samples.
A t-test will output a t-statistic and a p-value; the more important of these is the p-value, which tells us the probability of the t-statistic being a random chance (so the lower, the better). Comparing the t-tests from the hypothetical BMP data yields drastically different results between paired and unpaired data. A paired t-test on the hypothetical BMP data in Table 1 yields a statistically significant result (t = 3.53, p = 0.02), but an unpaired t-test is insignificant (t = 0.49, p = 0.63). So for our paired t-test the interpretation says there is a 2% chance that the difference between inflow and outflow was due to random chance, but for the unpaired data there is a 63% chance that the difference between inflow and outflow was just random. By correctly assuming a statistical link between the inflow and outflow data from each storm, it is possible to show a statistically significant reduction in stormwater volume in the hypothetical BMP and state with 98% certainty that the volume reduction was due to the BMP and not random.
Conclusions
Monitoring BMPs is an important aspect of any installation, but the challenges of collecting samples are numerous. By keeping in mind the statistical link between inflow and outflow samples and focusing collection efforts on paired samples from each storm, monitoring personnel will save money and time. Perhaps more importantly, the tools I present here can help managers increase their ability to assess the efficacy of BMP pollutant and volume removal. Ultimately, paired data collection allows for more statistical descriptive power of how well BMPs are working to achieve their goals of stormwater infiltration and treatment.
Reference
McHugh, M. L. 2011. “Multiple Comparison Analysis Testing in ANOVA.” Biochemia Medica 21(3): 203-09.
