Explore and analyze the continuous chlorophyll data to be included in the NDFS synthesis manuscript. We will attempt to fit multiple models to predict weekly average chlorophyll fluorescence values. All models will only include representative stations for 4 habitat types - upstream (RD22), lower Yolo Bypass (STTD), Cache Slough complex (LIB), and downstream (RVB). This notebook explores running the chlorophyll models for each of the 4 stations separately while using weekly average flow as a continuous predictor. Additionally, we’ll add a GAM smooth for Week number term to account for seasonality in all the models.
The zooplankton analysis used separate models for each station to allow for differing flow lag terms. We need to take a look at this approach to be consistent and see if it makes sense for the chlorophyll analysis. Since STTD, LIB and RVB are all downstream of LIS at different distances, the zooplankton analysis used lagged flow values as a predictor variable - using 1 week lag for STTD, a two week lag for LIB, and a 3 week lag for RVB. We’ll try something similar in this analysis.
# Load packages
library(tidyverse)
library(scales)
library(knitr)
library(mgcv)
library(car)
library(gratia)
library(ggeffects)
library(emmeans)
library(multcomp)
library(patchwork)
library(here)
library(conflicted)
# Declare package conflict preferences
conflicts_prefer(dplyr::filter(), dplyr::lag(), dplyr::select())
Display current versions of R and packages used for this analysis:
devtools::session_info()
## ─ Session info ────────────────────────────────────────────────────────────────────────────────────────────────────────────
## setting value
## version R version 4.6.1 (2026-06-24 ucrt)
## os Windows 11 x64 (build 26200)
## system x86_64, mingw32
## ui RTerm
## language (EN)
## collate English_United States.utf8
## ctype English_United States.utf8
## tz America/Los_Angeles
## date 2026-10-26
## pandoc 3.6.3 @ c:\\Program Files\\Positron\\resources\\app\\quarto\\bin\\tools/ (via rmarkdown)
## quarto 1.9.38 @ C:\\Users\\dboswort\\AppData\\Local\\Programs\\Quarto\\bin\\quarto.exe
##
## ─ Packages ────────────────────────────────────────────────────────────────────────────────────────────────────────────────
## package * version date (UTC) lib source
## abind 1.4-8 2024-09-12 [1] CRAN (R 4.6.1)
## bslib 0.12.0 2026-08-04 [1] CRAN (R 4.6.1)
## cachem 1.1.0 2024-05-16 [1] CRAN (R 4.6.1)
## car * 3.1-5 2026-02-03 [1] CRAN (R 4.6.1)
## carData * 3.0-6 2026-01-30 [1] CRAN (R 4.6.1)
## cli 3.6.6 2026-04-09 [1] CRAN (R 4.6.1)
## codetools 0.2-20 2024-03-31 [1] CRAN (R 4.6.1)
## conflicted * 1.2.0 2023-02-01 [1] CRAN (R 4.6.1)
## devtools 2.5.2 2026-04-30 [1] CRAN (R 4.6.1)
## digest 0.6.39 2025-11-19 [1] CRAN (R 4.6.1)
## dplyr * 1.2.1 2026-04-03 [1] CRAN (R 4.6.1)
## ellipsis 0.3.3 2026-04-04 [1] CRAN (R 4.6.1)
## emmeans * 2.0.4 2026-07-15 [1] CRAN (R 4.6.1)
## estimability 2.0.0 2026-06-26 [1] CRAN (R 4.6.1)
## evaluate 1.0.5 2025-08-27 [1] CRAN (R 4.6.1)
## farver 2.1.2 2024-05-13 [1] CRAN (R 4.6.1)
## fastmap 1.2.0 2024-05-15 [1] CRAN (R 4.6.1)
## forcats * 1.0.1 2025-09-25 [1] CRAN (R 4.6.1)
## Formula 1.2-6 2026-08-03 [1] CRAN (R 4.6.1)
## fs 2.1.0 2026-04-18 [1] CRAN (R 4.6.1)
## generics 0.1.4 2025-05-09 [1] CRAN (R 4.6.1)
## ggeffects * 2.3.2 2025-12-16 [1] CRAN (R 4.6.1)
## ggokabeito 0.1.0 2021-10-18 [1] CRAN (R 4.6.1)
## ggplot2 * 4.0.3 2026-04-22 [1] CRAN (R 4.6.1)
## glue 1.8.1 2026-04-17 [1] CRAN (R 4.6.1)
## gratia * 0.11.2 2026-02-07 [1] CRAN (R 4.6.1)
## gtable 0.3.6 2024-10-25 [1] CRAN (R 4.6.1)
## here * 1.0.2 2025-09-15 [1] CRAN (R 4.6.1)
## hms 1.1.4 2025-10-17 [1] CRAN (R 4.6.1)
## htmltools 0.5.9 2025-12-04 [1] CRAN (R 4.6.1)
## insight 1.5.4 2026-09-05 [1] CRAN (R 4.6.1)
## jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.6.1)
## jsonlite 2.0.0 2025-03-27 [1] CRAN (R 4.6.1)
## knitr * 1.52 2026-09-06 [1] CRAN (R 4.6.1)
## lattice 0.23-1 2026-08-12 [1] CRAN (R 4.6.1)
## lifecycle 1.0.5 2026-01-08 [1] CRAN (R 4.6.1)
## lubridate * 1.9.5 2026-02-04 [1] CRAN (R 4.6.1)
## magrittr 2.0.5 2026-04-04 [1] CRAN (R 4.6.1)
## MASS * 7.3-66 2026-07-15 [1] CRAN (R 4.6.1)
## Matrix 1.7-6 2026-07-25 [1] CRAN (R 4.6.1)
## memoise 2.0.1 2021-11-26 [1] CRAN (R 4.6.1)
## mgcv * 1.9-4 2025-11-07 [1] CRAN (R 4.6.1)
## mirai 2.7.3 2026-09-24 [1] CRAN (R 4.6.1)
## multcomp * 1.4-32 2026-08-21 [1] CRAN (R 4.6.1)
## mvnfast 0.2.8 2023-02-23 [1] CRAN (R 4.6.1)
## mvtnorm * 1.4-2 2026-07-12 [1] CRAN (R 4.6.1)
## nanonext 1.10.3 2026-09-23 [1] CRAN (R 4.6.1)
## nlme * 3.1-171 2026-09-01 [1] CRAN (R 4.6.1)
## otel 0.2.0 2025-08-29 [1] CRAN (R 4.6.1)
## patchwork * 1.3.2 2025-08-25 [1] CRAN (R 4.6.1)
## pillar 1.11.1 2025-09-17 [1] CRAN (R 4.6.1)
## pkgbuild 1.4.8 2025-05-26 [1] CRAN (R 4.6.1)
## pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.6.1)
## pkgload 1.5.3 2026-06-15 [1] CRAN (R 4.6.1)
## purrr * 1.2.2 2026-04-10 [1] CRAN (R 4.6.1)
## R6 2.6.1 2025-02-15 [1] CRAN (R 4.6.1)
## RColorBrewer 1.1-3 2022-04-03 [1] CRAN (R 4.6.1)
## Rcpp 1.1.2 2026-07-05 [1] CRAN (R 4.6.1)
## readr * 2.2.0 2026-02-19 [1] CRAN (R 4.6.1)
## rlang 1.3.0 2026-07-05 [1] CRAN (R 4.6.1)
## rmarkdown 2.32 2026-09-01 [1] CRAN (R 4.6.1)
## rprojroot 2.1.1 2025-08-26 [1] CRAN (R 4.6.1)
## S7 0.2.2 2026-04-22 [1] CRAN (R 4.6.1)
## sandwich 3.1-3 2026-08-03 [1] CRAN (R 4.6.1)
## sass 0.4.10 2025-04-11 [1] CRAN (R 4.6.1)
## scales * 1.4.0 2025-04-24 [1] CRAN (R 4.6.1)
## sessioninfo 1.2.4 2026-06-04 [1] CRAN (R 4.6.1)
## statmod 1.5.2 2026-05-17 [1] CRAN (R 4.6.1)
## stringi 1.8.9 2026-08-04 [1] CRAN (R 4.6.1)
## stringr * 1.6.0 2025-11-04 [1] CRAN (R 4.6.1)
## survival * 3.8-12 2026-09-09 [1] CRAN (R 4.6.1)
## TH.data * 1.1-5 2025-11-17 [1] CRAN (R 4.6.1)
## tibble * 3.3.1 2026-01-11 [1] CRAN (R 4.6.1)
## tidyr * 1.3.2 2025-12-19 [1] CRAN (R 4.6.1)
## tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.6.1)
## tidyverse * 2.0.0 2023-02-22 [1] CRAN (R 4.6.1)
## timechange 0.4.0 2026-01-29 [1] CRAN (R 4.6.1)
## tweedie 3.1.0 2026-05-19 [1] CRAN (R 4.6.1)
## tzdb 0.5.0 2025-03-15 [1] CRAN (R 4.6.1)
## usethis 3.2.2 2026-09-10 [1] CRAN (R 4.6.1)
## vctrs 0.7.3 2026-04-11 [1] CRAN (R 4.6.1)
## withr 3.0.3 2026-06-19 [1] CRAN (R 4.6.1)
## xfun 0.61 2026-09-16 [1] CRAN (R 4.6.1)
## xtable 1.8-8 2026-02-22 [1] CRAN (R 4.6.1)
## yaml 2.3.12 2025-12-10 [1] CRAN (R 4.6.1)
## zoo 1.9-0 2026-07-31 [1] CRAN (R 4.6.1)
##
## [1] C:/R/win-library/4.6
## [2] C:/Program Files/R/R-4.6.1/library
## * ── Packages attached to the search path.
##
## ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Create custom functions used in this notebook:
# Create ggplot from CCF output
plot_ccf_obj <- function(ccf_obj, title) {
# Extract components into a data frame
ccf_df <- tibble(
lag = as.numeric(ccf_obj$lag),
acf = as.numeric(ccf_obj$acf)
)
# Compute exact 95% Confidence Interval bounds based on the sample size used
n_obs <- ccf_obj$n.used
ci_limit <- qnorm((1 + 0.95) / 2) / sqrt(n_obs)
# Build the ggplot object
ccf_df |>
ggplot(aes(x = lag, y = acf)) +
geom_segment(aes(xend = lag, yend = 0), linewidth = 1, color = "darkblue") +
geom_hline(
yintercept = c(ci_limit, -ci_limit),
linetype = "dashed",
color = "red"
) +
geom_hline(yintercept = 0, color = "black") +
labs(
title = title,
subtitle = paste0("95% CI (n = ", n_obs, ")"),
x = "Lag (Weeks)",
y = "Cross-Correlation"
) +
scale_x_continuous(breaks = seq(min(ccf_df$lag), max(ccf_df$lag), by = 1)) +
# scale_y_continuous(limits = c(-1, 1), breaks = breaks_extended(n = 10)) +
theme_bw() +
theme(plot.subtitle = element_text(color = "gray30"))
}
# Create expression for chlorophyll for plots
chla_exp <- expression(
Chlorophyll ~ Fluoresence ~ (mu * g ~ L^{
-1
})
)
# Basic chlorophyll-flow scatter plots
plot_chla_flow <- function(df_data, flow_var, title) {
df_data |>
drop_na(Chla, {{ flow_var }}) |>
ggplot(aes(x = {{ flow_var }}, y = Chla)) +
geom_point() +
geom_smooth(method = "lm", formula = "y ~ x") +
theme_bw() +
labs(title = title, y = chla_exp)
}
# Check general GAM diagnostics
check_gam_diagnostics <- function(m_gam) {
cat("==================================================\n")
cat(" GAM MODEL DIAGNOSTICS \n")
cat("==================================================\n\n")
# Model Summary
cat("--- 1. MODEL SUMMARY ---\n")
print(summary(m_gam))
cat("\n")
# Basis Dimension Check (k.check)
cat("--- 2. BASIS DIMENSION CHECK (k.check) ---\n")
print(mgcv::k.check(m_gam))
cat(
"\n*Note: If p-value is low, the basis dimension 'k' might be too small.*\n\n"
)
# Normality Test of Residuals (Shapiro-Wilk)
cat("--- 3. NORMALITY TEST OF RESIDUALS ---\n")
resids <- residuals(m_gam, type = "deviance") # Default deviance residuals
sw_test <- shapiro.test(resids)
print(sw_test)
if (sw_test$p.value < 0.05) {
cat(
"Result: Reject H0. Residuals are NOT normally distributed (p < 0.05).\n"
)
} else {
cat(
"Result: Fail to reject H0. Residuals appear normally distributed (p >= 0.05).\n"
)
}
cat("\n")
# Generating Gratia Appraise Plots
cat("--- 4. GENERATING APPRAISE PLOTS ---\n")
cat(
"Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.\n"
)
print(gratia::appraise(m_gam))
# Partial Effect Plots (draw and mgcv plot)
cat("--- 5. GENERATING SMOOTH TERM PLOTS ---\n")
# Print the single selected smooth plot with residuals
print(gratia::draw(m_gam, select = 1, residuals = TRUE, rug = FALSE))
# Print the full terms plot grid from base mgcv package
# Save old par settings, force pages = 1 grid layout, then restore par
old_par <- par(no.readonly = TRUE)
plot(m_gam, pages = 1, all.terms = TRUE)
par(old_par)
cat("==================================================\n")
}
# Check GAM for serial autocorrelation
check_gam_autocorr <- function(m_gam, max_lag = 20) {
cat("==================================================\n")
cat(" GAM RESIDUAL AUTOCORRELATION CHECK \n")
cat("==================================================\n\n")
# Extract deviance residuals consistently
resids <- residuals(m_gam, type = "deviance")
# Statistical Test (Ljung-Box)
cat("--- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---\n")
# Use the user-defined max_lag (default is 20)
lb_test <- Box.test(resids, lag = max_lag, type = "Ljung-Box")
print(lb_test)
if (lb_test$p.value < 0.05) {
cat(
"Result: Reject H0. Significant residual autocorrelation detected (p < 0.05).\n"
)
cat(
" Your model might be missing a time/spatial trend or AR structure.\n"
)
} else {
cat(
"Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).\n"
)
}
cat("\n")
# Autocorrelation Function (ACF) Plot
cat("--- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---\n")
acf(resids, main = paste0("ACF of Residuals (Max Lag = ", max_lag, ")"))
cat("==================================================\n")
}
# Plot observed chlorophyll values vs fitted values from model
# Fitted values are unscaled and back-transformed to match the observed values
plot_obs_vs_model_fit <- function(m_gam, df_data) {
df_data_fit <-
fitted_values(m_gam, data = df_data) |>
mutate(fitted_bt = exp(.fitted) / 1000)
df_data_fit |>
ggplot(aes(x = fitted_bt, y = Chla)) +
geom_point() +
geom_abline(slope = 1, intercept = 0, color = "red") +
theme_bw() +
labs(
x = "Back-transformed Fitted Values",
y = "Observed Values"
)
}
# Calculate effects of flow on chlorophyll for each year holding the non-focal variables
# constant - marginal effects/adjusted predictions
calc_flow_yr_marg_eff <- function(m_gam, df_data, flow_var) {
# Calculate min and max flows for each year to narrow down x-axis in the effects plots
df_flow_yr_summ <- df_data |>
summarize(
min_flow = floor(min(.data[[flow_var]])),
max_flow = ceiling(max(.data[[flow_var]])),
.by = Year_fct
)
# Calculate marginal effects
as.data.frame(
predict_response(
m_gam,
terms = c(flow_var, "Year_fct"),
margin = "marginalmeans"
),
terms_to_colnames = TRUE
) |>
as_tibble() |>
# Narrow down range of flow values for each station
left_join(df_flow_yr_summ, by = join_by(Year_fct)) |>
filter(between(.data[[flow_var]], min_flow, max_flow)) |>
mutate(
Year_fct,
.data[[flow_var]],
# Back calculate model fits and confidence levels
across(c(predicted, conf.low, conf.high), \(x) exp(x) / 1000),
.keep = "none"
)
}
# Create effects plot of flow on chlorophyll by year
plot_eff_flow_by_yr <- function(df_eff, df_data, flow_var) {
df_eff |>
ggplot(aes(x = .data[[flow_var]], y = predicted)) +
geom_point(data = df_data, aes(y = Chla), alpha = 0.5) +
geom_line(linewidth = 1) +
geom_ribbon(
aes(ymin = conf.low, ymax = conf.high),
alpha = 0.15,
color = NA
) +
facet_wrap(vars(Year_fct), scales = "free_x") +
theme_bw() +
labs(x = "Flow (cfs)", y = chla_exp) +
scale_x_continuous(breaks = breaks_extended(6))
}
# Create boxplots of chlorophyll by year showing Tukey post-hoc results
boxplot_cont_yr <- function(em_obj, df_data) {
# Create table of contrasts and convert it to a tibble for plot
df_yr_cont <- em_obj |>
cld(sort = FALSE, Letters = letters) |>
as_tibble() |>
mutate(
group = str_trim(.group),
# back transform log-transformed results
across(c(emmean, lower.CL, upper.CL), ~ exp(.x) / 1000)
) |>
# Add min and max values of observed data to the Tukey post-hoc results and
# calculate vertical positioning of letters
left_join(
df_data |>
summarize(
max_val = max(Chla),
min_val = min(Chla),
.by = Year_fct
),
by = join_by(Year_fct)
) |>
mutate(
max_val = if_else(upper.CL > max_val, upper.CL, max_val),
y_pos = max_val + (max_val - min_val) / 10,
# Make all post-hoc contrast letters at same height equal to max of all
y_pos = max(y_pos)
) |>
select(
Year_fct,
emmean,
lower.CL,
upper.CL,
group,
y_pos
)
# Create boxplot showing Tukey post-hoc results
df_yr_cont |>
ggplot(aes(x = Year_fct, y = emmean, ymin = lower.CL, ymax = upper.CL)) +
geom_boxplot(
data = df_data,
aes(x = Year_fct, y = Chla),
inherit.aes = FALSE
) +
geom_crossbar(
color = "grey82",
fill = "grey",
alpha = 0.7,
linewidth = 0.1
) +
geom_point(color = "red") +
geom_text(aes(y = y_pos, label = group), size = 3) +
labs(x = "Year", y = chla_exp) +
theme_bw()
}
# Define file path for processed data
fp_data <- here("manuscript_synthesis/data/processed")
# Import weekly average water quality data
df_wq <- readRDS(file.path(fp_data, "wq_week_avg_2013-2019.rds"))
# Import weekly average flow data
df_flow <- readRDS(file.path(fp_data, "flow_week_avg_2013-2019.rds"))
# Create a vector for the factor order of StationCode
sta_order <- c("RD22", "STTD", "LIB", "RVB")
# We will use LIS flow data for all stations except for RD22
df_flow_c <- bind_rows(
expand_grid(
StationCode = c("STTD", "LIB", "RVB"),
df_flow |> filter(StationCode == "LIS") |> select(-StationCode)
),
df_flow |> filter(StationCode == "RD22")
)
# Prepare chlorophyll and flow data for exploration and analysis
df_chla_c1 <- df_wq |>
select(StationCode, Year, Week, Chla, SpCnd) |>
# Filter to only include representative stations for 4 habitat types - RD22, STTD, LIB, RVB
filter(StationCode %in% sta_order) |>
# Fill in missing weeks for each StationCode-Year combination
group_by(StationCode, Year) |>
complete(Week = seq.int(min(Week), max(Week), by = 1)) |>
ungroup() |>
# Join flow data to chlorophyll data
left_join(df_flow_c, by = join_by(StationCode, Year, Week)) |>
mutate(
# Scale and log transform chlorophyll values
Chla_log = log(Chla * 1000),
# Apply factor order to StationCode
StationCode = factor(StationCode, levels = sta_order),
# Add a column for Year as a factor for the model
Year_fct = factor(Year)
) |>
arrange(StationCode, Year, Week)
df_chla_c1 |>
drop_na(Chla, Flow) |>
summarize(
min_week = min(Week),
max_week = max(Week),
num_samples = n(),
.by = c(StationCode, Year)
) |>
arrange(StationCode, Year) |>
kable()
| StationCode | Year | min_week | max_week | num_samples |
|---|---|---|---|---|
| RD22 | 2014 | 39 | 45 | 7 |
| RD22 | 2015 | 30 | 45 | 16 |
| RD22 | 2016 | 25 | 38 | 14 |
| RD22 | 2017 | 28 | 44 | 17 |
| RD22 | 2018 | 28 | 45 | 18 |
| RD22 | 2019 | 28 | 45 | 18 |
| STTD | 2013 | 33 | 44 | 12 |
| STTD | 2014 | 30 | 45 | 14 |
| STTD | 2015 | 30 | 46 | 17 |
| STTD | 2016 | 25 | 38 | 14 |
| STTD | 2017 | 28 | 39 | 10 |
| STTD | 2018 | 29 | 42 | 14 |
| STTD | 2019 | 30 | 45 | 16 |
| LIB | 2013 | 29 | 46 | 18 |
| LIB | 2014 | 30 | 45 | 16 |
| LIB | 2015 | 27 | 46 | 20 |
| LIB | 2016 | 22 | 38 | 17 |
| LIB | 2017 | 28 | 44 | 17 |
| LIB | 2018 | 33 | 45 | 13 |
| LIB | 2019 | 28 | 45 | 14 |
| RVB | 2013 | 27 | 46 | 20 |
| RVB | 2014 | 30 | 45 | 16 |
| RVB | 2015 | 27 | 46 | 20 |
| RVB | 2016 | 22 | 38 | 16 |
| RVB | 2017 | 28 | 44 | 17 |
| RVB | 2018 | 28 | 45 | 18 |
| RVB | 2019 | 28 | 45 | 18 |
Looking at the sample counts and date ranges, we’ll only include years 2015-2019 for the analysis.
df_chla_c2 <- df_chla_c1 |>
filter(Year %in% 2015:2019) |>
mutate(Year_fct = fct_drop(Year_fct))
We’ll create another dataframe that has up to 2 lag variables for chlorophyll to be used in the models to help with serial autocorrelation.
df_chla_c2_lag <- df_chla_c2 |>
# Fill in missing weeks for each StationCode-Year combination
group_by(StationCode, Year) |>
# Create lag variables of scaled log transformed chlorophyll values
mutate(
lag1 = lag(Chla_log),
lag2 = lag(Chla_log, 2)
) |>
ungroup()
# Split up data by station to allow for models and plots to be created separately
ls_chla <- df_chla_c2_lag |>
nest(.by = StationCode, .key = "df_data") |>
deframe()
As stated above, the zooplankton analysis used lagged flow values as a predictor variable - using 1 week lag for STTD, a two week lag for LIB, and a 3 week lag for RVB. Let’s run a Cross-Correlation Function (CCF) analysis between the upstream LIS flow and chlorophyll values for STTD, LIB and RVB to provide empirical backing for the flow lag terms to use in each model. We’ll also run a CCF analysis between upstream LIS flow and specific conductance (SpCnd) values. Specific conductance is a more conservative measurement than chlorophyll and isn’t the response variable in the models, so it’s good to include it as an additional check. The daily average time-series plots of specific conductance (SpCnd) show a consistent increase in SpCnd during flow action periods indicating downstream transport of higher SpCnd water during flow pulses especially for STTD and less so for LIB and RVB.
ls_ccf_data <- ls_chla |> discard_at("RD22")
ls_ccf_plt_chla_allyrs <- ls_ccf_data |>
map(\(x) drop_na(x, Chla_log)) |>
map(\(x) ccf(x$Flow, x$Chla_log, lag.max = 5, plot = FALSE)) |>
imap(\(x, idx) {
plot_ccf_obj(x, paste0("Flow vs Chla CCF - ", idx, ", All Years"))
})
ls_ccf_plt_spcnd_allyrs <- ls_ccf_data |>
map(\(x) ccf(x$Flow, x$SpCnd, lag.max = 5, plot = FALSE)) |>
imap(\(x, idx) {
plot_ccf_obj(x, paste0("Flow vs SpCnd CCF - ", idx, ", All Years"))
})
ls_ccf_plt_chla_indiv_yrs <- ls_ccf_data |>
map(\(x) drop_na(x, Chla_log)) |>
map(\(x) nest(x, .by = Year_fct, .key = "df_data")) |>
map(
\(x) {
mutate(
x,
ccf_obj = map(
df_data,
\(df) ccf(df$Flow, df$Chla_log, lag.max = 5, plot = FALSE)
),
plt_ccf = map2(ccf_obj, Year_fct, plot_ccf_obj)
) |>
pull(plt_ccf)
}
) |>
imap(\(x, idx) {
wrap_plots(x, ncol = 3, axis_titles = "collect") +
plot_annotation(
title = paste0("Flow vs Chla CCF - ", idx, ", Individual Years")
)
})
ls_ccf_plt_spcnd_indiv_yrs <- ls_ccf_data |>
map(\(x) nest(x, .by = Year_fct, .key = "df_data")) |>
map(
\(x) {
mutate(
x,
ccf_obj = map(
df_data,
\(df) ccf(df$Flow, df$SpCnd, lag.max = 5, plot = FALSE)
),
plt_ccf = map2(ccf_obj, Year_fct, plot_ccf_obj)
) |>
pull(plt_ccf)
}
) |>
imap(\(x, idx) {
wrap_plots(x, ncol = 3, axis_titles = "collect") +
plot_annotation(
title = paste0("Flow vs SpCnd CCF - ", idx, ", Individual Years")
)
})
ls_ccf_plt_chla_allyrs$STTD
ls_ccf_plt_chla_indiv_yrs$STTD
The chlorophyll plots show a consistent positive peak in the cross-correlation value at 0-lag, which indicates that a non-lagged LIS flow term for STTD is the best choice.
ls_ccf_plt_spcnd_allyrs$STTD
ls_ccf_plt_spcnd_indiv_yrs$STTD
For specific conductance, the positive peak in the cross-correlation values are located at the -1 and 0 lag terms, which indicates that there is a slight lagged response of specific conductance values to increased LIS flow. This is probably because the lagged response is a few days, not a full week. With all of this information, I think we should use a non-lagged LIS flow term for STTD in the models. We’ll explore this further with some scatter plots of chlorophyll and flow comparing non-lagged flow with the lagged flow values.
plt_chla_flow_sttd <- ls_chla$STTD |>
drop_na(Chla) |>
plot_chla_flow(Flow, "No Flow Lag") +
facet_wrap(vars(Year), nrow = 1, scales = "free")
plt_chla_flow_sttd_lag1 <- ls_chla$STTD |>
mutate(Flow_lag1 = lag(Flow)) |>
drop_na(Chla, Flow_lag1) |>
plot_chla_flow(Flow_lag1, "1-Week Flow Lag") +
facet_wrap(vars(Year), nrow = 1, scales = "free")
plt_chla_flow_sttd /
plt_chla_flow_sttd_lag1 +
plot_layout(axis_titles = "collect") +
plot_annotation(title = "STTD")
The 1-week lag flow appears to have a weaker relationship with chlorophyll than non-lagged flow which confirms using the non-lagged LIS flow for the STTD model.
ls_ccf_plt_chla_allyrs$LIB
ls_ccf_plt_chla_indiv_yrs$LIB
The chlorophyll plots are less clear and harder to interpret than those for STTD. It looks like there is a 1-2 week lag between increased flows at LIS and increased chlorophyll values at LIB in 2015; however, there seems to be a flushing effect (decreased chlorophyll values a few weeks after increased flows at LIS) in 2019, and a initial increase in chlorophyll values followed by a possible flushing effect 4 to 5 weeks later in 2018.
ls_ccf_plt_spcnd_allyrs$LIB
ls_ccf_plt_spcnd_indiv_yrs$LIB
The specific conductance plots seem to be a little more clear. For all years, there is a 1-2 week lag between increased flows at LIS and increased specific conductance values at LIB. Looking at the individual years, the lag is between 0 and 2 weeks for most years.
ls_ccf_plt_chla_allyrs$RVB
ls_ccf_plt_chla_indiv_yrs$RVB
RVB shows an even stronger flushing effect (decreased chlorophyll values 2-4 weeks after increased flows at LIS) overall and for 2015, 2018, and 2019.
ls_ccf_plt_spcnd_allyrs$RVB
ls_ccf_plt_spcnd_indiv_yrs$RVB
For specific conductance, there is a 2-3 week lag between increased flows at LIS and increased specific conductance values at RVB for all years combined; however, this trend isn’t apparent across all of the individual year plots.
In summary, I think we should use an non-lagged LIS flow term for STTD, a two-week flow lag for LIB, and a 3-week flow lag for RVB. For LIB and RVB these lag values are consistent with the hydrodynamic modeling.
# Add lagged flow values for LIB and RVB
ls_chla_flow_lag <- ls_chla |>
map(\(x) group_by(x, Year)) |>
map_at("LIB", \(x) mutate(x, Flow_lag2 = lag(Flow, n = 2))) |>
map_at("RVB", \(x) mutate(x, Flow_lag3 = lag(Flow, n = 3))) |>
map(\(x) ungroup(x))
Let’s explore the data with some plots. We’ll plot the data in scatter plots of chlorophyll and flow for each Station, using the lagged flow values for LIB and RVB.
ls_chla_flow_plt <- ls_chla_flow_lag |>
imap(
\(x, idx) {
if (idx %in% c("RD22", "STTD")) {
plot_chla_flow(x, Flow, idx)
} else if (idx == "LIB") {
plot_chla_flow(x, Flow_lag2, idx)
} else if (idx == "RVB") {
plot_chla_flow(x, Flow_lag3, idx)
}
}
)
wrap_plots(ls_chla_flow_plt, ncol = 2, axis_titles = "collect_y")
Let’s break these scatterplots apart by year to see how these patterns vary annually.
ls_chla_flow_yr_plt <- ls_chla_flow_plt |>
map(\(x) x + facet_wrap(vars(Year), nrow = 1, scales = "free"))
wrap_plots(ls_chla_flow_yr_plt, ncol = 1, axis_titles = "collect_y")
We will attempt to fit generalized additive models (GAM’s) to each station’s data set to help account for seasonality in the data. We’ll try running each GAM using a two-way interaction between Year and Weekly Average Flow, and a cyclic penalized cubic regression spline smooth term for week number to account for seasonality (restricting the k-value to 5 to reduce overfitting). Initially, we’ll run the GAM’s without accounting for serial autocorrelation.
df_chla_rd22 <- ls_chla_flow_lag$RD22 |> drop_na(Flow, Chla_log)
m_gam_rd22 <- gam(
Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5),
data = df_chla_rd22,
method = "REML",
knots = list(Week = c(0, 52))
)
Lets look at the model summary and diagnostics:
check_gam_diagnostics(m_gam_rd22)
## ==================================================
## GAM MODEL DIAGNOSTICS
## ==================================================
##
## --- 1. MODEL SUMMARY ---
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5)
##
## Parametric coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 9.7354276 0.1157428 84.113 < 2e-16 ***
## Year_fct2016 -0.8259986 0.1708360 -4.835 7.51e-06 ***
## Year_fct2017 -0.9533764 0.2637595 -3.615 0.00056 ***
## Year_fct2018 -0.4106405 0.1514147 -2.712 0.00839 **
## Year_fct2019 -0.1324495 0.1488443 -0.890 0.37656
## Flow -0.0016110 0.0005249 -3.069 0.00304 **
## Year_fct2016:Flow 0.0018560 0.0008289 2.239 0.02829 *
## Year_fct2017:Flow 0.0325829 0.0141204 2.308 0.02396 *
## Year_fct2018:Flow 0.0005245 0.0006934 0.756 0.45192
## Year_fct2019:Flow -0.0001885 0.0006129 -0.308 0.75930
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 2.3 3 7.789 3.41e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## R-sq.(adj) = 0.581 Deviance explained = 63.8%
## -REML = 67.339 Scale est. = 0.12296 n = 83
##
## --- 2. BASIS DIMENSION CHECK (k.check) ---
## k' edf k-index p-value
## s(Week) 3 2.300173 0.9255603 0.21
##
## *Note: If p-value is low, the basis dimension 'k' might be too small.*
##
## --- 3. NORMALITY TEST OF RESIDUALS ---
##
## Shapiro-Wilk normality test
##
## data: resids
## W = 0.98104, p-value = 0.2594
##
## Result: Fail to reject H0. Residuals appear normally distributed (p >= 0.05).
##
## --- 4. GENERATING APPRAISE PLOTS ---
## Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.
## --- 5. GENERATING SMOOTH TERM PLOTS ---
## ==================================================
check_gam_autocorr(m_gam_rd22)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 27.174, df = 20, p-value = 0.1304
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
The diagnostic plots and stats look good. However, the residuals are autocorrelated.
Now, we’ll try to deal with the residual autocorrelation. We’ll run a series of models adding 1 and 2 lag terms and compare how well they correct for autocorrelation.
m_gam_rd22_lag1 <- gam(
Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5) + lag1,
data = df_chla_rd22,
method = "REML",
knots = list(Week = c(0, 52))
)
check_gam_autocorr(m_gam_rd22_lag1)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 21.281, df = 20, p-value = 0.3808
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
m_gam_rd22_lag2 <- gam(
Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5) + lag1 + lag2,
data = df_chla_rd22,
method = "REML",
knots = list(Week = c(0, 52))
)
check_gam_autocorr(m_gam_rd22_lag2)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 20.191, df = 20, p-value = 0.4461
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
The model with 2 lag terms seems to be okay with autocorrelation according to the acf plot. Let’s take a closer look at that one.
check_gam_diagnostics(m_gam_rd22_lag2)
## ==================================================
## GAM MODEL DIAGNOSTICS
## ==================================================
##
## --- 1. MODEL SUMMARY ---
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5) + lag1 +
## lag2
##
## Parametric coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.5399987 1.1482450 6.567 1.48e-08 ***
## Year_fct2016 -0.5894910 0.1923485 -3.065 0.00329 **
## Year_fct2017 -0.7554201 0.2681538 -2.817 0.00659 **
## Year_fct2018 -0.3341099 0.1518633 -2.200 0.03174 *
## Year_fct2019 -0.0696332 0.1447652 -0.481 0.63230
## Flow -0.0009265 0.0005115 -1.811 0.07519 .
## lag1 0.4566559 0.1321649 3.455 0.00103 **
## lag2 -0.2389602 0.1122388 -2.129 0.03745 *
## Year_fct2016:Flow 0.0007857 0.0007934 0.990 0.32605
## Year_fct2017:Flow 0.0311749 0.0155476 2.005 0.04956 *
## Year_fct2018:Flow 0.0004431 0.0006337 0.699 0.48714
## Year_fct2019:Flow -0.0001853 0.0005710 -0.325 0.74665
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 2.172 3 5.151 8e-04 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## R-sq.(adj) = 0.694 Deviance explained = 75%
## -REML = 56.205 Scale est. = 0.094491 n = 73
##
## --- 2. BASIS DIMENSION CHECK (k.check) ---
## k' edf k-index p-value
## s(Week) 3 2.17159 0.8836857 0.175
##
## *Note: If p-value is low, the basis dimension 'k' might be too small.*
##
## --- 3. NORMALITY TEST OF RESIDUALS ---
##
## Shapiro-Wilk normality test
##
## data: resids
## W = 0.98307, p-value = 0.4349
##
## Result: Fail to reject H0. Residuals appear normally distributed (p >= 0.05).
##
## --- 4. GENERATING APPRAISE PLOTS ---
## Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.
## --- 5. GENERATING SMOOTH TERM PLOTS ---
## ==================================================
anova(m_gam_rd22_lag2)
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5) + lag1 +
## lag2
##
## Parametric Terms:
## df F p-value
## Year_fct 4 4.118 0.00522
## Flow 1 3.281 0.07519
## lag1 1 11.938 0.00103
## lag2 1 4.533 0.03745
## Year_fct:Flow 4 1.513 0.20997
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 2.172 3.000 5.151 8e-04
The model diagnostics look really good. Before we proceed with this model for RD22, let’s take a closer look at how the back-transformed fitted values from the model match the observed values.
df_chla_rd22_lag <- df_chla_rd22 |> drop_na(lag1, lag2)
# Sample sizes for each year
df_chla_rd22_lag |> count(Year_fct)
## # A tibble: 5 Ă— 2
## Year_fct n
## <fct> <int>
## 1 2015 14
## 2 2016 12
## 3 2017 15
## 4 2018 16
## 5 2019 16
plot_obs_vs_model_fit(m_gam_rd22_lag2, df_chla_rd22_lag)
The model fit looks pretty decent. Note that variability does increase as the chlorophyll values increase. Now, let’s look more closely at its results.
Partial effect of flow on chlorophyll:
# Calculate effects of flow on chlorophyll holding the non-focal variables constant -
# marginal effects/adjusted predictions
df_rd22_flow_eff <- calc_flow_yr_marg_eff(
m_gam_rd22_lag2,
df_chla_rd22_lag,
"Flow"
)
# Create effects plot
plot_eff_flow_by_yr(df_rd22_flow_eff, df_chla_rd22_lag, "Flow")
# Estimated marginal means for year
em_rd22_yr <- emmeans(m_gam_rd22_lag2, ~Year_fct)
# Tukey post-hoc contrasts
pairs(em_rd22_yr)
## contrast estimate SE df t.ratio p.value
## Year_fct2015 - Year_fct2016 0.4981 0.156 58.8 3.201 0.0181
## Year_fct2015 - Year_fct2017 -2.8695 1.590 58.8 -1.809 0.3781
## Year_fct2015 - Year_fct2018 0.2826 0.123 58.8 2.298 0.1602
## Year_fct2015 - Year_fct2019 0.0912 0.118 58.8 0.773 0.9374
## Year_fct2016 - Year_fct2017 -3.3676 1.580 58.8 -2.136 0.2188
## Year_fct2016 - Year_fct2018 -0.2155 0.130 58.8 -1.652 0.4713
## Year_fct2016 - Year_fct2019 -0.4069 0.138 58.8 -2.939 0.0365
## Year_fct2017 - Year_fct2018 3.1521 1.590 58.8 1.984 0.2864
## Year_fct2017 - Year_fct2019 2.9607 1.580 58.8 1.870 0.3444
## Year_fct2018 - Year_fct2019 -0.1914 0.112 58.8 -1.707 0.4378
##
## P value adjustment: tukey method for comparing a family of 5 estimates
# Create boxplot showing Tukey post-hoc results
boxplot_cont_yr(em_rd22_yr, df_chla_rd22_lag)
The model seems way off for 2017. This is most likely because emmeans is using a Flow value outside of the range of flow values that occurred during 2017. I think this model is good for looking at the effects of flow by year, but not for differences between years.
df_chla_sttd <- ls_chla_flow_lag$STTD |> drop_na(Flow, Chla_log)
m_gam_sttd <- gam(
Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5),
data = df_chla_sttd,
method = "REML",
knots = list(Week = c(0, 52))
)
Lets look at the model summary and diagnostics:
check_gam_diagnostics(m_gam_sttd)
## ==================================================
## GAM MODEL DIAGNOSTICS
## ==================================================
##
## --- 1. MODEL SUMMARY ---
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5)
##
## Parametric coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 8.6132917 0.0666283 129.274 < 2e-16 ***
## Year_fct2016 0.4696569 0.1004782 4.674 1.68e-05 ***
## Year_fct2017 0.4404389 0.1706566 2.581 0.01228 *
## Year_fct2018 -0.6585791 0.0997395 -6.603 1.11e-08 ***
## Year_fct2019 -1.0636400 0.0985773 -10.790 8.98e-16 ***
## Flow 0.0027619 0.0004435 6.227 4.83e-08 ***
## Year_fct2016:Flow -0.0015045 0.0006544 -2.299 0.02495 *
## Year_fct2017:Flow 0.0145144 0.0051610 2.812 0.00661 **
## Year_fct2018:Flow -0.0002740 0.0005775 -0.474 0.63689
## Year_fct2019:Flow -0.0013625 0.0005152 -2.644 0.01039 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 8.792e-05 3 0 0.562
##
## R-sq.(adj) = 0.846 Deviance explained = 86.6%
## -REML = 43.239 Scale est. = 0.072301 n = 71
##
## --- 2. BASIS DIMENSION CHECK (k.check) ---
## k' edf k-index p-value
## s(Week) 3 8.79212e-05 0.8037782 0.0475
##
## *Note: If p-value is low, the basis dimension 'k' might be too small.*
##
## --- 3. NORMALITY TEST OF RESIDUALS ---
##
## Shapiro-Wilk normality test
##
## data: resids
## W = 0.97594, p-value = 0.1896
##
## Result: Fail to reject H0. Residuals appear normally distributed (p >= 0.05).
##
## --- 4. GENERATING APPRAISE PLOTS ---
## Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.
## --- 5. GENERATING SMOOTH TERM PLOTS ---
## ==================================================
check_gam_autocorr(m_gam_sttd)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 17.335, df = 20, p-value = 0.6311
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
Shapiro-Wilk normality test shows that the residuals are normal, the diagnostic plots look really good. The ACF plot and the Box-Ljung test indicate no serial autocorrelation. Note that the smooth term is not significant and it’s essentially a flat line. Let’s look at the ANOVA results for the model.
anova(m_gam_sttd)
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow + s(Week, bs = "cc", k = 5)
##
## Parametric Terms:
## df F p-value
## Year_fct 4 70.083 < 2e-16
## Flow 1 38.779 4.83e-08
## Year_fct:Flow 4 5.163 0.0012
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 8.792e-05 3.000e+00 0 0.562
All parametric terms are significant in this model. Let’s take a closer look at how the back-transformed fitted values from the model match the observed values.
# Sample sizes for each year
df_chla_sttd |> count(Year_fct)
## # A tibble: 5 Ă— 2
## Year_fct n
## <fct> <int>
## 1 2015 17
## 2 2016 14
## 3 2017 10
## 4 2018 14
## 5 2019 16
plot_obs_vs_model_fit(m_gam_sttd, df_chla_sttd)
The model fit looks pretty decent. Note that variability does increase as the chlorophyll values increase. Now, let’s look more closely at its results.
Partial effect of flow on chlorophyll:
# Calculate effects of flow on chlorophyll holding the non-focal variables constant -
# marginal effects/adjusted predictions
df_sttd_flow_eff <- calc_flow_yr_marg_eff(m_gam_sttd, df_chla_sttd, "Flow")
# Create effects plot
plot_eff_flow_by_yr(df_sttd_flow_eff, df_chla_sttd, "Flow")
# Estimated marginal means for year
em_sttd_yr <- emmeans(m_gam_sttd, ~Year_fct)
# Tukey post-hoc contrasts
pairs(em_sttd_yr)
## contrast estimate SE df t.ratio p.value
## Year_fct2015 - Year_fct2016 -0.400 0.0973 61 -4.110 0.0011
## Year_fct2015 - Year_fct2017 -1.114 0.3860 61 -2.886 0.0414
## Year_fct2015 - Year_fct2018 0.671 0.0973 61 6.899 <0.0001
## Year_fct2015 - Year_fct2019 1.127 0.0952 61 11.841 <0.0001
## Year_fct2016 - Year_fct2017 -0.714 0.3870 61 -1.845 0.3578
## Year_fct2016 - Year_fct2018 1.071 0.1020 61 10.538 <0.0001
## Year_fct2016 - Year_fct2019 1.527 0.0996 61 15.328 <0.0001
## Year_fct2017 - Year_fct2018 1.785 0.3870 61 4.612 0.0002
## Year_fct2017 - Year_fct2019 2.241 0.3870 61 5.797 <0.0001
## Year_fct2018 - Year_fct2019 0.456 0.0996 61 4.574 0.0002
##
## P value adjustment: tukey method for comparing a family of 5 estimates
# Create boxplot showing Tukey post-hoc results
boxplot_cont_yr(em_sttd_yr, df_chla_sttd)
Again, the model seems off for 2017. This is most likely because emmeans is using a Flow value outside of the range of flow values that occurred during 2017. I think this model is good for looking at the effects of flow by year, but not for differences between years.
df_chla_lib <- ls_chla_flow_lag$LIB |>
select(-Flow) |>
drop_na(Flow_lag2, Chla_log)
m_gam_lib <- gam(
Chla_log ~ Year_fct * Flow_lag2 + s(Week, bs = "cc", k = 5),
data = df_chla_lib,
method = "REML",
knots = list(Week = c(0, 52))
)
Lets look at the model summary and diagnostics:
check_gam_diagnostics(m_gam_lib)
## ==================================================
## GAM MODEL DIAGNOSTICS
## ==================================================
##
## --- 1. MODEL SUMMARY ---
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow_lag2 + s(Week, bs = "cc", k = 5)
##
## Parametric coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.6658256 0.0942857 81.304 < 2e-16 ***
## Year_fct2016 0.6452778 0.1549420 4.165 0.000100 ***
## Year_fct2017 0.0384005 0.1587787 0.242 0.809711
## Year_fct2018 -1.8838565 0.1503589 -12.529 < 2e-16 ***
## Year_fct2019 -0.5873082 0.1625333 -3.613 0.000614 ***
## Flow_lag2 0.0020176 0.0006761 2.984 0.004091 **
## Year_fct2016:Flow_lag2 -0.0016994 0.0009587 -1.773 0.081315 .
## Year_fct2017:Flow_lag2 -0.0029788 0.0030084 -0.990 0.326019
## Year_fct2018:Flow_lag2 -0.0005373 0.0008587 -0.626 0.533858
## Year_fct2019:Flow_lag2 -0.0020531 0.0007693 -2.669 0.009752 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 2.237 3 12.15 6.43e-07 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## R-sq.(adj) = 0.867 Deviance explained = 88.7%
## -REML = 71.522 Scale est. = 0.15383 n = 73
##
## --- 2. BASIS DIMENSION CHECK (k.check) ---
## k' edf k-index p-value
## s(Week) 3 2.236972 1.093003 0.735
##
## *Note: If p-value is low, the basis dimension 'k' might be too small.*
##
## --- 3. NORMALITY TEST OF RESIDUALS ---
##
## Shapiro-Wilk normality test
##
## data: resids
## W = 0.97299, p-value = 0.1185
##
## Result: Fail to reject H0. Residuals appear normally distributed (p >= 0.05).
##
## --- 4. GENERATING APPRAISE PLOTS ---
## Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.
## --- 5. GENERATING SMOOTH TERM PLOTS ---
## ==================================================
check_gam_autocorr(m_gam_lib)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 61.921, df = 20, p-value = 3.577e-06
##
## Result: Reject H0. Significant residual autocorrelation detected (p < 0.05).
## Your model might be missing a time/spatial trend or AR structure.
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
The diagnostic plots and stats look good. However, the residuals are autocorrelated.
Now, we’ll try to deal with the residual autocorrelation. We’ll run a series of models adding 1 and 2 lag terms and compare how well they correct for autocorrelation.
m_gam_lib_lag1 <- gam(
Chla_log ~ Year_fct * Flow_lag2 + s(Week, bs = "cc", k = 5) + lag1,
data = df_chla_lib,
method = "REML",
knots = list(Week = c(0, 52))
)
check_gam_autocorr(m_gam_lib_lag1)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 18.323, df = 20, p-value = 0.5661
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
m_gam_lib_lag2 <- gam(
Chla_log ~ Year_fct * Flow_lag2 + s(Week, bs = "cc", k = 5) + lag1 + lag2,
data = df_chla_lib,
method = "REML",
knots = list(Week = c(0, 52))
)
check_gam_autocorr(m_gam_lib_lag2)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 14.067, df = 20, p-value = 0.8271
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
The model with 1 lag term seems to be okay with autocorrelation according to the acf plot. Adding a second lag term seems unnecessary. Let’s take a closer look at the model with 1 lag term.
check_gam_diagnostics(m_gam_lib_lag1)
## ==================================================
## GAM MODEL DIAGNOSTICS
## ==================================================
##
## --- 1. MODEL SUMMARY ---
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow_lag2 + s(Week, bs = "cc", k = 5) +
## lag1
##
## Parametric coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.8585321 0.9062687 3.154 0.00254 **
## Year_fct2016 0.2273304 0.1603064 1.418 0.16147
## Year_fct2017 0.0046297 0.1381081 0.034 0.97337
## Year_fct2018 -0.6863587 0.2719895 -2.523 0.01436 *
## Year_fct2019 -0.2074296 0.1644704 -1.261 0.21225
## Flow_lag2 0.0007543 0.0006048 1.247 0.21724
## lag1 0.6255969 0.1173114 5.333 1.63e-06 ***
## Year_fct2016:Flow_lag2 -0.0008179 0.0008387 -0.975 0.33348
## Year_fct2017:Flow_lag2 -0.0010118 0.0026491 -0.382 0.70389
## Year_fct2018:Flow_lag2 -0.0008527 0.0007669 -1.112 0.27072
## Year_fct2019:Flow_lag2 -0.0008209 0.0007028 -1.168 0.24750
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 1.513 3 1.74 0.0352 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## R-sq.(adj) = 0.898 Deviance explained = 91.5%
## -REML = 61.32 Scale est. = 0.11801 n = 71
##
## --- 2. BASIS DIMENSION CHECK (k.check) ---
## k' edf k-index p-value
## s(Week) 3 1.512889 1.031146 0.555
##
## *Note: If p-value is low, the basis dimension 'k' might be too small.*
##
## --- 3. NORMALITY TEST OF RESIDUALS ---
##
## Shapiro-Wilk normality test
##
## data: resids
## W = 0.94609, p-value = 0.004201
##
## Result: Reject H0. Residuals are NOT normally distributed (p < 0.05).
##
## --- 4. GENERATING APPRAISE PLOTS ---
## Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.
## --- 5. GENERATING SMOOTH TERM PLOTS ---
## ==================================================
anova(m_gam_lib_lag1)
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow_lag2 + s(Week, bs = "cc", k = 5) +
## lag1
##
## Parametric Terms:
## df F p-value
## Year_fct 4 1.694 0.164
## Flow_lag2 1 1.556 0.217
## lag1 1 28.439 1.63e-06
## Year_fct:Flow_lag2 4 0.437 0.782
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 1.513 3.000 1.74 0.0352
The model diagnostics look a litle worse than the model without lag terms, but it’s corrected for autocorrelation. Before we proceed with this model for LIB, let’s take a closer look at how the back-transformed fitted values from the model match the observed values.
df_chla_lib_lag <- df_chla_lib |> drop_na(lag1)
# Sample sizes for each year
df_chla_lib_lag |> count(Year_fct)
## # A tibble: 5 Ă— 2
## Year_fct n
## <fct> <int>
## 1 2015 18
## 2 2016 15
## 3 2017 15
## 4 2018 12
## 5 2019 11
plot_obs_vs_model_fit(m_gam_lib_lag1, df_chla_lib_lag)
The model fit looks pretty decent. Note that variability does increase as the chlorophyll values increase. Now, let’s look more closely at its results.
Partial effect of flow on chlorophyll:
# Calculate effects of flow on chlorophyll holding the non-focal variables constant -
# marginal effects/adjusted predictions
df_lib_flow_eff <- calc_flow_yr_marg_eff(
m_gam_lib_lag1,
df_chla_lib_lag,
"Flow_lag2"
)
# Create effects plot
plot_eff_flow_by_yr(df_lib_flow_eff, df_chla_lib_lag, "Flow_lag2")
# Estimated marginal means for year
em_lib_yr <- emmeans(m_gam_lib_lag1, ~Year_fct)
# Tukey post-hoc contrasts
pairs(em_lib_yr)
## contrast estimate SE df t.ratio p.value
## Year_fct2015 - Year_fct2016 -0.1867 0.157 58.5 -1.186 0.7591
## Year_fct2015 - Year_fct2017 0.0456 0.228 58.5 0.200 0.9996
## Year_fct2015 - Year_fct2018 0.7287 0.264 58.5 2.756 0.0577
## Year_fct2015 - Year_fct2019 0.2482 0.163 58.5 1.525 0.5507
## Year_fct2016 - Year_fct2017 0.2323 0.252 58.5 0.921 0.8878
## Year_fct2016 - Year_fct2018 0.9154 0.348 58.5 2.632 0.0775
## Year_fct2016 - Year_fct2019 0.4349 0.227 58.5 1.913 0.3223
## Year_fct2017 - Year_fct2018 0.6831 0.321 58.5 2.130 0.2214
## Year_fct2017 - Year_fct2019 0.2026 0.254 58.5 0.796 0.9306
## Year_fct2018 - Year_fct2019 -0.4805 0.214 58.5 -2.245 0.1780
##
## P value adjustment: tukey method for comparing a family of 5 estimates
# Create boxplot showing Tukey post-hoc results
boxplot_cont_yr(em_lib_yr, df_chla_lib_lag)
This time the model seems off for 2016. This is most likely because the lag1 term is explaining much of the variance. I think this model is okay for looking at the effects of flow by year, but not for differences between years.
df_chla_rvb <- ls_chla_flow_lag$RVB |>
select(-Flow) |>
drop_na(Flow_lag3, Chla_log)
m_gam_rvb <- gam(
Chla_log ~ Year_fct * Flow_lag3 + s(Week, bs = "cc", k = 5),
data = df_chla_rvb,
method = "REML",
knots = list(Week = c(0, 52))
)
Lets look at the model summary and diagnostics:
check_gam_diagnostics(m_gam_rvb)
## ==================================================
## GAM MODEL DIAGNOSTICS
## ==================================================
##
## --- 1. MODEL SUMMARY ---
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow_lag3 + s(Week, bs = "cc", k = 5)
##
## Parametric coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.6186252 0.0490292 155.390 < 2e-16 ***
## Year_fct2016 0.0151918 0.0825560 0.184 0.8546
## Year_fct2017 -0.3390735 0.0811084 -4.181 9.40e-05 ***
## Year_fct2018 -0.3361156 0.0709362 -4.738 1.32e-05 ***
## Year_fct2019 -0.6872073 0.0734889 -9.351 2.08e-13 ***
## Flow_lag3 -0.0006823 0.0003611 -1.890 0.0636 .
## Year_fct2016:Flow_lag3 -0.0002422 0.0005034 -0.481 0.6321
## Year_fct2017:Flow_lag3 0.0001015 0.0015249 0.067 0.9471
## Year_fct2018:Flow_lag3 0.0004211 0.0004193 1.004 0.3192
## Year_fct2019:Flow_lag3 0.0005831 0.0003851 1.514 0.1351
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 2.711 3 8.669 3.51e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## R-sq.(adj) = 0.761 Deviance explained = 79.9%
## -REML = 29.632 Scale est. = 0.037955 n = 74
##
## --- 2. BASIS DIMENSION CHECK (k.check) ---
## k' edf k-index p-value
## s(Week) 3 2.711112 0.9691104 0.3625
##
## *Note: If p-value is low, the basis dimension 'k' might be too small.*
##
## --- 3. NORMALITY TEST OF RESIDUALS ---
##
## Shapiro-Wilk normality test
##
## data: resids
## W = 0.95785, p-value = 0.01469
##
## Result: Reject H0. Residuals are NOT normally distributed (p < 0.05).
##
## --- 4. GENERATING APPRAISE PLOTS ---
## Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.
## --- 5. GENERATING SMOOTH TERM PLOTS ---
## ==================================================
check_gam_autocorr(m_gam_rvb)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 23.196, df = 20, p-value = 0.2793
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
The diagnostic plots and stats look good. However, the residuals are slightly autocorrelated.
Now, we’ll try to deal with the residual autocorrelation. We’ll run a series of models adding 1 and 2 lag terms and compare how well they correct for autocorrelation.
m_gam_rvb_lag1 <- gam(
Chla_log ~ Year_fct * Flow_lag3 + s(Week, bs = "cc", k = 5) + lag1,
data = df_chla_rvb,
method = "REML",
knots = list(Week = c(0, 52))
)
check_gam_autocorr(m_gam_rvb_lag1)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 10.975, df = 20, p-value = 0.9469
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
m_gam_rvb_lag2 <- gam(
Chla_log ~ Year_fct * Flow_lag3 + s(Week, bs = "cc", k = 5) + lag1 + lag2,
data = df_chla_rvb,
method = "REML",
knots = list(Week = c(0, 52))
)
check_gam_autocorr(m_gam_rvb_lag2)
## ==================================================
## GAM RESIDUAL AUTOCORRELATION CHECK
## ==================================================
##
## --- 1. LJUNG-BOX TEST FOR SERIAL CORRELATION ---
##
## Box-Ljung test
##
## data: resids
## X-squared = 14.431, df = 20, p-value = 0.808
##
## Result: Fail to reject H0. No significant residual autocorrelation (p >= 0.05).
##
## --- 2. PLOTTING AUTOCORRELATION FUNCTION (ACF) ---
## ==================================================
The model with 1 lag term seems to be okay with autocorrelation according to the acf plot. Adding a second lag term seems unnecessary. Let’s take a closer look at the model with 1 lag term.
check_gam_diagnostics(m_gam_rvb_lag1)
## ==================================================
## GAM MODEL DIAGNOSTICS
## ==================================================
##
## --- 1. MODEL SUMMARY ---
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow_lag3 + s(Week, bs = "cc", k = 5) +
## lag1
##
## Parametric coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4.4374656 0.7094637 6.255 4.39e-08 ***
## Year_fct2016 -0.0107512 0.0794509 -0.135 0.892807
## Year_fct2017 -0.2489971 0.0822884 -3.026 0.003631 **
## Year_fct2018 -0.2303186 0.0731982 -3.147 0.002558 **
## Year_fct2019 -0.3919876 0.0968250 -4.048 0.000148 ***
## Flow_lag3 -0.0008376 0.0003254 -2.574 0.012522 *
## lag1 0.4188465 0.0925774 4.524 2.87e-05 ***
## Year_fct2016:Flow_lag3 0.0002272 0.0004622 0.492 0.624815
## Year_fct2017:Flow_lag3 0.0002050 0.0014570 0.141 0.888570
## Year_fct2018:Flow_lag3 0.0005158 0.0003990 1.293 0.201024
## Year_fct2019:Flow_lag3 0.0005877 0.0003660 1.606 0.113506
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 1.199 3 0.858 0.124
##
## R-sq.(adj) = 0.775 Deviance explained = 81%
## -REML = 23.535 Scale est. = 0.03485 n = 73
##
## --- 2. BASIS DIMENSION CHECK (k.check) ---
## k' edf k-index p-value
## s(Week) 3 1.198977 0.8776472 0.1025
##
## *Note: If p-value is low, the basis dimension 'k' might be too small.*
##
## --- 3. NORMALITY TEST OF RESIDUALS ---
##
## Shapiro-Wilk normality test
##
## data: resids
## W = 0.95419, p-value = 0.009799
##
## Result: Reject H0. Residuals are NOT normally distributed (p < 0.05).
##
## --- 4. GENERATING APPRAISE PLOTS ---
## Displaying: Residual QQ-plot, Residuals vs Linear Predictor, Histogram, and Observed vs Fitted.
## --- 5. GENERATING SMOOTH TERM PLOTS ---
## ==================================================
anova(m_gam_rvb_lag1)
##
## Family: gaussian
## Link function: identity
##
## Formula:
## Chla_log ~ Year_fct * Flow_lag3 + s(Week, bs = "cc", k = 5) +
## lag1
##
## Parametric Terms:
## df F p-value
## Year_fct 4 4.882 0.00176
## Flow_lag3 1 6.623 0.01252
## lag1 1 20.469 2.87e-05
## Year_fct:Flow_lag3 4 0.770 0.54899
##
## Approximate significance of smooth terms:
## edf Ref.df F p-value
## s(Week) 1.199 3.000 0.858 0.124
The model diagnostics look a litle worse than the model without lag terms, but it’s corrected for autocorrelation. Before we proceed with this model for RVB, let’s take a closer look at how the back-transformed fitted values from the model match the observed values.
df_chla_rvb_lag <- df_chla_rvb |> drop_na(lag1)
# Sample sizes for each year
df_chla_rvb_lag |> count(Year_fct)
## # A tibble: 5 Ă— 2
## Year_fct n
## <fct> <int>
## 1 2015 17
## 2 2016 12
## 3 2017 14
## 4 2018 15
## 5 2019 15
plot_obs_vs_model_fit(m_gam_rvb_lag1, df_chla_rvb_lag)
The model fit looks pretty decent. Note that variability does increase as the chlorophyll values increase. Now, let’s look more closely at its results.
Partial effect of flow on chlorophyll:
# Calculate effects of flow on chlorophyll holding the non-focal variables constant -
# marginal effects/adjusted predictions
df_rvb_flow_eff <- calc_flow_yr_marg_eff(
m_gam_rvb_lag1,
df_chla_rvb_lag,
"Flow_lag3"
)
# Create effects plot
plot_eff_flow_by_yr(df_rvb_flow_eff, df_chla_rvb_lag, "Flow_lag3")
# Estimated marginal means for year
em_rvb_yr <- emmeans(m_gam_rvb_lag1, ~Year_fct)
# Tukey post-hoc contrasts
pairs(em_rvb_yr)
## contrast estimate SE df t.ratio p.value
## Year_fct2015 - Year_fct2016 0.00109 0.0785 60.8 0.014 1.0000
## Year_fct2015 - Year_fct2017 0.24028 0.1200 60.8 2.009 0.2743
## Year_fct2015 - Year_fct2018 0.20838 0.0716 60.8 2.909 0.0391
## Year_fct2015 - Year_fct2019 0.36699 0.0932 60.8 3.936 0.0019
## Year_fct2016 - Year_fct2017 0.23919 0.1290 60.8 1.861 0.3494
## Year_fct2016 - Year_fct2018 0.20729 0.0871 60.8 2.381 0.1346
## Year_fct2016 - Year_fct2019 0.36590 0.1070 60.8 3.417 0.0096
## Year_fct2017 - Year_fct2018 -0.03190 0.1160 60.8 -0.275 0.9987
## Year_fct2017 - Year_fct2019 0.12671 0.1210 60.8 1.046 0.8327
## Year_fct2018 - Year_fct2019 0.15861 0.0788 60.8 2.012 0.2726
##
## P value adjustment: tukey method for comparing a family of 5 estimates
# Create boxplot showing Tukey post-hoc results
boxplot_cont_yr(em_rvb_yr, df_chla_rvb_lag)
The year contrasts look okay, slightly off for 2016 and 2019 which is most likely because the lag1 term is explaining much of the variance. I think this model is okay for looking at the effects of flow by year, and could be used to look at differences between years.
After looking at the results more closely from these models, I think they are adequate for predicting the effects of flow by year, but not at the differences between years.