2. Random Forest_Council_github.Rmd

---
title: "2.from kyle_Random Forest Council (Neural Network)" 
#Here, we use the Random Forest model to gapfill data based on predictors and real data - it runs simulations and learns patterns, attempting to fill in gaps based on training data and validation data 
#Random Forest is used to do a lot of the gap-filling --we could try to improve it by increasing the trees to allow for more complex relationships and we can clean the data a little more to take out outliers (which Dani did Oct 28, 2024 - we use the re-cleaned version here)

##For Council, NOTE: primary wind direction is N-S at this location, S is thermokarst and degradation which can be a big methane source, so wind direction could be important at this site --> NOTE: model is elevating CH4 in the winter, may need to revisit training / other relationships we can use to make sure it's more reasonable (no real winter data available to better train model with)

output: html_document
date: "2024-09-17"
---
#Set working directory
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE,dev = 'png')
#knitr::opts_knit$set(root.dir = 'C:/Users/karndt.WHRC/Desktop/sites/council/data') #Edit this to set your working directory
```


#Load in required packages
```{r,error=FALSE,warning=FALSE}
rm(list = ls()) #clear obj from environment 
#gc() #garbage collection to free up R memory
library(data.table)
library(ggplot2)
library(caret)
library(randomForest)
library(Boruta)
#test
Sys.setenv(TZ='UTC')
```

#Load in site data & make training datasets 
```{r,error=FALSE,warning=FALSE}
cf = fread("C:/Users/kkent/Documents/Council Data/Council BASE gapfilling/council_2016_2023_era.csv") #using the re-cleaned df from Dani 

#subset out by which has complete air t data 
cf = cf[complete.cases(cf$airt.eramod),]

```

#Create subset of data to use for RF training -- use only ERA5 variables for training since RF needs a continuous dataset (no NAs)

```{r}
#create a sub data frame of the variables you want for a neural network (random forest)
#KK Nov 27: added in u, v, wd, and cardinal direction from era df; also changed FC and FCH4 to FC.c, FCH4.c, and SWC_1_1_1.c to use re-cleaned data --> adj to WD from actual data and Cardinal_Direction from actual data as these seem to be era5 data adj by actual data and this uses the actual SWC data from site.



#create subset of the variables wanted for training RF
ncf = data.frame(cf$date,cf$FC.c,cf$FCH4.c,
                 cf$airt.eramod, cf$wd, cf$cardinal_direction, cf$rh.eramod,cf$rad.eramod,
                 cf$ws.eramod,cf$tsoil.eramod,#cf$SWC_1_1_1.c,cf$SWC_2_1_1.c, 
                 cf$SWC_3_1_1.c, #SWC 3 = the tussock location 
                 cf$h.eramod,cf$le.eramod)
#SWC also broken up by location
#  SWC_1_1_1 % Soil water content (15cm depth) – margin pond
# SWC_2_1_1 % Soil water content (15cm depth) – lichen/berries
# SWC_3_1_1 % Soil water content (15cm depth) - tussock ***** switch to using this one ** (used SWC1 before, by margin pond, switching to SWC3, tussock)

#another way to subset a new df 
# library(dplyr)
# 
# # Select specific variables from 'cf' and store them in 'ncf'
# ncf <- cf %>%
#   select(date, FC.c, FCH4.c, airt.eramod, WD, Cardinal_Direction, rh.eramod, rad.eramod, ws.eramod, tsoil.eramod, SWC_1_1_1.c, h.eramod, le.eramod)  

# View the new dataframe
head(ncf)

#rename for easier names
names(ncf) = c('date','nee',"fch4",'tair', 'wd', 'cardDir','rh','rg','ws','tsoil','swc','h','le')

#make card dir as factor - 8 levels 
ncf$cardDir <- as.factor(ncf$cardDir)

```
#SWC gapfilling through RF 

####Calc VPD from air temp and RH
```{r}
#calculate VPD from air t and RH
svp = 610.7*10^((7.5*ncf$tair)/(237.3+ncf$tair))
ncf$vpd = ((100 - ncf$rh)/100)*svp  
```

#create training dataset and testing dataset 
```{r}
#*KK edit 12/27/24 - changing to SWC3 to reflect tussock 

orig = Sys.time() #starting timer 

set.seed(123)#sets the start point of models, good for repeatability
cc = ncf[complete.cases(ncf$swc),]#create a gap free data set of the target variable

#use 80% of data set as training set and 20% as test set
sample = sample(c(TRUE, FALSE), nrow(cc), replace=TRUE, prob=c(0.8,0.2))
train  = cc[sample, ]
test   = cc[!sample, ]

#compare to ensure all data sets are representative of total data
hist(cc$swc)
hist(train$swc)
hist(test$swc)

#run random forest to predict missing SWC values 
rf.swc = randomForest(formula = swc ~ tair + rh + rg + ws + wd + tsoil + vpd + le + h, data = train, ntree = 800)
#per OOB errors, optimal # of tress for SWC gapfilling (using SWC3, tussock), is 800 
#used ntree=150 before 

#predict it on the full data set
swc.rf = predict(object = rf.swc, newdata = ncf)
ncf$swc.rf = swc.rf #add the variable to the main data frame


Sys.time() - orig #stop timer --> about  5 min for 150, about 50 min for 800
```


#Check out the SWC gap filling
```{r}
#time series plot of gap filled vs real
ggplot(data = ncf)+
  geom_point(aes(date,swc.rf*100,col='RF'))+
  geom_point(aes(date,swc*100,col='Measured'))+
  scale_y_continuous("SWC (%)")


#validation dataset from only test data
val = merge(test,ncf,by = "date",all.x = T)

ggplot(data = val,aes(swc.rf.y,swc.x))+theme_bw()+geom_abline(slope = 1,intercept = 0,col='red')+
  geom_point(alpha=0.1)+
  scale_x_continuous(limits = c(0,70),"RF SWC (%)")+
  scale_y_continuous(limits = c(0,70),"Measured SWC (%)")



#summary stats on gap filling
summary(lm(val$swc.x ~ val$swc.rf.y)) #R2 = 0.97, slope = 1.02

#Fig for council paper supp material 
SWC_RF <- ggplot(data = val,aes(swc.rf,swc.x))+theme_bw()+geom_abline(slope = 1,intercept = 0,col='red')+
  geom_point(alpha=0.1)+
  scale_x_continuous(limits = c(0,70),"RF SWC (%)")+
  scale_y_continuous(limits = c(0,70),"Measured SWC (%)")+
    annotate(geom = 'text',x = 10, y = 70,label=expression(R^2~"= 0.98"),size = 5)+
  annotate(geom = 'text',x = 10,y = 65,label=expression(Slope~"= 1.02"), size = 5)
SWC_RF

ggsave("C:/Users/kkent/Documents/Github Flux Network/Council_Flux_Analysis_Paper/Council Figures/SWC_RF_Validation.png", SWC_RF, 
       width = 10, height = 7, dpi = 600)



#create final gap free soil moisture
ncf$swc = ifelse(is.na(ncf$swc),ncf$swc.rf,ncf$swc)



#Create an interaction term between swc & soil temp 
ncf$TS_SWC_interact <- ncf$tsoil * ncf$swc



```
#NEE *********************

####Look at range of data 
```{r}
ggplot(data = ncf, aes(x=date, y = nee)) + geom_point()

```


####Create NEE dataset & Examine variable importance
```{r}
set.seed(123)
cc = ncf[complete.cases(ncf$nee),] #complete NEE data only 
cc = subset(cc,cc$nee < 12 & cc$nee > -14) #the range of nee for ncf dataset is -16 - 10 and there are no real major outliers, so this range captures the data well, otherwise use these to set limits on the spread of data to exclude  large outliers and tidy up the model - may need to adjust these limits based on your data*

#this attempts to establish which of the variables have the most important influence on NEE - kyle commented this out, may not be necessary & can take a little while to run but cool to see 

#added in wd (degrees)
boruta = Boruta(nee ~ tair + rh + rg + ws + wd + tsoil + swc + vpd + le + h,data = cc,doTrace = 2,maxRuns = 100)

#plots the variable importance from boruta 
plot(boruta,las = 2)
```


#Let's try to fill NEE with random forest

####Create training and testing dataset 
```{r,error=FALSE,warning=FALSE}
#use 80% of data set as training set and 20% as test set
sample.nee = sample(c(TRUE, FALSE), nrow(cc), replace=TRUE, prob=c(0.8,0.2))
train.nee  = cc[sample.nee, ]
test.nee   = cc[!sample.nee, ]

#Make sure the patterns here look similar so you know the training and testing datasets are representative of the overall dataset 
hist(cc$nee)
hist(train.nee$nee)
hist(test.nee$nee)

#sum(is.na(cc$swc)) #make sure there are no NAs in dataset or RF model won't work 
```
Find the optimum number of trees for the RF model using minimum out-of-bag error (OOB error)
-optimal number of trees for nee = 850, but minimal improvement after 600/650

A higher %IncMSE value means that the variable is more important for the model. Removing this variable would result in a larger increase in the model's prediction error.
A lower %IncMSE value means that the variable is less important. Removing this variable would result in a smaller increase in the model's prediction error.

#Training RF; include: wd, SWC3 tussuck, swc*tsoil interaction

####Training various versions of the model to determine best one, altering interaction effect, ntree #, and checking with lm agreement & RMSE 
```{r}
orig = Sys.time() #starting timer to see how long this takes 

#original RF model, no wind 
# rfnee = randomForest(formula = nee ~ tair + rh + rg + ws + tsoil + swc + vpd + le + h,data = train.nee,ntree = 150,importance=T)
# 
# pnee = predict(object = rfnee,newdata = ncf)
# 
# cf$rfnee = pnee

# Extract variable importance
# importance_values1 <- importance(rfnee, type = 1)  # type = 1 for %IncMSE
# 
# # Print the importance values
# print(importance_values1)
# 
# # Plot the importance values
# varImpPlot(rfnee, type = 1, main = "Variable Importance (%IncMSE)")
#Results:
#        %IncMSE
# tair  16.62267
# rh    34.39371
# rg    18.50870
# ws    44.29823 --> 2
# tsoil 29.00674
# swc   57.89705 --> 1
# vpd   17.67771
# le    28.87872
# h     36.95337 --> 3


#Final model (1.16.2025) --> using wd (degrees) and swc*tsoil interaction effect 
rfnee.2 = randomForest(formula = nee ~ tair + rh + rg + ws + wd + tsoil + swc + vpd + le + h + TS_SWC_interact, data = train.nee, ntree = 850, importance=T)

pnee.2 = predict(object = rfnee.2,newdata = ncf)

cf$rfnee.2 = pnee.2

# Extract variable importance
importance_values <- importance(rfnee.2, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values)

# Plot the importance values
varImpPlot(rfnee.2, type = 1, main = "NEE: Variable Importance (%IncMSE)")

#Results from ntree = 850
#                   %IncMSE
# tair             41.45073
# rh               73.40076
# rg               46.32502
# ws               82.69876
# wd              113.00751
# tsoil            57.68105
# swc             105.20490
# vpd              46.89792
# le               60.09180
# h                74.56083
# TS_SWC_interact  53.37965

#removing swc*tsoil interaction 
rfnee.nointeract = randomForest(formula = nee ~ tair + rh + rg + ws + wd + tsoil + swc + vpd + le + h, data = train.nee, ntree = 850, importance=T)


pnee.nointeract = predict(object = rfnee.nointeract, newdata = ncf)

cf$rfnee.nointeract = pnee.nointeract

# Extract variable importance
importance_values_nointeract <- importance(rfnee.nointeract, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values_nointeract)

# Plot the importance values
varImpPlot(rfnee.nointeract, type = 1, main = "NEE: Variable Importance (%IncMSE), ntree = 850, no interact")
#Results:
#        %IncMSE
# tair   43.68468
# rh     81.55038
# rg     42.04323
# ws     92.62934
# wd    123.02509
# tsoil  70.79813
# swc   125.02074
# vpd    47.83887
# le     58.24990
# h      77.54589



#ntree = 1000, wd (degrees), and swc*tsoil interaction effect 
rfnee.1000interact = randomForest(formula = nee ~ tair + rh + rg + ws + wd + tsoil + swc + vpd + le + h + TS_SWC_interact, data = train.nee, ntree = 1000, importance=T)

pnee.1000interact = predict(object = rfnee.1000interact,newdata = ncf)

cf$rfnee.1000interact = pnee.1000interact

# Extract variable importance
importance_values_1000interact <- importance(rfnee.1000interact, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values_1000interact)

# Plot the importance values
varImpPlot(rfnee.1000interact, type = 1, main = "NEE: Variable Importance (%IncMSE)")
#Results:
#                   %IncMSE
# tair             43.56984
# rh               73.43686
# rg               47.31644
# ws               91.75861
# wd              122.65821
# tsoil            61.99925
# swc             114.34163
# vpd              51.96064
# le               60.12734
# h                79.40722
# TS_SWC_interact  62.15551


#ntree = 1000, wd (degrees), *NO* interaction effect 
rfnee.1000 = randomForest(formula = nee ~ tair + rh + rg + ws + wd + tsoil + swc + vpd + le + h, data = train.nee, ntree = 1000, importance=T)

pnee.1000 = predict(object = rfnee.1000,newdata = ncf)

cf$rfnee.1000 = pnee.1000

# Extract variable importance
importance_values_1000 <- importance(rfnee.1000, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values_1000)

# Plot the importance values
varImpPlot(rfnee.1000, type = 1, main = "NEE: Variable Importance (%IncMSE)")
#Results
#        %IncMSE
# tair   46.87582
# rh     83.79680
# rg     48.27433
# ws     96.58794
# wd    125.74901
# tsoil  81.53861
# swc   131.51913
# vpd    53.56545
# le     64.36027
# h      82.82959



Sys.time() - orig #closing timer --> ntree = 650 took 12 min // 850 trees took roughly 23 min --> then took 16 min --> ntree=1000 took 20 min 
```
#Importance plot figure to save 
```{r}
library(dplyr)
library(ggplot2)

#Make varImport a new dataset 
# Extract variable importance
importance_values_nee <- importance(rfnee.2, type = 1)  # type = 1 for %IncMSE

# Convert the importance values to a data frame
NEEimportance_df <- as.data.frame(importance_values_nee)

# Add the variable names (row names) as a column
NEEimportance_df$Variable <- rownames(NEEimportance_df)

# Rename columns for clarity
colnames(NEEimportance_df) <- c("Importance", "Variable")


# Step 1: Sort the dataframe by Importance (ascending order)
NEEimportance_df <- NEEimportance_df %>%
  arrange(Importance)

# Step 2: Create the ggplot with the sorted data
importance_plot_NEE <- ggplot(NEEimportance_df, aes(x = reorder(Variable, Importance), y = Importance)) +
  # Add dotted lines aligned with variables
  geom_vline(xintercept = 1:nrow(NEEimportance_df), linetype = "dotted", color = "grey70") +
  # Points and other elements
  geom_point(size = 4, color = "black", fill = "white", shape = 21) +
  labs(title = "NEE: Variable Importance (%IncMSE)", x = "Variable", y = "%IncMSE") +
  coord_flip() +
   theme_bw() +
  theme(
    panel.grid.major = element_blank(),  # Remove all major grid lines
    panel.grid.minor = element_blank(),  # Remove all minor grid lines
    axis.text.x = element_text(angle = 0, hjust = 0.5, size = 14, face = "bold"),
    axis.text.y = element_text(size = 14, face = "bold"),
    axis.title.x = element_text(size = 16, face = "bold"),
    axis.title.y = element_text(size = 16, face = "bold"),
    plot.title = element_text(size = 20, face = "bold")
  )

# Print the plot
print(importance_plot_NEE)


#Save image 
ggsave(
  filename = "C:/Users/kkent/Documents/Github Flux Network/Council_Flux_Analysis_Paper/Council Figures/Importance_plot_IncMSE_NEE.png",  # File name and extension
  plot = importance_plot_NEE,   # The plot to save (default is the last plot created)
  width = 18,                 # Width in inches
  height = 10,                # Height in inches
  dpi = 600,                 # Resolution in dots per inch
  units = "cm"               # Units for width and height (can be "in", "cm", or "mm")
)

#ggsave("my_plot.pdf", width = 8, height = 6, dpi = 300, units = "in")

```


#RSME assessment of the different versions of the RF model 
```{r}
#merge the predicted datasets with the orig to validate 
test.data.nee = merge(test.nee,cf,by = 'date',all.x = T)

#original, no wd, ntree = 150
rmse_nee <- sqrt(mean((test.data.nee$FC.c - test.data.nee$rfnee)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 150 ", rmse_nee)) #RSME =  1.51710632421705"

#with wd, ntree 150, no interaction 
rmse_nee150 <- sqrt(mean((test.data.nee$FC.c - test.data.nee$rfnee.150)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 150 ", rmse_nee150)) #RSME 1.49099361362971 --> better RMSE, but since it's much lower number of trees, could be an artifact of not generalizing well / capturing noise 

#with wd, ntree 1000, no interaction 
rmse_nee1000 <- sqrt(mean((test.data.nee$FC.c - test.data.nee$rfnee.1000)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 1000 ", rmse_nee1000)) #RSME  1.5125100083716 

#with wd, ntree 1000, WITH* swc*tair interaction 
rmse_nee1000interact <- sqrt(mean((test.data.nee$FC.c - test.data.nee$rfnee.1000interact)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 1000 ", rmse_nee1000interact)) #RSME 1.50456892078326"--> very slightly better fit 

#with wd, interaction, ntree = 850 --> **FINAL MODEL**
rmse_nee_interact <- sqrt(mean((test.data.nee$FC.c - test.data.nee$rfnee.2)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 850, swc*tsoil interaction ", rmse_nee_interact)) #RSME   1.50520844797875  --> actually a bit worse, maybe remove interaction?

#with wd, *NO interaction, ntree = 850
rmse_nee_nointeract <- sqrt(mean((test.data.nee$FC.c - test.data.nee$rfnee.nointeract)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 850, no interaction ", rmse_nee_nointeract)) #RSME   1.51357213649105  --> worse, keep interaction 
```

####Time series plots
```{r,error=FALSE,warning=FALSE}
#This overlays the random forest modeled data points with the real data points so you can observe model agreement 
#ntree = 850; plot of final version of RF model rfnee.2
ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfnee.2,col='RF'),alpha=0.2)+
  geom_point(aes(date,FC.c,col='Real'),alpha=0.2)+
  labs(title = "with WD (degrees), ntree=850")+
  scale_x_datetime(limits = as.POSIXct(c("2019-06-15","2019-06-30")))
#plot of all years; tree = 850
ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfnee.2,col='RF'),alpha=0.2)+
  geom_point(aes(date,FC.c,col='Real'),alpha=0.2)+
  geom_hline(yintercept = 0)+
    labs(title = "with WD (degrees), ntree=850")


#ntree = 850 and ntree = 1000 look very similar*


#Results very similar, also seems to be predicting slightly lower/more conservative nee compared to site data 
```

####Validation - looking at agreement between modeled data and real data 
```{r,error=FALSE,warning=FALSE}
#merge the predicted datasets with the orig to validate, if you haven't already for the RMSE comparisons
test.data.nee = merge(test.nee,cf,by = 'date',all.x = T)

#changed to FC.c to use re-cleaned variable 

#Added wd (degrees) - RF ntree = 850, with interaction *FINAL MODEL********
summary(lm(test.data.nee$FC.c ~ test.data.nee$rfnee.2)) #R2 = 0.651; slope = 1.03; p<0.001

councilnee2 = ggplot(data = test.data.nee,aes(rfnee.2,FC.c))+theme_bw()+
  geom_hline(yintercept = 0,lty=2)+
  geom_vline(xintercept = 0,lty=2)+
  geom_point(alpha=0.2)+
  scale_fill_viridis_c()+
  geom_abline(slope = 1,intercept = 0,col='red',lty=1)+
  scale_x_continuous(limits = c(-20,20),expression('Random Forest NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  scale_y_continuous(limits = c(-20,20),expression('Eddy Covariance NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  annotate(geom = 'text',x = 6, y = -8,label=expression(R^2~"= 0.65"),size = 3)+
  annotate(geom = 'text',x = 6,y = -10,label=expression(Slope~"= 1.03"),size = 3)+
  theme(text = element_text(size = 8)) +
  labs(title = "with WD (degrees), ntree = 850, swc*tsoil interaction")

councilnee2 


#with wd (degrees) - RF ntree = 850, *NO interaction 
summary(lm(test.data.nee$FC.c ~ test.data.nee$rfnee.nointeract)) #R2 = 0.647; slope = 1.03; p<0.001

councilnee_nointeract = ggplot(data = test.data.nee,aes(rfnee.nointeract,FC.c))+theme_bw()+
  geom_hline(yintercept = 0,lty=2)+
  geom_vline(xintercept = 0,lty=2)+
  geom_point(alpha=0.2)+
  scale_fill_viridis_c()+
  geom_abline(slope = 1,intercept = 0,col='red',lty=1)+
  scale_x_continuous(limits = c(-20,20),expression('Random Forest NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  scale_y_continuous(limits = c(-20,20),expression('Eddy Covariance NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  annotate(geom = 'text',x = 6, y = -8,label=expression(R^2~"= 0.65"),size = 3)+
  annotate(geom = 'text',x = 6,y = -10,label=expression(Slope~"= 1.03"),size = 3)+
  theme(text = element_text(size = 8)) +
  labs(title = "with WD (degrees), ntree = 150, no interaction")

councilnee_nointeract  



#Added wd (degrees) - RF ntree = 1000, NO swc*tsoil interaction 
summary(lm(test.data.nee$FC.c ~ test.data.nee$rfnee.1000)) #R2 = 0.647, slope = 1.03

councilnee_1000nointeract = ggplot(data = test.data.nee,aes(rfnee.1000,FC.c))+theme_bw()+
  geom_hline(yintercept = 0,lty=2)+
  geom_vline(xintercept = 0,lty=2)+
  geom_point(alpha=0.2)+
  scale_fill_viridis_c()+
  geom_abline(slope = 1,intercept = 0,col='red',lty=1)+
  scale_x_continuous(limits = c(-20,20),expression('Random Forest NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  scale_y_continuous(limits = c(-20,20),expression('Eddy Covariance NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  annotate(geom = 'text',x = 6, y = -8,label=expression(R^2~"= 0.65"),size = 3)+
  annotate(geom = 'text',x = 6,y = -10,label=expression(Slope~"= 1.03"),size = 3)+
  theme(text = element_text(size = 8)) +
  labs(title = "with WD (degrees), ntree = 1000, no interaction")

councilnee_1000nointeract

#Added wd (degrees) - RF ntree = 1000, WITH* swc*tsoil interaction 
summary(lm(test.data.nee$FC.c ~ test.data.nee$rfnee.1000interact)) #R2 = 0.6515, slope = 1.02

councilnee_1000interact = ggplot(data = test.data.nee,aes(rfnee.1000interact,FC.c))+theme_bw()+
  geom_hline(yintercept = 0,lty=2)+
  geom_vline(xintercept = 0,lty=2)+
  geom_point(alpha=0.2)+
  scale_fill_viridis_c()+
  geom_abline(slope = 1,intercept = 0,col='red',lty=1)+
  scale_x_continuous(limits = c(-20,20),expression('Random Forest NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  scale_y_continuous(limits = c(-20,20),expression('Eddy Covariance NEE ('*mu*mol~CO[2]~m^-2~s^-1*')'))+
  annotate(geom = 'text',x = 6, y = -8,label=expression(R^2~"= 0.65"),size = 3)+
  annotate(geom = 'text',x = 6,y = -10,label=expression(Slope~"= 1.02"),size = 3)+
  theme(text = element_text(size = 8)) +
  labs(title = "with WD (degrees), ntree = 1000, swc*tsoil interaction")

councilnee_1000interact

#NOTE: ntree = 850 and ntree = 1000 appear essentially the same 

```

#Methane 

####Examine variable importance for FCH4
```{r}
set.seed(123)
cc = ncf[complete.cases(ncf$fch4),]

#added wd (degrees)
boruta = Boruta(fch4 ~ tair + rh + rg + ws + wd + tsoil + swc + vpd + le + h,data = cc,doTrace = 2,maxRuns = 100)
boruta$finalDecision #indicates all above are confirmed important variables via a list 
plot(boruta,las = 2) #plots the variables; green = important, yellow = tentative, red = not important 


```


#try random forest for CH4 

####Subset training and testing datasets 
```{r}
#looking at range of data 
ggplot(data = cc, aes(x=date, y = fch4)) + geom_point()


#use 80% of data set as training set and 20% as test set, expanded to try to represent the larger points better
#cc = subset(cc,cc$fch4 < 50) #50 cuts off what looks like real data for one of the years...adj to perhaps 60 
cc = subset(cc,cc$fch4 < 60)
cc = subset(cc,cc$fch4 > -25)
sample.ch4 = sample(c(TRUE, FALSE), nrow(cc), replace=TRUE, prob=c(0.8,0.2))
train.ch4  = cc[sample.ch4, ]
test.ch4   = cc[!sample.ch4, ]

hist(cc$fch4)
hist(train.ch4$fch4)
hist(test.ch4$fch4)

summary(cc$fch4)
summary(train.ch4$fch4)
summary(test.ch4$fch4)


```
####RF model for CH4 --> ntree = 150, with wd, just to see 

```{r}
# Train the Random Forest model - no interaction 
rf_model_CH4 = randomForest(formula = fch4 ~ tair + rh + rg + ws + wd + tsoil + vpd + swc + h + le, data = train.ch4,importance = TRUE, ntree = 150)
# Extract variable importance
importance_values <- importance(rf_model_CH4, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values)

# Plot the importance values
varImpPlot(rf_model_CH4, type = 1, main = "FCH4: Variable Importance (%IncMSE), ntree = 150, no interaction")
#Results:
#        %IncMSE
# tair  28.16018
# rh    35.06199
# rg    23.75861
# ws    45.06025 --> 3
# wd    53.83652 --> 2
# tsoil 40.79492
# vpd   25.60573
# swc   70.87396 --> 1
# h     31.45271
# le    31.76639

```



#### RF model for CH4 --> per exploratory modeling (in Rmd fild Step 1.5), found optimum number of trees for CH4 was 950 per OOB error rate and improved from adding swc*tsoil interaction 
```{r}
orig = Sys.time() #set timer

#ntree = 950, added wd (degrees) & swc*tsoil interaction -- used this version of the model***********  
rfch4.2 = randomForest(formula = fch4 ~ tair + rh + rg + ws + wd + tsoil + vpd + swc + h + le + TS_SWC_interact, data = train.ch4, ntree = 950, importance = TRUE)

pch4.2 = predict(object = rfch4.2, newdata = ncf)

cf$rfch4.2 = pch4.2

# Extract variable importance
importance_values_ch4 <- importance(rfch4.2, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values_ch4)

# Plot the importance values
varImpPlot(rfch4.2, type = 1, main = "FCH4: Variable Importance (%IncMSE), ntree = 950")

#Results
#                   %IncMSE
# tair             71.33973
# rh              102.17358
# rg               64.94474
# ws              130.02763
# wd              160.38979
# tsoil            76.85577
# vpd              65.92918
# swc             138.76444
# h                86.83067
# le               76.71211
# TS_SWC_interact  71.71983

#ntree = 1000, added wd (degrees) & swc*tsoil interaction -- used this version of the model***********  
rfch4.1000 = randomForest(formula = fch4 ~ tair + rh + rg + ws + wd + tsoil + vpd + swc + h + le + TS_SWC_interact, data = train.ch4, ntree = 1000, importance = TRUE)

pch4.1000 = predict(object = rfch4.1000, newdata = ncf)

cf$rfch4.1000 = pch4.1000

# Extract variable importance
importance_values_ch4.1000 <- importance(rfch4.1000, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values_ch4.1000)

# Plot the importance values
varImpPlot(rfch4.1000, type = 1, main = "FCH4: Variable Importance (%IncMSE), ntree = 1000")
#Results
#                   %IncMSE
# tair             68.87706
# rh              100.40642
# rg               64.18042
# ws              143.98849
# wd              165.49987
# tsoil            86.94837
# vpd              67.81461
# swc             145.68184
# h                93.27058
# le               84.19241
# TS_SWC_interact  74.92690


#ntree = 1000, added wd (degrees), *NO* interaction  
rfch4.1000.nointeract = randomForest(formula = fch4 ~ tair + rh + rg + ws + wd + tsoil + vpd + swc + h + le, data = train.ch4, ntree = 1000, importance = TRUE)

pch4.1000.nointeract = predict(object = rfch4.1000.nointeract, newdata = ncf)

cf$rfch4.1000.nointeract = pch4.1000.nointeract

# Extract variable importance
importance_values_ch4.1000.nointeract <- importance(rfch4.1000.nointeract, type = 1)  # type = 1 for %IncMSE

# Print the importance values
print(importance_values_ch4.1000.nointeract)

# Plot the importance values
varImpPlot(rfch4.1000.nointeract, type = 1, main = "FCH4: Variable Importance (%IncMSE), ntree = 1000")
#Results
#        %IncMSE
# tair   78.49971
# rh    106.15139
# rg     66.63705
# ws    135.24316
# wd    167.94026
# tsoil 109.61671
# vpd    67.40325
# swc   189.68935
# h      95.00363
# le     83.87560


Sys.time() - orig #stop timer --> about 7 min???; 1000 = 10 min; very quick
```
#Importance plot figure to save 
```{r}
library(dplyr)
library(ggplot2)

#Make varImport a new dataset 
# Extract variable importance
importance_values_ch4 <- importance(rfch4.2, type = 1)  # type = 1 for %IncMSE

# Convert the importance values to a data frame
CH4importance_df <- as.data.frame(importance_values_ch4)

# Add the variable names (row names) as a column
CH4importance_df$Variable <- rownames(CH4importance_df)

# Rename columns for clarity
colnames(CH4importance_df) <- c("Importance", "Variable")

# Step 1: Sort the dataframe by Importance (ascending order)
CH4importance_df <- CH4importance_df %>%
  arrange(Importance)

# Step 2: Create the ggplot with the sorted data
importance_plot_FCH4 <- ggplot(CH4importance_df, aes(x = reorder(Variable, Importance), y = Importance)) +
  # Add dotted lines aligned with variables
  geom_vline(xintercept = 1:nrow(CH4importance_df), linetype = "dotted", color = "grey70") +
  # Points and other elements
  geom_point(size = 4, color = "black", fill = "white", shape = 21) +
  labs(title = "CH4: Variable Importance (%IncMSE)", x = "Variable", y = "%IncMSE") +
  coord_flip() +
  # scale_y_continuous(breaks = seq(60, 160, by = 10)) +  # Custom breaks at every 10 units
  theme_bw() +
  theme(
    panel.grid.major = element_blank(),  # Remove all major grid lines
    panel.grid.minor = element_blank(),  # Remove all minor grid lines
    axis.text.x = element_text(angle = 0, hjust = 0.5, size = 14, face = "bold"),
    axis.text.y = element_text(size = 14, face = "bold"),
    axis.title.x = element_text(size = 16, face = "bold"),
    axis.title.y = element_text(size = 16, face = "bold"),
    plot.title = element_text(size = 20, face = "bold")
  )

# Print the plot
print(importance_plot_FCH4)


#Save image 
ggsave(
  filename = "C:/Users/kkent/Documents/Github Flux Network/Council_Flux_Analysis_Paper/Council Figures/Importance_plot_IncMSE_CH4.png",  # File name and extension
  plot = importance_plot_FCH4,   # The plot to save (default is the last plot created)
  width = 18,                 # Width in inches
  height = 10,                # Height in inches
  dpi = 600,                 # Resolution in dots per inch
  units = "cm"               # Units for width and height (can be "in", "cm", or "mm")
)

#ggsave("my_plot.pdf", width = 8, height = 6, dpi = 300, units = "in")

```


#Validation - looking at agreement between modeled data and real data 

```{r}
#changed to FCH4.c to use the re-cleaned data 

#merge datasets if to add the predicted methane to cf dataset for validation 
test.data.ch4 = merge(test.ch4,cf,by = 'date',all.x = T)

#with wd, with swc*tsoil interaction, ntree = 950
summary(lm(test.data.ch4$FCH4.c ~ test.data.ch4$rfch4.2)) #R2 = 0.485; slope = 1.018


councilch4 = ggplot(data = test.data.ch4,aes(rfch4.2,FCH4.c))+theme_bw()+
  geom_hline(yintercept = 0,lty=2)+
  geom_vline(xintercept = 0,lty=2)+
  geom_point(alpha=0.2)+
  scale_fill_viridis_c()+
  geom_abline(slope = 1,intercept = 0,col='red',lty=1)+
  scale_x_continuous(limits = c(-50,150),expression('Random Forest '*CH[4]~flux~" ("*mu*mol~CH[4]~m^-2~s^-1*')'))+
  scale_y_continuous(limits = c(-50,150),expression('Eddy Covariance '*CH[4]~flux~" ("*mu*mol~CH[4]~m^-2~s^-1*')'))+
  annotate(geom = 'text',x = 100,y = 20,label = expression(R^2~"= 0.48"),size = 3)+
  annotate(geom= 'text',x = 100,y = 10,label = expression(Slope~"= 1.018"),size = 3)+
  theme(text = element_text(size = 8))

councilch4

#with wd, with swc*tsoil interaction, ntree = 1000
summary(lm(test.data.ch4$FCH4.c ~ test.data.ch4$rfch4.1000)) #R2 = 0.483; slope = 1.017 --> same 

councilch4.1000 = ggplot(data = test.data.ch4,aes(rfch4.1000,FCH4.c))+theme_bw()+
  geom_hline(yintercept = 0,lty=2)+
  geom_vline(xintercept = 0,lty=2)+
  geom_point(alpha=0.2)+
  scale_fill_viridis_c()+
  geom_abline(slope = 1,intercept = 0,col='red',lty=1)+
  scale_x_continuous(limits = c(-50,150),expression('Random Forest '*CH[4]~flux~" ("*mu*mol~CH[4]~m^-2~s^-1*')'))+
  scale_y_continuous(limits = c(-50,150),expression('Eddy Covariance '*CH[4]~flux~" ("*mu*mol~CH[4]~m^-2~s^-1*')'))+
  annotate(geom = 'text',x = 100,y = 20,label = expression(R^2~"= 0.48"),size = 3)+
  annotate(geom= 'text',x = 100,y = 10,label = expression(Slope~"= 1.017"),size = 3)+
  theme(text = element_text(size = 8))

councilch4.1000


#with wd, *NO* interaction, ntree = 1000
summary(lm(test.data.ch4$FCH4.c ~ test.data.ch4$rfch4.1000.nointeract)) #R2 = 0.482; slope = 1.02 

councilch4.1000.nointeract = ggplot(data = test.data.ch4,aes(rfch4.1000.nointeract,FCH4.c))+theme_bw()+
  geom_hline(yintercept = 0,lty=2)+
  geom_vline(xintercept = 0,lty=2)+
  geom_point(alpha=0.2)+
  scale_fill_viridis_c()+
  geom_abline(slope = 1,intercept = 0,col='red',lty=1)+
  scale_x_continuous(limits = c(-50,150),expression('Random Forest '*CH[4]~flux~" ("*mu*mol~CH[4]~m^-2~s^-1*')'))+
  scale_y_continuous(limits = c(-50,150),expression('Eddy Covariance '*CH[4]~flux~" ("*mu*mol~CH[4]~m^-2~s^-1*')'))+
  annotate(geom = 'text',x = 100,y = 20,label = expression(R^2~"= 0.482"),size = 3)+
  annotate(geom= 'text',x = 100,y = 10,label = expression(Slope~"= 1.017"),size = 3)+
  theme(text = element_text(size = 8))

councilch4.1000.nointeract

```

#RSME of RF ntree 150 and ntree 1000
```{r}
#ntree 150, no wd
rmse_noWD<- sqrt(mean((test.data.ch4$FCH4.c - test.data.ch4$rfch4)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 150 ", rmse_noWD)) #RSME 6.11332419169469"

#ntree 150, using wd
rmse150 <- sqrt(mean((test.data.ch4$FCH4.c - test.data.ch4$rfch4.2)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 150 ", rmse150)) #RSME 5.88378022838222"


#ntree 950, using wd and swc*tsoil interaction --> FINAL MODEL 
rmse950 <- sqrt(mean((test.data.ch4$FCH4.c - test.data.ch4$rfch4.2)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 950 with interact ", rmse950)) #"RMSE: 6.66709295747304"


#ntree 1000, using wd and swc*tsoil interaction 
rmse1000 <- sqrt(mean((test.data.ch4$FCH4.c - test.data.ch4$rfch4.1000)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 1000 ", rmse1000)) #RSME 6.66709774348876" 

#ntree 1000, using wd and swc*tsoil interaction 
rmse1000.nointeract <- sqrt(mean((test.data.ch4$FCH4.c - test.data.ch4$rfch4.1000.nointeract)^2, na.rm = TRUE))
print(paste("RMSE: ntree = 1000, no interact ", rmse1000.nointeract)) #RSME 6.6739047968314" --> slightly worse, keep interaction 



```

#Plots and Validation
```{r}
#changed to FCH4.c to use re-cleaned data 
#Plots modeled vs real data among several dates to observe agreement 

ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
  labs(title = "FCH4 2019 July 1 - July 30")+
  scale_x_datetime(limits = as.POSIXct(c("2019-07-01","2019-07-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))

ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
   labs(title = "FCH4 2017")+
  scale_x_datetime(limits = as.POSIXct(c("2017-01-01","2017-12-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))

ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
  labs(title = "FCH4 2018")+
  scale_x_datetime(limits = as.POSIXct(c("2018-01-01","2018-12-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))

ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
  labs(title = "FCH4 2019")+
  scale_x_datetime(limits = as.POSIXct(c("2019-01-01","2019-12-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))

ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
  labs(title = "FCH4 2020")+
  scale_x_datetime(limits = as.POSIXct(c("2020-01-01","2020-12-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))

ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
  labs(title = "FCH4 2021")+
  scale_x_datetime(limits = as.POSIXct(c("2021-01-01","2021-12-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))

ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
  labs(title = "FCH4 2022")+
  scale_x_datetime(limits = as.POSIXct(c("2022-01-01","2022-12-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))


ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.5)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.5)+
  labs(title = "FCH4 2023")+
  scale_x_datetime(limits = as.POSIXct(c("2023-01-01","2023-12-30")))
 # scale_y_continuous(limits = c(-0.05,0.05))


ggplot(data = cf)+theme_bw()+
  geom_point(aes(date,rfch4.2,col='RF'),alpha=0.2)+
  geom_point(aes(date,FCH4.c,col='Real'),alpha=0.2)


```

#Observing agreement between modeled and real NEE and CH4 flux data 
```{r}
library(cowplot)

#png(filename = 'C:/Users/karndt.WHRC/Desktop/sites/YKD/plots/2023 yk2unburned gapfilling validation.png',width = 6,height = 4,units = 'in',res = 1000)
plot_grid(councilnee2,councilch4)
#dev.off()
```
#Rename refnee.2 and fch4.2
```{r}
library(dplyr)

#removing all the other exploratory rf nee and rf ch4 model results from the cf dataset to leave the two chosen models (which include the wd in degrees, SWC for the tussock location, swc*tsoil interaction effect, and respective optimal number of trees))

# cf2 <- cf %>%
#   select(-rfnee.nointeract, -rfnee.1000interact, -rfnee.1000, -rfch4.1000, -rfch4.1000.nointeract)
# 
# cf2 <- cf2 %>%
#   rename(rfnee = rfnee.2, rfch4 = rfch4.2)

#for when you re-run and don't include all the exploratory models 
cf <- cf %>%
  rename(rfnee = rfnee.2, rfch4 = rfch4.2)
```


#Save the gapfilled data - double check you use the correct cf vs cf2 depending on how you ran this 
```{r,error=FALSE,warning=FALSE}
#write.csv(x = cf,file = "./council_2017_2023_gf.csv",row.names = F) #_gf to indicate it's been gapfilled 
write.csv(x= cf, file = "C:/Users/kkent/Documents/Council Data/Council BASE gapfilling/council_2017_2023_gf_tuss.csv")  #added "tuss" to differentiate lichen/berries, margin pond, and tussock locations for SWC and temp 
```