--- title: "mMARCH.AC: An Open-Source R/R-Markdown Package for Processing of Accelerometry Data with GGIR in Motor Activity Research Consortium for Health (mMARCH) " author: "Wei Guo" date: '`r format(Sys.Date(), "%B %d, %Y")`' output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{mMARCH.AC} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) library(knitr) ``` # Introduction ## What is GGIR? [GGIR](https://CRAN.R-project.org/package=GGIR) is an R-package to process multi-day raw accelerometer data for physical activity and sleep research. GGIR will write all output files into two sub-directories of ./meta and ./results. GGIR is increasingly being used by a number of academic institutes across the world. ## What is mMARCH.AC? [mMARCH.AC](https://CRAN.R-project.org/package=mMARCH.AC) is an R-package to data processing after running GGIR for accelerometer data. In detail, all necessary R/Rmd/shell files were generated for data processing after running GGIR for accelerometer data. Then in module 1, all csv files in the GGIR output directory were read, transformed and then merged. In module 2, the GGIR output files were checked and summarized in one excel sheet. In module 3, the merged data was cleaned according to the number of valid hours on each night and the number of valid days for each subject. In module 4, the cleaned activity data was imputed by the average ENMO over all the valid days for each subject. Finally, a comprehensive report of data processing was created using Rmarkdown, and the report includes few explortatory plots and multiple commonly used features extracted from minute level actigraphy data in module 5-7. This vignette provides a general introduction to mMARCH.AC. ## Software Architecture The R package mMARCH.AC has been released with an open-source GPL-3 license on CRAN, and mMARCH.AC can run on Windows and Linux. Parallel computing in Linux is recommended due to the memory requirements associated with reading in multiple of the large data files. The package contains one primary function for users which, when run, generates all necessary R/R Markdown/shell executable files for data processing after running GGIR for accelerometer data; load, read, transform and merge long activity data; examine and summarize GGIR outputs; clean the merged activity data according to the number of valid hours per night and the number of valid days per subject; activity data imputation by taking the average across the valid days for each subject; build a comprehensive report of data processing and exploratory plots; extract multiple commonly used features and study feature structure by the covariance decomposition. Figure 1 presents a flowchart for each step in this process which is described in greater detail below. The procedure, R functions, inputs, and outputs are all described in this package vignette. In addition, more documentation and example data could be found in mMARCH.AC repository on GitHub (URL: https://github.com/WeiGuoNIMH/mMARCH.AC). # Mirroring the GGIR structure of processing individual data files in multiple parts, the mMARCH.AC package is split into seven modules, grouping functionalities in logical processing order. The modules are numbered from 1 to 7. modules 1 to 4 are dedicated to data processing. modules 5 to 7 are dedicated to producing R Markdown reports of data cleaning, feature extraction, and unsupervised covariance decomposition via the joint and individual variance explained (JIVE) method, respectively. These seven modules are carried out sequentially with milestone data automatically being saved locally. To use mMARCH.AC, the first step for users is to install and load the mMARCH.AC package. Then, users run the create.shell() function which creates a single R script. The newly created R script, Studyname_module0.maincall.R, is then edited by users, allowing for the specification of arguments relevant for each of the seven modules. All optional arguments and their defaults are described in the package vignette. In addition, for users with access to a cluster for parallel processing, a shell function, named as module9_swarm.sh is created which can parallelizes the processing of individual files with minor modifications by the user. These modifications are described in the package vignette. Computationally, module 1 is the most time-consuming task, taking up at least 60% of the processing time, which the activity data in .csv format was transformed and merged. Generally, module 1 takes about 10~30 minutes to process a file with 14 days of data recorded at 30 Hz on a GeneActiv device in processor cores of 36 x 2.3 GHz (Intel Gold 6140). All output created for each module is described in the package vignette. Briefly, module 1 and module 2 output are saved using a directory structure with a depth of two, containing output data and summary for all participants. The reports for modules 5 to 7 are saved in .html format and are generated using R Markdown (.Rmd) files. These .Rmd files are included in the output, users the flexibility to adapt the source code to their research purpose. # **Figure 1: Overview of main steps and output in mMARCH.AC workflow.** ```{r pdfplot, echo=F, out.width="700px", out.height="700px"} include_graphics("Figure1.workflow.pdf") ``` # ## Software Dependencies All mMARCH.AC code is written in R and reports generated in R Markdown. The R packages ActFrag and ActCR are used for the calculation of certain physical activity and circadian rhythmicity features. The R package r.jive is used to perform the feature interaction analysis and to study the joint and individual variation structure by JIVE. # # Setting up your work environment ## Install R and RStudio [Download and install R](https://cran.r-project.org/) [Download and install RStudio](https://www.rstudio.com/products/rstudio) (optional, but recommended) Download GGIR with its dependencies, you can do this with one command from the console command line: ```{R,eval=FALSE} install.packages("mMARCH.AC", dependencies = TRUE) ``` ## Prepare folder structure 1. folder of .bin files for GGIR or a file listing all .bin files - R program will check the missing in the GGIR output by comparing with all raw .bin files 2. foder of the GGIR output with two sub-folders - meta (./basic, ./csv, etc) - results (module*summary*.csv) # Quick start ## Create a template shell script of mMARCH.AC ```{R,eval=FALSE} library(mMARCH.AC) create.shell() ``` The function will create a template shell script of mMARCH.AC in the current directory, names as *STUDYNAME_module0.maincall.R*. ```{bash, eval=FALSE} cat STUDYNAME_module0.maincall.R ``` ```{r,eval=FALSE} argv = commandArgs(TRUE); print(argv) print(paste("length=",length(argv),sep="")) mode<-as.numeric(argv[1]) print(c("mode =", mode)) # (Note) Please remove the above lines if you are running this within R console # instead of submitting jobs to a cluster. ######################################################################### # (user-define 1) you need to redefine this according different study!!!! ######################################################################### # example 1 filename2id.1<-function(x) unlist(strsplit(x,"\\."))[1] # example 2 (use csv file =c("filename","ggirID")) filename2id.2<-function(x) { d<-read.csv("./mMARCH.AC/inst/extdata/example/filename2id.csv",head=1,stringsAsFactors=F) y1<-which(d[,"filename"]==x) if (length(y1)==0) stop(paste("Missing ",x," in filename2id.csv file",sep="")) if (length(y1)>=1) y2<-d[y1[1],"newID"] return(as.character(y2)) } ######################################################################### # main call ######################################################################### mMARCH.AC.shell<-function(mode,filename2id=NULL){ library(mMARCH.AC) packageVersion("mMARCH.AC") # ?mMARCH.AC.maincall # run help to see all argumengts ######################################################################### # (user-define 2) Fill in parameters of your ggir output ########################################################################## currentdir = studyname = bindir = outputdir = setwd(currentdir) rmDup=FALSE # keep all subjects in mMARCH.AC PA.threshold=c(50,100,400) part5FN="WW_L50M100V400_T5A5" epochIn = 5 epochOut = 60 use.cluster = FALSE log.multiplier = 9250 QCdays.alpha = 7 QChours.alpha = 16 QCnights.feature.alpha = c(0,0,0,0) DoubleHour= "average" QC.sleepdur.avg=NULL QC.nblocks.sleep.avg=NULL useIDs.FN=NULL Rversion="R" desiredtz="US/Eastern" RemoveDaySleeper=FALSE NfileEachBundle=20 holidayFN=NULL trace=FALSE ######################################################################### # remove duplicate sample IDs for plotting and feature extraction ######################################################################### if (mode==3 & rmDup){ # step 1: read ./summary/*remove_temp.csv file (output of mode=2) keep.last<-TRUE #keep the latest visit for each sample sumdir<-paste(currentdir,"/summary",sep="") setwd(sumdir) inFN<-paste(studyname,"_samples_remove_temp.csv",sep="") useIDs.FN<-paste(sumdir,"/",studyname,"_samples_remove.csv",sep="") ######################################################################### # (user-define 3 as rmDup=TRUE) create useIDs.FN file ######################################################################### # step 2: create the ./summary/*remove.csv file manually or by R commands d<-read.csv(inFN,head=1,stringsAsFactors=F) d<-d[order(d[,"Date"]),] d<-d[order(d[,"newID"]),] d[which(is.na(d[,"newID"])),] S<-duplicated(d[,"newID"],fromLast=keep.last) #keep the last copy for nccr d[S,"duplicate"]<-"remove" write.csv(d,file=useIDs.FN,row.names=F) } ######################################################################### # call afterggir ######################################################################### setwd(currentdir) mMARCH.AC.maincall(mode=mode, useIDs.FN=useIDs.FN, currentdir=currentdir, studyname=studyname, bindir=bindir, outputdir=outputdir, epochIn=epochIn, epochOut=epochOut, log.multiplier=log.multiplier, use.cluster=use.cluster, QCdays.alpha=QCdays.alpha, QChours.alpha=QChours.alpha, QCnights.feature.alpha=QCnights.feature.alpha, DoubleHour= DoubleHour, QC.sleepdur.avg=QC.sleepdur.avg, QC.nblocks.sleep.avg=QC.nblocks.sleep.avg, Rversion=Rversion, filename2id=filename2id, PA.threshold=PA.threshold, desiredtz=desiredtz, RemoveDaySleeper=RemoveDaySleeper, part5FN=part5FN, NfileEachBundle=NfileEachBundle, holidayFN=holidayFN, trace=trace) } ######################################################################### mMARCH.AC.shell(mode) ######################################################################### # Note: mMARCH.AC.shell(mode) # mode = 0 : creat sw/Rmd file # mode = 1 : data transform using cluster or not # mode = 2 : summary # mode = 3 : clean # mode = 4 : impu ``` ## Edit shell script Three places were marked as "user-define" and need to be edited by user in the *STUDYNAME_module0.maincall.R* file. Please rename the file by replacing your real studyname after the edition. ### 1. Define the function filename2id( ) This user-defined function will change the filename of the raw accelerometer file to the short ID. For example, the first example change "0002__026907_2016-03-11 13-05-59.bin" to new ID of "0002". If you prefer to define new ID by other way, you could create a .CSV file including "filename" and "newID" at least and then defined this function as the second example. The new variable of "newID", included in the output files, could be used as the key ID in the summary report of mMARCH.AC and be used to define the duplicate samples as well. ### 2. Parameters of mMARCH.AC.maincall() User needs to define the following parameters as follows, Variables | Description ----------------- | ---------------------------------------------------- rmDup | Set rmDup = TRUE if user want to remove some samples such as duplicates. Set rmDup = FALSE if user want to keep all samples. mode | Specify which of the five modules need to be run, e.g. mode = 0 makes that all R/Rmd/sh files are generated for other modules. When mode = 1, all csv files in the GGIR output directory were read, transformed and then merged. When mode = 2, the GGIR output files were checked and summarized in one excel sheet. When mode = 3, the merged data was cleaned according to the number of valid hours on each night and the number of valid days for each subject. When mode = 4, the cleaned data was imputed. useIDs.FN | Filename with or without directory for sample information in CSV format, which including "filename" and "duplicate" in the headlines at least. If duplicate="remove", the accelerometer files will not be used in the data analysis of module 5-7. Defaut is NULL, which makes all accelerometer files will be used in module 5-7. currentdir | Directory where the output needs to be stored. Note that this directory must exist. studyname | Specify the study name that used in the output file names bindir | Directory where the accelerometer files are stored or list outputdir | Directory where the GGIR output was stored. epochIn | Epoch size to which acceleration was averaged (seconds) in GGIR output. Defaut is 5 seconds. epochOut | Epoch size to which acceleration was averaged (seconds) in module1. Defaut is 5 seconds. flag.epochOut | Epoch size to which acceleration was averaged (seconds) in module 3. Defaut is 60 seconds. log.multiplier | The coefficient used in the log transformation of the ENMO data, i.e. log( log.multiplier * ENMO + 1), which have been used in module 5-7. Defaut is 9250. use.cluster | Specify if module1 will be done by parallel computing. Default is TRUE, and the CSV file in GGIR output will be merged for every 20 files first, and then combined for all. QCdays.alpha | Minimum required number of valid days in subject specific analysis as a quality control step in module2. Default is 7 days. QChours.alpha | Minimum required number of valid hours in day specific analysis as a quality control step in module2. Default is 16 hours. QCnights.feature.alpha | Minimum required number of valid nights in day specific mean and SD analysis as a quality control step in the JIVE analysis. Default is c(0,0), i.e. no additional data cleaning in this step. DoubleHour | Specify the method of processing the double hours for days that daylight saving time starts and ends for example. In detail, DoubleHour = c("average","earlier","later"). The acceleration data was averaged on double hours when DoulbeHour="average". Only the acceleration data in the earlier occurrence was remained for double hours while the other duplicate data were ignored when DoulbeHour="earlier". Only the acceleration data in the later occurrence was remained for double hours while the other duplicate data were ignored when DoulbeHour="later". Default is "average". QC.sleepdur.avg | As taking the deault value of QC.sleepdur.avg=c(3,12), individuals were excluded with an average sleep duration <3 hour or >12 hour. QC.nblocks.sleep.avg | As taking the deault value of QC.nblocks.sleep.avg=c(5,30), individuals were excluded with an average number of nocturnal sleep episodes <5 or >30. Rversion | R version, eg. "R/3.6.3". Default is "R". filename2id | User defined function for converting filename to sample IDs. Default is NULL. PA.threshold | Threshold for light, moderate and vigorous physical activity. Default is c(50,100,400). desiredtz | desired timezone: see also https://en.wikipedia.org/wiki/Zone.tab. Used in g.inspectfile(). Default is "US/Eastern". RemoveDaySleeper | Specify if the daysleeper nights are removed from the calculation of number of valid days for each subject. Default is FALSE. part5FN | Specify which output is used in the GGIR part5 results. Defaut is "WW_L50M125V500_T5A5", which means that part5_daysummary_WW_L50M125V500_T5A5.csv and part5_personsummary_WW_L50M125V500_T5A5.csv are used in the analysis. NfileEachBundle | Number of files in each bundle when the csv data were read and processed in a cluster. Default is 20. trace | Specify if the intermediate results is printed when the function was executed. Default is FALSE. ### 3. Subset of samples (optional) The mMARCH.AC package not only simply transform/merge the activity and sleep data, but it also can do some prelimary data analysis such as principle componet analysis and feature extraction. Therefore, the basic data clean will be processed first as follows, - data clean by removing valid days/samples defined by minimum required number of valid hours/days in the activity data - remove duplicate samples If you prefer to use all samples, just skip this module and use `rmDup=FALSE` as the default. Otherwise, if you want to remove some samples such as duplicates, there are two ways as follows, - Edit R codes of "step 2" in this module. For example, the template will keep the later copy for duplicate samples - Remove R codes of "step 2" in this module, and create *studyname_samples_remove.csv* file by filling "remove" in the "duplicate" column in the template file of *studyname_samples_remove_temp.csv*. The data will be kept unless duplicate="remove". ## Run R script ```{R,eval=FALSE} mMARCH.AC.shell(mode,filename2id) ``` Variables | Description ----------------- | ---------------------------------------------------- mode | Specify which of the five modules need to be run, e.g. mode = 0 makes that all R/Rmd/sh files are generated for other modules. When mode = 1, all csv files in the GGIR output directory were read, transformed and then merged. When mode = 2, the GGIR output files were checked and summarized in one excel sheet. When mode = 3, the merged data was cleaned according to the number of valid hours on each night and the number of valid days for each subject. When mode = 4, the cleaned data was imputed. filename2id | This user-defined function will change the filename of the raw accelerometer file to the short ID for the purpose of identifying duplicate IDs. ## Run script in a cluster ```{R,eval=FALSE} #!/bin/bash # #$ -cwd #$ -j y #$ -S /bin/bash source ~/.bash_profile module load R ; R --no-save --no-restore --args < studyname_module0.maincall.R 0 R --no-save --no-restore --args < studyname_module0.maincall.R 1 R --no-save --no-restore --args < studyname_module0.maincall.R 2 R --no-save --no-restore --args < studyname_module0.maincall.R 3 R --no-save --no-restore --args < studyname_module0.maincall.R 4 R -e "rmarkdown::render('module5_studyname_Data_process_report.Rmd' )" R -e "rmarkdown::render('module6_studyname_NonWear.report.Rmd' )" R -e "rmarkdown::render('module7a_studyname_calculate_newfeatures.Rmd' )" R -e "rmarkdown::render('module7b_studyname_merge_GGIRfeatures.Rmd' )" R -e "rmarkdown::render('module7c_studyname_runJIVE.Rmd' )" R -e "rmarkdown::render('module7d_studyname_calculate_WD_WE_avg_features.Rmd' )" ``` # # # Software Functionalities ## Module 1 The functions of Module 1 read, merge, and transform the lower resolution summary of raw acceleration data for all subjects. Raw data are summarized using Euclidian Norm Minus One (ENMO) (Migueles et al., 2019), a rotationally invariant measure of volume of acceleration, in epochs specified by the user in their call to GGIR. Merged data, originally in long format, are saved in wide format, an intuitive data structure for accelerometer data (Leroux 2019). Depending on the time zone on which the devices were initialized, a day may have between 23 and 25 hours due to daylights savings time. On daylight savings crossovers, ENMO is averaged for duplicate timestamps between 1:00 AM and 2:00 AM, allowing for straightforward comparisons to standard 24-hour days. In addition to recording ENMO at each epoch, the angle of the z-axis (ANGLEZ) relative to the horizontal plane (degrees), used for estimating sleep periods, is merged and saved into an excel file. Additional data recorded by certain devices are merged for all participants and saved under the ./data directory including the light mean, light peak, temperature mean, clipping score, and the Euclidian norm metric (EN) if available. The activity data in the GGIR output is formatted in long csv-spreadsheets as follows, ```{r,echo=F} input<-rbind(c("timestamp ","ENMO ","anglez "), c("2017-11-30T00:00:00+0100",8e-04,-32.5758), c("2017-11-30T00:00:05+0100",0.0198,-25.5726), c("2017-11-30T00:00:10+0100",0.0177,3.7972), c("2017-11-30T00:00:15+0100",0.0118,6.7154), c("2017-11-30T00:00:20+0100",0.0106,10.0357), c("2017-11-30T00:00:25+0100",0.0341,21.0143), c("2017-11-30T00:00:30+0100",0.1708,19.5008), c("......","......","......"), c("2017-11-30T23:59:55+0100",0.1504,-0.596)) output<-rbind(c( "Date","0:00:00","0:00:05","0:00:10","0:00:15","0:00:20","0:00:25","0:00:30","......","23:59:55"), c( "11/30/2017","8.00E-04",0.0198,0.0177,0.0118,0.0106,0.0341,0.1708,"......",0.1504)) kable(input,format = "html") ``` Each row represents the corresponding ENMO and ANGLEZ values at a timestamp per 5 seconds epoch, which is specified by GGIR parameter (windowsizes) when running GGIR. After running Module 1, the ENMO and ANGLEZ data are transformed into wide matrix in which each row represents 24 hours data for a day. For example, the ENMO data is formated as follows, ```{r,echo=F} kable(output) ``` Finally, the data was merged for all days and all subjects. ## Module 2 The functions of Module 2 create descriptive variables of all accelerometer files output by GGIR. In Module 2, an excel file is output under the ./summary directory, which includes ten pages as follows, (1) List of files in the GGIR output (2) Summary of numbers of output files (3) List of duplicate IDs (4) ID errors (5) Number of valid days (6) Table of number of valid/missing days (7) Missing pattern (8) Frequency of the missing pattern (9) Description of all accelerometer files (10) device information. Multiple plots are generated in a pdf file including the number of valid hours, days, missing pattern, etc. First, the completeness of GGIR run is inspected by detecting the missing files in each GGIR step when user specified the path of GGIR output. Additionally, the descriptive variables are created to form a comprehensive data quality report based on summary results generated by GGIR for the purpose of examining data quality in the Module 2 of mMARCH.AC. ## Module 3 and Module 4 The functions of Module 3 introduce 'flag' variables for data cleaning of the merged ENMO and ANGLEZ data. By default, days with more than 16 hours are marked as valid days and subjects with more than 7 valid days are marked as valid samples. Users can set these two parameters in the call to the main function (QCdays.alpha = 7, QChours.alpha = 16). Further, the data can be aggregated (averaged) in lower resolution epochs (e.g. minute-level or hour-level) as desired by users. In Module 4, ENMO data during estimated non-wear periods are imputed by taking the subject-level mean over all the valid days for each subject at that time, limiting potential bias in imputation due to diurnal patterns of activity. All output is saved under the ./data directory. In the output file, the following description variables are included: (1) number of valid hours; (2) missing pattern for each subject; (3) non-wear time in minutes; (4) an indicator variable to indicate if the visit should be removed for having multiple visits for some subjects which might lead to invalidity of independent and identically distribution in statistics and (5) the number of missing values after imputation. When the number of missing values after imputation is not zero, it means the activity data is missing on same timestamps among all days and therefore could not be imputed, therefore, such samples would be removed when systematic missingness was observed. ## Module 5 The functions of Module 5 generate a comprehensive report in .html format. This report includes data quality checks and an exploratory data analysis using valid days of data. First, the numbers and missingness of GGIR output files are summarized for all GGIR parts. Second, duplicate samples are checked and marked, where the duplication might be caused by having multiple visits for some samples but only one visit will be kept in the data analysis such as functional principal component analysis (FPCA). Third, as shown in Figure 1, data quality is presented visually using the number of valid days, non-wear time, and missing data pattern. As an exploratory analysis, the data correlation, and the output of FPCA analysis are plotted in the report. ## Module 6 (Optional) Estimated non-wear periods are loaded from the $M\$metalong\$nonwearscore$ variable of the R data that was stored in the folder of /meta/basic of the GGIR output, which generate the matrix to clarify when data was imputed for each long epoch time window and the reason for imputation. This function will generate a non-wear matrix at minute level, coded as 0/1 for wear/non-wear time. This function will generate a non-wear matrix at minute level, and it could be skipped if user chose to use the imputation data in the JIVE application as default. ## Module 7 In Module 7, 88 features were extracted from minute level activity data of three domains of sleep, physical activity, and circadian rhythmicity, which were based on outputs from GGIR v2.4.0 and calculated by R ActFrag and ActCR packages. The standard deviation across days on each subject was also created for each feature. The weekday and weekend specific features were extracted as well since most features in the sleep and physical activity showed significant difference between weekdays (Monday to Friday) and weekends (Saturday and Sunday). Specially, the sleep features on weekends are calculated based on sleep periods from Friday night to Saurday morning, Saturday night to Sunday morning, and last night to a holiday morning. Furthermore, holdiays could be counted as weekends if user provide the holiday information through the argument of *holidayFN* in the *afterggir()* function. In brief, sleep domain referred to sleep duration, midpoint, efficiency, etc. Physical activity referred to daily motor activity such as sedentary behavior, light, and moderate-to-vigorous physical activity (MVPA). Circadian rhythms were natural rhythms that regulates the sleep-wake cycle within every 24 hours. For example, the cosinor curve and FPCA analysis were used in modeling of biological rhythms. A comprehensive list of all features could be found in the supplementary table and more detailed definition could be found in the mMARCH.AC paper, GGIR manual and Di et al.'s publication in 2019. All features listed are calculated using minute-level activity data by default. Several duplicate features were also extracted from GGIR results as an additional data source. As an integral module of mMARCH.AC package, we proposed to use a multi-modal integrative technique, Joint and Individual Variance Explained (JIVE), to efficiently and simultaneously deal with multiple features representing physical activity (PA), sleep (SL), and circadian rhythmicity (CR) (Di et al., 2019; Lock et al., 2013; O'Connell and Lock, 2016). The integrative analysis of multiple derived variables in each of these three domains assessed by accelerometry is applied to explain the joint and individual variation in these three domains. Conceptually, JIVE assumes four groups of latent variables that generate: i) individual variation captured by PA, SL, CR; and ii) joint variation by PA, SL, CR. These latent variables (JIVE scores) can be estimated and then used as novel measures capturing both individual and joint variability of measures of sleep, physical activity and circadian rhythmicity. JIVE has been applied to multivariate actigraphy derived features to address the interdependence between domains of physical activity, sleep, and circadian rhythm (Di et al. 2019). JIVE decomposition utilized the subject level means of all 88 features including those duplicated features from GGIR outputs. Heatmaps of the estimated joint structure of the full JIVE decomposition are created, with examples shown in Figure 1. JIVE scores are estimated for both joint and individual variation structures. Building the activity features could enrich the markers for understanding genetic contribution of human diseases. # # Running mMARCH.AC and Inspecting the results ## Input and output of Module 0 * Command = mMARCH.AC.shell(mode=0) * Output folder = ./ Output | Description -------------- | ------------------------------------------------ module1a_data.transform.R (use.cluster=TRUE, optional)| R code for data transformation and merge for every 20 files in each partition. When the number of .bin files is large ( > 1000), the data merge could take long time, user could split the job and submit the job to a cluster for parallel computing. module1b_data.transform.sw (use.cluster=TRUE, optional)| Submit the job to a cluster for parallel computing module1c_data.transform.merge.sw (use.cluster=TRUE, optional)| Merge all partitions for the ENMO and ANGLEZ data module5_studyname_Data_process_report.Rmd |R markdown file for generate a comprehensive report of data processing and explortatory plots. module6_studyname_NonWear.report.Rmd |R markdown file for generate a report of nonwear score. module7a_studyname_calculate_newfeatures.Rmd |Extract some features from the actigraphy data using R module7b_studyname_merge_GGIRfeatures.Rmd |Extract other features from the GGIR output and merge all features together module7c_studyname_runJIVE.Rmd |Perform JIVE Decomposition for All Features using r.jive module7d_studyname_calculate_WD_WE_avg_features.Rmd |Extract some weekday/weekend specific features from the actigraphy data using R module9_swarm.sh |shell script to submit all jobs to the cluster ## Input and output of Module 1 + Main input files: csv files under /meta/csv folders of GGIR output + Command = mMARCH.AC.shell(mode=1) + Output folder = ./data Output | Description -------------- | --------------------------------------- studyname_filesummary_csvlist.csv | File list in the ./csv folder of GGIR studyname_filesummary_Rdatalist.csv | File list in the ./basic folder of GGIR All_studyname_ANGLEZ.data.csv | Raw data of ANGLEZ after merge All_studyname_ENMO.data.csv | Raw data of ENMO after merge nonwearscore_studyname_f0_f1_Xs.csv | Data matrix of nonwearscore nonwearscore_studyname_f0_f1_Xs.pdf | Plots for nonwearscore plot.nonwearVSnvalidhours.csv | Nonwear data for plot plot.nonwearVSnvalidhours.pdf | Nonwear plots lightmean_studyname_f0_f1_Xs.csv | Data matrix of lightmean lightpeak_studyname_f0_f1_Xs.csv | Data matrix of lightpeak temperaturemean_studyname_f0_f1_Xs.csv | Data matrix of temperaturemean clippingscore_studyname_f0_f1_Xs.csv | Raw data of clippingscore EN_studyname_f0_f1_Xs.csv | Data matrix of EN *f0 and f1 are the file index to start and finish with *Xs is the epoch size to which acceleration was averaged (seconds) in GGIR output ## Input and output of Module 2 + Main input files + ./data/All_studyname_ENMO.data.csv + GGIR results: part2, part4 and part5 (please specify $part5FN$ in the main function) + GGIR raw data when bindir was specified + Command = mMARCH.AC.shell(mode=2) + Output folder = ./summary Output | Description -------------- | ----------------------------------------- studyname_ggir_output_summary.xlsx | Description of all accelerometer files in the GGIR output. This excel file includs 9 pages as follows, (1) List of files in the GGIR output (2) Summary of files (3) List of duplicate IDs (4) ID errors (5) Number of valid days (6) Table of number of valid/missing days (7) Missing patten (8) Frequency of the missing pattern (9) Description of all accelerometer files. part2daysummary.info.csv | Intermediate results for description of each accelerometer file. studyname_ggir_output_summary_plot.pdf | Some plots such as the number of valid days, which were included in the module5_studyname_Data_process_report.htm file as well. studyname_samples_remove_temp.csv | Create studyname_samples_remove.csv file by filling "remove" in the "duplicate" column in this template. If duplicate="remove", the accelerometer files will not be used in the data analysis of module5. ## Input and output of Module 3 + Main input file: ./data/All_studyname_ENMO.data.csv + Command = mMARCH.AC.shell(mode=3) + Output folder = ./data Output | Description -------------- | -------------------------------------- flag_All_studyname_ANGLEZ.data.Xs.csv | Adding flags for data cleaning of the raw ANGLEZ data flag_All_studyname_ENMO.data.Xs.csv | Adding flags for data cleaning of the raw ENMO data IDMatrix.flag_All_studyname_ENMO.data.60s.csv | ID matrix *Xs is the epoch size to which acceleration was averaged (seconds) in GGIR output ## Input and output of Module 4 + Main input file: ./data/flag_All_ studyname _ENMO.data.5s.csv + Command = mMARCH.AC.shell(mode=4) + Output folder = ./data Output | Description -------------- | -------------------------------------------------------------------------------- impu.flag_All_studyname_ENMO.data.60s.csv | Imputation data for the merged ENMO data, and the missing values were imputated by the average ENMO over all the valid days for each subject. ## Description of flag variables in the output data Variable | Description -------------- | -------------------------------------------------------------------------------- filename | accelerometer file name Date | date recored from the GGIR part2.summary file id | IDs recored from the GGIR part2.summary file calender_date | date in the format of yyyy-mm-dd N.valid.hours | number of hours with valid data recored from the part2_daysummary.csv file in the GGIR output N.hours | number of hours of measurement recored from the part2_daysummary.csv file in the GGIR output weekday | day of the week-Day of the week measurementday | day of measurement-Day number relative to start of the measurement newID | new IDs defined as the user-defined function of filename2id(), e.g. substrings of the filename missing | "M" indicates missing for an invalid day, and "C" indicates completeness for a valid day Ndays | number of days of measurement ith_day | rank of the measurementday, for example, the value is 1,2,3,4,-3,-2,-1 for measurementday = 1,...,7 Nmiss | number of missing (invalid) days Nnonmiss | number of non-missing (valid) days misspattern | indicators of missing/nonmissing for all measurement days at the subject level RowNonWear | number of columnns in the non-wearing matrix NonWearMin | number of minutes of non-wearing remove16h7day | indicator of a key qulity control output. If remove16h7day=1, the day need to be removed. If remove16h7day=0, the day need to be kept. duplicate | If duplicate="remove", the accelerometer files will not be used in the data analysis of module5. ImpuMiss.b | number of missing values on the ENMO data before imputation ImpuMiss.a | number of missing values on the ENMO data after imputation KEEP | The value is "keep"/"remove", e.g. KEEP="remove" if remove16h7day=1 or duplicate="remove" or ImpuMiss.a>0 ## Input and output of Module 5 + Main input files + ./summary/studyname_ggir_output_summary.xlsx + ./summary/part24daysummary.info.csv + ./data/plot.nonwearVSnvalidhours.csv + ./data/impu.flag_All_studyname_ENMO.data.flag.epochOuts.csv + Command: run module5_studyname_Data_process_report.Rmd + Output folder = ./ Output | Description -------------- | -------------------------------------------------------------------------------- module5_studyname_Data_process_report.html | A comprehensive report of data processing and explortatory plots. ## Input and output of Module 6 + Main input file: nonwearscore_studyname_01_xx_900s.csv + Command = run module6_studyname_NonWear.report.Rmd + Output folder = ./ Folder | Output | Description -------------- | -------------- | -------------------------------------------------------------------------------- ./ |module6_studyname_NonWear.report.html | A report of nonwear score. ./data | JIVEraw_nonwearscore_studyname_1_5_Xs.csv | Imputation data matrix of nonwearscore (1/0) ./data | JIVEimpu_nonwearscore_studyname_1_5_Xs.csv | Data matrix of nonwearscore (1/0/NA) *f0 and f1 are the file index to start and finish with *Xs is the epoch size to which acceleration was averaged (seconds) in GGIR output ## Input and output of Module 7a + Main input + ./data/impu.flag_All_studyname_ENMO.data.flag.epochOuts.csv + GGIR: /results/QC/part4_nightsummary_sleep_full.csv + Command = run module7a_studyname_calculate_newfeatures.Rmd + Output folder = ./ Output | Description -------------- | -------------------------------------------------------------------------------- module7_studyname_all_features_dictionary.xlsx | Description of features module7a_studyname_calculate_newfeatures.html | Extract some features from the actigraphy data using R module7a_studyname_new_features_page1_features.csv | List of some features module7a_studyname_new_features_page2_face_day_PCs.csv | Function PCA at the day level using fpca.face( ) module7a_studyname_new_features_page3_face_subject_PCs.csv | Function PCA at the subject level using fpca.face( ) module7a_studyname_new_features_page4_denseFLMM_day_PCs.csv | Function PCA at the day level using denseFLMM( ) module7a_studyname_new_features_page5_denseFLMM_subject_PCs.csv | Function PCA at the subject level using denseFLMM( ) ## Input and output of Module 7b + Main inputs + GGIR: part2_summary.csv + GGIR: part2_daysummary.csv + GGIR: part4_nightsummary_sleep_cleaned.csv + GGIR: /results/QC/part4_nightsummary_sleep_full.csv + GGIR: part5_daysummary_part5FN.csv + module7_studyname_all_features_dictionary.xlsx + Command = run module7b_studyname_merge_GGIRfeatures.Rmd + Output folder = ./ Output | Description -------------- | -------------------------------------------------------------------------------- module7b_studyname_merge_GGIRfeatures.html | Extract other features from the GGIR output and merge all features together module7b_studyname_all_features_1_day.csv | Raw data of all features module7b_studyname_all_features_2_dayclean.csv | Keep sample with valid ENMO inputs module7b_studyname_all_features_2_dayclean.csv.log | Log file of each variable of module5b_studyname_all_features_2_dayclean.csv plot_module7b_studyname_all_features_2_dayclean.csv.pdf |Plot of each variable of module5b_studyname_all_features_2_dayclean.csv module7b_studyname_all_features_3_subject.csv |Average variable at the subject level module7b_studyname_all_features_3_subject.csv.log | Log file of each variable of module5b_studyname_all_features_3_subject.csv plot_module7b_studyname_all_features_3_subject.csv.pdf | Plot of each variable of module5b_studyname_all_features_3_subject.csv module7b_studyname_all_features_4_subjectSD.csv |subject level SD of each feature ## Input and output of Module 7c + Main inputs + module7b_studyname_all_features_3_subject.csv + module7b_studyname_all_features_4_subjectSD.csv + Command = run module7b_studyname_merge_GGIRfeatures.Rmd + Output folder = ./ Output | Description -------------- | -------------------------------------------------------------------------------- module7c_studyname_runJIVE.html | Perform JIVE Decomposition for All Features using r.jive module7c_studyname_jive_Decomposition.csv | Joint and individual structure estimates module7c_studyname_jive_predScore.csv |PCA scores of JIVE ( missing when jive.predict failes) ## Input and output of Module 7d + Main input: ./data/impu.flag_All_studyname_ENMO.data.flag.epochOuts.csv + Command = run module7d_studyname_calculate_WD_WE_avg_features.Rmd + Output folder = ./ Output | Description -------------- | ------------------------------------- module7d_studyname_calculate_WD_WE_avg_features.html | Extract some weekday/weekend specific features from the actigraphy data using R module7d_studyname_new_features_page1.csv | Perform JIVE Decomposition for All Features using r.jive module7d_weekday_studyname_all_features_3_subject.csv | subject level mean of each feature on weekday module7d_weekday_studyname_new_features_page4_denseFLMM_day_PCs.csv | Function PCA at the day level using denseFLMM( ) on weekday module7d_weekday_studyname_new_features_page5_denseFLMM_subject_PCs.csv | Function PCA at the subject level using denseFLMM( ) on weekday module7d_weekend_studyname_all_features_3_subject.csv | subject level mean of each feature on weekend module7d_weekend_studyname_new_features_page4_denseFLMM_day_PCs.csv | Function PCA at the day level using denseFLMM( ) on weekend module7d_weekend_studyname_new_features_page5_denseFLMM_subject_PCs.csv | Function PCA at the subject level using denseFLMM( ) on weekend ## Description of features of domains of physical activity, sleep and circadian rhythmicity **Sleep Domain** ```{r s1,echo=F} library(xlsx) library(knitr) library(kableExtra) feaFN<-system.file("template", "features.dictionary.xlsx", package = "mMARCH.AC") #feaFN<-"/data/guow4/project0/GGIR/postGGIR/postGGIR_compile/v2/mMARCH.AC/inst/template/features.dictionary.xlsx" dict<-read.xlsx(feaFN,head=1,sheetName="dictionary",stringsAsFactors=F) dict.SL<-dict[which(dict[,"Domain"]=="SL"),c("Variable","Description","level","Source")] dict.PA<-dict[which(dict[,"Domain"]=="PA"),c("Variable","Description","level","Source")] dict.CR<-dict[which(dict[,"Domain"]=="CR"),c("Variable","Description","level","Source")] row.names(dict.SL)<-NULL row.names(dict.PA)<-NULL row.names(dict.CR)<-NULL kable(dict.SL) %>% kable_styling(bootstrap_options = c("striped", "hover")) ``` **Physical Activity Domain** ```{r s2,echo=F} kable(dict.PA) %>% kable_styling(bootstrap_options = c("striped", "hover")) ``` **Circadian Rhythmicity Domain** ```{r s3,echo=F} kable(dict.CR) %>% kable_styling(bootstrap_options = c("striped", "hover")) ``` Note (1) The column of level indicates the feature was calculated as subject level or day level. (2) The column of source indicates the feature source. R indiciates the feature was calculated based on activity data. Part2, part4 and part5 in 'Source' column indicate the feature was extracted from summary results from GGIR outputs. ```{r s4,eval=FALSE,include=FALSE} d1<-read.xlsx("mMARCH.AC.output.description.xlsx",sheetName="output.format") cd /data/guow4/project0/GGIR/postGGIR/postGGIR_compile/v2/mMARCH.AC/vignettes R -e "rmarkdown::render('mMARCH.AC.Rmd' )" ``` # Citing mMARCH.AC + Guo W., Leroux A., Shou S., Cui L., Kang S., Strippoli M.P., Preisig M., Zipunnikov V., Merikangas K. (2022) Processing of accelerometry data with GGIR in Motor Activity Research Consortium for Health (mMARCH). (Manuscript) # Reference + Di J., Spira A., Bai J., Urbanek J., Leroux A., Wu M., ..., Zipunnikov V. (2019). Joint and individual representation of domains of physical activity, sleep, and circadian rhythmicity. Statistics in Biosciences, 11(2), 371-402. doi:10.1007/s12561-019-09236-4 + Lock E.F., Hoadley K.A., Marron J.S., Nobel A.B. (2013). Joint and individual variation explained (JIVE) for integrated analysis of multiple data types. Ann Appl Stat 7(1), 523-542. https://doi.org/10.1214/12-AOAS597 + O'Connell M.J., Lock E.F. (2016). R.JIVE for exploration of multi-source molecular data. Bioinformatics, 32(18), 2877-2879. # Websites + mMARCH.AC download: https://cran.r-project.org/package=mMARCH.AC + mMARCH.AC vignette: https://cran.r-project.org/package=mMARCH.AC + Example data and documents: https://github.com/WeiGuoNIMH/mMARCH.AC Please contact wei.guo3@nih.gov to ask questions.