Chapter 11 Data Files
The organization of a lasertrapr project folder is: LASERTRAPR -> PROJECT -> CONDITIONS -> DATE -> OBS
The application picks the location of the lasertrapr for you and this cannot be moved. The app automatically saves data and configuration files in specific location. The following are descriptions of the types of files that can be found within your lasertrapr folder. All files are plain text (.csv) files that can be opened in any text editor or read into R/python/etc easily.
11.1 Observation folder files
There are several files you will find within an observation folder:
11.1.1 trap-data.csv
When you upload your trapping data traces, the files are copied and saved into a new “obs-##” folder. The actual raw data is saved in a filed called “trap-data.csv”. Column descriptions:
The project, conditions, date, and obs folders are legacy columns from early versions of the app.
raw_bead: the exact data from the uploaded file, copied verbatim. For 2-channel datasets, you will have raw_bead_1 and raw_bead_2. This is typically your data in volts, but for Lumicks data, they supply their data in pN, so the raw_data columns is in pN for Lumicks data.
trap_position: This column is special to data uploaded from the Muscle Biophysics Lab at UMass Amherst for their custom uploader. This is the position of the AOD. The value will be 0 for non-feedback experiments.
processed_bead: This is the results of the raw_data after being “processed” by the “Clean & Process” tab. This is the data that is used by the analyzers. Note for 2 channel data, you will have processed_bead_1 and processed_bead_2.
rescaled_mini_data: Mini-ensemble data only. This is the processed_bead data that has had the baseline+event micro-baseline corrections applied during the mini-ensemble analysis. Therefore, the data has been “rescaled”.
run_mean_overlay: Mini-ensemble data only. This is the calculated running mean of the rescaled mini data.
aod_position: Greenberg lab data only.
feedback_filter_position: Greenberg lab data only
11.1.2 measured-events.csv
These “measured-events” files save the output of the analyzers. Each row is one event and columns are variables describing different aspects of the event:
displacement_nm: the displacement of the event in nanometers. Relative measurement for HM/CP based analyzers, absolute measurement for the mini analyzer. For IFC data, this is the displacement of the motor bead.
absolute_displacements_nm: IFC only. Absolute force displacement position. Traces are usually not “Zero’d”.
baseline_position_prior_nm*: IFC only. the mean baseline position for a few milliseconds before the start of the event. Used to make a relative displacement calculation (what is reported as displacement_nm).
force_pn. Or just “force” for files generated by older versions of the app. This is the force in piconewtons calculated by multiplying displacement_nm by nm2pn.
time_on_ms. The total attachment duration in milliseconds. Fitting these rates gives a detachment rate for the specific conditions the data collected under.
time_on_s: IFC only. Time on in seconds.
time_off_ms. The baseline duration before the event in milliseconds. The first event is always NA because there is no previous event to determine the “time between events”. Fitting these values gives an apparent attachment rate.
analyzer: reporting what analyzer generated this file.
index: event number
hmm_state: HM-Model only, should be 2.
num_windows: the attachment duration in window width length.
length_hz: crude conversion between the number of windows and hz in datapoints. A crude estimate of time on. Not recommended to use.
start: HM/CP models only. The start in datapoints of the event identified by the changepoint algorithm.
stop: HM/CP models only. The stop in datapoints of the event identified by the changepoint algorithm.
cp_found_start: TRUE/FALSE indicating if the changepoint analysis was successful in running at the start of the event.
cp_found_stop: TRUE/FALSE indicating if the changepoint analysis was successful in running at the stop of the event.
is_positive: HM/CP models only. Whether the event is positive (TRUE) or not (FALSE).
keep: analyzers decision on whether to keep/trust this measurement. TRUE/FALSE.
cp_displacements: HM/CP only. Analyzers first estimate displacement, force, time on, time off from the HM model/running window data only before applying the changepoint and re-measuring these parameters. The changepoint measurements are better and then overwrite the results of the HM Model estimates. So this should be the same as displacement_nm.
cp_forces: should be same as force_pn. Same as above.
trap_stiffness: Equipartition value of the baseline noise before the event. Note, this is calculated on a pre-tensed filament so is a “system” stiffness not necessarily trap stiffness.
myo_stiffness: Equipartition value of the whole changepoint identified event. Attempt to quantify the stiffness of the binding duration. Perhaps a relative change calculation could be more meaningful between the trap_stiffness and myo_stiffness due to effects of pre-tension on these metrics.
front_signal_ratio: not really used for anything. The signal to noise ratio of the events compared to the baseline before for the start of the event.
back_signal_ratio: not really used for anything. The signal to noise ratio of the events compared to the baseline before for the end of the event.
cp_time_on_dp: changepoint measurement of the attachment duration in datapoints.
cp_time_on_ms: the cp_time_on_dp divided by hz/1000 to get time on in milliseconds. should be the same as time_on_ms.
hm_event_start: the data point estimation of the start of the event from the HM-Model running window data.
hm_event_stop: the data point estimation of the start of the event from the HM-Model running window data.
cp_event_start_dp. HM/CP models only. A better named start column indicating its from the changepoint analysis and the units are datapoints. Same as start
cp_event_stop_dp. HM/CP models only. A better named stop column indicating its from the changepoint analysis and the units are datapoints. Same as stop.
conversion: an estimate conversion factor between running window time and real time.
peak_nm_index: For HM/CP models - the time in seconds at which the peak displacement is calculated for displacement method = “peak”. For methods = “avg”, it is an estimated time of the events running window that has the greatest value. For method = “avg” this is not particularly useful for any other analysis except the app uses it to place the displacement markers in the analyzed traces over the events. For mini-ensemble, this the peak displacement in datapoints.
mid_point_index_dp: IFC only. Similar to peak_nm_index. Used to put the displacement marker in the middle of the event during analyzer visualization.
event_user_excluded. TRUE/FALSE. Did the user manually exclude this event? FALSE means keep in further analysis. TRUE means user does not want event, filter it out.
11.1.3 hm-model-data.csv
This is data related to the running window transformation and HM-Model for those analyzers.
run_mean: the running mean signal.
run_var: the running variance signal.
state: 1 or 2 indicating HM-Model state classification.
index: window number.
var_signal_ratio: the same number for every window. the approximate “signal to noise ratio” of the variance dimension. The HM-Model estimates the distribution of the underlying states this is a ratio between the state 1 vs state 2 variance.
11.1.4 event-frequency.csv
Results of the event frequency analysis.
seconds: the one second interval. 1 = first second interval, 2 = second second interval etc.
begin: datapoint at which the second interval begins
end: the datapoint at which the second interval ends
freq_start: the number of events whose changepoint identified start occurred within that one second interval.
11.1.5 ensemble-data.csv
Results of the “Prep Ensemble” button. For internal use by the app. The data is the SUM of all displacement/position data for each event within an observation. This data is then read by avg_ensembles(), summed with all the other files and then divided by the number of events by reading in options-prep-ensemble.csv, which has an “event” count column for each obs folder.
direction: values are “forward” or “backwards” to designate whether the data is attributed to the forwards or backwards ensemble average.
forward_backward_index: a continuous time in datapoints used to plot the forwards/backwards ensembles on the same x-axis.
ensemble_index: relative time index for forwards and backwards ensembles respectively.
data: the SUM of all the respective relative datapoints for each event in the data trace within the observation folder. This data will be SUMMED with all the other data columns from other observations and then divided by the total number of events for the average.
11.1.6 options-prep-ensemble.csv
The user selected options from “Prep Ensembles” are saved here along with the total number of “events” in the event column. The avg_ensemble() function reads this file sums all the events and sums all the data columns from the ensemble-data.csv files to make the final average.
11.1.7 substeps.csv
Analysis of the substeps that occurs during the ensemble averaging procedure.
bead: the “bead” or channel the measurements were made on. You can average values between the beads. Value of 1 or 2.
event_id: the event number the data was measured on within the trace.
prior_unbound_position_nm: this is the absolute position in nm of the baseline a few milliseconds before the start of the events.
bead_position_substep_1_nm: the is the absolute average position of the bead for a few milliseconds after the changepoint identified start of the event.
substep_1_nm: the relative displacement between prior_unbound_position_nm and bead_position_substep_1_nm
bead_position_substep_2_nm: the is the absolute average position of the bead for a few milliseconds before the changepoint identified stop of the event.
substep_2_nm: the difference between bead_position_substep_2_nm and bead_position_substep_1_nm
total_step_nm: the sum of substep_1_nm and substep_2_nm.
after_unbound_position_nm: the absolute position average of the baseline over a few milliseconds after the changepoint identified stop of the event.
substep_2_nm_alt: alternative substep 2 calculation. Calculated as (bead_position_substep_2_nm minus after_unbound_position_nm) MINUS (bead_position_substep_1_nm minus prior_unbound_position_nm).
total_step_nm_alt: alternative total step calculation. Calculated as bead_position_substep_2 minus after_unbound_position_nm
11.1.8 options.csv
This file is used as a configuration / settings file for the app to automatically find / filter / keep track of certain user inputs.
hz: sampling frequency
processor: modification applied to the raw_bead signal to transform into processed_bead. Options are “none”, “remove_base”, “remove_mv”, and “detrend”.
include: TRUE/FALSE depending on whether user selected Include “Yes” or “No” in “Clean and Process” tab.
mv2nm: the millivolt to nanometer conversion factor.
mv2nm2: for 2 channel data sets, the volt to nanometer conversion for the second bead.
nm2pn: the nanometer to piconewton conversion
nm2pn2: for 2 channel data set, the nanometer to piconewton conversion for the second bead.
analyzer: which analyzer was used. Options are “hm/cp”, “covar”, “force_balance”, “mini”, or “ifc”.
report: throughout different analyses a “report” note is generated to detail if the analysis was successful or failed somewhere. If report = “success”, the analysis finished correctly. If other value, the trace was not completely analyzed.
review: TRUE/FALSE depending whether user clicked “Yes” or “No” in the review box of the “Analyzer” tab of the question “Should analysis be accepted?”
channels: the number of channels in the data. 1 or 2.
w_width: HM-models only. the user selected window width in datapoints.
w_slide: HM-models only. The user selected window advancement method.
em_random_start: HM-models only. TRUE/FALSE depending if the user selected EM Random Start.
use_channels: HM-Models only. “Mean/Var” or “Var” depending on whether the user selected to use the mean and variance dimensions for the HM-Model or just the variance.
front_cp_method: HM/CP models only. The user selected method for applying the changepoint. “Mean/Var” or “Var”.
back_cp_method: HM/CP models only. The user selected method for applying the changepoint. “Mean/Var” or “Var”.
cp_running_var_window: HM/CP models only. the user selected window width for applying the running variance window. The “Mean/Var” option uses the raw data, this options does not apply.
displacement_type: HM/CP models only. the user selected displacement measurement type. Options are “avg” or “peak”.
status: Whether the observation has been analyzed or not. Value of “analyzed” means it has been sent through an analyzer.
w_width_ms: mini ensemble only, the user chosen window width in milliseconds.
displacement_threshold: mini ensemble only. the user selected displacement threshold
time_threshold_ms: mini ensemble only. the user selected minimum time threshold.
feedback_motor_bead: only used for IFC data. The channel that is the motor bead. 1 or 2.
preferred_channel: for 2 channel data. the user selected preferred channel to be used for certain analyses.
lab: The lab that the data came from. Options are “unknown”, “greenberg”, “debold”, or “lumicks”.
original_filename: the original filename that the trap data was uploaded from.
11.1.9 header.csv
If “header” option was selected during data upload, the values are extracted and stored her verbtim along with the app putting those values where it wants in options.csv etc. This is more of a preservation file.
Columns are V1 and V2 which represent KEY-VALUE pair of the header, which should be in KEY-VALUE format to be properly read and parsed.
11.2 Summary folder files
The summary folder is created in project -> summary when the “summarize” button is first hit.
The files in the summary folder will vary on type of analysis. The most important one will be the one that ends in “*_all-measured-events.csv”.
11.2.1 all-measured-events.csv
This is one file that has all of the “measured-events.csv” files combined into one csv file. Note, this is the results of the R function rbind_measured_events() supplied by the lasertrapr R package. It finds and reads all relevant files and applies appropriate filters to only include data that you want to include and filters out events you excluded. The columns are all described in “measured-events.csv” above and depend on the specific analyzer used.
11.2.2 summary-data.csv
An csv file with selected pre-calculated averages of all-measured-events.csv. Recommended you calculate these values yourself for double checking.
11.2.3 ensemble-average-models.txt
Fitted model output of the exponential fitting of the ensemble averages. Verbatim R output.