omitted flashes: repeated time values, also consecutive!

[farznaj]

@dougollerenshaw

Hi, I am looking at the omitted flashes of session_id : 814281477

Two issues:

1. two of the omitted flashes are repeated in the start times below (check the first two): (dataset.stimulus_table)

4649 3779.89708 4650 3779.89708 4679 3801.66484 4696 3814.42524 4755 3858.72810 4770 3869.98726 4793 3886.51745 4813 3901.54636 4814 3902.29697

2. when I look at the timing between omitted flashes, it seems a number of them are 750ms apart (see below), which I guess means they happen right after each other. Is that possible? I thought there was some rule about not having 2 omissions in a row.

np.sort(np.diff(list_omitted))

array([ 0. , 0.75055, 0.7506 , 0.75061, 0.75061, 0.75062, 1.50119, 1.50119, 1.50121, 1.50123, 1.50123, 1.50123, 1.50123, 1.50124, 1.50125, 1.50125, 1.50126, 1.50126,

Thanks. Farzaneh

[nickponvert]

Hi, the 0 value (a double-logged omitted flash, it seems to me) makes me realize that we only fixed #511 in the production analysis branch as of now. @farznaj were you using the master branch when you ran into this issue? There was a bug in camstim where some omitted flashes were being logged twice very occasionally (which we might be seeing here), and also were sometimes being logged in both the stimulus log and the dropped frame log (usually with an offset of 1 frame). However, if I understand correctly (please correct me @dougollerenshaw) there is a 5% chance for each non-change flash to be omitted, so it is possible to have one omitted flash right after another (just pretty unlikely). How many of the 750ms intervals were you seeing compared to the total number of flashes/omitted flashes?

[dougollerenshaw]

Nick nailed it!

On May 8, 2019, at 7:25 PM, Nick Ponvert [email protected] wrote:

Hi, the 0 value (a double-logged omitted flash, it seems to me) makes me realize that we only fixed #511 in the production analysis branch as of now. @farznaj were you using the master branch when you ran into this issue? Also, which data file were you loading? There was a bug in camstim where some omitted flashes were being logged twice very occasionally (which we might be seeing here), and also were sometimes being logged in both the stimulus log and the dropped frame log (usually with an offset of 1 frame). However, if I understand correctly (please correct me @dougollerenshaw) there is a 5% chance for each non-change flash to be omitted, so it is possible to have one omitted flash right after another (just pretty unlikely). How many of the 750ms intervals were you seeing compared to the total number of flashes/omitted flashes?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[farznaj]

Hi Nick, I am using the "production analysis" branch (I only cloned this branch of visbeh).

I am getting 5 consecutive flashes in 4815 flashes, which after subtracting out "omitted" and changed images from 4815, the fraction consecutive changes to 0.0012. It is much less than 5%.

Let me make sure I fully understand how omissions occur. Can you or @dougollerenshaw evaluate my statements below? :)

1. There is 5% chance for a flash to be assigned as omitted.
2. Once a flash is assigned as omitted, there is 5% chance for it to be over-written as non-omitted? and 95% chance to remain as omitted.
3. Question: once a flash is assigned as omitted, how does it change the probability of next flash being assigned as omitted? This question will answer why we are seeing 0.1-0.2 % consecutive flashes, I guess.

Regarding the data, it is mesoscope data, and the session details are as follows:

session_id = 814281477

 lims_id  area depth

0 816242121 LM 75 0 816242105 LM 150 0 816242095 LM 225 0 816242114 LM 280 0 816242091 VISp 75 0 816242073 VISp 150 0 816242065 VISp 225 0 816242080 VISp 280

Thanks, Farzaneh

[dougollerenshaw]

• There is 5% chance for a flash to be assigned as omitted.

True

• Once a flash is assigned as omitted, there is 5% chance for it to be over-written as non-omitted? and 95% chance to remain as omitted.

True, but this bug has been fixed in camstim. Check with @rhytnen to get the version number where the fix was applied and compare that against the camstim version that generated the data you're analyzing. If you're still seeing this after the version where the fix was applied, please report it as a bug in camstim.

• Question: once a flash is assigned as omitted, how does it change the probability of next flash being assigned as omitted? This question will answer why we are seeing 0.1-0.2 % consecutive flashes, I guess.

The probabilities are independent.

[nickponvert]

@matchings and I merged some changes into production analysis about a week ago (#529 and #531) that should be correcting for the camstim bug, but the fact that you have one difference of 0 between times of omitted flashes makes me think that it isn't working correctly. @farznaj would you mind posting / sending me the code or notebook where you are doing this analysis so I can check it out?

[dougollerenshaw]

Oh wait, I just re-read my answer to @farznaj above and realized that I got one thing wrong. The answer to the second question should be "false". The probabilities are backwards. After a flash was chosen as omitted, there is a 5% chance that it will be chosen as omitted again on the second time through the algorithm, and a 95% chance it'll be actually displayed. Making matters more complicated, this was only the case for the first flash in the trial, which was the only time that camstim was calling that function twice.

But again, this bug was addressed many months ago in camstim (http://aibspi/braintv/camstim/merge_requests/35), so you should only be seeing this now if you're looking at old data, or if the mesoscope hasn't been updated.

[farznaj]

@matchings and I merged some changes into production analysis about a week ago (#529 and #531) that should be correcting for the camstim bug, but the fact that you have one difference of 0 between times of omitted flashes makes me think that it isn't working correctly. @farznaj would you mind posting / sending me the code or notebook where you are doing this analysis so I can check it out?

@nickponvert ; Hi; see below for the code:

session_id = 814281477 Session_obj = LimsOphysSession(lims_id=session_id) list_mesoscope_exp = Session_obj.data_pointer['ophys_experiment_ids'] dataset = VisualBehaviorOphysDataset(list_mesoscope_exp[0], cache_dir=cache_dir) dataset.stimulus_table list_omitted = table_stim[table_stim['omitted']==True]['start_time']

[nickponvert]

@matchings and I merged some changes into production analysis about a week ago (#529 and #531) that should be correcting for the camstim bug, but the fact that you have one difference of 0 between times of omitted flashes makes me think that it isn't working correctly. @farznaj would you mind posting / sending me the code or notebook where you are doing this analysis so I can check it out?

@nickponvert ; Hi; see below for the code:

session_id = 814281477 Session_obj = LimsOphysSession(lims_id=session_id) list_mesoscope_exp = Session_obj.data_pointer['ophys_experiment_ids'] dataset = VisualBehaviorOphysDataset(list_mesoscope_exp[0], cache_dir=cache_dir) dataset.stimulus_table list_omitted = table_stim[table_stim['omitted']==True]['start_time']

@farznaj Did you run convert_level_1_to_level_2 on the session first, or are you loading h5 files (the output files from that conversion) that might have been generated a while ago? If they were generated before the fix was in place, or if they were generated with a different branch than production analysis, then that would explain why you still see that one repeated omitted flash.

[farznaj]

Oh wait, I just re-read my answer to @farznaj above and realized that I got one thing wrong. The answer to the second question should be "false". The probabilities are backwards. After a flash was chosen as omitted, there is a 5% chance that it will be chosen as omitted again on the second time through the algorithm, and a 95% chance it'll be actually displayed. Making matters more complicated, this was only the case for the first flash in the trial, which was the only time that camstim was calling that function twice.

But again, this bug was addressed many months ago in camstim (http://aibspi/braintv/camstim/merge_requests/35), so you should only be seeing this now if you're looking at old data, or if the mesoscope hasn't been updated.

Thanks @dougollerenshaw . If the probability of the an omission remaining as an omission is 5%, the final omission rate would be .05^2 = .0025 ....which is quite low. Is this what you actually meant? How much has this bug affected the omission rate (ie from .05 to what fraction?)

Is there a way to know how many "trials" are in a session? (so we can estimate how much this bug that occurred on the first flash of a trial has changed things).

When did the fix in camstim happen? Does this mean that if we are dealing with experiments before that date, what dataset.stimulus_table calls "omitted", is actually not omitted and there is no way for us to know? Or is there an analysis that can reveal that?

The metadata of the session I am looking at is as follows, and here I am getting ~.05 omissions : sum(omissions) / (len(stimulus_table) - len(changed_flash)).

ophys_experiment_id 816242065 experiment_container_id None targeted_structure VISp imaging_depth None specimen_driver_line Vip-IRES-Cre cre_line Vip-IRES-Cre reporter_line Ai148 full_genotype Vip-IRES-Cre;Ai148 session_type behavior_session_225 stage OPHYS_0_images_A_habituation donor_id 421137 experiment_date 2019-01-28 specimen_id 760928877 session_id 814281477 project_id MesoscopeDevelopment rig MESO.1 ophys_frame_rate 11 stimulus_frame_rate 60 Name: 816242065, dtype: object

Thanks.

[farznaj]

@matchings and I merged some changes into production analysis about a week ago (#529 and #531) that should be correcting for the camstim bug, but the fact that you have one difference of 0 between times of omitted flashes makes me think that it isn't working correctly. @farznaj would you mind posting / sending me the code or notebook where you are doing this analysis so I can check it out?

@nickponvert ; Hi; see below for the code: session_id = 814281477 Session_obj = LimsOphysSession(lims_id=session_id) list_mesoscope_exp = Session_obj.data_pointer['ophys_experiment_ids'] dataset = VisualBehaviorOphysDataset(list_mesoscope_exp[0], cache_dir=cache_dir) dataset.stimulus_table list_omitted = table_stim[table_stim['omitted']==True]['start_time']

@farznaj Did you run convert_level_1_to_level_2 on the session first, or are you loading h5 files (the output files from that conversion) that might have been generated a while ago? If they were generated before the fix was in place, or if they were generated with a different branch than production analysis, then that would explain why you still see that one repeated omitted flash.

@nickponvert ah I think you caught it correctly. Let me check with Jerome as he ran the convert part.

[dougollerenshaw]

@farznaj:

• The PR was submitted to camstim on 11/26/18. See the link above. If you can't see that link, contact @rhytnen for access. I don't know what their actual deployment schedule was.

• The number of trials is equal to the number of rows in the extended trial dataframe.

[nickponvert]

@dougollerenshaw @farznaj @matchings @jeromelecoq This seems resolved but I think we need to consolidate some documentation on this.

[farznaj]

hmmm ... my memory is weak here ... i can't remember how it got resolved...

[nickponvert]

we pushed changes into vba and the sdk to deal with the double-logged omitted flashes. but we still need to document the probability of double-omissions somewhere.