Introducing the new VFLAG for 'duplicates'

[Osanna123]

As discussed at the HARSAT workshop, ICES has introduced a new VFLAG 'E' Exclude from the standard assessment procedure.
If a data submitter has two replicate results for the same PARAM from the same sample (due to f.ex. different methods of analysis), they can use VFLAG E for the replicate that should not be used in a standard assessment. This should be taken into account when only one of the replicates is picked for the assessment.

[RobFryer]

@Osanna123 This looks great. I've got several questions, mostly to get agreement on how VFLAG 'E' will be used.

Will a measurement with VFLAG 'E' always be excluded from a (harsat) assessment or only if it is a replicate measurement? Specifically, what will happen in the following situations:

• measurement A has VFLAG E, and there is another replicate measurement which does not have VLFAG E
• measurement A has VFLAG E, but there is no other replicate measurement
• measurement A has VFLAG E, so does replicate B, but there is no other replicate measurement; i.e. all replicates have VFLAG E

Will this get fully documented in RECO? Will need to explain what is meant by the 'standard assessment procedure'; something like the harsat assessment of temporal trends perhaps?

Once we've got clarification, I'll implement this in the code.

[RobFryer]

@Osanna123 On reflection, I wonder whether we are over-complication things by tying this in with replicates. Simpler to just make VFLAG = E means exclude from harsat style assessments. This would then be generalisable to other situations that we haven't yet thought of.

Perhaps we should discuss at the harsat workhop on Friday?

[Osanna123]

Hi @RobFryer it's a good idea to discuss this on Friday.
I think, we might need to hear others what the rules around this flag should be. I can improve the description in RECO.

[RobFryer]

@Osanna123 @HansMJ @annelaerkes @hallareinert @DagHjermann @swamap
Can I close this issue now? VFLAG = "E" is now described in RECO as "Exclude from the standard assessment procedure" and could be used for a variety of purposes; e.g. Swedish replicates analysed by different labs where one is preferred to the other (but both are valid data); or Halle's foetus data which shouldn't be used in the standard time series assessment (but again are valid data).
If agreed, then I will add the exclusion to tidy_data, which is where we also delete suspect data (VFLAG = "S") etc. Remember that all the data will still be read into harsat, so they won't be lost from sight.

[DagHjermann]

Sounds good to me!

[HansMJ]

Yes sounds good. Straight forward implementation without any interpretation needed.
Will 'E' apply equally for AMAP, HELCOM and OSPAR then?

[hallareinert]

So, can someone please clarify: when should I use VFLAG = S compared to using VFLAG = E?
Do they have the same end goal - ie to be excluded from HARSAT assessments? Or is there a reason to choose one over the other?

[DagHjermann]

@hallareinert We have some data which are not suspect (VFLAG = S) but still should be excluded. For instance, a few years ago we measured PCB in super high concentrations in liver from three cod from the Sørfjord in Norway (100 000s of ug/kg, so actually several %) . The researchers thought something must be wrong, so they took new samples from those livers and reanalysed them - same result. So the concentrations were real, but these three fish (out of 15 in total that year) really messed up Rob's time series analyses. So these are a good example of VFLAG = S I guess.

[swamap]

From my point of view it would be preferable if there were some flexibility in use of this flag so that it could be applied differently for different organizations (or types of work). Does it have to be a single character? Could we have at least e.g. EO – excluded for OSPAR assessments EH ... ditto for HELCOM EA .... ditto for AMAP S – excluded from all (suspect data)? could also then potentially have ET .. exclude from temporal trend assessments (perhaps that’s same as omitting T from purpose of monitoring? ... otherwise might need to combine single character E VFLAG with content of the ORGanizations field (e.g. if VFLAG=E and ORG code includes H then omit from a HELCOM assessment?) Cheers Simon From: Hans Mose Jensen ***@***.***> Sent: 02 December 2024 08:07 To: osparcomm/HARSAT ***@***.***> Cc: swamap ***@***.***>; Mention ***@***.***> Subject: Re: [osparcomm/HARSAT] Introducing the new VFLAG for 'duplicates' (Issue #493) Yes sounds good. Straight forward implementation without any interpretation needed. Will 'E' apply equally for AMAP, HELCOM and OSPAR then? — Reply to this email directly, view it on GitHub <#493 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR5CXRYSD2HUOIMNF4H3HGL2DQIK7AVCNFSM6AAAAABRNLC3XSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJQHAZTINBZHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RobFryer]

I've summarised the present VFLAG situation and then suggested some solutions. I'm afraid that as I wrote, it became more and more complicated (and also led to more general thoughts about the matching of data to stations). So please have a read and then I will set up a meeting to discuss what might work.

Remember, we are only talking about ICES extractions here. And remember that one reason for all the complications is that data have been submitted inconsistently over time and there is a huge overhead in getting legacy data resubmitted.

Current situation

All data with VFLAG = S are filtered out of the ICES extraction and not used. There aren't many records with VFLAG = S (in the OSPAR extraction at least which covers all data in the OSPAR area). This process happens in tidy_data, so it is possible to see all the data that have VFLAG = S before they are deleted.

According to RECO, VFLAG = S covers "Suspect value - considered suspect by originator on the basis of quality control or recorder/instrument/platform performance". This relates to the analytical measurement, but could also cover the situation where there might have been contamination of the sample during (pre-)processing.

Some data are currently excluded from the OSPAR assessment by ad-hoc corrections. It would be nice to make this more automatic through identifying the relevant data appropriately in submissions. Here are some examples of genuine reasons for excluding data.

Reasons for excluding data

There is a desire to exclude data from a 'standard harsat assessment' (whatever that means) without having to do ad-hoc corrections in the code. Here are four examples where data should be excluded but where using VFLAG = S probably wouldn't be the correct way to go about it.

• Halle has submitted contaminants in pilot whale foetus. These measurements are valid (VFLAG = S would be inappropriate), but are not to be considered for a big assessment. At present, this is dealt with in the OSPAR assessments by an ad-hoc correction which deletes these data.
• Sweden have measured some samples twice using different laboratories. (There was a change in laboratory and this was to allow inter-laboratory comparisons.) One laboratory had much lower limits of detection and the measurements from this laboratory are to be used in harsat assessments (since currently harsat isn't clever enough to deal with replicate measurements properly). Currently harsat rejects one replicate at random. An ad-hoc correction could be used to fix this.
• France have mytilus data from autumn and winter. Both are valid, but concentrations differ between seasons. Winter concentrations are to be used for OSPAR assessments. Currently implemented by an ad-hoc correction.
• Dag's example of three cod with super high concentrations of PCBs (much higher than all other samples in the time series). The measurements are 'real' (in the sense that repeat analysis of the samples gave similarly high values), but they are not representative of the population being sampled. Maybe the samples were contaminated during processing in which case VFLAG = S would be appropriate. Or maybe they come from fish with very unusual life histories, in which case VFLAG = S would be stretching things. They certainly cause problems in a time series assessment (because the routines are not robust in the statistical sense). I don't know whether these data are marked with VFLAG = S or whether Dag has deleted them from the ICES database, but I haven't noticed them for a while.

Solutions

Before we start, some more information about relevant fields used in the data extraction.

At present, the PURPM submitted with the data is not used at all. Instead, the purpose of monitoring is identified at the station level. That is, we assume that all data collected at a station marked in the station dictionary with station_PURPM = T are for temporal monitoring purposes. All other data are excluded. We have gone down this route because there have been many inconsistencies in the way PURPM has been used in data submissions and because it is so much easier to update the station_PURPM field in the station dictionary than it is to resubmit legacy data with updated values of PURPM.

Monitoring program information is available at both the measurement (MPROG) and station (station_programgovernance) level and can be used to match data to stations. Data are excluded if there is no match. (Note that MPROG information is simplified in the extraction to is_ospar_monitoring, is_amap_monitoring, is_helcom_monitoring.) However, the default for HELCOM and AMAP is currently to ignore program governance (i.e. use all data). (This can easily be changed, and indeed Simon we should discuss this as I have just noticed some other potential anomalies for AMAP.) The default for OSPAR uses program governance, but is rather convoluted because of the way both OSPAR and AMAP stations are involved.

Possible solutions:

1. Use the PURPM field submitted with the data. Only use measurements with PURPM = T. Would potentially require a massive overhaul of legacy data submissions.
2. Use the MPROG field submitted with the data and exclude data which aren't marked for a specific purpose. Wouldn't currently work for HELCOM where all data are used, regardless of MPROG.
3. Get data submitters to resubmit legacy data and to exclude the measurements that shouldn't be included in standard assessments. But this isn't desirable, because the data base loses data that are valid (for other purposes).
4. Use VFLAG = E to exclude data. This could be switched on or off in tidy_data so that different users could choose whether to implement it. For example, AMAP could retain everything with VFLAG = E and OSPAR could delete everything with VFLAG = E.
5. Use VFLAG = E to always exclude data from a harsat assessment, and VFLAG = EA to exclude only from an AMAP assessment; VFLAG = EO to exclude only from an OSPAR assessment; and VFLAG = EH to exclude only from a HELCOM assessment; and VFLAG = EO~EA to exclude from both an AMAP and an OSPAR assessment, but not a HELCOM assessment

@swamap I understand your concerns about flexibility, but can you think of an example where using VFLAG under option 4 might not meet your needs and where option 5 would be preferable.

[DagHjermann]

I think (4) seems like a good solution which is relatively easy to implement and administrate.
(1) seems unrealistic
(2) doesn't work for HELCOM
(3) is what I did for the massively PCB-contaminated cod, I believe (but it didn't feel right - see comment below)
(5) seems a bit of overkill? At least if there only is a hypothetical need for it (at present). If somebody needs to "unexclude" some data for some specific purpose, it's always possible to manipulate VFLAG after read_data() and before tidy_data().

Comment on the few cod with PCB concentrations up to 427000 ug/kg: contamination is unlikely to cause such massive concentrations. The likely reason was that the local power station at the seaside (picture) was renovated the year before, using sandblasting to remove the old plaster and paint. I had a faint memory that they actually published a paper on it, and now I found it:
https://www.sciencedirect.com/science/article/pii/S0025326X0500531X

[hallareinert]

Thanks everyone for the input!
I think it's great to have the distinction between suspect data (VFLAG = S) and some way of excluding data that are valid but not fit for purposes (ie time trend monitoring). I am not sure what the ideal solution is for all purposes, but easiest solution tends to be the best :)
I guess generally we should exclude any option that requires a massive overhaul of legacy data - right? Doesn't seem feasible from what I can tell by Robs' chasing... Although as I see it, wouldn't all of these solutions require changing legacy data at least to some extent?

1. This would require changing legacy data. I guess it could be an option for future data? And then it could work in combination with fx is_amap_monitoring or MPROG, giving the same option as Simon's suggestion, but instead of excluding data, we choose what to include? This would only work though if we can identify and "mark" legacy data that should be included in the desired assessment.
2. I don't see this working for either suspect or replicate/duplicate values?
3. Same as with one, probably not an option, AND would results in loss of data.
4. Easy solution, at least for data submissions moving forward. What about legacy data, wouldn't they require to be resubmitted for this solution as well? And I guess it's not a flexible solution... But if it is possible to manipulate VFLAG after read_data() then this would be the best solution as I see it. And there is always the opportunity of treating the dataset as external data that has been edited as needed.
5. I understand the desire to introduce more options about data exclusion/inclusion and it would be a tidy solution eventually I'm sure. But as I see it, it adds complexity to both data submissions and analysis.

I think for our data, the VFLAG = S and VFLAG = E options should suffice. But then again, I have yet to be involved in the AMAP data analysis, so I might still change my mind! :)

Ps early morning ramblings, hopefully it makes sense!

[Osanna123]

Dear all, thank you for your comments so far.

Thank you for the good examples, Rob. I think, there are several countries that have valid replicates like Sweden. If you ask them, they can easily specify that data from one ALABO or with one METOA should be prioritized compared to another. The thing is that we don't ask, and random picking for the assessments is not what the submitters want. VFLAG E has its flaws, but it is transparent and easy to implement, so perhaps it could help?

Regarding Rob's solutions, comments on behalf of DOME and our data submitters:

1. Using the submitted PURPM might be important for the samplings made without a station reference (mainly legacy data). But i don't see our submitters resubmitting all data to fix them. This is not helpful with deciding what replicate records to use.
2. This is an option to work towards in a long run, but will not replace VFLAGs or option 1.
3. No go. No. Not an option.
4. Seems kindda like an obvious choice?
5. Most of our data submitters don't have enough knowledge what has to be marked for exclusion for OSPAR, HELCOM, or AMAP. If we overcomplicate the VFLAG options, they will not report them at all. Even E might raise some reporting issues.