Meet Jim Huggett LGC, Helen Parkes LGC and Mojca Milavec NIB
What we find with the methods is that they are very precise. You do them once, you do them again and again and you get high precision and very good agreement. But that can be dangerous because that very precision can lead to biased results. Jim Huggett, Principal Scientist, LGC
TEA: Hello! This is Tea from BioSistemika here at Piran, the pearl of the Slovenian Mediterranean coast. I am here with the very interesting team behind the INFECT-MET project. So, if you could introduce yourselves to our Splice audience…
MOJCA: Hello, my name is Mojca Milavec, I am from the National Institute of Biology that has hosted two meetings of the European Metrology Research Project over the last two days. One is INFECT-MET and the other is BioSITrace.
JIM: Hi there! My name is Jim Huggett, I am from LGC. I am working and running the groups looking at some of the molecular methods we are investigating in both studies.
HELEN: Hello! I am Helen Parkes, also from LGC. I tend to champion international bio-measurement comparability. I don’t do the technical work, I just try to make sure that the work we do has an impact. We work very closely with stake holders and feed into international regulation. We generate results which have meaning to the general public and public health.
We are trying to get metrological support, i.e. accurate and repeatable measurements to help in the health sector and achieve better diagnostics.
TEA: Great! Mojca, could you introduce a little bit of the whole idea behind the project and what you have been working on over the past few days?
MOJCA: We started with the INFECT-MET Project. This Project deals with metrology for respiratory diseases. We are trying to get metrological support, i.e. accurate and repeatable measurements to help in the health sector and achieve better diagnostics. We are working with several RTI-causing-microorganisms (Respiratory Tract Infections). I was mostly involved in establishing good methods, we call them higher order methods, for the detection of three model organisms. We have chosen one bacterium and two viruses. We started as it should be started with a method: first get a good extraction method that gives us good nucleic acid which can then be used in further analysis and getting good final results.
TEA: So basically when we are talking about public health protection and rapid diagnostics of infectious diseases, we know that methods like Real-Time PCR, digital PCR, sequence analysis etc., offer great potential to improve the management of infectious diseases when compared to traditional methods. What was actually the bigger vision behind all this work you have been doing here?
MOJCA: We developed methods that could be used as good methods to which all other methods can be compared. In our studies we chose digital PCR and we tested how good this method is so that we can have a method that can be used for value assignment of reference materials and also so that laboratories can compare their methods to this method and see how close their results are to our method.
The idea in terms of the bigger vision is that the work we do should be accepted internationally, so we feed our results into the international regulatory framework and the standards framework.
HELEN: And if I can actually add … obviously, when we’ve done the technical work and looked in the lab at reliability, reproducibility and all those factors which will affect how well digital PCR works, we’re developing guidelines and reference materials which will help support other users and stakeholders in the field. The idea in terms of the bigger vision is that the work we do should be accepted internationally, so we feed our results into the international regulatory framework and the standards framework. That means, for example, that a hospital laboratory that is using digital PCR would (quite frequently) need to be accredited to particular laboratory standards, for example using ISO standardization for clinical measurements or perhaps working to the Clinical and Laboratory Standards Institute to get their accreditation. With the methods that we develop will actually be written into these standards as good practice. So basically we’ll be developing good practice. But OK … back to the technical side of things…
A huge amount of research is performed, many Nature papers published, but a very small amount of this work actually makes it to the patient.
TEA: Jim, can you tell us more about why you chose digital PCR? What would be the main benefit of this technology?
JIM: One of the potentials of digital PCR is its ability to essentially count molecules. This is one of the areas that the BioSITrace Project, which is complementing, it’s funded by the same body as INFECT-MET, is looking at. If we’re able to count molecules in the way the digital PCR performs, we are no longer necessarily, and I say necessarily because it’s not yet been proven, dependent on a calibrator. When you think of pretty much all biological measurement methodologies, whether they are ELISA or qPCR etc. for quantification, you require some kind of calibrator. That is OK from the point of view of performing an experiment on your own laboratory, but the big issue comes out on how you make that calibrator reproducible? How do you make my calibrator similar to your calibrator? This is a major challenge for reproducibility.
One of the reasons for this is because research is performed in a manner that is not easy to reproduce and one of the areas INFECT-MET is looking at as well, as described by Mojca and Helen, is these advanced methods.
As Helen and Mojca have described, we are looking to assist applied methods that are being performed at the moment in the clinics but additionally we are also interested in new methods, the application of digital PCR and also NGS. Specifically we are trying to describe how that can be applied in a reproducible manner. Because one of the shames about the application of molecular biology to preclinical research, for looking at biomarkers for example, is how little is actually translated to aid patients. A huge amount of research is performed, many Nature papers published, but a very small amount of this work actually makes it to the patient. One of the reasons for this is because research is performed in a manner that is not easy to reproduce and one of the areas INFECT-MET is looking at as well, as described by Mojca and Helen, is these advanced methods. Coming back to digital PCR, that also touches on other aspects of BioSITrace, which is looking at the enumeration of biological entities, i.e. looking at protein molecules and cells, and it is the counting that is crucial. In biology we tend to be dependent upon the mole (measurement unit). But this is potentially initiative… If you think of a PCR reaction where you would describe the primer concentration as milimolar (mM), unless you know the sequence and size of that primer, then that will actually vary in exactly how many molecules are present. Helen?
This is actually quite interesting because, a lot of times in terms of the measurements that are carried out in all these areas, there is a huge variance in the actual accuracy of results that are achieved and the lack of comparability of results. That’s when it really starts to matter.
HELEN: So basically people think of the SI (The International System of Units) etc. But we have to be sure, when we are counting for our copy number determination or, for example, the number of infectious disease HIV particles or tuberculosis bacterium, we need to know what the numbers are. We need to be sure when we are doing the so called traceable measurements that we know what the measurement uncertainty is around the detection method that we are using for our counting. By that I mean the “confidence”: how sure we are that we are getting the right number. This is actually quite interesting because, a lot of times in terms of the measurements that are carried out in all these areas, there is a huge variance in the actual accuracy of results that are achieved and the lack of comparability of results. That’s when it really starts to matter. For example, one of the things that we’re looking at in the BioSITrace Project and using digital PCR for is looking at circulating cell-free DNA and using KRAS (Kirsten Rat Sarcoma viral oncogene homolog) as a model. We are developing a panel of reference materials that we can use (obviously it is a model system for our study) but ultimately there’s a goal to support medical measurements in that area. But we are using those to enable us to very precisely determine how accurate the measurement method that we’re using can be and, what is perhaps more important, everything that it needs to be to contribute to how reliable a method will be. Basically as I said, considering the quality criteria, we then feed those into developing standards. We also want to develop reference methods that other people can then use and get those accepted internationally. So, really just put a little bit of scientific rigor into supporting that higher level of measurement developing reference materials that then can support other people’s measurements for these critical areas and reference methods so that they can then go into international standards that everybody uses. That gives us the ability to measure and hopefully, support more comparable measurement.
…the uptake of NGS in hospitals and clinical laboratories and it seems to have taken the lead…
JIM: I guess digital PCR is one of the potential methods that could be the higher order method because of the fact that it can count without the need of a calibrator. But I guess my word of caution would be – we need to prove this. And that is a lot of work which we are doing (on both projects).
TEA: And do you also recognize the potential of NGS in clinical applications? That is also quite an interesting subject, because from one point of view we see that NGS is not actually there yet. But on the other hand it does show great potential in this area.
NGS is a phenomenally powerful method. I certainly believe it will change everything: the only question is when.
HELEN: Yes, it is recognized and we know for a fact that there is an uptake of NGS in the clinical sector across Europe: we belong to the EuroGentest Consortium that is looking at the development of the need for genetic standards across Europe… They did a study on the uptake of NGS in hospitals and clinical laboratories and it seems to have taken the lead: certainly, they missed out on PCR and digital PCR and went straight to sequencing. But they don’t really understand the power of … the variability that can be achieved with that. I think, Jim, maybe if you give an example of the work we’ve done on metagenomics under the INFECT-MET Project, that would give an example of how we are looking at NGS to describe this variability. We have a lovely picture on the slide, but it is a bit difficult to describe it in words…
JIM: NGS is a phenomenally powerful method. I certainly believe it will change everything: the only question is when. Metagenomics, i.e. the study of the microbiome, is arguably one of the most complex applications of NGS.
But because sequencing gives you numbers, people are plotting abundances.
HELEN: But it illustrates the point…
JIM: Yes, of course. It is a great example. But because sequencing gives you numbers, people are plotting abundances. We’ve published a couple of papers looking at how one may support that from the point of view of calibration and where the reference material may be able to support a measurement. Different instruments can provide wildly different results. Different techniques, even different library preparation methods, can give you different abundances. You may see different things being prominent simply because of the method that is being used and it comes down to the mantra that all measurement is estimation.
I think there’s the old adage “Rubbish in, rubbish out” which really says that we actually have to work very hard on looking at the quality of all parts of NGS.
HELEN: And certainly what we see at the international level, when we are looking at international standards, is an attempt to try to get more quality into international use of NGS. Particularly, for example, the Chinese are leading on this. They’ve got huge high-throughput sequencing facilities. But what we really need is not only this … there has been international effort to concentrate on the bioinformatical aspect, which is incredibly important, of course, and can lead to highly variable results. I think there’s the old adage “Rubbish in, rubbish out” which really says that we actually have to work very hard on looking at the quality of all parts of NGS. That actually means looking at PCR, we’re doing in the first place aspects of the procedure, and not only how we actually look at the results. So again, a lot of the work that we are doing is around that end.
What we find with the methods is that they are very precise. You do them once, you do them again and again and you get high precision and very good agreement. But that can be dangerous because that very precision can lead to biased results.
JIM: What we find with the methods is that they are very precise. You do them once, you do them again and again and you get high precision and very good agreement. But that can be dangerous because that very precision can lead to biased results.
HELEN: Yes, precisely wrong, unfortunately.
It is incredibly powerful, but too powerful in the wrong hands sometimes…
JIM: We need to consider the reproducibility again. But I am an advocate; I think NGS has a lot to offer.
HELEN: It is incredibly powerful, but too powerful in the wrong hands sometimes… Well, we are out to support our diagnostics manufacturers and our clinicians and that’s the whole idea of national measurement institutes. Projects that we have described (INFECT-MET & BioSITrace) are actually partnerships across Europe. While we’re here in wonderful Slovenia in Piran and enjoying that, we have to say that obviously, we’re hosted by Slovenia (NIB) and around the table we’ve had France, LNE (the National Measurement Institute from France), PTB from Germany, LGC from the UK, UME from Turkey and JRC-IRMM (the Joint Research Centre for the European Union) based in Belgium. So these are very integrated and important European efforts. What we do as national measurement institutes is try to support with good measurement, in this case very good molecular measurement, where we are actually in the world. The world’s national measurement institutes NMI, LGC, IRMM, NIST and NIB lead the world in digital PCR. Any conference, they are at it.
We do have leading edge technologies but our role is to understand how we can actually look at the performance of bacteria, reduce the measurement uncertainty, make sure people use the technology in the best possible way and get the most comparable results across the world.
We understand the need for precision and accuracy in one lab, but nowadays people are global travelers, research is shared and we need to be sure we can actually understand the results we’re all getting to have a level of comparability and that we are comparing apples with apples and not apples with pears. I think that goes across everything we’ve done, from both the infectious disease diagnostics right through to cell marker experiments and everything we do as national measurement institutes.
We understand the need for precision and accuracy in one lab, but nowadays people are global travelers, research is shared and we need to be sure we can actually understand the results we’re all getting to have a level of comparability and that we are comparing apples with apples and not apples with pears.
TEA: What would be your advice for young research scientists, postdocs pursuing a career in the field of molecular biology, people who are actually very eager to learn about new techniques and move the research and clinical applications forward?
JIM: I have a suggestion, when I show my age to anybody listening from the UK…
Take nobody’s word for it!
HELEN: (Laughs) And he is the youngest one, we should say this!
JIM: When I was a child there was a science program called “Take Nobody’s Word for It” and I think in molecular measurement and molecular biology that really sums it up. We have a number of fantastic kits that allow us to do extraction, perform reactions and do various things and they are often very useful. But just because they come in a nice package doesn’t mean you should not evaluate them in your own hands. I think that would be one piece of advice I would give to a new scientist starting a PhD, and it is often it is against the will of their supervisor: are they sure the method is working?
Just because if it works once, don’t write it down and carry on. That is the role we play. Because for applications in hospital medicine, in industry, you actually have to be sure you have robust and reliable measurements.
HELEN: Yes, just because if it works once, don’t write it down and carry on. That is the role we play. We have to be sure things work over and over and over. Because for applications in hospital medicine, in industry, you actually have to be sure you have robust and reliable measurements. The academic front is about the question: “Does it work? Yes! Tick that box and move on. I found out what I want to”. But it might not necessarily be the right thing and it might not have worked yesterday and it might not work tomorrow. Put a little bit of caution and skepticism about the quality and the output of what you’re doing. Just question why, what, etc.
JIM: This field essentially we are talking about is metrology. It was developed two hundred years ago. And started in physics…
HELEN: No! Not two hundred years ago, it is an ancient teaching… (laughs)
JIM: (laughs) Ok, the metric system is about two hundred years old and the science of metrology comes hand in hand with that.
HELEN: Metrology is the measurement of science.
JIM: Yes, the science of measurement. And the chemists took it on 50 or 60 years ago. Now the biologists are beginning to realize that it is the case and it is happening and it is great to be involved in this.
TEA: Mojca, anything to add for the end? Before we let you go?
MOJCA: Well, I think we have had three good days here in Piran, a lot of discussions and a lot of progress on good projects. I hope that such cooperation will continue because I think we can make a change!
TEA: Great!
The great thing though is that the actual stake holders; clinical chemists and hospital molecular biologists is that they really want to make their measurements right.
HELEN: And we are! Slowly, slowly… the great thing though is that the actual stake holders; clinical chemists and hospital molecular biologists is that they really want to make their measurements right. They are actually working and collaborating with us. So all the projects we’ve described actually have a host of very relevant stakeholders and collaborators coming from the in vitro diagnostics industry, hospitals and the instrument manufacturers. Everybody wants to get it right.
TEA: Of course, yes. Thank you very much for this interesting interview. Well, we hope to see you again in Piran anytime…
HELEN: (laughs) Oh we have to come to Piran!
TEA: Thank you very much, all of you!
HELEN: Thank you!
