Poster · 2020
Real Time Quality Control for Subjective Endpoints in Clinical Trials
Morinan G, Peng Y, O’Keeffe J
Objective
Here we present a system which is capable of performing nearly-real time monitoring of UPDRS part-3 motor assessments using a combination of multidimensional statistics and computer vision.
Background
The development of neurotherapeutics is often reliant on subjective end-points. These are subject to issues such as inter-rater and intra-rater variability, rater bias, rater drift or simply poor quality assessment. Consequently quality control is a major concern for those performing clinical trials in this field, and has led to costly innovations such as central rating. Intelligent systems, so-called A.I., are of great interest in this regard because they offer the prospect of an automated “second opinion” in real time, which does not suffer from the problems associated with subjectivity.
Methods
Firstly, a large data-set of over 15,000 UPDRS part-3 assessments was used to characterise a high dimensional feature space, induced by treating a part-3 assessment as a 33 element feature vector. Using a specific distance metric as a test statistic this allowed a p-value to be allocated to a part-3 assessment immediately and in real time.
Secondly, computer vision was used to analyse videos and estimate the correct UPDRS score, allowing a second p-score to be allocated on the basis of this independent analysis within a few minutes of an assessment being completed.1 Combining these two filters results in a highly sensitive system, which could be used to vastly reduce the cost of quality control by focusing attention onto those assessments which appear to be highly improbable, under the assumption of high quality and consistent rating.
Results
The system was validated using a high quality video and rating data set, collected across multiple clinical sites and scored by trained assessors, which we were able to artificially “corrupt”. The system was shown to be both sensitive and specific for detecting “corrupted” assessments.
Conclusions
To our knowledge this system represents the first near-real time intelligent quality control system, which it is apparent could be extended to encompass many more subjective endpoints in this and other conditions.
Figures






Conclusion
These results demonstrate a non-parametric system that can distinguish authentic from inauthentic data. This can be applied on a macro scale and in real-time to identify clinical sites where the data may be inconsistent. The addition of a computer vision analysis improves performance through a direct assessment of source data, rather than just derivative data such as clinical ratings.
References
- Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S. & Sheikh, Y. A. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).↑a↑b
- Marek, K., Jennings, D., Lasch, S., Siderowf, A., Tanner, C., Simuni, T., Coffey, C., Kieburtz, K., Flagg, E., Chowdhury, S. et al. The Parkinson Progression Marker Initiative (PPMI). Progress in Neurobiology 95(4), 629–635 (2011).↑a↑b
About this poster
This research was originally presented as a poster by Machine Medicine Technologies. This HTML version has been reformatted for accessibility and web presentation, while preserving the content of the original poster.