Pragmatic Free Trial Meta
Pragmatic Free Trial Meta is a free and non-commercial open data platform and infrastructure that supports research on pragmatic trials. It collects and distributes cleaned trial data, ratings and evaluations using PRECIS-2. This allows for a variety of meta-epidemiological analyses to compare treatment effect estimates across trials of various levels of pragmatism.
Background
Pragmatic studies are increasingly acknowledged as providing evidence from the real world to support clinical decision-making. However, the use of the term "pragmatic" is inconsistent and its definition and assessment requires clarification. Pragmatic trials must be designed to inform clinical practice and policy decisions, rather than confirm an hypothesis that is based on a clinical or physiological basis. A pragmatic trial should aim to be as similar to real-world clinical practice as possible, including in the recruitment of participants, setting and design, the delivery and execution of the intervention, and the determination and analysis of outcomes and primary analysis. This is a significant difference between explanatory trials, as defined by Schwartz & Lellouch1, which are designed to test the hypothesis in a more thorough way.
Studies that are truly pragmatic should not attempt to blind participants or healthcare professionals, as this may cause bias in estimates of the effects of treatment. Practical trials also involve patients from different healthcare settings to ensure that their results can be generalized to the real world.
Furthermore, trials that are pragmatic must concentrate on outcomes that are important to patients, like the quality of life and functional recovery. This is particularly relevant for trials that involve surgical procedures that are invasive or may have dangerous adverse effects. The CRASH trial29, for instance focused on the functional outcome to compare a two-page report with an electronic system to monitor the health of patients admitted to hospitals with chronic heart failure. Similarly, the catheter trial28 utilized urinary tract infections caused by catheters as its primary outcome.
In addition to these characteristics pragmatic trials should also reduce the requirements for data collection and trial procedures to cut down on costs and time commitments. Finaly these trials should strive to make their findings as relevant to real-world clinical practices as they can. This can be accomplished by ensuring that their analysis is based on an intention-to treat method (as described in CONSORT extensions).
Many RCTs which do not meet the requirements for pragmatism however, they have characteristics that are in opposition to pragmatism, have been published in journals of different types and incorrectly labeled pragmatic. This could lead to false claims of pragmatism and the use of the term should be standardized. The development of the PRECIS-2 tool, which offers a standard objective assessment of pragmatic features is a good initial step.
Methods
In a practical study it is the intention to inform policy or clinical decisions by demonstrating how an intervention could be integrated into routine treatment in real-world situations. This differs from explanation trials, which test hypotheses about the cause-effect relationship in idealised conditions. Therefore, pragmatic trials might have lower internal validity than explanatory trials and might be more susceptible to bias in their design, conduct, and analysis. Despite their limitations, pragmatic studies can provide valuable data for making decisions within the context of healthcare.
The PRECIS-2 tool evaluates the degree of pragmatism in an RCT by assessing it across 9 domains that range from 1 (very explanatory) to 5 (very pragmatic). In this study the domains of recruitment, organisation and flexibility in delivery, flexible adherence, and follow-up received high scores. However, the primary outcome and the method of missing data were scored below the practical limit. This suggests that it is possible to design a trial using excellent pragmatic features without damaging the quality of its outcomes.
It is hard to determine the degree of pragmatism within a specific trial since pragmatism doesn't possess a specific attribute. Some aspects of a research study can be more pragmatic than other. A trial's pragmatism can be affected by modifications to the protocol or the logistics during the trial. Koppenaal and colleagues discovered that 36% of 89 pragmatic studies were placebo-controlled, or conducted prior to the licensing. The majority of them were single-center. They are not close to the norm and are only called pragmatic if their sponsors agree that the trials are not blinded.
Furthermore, a common feature of pragmatic trials is that the researchers try to make their results more meaningful by analysing subgroups of the trial sample. However, this often leads to unbalanced results and lower statistical power, increasing the risk of either not detecting or misinterpreting the results of the primary outcome. This was a problem in the meta-analysis of pragmatic trials because secondary outcomes were not adjusted for covariates' differences at the time of baseline.
Additionally, studies that are pragmatic can pose difficulties in the collection and interpretation of safety data. This is due to the fact that adverse events are usually self-reported, and are prone to delays, errors or coding variations. It is essential to improve the quality and accuracy of the results in these trials.
Results
Although the definition of pragmatism does not require that all trials are 100 100% pragmatic, there are benefits of including pragmatic elements in clinical trials. These include:
Increased sensitivity to real-world issues, reducing cost and size of the study, and enabling the trial results to be faster implemented into clinical practice (by including patients from routine care). However, pragmatic trials may also have disadvantages. For instance, the right kind of heterogeneity can allow the trial to apply its results to many different patients and settings; however, the wrong type of heterogeneity can reduce assay sensitivity, and thus reduce the power of a trial to detect minor treatment effects.
A number of studies have attempted to categorize pragmatic trials, with a variety of definitions and scoring systems. Schwartz and Lellouch1 have developed a framework to distinguish between explanation-based trials that support a physiological or clinical hypothesis, and pragmatic trials that inform the selection of appropriate treatments in the real-world clinical setting. The framework was composed of nine domains that were scored on a 1-5 scale with 1 being more lucid while 5 was more practical. The domains covered recruitment and setting up, the delivery of intervention, flexible compliance and primary analysis.
The initial PRECIS tool3 featured similar domains and scales from 1 to 5. Koppenaal et al10 devised an adaptation of this assessment dubbed the Pragmascope that was easier to use in systematic reviews. They discovered that pragmatic systematic reviews had a higher average score in most domains but lower scores in the primary analysis domain.
This difference in primary analysis domain can be due to the way in which most pragmatic trials approach data. Some explanatory trials, however do not. The overall score was lower for pragmatic systematic reviews when the domains of organisation, flexible delivery, and follow-up were combined.
It is crucial to keep in mind that a study that is pragmatic does not necessarily mean a low-quality study. In fact, there is increasing numbers of clinical trials that use the term "pragmatic" either in their abstracts or titles (as defined by MEDLINE but which is neither precise nor sensitive). The use of these terms in abstracts and titles could indicate a greater understanding of the importance of pragmatism but it is unclear whether this is manifested in the content of the articles.
Conclusions
As appreciation for the value of evidence from the real world becomes more widespread the pragmatic trial has gained popularity in research. They are clinical trials that are randomized that evaluate real-world alternatives to care rather than experimental treatments under development. They involve populations of patients that are more similar to the patients who receive routine care, they employ comparisons that are commonplace in practice (e.g. existing drugs) and depend on the self-reporting of participants about outcomes. This method can help overcome limitations of observational studies which include the biases associated with reliance on volunteers and limited availability and coding variability in national registry systems.
Other advantages of pragmatic trials are the possibility of using existing data sources, as well as a higher likelihood of detecting meaningful changes than traditional trials. However, they may be prone to limitations that compromise their validity and generalizability. Participation rates in some trials may be lower than expected due to the health-promoting effect, financial incentives, or competition from other research studies. The need to recruit individuals in a timely fashion also restricts the sample size and the impact of many practical trials. Additionally, some pragmatic trials do not have controls to ensure that the observed differences aren't due to biases in trial conduct.
The authors of the Pragmatic Free Trial Meta identified RCTs published from 2022 to 2022 that self-described themselves as pragmatic. The PRECIS-2 tool was employed to assess the pragmatism of these trials. It includes areas like eligibility criteria as well as recruitment flexibility, adherence to intervention, and follow-up. They found 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or above) in at least one of these domains.
Trials that have a high pragmatism score tend to have broader eligibility criteria than traditional RCTs which have very specific criteria that are not likely to be found in the clinical setting, and include populations from a wide variety of hospitals. These characteristics, according to the authors, may make pragmatic trials more relevant and useful in everyday clinical. However they do not ensure that a study is free of bias. Furthermore, the pragmatism of the trial is not a predetermined characteristic; a pragmatic trial that doesn't possess all the characteristics of an explanatory trial can yield valid and useful results.