中国地区询价 
1 (250) 388-3531 
+33 4 86 68 68 10 
How Feasibility Studies Solve Three Frequently Asked Testing Questions
Three frequently asked questions from lab clients are:
- How many samples do I need?
- What acceptance criteria should I use?
- Can I submit first results to a regulatory body?
“Run a feasibility study first” is usually the best answer to all these questions. This ViVitro Labs Tech Brief explains how and why a feasibility study should be part of any new device testing procedure.
Feasibility studies are always recommended for first time lab customers. They help answer unknown questions including required sample size and acceptance criteria. They also help de-risk lengthy verification and validation studies by identifying device or procedural issues as quickly and cheaply as possible. These initial studies are often the starting point to successfully bring a device to market.
Acceptance criteria and the corresponding sample size required for a test are closely interrelated. An iterative approach is often required to correctly define them. Performing feasibility studies early in the device development process ensures testing occurs as quickly and efficiently as possible. Feasibility studies are also recommended to help ensure the success of large investment Verification & Validation (V&V) studies.
1. Sample size – How many samples do I need?
When it is not feasible to test every single device, use discrete sampling based on a small subset of devices to make conclusions about the overall safety or performance of a product. While some standards have clearly defined sample size requirements, many do not. When there is not a clearly defined requirement, it is up to the manufacturer to specify and justify the sample sizes.
Two common ways of determining sample size:
The first method is to use an Acceptance Quality Level (AQL) table. This is useful for pass/fail testing or attribute testing. The AQL method uses tables and charts that allow sample size to be determined based on production volume (lot size), acceptable level of defects, and the level and severity of inspection to be performed.
For devices with costly manufacturing processes such as heart valves, LVADs, AAAs and TAAs, the lot size is typically very small. However, with simpler devices such as guidewires the lot size may start to increase. Device cost cannot be the only factor in selecting sample size, as it is not adequate justification from a regulatory perspective. Justification can be based on class of medical device, criticality of tested parameters, destructive or not destructive testing, or reliability of the process (manual/automatic, stable/unstable). Lot history of past production runs, known as the severity of inspection, can reduce the burden of inspection, and should be considered when determining the appropriate sample size.
ISO 16269-6:2014 gives more guidance on this sampling method. Note that this method requires a normal distribution. A Shapiro-Wilk test can be used to ensure the presence of a normal distribution. If sampling from several different groups, a Levene’s Test should also be applied to ensure equality of variances.
The second common method is a statistical approach using a corresponding confidence and reliability level. The confidence interval may look something like the following:
“This testing shows statistically with at least a 95 % confidence that 99,9 % of the devices (reliability) will not fail at or below the acceptance criteria.”
Some standards give acceptable values for confidence and reliability levels, but most often it will be up to the manufacturer to specify and justify acceptable values based on a risk analysis. The risk analysis depends on the potential failure mode of the device or component, the associated harm, and the anticipated frequency. Failure Modes and Effects Analysis (FMEA) is one of the most common approaches to understand risks associated with a device. For regulatory purposes, acceptance criteria, confidence interval, and reliability level can all be captured in an FMEA document.
In either approach a feasibility study can determine the variability among samples which will help guide the manufacturing process refinement and/or future sample size consideration used for V&V studies. Knowing the averages and standard deviations for a particular attribute is a critical step. Often the only way to determine these are through measurement.
No matter what approach is used, the more samples used, the closer the sample will be to the actual population. There is, however, a real tradeoff between the sample size and the associated cost and time implications. Producing large volumes of devices, particularly at the early stage, is very cost and time intensive and not feasible for many startup ventures. Multiple factors need to be considered, including funding, time to market, and labor resources. Regulators will not be interested in these constraints. Having robust justifications for worst case testing is the best method to reduce sample size. Robust worst-case justification can be a complex process and may require additional studies or methods such as Finite Element Analysis (FEA) or numerical simulations. The cost of providing these justifications should be balanced against running a study with a large number of samples.
An empirical rule of thumb is the minimum statistical significance is between three to five samples. A challenge with only using three devices is that if an outlier is included in the sample, it will be exceedingly difficult to determine an accurate average. For this reason, using five devices is a more conservative approach to help identify and justify any potential outliers. Testing a single device may be appropriate during the very early proof-of-concept phase, but beyond that it can be a risky approach and is generally not recommended.
Either Acceptance Quality Level (AQL) tables or confidence intervals are useful tools for determining and justifying sample sizes. A bare minimum 3-5 samples are generally recommended. A feasibility study can be helpful in determining the variability among samples.
2. What acceptance criteria should I use?
Acceptance criteria are critical to design verification and validation for medical devices. Manufacturers can demonstrate that their device is safe and effective by testing against predefined acceptance criteria. Acceptance criteria are most important during the late-stage device development process, but consideration should be given to developing them as early as possible in the design process.
Methods to define and justify appropriate acceptance criteria
Textbooks or engineering handbooks can be used as a reference for basic parameters such as simple material property characterization or dimensional tolerances. Often with medical devices these acceptance criteria are not so easily obtained and defined.
A more common approach to defining acceptance criteria is to derive them. Use engineering fundamentals and physical equations to derive acceptance criteria. For example, knowing that Pressure = Force / Area, a known pressure acting over a known area can be used to determine an unknown force on a component or device. In some cases, the situation is too complex to model using equations.
A literature search is used for clinical issues that are challenging to break down into physical equations. Several sources may be combined and synthesized together to determine the required information. A common example is investigations pertaining to anatomy and physiology, such as determining anatomical dimensions, displacements, or loading. ViVitro Labs expertise includes performing literature reviews using the latest scientific research built on seminal knowledge and decades of experience.
When none of the previous options are sufficient, the only remaining option is a feasibility study. Often comparative samples or predicate devices are used as part of the study to help derive acceptance criteria. These types of studies are typically conducted as non-inferiority studies. If the device is a “Me Too” device, it may be straight forward to identify which samples should be acquired from a competitor. ViVitro Labs stocks some devices and can aid in obtaining others if required.
Novel and innovative devices are more challenging to select the closest device on the market that is applicable to the acceptance criteria in question. Each different parameter may require a different comparative device, leading to a large number of required feasibility studies and devices. Some devices may not have comparative samples, so feasibility testing on the novel device can be used to inform and define acceptance criteria.
Acceptance criteria are mandatory to prove a device is safe and effective. There are a variety of approaches that can be used to determine acceptance criteria, including material properties or literature reviews. Often the best approach is to run a feasibility study. These studies may involve existing competitive devices already on the market but can be leveraged to gain a deeper understanding of how a medical device performs.
3. Can I submit first results to a regulatory body? Not without a feasibility study.
Although you can submit a first report to the regulatory authorities, it is unwise, because first tests rarely pass regulatory submissions for a variety of reasons. Unexpected results can arise from many sources including unanticipated device behavior in the test apparatus or under the specified test conditions or improper steps laid out in a test protocol. Finally, the test samples used during a feasibility study may not integrate all the design features of the clinical unit and may not be representative of the final product. All of these issues are mitigated by first performing a small feasibility test on a subset of samples, often without the cost of a control or reference device.
No one wants to see a device failing a V&V study as this can introduce significant program delays and increase development costs unnecessarily. To prevent unexpected results during lengthy and costly V&V studies, we recommend (and some tests require) that a feasibility test be performed prior to a V&V study.
Many companies try to jump straight to V&V studies in order to save time and money, but their efforts rarely work and often end up increasing cost and delaying projects instead. A device performing well on in-house equipment does not guarantee its performance will be the same on ISO 17025 accredited testing platforms. In some cases it is not possible to provide appropriate justifications for test conditions, acceptance criteria, or sample size, without first performing feasibility work to understand a device more deeply.
Feasibility studies assess the device under controlled conditions. These studies are often executed very quickly with a limited number of samples. Conducting feasibility studies early in the development of a medical device helps to inform acceptance criteria and sample size considerations that will be required during device verification and validation). Feasibility studies allow manufacturers to begin building trust with vendors which can be critical in during final device approvals. Feasibility studies also allow the lab to become familiar with the specificity of a particular medical device. This can help development of more robust V&V studies which in turn decreases the risk of unexpected results. V&V studies typically require much more time, samples, and investment than feasibility studies. Performing a feasibility test prior to a V&V study helps ensure the V&V is a success.
Key to a properly executed feasibility study are the protocol and report
These elements provide the documentation and traceability required to fully define the acceptance criteria. They will form the foundation of a medical device submission package. The protocols and reports should contain an appropriate level of detail and justification relative to the stage of device development.
Feasibility studies can also be used as inputs or validation for in-silico approaches like Finite Element Analysis (FEA) or Computational Fluid Dynamics (CFD). These tools allow designers to further understand and evaluate how devices will perform under a wide range of conditions and configurations. In-vitro feasibility studies give confidence to design teams prior to engaging in animal studies. Conducting in-vitro feasibility studies before undergoing pre-clinical animal testing ensures there are no surprises when testing with animals. The animal lab can focus on key questions that can only be answered in an animal model.
Samples can often be re-used to demonstrate multiple acceptance criteria for both feasibility testing and submissions to a regulatory body. Tests like failure mode and destructive testing can provide significant information about the safety factors of the device in relation to its nominal operating conditions but end up damaging a sample. To get the most possible information from a sample before being destroyed, samples should progress
Please contact our sales team, Doug Bigrigg, or Jérémie Decressac to discuss your testing needs and how feasibility testing can reduce costs and save money for your regulatory submission.