A pilot evaluation of a computer-based psychometric test battery designed to detect impairment in patients with cirrhosis

Introduction

Hepatic encephalopathy (HE) is one of the most debilitating consequences of liver disease and is characterized as diffuse brain dysfunction caused by liver insufficiency and/or portosystemic shunting.¹ Due to the complex pathogenesis, the neurological and/or psychiatric manifestations of the disease vary according to the extent of its severity.

The full clinical presentation manifests in patients with overt HE (OHE), which is an event that defines liver decompensation. Numerous neurological and neurocognitive domains are affected, which have protean manifestations including extrapyramidal dysfunction, asterixis, myelopathy, progressive memory impairment, disorientation for time and space, acute confusion and coma.² Conversely, in minimal HE (mHE), there is none of the clinical or obvious cognitive dysfunctions of OHE and it is only detectable by psychometric testing of psychomotor speed, executive functions or neurophysical alterations.³ Even though mHE is blanketed by its subclinical presentation, it may have a significant impact on activities of daily living, such as impairment of the ability to drive or operate machinery safely, owing to impaired cognitive and locomotive function.⁴

Psychometric testing to identify early HE development is crucial for initiating treatment and halting disease progression to OHE, reducing the overall burden of disease to the local health care system. Furthermore, it is important to assess those with suspected mHE for the ability to drive or operate machinery safely, to minimize the personal and societal consequences of accidents.⁴

In a report by the United Kingdom Health Protection Agency, it was estimated that by 2020, there would be a significant increase in the number of people living with virally related chronic liver disease in the UK and a 10% rise in cirrhosis prevalence.⁵ With the added burden of alcohol misuse and non-alcohol-related fatty liver disease, the number of deaths from hepatic disorders is rising in the UK, compared to other parts of Europe. There is increasing pressure for quick, cost-effective ways of detecting mHE for appropriate management and treatment, thereby reducing the impact on affected individuals and their carers.

The psychometric test battery recommended by the recent European Association for the Study of the Liver/American Association for the Study of Liver Diseases guidelines is the Psychometric Hepatic Encephalopathy Score (PHES).⁶ This collection of neurocognitive tests has been designed to examine motor speed, visual perception, visual–spatial orientation, visual construction, concentration and attention and, to a lesser degree, memory.⁶ There are four different versions of each of the tests that make up the PHES to prevent patients who require repeated tests from learning and recognizing them.

The components are

1. Number connection test A: measures concentration, mental tracking and audio visuomotor speed.
2. Number connection test B: measures concentration, mental tracking and audio visuomotor speed but with more complexity than Test A.
3. Digit symbol test (DST): measures psychomotor and visuomotor speed.
4. Line tracing test (LTT): measures visuomotor and visuospatial components for both speed and accuracy.
5. Serial dotting test (SDT): measures psychomotor speed.

The evaluation starts with a practice phase consisting of a small sample of items, so that the patient can become familiar with how each test is designed and what is required of them. The components are basic, do not require expensive equipment and can be performed in most clinical and office settings, if time allows. However, the ideal environment is a quiet room with good lighting to standardize conditions. Although feasible, completing the assessment by the bedside on a busy ward is rarely appropriate because of noise and distractions.

The PHES test has its limitations, given that it can be influenced by education, cultural background, lack of sleep, emotional upset or language difficulties.⁴ It has been adapted for use in Spain, Italy, UK and India.^7–10 However, it is not used in the USA, where alternatives, such as the more-time-consuming repeatable battery for the assessment of neuropsychological status battery, which consists of 12 subtests, are used. As psychometric testing is a reimbursable cost, and to comply with the requirements of US health insurance companies, the testing is more intensive than in Europe.¹¹ Although the PHES has been adopted in many centers worldwide to screen for the presence of mHE, its clinical utility has been criticized. Kircheis et al performed a placebo-controlled, clinical multicenter trial with 217 cirrhosis patients, to study the discrepancy in the expected and observed severity of HE.¹² The study revealed that up to 50% of the patients were wrongly allocated with regard to HE severity because of the difficulty in differentiating mild OHE from mHE. Moreover, mistakes were made in analyzing and scoring the results of PHES. The contradictory results prompt recognition of other psychometric investigations of mHE.

As an alternative to paper-and-pencil–based cognitive tests, several computer-based cognitive tests have been developed, some of which have become too expensive for use in routine clinical settings. However, Cogstate^™ (Cogstate Inc., New Haven, CT, USA) has developed a number of computer-based tests that are already in use for human immunodeficiency virus (HIV) patients with cognitive difficulties and have the potential to be adapted for use in mHE.¹³ The battery comprises a series of computer-based adaptations of standard neuropsychological tests that assess a range of cognitive functions, including psychomotor speed, attention, learning, and visual and verbal working memory. The battery can be customized to test the cognitive functions appropriate for certain diseases and has been validated for use in patients suffering from Alzheimer’s disease, schizophrenia, mild traumatic brain injury and HIV.^13,14 The battery typically takes between 20 and 40 minutes to complete depending on the number of cognitive domains assessed and gives individuals 1) various card games on a green background, 2) maze games chasing the targets pathway around the maze and memorizing it, 3) pairing shapes and memorizing where they are with places covered on the screen by colored balls. The first and last task involve an idealized “shopping list,” which is read out to each participant at the beginning of the battery. Participants are 1) asked to recall immediately after hearing the list, and 2) then recall again at the very end of the session without hearing the list again, once they have completed the other tasks.

Normative data generated from adults from 18 to 89 years in age are available for the Cogstate tests used in this study. The data are derived from a healthy population of subjects in a series of clinical trials, research and academic studies. The baseline sessions were included in the normative database and only a single session was included for each subject. The participants were recruited from countries in North and South America, Europe, Asia and Australia.¹³

We hypothesized that a tailor-made computer battery would be more discriminant in mHE diagnosis than the standard PHES battery and that it would minimize both ceiling and floor effect of PHES, when highest and lowest scores are unable to discriminate the patient’s level of ability. Based on the hypotheses, we 1) compare individual Cogstate battery tests with PHES tests, and 2) derive a simple Cogstate test battery that can be used for detection of mHE in routine clinical practice.

Patients and methods

The patient population consisted of 27 subjects (28 males, nine females of mean [range] age 56.9 [36–69] years) with biopsy-proven cirrhosis, attending hepatology outpatient clinics between April 2015 and April 2016 at the Imperial College Healthcare Trust, London, UK. The underlying etiology of the cirrhosis was hepatitis C infection (n=11), alcohol misuse (n=6), non-alcoholic fatty liver disease (NAFLD) (n=2), hepatitis C and NAFLD (n=1); hepatitis C and alcohol misuse (n=1), autoimmune hepatitis (n=2), hepatitis B infection (n=1), primary sclerosing cholangitis (n=1), biliary (n=1) and hemochromatosis (n=1). All subjects were able to understand and communicate in the English language. All were given a minimum of 48 h to read the patient information sheet, prior to recruitment and all gave informed, written consent, according to the guidelines adopted by the 18th World Medical Assembly in the 1964 Declaration of Helsinki on Human Rights (World Medical Assembly, 1964), and in accordance with the London – Fulham Research Ethics Committee approval (LREC reference no. 05/Q0411/71).

Exclusion criteria were recent excess alcohol consumption within the preceding 6 months, current intravenous or nasal illicit drug use, usage of psychoactive and antipsychotic drugs, known cerebrovascular disease, consumption of drinks containing caffeine 2 h prior to planned psychometric testing, or current or previous clinical evidence of OHE.

All subjects were examined clinically and had no neurological abnormalities, nor any clinical evidence of OHE. Each had blood drawn for standard liver biochemistry with Child–Pugh and United Kingdom end-stage liver disease scores being calculated to grade the functional level of cirrhosis for each patient.^15,16 These were used as co-variables with PHES and Cogstate results.

All subjects underwent both the English PHES battery testing and the modified Cogstate battery tests. To avoid test bias, the order of testing was alternated: participants with an odd subject number were tested with the computer-based Cogstate battery first and the PHES battery second, and the subjects with even subject numbers were tested with the PHES battery first and the Cogstate battery second.

The PHES battery consisted of five paper-and-pencil–based tests: the number connection tests A and B (NCT-A/B), DST, SDT and LTT.⁶

The scoring of the PHES is performed by taking account of 1) the age of the subject, 2) time of test completion, and 3) accuracy of test completion with a score for healthy controls, which ranges from ≥−2 to +5. Scores of <−2 to −15 indicate varying levels of HE from mHE through to OHE.

Test results within ±1SD from the age-adjusted mean were scored with 0 points, those between −1 and −2SD were scored −1, those between −2 and −3SD beyond the mean were scored −2 points and those worse than −3SD were scored with −3 points. Results better than means + 1SD were scored +1, which allows score results to range from +6 to −18 points. The final English PHES z score (2 decimal points) was normalized for the UK-based population.¹⁷ It has taken into account ethnicity, years of education, whether subjects were educated in the UK or abroad, and weekly alcohol intake, measured in grams.

The Cogstate battery consisted of

1. International shopping list (ISL). A list of 12 items is read out aloud to the participant three times. An ISL score (total number of correct answers across all three learning trials [ISL cor]) is derived from the number of correctly remembered items.
2. Chase test (CT). The participant has to chase the target in the grid following the exact journey the target has taken in the grid. The grid is 10×10 in size and travel up, down and side to side by one or two squares is allowed.
3. Groton Maze learning test (GMLT). The participant has to remember the 28-step pathway that is used to get from the top of the grid to the bottom. The grid is 10×10 in size. The pattern can travel up, down and side to side, but not diagonally, while travel must be only by one square at a time and participants cannot move back on the pathway. Feedback is given with visual and auditory cues to indicate whether the selected box is correct or incorrect.
4. Detection task. In this test, the playing cards all depict the same joker. The subject is asked to press the Yes key as soon as the card in the center of the screen turns face up. The software measures the speed and accuracy of each response.
5. Identification test (IDN). In this test, the playing cards are all either red or black jokers. The subject is asked whether the card displayed in the center of the screen is red. The subject responds by pressing the Yes key when the joker card is red and No when it is black. The software measures the speed and accuracy of each response.
6. One card learning task (OCL). In this test, the playing cards are identical to those found in a standard deck of 52 playing cards (without the joker cards). The subject is asked whether the card displayed in the center of the screen was seen previously in this test. The subject responds by pressing the Yes or No key. The software measures the speed and accuracy of each response.
7. One back task (ONB). In this test, the playing cards are identical to those found in a standard deck of 52 playing cards (without the joker cards). The subject is asked whether the card displayed in the center of the screen is the same as the card presented immediately previously. The subject responds by pressing the Yes or No key. Because no card has been presented yet on the first trial, a correct first response is always No. The software measures the speed and accuracy of each response.
8. Continuous paired associate learning task (CPAL). This test consists of a single amoeboid shape displayed in the center of the screen surrounded by a number of blue-filled circles. Beneath all but two of the blue spheres are amoeboid shapes, one of which matches the central display; the two remaining circles are distractors. In the exposure phase of the test all of the to-be-remembered pattern–location associations are presented on the computer screen simultaneously. After the exposure phase, a pattern in shown in the center, and the subject is required to select the peripheral location where an identical pattern is hidden beneath the blue sphere. The participant has to identify where the picture is located that matches the picture in the center target. All the pictures are covered by blue balls. Once identified, the center picture changes until each picture is paired.
9. International shopping list delayed recall (ISLR). The final task is a return to the shopping list, but this time the participant is asked to remember items on the list that were read out at the beginning of the computer battery testing. This, therefore, involves delayed recall.

The Cogstate battery was customized specifically for liver disease patients, based on those tests that were discriminant for mHE with eight of the 11 Cogstate tests used in a modified battery: 1) international shopping list test, 2) CT, 3) GMLT, 4) detection test (DET), 5) IDN, 6) OCL, 7) ONB, and 8) ISLR (Cogstate Inc.).¹⁸

Consequently, we selected the following variables from each Cogstate test:

The individual patient scores for each Cogstate test were referenced to average test scores that were derived from an age-matched healthy population. This was performed automatically by the Cogstate system (Cogstate Inc.).

A Spearman’s correlation (r) analysis was performed to compare the individual subtest from the Cogstate and PHES battery. Results were visualized in a heatmap.

A final Cogstate test score (CS_total) was then derived 1) by combining the referenced subtest scores with equal weightings, and 2) by linear modeling using a non-negative linear least-squares approach,¹⁹ as implemented in the R package nnls, which is available via the Comprehensive R Archive Network (www.cran.r-project.org). The optimal CS_total cutoff value for diagnosing mHE was then determined with the Youden’s J statistic using the dichotomous mHE classification determined by a PHES value of <−2.

Results

A total of 27 participants completed both the PHES and the Cogstate battery tests. The median PHES score was −5 (range 0 to −13). Using a PHES total cutoff value of <−2, classified 23 patients (85%) as mHE and five patients (15%) as non-mHE.

Equivalence (and nonequivalence) of PHES and Cogstate subtests

Correlation between PHES and Cogstate battery tests is visualized by the heatmap (Figure 1) and the correlation table (Table 1). The highest correlation between test scores was observed for Cogstate’s ISL and PHES’ Serial Dotting task (ρ=0.51), followed by IDN and Line Tracing Time (ρ 0.45) (Figure 1). Another high correlation magnitude was observed for both the Cogstate ISL and ISLR with PHES digital symbol task (ρ=0.42 and 0.41, respectively).

Figure 1 Heatmap demonstrating correlations between subtests of the PHES test (x-axis) and subtests of Cogstate battery (y-axis). Abbreviations: CPAL, continuous paired associated learning; DS, digital symbol task; IDN, identification test; ISL, international shopping list; ISLR, international shopping list delayed recall; LT-Error, line tracing test error; LT-Time, line tracing test time; NCT-A, number connection test A; NCT-B, number connection test B; OCL, one card learning; ONB, one back task; PHES, Psychometric Hepatic Encephalopathy Score; SD, serial dotting test; Total.lmW, Cogstate total score using non-negative linear model weighting approach; Total.equalW, Cogstate total score using equal weighting approach.

Table 1 Correlations between subtests of Cogstate battery and subtests of the PHES test Notes: **Correlation is significant at the 0.01 level (two-tailed). *Correlation is significant at the 0.05 level (two-tailed). Abbreviations: CPAL, continuous paired associated learning; DS, digital symbol; IDN identification test; ISL, international shopping list; ISLR, international shopping list delayed recall; LT-Error, line tracing error; LT-Time, line tracing time; NCT-A, number connection test A; NCT-B, number connection test B; OCL, one card learning; ONB, one back task; PHES, Psychometric Hepatic Encephalopathy Score; SD, serial dotting test; Sig., significance.

CPAL showed mainly low-magnitude correlations (maximally with PHES SDT, ρ=0.24), and similar observations were made for ONB (ρ=−0.21 with DST) and OCL (ρ=0.13 with LTT error).

Tables 2 and 3 both show a strong correlation between the ISL task from Cogstate and PHES total score. This is the only task that is present in all the independent variables in the analysis-of-covariance table, where the P values are <0.05.

Table 2 Model summary regression: Cogstate predicting PHES total Notes: aPredictors: (Constant), ONB, CPAL, OCL, ISLR, ISL, IDN. bPredictors: (Constant), ONB, OCL, ISLR, ISL, IDN. cPredictors: (Constant), OCL, ISLR, ISL, IDN. dPredictors: (Constant), OCL, ISL, IDN. ePredictors: (Constant), ISL, IDN. fPredictors: (Constant), ISL. Abbreviations: CPAL continuous paired associated learning; IDN, identification test; ISL, international shopping list; ISLR, international shopping list delayed recall; OCL, one card learning; ONB, one back task; PHES, Psychometric Hepatic Encephalopathy Score; Sig. F change, significance of F change.

Table 3 Analysis of covariance of Cogstate (independent variable) with PHES Notes: bPredictors: (Constant), ONB, CPAL, OCL, ISLR, ISL, IDN. cPredictors: (Constant), ONB, OCL, ISLR, ISL, IDN. dPredictors: (Constant), OCL, ISLR, ISL, IDN. ePredictors: (Constant), OCL, ISL, IDN. fPredictors: (Constant), ISL, IDN. gPredictors: (Constant), ISL. Abbreviations: CPAL, continuous paired associated learning; IDN, identification test; ISL, international shopping list; ISLR, international shopping list delayed recall; OCL, one card learning; ONB, one back task; PHES, Psychometric Hepatic Encephalopathy Score; Sig, significance.

Cogstate battery as a tool for mHE diagnosis

For each participant, the Cogstate CPAL, IDN, ISL, ISLR, OCL and ONB scores were linearly combined to form a composite total Cogstate score. Assigning equal weightings to each subscore was not able to recover the mHE/non-mHE class membership distribution determined by a PHES total score of <−2 (Figure 2, upper panel).

Figure 2 Evaluation of Cogstate battery utility in mHE diagnosis by equal weights approach (upper panel) and linear model approach (lower panel). Note: The PHES cutoff value for mHE was a score of <−2. Abbreviations: PHES, Psychometric Hepatic Encephalopathy Score; mHE, minimal hepatic encephalopathy.

However, in a non-negative linear modeling approach, individual weightings were assigned to the ONB, identification test (IDN) and the ISLR tasks (model coefficients: 1.69, 1.54 and 1.21, respectively). The CPAL task was assigned a slightly lower weighting (coefficient =0.80), and the ISL and ONB were assigned coefficients of zero, indicating that the subtests may be limited for mHE diagnosis. For participants classified as non-mHE, the final Cogstate total score ranged from 2.5 to −5.7 with a median value of −0.2. Participants classified as having mHE had on average lower Cogstate scores with a median value of −4.7, ranging from 2.7 to −11.2 (Figure 2, lower panel).

Discussion

Our main findings indicate that certain components of the PHES battery may not be as diagnostically discriminant for mHE as had been previously reported, despite the battery being widely regarded by the International Community as the gold standard. The test battery relies on patients being able to count, follow instructions and recall the Roman alphabet. In the UK, language can be a problem if English is not a patient’s first medium, or if they are illiterate, thereby limiting its use in a multicultural setting. We controlled for this by recruiting only those patients who can maintain communication in the English language. Centers in India have addressed this issue by replacing recall of the Roman alphabet with figure connection tests as an alternative.⁹

We found that physical problems also inhibited patients from performing well in the PHES test, particularly during the line tracing task (subtest 5 of the PHES battery), where a steady hand is required as part of the test. A subject during the study with a previous arm injury subsequently found the test difficult. Similar physical limitations in patients with good cognition may lead to false-positive results.

The serial dotting (PHES subtest 4) test results of the 27 subjects varied between −2 and −3 without a range in the distribution from the total PHES score. It would seem appropriate in future to consider whether this subtest is useful in larger-scale studies with a view to developing a more streamlined four-component battery, rather than a five-component battery.

In the present study, we limited the number of people scoring the PHES test to two and both of them independently corroborated the other’s results. To ensure optimal results, we observed that the surrounding environment for cognitive testing needs careful consideration for both PHES and Cogstate, as it may have an influence on how a patient performs during the testing. The room needs to be in a quiet location, with good lighting and a comfortable temperature with mobile phones switched off.

The Cogstate battery was well received by patients: several expressed that they enjoyed the test in feedback received. Some found using the mouse to navigate around the Groton Maze and Chase the Target quite difficult because of limited experience with computers. Despite a strong correlation, the Groton Maze was removed from the test battery after the eleventh patient as it took an average of 60–90 minutes to complete per person, which is a major limitation in our aim to devise a practical battery for use in clinical scenarios. Once removed, the full modified Cogstate took an average of 20–40 minutes. CPAL err also correlated poorly, but this may be because of the level of task difficulty. Measures of psychomotor speed, CPAL err and Serial Dotting correlated poorly.

The ISL and the ISLR also had strong positive correlations with the PHES DST. These tests did not require patients to read English because the items were read to them by the person supervising the test. They had to repeat back the items they could remember. The shopping test seemed a very practical, but simple test that would be very relevant to daily activities of living, which most nursing frameworks advocate.²⁰

With regard to practice effects, the PHES test may have a learning effect on repeat testing. Patients may be able to remember the components from previous testing, which could influence results.^4,7–10 Conversely, the computer Cogstate battery was designed to be repeatable even over brief periods of retesting.

We, therefore, suggest for clinical testing that an adapted form of PHES without the Serial Dotting (four components instead of five) should be further evaluated. However, in order to adjust the PHES total score, an age-matched healthy control group would be required for validation, needing further research into this area. Modification of the current PHES battery has previously been suggested by Riggio et al, who identified that a simplified PHES without NCT-A/B was as efficient as PHES in detecting mHE and predicting subsequent occurrence of OHE.²¹

We recommend Cogstate to be used for drug studies and clinical trials, as it is a time-efficient test and the analysis is simple. The modified form is potentially suitable for clinical practice. However, the use of the ISL and the ISLR would be both easy to facilitate and be discriminatory in clinical practice, perhaps in combination with a modified four-component PHES test.

Further research should aim at increasing the sample size of subjects with an equal number of healthy controls, as well as focusing on its diagnostic efficacy within the different stages of HE. If the preliminary findings of our research are validated, then development of a “smartphone App” would be indicated, incorporating an adapted version of the ISL from the Cogstate battery and the Digit Symbol task from the PHES battery, with a simple scoring system that would be easy, quick and assessable to use in clinical practice.

Acknowledgments

The Cogstate battery was provided free of charge for this study. All authors acknowledge the support of the United Kingdom National Institute for Health Research Biomedical Research Centre at Imperial College London for infrastructure support. NAC was supported by a grant from the Trustees of St Mary’s Hospital, Paddington (London, UK). MMEC is supported by a Fellowship from the Sir Halley Stewart Trust (Cambridge, UK). MMEC and SDT-R hold grants from the United Kingdom Medical Research Council and the Wellcome Trust Institutional Strategic Support Fund. We would like to express our gratitude to the Stratified Medicine Graduate Training Programme in Systems Medicine and Spectroscopic Profiling (STRATiGRAD) for funding TK.

Disclosure

The abstract of this paper was presented at the International Society for Hepatic Encephalopathy and Nitrogen Metabolism (ISHEN) Conference 2017 as a poster presentation with interim findings. The poster’s abstract was published in Volume 7, Supplement 1 (February 2017) in the Journal of Clinical and Experimental Hepatology (10.1016/j.jceh.2017.01.037). The authors report no other conflicts of interest in this work.

References

1.		Vilstrup H1, Amodio P, Bajaj J, et al. Hepatic encephalopathy in chronic liver disease: 2014 Practice Guideline by the American Association for the Study of Liver Diseases and the European Association for the Study of the Liver. Hepatology. 2014;60(2):715–735.
2.		Ferenci P, Lockwood A, Mullen K, Tarter R, Weissenborn K, Blei AT. Hepatic encephalopathy–definition, nomenclature, diagnosis, and quantification: final report of the working party at the 11th World Congresses of Gastroenterology, Vienna, 1998. Hepatology. 2002;35(3):716–721.
3.		Guerit J, Amantini A, Fischer C, et al. Neurophysiological investigations of hepatic encephalopathy: ISHEN practice guidelines. Liver Int. 2009;29(6):789–796.
4.		Amodio P, Montagnese S, Merkel C. Attention: minimal hepatic encephalopathy and road accidents. Hepatology. 2012;55(4):985–987.
5.		Health Protection Agency. (2013). Health Protection Agency annual report and accounts 2012 to 2013. Available from: www.gov.uk/government/uploads/system/uploads/attachment_data/file/246760/0174.pdf. Accessed August 10, 2017.
6.		Weissenborn K, Ennen JC, Schomerus H, Rückert N, Hecker H. Neuropsychological characterization of hepatic encephalopathy. J Hepatol. 2001;34(5):768–773.
7.		Romero Gomez M, Cordoba J, Jover R, et al; Red Nacional de Investigacion en Encefalopatia Hepatica. [Normality tables in the Spanish population for psychometric tests used in the diagnosis of minimal hepatic encephalopathy]. Med Clin (Barc). 2006;127(7):246–249. Spanish.
8.		Amodio P, Campagna F, Olianas S, et al. Detection of minimal hepatic encephalopathy: normalization and optimization of the Psychometric Hepatic Encephalopathy Score. A neuropsychological and quantified EEG study. J Hepatol. 2008;49(3):346–353.
9.		Dhiman RK, Kurmi R, Thumburu KK, et al. Diagnosis and prognostic significance of minimal hepatic encephalopathy in patients with cirrhosis of liver. Dig Dis Sci. 2010;55(8):2381–2390.
10.		Weissenborn K. PHES: one label, different goods?! J Hepatol. 2008;49(3):308–312.
11.		Mooney S, Hasssanein TI, Hilsabeck RC, et al; UCSD Hepatology Neurobehavioral Research Program. Utility of the repeatable battery for the assessment of neuropsychological status (RBANS) in patients with end-stage liver disease awaiting liver transplant. Arch Clin Neuropsychol. 2007;22(2):175–186.
12.		Kircheis G, Fleig WE, Görtelmeyer R, Grafe S, Häussinger D. Assessment of low-grade hepatic encephalopathy: a critical analysis. J Hepatol. 2007;47(5):642–650.
13.		Maruff P, Thomas E, Cysique L, et al. Validity of the CogState brief battery: relationship to standardized tests and sensitivity to cognitive impairment in mild traumatic brain injury, schizophrenia, and AIDS dementia complex. Arch Clin Neuropsychol. 2009;24(2):165–178.
14.		Lim YY, Ellis KA, Harrington K, et al, The Aibl Research Group. Use of the CogState Brief Battery in the assessment of Alzheimer’s disease related cognitive impairment in the Australian Imaging, Biomarkers and Lifestyle (AIBL) study. J Clin Exp Neuropsychol. 2012;34(4):345–358.
15.		Pugh R, Murray-Lyon I, Dawson J, Pietroni M, Williams R. Transection of the oesophagus for bleeding oesophageal varices. Br J Surg. 1973;60(8):646–649.
16.		Barber K, Madden S, Allen J, Collett D, Neuberger J, Gimson A; United Kingdom Liver Transplant Selection and Allocation Working Party. Elective liver transplant list mortality: development of a United Kingdom end-stage liver disease score. Transplantation. 201192(4):469–476.
17.		Marks M, Jackson C, Montagnese S, et al. 300 Derivation of a Normative UK Database for the Psychometric Hepatic Encephalopathy Score (PHES): Confounding Effect of Ethnicity and test Scoring. J Hepatol. 2008;48(Suppl 2):S119.
18.		Morgan MY, Amodio P, Cook NA, et al. Qualifying and quantifying minimal hepatic encephalopathy. Metab Brain Dis. 2016;31(6):1217–1229.
19.		Lawson CL, Hanson RJ. Solving least squares problems; SIAM: 1995; Vol. 15.
20.		Roper N. A model for nursing and nursology. J Adv Nurs. 1976;1(3):219–227.
21.		Riggio O, Ridola L, Pasquale C, et al. A simplified psychometric evaluation for the diagnosis of minimal hepatic encephalopathy. Clin Gastroenterol Hepatol. 2011;9(7):613–616. e1.