0022-5347/03/1705-1756/0
THE JOURNAL OF UROLOGY®
Copyright © 2003 by AMERICAN UROLOGICAL ASSOCIATION
Vol. 170, 1756 –1760, November 2003
Printed in U.S.A.
DOI: 10.1097/01.ju.0000092696.20128.9b
BLADDER CANCER FACTS: ACCURACY OF INFORMATION ON THE
INTERNET
CHERYL T. LEE,* CLAYTON A. SMITH, JANETTE M. HALL, W. BEDFORD WATERS
AND J. SYBIL BIERMANN
From the Departments of Urology (CTL) and Orthopedic Surgery (CAS, JMH, JSB), University of Michigan, Ann Arbor, Michigan, and
Division of Urology (WBW), University of Tennessee, Knoxville, Tennessee
ABSTRACT
Purpose: Patients continually seek information regarding the etiology, diagnosis, management
and treatment of bladder cancer. The Internet has a growing number of health web sites and it
is a tremendous resource for medical information. We examined the accuracy and completeness
of bladder cancer sites on the World Wide Web.
Materials and Methods: Internet searching was performed by retrieving the first 30 universal
resource locators from 8 popular search engines using the search term bladder cancer. A total of
38 independent web sites were evaluated. Other sites were inaccessible, duplicates or only
contained linked pages. Two reviewers evaluated the accuracy and completeness of information
using a predetermined 41-point checklist rating instrument that evaluated essential information related to bladder cancer. The ␬ statistic was used to evaluate interrater variability.
Results: The mean ␬ statistic for evaluable variables was 0.70. Most nonevaluable variables
had excellent agreement, indicating good overall interrater reliability. No rating factor was
present on 100% of sites. Eight factors were present and accurate on 80% to 90% of web pages and
related to signs/symptoms, risk factors, diagnostic tests and treatment of early stage disease. Six
factors were inaccurate on 32% of sites, including incidence (7), staging (3), recurrence (1), and
treatment of early invasive (1) and metastatic (1) disease.
Conclusions: Bladder cancer information retrieved from the majority of medical web sites was
incomplete. However, general information relating to presentation, diagnosis, staging and treatment
of low stage disease was present and accurate on most sites. Although inaccurate information was
detected on 32% of the sites, it tended to be related to outdated information. It is reasonable to refer
patients to select comprehensive web sites to obtain pertinent information about bladder cancer.
KEY WORDS: bladder, bladder neoplasms, Internet
Bladder cancer continues to be a significant health problem with an estimated 57,400 new cases and 12,500 deaths in
2003.1 Few lay persons are aware of basic facts related to the
disease. Traditionally access to health information was provided through television, books and newspaper features. Increasingly patients and their families seek information on
own health conditions and actively participate in treatment
decisions. The Internet has become a popular resource because of the relative ease of researching multiple topics simultaneously. Its availability conveniently extends to the
patient home, or local hospital, library or workplace.
Although a growing number of available medical sites on
the World Wide Web provide information for lay persons,
there are no mandatory guidelines for the presentation and
content of the information provided. Sites are not required to
update information and they may potentially relay inaccurate or outdated facts to patients.2 Consequently patients
may confront physicians with misguided complementary and
nonstandard treatment practices advertised in the media
and on the Internet. To counsel effectively patients on the
risks and benefits of these practices and refer them to reliable sources of health education it is necessary to assess
objectively the information widely accessible to them.3, 4
To our knowledge no peer reviewed studies have rigorously
evaluated sites related to bladder cancer information. We
examined the accuracy and completeness of bladder cancer
web sites and identified those containing reliable information
regarding the etiology, risk, presentation, signs/symptoms,
treatments and prognosis of bladder cancer.
METHODS
In March 2002 Digital Media Audience Reach ratings were
obtained from Jupiter Media Metrix to determine the 8 most
frequently used Internet search engines and their universal
resource locators (URLs).5 In order of use they were Microsoft
Network, Yahoo!, Google, America Online, Ask Jeeves,
LookSmart, InfoSpace and Netscape. The term bladder cancer
was queried from each search engine in May 2002. The first 30
URLs were retrieved from each engine, resulting in a list of 240
resource locators. Each URL and its respective web page were
examined for duplication and pertinence to the study using a
method modified from Biermann et al.2 The whole corresponding web site was then evaluated. An evaluable web site was
accessible and attempted to disseminate medical information
pertaining to bladder cancer. Inaccessible URLs were excluded
from study, as were link pages, nonEnglish sites and sites only
peripherally related to bladder cancer. Peripheral sites mentioned bladder cancer in an unrelated context. Duplicated sites,
defined as a web site already returned from any preceding
search engine, were only counted once.
A checklist rating system was developed based on recommendations from recent bladder cancer consensus panels,
practice guidelines from the National Comprehensive Cancer
Network (http://www.nccn.org/physician_gls/index.html) and
Accepted for publication June 20, 2003.
* Corresponding author: Department of Urology, University of
Michigan, 1500 East Medical Center Dr., Box 0330, Ann Arbor,
Michigan 48109-0330 (telephone: 734-615-6662; FAX: 734-936-9127;
e-mail: ctlee@umich.edu).
1756
1757
BLADDER CANCER FACTS
expert review of standard practice patterns.6 – 8 Two academic urological oncologists (CTL and WBW) used this
checklist to assess and rate Internet retrieved information
from each evaluable web site. The list included general information related to bladder cancer, risk factors, screening,
diagnosis, staging, treatment, prevention and other factors.
Checklist factors within each site were determined to be
present if there was any mention, absent if there was no
mention or present but inaccurate if the factor was associated with incorrect or outdated information.
Prior to rating all of the information obtained the validity
and reliability of the rating system were determined using a
sample of 13 random sites, which were reviewed by the 2
raters. The level of agreement between evaluator ratings was
tabulated using Cohen’s ␬ statistic. Due to good to excellent
agreement the remaining web sites were divided between the
evaluators and reviewed.
RESULTS
An analysis of 240 retrieved URLs revealed 38 unique and
evaluable Web sites (16%). The majority of generated sites
were duplicated (72 or 30%) or peripheral (75 or 31%). Linked
pages or inaccessible links were detected at URL review in 28
(12%) and 27 (11%) cases, respectively. The corresponding
URLs for all evaluable sites are listed in the Appendix.
The checklist rating instrument identified 41 factors used to
assess the completeness of each web site (table 1). For each
factor evaluated by the 2 reviewers ␬ analysis was performed.
The mean ␬ statistic for evaluable variables in the initial review
of 13 samples was 0.70, indicating good overall interrater reliability. It included 9 factors with a ␬ of 1.00, denoting perfect
interrater reliability, 8 with a ␬ of between 0.74 and 0.85,
denoting good to excellent interrater reliability and 6 with a ␬ of
between 0.65 and 0.73, denoting good interrater reliability. For
the 12 factors with a ␬ of less than 0.65 the reviewers agreed in
evaluating that factor in 11 of 13 sites (3 factors), 10 of 13 sites
(5) and 9 of 13 sites (4). A ␬ statistic could not be generated for
6 factors for which reviewer agreement was observed in all
except 1 site (1 factor), 2 sites (2) and 4 sites (3).
Table 1 lists the number of sites that included a given
checklist factor for the whole cohort of 38. No factors were
present or absent on 100% of sites. Eight factors were present
and accurate on 80% to 90% of web sites, while there were 5
on 70% to 79%, 5 on 60% to 69%, 7 on 50% to 59% and 16 on
less than 50%. The top 8 factors present and accurate on sites
involved signs/symptoms, tobacco and chemical exposures as
TABLE 1. Bladder cancer checklist rating system and results
Checklist Factor
General information:
Definition
Histology
Incidence
Grade
Staging system
Signs/symptoms
Etiology/risk factors:
Tobacco
Chemical exposure
Chemotherapy
Radiation therapy
Chronic infection
Chronic indwelling catheter
Prevention (smoking cessation)
Screening:
Urinalysis
Urine cytology
Evaluation/diagnosis:
Urine cytology
Excretory urography/ultrasound ⫹ retrograde pyelography
Cystoscopy
TURBT
Staging:
TURBT
Computerized imaging (computerized tomography/magnetic resonance imaging)
Chest x-ray
Examination under anesthesia
Bone scan (some cases)
Treatment:
Superficial (Ta/CIS)
Early invasive (T1)
Muscle invasive (T2)
Extravesical (T3/T4)
Metastatic (N or M ⫹)
Urinary diversion:
Incontinent stoma
Continent stoma
Orthotopic neobladder
Prognosis (survival rate)
Surveillance:
Cytology
Cystoscopy
Upper tract imaging/computerized tomography
Recurrence:
Rates
Signs/symptoms
Research:
Clinical trial
Links
Support groups
No. Sites (%)
Present
Absent
29 (76)
18 (47)
23 (61)
16 (42)
20 (53)
33 (87)
9 (24)
20 (53)
8 (21)
22 (58)
16 (39)
5 (13)
32 (84)
31 (82)
8 (21)
7 (18)
14 (37)
3 (8)
22 (58)
5 (13)
7 (18)
30 (79)
31 (82)
24 (63)
35 (92)
16 (42)
21 (55)
18 (47)
17 (45)
20 (53)
26 (68)
31 (82)
31 (82)
31 (82)
12 (32)
7 (18)
7 (18)
7 (18)
25 (66)
28 (73)
18 (47)
17 (45)
16 (42)
13 (34)
10 (27)
20 (53)
21 (55)
22 (58)
32 (84)
31 (82)
30 (79)
28 (74)
27 (71)
6 (16)
7 (18)
7 (18)
10 (26)
10 (26)
26 (68)
22 (58)
25 (66)
21 (55)
12 (32)
16 (42)
13 (34)
17 (45)
12 (32)
18 (47)
7 (18)
26 (68)
20 (53)
31 (82)
18 (47)
6 (16)
19 (50)
32 (84)
21 (55)
20 (53)
14 (37)
17 (45)
18 (47)
24 (63)
Present ⫹ Inaccurate
7 (18)
3 (8)
1 (3)
1 (3)
1 (3)
1 (3)
1758
BLADDER CANCER FACTS
risk factors, the diagnostic role of cystoscopy/excretory urography and transurethral bladder tumor resection, (TURBT),
and treatment for superficial and early invasive disease. The
5 factors that were most absent related to risk factors (chemotherapy, radiation therapy and chronic catheterization),
surveillance using upper tract imaging or computerized tomography, and signs and symptoms of tumor recurrence.
Table 2 lists sites with the highest number of accurate
checklist factors. Only 1 site (3%) had more than 90% of
factors present and accurate, while 4 (11%) had at least 85%
and 10 (26%) included 73%. A total of 12 sites (32%) incorporated fewer than 50% of the suggested factors, 4 (11%) had
16 to 20, 3 (8%) had 10 to 15 and 5 (13%) had fewer than 10.
Six factors were present but inaccurate on 12 sites (32%).
These factors related to incidence (7 sites), staging (3), risk
factors/tobacco (1), recurrence (1), and treatment of muscle
invasive (1) and metastatic (1) disease (table 3).
DISCUSSION
The management of bladder cancer is complex. Signs and
symptoms of the disease are commonly ignored or observed
until disease progression occurs, often delaying diagnosis.
Treatment, surveillance and prognosis vary substantially depending on disease stage. Interventions range from office fulguration to radical surgery to systemic therapy. Consequently
disease management can be difficult for patients to understand.
Fortunately patients now have medical information at their
immediate disposal day and night via the Internet. More than
149 million Americans used the Internet in 2001 and it is
predicted that this number may increase to 193 million in 2004
and 236 million in 2007.9 A significant number of patients are
using this resource to educate themselves and their families on
the disease state, choose physicians and orchestrate treatment
plans.4 The process of self-education is applauded. However, the
lack of peer reviewed material on the Internet makes it difficult
to assure patients that they are gaining accurate information.
Therefore, it is incumbent on physicians to be aware of the
material to which patients are exposed.
To provide guidance to patients and facilitate the search
for quality resources physicians must scrutinize web sources.
To our knowledge there is currently no such evaluation of
bladder cancer web sites. To provide an objective assessment
of these sites a checklist rating instrument was developed
that summarizes basic aspects of standard bladder cancer
management and general disease information that would be
relevant to a wide spectrum of patients with bladder cancer.
Although evaluable sites in the current study were rated
using this checklist, it is clear that other aspects of bladder
cancer management could be used to assess these sites. The
rating instrument did not assess the quantity of site information provided, but rather the accuracy. Moreover, the instrument was applied only to the content on a given site and
not to any externally linked pages because each site was
judged according to its independent information provided for
readers. Since the current study objective was to determine
the accuracy and completeness of these sites, the checklist
rating instrument, which reviewed 41 core factors that any
comprehensive site should include, seemed reasonable.
Of the 41 factors present on the majority of sites 8 represented basic information that patients seek. Patients are typically concerned with bladder cancer etiology, tumor prevention,
cancer extent, available treatments, prognosis and recurrence.
Interestingly only 18% of sites described the signs and symp-
toms of recurrent cancer. Perhaps the 87% of sites that detailed
signs and symptoms of initial disease diagnosis considered that
this description was also adequate for recurrence. Although
84% of sites mentioned tobacco as a major risk factor for bladder
cancer, only 58% described smoking cessation as a method of
disease prevention. Despite the need for prolonged followup
evaluations in this disease fewer than half of the sites discussed
surveillance. Although tumor stage and grade are important
variables determining patient outcome, only 53% of sites described the staging system accurately and only 42% defined
tumor grade.10, 11
The discussion of treatment options was not always complete. Although 74% to 84% of sites reviewed options for
disease treatment that might include cystectomy, only 58%
to 68% then went on to list options for urinary diversion.
Despite the fact that patients are often most concerned about
survival after treatment, only 55% of sites provided survival
rates. Support group networks can be invaluable for patients
during and after treatment but they were discussed infrequently in 37% of sites. The Internet may also be an excellent
tool to recruit patients for clinical research studies but little
more than half of the sites included such information.
Internet searches can be time-consuming and exhausting,
particularly for the aging population that acquires bladder cancer.3 A large number of duplicated and inaccessible sites were
found using the most popular search engines. Peripheral websites were another source of frustration since they casually mention a topic or use it to lure in unsuspecting patients for sales
promotions or unsubstantiated treatments posed to any patient
with cancer. Ultimately after retrieving 240 URLs and, thus, 240
potential sites only 17% were unique, accessible and evaluable.
When one finally has a list of sites, it can be confusing as to
which is the best. Table 2 lists the most comprehensive sites.
These sites were created at or borrowed from the National
Cancer Institute, or they were part of large Internet cancer
sites. Other sites that were produced purely by industry were
often directed at individual products and provided specific disease management information vs broad, overall disease related
material. Directed sites were generally brief, whereas more
comprehensive sites were lengthy, possibly overwhelming patients. These sites combat this problem by segmenting the site
into numerous topic headings that allow the patient to search
for specific topics and avoid others that are not pertinent to
them, making the site manageable and easily navigable.
Inaccurate information in this study generally reflected outdated information. Although few blatantly false statements
were noted, an inaccuracy rate of 32% of sites is disturbing.
Internet studies evaluating other cancers, including melanoma
and sarcoma, indicated an inaccuracy rate of 14% and 6%,
respectively.2, 12 Clearly the Internet is a fluid medium that
changes rapidly. Periodic assessment and maintenance may be
cumbersome but are necessary to maintain updated and accurate information. The highest number of inaccurate factors was
seen in the incidence category with 18% of sites presenting
outdated (prior to 2001) cancer statistics. Of the sites 8% provided inaccurate staging information or used data prior to the
1997 American Joint Committee on Cancer staging system.
It has been suggested that physicians can improve Internet
based medical sites by contributing patient material or converting accepted disease guidelines into lay terminology.12 These
efforts would add substance to our web sites that may in fact
help shorten clinic visits or answer frequently asked questions.
TABLE 2. Sites containing the highest number of checklist factors
Site No.
URL
No. Accurate Factors (%)
8
20
32
33
http://www.cancer.gov/cancer_information/cancer_type/bladder/
http://www.nci.nih.gov/cancer_information/cancer_type/bladder/
http://www.1uphealth.com/medical/disease/cancer-disease/bladder-cancer-1.html
http://www.cancerlinksusa.com/bladder/home.html
39 (95)
36 (88)
36 (88)
36 (88)
1759
BLADDER CANCER FACTS
TABLE 3. Inaccurate site information
Site No.
Checklist Factor
Inaccuracy
2, 15–18, 20, 27
2, 21, 24
15
3
4
13
General information (incidence)
General information (staging system)
Etiology/risk factors (tobacco)
Treatment (muscle invasive T2)
Treatment (metastatic N or M⫹)
Recurrence (rates)
Old or outdated data on incidence rates
1996 American Joint Committee on Ca staging, T3a is organ confined disease
Underestimates tobacco use risk
TURBT provides adequate local control
“No established chemotherapy exists for bladder cancer”
Inaccurate values
Physicians must also periodically assess web site information.
When considering web sites to be offered to patients, it is probably best to have a select number of available URLs. Cancer
organizations with significant resources provide patients with
an opportunity to obtain cutting edge information from leaders
in the field and we should take advantage of these efforts.
CONCLUSIONS
Bladder cancer information retrieved from most medical web
sites is incomplete. However, general information related to
presentation, diagnosis, staging and treatment of low stage
disease is present on most sites. Although inaccurate information was detected on 32% of sites, outdated data is often the
problem rather than outwardly false statements. This finding
highlights the necessity of regular review and periodic updates
of any web site. Nevertheless, it is reasonable to refer patients
to select comprehensive web sites to obtain pertinent information about bladder cancer. This active referral can minimize
random and exhaustive Internet query of inadequate sites and
provide effective patient education.
APPENDIX: EVALUABLE WEB SITES
Engine (web site No.)
Microsoft Network:
1
2
3
4
5
6
7
8
9
10
11
12
13
Yahoo!:
14
15
16
17
18
19
Google:
20
21
22
23
24
America Online:
25
26
Ask Jeeves:
27
28
29
30
31
LookSmart:
32
33
34
InfoSeek:
35
36
37
Netscape:
38
URL
http://www.yourcancerrisk.harvard.edu/hccpquiz.pl?func⫽start&quiz⫽bladder
http://www.cancernews.com/bladder.htm
http://www.urologyinstitute.com/html/bladder_cancer.html
http://www.matritech.com/bladdercancer.html
http://www.mayoclinic.com/findinformation/diseasesandconditions/invoke.cfm?id⫽DS00177&
http://www.crha-health.ab.ca/hlthconn/items/ca-blad.htm
http://www.med.wayne.edu/urology/DISEASES/bladdercancer.html
http://www.cancer.gov/cancer_information/cancer_type/bladder/
http://www.findarticles.com/cf_dls/m0999/1998_Nov_14/53340585/p1/article.jhtml
http://blcwebcafe.org/default.asp
http://www.findarticles.com/cf_dls/m3225/12_61/62829334/p1/article.jhtml
http://www.findarticles.com/cf_dls/g2603/0002/2603000210/p1/article.jhtml
http://www.findarticles.com/cf_dls/g2601/0002/2601000204/p1/article.jhtml
http://www.oncolink.com/templates/types/section.cfm?c⫽21&s⫽66
http://www.btastat.com/
http://www.holisticurology.com/
http://www.urologychannel.com/bladdercancer/index.shtml
http://www.thecancer.info/bladder/
http://www.ricancercouncil.org/facts/bladfacts.htm
http://www.nci.nih.gov/cancer_information/cancer_type/bladder/
http://www.cancerhelp.org.uk/help/default.asp?page⫽2680
http://www.cancerbacup.org.uk/info/bladder.htm
http://www.meb.uni-bonn.de/cancernet/101206.html
http://www.associatedurologists.com/bladder.html
http://www.focusonurology.com/script/main/forum.asp?articlekey⫽292&rd⫽1
http://www.cnn.com/HEALTH/9905/05/bladder.cancer/
http://telescan.nki.nl/bladder2.html
http://www.canceradvice.co.uk/
http://www.healthatoz.com/
http://www.royalmarsden.org/patientinfo/booklets/cancer_bladder/index.asp
http://www.afud.org/conditions/bc.html
http://www.1uphealth.com/medical/disease/cancer-disease/bladder-cancer-1.html
http://www.cancerlinksusa.com/bladder/home.html
http://www.bladder-disorders.com/
http://www.umm.edu/cancer/index.html?source⫽goto
http://real-essiac.com/a.bladder_cancer/k.bladder_cancer
http://www.ccspublishing.com/journals2/Bladder_Cancer.htm
http://my.webmd.com/condition_center_hub/can
1760
BLADDER CANCER FACTS
REFERENCES
1. Jemal, A., Murray, T., Samuels, A., Ghafoor, A., Ward, E. and
Thun, M. J.: Cancer statistics, 2003. CA Cancer J Clin, 53: 5,
2003
2. Biermann, J. S., Golladay, G. J., Greenfield, M. L. and Baker,
L. H.: Evaluation of cancer information on the Internet. Cancer, 86: 381, 1999
3. Sacchetti, P., Zvara, P. and Plante, M. K.: The Internet and
patient education—resources and their reliability: focus on a
select urologic topic. Urology, 53: 1117, 1999
4. Hellawell, G. O., Turner, K. J., Le Monnier, K. J. and Brewster,
S. F.: Urology and the Internet: an evaluation of internet use
by urology patients and of information available on urological
topics. BJU Int, 86: 191, 2000
5. Sullivan, D.: Jupiter Media Metrix Search Engine Ratings.
Available at http://www.searchenginewatch.com/reports/mediametrix.html. Accessed March 2002
6. Fleshner, N. E., Herr, H. W., Stewart, A. K., Murphy, G. P.,
Mettlin, C. and Menck, H. R.: The National Cancer Data Base
report on bladder carcinoma. The American College of Surgeons
Commission on Cancer and The American Cancer Society.
Cancer, 78: 1505, 1996
7. Smith, J. A., Jr., Labasky, R. F., Cockett, A. T. K., Fracchia, J. A.,
8.
9.
10.
11.
12.
Montie, J. E. and Rowland, R. G.: Bladder Cancer Clinical
Guidelines Panel summary report on the management of nonmuscle invasive bladder cancer (stages Ta, T1 and Tis). J Urol,
162: 1697, 1999
The NCCN Bladder Cancer Guideline, The Complete Library of
Practice Guidelines in Oncology. Rockledge, Pennsylvania:
National Comprehensive Cancer Network, version 1, 2002.
Available at http://www.nccn.org/physician_gls/index.html,
Accessed August 2002.
Computer Industry Almanac, Inc. Available at http://www.c-ia.com. Accessed June 2002
Stein, J. P., Lieskovsky, G., Cote, R., Groshen, S., Feng, A. C. and
Boyd, S. et al: Radical cystectomy in the treatment of invasive
bladder cancer: long-term results in 1,054 patients. J Clin
Oncol, 19: 666, 2001
Millán-Rodrı́guez, F., Chéchile-Toniolo, G., Salvador-Bayarri, J.,
Palou, J. and Vicente-Rodriguez, J.: Multivariate analysis of
the prognostic factors of primary superficial bladder cancer.
J Urol, 163: 73, 2000
Bichakjian, C. K., Schwartz, J. L., Wang, T. S., Hall, J. M.,
Johnson, T. M. and Biermann, J. S.: Melanoma information on
the Internet: often incomplete—a public health opportunity?
J Clin Oncol, 20: 134, 2002