Clean and filter the bibliographic data

Database records are messy. The same concept appears as “AI”, “artificial intelligence” and “artificial intelligence (AI)”; your own search terms appear in every record and dominate every keyword map; one author appears under two spellings while two different authors share the same initials. Cleaning takes time, but it is what separates an insightful map from an artefact of the data.

Step 1: check completeness

The Data completeness page shows, for each field, the share of records where it is missing. Use it to decide which analyses are trustworthy. As a rule of thumb, a field missing in less than 10% of records is fine, 10% to 30% is acceptable with a caveat, and above 30% the analysis based on it should be dropped or clearly flagged. Typical gaps are author keywords in older papers, affiliations in some document types and cited references in non-Core collections.

Step 2: apply the inclusion criteria

The Filter documents page restricts the collection by publication year, document type, language, source and topic terms. Apply the criteria you decided in Lesson 2, for example articles and reviews only, from 2008 to the last complete year. In the demo data, the 418 documents include 315 articles, 63 reviews and 40 conference papers; a strict article-and-review filter would keep 378. Every filter is recorded and counted in the PRISMA diagram. Avoid filtering on the current year: it is incomplete and makes the last point of the annual production curve drop artificially.

Step 3: merge keyword synonyms

Keyword analyses count terms literally, so variants split the counts. On the Keyword cleaning page, define synonym groups, each mapped to one preferred term:

• abbreviations and full forms: “AI” = “artificial intelligence”; “EHR” = “electronic health records”;
• singular and plural, and spelling variants: “wearable device” = “wearable devices”; “behavior change” = “behaviour change”;
• hyphenation and word order: “m-health” = “mhealth” = “mobile health”.

Only merge true synonyms. “Machine learning” and “deep learning” are related but distinct concepts; merging them hides the rise of deep learning that the trend analysis would otherwise show.

Step 4: remove uninformative terms

Your search terms appear in nearly every record by construction, so they sit at the centre of every keyword map and link to everything. Remove them (or the most dominant ones) from keyword analyses, or at least mention their effect. Also remove generic index terms such as “human”, “article”, “female”, “priority journal” and “controlled study”, which databases add automatically. A stop-word list is applied to title and abstract words; extend it with terms that are generic in your field.

Step 5: author names and affiliations

Author names are standardised as surname plus initials (HANSEN W). This means that two authors with the same surname and initials are merged, and one author who sometimes uses a middle initial is split. Check the top 20 authors by hand: open their documents and confirm that they belong to one person. Scopus Author IDs and ORCID identifiers, where exported, help. Affiliations vary even more (“Univ. of X”, “University X, Department of Y”); country-level analyses are much more reliable than institution-level ones.

Step 6: document the cleaning

Report what you cleaned and how: the filters, the number of synonym groups and examples, the removed terms, and how author names were checked. Save the synonym list as part of your project; reviewers sometimes ask for it, and you will want it for an update of the analysis.

The data table

The Data table page lets you browse and search the records. Use it to spot obvious problems: errata, retractions, editorials that slipped through the document type filter, or off-topic records retrieved by an ambiguous term. Remove them with a reason, and count them in PRISMA.