OBJECTIVE: Inadequate information transfer during transitions in healthcare is a major patient safety issue. Aim of this study was to pilot a review of medical records to identify transitional safety incidents (TSIs) for use in a large intervention study and assess its reliability and validity. DESIGN: A retrospective medical record review study. SETTINGS AND PARTICIPANTS: Combined primary and secondary care medical records of 301 patients who had visited their general practitioner and the University Medical Center Utrecht, the Netherlands, in 2013 were randomly selected. Six trained reviewers assessed these medical records for presence of TSIs. OUTCOMES: To assess inter-rater reliability, 10% of medical records were independently reviewed twice. To assess validity, the identified TSIs were compared with a reference standard of three objectively identifiable TSIs. RESULTS: The reviewers identified TSIs in 52 (17.3%) of all transitional medical records. Variation between reviewers was high (range: 3-28 per 50 medical records). Positive agreement for finding a TSI between reviewers was 0%, negative agreement 80% and the Cohen's kappa -0.15. The reviewers identified 43 (22%) of 194 objectively identifiable TSIs. CONCLUSION: The reliability of our measurement tool for identifying TSIs in transitional medical record performed by clinicians was low. Although the TSIs that were identified by clinicians were valid, they missed 80% of them. Restructuring the record review procedure is necessary.