Good quality CAPI research relies on trustworthy interviews. You need to be sure fieldwork integrity isn’t compromised by interviewers taking shortcuts, submitting false responses. Nfield CAPI now builds on standard verification checks with an Audit Trail / Logging feature that highlights suspicious input patterns. By analyzing statistical details, this enables you to identify when interviewers are likely to be cheating the system, and take appropriate action.
Building on the basics
Traditional quality control components such as field validation, logic checking, ordering and randomization are, of course, all supported in Nfield. For face-to-face CAPI interviews, additional features such as GPS locations, photographs and audio recordings can also be activated to provide evidence of interviews taking place. (More information about these can be found at Nfield’s quality control options for rock-solid CAPI data.)
All this gets you off to a great start, but still leaves room for cheating. To further close off the possibility of falsified responses, you need to look more closely at response input patterns. Nfield CAPI’s Audit Trail function enables you to spot the tell-tale signs of dishonest interviewer behavior, such as:
Within an interview:
- Short overall interview time1
- Long answer time for a simple question1
- Short answer time for a difficult question1
- Interview happened in an incorrect location2
Within an interviewer’s individual behavior:
- Gap between two interviews is shorter than the expected travel time between locations2
- Completing an overly high number of interviews in a day for the interviewer’s known ability2
Within an interviewer’s comparable performance:
- Much shorter interview times than average1
- Higher-than-average percentage of inconsistency between GPS locations and respondent addresses2
- Higher-than-average percentage of unentered respondent tracing information (failure to enter phone number, personal details, etc.)3
- Lower-than-average number of visits2
- Higher-than-average number of interviews/visits per day2
- Significantly higher or lower numbers for project-specific tasks. E.g. collecting samples of hair for laboratory analysis as part of a haircare survey.
The data for annotated items 1, 2 and 3 can be found in the Nfield downloads:
- 1 the new audit log file ending with “-auditlog.csv”
- 2 paradata csv file ending with “.csv”
- 3 survey result data file ending with “.dat”
Comparing performance via Audit Trail
Nfield CAPI’s new Audit Trail feature records how long an interviewer stayed on a page and which button they clicked to move on. The results are provided in a csv file. The example below shows how this might look for a single interview.
You can compare each result line against your reasonable expectations and set benchmarks to flag up interviews that fall outside these parameters.
The best method for reviewing this is to build a report (in Microsoft PowerBI or other reporting system) to show suspicious interviews. Incorporating a drill-down function will give you an easy way to view more information.
The two illustrations below show how your reports might look.
Here, the results are sorted to show interview durations in descending order.
Intelligence at your fingertips
Armed with these insights, you can better fortify the quality of your CAPI surveys by eliminating falsified responses. Not only can you challenge interviewers whose input appears suspicious, and remove fake responses from your results, you can also use this feature to deter dishonest interviewer behavior from taking place at all.
And in case you are wondering, this Audit Trail feature will also be introduced for Nfield Online in the near future.
