Exam Design Principles

• Onsite interview processes are broken. They don’t cover enough, and yet can be hours long. The bulk of the assessment needs to be done offline, well before the interview process begins.
• There are many factors that make a software engineer great. The CSPA must be able to assess candidates on many dimensions, both objectively and subjectively.
• We should cover as many topics as possible, knowing fully that candidates will inevitably be strong in certain areas and weak in others. This gives candidates a chance to prove their expertise, and mitigates risk of hitting a knowledge blind spot. Furthermore, we should reward the candidate’s strengths and de-emphasize weaknesses.
• Programming languages change. Libraries and frameworks change. We should focus on underlying concepts and principles rather than implementations and programming languages. Deeply understanding core fundamentals lead to more adaptable candidates.
• However, practical application of that knowledge is equally important. We should design programming projects from actual real-life problems engineers face.
• To maximize utility to employers, the exam conditions should mimic the work environment as closely as possible. For example, using their own machine (not a whiteboard), the use of familiar IDEs and other software, access to Google, and time constraints can all affect the candidate’s performance.
• Programming languages change. Libraries and frameworks change. Technology changes (very fast!). We must constantly evolve the test to cover the topics and technologies employers care about the most.