Psychometrician- Assessment EngineerinG
About This Position
As Psychometrician- Assessment Engineering, you’ll focus on principled assessment design, and work with our team to maintain the psychometric integrity of ARRT’s certification programs. You’ll also:
- Systematically evaluate modern testing methodologies and innovative types of items.
- Help increase understanding through psychometric research.
- Disseminate information by publishing in professional journals and presenting at professional conferences.
- Collaborate with ARRT’s Board of Trustees, ARRT’s examination committees, Ph.D. programs in educational measurement or psychometrics, and other organizations.
Key Duties and Responsibilities
- Coordinate the cognitive modeling of information, and then use those models to modify content specifications and construct exam item templates.
- Update and enhance item writing workshop materials to include cognitive modeling/assessment engineering principles.
- Design and evaluate innovative types of items that meet the cognitive modeling parameters ARRT examinees demonstrate.
- Apply cognitive modeling and/or assessment engineering techniques that could enhance the quality of certification exam scores and assessment results.
- Use data mining, cognitive modeling, and language processing tools to model the difficulty of items, helping to predict item difficulty before we pilot new items.
Cognitive Modeling Leadership Support
Provide leadership and technical support for, consult with staff on, and manage initiatives possibly including, but not limited to:
- Using cognitive modeling and/or assessment engineering techniques create forms of knowledge assessment appropriate for alternate forms of recognition.
- Helping ARRT staff investigate issues related to maintaining certification and registration.
- Consulting with staff on other psychometric issues (e.g., standards setting, job analysis, survey development).
- Analyzing data and drafting reports for the Board of Trustees and/or senior staff.
- Conducting applied psychometric research in support of ARRT’s mission.
- Participate in activities that promote ARRT’s goodwill and commitment to the professional community.
- Present papers and conduct workshops at conferences related to psychometrics and/or medical imaging.
- Publish research-based articles.
- Attend occasional state or interorganizational meetings.
Exam Content and Eligibility Materials
- Oversee periodic practice analysis projects for assigned programs, including selecting committee members, developing surveys, analyzing data, and writing reports.
- Determine psychometric quality assurance criteria for reviewing item statistics, exam statistics, and examinee comments. Ensure the criteria are met.
- Review exams and comment on items before examination committee meetings in accordance with psychometric quality assurance criteria.
Practice Analysis Committees
- Where possible, develop procedures for standard presentations of practice analysis/job analysis survey data. Add documentation of those processes to the Exam Development Manual
Examination Scoring, Analysis, and Reporting
- Determine psychometric quality assurance criteria for exam scoring, analysis, and reporting. Ensure the criteria are met.
- Periodically review item statistics, exam results, and examinee comments to ensure accuracy of keys, passing scores, and exam content.
- Serve as back-up for exam results processing, scoring, and quality control for all programs.
- Maintain procedures manual and other documentation required for processing and scoring exams.
- Analyze exam results. Collaborate with department staff to produce and disseminate annual examination summary reports and technical reports.
- Respond to letters and calls regarding score reporting.
- Recommend enhancements to scoring procedures and reports.
Item Calibration and Test Form Scaling
- Determine psychometric quality assurance criteria for item calibration and test form scaling, and ensure the criteria are met.
- Prepare and/or follow schedules for item calibration (initial, pilot, and recalibrations) for all exam programs.
- Calibrate items as needed for selected exam programs.
- Calculate cut scores scaling equations for exam forms; monitor pass rates, cut scores, and other exam statistics.
- Periodically conduct studies to identify models and procedures for calibration and scaling; perform analyses as needed to troubleshoot calibration and scaling methods.
- Periodically meet with other staff to review and evaluate calibration and scaling procedures.
Psychometric Exam Security
- Design exam security program in concert with psychometric research findings.
- Set criteria for flagging suspected security breaches.
- Design procedure to systematically and psychometrically assess security risks.
- Alert Director of Psychometric Services to suspected security breaches.
- With other staff members, monitor outcomes from exam security program. Investigate any potential security breaches.
Essay Item and Case Study Essays
- Determine psychometric quality assurance criteria for developing and scoring case study essay questions; ensure the criteria are met.
- Systematize the development of case studies by preparing such things as instructional materials for authors and reviewers.
- Establish mechanisms for scoring essays based on the use of multiple graders. Include quality control procedures and policies for rescoring essay exams. Investigate the feasibility of automated (computerized) scoring.
- Establish methods for training and monitoring the performance of essay graders.
- Evaluate and adopt methods for setting performance standards, scoring, and scaling responses to essay questions, and for determining the reliability of essay scores and total scaled scores.
- Evaluate and adopt statistical models for monitoring the psychometric properties of graders and cases; develop specifications for incorporating statistical information into the item bank or other database.
- Determine long-term needs of the item bank and work with IT staff on ways to achieve and prioritize those goals.
- Review and edit ARRT publications—including website material—as requested.
- Assist with or prepare memos and reports for the Board of Trustees.
- Conduct standard-setting studies as needed; analyze data and prepare reports. Train new staff members to conduct standard-setting studies. Oversee and monitor any standard-setting work delegated to others once they are sufficiently prepared to undertake such tasks.
- Perform other duties as requested.
- Doctorate degree in Psychometrics, Quantitative Methods in Psychology, Educational Measurement, Educational Psychology, Industrial and Organizational Psychology, or a related field
- Experience in assessment engineering
- Five years of experience, including education, in applied and research Psychometrics
The Twin Cities (Minneapolis and St. Paul) is a vibrant and lively metropolitan area. It consistently appears on national lists of best places to live in the U.S., and we couldn't agree more. We're home to six professional sports teams, an impressive arts and theater scene, and numerous restaurants with James Beard-winning and -nominated chefs. We also have an abundance of lakes, trails, and outdoor activities, as well as dozens of higher education institutions within the metro area.