Treffer: External Correlates of Adult Digital Problem-Solving Process: An Empirical Analysis of PIAAC PSTRE Action Sequences

Computerized assessments and interactive simulation tasks are increasingly popular and afford the collection of process data, i.e., an examinee's sequence of actions (e.g., clickstreams, keystrokes) that arises from interactions with each task. Action sequence data contain rich information on the problem-solving process but are in a nonstandard, variable-length discrete sequence format. Two methods that directly extract features from the raw action sequences, namely multidimensional scaling and sequence-to-sequence autoencoders, produce multidimensional numerical features that summarize original sequence information. This study explores the utility of action sequence features in understanding how problem-solving behavior relates to cognitive proficiencies and demographic characteristics. This is empirically illustrated with the process data from the 2012 PIAAC PSTRE digital assessment. Regularized regression results showed that action sequence features are more predictive of examinees' demographic and cognitive characteristics compared to final outcomes. Partial least squares analysis further aided the identification of behavioral patterns systematically associated with demographic/cognitive characteristics.

As Provided

aid-asi1&gt;3.0.co;2-9</p> <anchor name="c7"></anchor>

Dillon, G., Boulet, J., Hawkins, R., & Swanson, D. (2004). Simulations in the United States Medical Licensing Examination™ (USMLE™). BMJ Quality & Safety, 13(Suppl 1), i41–i45. 10.1136/qshc.2004.010025

<anchor name="c8"></anchor>

Eichmann, B., Greiff, S., Naumann, J., Brandhuber, L., & Goldhammer, F. (2020). Exploring behavioural patterns during complex problem-solving. Computer Assisted Learning, 36(6), 933–956. 10.1111/jcal.12451

<anchor name="c9"></anchor>

Ercikan, K., Guo, H., & He, Q. (2020). Use of response process data to inform group comparisons and fairness research. Educational Assessment, 25(3), 179–197. 10.1080/10627197.2020.1804353

<anchor name="c10"></anchor>

Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and Activity, 5(3), 178–186. 10.1207/s15327884mca0503_3

<anchor name="c11"></anchor>

Fang, G., & Ying, Z. (2020). Latent theme dictionary model for finding co-occurrent patterns in process data. Psychometrika, 85(3), 775–811. 10.1007/s11336-020-09725-2

<anchor name="c12"></anchor>

Fleisher, M. S., & Tsacoumis, S. (2012). O* NET analyst occupational skills ratings: Procedures update. National Center for O* NET Development.

<anchor name="c13"></anchor>

Friedman, J., Hastie, T., & Tibshirani, R. (2009). glmnet: Lasso and elastic-net regularized generalized linear models. R package version, 1.4.

<anchor name="c14"></anchor>

Gómez-Alonso, C., & Valls, A. (2008, October). A similarity measure for sequences of categorical data based on the ordering of common elements. In Proceedings of the 5th International Conference Modeling Decisions for Artificial Intelligence, MDAI 2008, Sabadell, Spain, October 30-31, 2008 (pp. 134–145). Springer.

<anchor name="c15"></anchor>

Gao, Y., Cui, Y., Bulut, O., Zhai, X., & Chen, F. (2022). Examining adults' web navigation patterns in multi-layered hypertext environments. Computers in Human Behavior, 129, Article 107142. 10.1016/j.chb.2021.107142

<anchor name="c16"></anchor>

Gao, Y., Zhai, X., Bulut, O., Cui, Y., & Sun, X. (2022). Examining humans' problem-solving styles in technology-rich environments using log file data. Journal of Intelligence, 10(3), Article 8. 10.3390/jintelligence10030038

<anchor name="c17"></anchor>

Greiff, S., Wüstenberg, S., & Avvisati, F. (2015). Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92–105. 10.1016/j.compedu.2015.10.018

<anchor name="c18"></anchor>

Han, Y., Liu. H., & Ji, F. (2022). A sequential response model for analyzing process data on technology-based problem-solving tasks. Multivariate Behavioral Research, 57(6): 960–977. 10.1080/00273171.2021.1932403

<anchor name="c19"></anchor>

Hao, J., & Mislevy, R. J. (2019). Characterizing interactive communications in computer-supported collaborative problem-solving tasks: A conditional transition profile approach. Frontiers in Psychology, 10, Article 1011. 10.3389/fpsyg.2019.01011

<anchor name="c20"></anchor>

He, Q., Borgonovi, F., & Paccagnella, M. (2021). Leveraging process data to assess adults' problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166, Article 104170. 10.1016/j.compedu.2021.104170

<anchor name="c21"></anchor>

He, Q., Borgonovi, F., & Suárez-Álvarez, J. (2023). Clustering sequential navigation patterns inmultiple-sourcereading tasks with dynamic time warping method. Computer Assisted Learning, 39(3), 719–736. 10.1111/jcal.12748

<anchor name="c22"></anchor>

He, Q., Shi, Q., Tighe, E. (2023). Predicting problem-solving proficiency with hierarchical supervised models on response process. Psychological Test and Assessment Modeling, 65(1), 145–178. <a href="https://www.proquest.com/scholarly-journals/predicting-problem-solving-proficiency-with/docview/2799814448/se-2" target="_blank">https://www.proquest.com/scholarly-journals/predicting-problem-solving-proficiency-with/docview/2799814448/se-2</a>

<anchor name="c23"></anchor>

He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with N-grams. In Y.Rosen, S.Ferrara, & M.Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 750–777). IGI Global. 10.4018/978-1-4666-9441-5.ch029

<anchor name="c24"></anchor>

He, Q., von Davier, M., & Han, Z. (2019). Exploring process data in problem-solving items in computer-based large-scale assessments: Case studies in PISA and PIAAC. In H.Jiao, R. W.Lissitz, & A.Van Wie (Eds.), Data analytics and psychometrics: Informing assessment practices (pp. 53–76). Information Age Publishing.

<anchor name="c25"></anchor>

Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6), 417, 441. 10.1037/h0071325

<anchor name="c26"></anchor>

Kirsch, I., & Thorn, W. (2013). Foreword: The programme for international assessment of adult competencies: An overview [PIAAC Technical Report] (pp. 5–24). OECD.

<anchor name="c27"></anchor>

LaMar, M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67–88. 10.1007/s11336-017-9570-0

<anchor name="c28"></anchor>

Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755), 788–791. 10.1038/44565

<anchor name="c29"></anchor>

Liang, K., Tu, D., & Cai, Y. (2023). Using process data to improve classification accuracy of cognitive diagnosis model. Multivariate Behavioral Research, 58(5):969–987. 10.1080/00273171.2022.2157788

<anchor name="c30"></anchor>

Liao, D., He, Q., & Jiao, H. (2019). Mapping Background variables with sequential patterns in problem-solving environments: An investigation of United States adults' employment status in PIAAC. Frontiers in Psychology, 10, Article 646. 10.3389/fpsyg.2019.00646

<anchor name="c31"></anchor>

Liu, H., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, Article 1372. 10.3389/fpsyg.2018.01372

<anchor name="c32"></anchor>

Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29. 10.1002/j.2333-8504.2003.tb01908.x

<anchor name="c33"></anchor>

Nwakasi, C. C., Cummins, P. A., Mehri, N., Zhang, J., & Yamashita, T. (2019). Problem solving in technology-rich environments, adult education and training, and income: An International Comparison Using PIAAC Data. Grantee Submission.

<anchor name="c34"></anchor>

Nye, C. D., Su, R., Rounds, J., & Drasgow, F. (2012). Vocational interests and performance. Perspectives on Psychological Science, 7(4), 384–403. 10.1177/1745691612449021

<anchor name="c35"></anchor>

OECD. (2012). Literacy, numeracy and problem solving in technology-rich environments: Framework for the OECD survey of adult skills. <a href="https://www.oecd.org/skills/piaac/PIAAC%20Framework%202012--%20Revised%2028oct2013_ebook.pdf" target="_blank">https://www.oecd.org/skills/piaac/PIAAC%20Framework%202012--%20Revised%2028oct2013_ebook.pdf</a>

<anchor name="c36"></anchor>

OECD. (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems (Vol. V). <a href="https://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-V.pdf" target="_blank">https://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-V.pdf</a>

<anchor name="c37"></anchor>

OECD. (2016). Technical report of the survey of adult skills (PIAAC) (2nd ed.). <a href="https://www.oecd.org/skills/piaac/PIAAC_Technical_Report_2nd_Edition_Full_Report.pdf" target="_blank">https://www.oecd.org/skills/piaac/PIAAC_Technical_Report_2nd_Edition_Full_Report.pdf</a>

<anchor name="c38"></anchor>

Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9, Article 2231. 10.3389/fpsyg.2018.02231

<anchor name="c39"></anchor>

Qiao, X., Jiao, H., & He, Q. (2023). Multiple-group joint modeling of item responses, response times, and action counts with the Conway-Maxwell-Poisson distribution. Journal of Educational Measurement, 60(2), 255–281. 10.1111/jedm.12349

<anchor name="c40"></anchor>

Schleicher, A. (2008). PIAAC: A new strategy for assessing adult competencies. International Review of Education, 54(5–6), 627–650. 10.1007/s11159-008-9105-0

<anchor name="c41"></anchor>

Steinfeld, E., & Maisel, J. (2012). Universal design: Creating inclusive environments. Wiley & Sons.

<anchor name="c42"></anchor>

Tang, X., Wang, Z., He, Q., Liu, J., & Ying, Z. (2020). Latent feature extraction for process data via multidimensional scaling. Psychometrika, 85(2), 378–397. 10.1007/s11336-020-09708-3

<anchor name="c43"></anchor>

Tang, X., Wang, Z., Liu, J., & Ying, Z. (2021). An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology, 74(1), 1–33. 10.1111/bmsp.12203

<anchor name="c44"></anchor>

Tang, X., Zhang, S., Wang, Z., Liu, J., & Ying, Z. (2021). Procdata: An r package for process data analysis. Psychometrika, 86(4), 1058–1083. 10.1007/s11336-021-09798-7

<anchor name="c45"></anchor>

Tippins, N. T. (2015). Technology and assessment in selection. Annual Review of Organizational Psychology and Organizational Behavior, 2(1), 551–582. 10.1146/annurev-orgpsych-031413-091317

<anchor name="c46"></anchor>

Ulitzsch, E., He, Q., & Pohl, S. (2022). Using sequence mining techniques for understanding incorrect behavioral patterns on interactive tasks. Journal of Educational and Behavioral Statistics, 47(1), 3–35. 10.3102/10769986211010467

<anchor name="c47"></anchor>

Ulitzsch, E., He, Q., Ulitzsch, V., Molter, H., Nichterlein, A., Niedermeier, R., & Pohl, S. (2021). Combining clickstream analyses and graph-modeled data clustering for identifying common response processes. Psychometrika, 86(1), 190–214. 10.1007/s11336-020-09743-0

<anchor name="c48"></anchor>

Ulitzsch, E., Ulitzsch, V., He, Q., & Lüdtke, O. (2023). A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks. Behavior Research Methods, 55(3), 1392–1412.

<anchor name="c49"></anchor>

Wang, Z., Tang, X., Liu, J., & Ying, Z. (2023). Subtask analysis of process data through a predictive model. British Journal of Mathematical and Statistical, 76(1), 211–235. 10.1111/bmsp.12290

<anchor name="c50"></anchor>

Wehrens, R., & Mevik, B. H. (2007). The PLS package: principal component and partial least squares regression in R. Journal of Statistical Software, 18(2), 1–23. 10.18637/jss.v018.i02

<anchor name="c51"></anchor>

Wold, S., Sjöström, M., & Eriksson, L. (2002). Partial least squares projections to latent structures (PLS) in chemistry. In Encyclopedia of computational chemistry (Vol. 3). John Wiley and Sons. 10/02/0470845015.cpa012

<anchor name="c52"></anchor>

Xiao, Y., He, Q., Veldkamp, B., & Liu, H. (2021). Exploring latent states of problem-solving competence using hidden Markov model on process data. Computer Assisted Learning37(5), 1232–1247. 10.1111/jcal.12559

<anchor name="c53"></anchor>

Xiao, Y., & Liu. Y. (2024). A state response measurement model for problem-solving process data. Behavior Research, 56(1), 258–277. 10.3758/s13428-022-02042-9

<anchor name="c54"></anchor>

Xu, H., Fang, G., Chen, Y., Liu, J., & Ying, Z. (2018). Latent class analysis of recurrent events in problem-solving items. Applied Psychological Measurement, 42(6), 478–498. 10.1177/0146621617748325

<anchor name="c55"></anchor>

Xu, H., Fang, G., & Ying, Z. (2020). A latent topic model with Markov transition for process data. British Journal of Mathematical and Statistical, 73(3), 474–505. 10.1111/bmsp.12197

<anchor name="c56"></anchor>

Zhan, P., & Qiao, X. (2022). Diagnostic classification analysis of problem-solving competence using process data: An item expansion method. Psychometrika, 87(4), 1529–1547. 10.1007/s11336-022-09855-9

<anchor name="c57"></anchor>

Zhang, S., Wang, Z., Qi, J., Liu, J., & Ying, Z. (2023). Accurate assessment via process data. Psychometrika, 88(1), 76–97. 10.07/s11336-022-09880-8

<anchor name="c58"></anchor>

Zumbo, B. D., & Hubley, A. M. (2017). Understanding and investigating response processes in validation research (Vol. 26). Springer.

Submitted: November 1, 2022 Revised: June 23, 2023 Accepted: October 5, 2023