Redesigning an Educational Technology Course under a Competency-Based Performance Assessment Model

The transformational process of an educational technology online course redesign into competency-based learning approach is described. This process implied the understanding of paradigm change, the integration of cognitive theory foundations, the selection of a learning taxonomy, the application of performance and authentic assessment principles as well as the design of performance rubrics. As a result, acompetency-based performance assessment model and a student performance reportemerge.


Introduction
Higher education degrees have recently become inadequate in terms of demonstrating the extent to which graduates are capable of confronting the challenges posed by the knowledge-based economy (Caldwell & Cattermole, 2015;Chan, 2015).A widely reported issue by employers is the scarcity of employees who are able to transfer their acquired knowledge to solve problems and make decisions.As stated by McKinsey (2013), worldwide statistics show high levels of youth unemployment and a shortage of people with significant job skills.Seventy-five million youth are unemployed, half of these youngsters are not sure that their postsecondary education improved their chances of ISSN 1392-0340 E-ISSN 2029-0551 Pedagogika / 2017, t. 127, Nr. 3 getting a job, and almost 40% of employers have found weak levels of skill development in young workers.
The need for supplying industry sectors with an adequately prepared labor force places online education programs at the center of the stage because online education makes it possible for people to continue learning by taking advantage of the flexibility that the digital learning environments provide (Allen & Seaman, 2013;Everhart, Bushway, & Schejbal, 2016;Morcke, Dornan, & Eika, 2013;NCES, 2016).However, lifelong learning with technology by itself cannot adequately respond to the labor market's needs.The competency-basedlearning (CBL) model is a plausible alternative that focuses on assessing learner's performance (knowledge and skills applied for a reason in a certain context), rather than knowledge in isolation (OCDE, 2001).
Thus, given that the CBL model focuses on performance, it requires an educational paradigm transformation together with a change of course design and teaching and assessment practices (Caldwell & Cattermole, 2015;Chag, 2015;Koh, Tan,& Ng, 2012).Under CBL, the educational paradigm changes, as this model appeals to a holistic interpretation of the learning process, rather than a cross-sectional view of learning results.The change of course design and teaching occurs when class materials (readings, videos, multimedia, etc.), class assignments, and activities are intertwined yet aligned towards the acquisition of a predetermined set of skills or competencies.The assessment practices change, as this model emphasizes providing feedback during skills development, rather than giving a grade as a measure of knowledge acquisition.
Purpose of this study: to describe the transformational process of an online Educational Technology course into a CBL performance assessment approach.The process is encapsulated in a proposed model in which CBL performance assessment elements converge.
Objectives of the study: 1.To identify the challenging aspects of transforming objective model course development to CBL in higher education 2. To describe the basic processes and techniques for redesign courses to CBL.
3. Toestablish aredesign model of the development of intellectual capabilities of students.
Methods.A critical review of research literature: analysis of CBL design process as well as performance and authentic assessment requirements from diverse sources, the identification of problematic aspects while CBL is applied in different educational levels in general, and in higher education specifically.All these steps derived in the creation of integral CBL redesign model.This model has been applied on an online graduate educational course.

Literature Review Change of Paradigm
For many years, human activities have been analyzed from different philosophical perspectives to understand the way truth and reality emerge.In the educational field, as well as in other disciplines, both positivism and phenomenology paradigms have sparked debate about human understanding of findings and decisions.
One key philosopher in positivism is Comte, whose work gained strength during the years of the French Revolution (Kremer-Marietti, 1997).His paradigm established a pathway to define the scientific process and research methodologies with at least three main underlying principles: (1) Reality exists and it is measurable independently from particular human views or perceptions; (2) Existence is governed by natural laws that can be observed and measured, an attribute that allows for theorization and generalization; (3) The behavior of individuals can be predicted following a well-conducted, controlled research method that uses appropriate measures (Peca, 2000;Stone, 1992).Phenomenology, the counterpart of positivism, emerged with Heidegger and Gadamer's work.This qualitative paradigm arose from a social need to understand reality within a more holistic conception beyond the mere re-creation of someone else's meaning (Laverty, 2003;Patton, 1990).Phenomenology emphasizes the following: (1) reality is based on social constructs, and it only makes sense from an individual's point of view; (2) rules that govern behaviors and thoughts are directly related to context; and (3) the behaviors of individuals are unique.Qualitative research methods are ways to explain phenomena in a deeper way when considering certain contexts, not for generalizing, but for inductively constructing new pieces of knowledge.
The coexisting influences of the two antagonist paradigms in education allow us reflect on their impact and legacy.In the case of positivism, two redundant critical consequences are the high level of rationality and the systematic ordination of beliefs.Taken together, according to Thomas (1998) and Giroux (2001), these concepts constitute a dramatic limitation in explaining human learning processes.Under positivism, the learning process focuses on academic achievement (content) and not on skills development (procedures), generating a gap between learning about the content and learning about procedures.Hence, there is a lack of reflection on what is being learned and the usefulness of it (Stolovitch & Keeps, 2011).
On the contrary, Escudero (2003) emphasized some unequivocal influences of positivism in education, such as the definition and precision of learning expectations expressed through learning objectives.According to these authors, an unprecedented amount of information can be used to enhance the understanding of learning processes based on testing and statistics.
In the case of phenomenology, Jardine (1990) recognized that even if ambiguity is one of the weaknesses of its practice, a major strength is connectedness with the subjects of inquiry, which allows a deeper understanding of the learning phenomena in different contexts.For learners, qualitative feedback provides meaningful information about the gaps between learning progress and expected achievement, as opposed to precise but dry numerical scores and letter grades.
Nowadays, in education -as a discipline and as a process -both phenomenology and positivism coexist and influence instructional decisions.In planning the giving and taking of lectures and assessments, students and educators unlearn and relearn continuously by means of the two controversial approaches.A vivid example is the case of the CBL model, which is based on the phenomenological paradigm, with certain influences of positivism, since observation and measurement are part of its practices.

Change in the Assessment Focus
For years, tertiary education has suffered from a breaking off between discipline knowledge and practice (Caldwell & Cattermole, 2015;Chan, 2015).There is a shortage of educational programs emphasizing student assessment based on content knowledge acquisition in combination with the development of soft skills (Gerstein & Hershey, 2016) and hard skills.It is of concern that the role of student assessment in most courses is incomplete, as it measures content knowledge but fails to provide, to some extent, feedback on professional skills.
Nevertheless, higher education curricula have been reacting in several ways to the lack of alignment between graduate skills and labor-market demand.One of the strategies that have been pushing students away from traditional lectures into more real-life professional situations is the introduction of performance and authentic assessment features in educational practice (Palm, 2008).Performance assessment requires students to complete activities that imply the application of knowledge and skills.Several universities have already adopted this approach (e.g., Western Governors University, Singularity University, and Capella University).Hancock (2007) defined performance assessment as a process that involves observing and assessing students' behavior while the behavior is underway.Students are required to demonstrate the acquisition of knowledge and skills in one or more content areas, rather than just answering questions that assess declarative knowledge.Hence, defining a set of criteria to judge how well a task must be achieved is highly recommended.Moreover, students undergoing performance assessments may be asked to demonstrate their achievement after engaging in individual or group activities by producing an extended written or spoken answer or creating a specific service, product, or artifact.
Within the performance assessment framework, an authentic assessment takes place.It can be defined as the kind of assessment that is applied in real life or a natural situation in conjunction with variables that normally interact.In higher education, authentic assessment is versatile.For example, it can be applied to understand how deeply students' knowledge and skills have been developed through a game in the same way as in a real-life project (Gao & Grisham-Brown, 2011;Frey, Schmitt & Allen, 2012).
Recent research on this topic has shown that performance and authentic assessment is not an isolated but a strategic practice for increasing learning and educational achieve- ment.For instance, in Singapore (Koh et al., 2012), a fundamental educational change was proposed to leave dependence on rote learning or repetitive tests behind.Instead, students' experience in schools should be more engaging through innovative and effective teaching approaches and strategies for discovery and experiential learning under differentiated teaching, lifelong learning skills, and character building.According to Koh et al., the Singapore Ministry of Education established at least six strategies to move schools' authentic assessment practices forward: (a) promote higher-order thinking classroom activities, (b) practice knowledge criticism, (c) use manipulation and application instead of just description as a path for learning, (d) engender sustained writing for expressing the known and unknown aspects of situations, (e) connect to the real world, and (f) build teacher capacity to conduct performance and authentic assessment.Tita and Tita (2016) reported positive results from an authentic assessment experience in higher education.The research was conducted with 860 business students in a threeyear period (from 2012 to 2015).The authors demonstrated the relationship between using simulation programs or games and the reached level of performance.Furthermore, authentic assessment enhanced cooperation, fostered responsibility, and improved students' emotional management through gaming.The authors concluded that simulating various aspects of professional environment and real industrial processes allowed students to transfer knowledge and apply problem-solving skills.
In a case study conducted by Sherrett, Nefcy, Gummer, & Koretsky (2013), using a regular task-simulation design in the chemistry industry, the researchers not only understood the learning process deeply but identified 14 competencies that students had to develop in their professional studies to demonstrate higher performance within the most common real-life situations related to this discipline.
It can be concluded that knowledge is meaningful when it is applied in a context for a certain purpose.While measuring academic knowledge acquisition is a useful practice, defining expected competency levels and providing performance feedback to students based on standardsare gradually becoming relevant, thus evincing the legacy of positivism and phenomenology interacting in a crossbred educational approach.

Curriculum Design Change Background
Higher education institutions worldwide are being questioned about their contribution towards social and economic development.For instance, originating from the Bologna Process in 1999, the European Higher Education Area (EHEA) was created to ensure more compatibility among higher education systems in enhancing student competencies development related to employability and lifelong learning (Želvys & Akzholova, 2016).In Australia, The Tertiary Education Quality and Standards Agency (TEQSA) was established to ensurethat undergraduates fulfill new standards by assessing the needs and preparedness of individual students and cohorts, instituting a formative assessment that provides feedback on academic progress, and providing student support for com- Pedagogika / 2017, t. 127, Nr. 3 petency development and continuous improvement.In the United States, by the first decade of this century, the U.S. Department of Education's (DOE) had provided further recognition of CBL in the quest for better shaping of higher education curricula.Thus, the Commission on the Future of Higher Education created a comprehensive plan in 2015 for increasing intellectual capital, social mobility, and leadership by emphasizing acquisition of appropriate skills (Calhoun, Wrobel, & Finnegan, 2011).
From Objectives-Centered to Competency-Based Learning Several decades ago, the same issues and concerns of contemporary initiatives to improve higher education were addressed by scholars.Within the prevalent positivist framework of his era, Tyler (1949) established the foundations of curriculum design based on learning objectives following four steps: (a) identification of the learning objectives students must fulfill at the end of the learning process or event, (b) determination of learning experiences (assignments) in connection to desired competencies, (c) arrangement of learning experiences in a timeline within the course, and (d) design of the assessment methodologies (e.g.tests, quizzes, and essays).
Then, one of Tyler's students, Bloom (1956), addressed cognition in his taxonomy of the cognitive domain, establishing levels used for educational goals: knowledge, comprehension, application, analysis, synthesis, and evaluation.Later, in 1966, Gagne highlighted the importance of categorizing learning goals according to the type of outcome they represent in such a way that teachers could plan testing in a more concrete matter.Gagne recognized the existence of external conditions that could improve or diminish learning (Driscoll, 2004) and suggested that giving reflective feedback -after measuring for quality and pertinence outcomes -is essential in enhancing students' strategic thinking.
At the beginning of 21st century, Voorhees (2001) explained that the bridge between the traditional paradigm (positivism) and the new learning revolution can be found in competency-based approaches.According to the author, Competencies are the result of integrative learning experiences in which skills, abilities, and knowledge interact to form learning bundles that have currency in relation to the task for which they are assembled.Demonstrations are the results of applying competencies.It is at this level that performance-based learning can be assessed (p.knowledge utilization within the content domains of information, mental procedures, and psychomotor procedures. In summarizing the literature reviewed on curriculum design change and its evolution towards CBL implementation worldwide, it can be said that the change of perspective (paradigm) is one of the most crucial challenges that CBL confronts before it can flourish in higher education.In the words of Ford, "The application of competency-based education models to degree programs can be described as either evolutionary or revolutionary" (Ford, 2014, p. 4).
From this standpoint, the evolution of the teaching perspective implies a change towards a curriculum design whereby learning goals (objectives) acquire meaning when conceived as competencies linked to performance assessment in real context experiences for skill development and applied knowledge.It is a demanding process that involves its own degree of complexity.

Redesign Model Towards CBL in Online Courses
The current educational challenges to improve degree programs are positioning the CBL approach as a viable alternative for transforming higher education.An online search on specialized websites revealed a vast academic production of content related to CBL application in tertiary education.However, few sources have described what teachers or instructional designers have to do to apply CBL principles to a course subject, and none of the studies have provided a concrete model for designing courses under the approach (searched using Google Scholar, 07 14 2016).
Given the need for an instructional design prototype that integrates CBL, the researchers worked out a CBL transformational process model based on the findings of Voorhees (2001), Jones et al. (2002), and Gallardo and Valenzuela (2014).The following sections describe: (a) the transformational process to CBL, (b) the change from learning objectives to competencies, (c) the performance and authentic assessment through assignments, (d) the rubric design, and (e) the researchers' redesign model.

Transformational Process to CBL
The Technology Leadership Certificate offered by a public university in Texasis a component of a set of online graduate educational technology programs.In a first phase, the certificate's main goal is the use of technology for teaching purposes.Courses were designed based on textbooks and open sources.
In a second phase, courses were redesigned for standards-based assessment (AECT, 2012), which Ford depicts as "measurable, behavioral objectives . . .[that] specify what a learner should be able to 'do' and at what level" (Ford, 2014, p. 1).Under that scenario, project-based learning was taking place with relevant, on-the-job type projects in which "the competency component was managed through the implementation of e-portfolios showcasing individual student accomplishments" (Ford, 2014, p. 3), and rubrics were designed for each class project.In a third phase, researchers realized that in working with CBL, the assessment of projects would need to evaluate the competencies students would need to further increase their performance in the workplace.
The adoption of the CBL came as part of a collaborative initiative including one faculty at a private university in the northeastern part of Mexico and one faculty at the aforesaid public university in Texas.Under this research joint venture, a first course was redesigned.
As the CBL model states, competencies not only promote knowledge per se but also transfer or apply content in particular contexts and with specific purposes.Hence, the main competence defined by the researches was that at the end of the course, student must be able to use social media to lead, motivate, and educate coworkers, associates, or colleagues in the selection and use of educational technology tools for educational, professional development, or training purposes.

Change from Learning Objectives to Competencies
The original learning objectives were listed in the course syllabus as follows: (a) identify a variety of multimedia, web, and social media tools for communication, collaboration, and learning; (b) describe scientifically based instructional research for the use of multimedia, web-based, and social media tools in online learning environments; (c) pinpoint best practices for the design and development of multimedia presentations; and (d) apply a variety of multimedia, web, and social media tools for communication, collaboration, and learning.
Student assessments were originally (and will continue to be) based on three of the standards instituted by the Learning Environments Standards of the Association for Educational Communications and Technology (AECT, 2012), as depicted in Table 1.

Standards Indicators 1 Content Knowledge
Creating.Candidates demonstrate the ability to create instructional materials and learning environments using a variety of systems approaches.
Using.Candidates demonstrate the ability to select and use technological resources and processes to support student learning and to enhance their pedagogy.

Content Pedagogy
Using.Candidates implement appropriate educational technologies and processes based on appropriate content pedagogy.
Ethics.Candidates design and select media, technology, and processes that emphasize the diversity of our society as a multicultural community.

Learning Environments
Creating.Candidates create instructional design products based on learning principles and research-based best practices.
ISSN 1392-0340 E-ISSN 2029-0551 Pedagogika / 2017, t. 127, Nr. 3 Subsequently, the original learning objectives were disregarded and based on the course main competency and the set of AECT substandards in Table 1, and five sub-competencies were defined as follows: 1. Create materials for e-learning environments using appropriate pedagogy in a variety of educational technologies.2. Support learning with the selection and use of technological resources and processes that enhance pedagogy based on the literature.3. Integrate social diversity and multicultural community characteristics in designing learning processes while selecting appropriate media and technologies.4. Plan strategies to foster a learning community that empowers learners with diverse backgrounds, characteristics, and abilities.5. Identify the convergences between learning with technology and trends in the educational field and one's own professional experiences.

Restructuring Assignments for Performance and Authentic Assessment
Once main competency and sub-competencies were set, three learning activities were redesigned and aligned to help students progressively provide evidence of competency achievement.From the three redesigned assignments in the course, the final project was considered the most important, as evidence had to be shown regarding all five sub-competencies (descriptions of assignments 1, 2, & 3 are available in Appendix A).
The third and final assignment requires students to create a plan to motivate and lead colleagues and coworkers (by means of a social media tool) to use technology for educational or training purposes.Students are required to write a plan following a preset structure and to record a video presentation using the plan as a script.The plan should be situated in the context of the student's workplace or daily activities environment and refer to a real technology-related learning issue of concern at the moment the student is taking the course.Given the nature of the assignment, the sub-competencies framework would become evident in the final product, as students would have to demonstrate capabilities to create materials, support learning decisions based on the literature, integrate multicultural features, formulate strategies, and discuss trends in technology-based learning in an online forum with classmates.

Rubric Design
A rubric is a helpful assessment tool in organizing, synthesizing, and outlining performance requirement features in relation to certain criteria (Brookhart, 2013;Moskal, 2000).One of the rubrics' main advantages is the progressive display of performance levels.This lower-to-higher parameter helps both teachers and students to guide the actions towards achieving the expected competencies, focus on relevant aspects of the activity, and grade and give evidence-based feedback.In the case of the course subject of this article, a rubric for the third assignment was designed under CBL model principles to assess student performance in an authentic way by aligning sub-competencies as the assessment criteria for student levels of performance.First, the alignment of the main competency, sub-competencies, and the nature of the project were taken into account in defining which knowledge and skills students should demonstrate in the activity.Second, a learning taxonomy (Marzano & Kendall, 2008) was introduced as the theoretical framework that supported performance expectations related to the cognitive process.The taxonomy component allows the instructor to make clear differences from one performance level to another, as recommended by Goldberg (2014), who studied rubrics in terms of avoiding blind spaces or extreme similarities from one level to another.
Thus, the rubric sketch in Figure 1 encompasses two axes: Vertically, the rows convey criteria based on the sub-competencies to be demonstrated by the student in the third assignment.Horizontally, Marzano and Kendall's (2008) levels of processing are depicted in the column headers for each of the sub-competences, as the expected levels can vary depending on the level of the course (e.g., core course and advanced course) and the specific requirements of the assignment.The student levels of processing were enumerated from one to four according to Marzano and Kendall's cognitive scale.
The complete rubric has three additional columns on the right side, one for the student to know about the maximum points possible on each row, one for the instructor to type a score, and most importantly, one for the instructor to make qualitative comments (see the end of Appendix B).Once the assignments are graded, a two-page student performance report (SPR) is originated for each student by using software created at the private university in Mexico to record individual students' performance levels.The SPR contains three sections: one for general comments, one radar chart showing any resulting contrasts between expected and achieved cognitive processing levels, and a final section for the rubric itself.

The CBL Redesign Model (CBL-RM)
Redesigning the course process into a competency-based assessment design led the researchers to the delineation of a model with the elements involved in a CBL performance assessment converge.Figure 2 contains the researchers' proposed model for online course designers who are considering CBL for higher education programs of study.The model is intended to resemble the integration of emergent socioeconomic demands and quality education standards linked to real-life job situations, as well as technological tools (learning management systems, etc.) in the making of decisions for curriculum configuration.
The CBL-RM in Figure 2 can be interpreted by curriculum designers in two ways: by dimensions and by steps or segments.
From the dimensional point of view, the outside circle represents the societal environment, while the inner part represents a holistic set of key curriculum elements where the course transformational process occurs.The area labeled inner-educational issues serves to identify gaps and to trigger curriculum decisions to support a more holistic professional profile.
From the steps or segments point of view, the peripheral elements may serve to guide designers in conjugating the inner and outer parts of each of the triangular pieces.Steps 5 and 6 meritspecial attention.The researchers suggest a back-and-forth revision of both the statement of sub-competencies and the redesign or creation of assignments/projects to ensure mutual compliance.From the steps or segments point of view, the peripheral elements may serve to guide designers in conjugating the inner and outer parts of each of the triangular pieces.Steps 5 and 6 meritspecial attention.The researchers suggest a back-and-forth revision of both the statement of sub-competencies and the redesign or creation of assignments/projects to ensure mutual compliance.

Incoming and Future Research
Once the CBL-RM was formulated, the curriculum redesign process of the Technology Leadership Certificate Program was undertaken for the first course in the program.Student perceptions about SPR utility were to be collected at the end of classes in summer 2016 and fall 2016.The purpose of such a study collecting this data would be to identify areas of the course that may need adjustments within the framework of steps 5 to 8 to focus student performance assessment, as well as to consider adjustments in the rubric and in the management of the SPR if necessary.
Subsequent research may include observations of the model as applied to other courses in the certificate program.Furthermore, it would be useful to conduct research on faculty experiences in adopting the CBL-RM in different disciplines.Then, researchers should investigate learning activities outside of campus to measure the impact on employees' performance and to garner employers' perspectives as stakeholders.Employers' feedback could help in the potential evolution of the CBL-RM, especially on steps 3 and 4. Project 2: Each student will use a free social bookmarking application (content curator) to assemble web pages, articles, blog posts, images, and videos -all commonly related to the specific Educational Technology topic of the final assignment (Project 3) -and will post insights on every article collected.Students may invite classmates and other users to follow the topic or optionally use social media (Twitter, Facebook, LinkedIn, or other) to increase audience.
Project 3: Using a predetermined structure for the content, students will write a script describing how each student would use social media to motivate and lead colleagues and coworkers to utilize educational technology tools for educational or training purposes.Based on the script, students will record video presentations and post their videos in a closed group on Facebook (or a similarly functioning social media platform) and make comments on their own and their classmates' video presentations.
5) Jones, Voorhees and Paulson (2002) established the bases of the instructional design model concerning CBL.In contrast to what Tyler had exposed 50 years earlier, Jones, Voorhees, and Paulson's model proposed (a) an assessment and performance demonstration; (b) competencies that integrate, abilities, skills, and knowledge; (c) learning experiences that permit the development of those abilities, skills, and knowledge; and (d) traits and characteristics of students as the foundations for learning.Then, for ensuring a better understanding of what happens when the human brain is working on learning processes, Marzano and Kendall's new taxonomy (2008) enriched Bloom's taxonomy by declaring four levels of processing: retrieval, comprehension, analysis, and ISSN 1392-0340 E-ISSN 2029-0551 Pedagogika / 2017, t. 127, Nr. 3

Fig. 1 .
Fig. 1.Rubric sketch showing sub-competencies and levels of processing an example of how feedback and the radar graph appear in the SPR once completed.
, t. 127, Nr. 3 curriculum decisions to support a more holistic professional profile.