Why assessment designers need to know how to read a unit of competency
- Specialised VET Services
- 15 hours ago
- 5 min read
A study on ASQA's case study about using AI in assessment
This week, the National VET Regulator shared their special AI edition of the IQ publication. In it, ASQA gave us a case study to illustrate whether AI could be used in an assessment for the unit of competency BSBPEF502 Develop and use emotional intelligence.
We'll let you read the ASQA case study for yourself, but here, wish to point out the very good case it makes for:
a) understanding the assessment conditions as mandatory assessment context
b) understanding how to unpack a unit of competency
c) the adage "just because we can... doesn't mean we should"
The aim of this article is not to take a swipe at the regulator - their recent efforts to fulfil their ongoing obligations under Clause 1.4 of the Standards for VET Regulators 2015 is welcome.
By the way, Clause 1.4 states:
"The VET Regulator provides general education and guidance materials to RTOs to assist them to comply with the Standards for RTOs."
Instead, this article aims to unpack the example given as part of the latest educative efforts, and use it for further teaching purposes.
Why are we doing this? Because in the ASQA case study, they cite the example of a provider that looks to redesign its assessments for the emotional intelligence unit and seeks to evaluate whether an AI avatar could be designed to interact with the students as part of the assessment.
The intention is to skirt the challenges to organise role plays with trainers or volunteers and instead use the avatar in simulated workplace conversations.
The case study goes on to say: Before making any changes, the provider thoroughly reviewed the unit's performance criteria, assessment requirements and assessment conditions to determine whether simulated interactions and AI avatars were appropriate.
Here is the first issue we have. The unit of competency was endorsed in October 2020 - nearly two years before ChatGPT hit the mainstream market and the subsequent rapid development of the large language model technology that allows advanced text-to-speech and lifelike, interactive video avatars. So, it is impossible that there would be any indication in the positive, in any of the unit's performance criteria, assessment requirements and assessment conditions, that use of AI avatars would be appropriate for this unit of competency.
While it could be argued that not explicitly saying "you can/cannot use an avatar here" is not definitive permission either way, it's the unpacking of the unit requirements as a whole that point us in the right direction.
That, and the fact even AI admits that avatars have no emotions:
"No, digital and AI avatars do not actually feel or experience emotions, but they can be programmed or prompted to display simulated emotional expressions. Avatars lack a nervous system, consciousness, or brain chemistry, meaning they cannot genuinely feel happiness, sadness, anger, or fear. Any emotional response you see is just software processing text, audio, or user commands to trigger specific outputs."
- Google Gemini 27.8.26 in response to the query "do avatars have emotions"
When we begin to unpack the unit, let's go straight to the assessment conditions (AC) - the non-negotiable contexts for valid assessment to occur.
From the Standards for Training Packages 2012 (in place at time of the unit's endorsement):
You'll note the description in the table above states that assessment conditions are mandatory and specify "the conditions under which evidence for assessment must be gathered, including any details of equipment and materials, contingencies, specifications, physical conditions, relationships with team members and supervisor, relationship with client/customer, and timeframe."
Assessment condition 1 of the BSBPEF502 unit states (emphasis added):
Skills in this unit must be demonstrated in a workplace or simulated environment where the conditions are typical of those in a working environment in this industry.
I'm curious to know in which industry it is a requirement to display emotional intelligence when interacting with avatars?? Notwithstanding that avatars are not human, are typically sycophantic in their responses (meaning they will flatter and give positive feedback even when not warranted - and this is mentioned in the ASQA case study by the way), and avatars do not display emotion. Which is the crux of the unit's outcome. The whole unit is about emotional intelligence. Not just communication, not just leadership skills, but the ability to understand, use, and manage your own emotions in positive ways, as well as recognise and influence the emotions of others. Emotionally intelligent people typically have self-awareness, self-regulation, empathy (recognising, understanding, and sharing what other people are feeling), and the social skills to build strong relationships and manage conflict constructively.
Any student undertaking this unit must expect to exit with the skills and knowledge required to develop and use emotional intelligence to increase self-awareness, self-management, social awareness and relationship management in the workplace.
Surely proper development of emotional intelligence (EI) cannot occur in an environment devoid of others and others' emotions?
Further the performance evidence requires the student is able to demonstrate skills including (but not limited to):
Analysing interactions with co-workers, and their own personal emotional responses and triggers
To apply techniques that promote flexibility in interactions with others
Show consideration for the emotions of others
Promote development of emotion intelligence in others on at least one occasion
The foundation skills requirements say the student must be able to demonstrate the following inclusions (but not limited to):
Leads a collaborative approach, using inquiring and inclusive techniques, to develop understanding and skills that enhances individuals’ emotional intelligence
Identifies personal attributes and considers the impact on others and modifies approach to support development
PC 2.1 asks for evidence of being able to "Analyse and document emotional responses of co-workers"
PC 2.4 asks for evidence of being able to "Apply techniques that show consideration for the emotions of others when making decisions"
PC3.2 ask for evidence the student can "Develop tasks for assisting others to understand effect of personal behaviour and emotions on others in the workplace"
Even in this snapshot of selected unit requirements, a clear picture is emerging regarding the need for "others" - other human beings with feelings, perspectives, own levels of EI. You may have noticed some of the bolded text above - this clearly indicates the need for assessment to occur in relation to, and in conjunction with, other people.
Luckily, the ASQA case study went on to illustrate why the use of avatars would not be valid but the concern is that it does describe a process with the newly developed assessment tool that even got to the point of "test[ing] and review[ing] it with a group of assessors, industry representatives and instructional designers."
Had the unit been properly unpacked with a view to quality, compliant assessment, there is no way the assessment tool would have been designed, developed or tested to include use of AI avatars! In a real life scenario, what a waste of time and effort that would have been. This is where experienced instructional design and the ability to properly interpret competency requirements is invaluable. I have no problem with the idea of using the technology for learning and practice, but as shown above, the assessment must definitely use people.
Undoubtedly, technology such as AI brings all manner of efficiencies, personalised learning benefits and opportunity. However, sound instructional design will always look past the "bright and shiny" and focus on the true purpose and intent of the exercise. And in this case, on the compliance aspects as well.
Just because we can access technology like AI avatars, does not mean we should. Especially when the unit of competency calls for something different.
This article is AITAS Rating 1
What is the AITAS?
27.8.26
