An assessment delivery engine to administer low- and high-stakes assessments. Available in three
The QTI Assessment player is a Java web applications provided in war packaging.
It is possible to show the same assessment in different styles and lay-outs
Safari,Chrome, IE9 and above, Firefox. The player renders the QTI to an actual assessment that humans
can see in a browser.
Screen resolutions of candidates may differ. Ideally you want the assessment to render correctly on all
In order to do a test a certain person has to be combined with a certain test for a certain time period.
This is called a session. The player has an API to create these sessions from for example, your student
Assesssments may contain audio and video files. In order to be sure that these files will be played
successfully on the candidate’s computer the player has an audio and video check before the actual
assessment starts. The tests are conditional when the assessment does not have a video fragment in it
the Video test is skipped.
QTI specifies different interaction types. Basically interaction types are high-level representations of
question types. One interaction type can represent different question types depending on the visual
Multiple choice with text or images or both
Multi select with text or images or both
Text entry interaction
Fill in the blanks
Inline Choice Interaction
Fill in the blanks drop down
Gap match Interaction
Fill in the blanks drag and drop
Graphic gap Match Interaction
Fill in the blanks drag and drop images
Order images text or both
Hot text interaction
Text hotspot, select sentences words or paragraphs in a text
Hot spot Interaction
Image hotspot, select hotspots on an image
Position Object interaction
Place 1 or more objects on a canvas
Select point interaction
Select 1 or more points on a canvas
Graphic order interaction
Decide order of hotpsots on a image
Graphic associate Interaction
Connect hotspots on a image
Set a value by moving a slider
Determine min and max number of plays for audio or media file. Determine how many times an audio or
media file is played.
Time based assessment, test-part and item
It is possible to set a maximum time on a assessment an item and a testpart.An item is what is shown in
one screen. Normally this is called a question, although more questions can be shown in one screen.
Every item has a number to represent it in the navigation A test-part is a collection of questions that
has to be closed at the end. When closed it can not be opened again. An assessment must contain at least
one test-part but can contain more.
To make the processing of the input of the candidate easier.Trim –removes whitespaces, Decimal notation
– changes localized to uniform notation, Diacritcs – changes diacritics to charactrer without diacritic
There are several tools available in the player. You can use the QTI to decide if and where a certain
tool is available
The player offers a navigation that allows you to only move forward through the assessment
The player offers a navigation that allows you to move freely through the assessment
Show answered state
The player can show which questions are answered and which are not
The player can show an overview that can show all the questions in the test-part, the answered state and
the marked state. It can also be used to navigate
The player can handle adaptive testing. An adaptive assessment has an algorithm that selects the next
item to show to the candidate based on the answers the candidate has given.
The player can show a result screen after submitting the assessment. Different ways of showing the
results are possible The amount of time that the results are shown can also be set
Many interactions between the player and the candidate can be logged. The following events can be
logged:Navigation changes, Answers, Session started / ended, Item / Assessment imported
To prevent more than one browser showing the same session we use secure cookies HMAC and we are OWASP
This is an extention to the Assessment Delivery Engine. The process of securing an endpoint.
The candidate can only access authorised features. An example of this would be turning the
device into a 'Thin Client' by locking it down so that an end user can only connect to predefined
applications, like taking the assessment only.
A Front-end application to monitor candidate's assessment progress. It is used in test centers or at test
location. It provides the supervisor - for example - an overview about:
candidate's seats (location)
who is taking which test
if the test is started
individual progress made (e.g. time left)
if the test is submitted
There is an option for the supervisor to intervene and pauze, re-start or hand-in the test per
The Central Office for Driver licenses (Dutch: Centraal Bureau voor Rijbewijzen (CBR)) is the
appointed by the department of infrastructure that assesses driving skills and medical fitness of
(car, truck, boat, and plane).
The CBR has 5 offices, 55 practical exam locations and 30 theoretical exam locations.
In total 1.500.000 exams are held each year; 800.000 are theoretical exams.
The main part of these theoretical exams are audio-visual and collective which are held on a certain time on
a certain location. All candidates are exposed to yes/no questions referring to a series of slides and
In view of the current individualization of our society CBR wanted to change the current process of
registering, planning and taking of all 800.000 theoretical exams. Instead of the collective exams it should
be possible to take an exam individually whenever a candidate wishes to. Reasons are:
Efficiency: candidates can work in their own tempo and finish exams faster than they would in a group;
Fraud prevention: individual exams make different versions of the same exam possible and cheating virtually
Service: candidates can take an exam on a date and at a time they prefer.
The new assessment platform for CBR allows to take an exam individually and at a preferred moment.
It includes features such as a calculator, reference tool and a time-based functionality (complete
the exam in a certain time).
The new system is based on Trifork’s QTI (Question Test Interoperability) Assessment Delivery Engine, using
a world-wide Open Standard. The client lockdown allows secure, high-stake digital assessments by making it
impossible to start up other applications or browse on the internet while taking a test and therefore
prevent from fraud. The whole system is completely touch based (including a touch(on)screen keyboard) and
displayed on a high resolution touchscreen.
Trifork also integrated a text-to-speech functionality for candidates with reading difficulties. And on top
of that Trifork delivered an on-premise proctoring exam monitor which helps supervisors to manage the active
exam sessions on a certain location (who is taking which exam, how much time is spent and left, how many
questions are answered and left). Furthermore the supervisor can pause and abort individual exams or all the
exams that are active on a location.
This project was a European tender, fixed price and fixed date. Not all requirements specified in the tender
where implemented in the end because new requirements popped up during implementation. Because Trifork used
an agile approach the product owner had the freedom to decide every sprint which functionality was
implemented. Since the project was fixed price this meant that the product owner had to exchange
functionality when out-of-scope functionality was added to a sprint.
CBR has done a major overhaul of their process. The center piece is Trifork’s QTI Assessment
Delivery Engine. Another major part of the process supervising the exam is also supported by software
created by Trifork, the exam monitor.
Four Swiss cantons of Aargau, Basel-Landschaft, Basel-Stadt and Solothurn (Bildungsraum
have set themselves a formidable goal to improve their educational system within the upcoming years.
For this major challenge the Institut for Bildungsevaluation, University of Zurich (IBE) and the Dutch
assessment institute Cito were selected to compose all the requirements for the new system. IBE and Cito
defined the advanced set of needs to redefine the educational system for the Swiss children between the
age of 4 and 13 year. This formed the new e-learning portal Mindsteps which enables the students and
teachers to have access to a very advanced learning system for both high-stake and low-stake assessment
Via the portal Mindsteps the students can be tested on a regular basis on their knowledge of different
topics throughout the year. Both the students and teachers have access to an item database which can be used
during the whole school year. Teachers can use the item database for developing their tailor-made computer
based low-stake tests or tailor-made computer based learning units. Both can be allocated to a student or to
a specific group of students. Students can use the item database for learning wherever they are (outside of
The primary function of the item database for teachers and students is to deliver data driven information
about the learning processes of students. By means of differentiation within each class or learning group
this instrument is adapted and advanced to support the individual learning progress.
The technical basis for the project is a well-designed IT-Solution. All items are stored in an item database
(Aufgaben-Datenbank). A web application has been developed for administering high-stake, and low-stake
testing as well as learning.
Trifork was asked to solve the most challenging part of the project; providing the digital assessment
platform: the heart of the e-learning portal Mindsteps.
“Major reason for selecting Trifork as partner has been their QTI Assessment Delivery Engine product in
combination with their extensive knowledge of large-scale digital assessment systems”, says Urs Moser, head
of IBE. “We were especially impressed of their work done for the Dutch Board of Exams where they have
implemented one of the most advanced digital assessment engines in an environment suited for 30.000
Trifork’s QTI Assessment Delivery Engine is a quality product which enables digital assessments of the
future for many organisations. Built with the latest technologies, the platform is an ultra scalable
solution for any serious assessment organisation. The low-stakes requirements of IBE creates the challenge
to support many type of devices, something the Assessment Delivery Engine is very capable of with a complete
HTML5 based implementation.
The Assessment Delivery Engine is completely based on IMS QTI, the worldwide standard for digital
assessments. Thru the support of this standard Trifork is interoperable with many external assessment
systems, like the authoring system of Cito’s Questify Builder.
With an agile project approach, Trifork has delivered the right functionality on time for the project. This
has enabled the successful delivery of the Mindsteps portal, providing a solid basis for the future plans of
“Trifork has proven to be a worthy partner for us on both the technological and the digital assessment
domain. In a short timespan they were capable of delivering a very advanced digital assessment system which
enables us to meet our current and future needs”
QTI Assessment Delivery Engine
Trifork’s assessment delivery engine is a state-of-the-art player capable of delivering and automatically
scoring QTI assessments. This engine delivers the latest version of IMS QTI assessments to any HTML capable
Trifork is a trusted QTI implementation partner for many customers within the Educational domain, including
integration of the engine within existing assessment environments. With the use of client-lock-down
technology, this engine is now also capable of securely delivering the examinations. This combined with the
interoperability between several author and assessment environments proves it to be a very powerful tool.
The QTI Assessment Delivery Engine makes it possible to publish, deliver and score assessments for
practicing, assessing and examining within any assessment environment.
College voor Toetsen en Examens
The College voor Toetsen en Examens (het CvTE) is an independent administrative board, responsible
for all exams in secondary education and exams for Dutch as second language (necessary for obtaining a
The mission of this board is to guarantee the quality of the exams and to provide circumstances that
assure a smooth examining. Moving all online school exams to the QTI 2.1 format is part of the strategy of
succeeding in this mission.
The system the CvTE has in place now is capable of planning, presenting and correcting exams. This system is
developed over ten years ago not meeting today’s standards and witholding any innovation on future
development. The CvTE decided that a new system for digital examining was a logical investment. Further
development on the old system would not be cost effective, since the new features CvTE needed, would have a
major impact on the overall architecture of the system. Data storage and retrieval and the planning module
are the responsibility of DUO (Dienst Uitvoering Onderwijs). The modules to preview, view, correct, review
and practice exams with, are Trifork’s responsibility.
The most visible improvement is the user interface. For all the tools Trifork created a new interaction and
visual design. The new design had to preserve all the things that people liked after ten years of use. In
the same time several tools were introduced and embedded in the examining environment; calculator, notepad,
magnifier, spell-check, protractor, ruler, marker, special characters, and formula editor.
A major audible improvement is the text to speech (TTS) integration. Trifork integrated Acapela TTS which
makes it possible to have all text based content read to you. Trifork supports Dutch and 3 other languages.
The TTS switches language automatically when the language within a text changes.
The content of the exams is provided by Cito, and needs to be imported into the assessment platform. The
results of each candidate also need to be reported back to Cito, for analysis. The system uses QTI 2.1 to
exchange assessments and results. With QTI 2.1 CvTE can accept a solid, well proven and world-wide adapted
content standard. This enables them to take in content from various sources and stay independent of their
To prevent cheating, Trifork has developed a client lock-down browser. This browser prevents access to any
other application during the assessment. A browser is provided for Windows, Linux and Mac. The assessment
system does a browser check when a candidate tries to start an assessment. Access with regular browsers is
translates the QTI to HTML5 within the student’s browser. Since QTI describes the entire assessment
including the answers it would impose a security risk to sent this to the browser of the candidate. For this
reason Trifork created a rendering filter that strips all the data that should stay secret.
The storage and retrieval of questions and given answers is the responsibility of DUO (Dienst Uitvoering
Onderwijs). Together with DUO, Trifork has designed a DAL (Data Abstraction Layer), which acts as the
interface between the assessment environment and the storage infrastructure. For local development, and for
the offline preview environment used by assessment constructors, Trifork has provided an in-memory
implementation of the DAL.
The authentication process is part of the logistical application built by DUO. The assessment engine itself
does not need to know about user details. However, it does need to be able to restrict access to only
authorized candidates. For access security, Trifork has implemented HMAC validation on the URL that launches
the assessment engine.
Besides the QTI player in which a student makes his assessment Trifork also created a correction component
for the teachers to score manually, and a preview component for test constructors in which they can view the
assessment they are creating and all the metadata that comes with it.
Although the project was a European tender with a fixed price and fixed date, CvTE managed to keep the
requirements specified in de tender on a level that made it possible to fill in the details just before
implementation. Some functionality was implemented gradually in several iterations. The product owner made
full use of the potential that the scrum process offers and was always looking for functionality that added
the most value.
The new system will provide a robust platform to facilitate the growing number of digital exams. Most of the
feedback that has been generated during field test and during the real exams in which users encountered the
new system are very positive. To maintain this quality the system will be evaluated constantly and will be
improved in the years to come. Last but not least the project started in winter 2011 and has been delivered
in spring 2013, on time and within budget.
Trifork Learning Solutions an e-Assessment software integrator
English and/or Dutch
European Union applicants only
You - as backend developer – can make a difference within our software development team.
We want you to help us build technology that complies to good, fair testing. You are
the educate, skill and employ branch ranging from governmental bodies, test and educational
publishers, awarding bodies and certification organizations who use our QTI technology.
We invite you to strive for equal opportunities for every pupil, student and candidate
taking an assessment. Preventing arbitrariness in the decision of passing or failing. Taking an assessment
should be fun whereby data integrity is non-negotiable.
It is not our intention that you deliver end-to-end solutions. Your focus is interoperability
with a sense of best-of-breed mentality and a sharp eye for integrating with the proper
applications that our clients already have. When you add this with your customization magic
it brings the competitive advantage of your clients to life…
What you are working on is:
Developing new products and modules set out on our roadmap for 2016/17
Raising your voice to making improvements and set a new pace to inspire the team
Becoming a full stack developer (if you already are that is exciting)
Maintaining and upgrading our QTI Engine and Player, Previewer, Reviewer, Importer, Lockdown Browser and
Extra features like our tools: calculator, text to speech, MathML, ruler, monocular
Extending our add-ons and extensions like: templating, inline feedback, item randomization,
assessment review modus, corrector module, test inspector, media interactions, responsive design,
Extending our APIs like: Import API, Corrector API, Session Management API, Session Monitor API,
Analytics API, and Transcoding API
Providing support to our clients: reproducing, fixing bugs and offer consulting
Taking part in – international - client and sales meetings
Presenting your solutions to the team and clients
Being an example to internship(s)
With what kind of knowledge do you need to do that?
We encourage you to work with us on agile and test driven principles. Help us towards our goal of
continuously delivering the best possible solutions for our clients. Knowledge and experience in delivering
robust, highly scalable applications is a definite plus!
Ambition to grow towards a full stack developer
Comfortable to speak and write English
Apply Agile development processes like Scrum and Kanban.
Understand what performance and scalability means in high stakes testing
Working with T-Learning Solutions means
You have your own desk and a permanent team. You will not be outsourced to clients. These are our top 10
Salary we offer an excellent salary that goes with your age, training and experience.
Pay rise you receive – in case of proper performance – an annual pay rise as percentage of
Bonus If our business does well and you must profit from that too. Based on annual performance
indicators like revenue growth and EBITDA you can add a bonus to your annual wages.
Holiday fee You receive each year a holiday fee of 8,33% of your annual gross salary, which
complies with an extra month pay.
Holidays You can spend 25 holidays per year.
Pension We pay 50% of the maximum pension fee within the current premium system.
Working hours 32-40 hours a week. You can plan your hour with flexibility in close
with your team. If possible you can work occasionally from home.
Equipment A laptop is provided.
Lunch lunch is taken care of. You can make your own sandwich toast and boil your eggs
in our dining kitchen. Sodas and refreshments are available throughout the day.
Shares You can buy shares in Trifork Holding.
We have an office in a stunning industrial fashion building on the 2nd floor.
You can brew your own Nespresso.
There is a cinema on ground floor operated by UIP.
10 minutes from Amsterdam city center.
Come and join our team. You will experience how fascinating the assessment industry is… and it will not
let you go… This is why we are the best and our clients feel that.