Award data migration is the reason most programmes stay on a portal they have outgrown. The entries are the easy part — a competent export handles those in an afternoon. What blocks the move is everything around them: the judging record, the files, and the Chinese characters that arrive on the other side as question marks.
Our audit of Hong Kong award portals found 10 of 11 programmes running bespoke systems, some commissioned a decade ago. This is written for the people running them.
On this page
- What actually blocks the move
- The award data migration inventory
- The Hong Kong encoding trap
- When to cut over
- How to prove it worked
What actually blocks the move
Ask a programme why it has not replaced its portal and the answer is usually “the historical data”. Press further and the concern is rarely the entries themselves. It is the judging record — who assessed what, what conflicts were declared, what scores were given and revised, and why.
That concern is correct, and it is the part most migrations quietly abandon. A bespoke portal typically stores judging data in a shape nobody documented, built by a developer who has since left, with the reasoning encoded in application logic rather than in the database. Exporting the entries is trivial. Exporting a defensible account of how decisions were reached is the actual project.
Programmes that skip it discover the loss at the worst possible moment — when someone challenges a past result and the record has become a spreadsheet of final scores with no working behind it.
The award data migration inventory
Work through this before anyone writes an export script. The third element in each row is the one people forget.
Structured field values across all past cycles.
Watch Form structures change between years. A 2019 entry may have fields that no longer exist, and flattening them to fit the current form silently destroys the historical answer.Supporting documents, images, video, signed declarations.
Watch Checksum every file before and after. Bulk transfers truncate large files without error, and nobody opens a 2018 PDF to check.Per-criterion scores by judge, with assignment history.
Watch Raw and adjusted scores must both come across, plus which normalisation was applied. Adjusted scores alone cannot be recomputed or defended.Who declared what, and who decided the outcome.
Watch Frequently held as free text or, worse, in email. If it never existed in the system it cannot be migrated — record that gap explicitly rather than leaving it implied.Score changes with author, timestamp and reason.
Watch Often the largest table and often the least complete. Decide deliberately how far back to carry it rather than letting volume decide for you.Unsuccessful entries and documents beyond your stated retention period.
Watch Migrating these carries a PDPO problem into a new system and restarts its life. A migration is the cleanest deletion opportunity a programme ever gets.Judges and entrants with no activity for several cycles.
Watch Migrating credentials and contact details for people who last engaged in 2017 expands your exposure for no operational benefit.Treat the migration as a retention exercise. The obligation under DPP2 is to keep personal data no longer than necessary, and a migration forces you to look at every record you hold — often for the first time. Deciding what not to move is the same decision as deciding what to delete, and it is far easier to make now than during an audit. Our PDPO compliance guide sets out a defensible retention shape.
The Hong Kong encoding trap
This one is specific to the region and it damages exactly the data you can least afford to damage: people’s names.
Older Hong Kong systems commonly store Chinese text in Big5 or in Big5-HKSCS — the Hong Kong Supplementary Character Set, which extends Big5 with several thousand characters used in Hong Kong and in written Cantonese that base Big5 never covered. HKSCS-2008 reached 5,009 such characters.
Three failures follow, in increasing order of how long they take to notice.
Decoding HKSCS as plain Big5. The obvious one. Characters outside base Big5 come through as mojibake, and it is visible immediately — assuming anyone is looking at the Chinese records rather than the English ones.
data.decode(‘big5’)
// correct for Hong Kong legacy data
data.decode(‘big5hkscs’)
Silent substitution. Not every HKSCS character has an unambiguous Unicode mapping. Conversion libraries replace what they cannot map with a question mark or a replacement character, and they do it without raising an error. A name loses one character and nothing in the log records it.
Private Use Area legacy. Before Unicode caught up, Hong Kong characters were parked in Unicode’s Private Use Area — 1,686 of them under HKSCS-2001. PUA code points mean different things in different systems, so text stored that way can arrive interpreted as entirely unrelated characters. Later Unicode versions gave proper code points to all but a handful, but data written under the old regime still carries the old values.
Check your target column type. Some HKSCS characters sit outside the Basic Multilingual Plane and need four-byte UTF-8. A MySQL column declared utf8 holds only three bytes per character and will reject or truncate them — utf8mb4 is required. This is a schema decision made early and discovered late, usually by a person whose name will not save.
The practical defence is a character-level audit rather than a spot check: extract every distinct character present in the source data, convert, and compare the two sets. Anything that vanished or became a replacement character shows up immediately. Do this before the real run, not after. It pairs with the wider issues covered in our guide to running a bilingual award programme.
When to cut over
Between cycles, and not close to either end. The window opens once the previous cycle’s results are announced and judging access has been revoked, and closes well before the next call for entries goes out.
Migrating mid-cycle means reconciling entries that changed during the transfer, which is a genuinely hard problem and an unnecessary one. Migrating immediately before a launch means discovering encoding faults while entrants are already submitting. Both happen because migration gets scheduled around a contract renewal date rather than around the programme calendar.
Keep the old system readable — not writable — for at least one full cycle after cutover. It costs little and it is the only way to settle a question about what the source data actually said.
How to prove it worked
Row counts and file checksums are necessary and not sufficient. They prove data arrived, not that it still means the same thing.
The test worth running is this: take last cycle’s migrated data and recompute the results in the new system. The final ranking should be identical to what was announced.
If the rankings differ, you have learned something important. Either the migration lost or altered scoring data, or the new system applies a different normalisation method than the old one did. Both are serious, both are fixable now, and both are extremely expensive to discover during a live cycle. A migration that reproduces a known result is one you can defend.
Add three cheaper checks alongside it: the distinct-character comparison described above, a sample of historical entries opened and read in full by a human, and confirmation that at least one complete judging trail — assignment through conflict declaration through revision — is reconstructable end to end.
Migration capability is worth putting to vendors directly, alongside the questions in our RFP guide. Ask specifically who performs the extraction from your legacy system, how HKSCS data is handled, and whether the reproduce-last-year’s-results test forms part of their acceptance criteria. A vendor who has done this before will recognise the test immediately.
Planning a move off a legacy portal? Book a live demo or see pricing.
