UL’s Director of Engineering Benchmarks Jani Joki said the phones will appear unranked, and without PCMark for Android scores, at the bottom of its list.
For those who came in late, MediaTek was discovered to have installed software called “sports settings” which recognised a benchmark test was taking place and reset the chip to function differently to pass it.
No one benefits when a company tries to manipulate benchmark scores to its advantage. That's why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software, Joki said.
UL said that it has temporarily delisted the PCMark for Android scores for all devices that use the following MediaTek chipsets:
- MediaTek Helio G90
- MediaTek Helio G70
- MediaTek Helio P95
- MediaTek Helio P90
- MediaTek Helio P65
- MediaTek Helio P60
- MediaTek Helio P20
- MediaTek Helio A22
In total, more than 50 devices from more than 25 different vendors have been delisted including several recent, high-performing models such as the Xiaomi Redmi Note 8 Pro, Oppo Reno3 Pro, Vivo Y19, and Realme 6.
MediaTek has since responded with a post on its website, with a response which does not impress Joki.
"MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets."
Joki said that using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an "accepted industry standard". It is, in fact, the very opposite of the accepted standard.
Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device's true performance in everyday use.
“A device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.”
