cpufreq: armada-37xx: forbid cpufreq for 1.2 GHz variant

[erdoukki]

openwrt/openwrt@f407b2f

How can we contact Marvell to have the needed information ?

[pali]

@kostapr: Hi! Could you look at this bug report?

[kostapr]

I have no problem with this patch. BTW, Ken Ma, Igal Liberman and Victor Gu are not with Marvell anymore. For the future Armada-related patches, please add Stefan (stefanc@marvell.com) and Nadav (nadavh@marvell.com) to the CC list.
Reviewed-by: Konstantin Porotchkin kostap@marvell.com

[robimarko]

@kostapr But that patch is not a solution, it's just a hotfix to get the devices booting and not constantly crashing due to voltage issues.
The solution cant be just to disable cpufreq and force the lowest frequency

[kostapr]

@robimarko I cannot comment on the problem. I personally think that not all 37xx dies are capable to work stable at 1.2GHz. However, this should be confirmed by HW design or production team at Marvell. Hopefully @haklai can add more on this matter.

[robimarko]

@kostapr Well that is normal, but the thing is that those marked and sold as 1.2GHz ones are having issues, personally, I have a lot of those in the field and they all crash currently if you allow them to scale.
I know that @pali has been trying to get solved for a while now.

[pali]

There is part order number 88F3720-xx-–BVB2C120-P123 of A37xx SoC which is designed for 1.2 GHz. This SoC die has below its Marvell logo marking C120 (speed code).

So @robimarko could you confirm that you have the right 37xx die which is designed for 1.2 GHz and in this case @kostapr or @haklai could you get more information about HW design / production team where is the issue?

[robimarko]

@pali I opened one of the Esspresobin Ultras I have and the SoC PN is: 88F3720-A0 C120
Even the stock ATF/WTMI and U-boot see it as a 1.2GHz model.
I am attaching the image as well.
https://imgur.com/A02jhaw

[pali]

@kostapr so for sure above @robimarko's SoC is designed for 1.2 GHz.

[stefanchulski]

@pali If you disable DFS feature and boot with 1.2GHz frequency only, do you see any crashes?

[pali]

@stefanchulski currently I do not have 1.2GHz variant of A3720 SoC.

@robimarko and @erdoukki could you please do required tests for @stefanchulski?

[erdoukki]

Sure, with pleasure, as usual...
Just give me the needed patch file or binary, please.
I will also check the CPU rerefence of my Ultra.

[robimarko]

@stefanchulski If I am seeing it correctly, it's using 1200MHz by default after booting as the kernel is not scaling it anymore.

[    2.305272] Unsupported CPU frequency 1200 MHz
root@OpenWrt:/sys/devices/system/cpu# cat /sys/kernel/debug/clk/cpu/clk_rate 
1200000000

I need to really stress test it before claiming that it's stable with the WTMI set VDD.
Which in my case is:
SVC REV: 5, CPU VDD voltage: 1.213V

But I have seen samples that use 1.26V as well, and I don't think that the CPUFreq has a way to know this and uses too low voltage for most boards.

UPDATE:
Even a couple of seconds of stress testing will crash it, so it's not stable at all:

root@OpenWrt:/# stress --cpu 2 --io 2 --timeout 1h
stress: info: [2444] dispatching hogs: 2 cpu, 2 io, 0 vm, 0 hdd
[   55.174519] ------------[ cut here ]------------
[   55.179312] Kernel BUG at do_undefinstr+0x27c/0x290 [verbose debug info unavailable]
[   55.187300] Internal error: Oops - BUG: 0 [#1] SMP
[   55.192237] Modules linked in: pppoe ppp_async iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD pppox ppp_generic nf_nat nf_flow_table nf_conntrack ipt_REJECT xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG slhc rtc_pcf8563 nf_reject_ipv4 nf_log_g
[   55.250897] CPU: 1 PID: 771 Comm: loop0 Not tainted 5.10.64 #0
[   55.256909] Hardware name: Globalscale Marvell ESPRESSOBin Ultra Board (DT)
[   55.264088] pstate: 00400005 (nzcv daif +PAN -UAO -TCO BTYPE=--)
[   55.270283] pc : do_undefinstr+0x27c/0x290
[   55.274503] lr : do_undefinstr+0x58/0x290
[   55.278633] sp : ffffffc0110839d0
[   55.282045] x29: ffffffc0110839d0 x28: ffffff8001777300 
[   55.287522] x27: 0000000000000000 x26: 00000000000000e0 
[   55.292999] x25: 0000000000000000 x24: ffffffc010b3f000 
[   55.298476] x23: 0000000060400005 x22: ffffffc0107b68e0 
[   55.303952] x21: 00000000009b4000 x20: 0000000000000000 
[   55.309429] x19: ffffffc011083a40 x18: 0000000000000000 
[   55.314905] x17: 0000000000000000 x16: 0000000000000000 
[   55.320382] x15: 0000000000000000 x14: 0000000000000000 
[   55.325858] x13: 00000000000001f9 x12: 0000000000000040 
[   55.331335] x11: ffffff8001000000 x10: 0000000000000005 
[   55.336812] x9 : 0000000000000001 x8 : 00000000830b65da 
[   55.342288] x7 : 0000000000000005 x6 : ffffffc011083a10 
[   55.347765] x5 : 0000000000000000 x4 : ffffffc010a3fd20 
[   55.353242] x3 : 00000000d5300000 x2 : 0000000000000000 
[   55.358719] x1 : ffffffc010b0b0e8 x0 : 0000000060400005 
[   55.364196] Call trace:
[   55.366716]  do_undefinstr+0x27c/0x290
[   55.370581]  el1_undef+0x2c/0x4c
[   55.373904]  el1_sync_handler+0x8c/0xd0
[   55.377855]  el1_sync+0x88/0x140
[   55.381183]  __hyp_text_end+0x5c/0x77c
[   55.385044]  __wait_for_common+0xe4/0x1e4
[   55.389175]  wait_for_completion_io+0x20/0x30
[   55.393667]  submit_bio_wait+0x4c/0x64
[   55.397532]  blkdev_issue_flush+0x74/0x94
[   55.401664]  blkdev_fsync+0x2c/0x4c
[   55.405258]  vfs_fsync+0x3c/0x7c
[   55.408587]  loop_queue_work+0x368/0x97c
[   55.412632]  kthread_worker_fn+0x100/0x1d0
[   55.416852]  loop_kthread_worker_fn+0x20/0x30
[   55.421341]  kthread+0x124/0x12c
[   55.424665]  ret_from_fork+0x10/0x3c
[   55.428352] Code: d5033fdf d51b4220 17ffffcf a9025bf5 (d4210000) 
[   55.434635] ---[ end trace 8fab771838008e64 ]---
[   55.439393] Kernel panic - not syncing: Oops - BUG: Fatal exception
[   55.445855] SMP: stopping secondary CPUs
[   55.449900] Kernel Offset: disabled
[   55.453494] CPU features: 0x0000002,00002008
[   55.457891] Memory Limit: none
[   55.461037] Rebooting in 3 seconds..

[pali]

My guess is that in wtmi firmware is missing some init sequence related to CPU voltage configuration. See function init_avs(): https://github.com/MarvellEmbeddedProcessors/A3700-utils-marvell/blob/master/wtmi/sys_init/avs.c

There is array otp_data[] filled by OTP values from SoC itself, but some bits are not used. And they are non-zero, so has some value, but there is no documentation what they mean... E.g. low 8 bits in otp_data[OTP_DATA_SVC_REV_ID]. Relevant header file: https://github.com/MarvellEmbeddedProcessors/A3700-utils-marvell/blob/master/wtmi/sys_init/avs.h

[pali]

But I have seen samples that use 1.26V as well, and I don't think that the CPUFreq has a way to know this and uses too low voltage for most boards.

CPUFreq driver armada-37xx-cpufreq.c know this, it grabs this value from OTP (but indirectly, it reads it from register which is filled by wtmi code, which fills it from OTP). Driver uses following Marvell algorithm:

• CPU max_freq uses vdd max(OTP, 1000mV)
• CPU max_freq/div1 uses vdd max(OTP-100mV, 1000mV)
• CPU max_freq/div2 uses vdd max(OTP-150mV, 1000mV)
• CPU max_freq/div3 uses vdd max(OTP-150mV, 1000mV)

For max_freq 1200 MHz are: div1=2, div2=4, div3=6; for 1000 MHz are: div1=2, div2=4, div3=5; and for 800 MHz are: div1=2, div2=3, div3=4.

But what is source of above Marvell algorithm and these constants (specially those substracted 100mV and 150mV for div1/2/3) I do not know. I was not able to find this documented neither in Armada 3720 Functional or Hardware specification.

And I suspect that these 100mV and 150mV constants are incorrect too as for CPU with max_freq=1GHz I had to do small adjustment in cpufreq driver.

I was told that Marvell reproduced this issue on their 3720 development board last year and was preparing some fix for it, including documentation/errata update. But I have not seen anything.

So it means that somebody in Marvell must have been aware of this issue and should have know more details about it (or somebody who is not with Marvell anymore as @kostapr wrote).

Also look at Armada 3720 Errata document, there is for a long time documented issue related to 1.2GHz mode.

[pali]

@stefanchulski: Do you need some more tests? Or is above crash confirmation with log from @robimarko enough?