Problem on AT91SAM9G25 - randomly is not rebooting

This forum is for users of Atmel's SAM Series and who are interested in using Linux OS.

Moderator: nferre

misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Problem on AT91SAM9G25 - randomly is not rebooting

Fri May 26, 2017 10:45 am

Hi all,
I have custom board with AT91SAM9G25. Previously I used kernel 3.13.5 with no reboot problems (but problems with serial line errors). After update to 4.1.0 serial line (and all other) was OK, excepting problems with software reboot. Sometimes (1x of 15-50 tries) board hangs instead of reboot.
I have added debug printk to /drivers/power/reset/at91-reset.c

Code: Select all

static int at91sam9g45_restart(struct notifier_block *this, unsigned long mode,
             void *cmd)
{
  printk("at91sam9g45_restart\n");//MM
  asm volatile(
    /*
     * Test wether we have a second RAM controller to care
     * about.
     *
     * First, test that we can dereference the virtual address.
     */
    "cmp  %1, #0\n\t"
    "beq  1f\n\t"

    /* Then, test that the RAM controller is enabled */
    "ldr  r0, [%1]\n\t"
    "cmp  r0, #0\n\t"

    /* Align to cache lines */
    ".balign 32\n\t"

    /* Disable SDRAM0 accesses */
    "1: str %3, [%0, #" __stringify(AT91_DDRSDRC_RTR) "]\n\t"
    /* Power down SDRAM0 */
    " str %4, [%0, #" __stringify(AT91_DDRSDRC_LPR) "]\n\t"
    /* Disable SDRAM1 accesses */
    " strne %3, [%1, #" __stringify(AT91_DDRSDRC_RTR) "]\n\t"
    /* Power down SDRAM1 */
    " strne %4, [%1, #" __stringify(AT91_DDRSDRC_LPR) "]\n\t"
    /* Reset CPU */
    " str %5, [%2, #" __stringify(AT91_RSTC_CR) "]\n\t"

    " b .\n\t"
    :
    : "r" (at91_ramc_base[0]),
      "r" (at91_ramc_base[1]),
      "r" (at91_rstc_base),
      "r" (1),
      "r" cpu_to_le32(AT91_DDRSDRC_LPCB_POWER_DOWN),
      "r" cpu_to_le32(AT91_RSTC_KEY | AT91_RSTC_PERRST | AT91_RSTC_PROCRST)
    : "r0");

  printk("at91sam9g45_restart asm done \n");//MM
  return NOTIFY_DONE;
}
and only first printk line prints (it is normal),but no other printout from ROMboot, like in the normal response if reboot works.
Yesterday I have tried also actual 4.4.68 kernel from https://github.com/linux4sam/linux-at91.git
I did not added full support to our board, only partial, but tried sw reboots, and also this kernel hangs in the same point. So I would prefer to fix my 4.1.0 version...
Is possible to do software reset in other way? (I can not use watchdog, because it is disabled in AT91Bootstrap which can be only flashed via SAM-BA and I have many boards in fields - it is problem to upgrade them remotely, if reboot can cause hang...)
I have almost no experiences with gdb, can you help me please, how to debug and fix this problem?
Thanks

successfull reboot outoput to serial line (on kernel 4.1.0 with some additional printks):

Code: Select all

The system is going down NOW!
Sent SIGKILL to all processes
Requesting system reboot
[   28.604000] reboot: Restarting system
[   28.604000] Local irq disable
[   28.604000] smp_send_stop
[   28.604000] do kernel restart (null)
omBOOT.604000] at91sam9g45_restart
 ba_offset = 0xb ...


AT91Bootstrap 3.5.3 (▒t ▒ec  9 10:06:17 CEST 2015)

unsuccessfull reboot outoput to serial line (on kernel 4.1.0 with some additional printks):

Code: Select all

[   26.780000] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
[   28.708000] reboot: Restarting system
[   28.708000] Local irq disable
[   28.708000] smp_send_stop
[   28.708000] do kernel restart (null)
[   28.708000] at91sam9g45_restart
and hang...
blue_z
Location: USA
Posts: 1507
Joined: Thu Apr 19, 2007 10:15 pm

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Wed May 31, 2017 2:13 am

You're rather sloppy/vague in your problem description.
How is this random event of "1x of 15-50 tries" distributed?
Is it just one board, or occurs with every board, or ???

Regards
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Tue Jun 13, 2017 8:56 am

Sorry,
I meant one restart in 15-50 "reboot" commands will fail... It seems to be board indipendent (I have tried it on three different custom boards, on six boards at all). My firmware test script logs-in to board (via ssh), and proceeds 'reboot' command (and measures time between reboots). I have also connected serial console to debug the last kernel outputs. After some number of successfull reboots, system hangs during the last phase of reboot (assembler).
Sorry for may bad English.
Regards
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Wed Jun 14, 2017 2:10 pm

Unfortunatelly, I have problems with assembler code. This is previous -not hanging- kernel (3.13.5) assembler code for at91sam9g45_restart (it was in the .S file):

Code: Select all

      .arm
/*
 * at91_ramc_base is an array void*
 * init at NULL if only one DDR controler is present in or DT
 */
      .globl  at91sam9g45_restart

at91sam9g45_restart:
      ldr r5, =at91_ramc_base   @ preload constants
      ldr r0, [r5]
      ldr r5, [r5, #4]      @ ddr1
      cmp r5, #0
      ldr r4, =at91_rstc_base
      ldr r1, [r4]

      mov r2, #1
      mov r3, #AT91_DDRSDRC_LPCB_POWER_DOWN
      ldr r4, =AT91_RSTC_KEY | AT91_RSTC_PERRST | AT91_RSTC_PROCRST

      .balign 32        @ align to cache line

      strne r2, [r5, #AT91_DDRSDRC_RTR] @ disable DDR1 access
      strne r3, [r5, #AT91_DDRSDRC_LPR] @ power down DDR1
      str r2, [r0, #AT91_DDRSDRC_RTR] @ disable DDR0 access
      str r3, [r0, #AT91_DDRSDRC_LPR] @ power down DDR0
      str r4, [r1, #AT91_RSTC_CR]   @ reset processor

      b .
I can see that there is swapped disabling DDR0 and DDR1 memories and little bit different comparison at the begginning... Can somebody compare it with my first post and try to make a suggestion how to modify "c-like" assembler in the first post according to this assemmbler, to try if it is solution?
If it is possible, please post this "c-like" code here (I am affraid, that I am not able to modify it by myself).
Thanks.
Martin
tsvoipio
Posts: 53
Joined: Wed Aug 19, 2015 9:44 pm

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Wed Jun 14, 2017 7:03 pm

misarm wrote:Unfortunatelly, I have problems with assembler code. This is previous -not hanging- kernel (3.13.5) assembler code for at91sam9g45_restart (it was in the .S file):

Code: Select all

      .arm
/*
 * at91_ramc_base is an array void*
 * init at NULL if only one DDR controler is present in or DT
 */
      .globl  at91sam9g45_restart

at91sam9g45_restart:
      ldr r5, =at91_ramc_base   @ preload constants
      ldr r0, [r5]
      ldr r5, [r5, #4]      @ ddr1
      cmp r5, #0
      ldr r4, =at91_rstc_base
      ldr r1, [r4]

      mov r2, #1
      mov r3, #AT91_DDRSDRC_LPCB_POWER_DOWN
      ldr r4, =AT91_RSTC_KEY | AT91_RSTC_PERRST | AT91_RSTC_PROCRST

      .balign 32        @ align to cache line

      strne r2, [r5, #AT91_DDRSDRC_RTR] @ disable DDR1 access
      strne r3, [r5, #AT91_DDRSDRC_LPR] @ power down DDR1
      str r2, [r0, #AT91_DDRSDRC_RTR] @ disable DDR0 access
      str r3, [r0, #AT91_DDRSDRC_LPR] @ power down DDR0
      str r4, [r1, #AT91_RSTC_CR]   @ reset processor

      b .
I can see that there is swapped disabling DDR0 and DDR1 memories and little bit different comparison at the begginning... Can somebody compare it with my first post and try to make a suggestion how to modify "c-like" assembler in the first post according to this assemmbler, to try if it is solution?
If it is possible, please post this "c-like" code here (I am affraid, that I am not able to modify it by myself).
Thanks.
Martin
At least the .balign directive is suspect. It will create a 0 - 30 byte hole in the code, and in 15 cases of 16 your code will get into never-never land there.

Try this instead:

b aligned
.balign 32
aligned:
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Thu Jun 15, 2017 8:31 am

Thanks tsvoipio,
I have tried to add

Code: Select all

b aligned
.balign 32
aligned:
instead of

Code: Select all

.balign 32
but problem still persists (after 37 successfull attempts system hung and didn't reboot in the same way).
Maybe it is not in the at91sam9g45_restart asm code, but in some other condition/state processor is in, before reboot.
I have unsuccessfully tried to rewrite older kernel's linux-3.13.5/arch/arm/mach-at91/at91sam9g45_reset.S to newer kernel's /drivers/power/reset/at91-reset.c, but i am doing some mistake in asm->'c-like' asm conversion (kernel ends with panic instead of reboot).
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Thu Jun 15, 2017 3:08 pm

I studied manual and I have added AT91_RSTC_EXTRST to AT91_RSTC_CR to see on external reset pin if reset is done (NRESET pin goes low for some time), and in case of reboot fail IS NOT (only in case of successfull reboot).
So I changed end of c asm function in such (dirty) way:

Code: Select all

    /* Reset CPU */
    " str %5, [%2, #" __stringify(AT91_RSTC_CR) "]\n\t"
    " b .\n\t"
to

Code: Select all

    /* Reset CPU */
    "reset:\n\t"
    " str %5, [%2, #" __stringify(AT91_RSTC_CR) "]\n\t"
    " b reset\n\t"
    " b .\n\t"
and I have now 200+ successfull reboot tries (i hope it is fixed now - I have no problem till now).
If somebody knows, what could happened,or how to debug it more, please feel free to suggst better solution.
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Mon Jun 19, 2017 8:00 am

After stronger testing, I see, that problem still persists, only is occuring with less probability (one of 35 to 200 tries). I tried to modify asm in several ways, but no luck.
After hang, system can be rebooted by external reset (if I pull it to ground) and I am suspecting, that asm reset instruction is not proceeding.

Code: Select all

str %5, [%2, #" __stringify(AT91_RSTC_CR) "]\n\t"
I don't know, how to continue, to debug and avoid this problem completely...
tsvoipio
Posts: 53
Joined: Wed Aug 19, 2015 9:44 pm

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Mon Jun 19, 2017 2:02 pm

Does anything ensure that there are no interrupts between the RAM switch-off and the reset instruction?

The interrupt handler will end in never-never land if it is attempting to run from the RAM which is switched off.

I just wonder why the DDR is switched off for a reboot.
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Mon Jun 19, 2017 2:24 pm

Thank you,
There is a notice

Code: Select all

* unless the SDRAM is cleanly shutdown before we hit the
* reset register it can be left driving the data bus and
* killing the chance of a subsequent boot from NAND
We are booting from NAND, so I don't want to disable it forever, but I can check if it will be better for now...

I will try also to check interrupts...
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Tue Jun 20, 2017 7:29 am

Thanks,
after skipping RAM shutdown, I had 250+ successfull tries and still no hang, I will continue today.
misarm
Location: EUROPE
Posts: 18
Joined: Wed Oct 03, 2007 9:31 am

Re: Problem on AT91SAM9G25 - randomly is not rebooting

Wed Jun 21, 2017 8:04 am

I have successfully rebooted 1150 times. We don't use shared pins for DDR and NAND, so I hope I can savely disable RAM shutdown.
Thank you (especially tsvoipio). for me is problem solved now.
MM

Return to “Linux”

Who is online

Users browsing this forum: No registered users and 1 guest