Interrupt Priority for the Scheduler

[alexandruradovici]

Pull Request Overview

This pull request adds awareness about interrupt priority to the Chip trait to be used by the scheduler.

The issue

The current implementation of the does not take into account the semantic meaning of the interrupts. For the scheduler trait, interrupts and deferred tasks are grouped into kernel work. The scheduler can only decide to instruct the kernel to perform kernel work, but is not able to instruct it about the order in which this work needs to be performed. This behavior leads to potential interrupt starvation.

For example, on the STM32 boards, take interrupts:

pub const DMA1_Stream0: u32 = 11;
pub const DMA1_Stream1: u32 = 12;
pub const DMA1_Stream2: u32 = 13;
pub const DMA1_Stream3: u32 = 14;
pub const DMA1_Stream4: u32 = 15;
pub const DMA1_Stream5: u32 = 16;
pub const DMA1_Stream6: u32 = 17;

pub const CAN1_TX: u32 = 19;
pub const CAN1_RX0: u32 = 20;
pub const CAN1_RX1: u32 = 21;

As can be observed, all DMA interrupts have numbers lower than the CAN interrupts. The STM32F4 users DMA to read and write to the serial port. Due to the way in which the scheduler handles kernel work all together, as long as there is UART traffic, CAN interrupt handling will be delayed. The kernel referrers to the cortex-m crate for interrupt handling, which always handles the lower interrupts first due to:

let bit = ispr.trailing_zeros();
return Some(block as u32 * 32 + bit);

The CAN interrupt handling works great, as long as there are no debug! messages and no application is writing to the console. As soon as there is any reading or writing to the serial port, CAN interrupts are delayed a few milliseconds, enough to drop packets.

Proposed solution

This PR introduces the concept of interrupts to the scheduler.

As interrupts can be represented in different on different platforms, we add an Interrupts trait to the Chip. This trait represents an interrupt set. The actual implementation of the set is left to the arch crate.

pub trait Interrupts: Copy {
    fn full() -> Self;
    fn empty() -> Self;

    fn is_set(&self, interrupt_number: u8) -> bool;
    fn is_empty(&self) -> bool;

    fn set(&mut self, interrupt_number: u8);
    fn clear(&mut self, interrupt_number: u8);
}

The Interrupts trait is added to the Chip trait as an associated type. The service_pending_interrupts receives an interrupt set that it should handle and the has_pending_interrupts returns an interrupt set.

pub trait Chip {
    // ...
    type Interrupts: Interrupts;
    // ...

    /// This function should loop internally until all interrupts in the provided set have been
    /// handled.
    fn service_pending_interrupts(&self, interrupts: Self::Interrupts);

    /// This function returns a set of interrupts that are pending
    fn has_pending_interrupts(&self) -> Self::Interrupts;
}

The

For the cortex-m crate, this is implemented as a bit map composed out of a tuple ([2; u128]), as the cortex-m defines a maximum of 240 interreupts.

#[derive(Copy, Clone)]
#[repr(transparent)]
pub struct CortexMInterrupts([u128; 2]);

The same idea applies to the deferred tasks, except that the task set can be represented as a u32, as there are a maximum of 32 deferred tasks.

The default implementation of the Scheduler trait changes for the unsafe functions:

    unsafe fn execute_kernel_work(&self, chip: &C) {
        chip.service_pending_interrupts(<C as Chip>::Interrupts::full());
        while DeferredCall::has_tasks().is_some() && !chip.has_pending_interrupts().is_empty() {
            DeferredCall::service_next_pending(0xFFFF_FFFF);
        }
    }

    /// implementation, which always prioritizes kernel work, but schedulers
    /// that wish to defer interrupt handling may reimplement it.
    unsafe fn do_kernel_work_now(&self, chip: &C) -> bool {
        !chip.has_pending_interrupts().is_empty() || DeferredCall::has_tasks().is_some()
    }

    /// Ask the scheduler whether to continue trying to execute a process.
    ///
    /// `id` is the identifier of the currently active process.
    unsafe fn continue_process(&self, _id: ProcessId, chip: &C) -> bool {
        chip.has_pending_interrupts().is_empty() && !DeferredCall::has_tasks().is_some()
    }

This allows the scheduler implementation to inform the arch crate about the order in which to handle the interrupts. The arch crate can still execute them in any arbitrary order, just that it is constrained to a masked set. To handle interrupts in a specific order, scheduler can than issue several service_pending_interrupts calls with a set of one single interrupt.

The same applies to deferred tasks.

This allows the scheduler to instruct the kernel about which tasks to handle as kernel work.

Testing Strategy

N/A

TODO or Help Wanted

Feedback

Documentation Updated

• Updated the relevant files in /docs, or no updates are required.

Formatting

• Ran make prepush.

[lschuermann]

Find and replace gone wrong?

[bradjc]

I know this was a quick PR for tockworld to illustrate, but at some point I at least need more description and comments to understand this change.

[alexandruradovici]

I added a detailed description.

[alexandruradovici]

Kind reminder to review and discuss this 😄

[bradjc]

If I'm reading this correctly this PR would allow the scheduler to learn which interrupts are pending, and specify which should run.

What I don't understand is how is the scheduler going to know what interrupt, say, 24 is, and whether it should request that the chip service interrupt 24 instead of, say, 13? I'm guessing that with this PR there would be a very custom scheduler that would know the meanings of each interrupt number.

The tension appears to be that deciding how interrupts should be prioritized is (or may be) a board-level/application-level decision, yet the code for deciding how to prioritize interrupts lives in the very shared arch/ crates. Right now Tock doesn't specify how chips prioritize interrupts, it just so happens that (I believe) all of our arch/ implementations prioritize the lowest number.

I see two or three options (assuming I'm understanding the problem correctly):

1. We do nothing, and instead add another implementation of nvic (or whatever interrupt manager code) such that individual boards can choose an implementation that matches their desired interrupt priority.
2. We change the interface between arch and chip so that the chip gets to decide which interrupt to run, not the arch. Then, maybe, we make the priority the chip uses configurable somehow, so a board can set it. I say maybe because I'm not convinced that there isn't a "right" priority that will work for all use cases.
3. We somehow create an interface for interrupt priority in the actual hardware and use that.

I don't think this has anything to do with scheduling, unless the scheduler needs to be able to say "if pending interrupt is 24, run apps, but if it is 13, service the interrupt". However, I don't think that is the case.

[phil-levis]

NB: #1181

[alexandruradovici]

I think it is still a problem of the scheduler. Here are some examples:

• we have an application that requires low latency, and while it is running it should only be preempted if there are certain interrupts pending (Ethernet or CAN for example)
• an application needs to be scheduled due to an external interrupt (another device asserts a GPIO)

The scheduler would be written in the board crate, so it would know the hardware that it runs on.

I think options 2 and 3 proposed by @bradjc are worth exploring.

[bradjc]

* we have an application that requires low latency, and while it is running it should only be preempted if there are certain interrupts pending (Ethernet or CAN for example)

This is an interesting use case that would directly affect the scheduler. There are two pieces:

1. How is an application marked as "important"? Do we have something specific for this case, or do we perhaps just use credentials and a custom scheduler knows which processes are priority based on some characteristic?
2. How does the scheduler know when an interrupt is important or not?

It seems like the design question is around how general of a mechanism do we want? A rough sketch, from specific to general:

1. The scheduler is given the list of pending interrupt numbers. A scheduler that wants to do this type of priority must know what the interrupt numbers mean, know which processes should be interrupted for which interrupts, and can implement the logic.
2. The scheduler is given whether any "priority" interrupts are pending, and can choose to prioritize specific applications over non-priority interrupts.
3. Interrupts are divided into multiple priorities, and the scheduler knows which priorities are pending. Each process is tagged with a priority, and a general scheduler can choose interrupts and processes based on priority.

* an application needs to be scheduled due to an external interrupt (another device asserts a GPIO)

This should happen if the app subscribes to that interrupt, right?

[hudson-ayers]

Even with this change, Tock cannot guarantee interrupt latencies because the kernel is non-preemptible: if the kernel thread is already executing some longer-running task when a CAN interrupt arrives, the CAN interrupt will not be serviced until the kernel task finishes. I understand that this PR would improve the status quo of interrupt latencies by a lot, but I do wonder how many real systems that require short interrupt handling latencies can actually be ok with that limitation. @alexandruradovici do you anticipate that the systems you care about are ok with that limitation, or are you looking at that as "the next problem to solve"?

I am trying to figure out whether this is the first step down a path that will ultimately require a preemptible kernel to be useful, because if so I think we need to have that conversation first.