Linux Kernel Modifications
From PEL Wiki
Contents |
Base Address Register (BAR) Problems
Funny thing: the Linux Kernel doesn't seem to like it that I have 256 GB of the address space allocated for my device. When I boot, it complains:
Cannot allocate resource region 3 of device 0000:05:01.0
then
Device 0000:05:01.0 not available because of resource collisions
I defined DEBUG in pci.h and set the iommu=force kernel parameter, and now instead of disabling my device, it kernel panics with:
Cannot allocate iommu bitmap
It turns out that something is going wrong before this point. After a little digging, it appears that the resource it can't allocate is 00000080-000000bf. That happens to be reserved. I need to figure out where it's pulling those numbers from, and get it to be something like 100000000-4100000000
Turns out that those numbers are the high 32-bits of the address that was assigned. I don't know what happened to them, but that is probably a good clue.
My device should be a bus master, but during LinuxBIOS boot, it is not. I don't know if I need to track that down, but it's good to remember in case of problems in the future.
It looks like LinuxBIOS is doing it right
base1: 0x8000000000 limit1: 0xfcffffffff size: 0x4000000000 align: 38 --snip-- PCI: 00:18.0 1a9 <- [0x8000000000 - 0xbfffffffff] prefmem <node 0 link 1> --snip-- PCI: 05:01.0 18 <- [0x8000000000 - 0xbfffffffff] prefmem64
At least up to that point. I'm still looking.
It turns out that the Linux Kernel couldn't handle the 64-bit BARS if they had a size larger than 4GB. The 32-bit size check returned zero, so they were disabled. YHLU from the LinuxBIOS list fixed it for me! Thanks.
He also gave me a CK804 fix that I need to look through better, because I don't understand what it does.
Unfortunately I decided to move the large BAR to BAR45 instead of BAR23 and now I have the same conflict messages. I need to track that down. Fixed. It was a typo. In my VHDL, I'd set the register to ffffff40 instead of ffffffc0. YHLU helped me track that down by suggesting trying setpci from Linux.
IOMMU
I'd forgotten that YHLU told me to run the SMP kernel if I wanted to enable the iommu. It looks like the iommu is still being disabled with 2.6.18, but I'm still running SMP.
Freeze
For some reason it's freezing up when I try to run the test. I'm not sure if it's trying to do something with my 256 GB of memory space or if there's some problem with interrupts.
Nanosleep timer resolution
It looks like the timer resolution is about 1 ms. I did this experiment because I couldn't see too much difference when I changed the sleep times to slow I/Os.
Slept 10 times 10 nsecs elapsed 0.058414 seconds Slept 10 times 100 nsecs elapsed 0.0559649 seconds Slept 10 times 1000 nsecs elapsed 0.0519829 seconds Slept 10 times 10000 nsecs elapsed 0.05199 seconds Slept 10 times 100000 nsecs elapsed 0.051991 seconds Slept 10 times 1000000 nsecs elapsed 0.079993 seconds Slept 10 times 10000000 nsecs elapsed 0.171977 seconds Slept 10 times 100000000 nsecs elapsed 1.04001 seconds Slept 100 times 10 nsecs elapsed 0.411983 seconds Slept 100 times 100 nsecs elapsed 0.419995 seconds Slept 100 times 1000 nsecs elapsed 0.419987 seconds Slept 100 times 10000 nsecs elapsed 0.419985 seconds Slept 100 times 100000 nsecs elapsed 0.42399 seconds Slept 100 times 1000000 nsecs elapsed 0.807991 seconds Slept 100 times 10000000 nsecs elapsed 1.612 seconds Slept 100 times 100000000 nsecs elapsed 10.4001 seconds Slept 1000 times 10 nsecs elapsed 4.02403 seconds Slept 1000 times 100 nsecs elapsed 4.02004 seconds Slept 1000 times 1000 nsecs elapsed 4.02404 seconds Slept 1000 times 10000 nsecs elapsed 4.02404 seconds Slept 1000 times 100000 nsecs elapsed 4.02404 seconds Slept 1000 times 1000000 nsecs elapsed 4.02804 seconds Slept 1000 times 10000000 nsecs elapsed 12.0042 seconds Slept 1000 times 100000000 nsecs elapsed 104.002 seconds
Categories: Pel | Myles | LinuxKernel
