PCIe RAM

From PEL Wiki

I'm keeping a list of Modified Linux Source Files.

Hitech Global Board talks talks about the board we will use if we need to go over the PCIe bus.

The trick here is to fool the Linux kernel to believe that the address space on the PCIe card is RAM early in the boot process. In order for that to happen, some other things need to happen. Bolded Items are done

1. The PCIe card and the bus that it hangs on have to be initialized before the PCI system is. PCIe Initialization
2. The Linux Resources need to be set up correctly
3. The ACPI(Advanced Power Configuration Interface)SRAT(Storage Resource Allocation Table) (node - memory mapping needs to be correct)
4. The Linux kernel needs to be loaded at the correct address Loading Linux

That means we need to set the bridge to respond to the address range (I chose 0x200000000 because it is after my RAM) at the same time the kernel reads the BIOS-supplied memory map. The problem with this is that we need to keep the PCI system from undoing that later in the boot process. It doesn't like PCI peripherals to be mapped into the same address space as RAM.

Right now I'm trying to figure out why the card still gets disabled when the address range is assigned a valid resource. It looks like it's because it is a display controller, and the system only wants one. I'm changing it to look like a memory controller. Card Disabled Page details my search, which ended at the X server. Oh Well, until I get my own card I won't be able to use my kernel in graphical mode.

Now I need to make sure that the ACPI system (specifically the SRAT table) gets set up correctly. It's unfortunate that there isn't a single source for RAM information available to the kernel.

I'm starting in _drivers/acpi/numa.c_, pretending there's an extra memory entry in the SRAT. I need to track down everywhere the information this changes gets used. It'll change the end_of_RAM variables. I'm printing those out.

It looks like end_pfn gets set correctly, but end_pfn_map doesn't. I'll find where it's set and see if it needs help.

In _mm/sparse.c_ memory_present() calls sparse_init()

When I add the extra SRAT entry I get a kernel direct mapping tables up to ffff.8102.1000.0000 PANIC: early exception rip 10 error ffff.ffff.8053.c582 cr2 0 PANIC: early exception rip ffff.ffff.8011.7e81 error 0 cr2 ffff.ffff.ff5f.d023

That looks like paging wasn't set up correctly. Do the mtrrs need to be set up too?

I changed some printk to early_printk

That gave me some extra output before the PANIC.

nodedata_phys 200000000 bootmap start 858942784 pages 66tion type 0x83

• doesn't reach*

setting up node 0 0-1ffff

and nodedata_phys is 11000 instead of 20000000

I added some printks in e820_find_area when I don't add my last memory area, I get nodedata_phys 11000 start 0 end 17fffffff pgdat_size 8192 find_e820_area: a000 in 8000-100000 find_e820_area: 2000 in 0-17fffffff find_e820_area: 30000 in 13000-17fffffff

when it I add it, I get find_e820_area: a000 in 8000-100000 find_e820_area: 2000 in 0-20fffffff find_e820_area: 42000 in 200002000-20fffffff

That actually looks right, so I don't know why nodedata_phys gets set wrong.

I'm wondering if the CONFIG_SPARSE_MEM is to blame. I took that out. That didn't help.

_In particular, accesses from the CPU can only hit in the MMIO address maps if the corresponding CPU memory type is of type IO. For accesses from I/O devices, the lookup is based on address only._

This quote from the AMD BIOS developer's manual makes me think that I haven't set up the MTRR's correctly.

I'm going to go back and move the graphics card back to where it belongs (0xc0000000), and hope that that makes it so that the MTRR's are set up correctly.

That gives me no errors, and top shows that I have 4GB + 256 MB RAM. I need to check by allocating RAM in that range to see if it is indeed working.

I decided to revisit my changes and upgrade the kernel. It wasn't too bad.

1. _diff -urN linux-x.y.z/ linux/ > my-patch_ after _make clean_
2. Download new kernel source
3. rpm -U kernel...src.rpm
4. rpmbuild -bp SOURCES/kernel..spec
5. cp source tree
6. apply patch _patch -p1 < my-patch_

Unable to handle kernel paging request at ffffffffffffffff RIP: <ffffffff802efe0c>{pskb_expand_head+182} PGD 103027 PUD 7c3d067 PMD 0 Oops: 0000 [1] SMP

This is the oops when I try to allocate memory that will start to use the Card.