openswitch-lan969x Part 2: Inital Software Bringup

The lan969x requires the following main components to boot:

1. DDR config
2. U-Boot
3. Arm Trusted Firmware
4. The Linux Kernel
5. A Root Filesystem

At the end of this part the board should be able to boot a linux distro, although, with only the minimum features needed to be able to ssh into the board over the managment port.

In the next part I will add support for loading device tree overlays from the carrier eeprom, to enable features spesific to each carrier board, such as the 10g sfp+ ethernet ports on this one.

DDR Config

The lan969x platform uses the DesignWare UMcTL2 DDR controller IP. Microchip has a config tool to generate a device tree snipped to configure the controller for a paticular board.

The ddr controller is initalized by ATF in plat/microchip/lan969x/common/ddr_umctl.c using the config parameters given in the devicetree.

To generate the config first clone the ddr-umctl repo: git clone git@github.com:microchip-ung/ddr-umctl.git

Three config files need to be created in the repo:

1. configs/profiles/openswitch_lan969x.yaml for the top level config which specifies which ddr chip config and which pcb spesific config files to use
2. configs/boards/openswitch_lan969x.yaml for board specific config, such as odt impedance, memory size, clock speed, etc
3. configs/memory/ddr4_512Mx16_fullbus_2B.yaml for ddr chip specific config (just address map configuration). I am just using this default file because it is already compatable with the ddr4 chip I am using.

for now I am just copying the contents of the board and profile files used by the eval board, and after testing I may need to adjust odt and drive impedance because by ddr layout is very different to the eval board.

After creating the config files, generate the dt config:

./scripts/gen_cfg.rb -f devicetree configs/profiles/openswitch_lan969x.yaml

Later this will be copied into the arm trusted firmware repo.

Porting U-Boot

Mainline u-boot does not support the lan969x yet, so I will use microchips u-boot fork: git clone git@github.com:microchip-ung/u-boot.git u-boot-ung

set the arch and cross compiler path: export ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

There are 3 main components that need to be created:

1. a defconfig file configs/qlab_lan969x_openswitch_defconfig which specifies which components to build and a few hardcoded parameters.
2. board/q-lab/lan969x_openswitch/* for board init files. the eval board does the following here: i) read a gpio pin to decide which device tree to load ii) check if we booted from emmc or spi nor flash to load the u-boot enviroment from the same source iii) setup the ddr memory map iv) reset the ethernet phys and read the mac addresses for each port from otp
3. the device tree for the board arch/arm/dts/lan969x_openswitch.dts. ideally this should be the same dt as used in linux, but sometimes u-boot drivers expect different properties.

Defconfig

I just copied over the eval board defconfig, and then went though the file and adjusted the config options to fit the board. for example updating CONFIG_DEFAULT_DEVICE_TREE to lan969x_openswitch and removing the dt selection components, since I only need one dt, etc.

I usually start be reading through the file to see what options have been selected, and then I run memuconfig to change them.

make qlab_lan969x_openswitch_defconfig

make menuconfig

after making changes select save in menuconfig, and save to .config, then exit and run make savedefconfig to create a cleaned version of .config, and then copy it back to your defconfig: mv defconfig configs/qlab_lan969x_openswitch_defconfig

Devicetree

I started out by copying the eval board dt arch/arm/dts/lan969x_ev09p11a.dts into a new file arch/arm/dts/lan969x_openswitch.dts and then adding the file into the makefile in arch/arm/dts so it would be built.

I only had to make a few minor changes to the dt for now: setting the model property, updating the managment port phy address, and changing the qspi flash config a bit.

Board files

again, based on the eval board code, except I removed all the code for reading the otp fues, and removed the dt selection logic.

I created a new config target: CONFIG_TARGET_OPENSWITCH_LAN969X based on the CONFIG_TARGET_LAN969X kconfig option in arch/arm/Kconfig and then updated my defconfig to select this new target and not the old one.

To get the board code in board/qlab/openswitch_lan969x/openswitch_lan969x.c to build a Kconfig file has to be created in board/qlab/openswitch_lan969x that sets SYS_BOARD and SYS_VENDOR to the names in the board dir path. u-boot uses these varables to decide which board directory to build. The Kconfig file also has to be sourced in arch/arm/Kconfig

Building

# set arch and cross compiler
# no need to do this if already exported in the same terminal session
export ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

# select defconfig you want to build
make qlab_lan969x_openswitch_defconfig

# build
make -j$(nproc)

Porting TF-A

First, clone the repo, and setup microchips docker run tool to be able to build:

# setup microchips docker-run tool
cd ~/Software
git clone git@github.com:microchip-ung/docker-run.git
ln -s ~/Software/docker-run/dr ~/.local/bin/dr

# get microchips tfa fork
git clone git@github.com:microchip-ung/arm-trusted-firmware.git atf-ung

usually you can just use a common tfa image per soc, such as on the imx8mm where there is a single tfa image shared by all imx8mm boards, but for the lan969x tf-a does the ddr controller and phy initalization, which needs pcb spesific info, so each board needs its own tfa image. Of course if your board has identical layout to the eval board you could just reuse the same image, if you don't care about the board name not matching.

I do care, so I created a new build target for my board.

This involved the following changes:

1. I copied over the ddr config I generated using ddr-umctl to plat/microchip/lan969x/fdts/lan969x_openswitch-ddr.dtsi
2. A platform.mk file in plat/microchip/lan969x/lan969x_openswitch which is just a copy-paste of the eval board platform.mk file
3. a minimal device tree for the board, whcih only contains the emmc and spi flash nodes, so tfa can load the next bootloader stage: plat/microchip/lan969x/lan969x_openswitch/fdts/lan969x_openswitch_tb_fw_config.dts this was just a copy paste of the eval board device tree
4. I had to modify microchips build scripts: build.rb build_platform.rb This is not part of upstream tfa, Microchip just hacked these on to the tfa repo. I had to update them to build the new openswitch platform I added
5. microchip's html5 firmware update tool: scripts/fwu/fwu.js This is another hacked on addition by microchip, It is a very nice tool, but I have no idea why they chose to place it in the tfa repo, but it is used to flash images onto lan969x-based boards over usb. I just had to update a line so I would detect the new build target I created.

To build, first copy over the u-boot image you built, and then build using the dr tool:

cp ../u-boot-ung/u-boot-fit-dtb.bin ./build

dr ./scripts/build.rb -p lan969x_openswitch --release --bl33_blob build/u-boot-fit-dtb.bin

The output image build/lan969x_openswitch/release/nor.gpt.gz can then be flashed using fwu.html

BL2U SPI Flash Troubleshooting

When trying to flash the nor.gpt.gz image using fwu.html I kept getting the error: "spi_nor_erase: Erase ready timeout" digging through the tfa source code I found it was comming from spi_nor.c and is caused by:

1. a sector erase is triggered by calling spi_nor_erase_sector()
2. spi_nor_wait_ready() is then called to wait for the erase to complete
3. this polls the spi flash chips status register for SPI_READY_TIMEOUT_US which is set to 40ms
4. the timeout is exceeded, which returns ETIMEOUT

The SPI flash chip I used has a erase time that is much higher than this (up to 400ms), which was what was causing the timeout:

Updating SPI_READY_TIMEOUT_US (which is defined in spi_nor.c and qspi.c) to 400ms fixed the issue and I was able to flash tfa + u-boot to the spi flash chip and boot the board:

Flashing u-boot and TF-A

Set the VCORE_[3:0] strapping pins to TF-A monitor mode, 9211600 baud, and connect to the board using fwu.html

Next, click download BL2U

Then go to the upload page and upload nor.gpt.gz from tfa, and click upload file

Then goto the write page, select nor flash, and click write flash image

After this completes set the boot mode switches back to spi nor flash mode.

Porting The Linux Kernel

TODO

Building a Root Filesystem

TODO