Sparx5
The Sparx5 family consists of the following SKUs:
-
VSC7546 - SparX-5-64 supports up to 64 Gbps of switch core bandwidth
-
VSC7549 - SparX-5-90 supports up to 90 Gbps of switch core bandwidth
-
VSC7552 - SparX-5-128 supports up to 128 Gbps of switch core bandwidth
-
VSC7556 - SparX-5-160 supports up to 160 Gbps of switch core bandwidth
-
VSC7558 - SparX-5-200 supports up to 200 Gbps of switch core bandwidth
1. Booting Sparx5
This section describes the Sparx5 boot sequence including the relevant HW and SW facilities.
1.1. Overview
The behavior of the Sparx5 CPU system after reset is determined by four strapping pins that are overlaid on GPIO pins. The value of these GPIOs is sampled shortly after releasing reset to the device.
This allows the Sparx5 system to decide the boot mode:
-
Main CPU booting from SPI Interface (CPU is SPI controller)
-
Sub CPU booting from SPI Interface (Sub CPU is SPI controller)
-
SPI device mode, Main CPU and Sub CPU are stopped
-
SPI device mode, MIIM device mode, Main CPU and Sub CPU are stopped
In the first listed mode Sparx5 can boot from an external boot media such as SPI NOR flash and load a boot loader image such as UBoot.
UBoot can then load drivers for other external media such as SPI NAND or eMMC and continue the boot process after having loaded final images from these media.
It is also possible to load all images from SPI NOR flash, but this is not as fast as using SPI NAND or eMMC.
During boot Sparx5s PCIe controller will also be enabled in device mode.
1.2. HW Facilities
This document will focus on the Sparx5 SoCs, but to fully explain the boot process and boot facilities, it is also necessary to consider the flash memories available on the board. The section will briefly describe the boot media supported by the Sparx5, and then describe which of these media is available on the Sparx5 evaluation boards offered by Microchip.
There are currently two Sparx5 evaluation boards available:
-
PCB134 - Providing 20 SPF+ ports with 10Gbps speed.
-
PCB135 - Providing 64 1Gbps Ethernet PHY ports and 4 25Gbps SFP+ ports
The two boards can be mounted with either SPI NAND or eMMC enabled (but not both).
1.3. Sparx5 boot media
To utilize the Main CPU of the Sparx5, the correct boot mode must be selected and a SPI NOR device with a boot image must be available at the SPI NOR memory address found in the memory map below:
Name Start Size ====================== ========== ================= Mirror SPI NOR Flash 0x000000000 0x100000000 (4GB) Mirror SPI NAND Flash 0x100000000 0x100000000 (4GB) SPI NOR Flash 0x400000000 0x100000000 (4GB) SPI NAND Flash 0x500000000 0x100000000 (4GB) Vcore Chip Regs 0x610000000 0x10000000 (256MB) CSR Chip Regs 0x620000000 0x10000000 (256MB) PCIe outbound 0x630000000 0x10000000 (256MB) Sub CPU system 0x640000000 0x10000000 (256MB) DDR Memory 0x800000000 0x200000000 (8GB)
Figure 1: Sparx5 Boot Mode memory map.
The bootloader will switch the Sparx5 from Boot mode to Normal mode which will remap the DDR memory down to address 0.
2. SPI NOR
During the boot process, the boot loader will read configuration from its environment partition in the SPI NOR device.
Based on the configuration it will decide where to find and load the next stage image. The following table shows the offset and name of where the different components are expected to be found when using the SPI NOR device.
MEDIA Partition No Name Size Offset ======= ============ =========== ================== ========== SPI NOR 0 UBoot 0x00100000 (1MB) 0x00000000 SPI NOR 1 Env 0x00040000 (256KB) 0x00100000 SPI NOR 2 Env.bk 0x00040000 (256KB) 0x00140000 SPI NOR 3 linux 0x01400000 (20MB) 0x00180000 SPI NOR 4 linux.bk 0x01400000 (20MB) 0x01580000 SPI NOR 5 rootfs_data 0x02000000 (32MB) 0x01580000
Figure 2: Partitions in the SPI NOR only boot mode.
The above partition table is used in a system that has only a SPI NOR media, so all images will be stored in this partition table.
For a system using a SPI NAND device only the first 3 partitions are needed.
The content of the linux
and linux.bk
partition is the raw binary FIT image
that is used by UBoot when booting from NOR only. This mode is explained later
in this document.
3. SPI NAND
A SPI NAND device uses a block design which does not allow it to be used as an execute-in-place device.
Instead the bootloader will read the content of the device in blocks and execute the image from DDR memory once the whole image has been loaded.
Here is the partition table for the Sparx5 SPI NAND device.
MEDIA Partition No Name Size Offset ======= ============ =========== ================== ========== SPI NAND 0 Boot0 0x04000000 (64MB) 0x00000000 SPI NAND 1 Boot1 0x04000000 (64MB) 0x04000000 SPI NAND 2 rootfs_data 0x08000000 (128MB) 0x08000000
Figure 3: SPI NAND Partitions
The Boot0
and Boot1
partitions contain a UBI filesystem with a FIT image file
in the root of the filesystem.
The UBI filesystem also contains the Linux root filesystem that the kernel mounts during boot. The BSP contains a root filesystem that can be used for this purpose.
The rootfs_data
partition can be used for storage when the Linux Kernel have
booted.
The partitions of both the SPI NOR and the SPI NAND devices can be shown with
the mtd list
command in the UBoot shell.
=> mtd list List of MTD devices: * nor0 - type: NOR flash - block size: 0x1000 bytes - min I/O: 0x1 bytes - 0x000000000000-0x000008000000 : "nor0" - 0x000000000000-0x000000100000 : "UBoot" - 0x000000100000-0x000000140000 : "Env" - 0x000000140000-0x000000180000 : "Env.bk" * spi-nand0 - device: spi-nand@e - parent: spi-master@600104000 - driver: spi_nand - type: NAND flash - block size: 0x20000 bytes - min I/O: 0x800 bytes - OOB size: 128 bytes - OOB available: 62 bytes - 0x000000000000-0x000010000000 : "spi-nand0" - 0x000000000000-0x000004000000 : "Boot0" - 0x000004000000-0x000008000000 : "Boot1" - 0x000008000000-0x000010000000 : "rootfs_data" =>
Figure 4: SPI Device Partitions
4. eMMC
An eMMC device is essentially a faster version of a SPI NAND device. It may use more that one data line and may also provide DMA functionality to increase the transfer rate compared to a SPI NAND type of device.
The partitions of the eMMC device can be inspected with the mmc
tool:
=> mmc part Partition Map for MMC device 0 -- Partition Type: EFI Part Start LBA End LBA Name Attributes Type GUID Partition GUID 1 0x00000022 0x00200021 "Boot0" attrs: 0x0000000000000000 type: 0fc63daf-8483-4772-8e79-3d69d8477de4 type: linux guid: 0e036503-0cc3-46c3-9790-1e65b9589954 2 0x00200022 0x00400021 "Boot1" attrs: 0x0000000000000000 type: 0fc63daf-8483-4772-8e79-3d69d8477de4 type: linux guid: 758a6c00-643e-4eb5-9561-f226556ade32 3 0x00400022 0x00700021 "Data" attrs: 0x0000000000000000 type: 0fc63daf-8483-4772-8e79-3d69d8477de4 type: linux guid: 1be9268f-39a3-4112-929a-04fc915b9b87
Figure 5: eMMC Device Partitions
There are 3 partition in a GPT (GUID Partition Table) and all 3 are linux
type
which means that the type is recognizable by the Linux Kernel’s filesystem
drivers and in this case this means that the EXT4 filesystem is used in these
partitions.
The Boot0
(and Boot1
) partitions will need to contain a EXT4 filesystem image
containing a root filesystem and a UBoot FIT image.
-
The UBoot FIT image must called
Image.itb
and must be placed in the root -
folder of the filesystem. The FIT image must contain a Linux kernel and device trees.
The Data partition can be used for storage when the Linux Kernel have booted.
5. Sparx5 Reference Boards
The next sections describes the different Sparx5 Reference Boards.
All examples assume that there is a working TFTP and DHCP server on the same subnet as the evaluation board.
6. Bootstrapping a SPI NAND Evaluation Board
The SPI NOR MX66L1G45G flash can be programmed with a standard SPI flash programmer.
Find the appropriate u-boot-fireant_pcb*.bin UBoot image file in the
arm64-armv8_a-linux-gnu/bootloaders/release
BSP folder.
6.1. UBoot environment for SPI NAND boot
Reset the board and eventually press any key to stop autoboot in order to get a U-Boot prompt.
First make sure that the following environment variables are defined as shown:
=> env print baudrate=115200 bootargs_extra=loglevel=4 bootcmd=run nand_boot bootdelay=3 console=ttyS0,115200n8 fdtcontroladdr=77ffd0310 loadaddr=740000000 mtdids=nor0=spi0.0,spi-nand0=spi0.14 mtdparts=mtdparts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk);spi0.14:64m(Boot0),64m(Boot1),-(rootfs_data) nand_bak=1 nand_boot=run nand_tryboot;env set nand_cur ${nand_bak};run nand_tryboot nand_cur=0 nand_dlup=dhcp ${nand_image};run nand_update nand_image=new.ubifs nand_mtdroot=root=ubi0:rootfs ro rootfstype=ubifs nand_swap=env set nand_cur ${nand_bak}; env save nand_tryboot=run nandload;setenv mtdroot ubi.mtd=Boot${nand_cur},2048 ${nand_mtdroot};run ramboot nand_update=sf probe;mtd erase Boot${nand_cur};ubi part Boot${nand_cur};ubi create rootfs -;ubi write ${fileaddr} rootfs ${filesize} nandload=sf probe;ubi part Boot${nand_cur};ubifsmount ubi0:rootfs;ubifsload - /Image.itb nor_boot=sf probe; env set active linux; run nor_tryboot; env set active linux.bk; run nor_tryboot nor_dlup=dhcp ${nor_image}; run nor_update nor_image=new.itb nor_only=env set mtdparts mtdparts=${nor_parts};env set bootcmd run nor_boot;env save nor_parts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk),20m(linux),20m(linux.bk),32m(rootfs_data) nor_tryboot=mtd read ${active} ${loadaddr}; run ramboot nor_update=sf probe; sf update ${fileaddr} linux ${filesize} pcb=pcb134 ramboot=run setup; bootm #${pcb} rootargs=root=/dev/ram0 rw rootfstype=squashfs setup=setenv bootargs console=${console} ${mtdparts} ${rootargs} ${mtdroot} fis_act=${active} ${bootargs_extra} stderr=serial@600100000 stdin=serial@600100000 stdout=serial@600100000 ubupdate=sf probe; sf update ${fileaddr} 0 ${filesize} ver=U-Boot 2019.10 (Mar 09 2022 - 12:41:04 +0100)fireant Environment size: 1705/8187 bytes
Figure 6: SPI NAND UBoot default environment
Some of these environment variables are command macros that can be used to simplify the boot process.
If there is a difference you should reset the environment, save it and restart the board to use the default environment.
This can be done with these 3 commands:
=> env default -a -f ... => saveenv ... => reset ... Hit any key to stop autoboot: 0 =>
6.2. Boot via Network
During development, it is often advantageous to load and execute the image via DHCP instead of loading it from flash. The ramboot setting above supports that, if you have a FIT image on the TFTP server.
Here are the commands to run to load a fit-sparx5.itb
file from the root of the
TFTP server:
=> dhcp fit-sparx5.itb NPI Port: Up ... => run ramboot ## Loading kernel from FIT Image at 740000000 ... Using 'pcb134' configuration ...
When this works satisfactorily you can commit this to the SPI NAND memory and update UBoot to use this media for booting.
6.3. Setting up SPI NAND boot
As UBoot expects a UBIFS image in the SPI NAND partition Boot0 (or Boot1) you need to produce a UBIFS image file and place it in your TFTP server.
If you have the Linux kernel, the device tree binary files and the rootfile system file then you can create a UBIFS image file for a PCB134 evaluation board by running the imggen.rb script like this:
$ external/support/scripts/imggen.rb -s sparx5 -o /tftpboot/ -k mscc-linux-kernel.bin.gz -d sparx5_pcb134_switchdev.dtb,pcb134 -r rootfs.squashfs -t ubifs-itb-bare -n new
Note that the device tree configuration is renamed by adding a ,pcb134
element
to the device tree binary file references.
Also note that the resulting file is generated in the /tftproot folder (used by the
TFTP server in this case) and has the name new
so that the folder will contain
the file new.ubifs
which is what the UBoot environment macros expect by
default.
Now you can load the image and write it to SPI NAND with this command:
=> run nand_dlup ... Filename 'new.ubifs'. ... Bytes transferred = 30601216 (1d2f000 hex) ... ubi0: attached mtd5 (name "Boot0", size 64 MiB) ... Creating dynamic volume rootfs of size 59170816 30601216 bytes written to volume rootfs =>
With the image in Boot0 time has now come to boot this image. This involve loading the Image.itb file from the UBIFS image and booting this as a FIT image containing a Linux kernel and a device tree.
Note that the UBoot environment expects a device tree configuration with the
name found in the pcb
environment variable. This was the reason for adding
the ,pcb134
to the imggen.rb scripts device tree reference.
As mentioned above the UBoot FIT image does not contain a root filesystem, so the root filesystem is mounted as the content of the Boot0 partition.
Now boot the image with this command:
=> run nand_boot SF: Detected mx66l1g45g with page size 256 Bytes, erase size 4 KiB, total 128 MiB ... ubi0: attached mtd5 (name "Boot0", size 64 MiB) ... Loading file '/Image.itb' to addr 0x740000000... ... Starting kernel ... ... vcoreiii login:
The kernel boots and show the Linux command shell. Now you are ready to use your system.
If this works as expected you can change the UBoot bootcmd
setting to use the
run nand_boot
command permanently.
7. Bootstrapping a eMMC Evaluation Board
The SPI NOR MX66L1G45G flash can be programmed with a standard SPI flash programmer.
Find the appropriate u-boot-fireant_pcb*.bin UBoot image file in the
arm64-armv8_a-linux-gnu/bootloaders/release
BSP folder.
7.1. UBoot environment for eMMC boot
Reset the board and press any key to stop autoboot in order to get a U-Boot prompt.
First make sure that the following environment variables are defined as shown:
=> env print baudrate=115200 bootargs_extra=loglevel=4 bootcmd=run mmc_boot bootdelay=3 console=ttyS0,115200n8 fdtcontroladdr=77ffce2d0 loadaddr=740000000 mmc_bak=2 mmc_boot=run mmc_tryboot;env set mmc_cur ${mmc_bak};run mmc_tryboot mmc_cur=1 mmc_dev=mmc 0 mmc_dlup=dhcp ${mmc_image};unzip ${fileaddr} ${mmcaddr};run mmc_update mmc_format=gpt guid ${mmc_dev} mmc_guid;gpt write ${mmc_dev} ${mmc_part}; env save mmc_image=new.ext4.gz mmc_part=uuid_disk=${mmc_guid};name=Boot0,size=1024MiB,type=linux;name=Boot1,size=1024MiB,type=linux;name=Data,size=1536MiB,type=linux mmc_swap=env set mmc_cur ${mmc_bak}; env save mmc_tryboot=run mmcload;setenv mtdroot root_next=/dev/mmcblk0p${mmc_cur}; run ramboot mmc_update=run mmcgetoffset;mmc write ${mmcaddr} ${mmc_start} ${filesize_512} mmcaddr=760000000 mmcgetoffset=part start ${mmc_dev} ${mmc_cur} mmc_start mmcload=ext4load ${mmc_dev}:${mmc_cur} ${loadaddr} Image.itb mtdids=nor0=spi0.0 mtdparts=mtdparts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk),20m(linux),20m(linux.bk) nor_boot=sf probe; env set active linux; run nor_tryboot; env set active linux.bk; run nor_tryboot nor_dlup=dhcp ${nor_image}; run nor_update nor_image=new.itb nor_only=env set mtdparts mtdparts=${nor_parts};env set bootcmd run nor_boot;env save nor_parts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk),20m(linux),20m(linux.bk),32m(rootfs_data) nor_tryboot=mtd read ${active} ${loadaddr}; run ramboot nor_update=sf probe; sf update ${fileaddr} linux ${filesize} pcb=pcb134_emmc ramboot=run setup; bootm #${pcb} rootargs=root=/dev/ram0 rw rootfstype=squashfs setup=setenv bootargs console=${console} ${mtdparts} ${rootargs} ${mtdroot} fis_act=${active} ${bootargs_extra} stderr=serial@600100000 stdin=serial@600100000 stdout=serial@600100000 ubupdate=sf probe; sf update ${fileaddr} 0 ${filesize} ver=U-Boot 2019.10 (Mar 09 2022 - 12:41:16 +0100)fireant Environment size: 1849/8187 bytes
Figure 7: eMMC UBoot default environment
As in the SPI NAND case some of these environment variables are command macros that can be used to simplify the eMMC boot process.
If you do not see the above environment you have either used the UBoot image when flashing, or you need to reset the environment.
Resetting the UBoot environment is done with the following commands:
=> env default -a -f ... => saveenv ... => reset ... Hit any key to stop autoboot: 0 =>
7.2. Boot via Network
Like described in the SPI NAND boot section of this document it is recommended to do a network boot of your images before writing the image to the eMMC device.
7.3. Setting up eMMC boot
Start by formatting the eMMC device with a GPT partition. You only need to do this once before writing images to the eMMC.
Use the mmc_format command:
m => run mmc_format Writing GPT: success!
This creates a mmc_guid
setting in the UBoot environment for the eMMC device,
that is used in the GUID Partition Table.
As UBoot expects an EXT4 image in the eMCC partition Boot0 (or Boot1) you need to produce a EXT4 image file and place it in your TFTP server.
If you have the Linux kernel, the device tree binary files and the rootfile system file then you can create a EXT4 image file for a PCB134 evaluation board by running the imggen.rb script like this:
$ external/support/scripts/imggen.rb -s sparx5 -o /tftpboot/ -k mscc-linux-kernel.bin.gz -d sparx5_pcb134_emmc_switchdev.dtb,pcb134_emmc -r rootfs.squashfs -t ext4-itb-bare -n new -z
Note that the device tree configuration is renamed by adding a ,pcb134_emmc
element
to the device tree binary file references.
Also note that the resulting file is generated in the /tftproot folder (used by the
TFTP server in this case) and has the name new
so that the folder will contain
the file new.ext4.gz
which is what the UBoot environment macros expect by
default.
Now you can load the image and write it to eMMC with this command:
=> run mmc_dlup ... Filename 'new.ext4.gz'. Load address: 0x740000000 ... MMC write: dev # 0, block # 34, count 213808 ... 213808 blocks written: OK =>
Before you can boot the eMMC disk image, you need to make two small changes in the UBoot environment.
First delete the rootargs
setting
=> setenv rootargs
Next change the mmc_tryboot
environment variable to use the root
argument
instead of the next_root
argument and to wait 1s for the eMMC filesystem to be
mounted.
This is done fairly easy by using the editenv
command like this:
=> editenv mmc_tryboot
When the setting looks like this:
run mmcload;setenv mtdroot root=/dev/mmcblk0p${mmc_cur} rootdelay=1; run ramboot
you can save the changes that you have made permanently with the command:
=> saveenv Saving Environment to SPI Flash... Erasing SPI flash...Writing to SPI flash...done Valid environment: 2 OK =>
You can now boot the system with the run mmc_boot
command.
If this works as expected you can change the bootcmd
setting to use the run
mmc_boot
command permanently.
8. Setting up SPI NOR only boot
No matter if you have a SPI NAND or a eMMC equipped evaluation board you can also select to store the boot image in SPI NOR flash only and use this image as the boot media.
This will not be as fast as booting SPI NAND or eMMC, but can be used as a fallback solution in case the first two options are not available for some other reason.
8.1. UBoot environment for SPI NOR only boot
As the first step you should reset the environment, save it and restart the board to use the initialized environment.
This can be done with these 3 commands:
=> env default -a -f ... => saveenv ... => reset ... Hit any key to stop autoboot: 0 =>
Now switch the to the SPI NOR only partition configuration with this command:
=> run nor_only ... =>
The SPI NOR flash is partitioned like this:
=> mtd list List of MTD devices: * nor0 - type: NOR flash - block size: 0x1000 bytes - min I/O: 0x1 bytes - 0x000000000000-0x000008000000 : "nor0" - 0x000000000000-0x000000100000 : "UBoot" - 0x000000100000-0x000000140000 : "Env" - 0x000000140000-0x000000180000 : "Env.bk" - 0x000000180000-0x000001580000 : "linux" - 0x000001580000-0x000002980000 : "linux.bk" - 0x000002980000-0x000004980000 : "rootfs_data" => boot
As you can see there are two 20MB Linux partitions (a main and a backup) that can be used to store a UBoot FIT image.
The UBoot environment settings that begin with nor_
allows you to store and
load images in these partitions.
8.2. Setting up SPI NOR only boot
When using the SPI NOR boot method the FIT image that you provide must have both the Linux Kernel, device trees and the root file system.
To generate that you can use the imggen.rb tool like this:
$ external/support/scripts/imggen.rb -s sparx5 -o /tftpboot/ -k mscc-linux-kernel.bin.gz -d sparx5_pcb134_emmc_switchdev.dtb,pcb134_emmc -r ws/mscc-brsdk-source-2021.02.7-999/output/build_arm64_standalone/images/rootfs.squashfs -t itb-rootfs -n new
Note that the device tree configuration is renamed by adding a ,pcb134_emmc
element to the device tree binary file references. Here you need to choose the
one that fits with your evaluation board and more specifically the pcb
setting
in the UBoot environment.
Also note that the resulting file is generated in the /tftproot folder (used by the
TFTP server in this case) and has the name new
so that the folder will contain
the file new.itb
which is what the UBoot environment nor_
macros expect by
default.
Now you can load the image and write it to SPI NOR with this command:
=> run nor_dlup ... Filename 'new.itb'. Load address: 0x740000000 ... SF: Detected mx66l1g45g with page size 256 Bytes, erase size 4 KiB, total 128 MiB device 0 offset 0x180000, size 0x11fc663 18859619 bytes written, 0 bytes skipped in 222.855s, speed 86910 B/s =>
You can now boot the system with the run nor_boot
command.
If this works as expected you can change the bootcmd
setting to use the run
nor_boot
command permanently.