Sparx5

The Sparx5 family consists of the following SKUs:

  • VSC7546 - SparX-5-64 supports up to 64 Gbps of switch core bandwidth

  • VSC7549 - SparX-5-90 supports up to 90 Gbps of switch core bandwidth

  • VSC7552 - SparX-5-128 supports up to 128 Gbps of switch core bandwidth

  • VSC7556 - SparX-5-160 supports up to 160 Gbps of switch core bandwidth

  • VSC7558 - SparX-5-200 supports up to 200 Gbps of switch core bandwidth

1. Booting Sparx5

This section describes the Sparx5 boot sequence including the relevant HW and SW facilities.

1.1. Overview

The behavior of the Sparx5 CPU system after reset is determined by four strapping pins that are overlaid on GPIO pins. The value of these GPIOs is sampled shortly after releasing reset to the device.

This allows the Sparx5 system to decide the boot mode:

  • Main CPU booting from SPI Interface (CPU is SPI controller)

  • Sub CPU booting from SPI Interface (Sub CPU is SPI controller)

  • SPI device mode, Main CPU and Sub CPU are stopped

  • SPI device mode, MIIM device mode, Main CPU and Sub CPU are stopped

In the first listed mode Sparx5 can boot from an external boot media such as SPI NOR flash and load a boot loader image such as UBoot.

UBoot can then load drivers for other external media such as SPI NAND or eMMC and continue the boot process after having loaded final images from these media.

It is also possible to load all images from SPI NOR flash, but this is not as fast as using SPI NAND or eMMC.

During boot Sparx5s PCIe controller will also be enabled in device mode.

1.2. HW Facilities

This document will focus on the Sparx5 SoCs, but to fully explain the boot process and boot facilities, it is also necessary to consider the flash memories available on the board. The section will briefly describe the boot media supported by the Sparx5, and then describe which of these media is available on the Sparx5 evaluation boards offered by Microchip.

There are currently two Sparx5 evaluation boards available:

  • PCB134 - Providing 20 SPF+ ports with 10Gbps speed.

  • PCB135 - Providing 64 1Gbps Ethernet PHY ports and 4 25Gbps SFP+ ports

The two boards can be mounted with either SPI NAND or eMMC enabled (but not both).

1.3. Sparx5 boot media

To utilize the Main CPU of the Sparx5, the correct boot mode must be selected and a SPI NOR device with a boot image must be available at the SPI NOR memory address found in the memory map below:

Name                   Start       Size
====================== ==========  =================
Mirror SPI NOR Flash   0x000000000 0x100000000 (4GB)
Mirror SPI NAND Flash  0x100000000 0x100000000 (4GB)
SPI NOR Flash          0x400000000 0x100000000 (4GB)
SPI NAND Flash         0x500000000 0x100000000 (4GB)
Vcore Chip Regs        0x610000000 0x10000000  (256MB)
CSR Chip Regs          0x620000000 0x10000000  (256MB)
PCIe outbound          0x630000000 0x10000000  (256MB)
Sub CPU system         0x640000000 0x10000000  (256MB)
DDR Memory             0x800000000 0x200000000 (8GB)

Figure 1: Sparx5 Boot Mode memory map.

The bootloader will switch the Sparx5 from Boot mode to Normal mode which will remap the DDR memory down to address 0.

2. SPI NOR

During the boot process, the boot loader will read configuration from its environment partition in the SPI NOR device.

Based on the configuration it will decide where to find and load the next stage image. The following table shows the offset and name of where the different components are expected to be found when using the SPI NOR device.

MEDIA    Partition No Name        Size               Offset
=======  ============ =========== ================== ==========
SPI NOR  0            UBoot       0x00100000 (1MB)   0x00000000
SPI NOR  1            Env         0x00040000 (256KB) 0x00100000
SPI NOR  2            Env.bk      0x00040000 (256KB) 0x00140000
SPI NOR  3            linux       0x01400000 (20MB)  0x00180000
SPI NOR  4            linux.bk    0x01400000 (20MB)  0x01580000
SPI NOR  5            rootfs_data 0x02000000 (32MB)  0x01580000

Figure 2: Partitions in the SPI NOR only boot mode.

The above partition table is used in a system that has only a SPI NOR media, so all images will be stored in this partition table.

For a system using a SPI NAND device only the first 3 partitions are needed.

The content of the linux and linux.bk partition is the raw binary FIT image that is used by UBoot when booting from NOR only. This mode is explained later in this document.

3. SPI NAND

A SPI NAND device uses a block design which does not allow it to be used as an execute-in-place device.

Instead the bootloader will read the content of the device in blocks and execute the image from DDR memory once the whole image has been loaded.

Here is the partition table for the Sparx5 SPI NAND device.

MEDIA     Partition No Name        Size               Offset
=======   ============ =========== ================== ==========
SPI NAND  0            Boot0       0x04000000 (64MB)  0x00000000
SPI NAND  1            Boot1       0x04000000 (64MB)  0x04000000
SPI NAND  2            rootfs_data 0x08000000 (128MB) 0x08000000

Figure 3: SPI NAND Partitions

The Boot0 and Boot1 partitions contain a UBI filesystem with a FIT image file in the root of the filesystem.

The UBI filesystem also contains the Linux root filesystem that the kernel mounts during boot. The BSP contains a root filesystem that can be used for this purpose.

The rootfs_data partition can be used for storage when the Linux Kernel have booted.

The partitions of both the SPI NOR and the SPI NAND devices can be shown with the mtd list command in the UBoot shell.

=> mtd list
List of MTD devices:
* nor0
  - type: NOR flash
  - block size: 0x1000 bytes
  - min I/O: 0x1 bytes
  - 0x000000000000-0x000008000000 : "nor0"
          - 0x000000000000-0x000000100000 : "UBoot"
          - 0x000000100000-0x000000140000 : "Env"
          - 0x000000140000-0x000000180000 : "Env.bk"
* spi-nand0
  - device: spi-nand@e
  - parent: spi-master@600104000
  - driver: spi_nand
  - type: NAND flash
  - block size: 0x20000 bytes
  - min I/O: 0x800 bytes
  - OOB size: 128 bytes
  - OOB available: 62 bytes
  - 0x000000000000-0x000010000000 : "spi-nand0"
          - 0x000000000000-0x000004000000 : "Boot0"
          - 0x000004000000-0x000008000000 : "Boot1"
          - 0x000008000000-0x000010000000 : "rootfs_data"
=>

Figure 4: SPI Device Partitions

4. eMMC

An eMMC device is essentially a faster version of a SPI NAND device. It may use more that one data line and may also provide DMA functionality to increase the transfer rate compared to a SPI NAND type of device.

The partitions of the eMMC device can be inspected with the mmc tool:

=> mmc part

Partition Map for MMC device 0  --   Partition Type: EFI

Part    Start LBA       End LBA         Name
        Attributes
        Type GUID
        Partition GUID
  1     0x00000022      0x00200021      "Boot0"
        attrs:  0x0000000000000000
        type:   0fc63daf-8483-4772-8e79-3d69d8477de4
        type:   linux
        guid:   0e036503-0cc3-46c3-9790-1e65b9589954
  2     0x00200022      0x00400021      "Boot1"
        attrs:  0x0000000000000000
        type:   0fc63daf-8483-4772-8e79-3d69d8477de4
        type:   linux
        guid:   758a6c00-643e-4eb5-9561-f226556ade32
  3     0x00400022      0x00700021      "Data"
        attrs:  0x0000000000000000
        type:   0fc63daf-8483-4772-8e79-3d69d8477de4
        type:   linux
        guid:   1be9268f-39a3-4112-929a-04fc915b9b87

Figure 5: eMMC Device Partitions

There are 3 partition in a GPT (GUID Partition Table) and all 3 are linux type which means that the type is recognizable by the Linux Kernel’s filesystem drivers and in this case this means that the EXT4 filesystem is used in these partitions.

The Boot0 (and Boot1) partitions will need to contain a EXT4 filesystem image containing a root filesystem and a UBoot FIT image.

  • The UBoot FIT image must called Image.itb and must be placed in the root

  • folder of the filesystem. The FIT image must contain a Linux kernel and device trees.

The Data partition can be used for storage when the Linux Kernel have booted.

5. Sparx5 Reference Boards

The next sections describes the different Sparx5 Reference Boards.

All examples assume that there is a working TFTP and DHCP server on the same subnet as the evaluation board.

6. Bootstrapping a SPI NAND Evaluation Board

The SPI NOR MX66L1G45G flash can be programmed with a standard SPI flash programmer.

Find the appropriate u-boot-fireant_pcb*.bin UBoot image file in the arm64-armv8_a-linux-gnu/bootloaders/release BSP folder.

6.1. UBoot environment for SPI NAND boot

Reset the board and eventually press any key to stop autoboot in order to get a U-Boot prompt.

First make sure that the following environment variables are defined as shown:

=> env print
baudrate=115200
bootargs_extra=loglevel=4
bootcmd=run nand_boot
bootdelay=3
console=ttyS0,115200n8
fdtcontroladdr=77ffd0310
loadaddr=740000000
mtdids=nor0=spi0.0,spi-nand0=spi0.14
mtdparts=mtdparts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk);spi0.14:64m(Boot0),64m(Boot1),-(rootfs_data)
nand_bak=1
nand_boot=run nand_tryboot;env set nand_cur ${nand_bak};run nand_tryboot
nand_cur=0
nand_dlup=dhcp ${nand_image};run nand_update
nand_image=new.ubifs
nand_mtdroot=root=ubi0:rootfs ro rootfstype=ubifs
nand_swap=env set nand_cur ${nand_bak}; env save
nand_tryboot=run nandload;setenv mtdroot ubi.mtd=Boot${nand_cur},2048 ${nand_mtdroot};run ramboot
nand_update=sf probe;mtd erase Boot${nand_cur};ubi part Boot${nand_cur};ubi create rootfs -;ubi write ${fileaddr} rootfs ${filesize}
nandload=sf probe;ubi part Boot${nand_cur};ubifsmount ubi0:rootfs;ubifsload - /Image.itb
nor_boot=sf probe; env set active linux; run nor_tryboot; env set active linux.bk; run nor_tryboot
nor_dlup=dhcp ${nor_image}; run nor_update
nor_image=new.itb
nor_only=env set mtdparts mtdparts=${nor_parts};env set bootcmd run nor_boot;env save
nor_parts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk),20m(linux),20m(linux.bk),32m(rootfs_data)
nor_tryboot=mtd read ${active} ${loadaddr}; run ramboot
nor_update=sf probe; sf update ${fileaddr} linux ${filesize}
pcb=pcb134
ramboot=run setup; bootm #${pcb}
rootargs=root=/dev/ram0 rw rootfstype=squashfs
setup=setenv bootargs console=${console} ${mtdparts} ${rootargs} ${mtdroot} fis_act=${active} ${bootargs_extra}
stderr=serial@600100000
stdin=serial@600100000
stdout=serial@600100000
ubupdate=sf probe; sf update ${fileaddr} 0 ${filesize}
ver=U-Boot 2019.10 (Mar 09 2022 - 12:41:04 +0100)fireant

Environment size: 1705/8187 bytes

Figure 6: SPI NAND UBoot default environment

Some of these environment variables are command macros that can be used to simplify the boot process.

If there is a difference you should reset the environment, save it and restart the board to use the default environment.

This can be done with these 3 commands:

=> env default -a -f
...
=> saveenv
...
=> reset
...
Hit any key to stop autoboot:  0
=>

6.2. Boot via Network

During development, it is often advantageous to load and execute the image via DHCP instead of loading it from flash. The ramboot setting above supports that, if you have a FIT image on the TFTP server.

Here are the commands to run to load a fit-sparx5.itb file from the root of the TFTP server:

=> dhcp fit-sparx5.itb
NPI Port: Up
...
=> run ramboot
## Loading kernel from FIT Image at 740000000 ...
   Using 'pcb134' configuration
...

When this works satisfactorily you can commit this to the SPI NAND memory and update UBoot to use this media for booting.

6.3. Setting up SPI NAND boot

As UBoot expects a UBIFS image in the SPI NAND partition Boot0 (or Boot1) you need to produce a UBIFS image file and place it in your TFTP server.

If you have the Linux kernel, the device tree binary files and the rootfile system file then you can create a UBIFS image file for a PCB134 evaluation board by running the imggen.rb script like this:

$ external/support/scripts/imggen.rb -s sparx5 -o /tftpboot/ -k mscc-linux-kernel.bin.gz -d sparx5_pcb134_switchdev.dtb,pcb134 -r rootfs.squashfs -t ubifs-itb-bare -n new

Note that the device tree configuration is renamed by adding a ,pcb134 element to the device tree binary file references.

Also note that the resulting file is generated in the /tftproot folder (used by the TFTP server in this case) and has the name new so that the folder will contain the file new.ubifs which is what the UBoot environment macros expect by default.

Now you can load the image and write it to SPI NAND with this command:

=> run nand_dlup
...
Filename 'new.ubifs'.
...
Bytes transferred = 30601216 (1d2f000 hex)
...
ubi0: attached mtd5 (name "Boot0", size 64 MiB)
...
Creating dynamic volume rootfs of size 59170816
30601216 bytes written to volume rootfs
=>

With the image in Boot0 time has now come to boot this image. This involve loading the Image.itb file from the UBIFS image and booting this as a FIT image containing a Linux kernel and a device tree.

Note that the UBoot environment expects a device tree configuration with the name found in the pcb environment variable. This was the reason for adding the ,pcb134 to the imggen.rb scripts device tree reference.

As mentioned above the UBoot FIT image does not contain a root filesystem, so the root filesystem is mounted as the content of the Boot0 partition.

Now boot the image with this command:

=> run nand_boot
SF: Detected mx66l1g45g with page size 256 Bytes, erase size 4 KiB, total 128 MiB
...
ubi0: attached mtd5 (name "Boot0", size 64 MiB)
...
Loading file '/Image.itb' to addr 0x740000000...
...
Starting kernel ...
...
vcoreiii login:

The kernel boots and show the Linux command shell. Now you are ready to use your system.

If this works as expected you can change the UBoot bootcmd setting to use the run nand_boot command permanently.

7. Bootstrapping a eMMC Evaluation Board

The SPI NOR MX66L1G45G flash can be programmed with a standard SPI flash programmer.

Find the appropriate u-boot-fireant_pcb*.bin UBoot image file in the arm64-armv8_a-linux-gnu/bootloaders/release BSP folder.

7.1. UBoot environment for eMMC boot

Reset the board and press any key to stop autoboot in order to get a U-Boot prompt.

First make sure that the following environment variables are defined as shown:

=> env print
baudrate=115200
bootargs_extra=loglevel=4
bootcmd=run mmc_boot
bootdelay=3
console=ttyS0,115200n8
fdtcontroladdr=77ffce2d0
loadaddr=740000000
mmc_bak=2
mmc_boot=run mmc_tryboot;env set mmc_cur ${mmc_bak};run mmc_tryboot
mmc_cur=1
mmc_dev=mmc 0
mmc_dlup=dhcp ${mmc_image};unzip ${fileaddr} ${mmcaddr};run mmc_update
mmc_format=gpt guid ${mmc_dev} mmc_guid;gpt write ${mmc_dev} ${mmc_part}; env save
mmc_image=new.ext4.gz
mmc_part=uuid_disk=${mmc_guid};name=Boot0,size=1024MiB,type=linux;name=Boot1,size=1024MiB,type=linux;name=Data,size=1536MiB,type=linux
mmc_swap=env set mmc_cur ${mmc_bak}; env save
mmc_tryboot=run mmcload;setenv mtdroot root_next=/dev/mmcblk0p${mmc_cur}; run ramboot
mmc_update=run mmcgetoffset;mmc write ${mmcaddr} ${mmc_start} ${filesize_512}
mmcaddr=760000000
mmcgetoffset=part start ${mmc_dev} ${mmc_cur} mmc_start
mmcload=ext4load ${mmc_dev}:${mmc_cur} ${loadaddr} Image.itb
mtdids=nor0=spi0.0
mtdparts=mtdparts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk),20m(linux),20m(linux.bk)
nor_boot=sf probe; env set active linux; run nor_tryboot; env set active linux.bk; run nor_tryboot
nor_dlup=dhcp ${nor_image}; run nor_update
nor_image=new.itb
nor_only=env set mtdparts mtdparts=${nor_parts};env set bootcmd run nor_boot;env save
nor_parts=spi0.0:1m(UBoot),256k(Env),256k(Env.bk),20m(linux),20m(linux.bk),32m(rootfs_data)
nor_tryboot=mtd read ${active} ${loadaddr}; run ramboot
nor_update=sf probe; sf update ${fileaddr} linux ${filesize}
pcb=pcb134_emmc
ramboot=run setup; bootm #${pcb}
rootargs=root=/dev/ram0 rw rootfstype=squashfs
setup=setenv bootargs console=${console} ${mtdparts} ${rootargs} ${mtdroot} fis_act=${active} ${bootargs_extra}
stderr=serial@600100000
stdin=serial@600100000
stdout=serial@600100000
ubupdate=sf probe; sf update ${fileaddr} 0 ${filesize}
ver=U-Boot 2019.10 (Mar 09 2022 - 12:41:16 +0100)fireant

Environment size: 1849/8187 bytes

Figure 7: eMMC UBoot default environment

As in the SPI NAND case some of these environment variables are command macros that can be used to simplify the eMMC boot process.

If you do not see the above environment you have either used the UBoot image when flashing, or you need to reset the environment.

Resetting the UBoot environment is done with the following commands:

=> env default -a -f
...
=> saveenv
...
=> reset
...
Hit any key to stop autoboot:  0
=>

7.2. Boot via Network

Like described in the SPI NAND boot section of this document it is recommended to do a network boot of your images before writing the image to the eMMC device.

7.3. Setting up eMMC boot

Start by formatting the eMMC device with a GPT partition. You only need to do this once before writing images to the eMMC.

Use the mmc_format command:

m => run mmc_format
Writing GPT: success!

This creates a mmc_guid setting in the UBoot environment for the eMMC device, that is used in the GUID Partition Table.

As UBoot expects an EXT4 image in the eMCC partition Boot0 (or Boot1) you need to produce a EXT4 image file and place it in your TFTP server.

If you have the Linux kernel, the device tree binary files and the rootfile system file then you can create a EXT4 image file for a PCB134 evaluation board by running the imggen.rb script like this:

$ external/support/scripts/imggen.rb -s sparx5 -o /tftpboot/ -k mscc-linux-kernel.bin.gz -d sparx5_pcb134_emmc_switchdev.dtb,pcb134_emmc -r rootfs.squashfs -t ext4-itb-bare -n new -z

Note that the device tree configuration is renamed by adding a ,pcb134_emmc element to the device tree binary file references.

Also note that the resulting file is generated in the /tftproot folder (used by the TFTP server in this case) and has the name new so that the folder will contain the file new.ext4.gz which is what the UBoot environment macros expect by default.

Now you can load the image and write it to eMMC with this command:

=> run mmc_dlup
...
Filename 'new.ext4.gz'.
Load address: 0x740000000
...
MMC write: dev # 0, block # 34, count 213808 ... 213808 blocks written: OK
=>

Before you can boot the eMMC disk image, you need to make two small changes in the UBoot environment.

First delete the rootargs setting

=> setenv rootargs

Next change the mmc_tryboot environment variable to use the root argument instead of the next_root argument and to wait 1s for the eMMC filesystem to be mounted.

This is done fairly easy by using the editenv command like this:

=> editenv mmc_tryboot

When the setting looks like this:

run mmcload;setenv mtdroot root=/dev/mmcblk0p${mmc_cur} rootdelay=1; run ramboot

you can save the changes that you have made permanently with the command:

=> saveenv
Saving Environment to SPI Flash... Erasing SPI flash...Writing to SPI flash...done
Valid environment: 2
OK
=>

You can now boot the system with the run mmc_boot command.

If this works as expected you can change the bootcmd setting to use the run mmc_boot command permanently.

8. Setting up SPI NOR only boot

No matter if you have a SPI NAND or a eMMC equipped evaluation board you can also select to store the boot image in SPI NOR flash only and use this image as the boot media.

This will not be as fast as booting SPI NAND or eMMC, but can be used as a fallback solution in case the first two options are not available for some other reason.

8.1. UBoot environment for SPI NOR only boot

As the first step you should reset the environment, save it and restart the board to use the initialized environment.

This can be done with these 3 commands:

=> env default -a -f
...
=> saveenv
...
=> reset
...
Hit any key to stop autoboot:  0
=>

Now switch the to the SPI NOR only partition configuration with this command:

=> run nor_only
...
=>

The SPI NOR flash is partitioned like this:

=> mtd list
List of MTD devices:
* nor0
  - type: NOR flash
  - block size: 0x1000 bytes
  - min I/O: 0x1 bytes
  - 0x000000000000-0x000008000000 : "nor0"
          - 0x000000000000-0x000000100000 : "UBoot"
          - 0x000000100000-0x000000140000 : "Env"
          - 0x000000140000-0x000000180000 : "Env.bk"
          - 0x000000180000-0x000001580000 : "linux"
          - 0x000001580000-0x000002980000 : "linux.bk"
          - 0x000002980000-0x000004980000 : "rootfs_data"
=> boot

As you can see there are two 20MB Linux partitions (a main and a backup) that can be used to store a UBoot FIT image.

The UBoot environment settings that begin with nor_ allows you to store and load images in these partitions.

8.2. Setting up SPI NOR only boot

When using the SPI NOR boot method the FIT image that you provide must have both the Linux Kernel, device trees and the root file system.

To generate that you can use the imggen.rb tool like this:

$ external/support/scripts/imggen.rb -s sparx5 -o /tftpboot/ -k mscc-linux-kernel.bin.gz -d sparx5_pcb134_emmc_switchdev.dtb,pcb134_emmc -r ws/mscc-brsdk-source-2021.02.7-999/output/build_arm64_standalone/images/rootfs.squashfs -t itb-rootfs -n new

Note that the device tree configuration is renamed by adding a ,pcb134_emmc element to the device tree binary file references. Here you need to choose the one that fits with your evaluation board and more specifically the pcb setting in the UBoot environment.

Also note that the resulting file is generated in the /tftproot folder (used by the TFTP server in this case) and has the name new so that the folder will contain the file new.itb which is what the UBoot environment nor_ macros expect by default.

Now you can load the image and write it to SPI NOR with this command:

=> run nor_dlup
...
Filename 'new.itb'.
Load address: 0x740000000
...
SF: Detected mx66l1g45g with page size 256 Bytes, erase size 4 KiB, total 128 MiB
device 0 offset 0x180000, size 0x11fc663
18859619 bytes written, 0 bytes skipped in 222.855s, speed 86910 B/s
=>

You can now boot the system with the run nor_boot command.

If this works as expected you can change the bootcmd setting to use the run nor_boot command permanently.