本编文章的内容主要是分析 boot/recovery 的启动过程,其中的 boot 就是 android 的kernel, 是整个 android 系统的核心。本文的分析是紧接着 aboot_init 的分析内容的,只是因为其重要性才单独成为一章。

以下是前两篇文章回顾:

高通(Qualcomm)LK源码深度分析

高通(Qualcomm)LK源码深度分析(二)

recovery boot & normal boot

recoverynormal 使用的是同一套加载流程,所以放在一起分析。在开始分析加载过程之前,先从aboot_init 进入加载过程前的代码入手:

if (target_is_emmc_boot()){if(emmc_recovery_init())dprintf(ALWAYS,"error in emmc_recovery_init\n");if(target_use_signed_kernel()){if((device.is_unlocked) || (device.is_tampered)){
#ifdef TZ_TAMPER_FUSEset_tamper_fuse_cmd();
#endif
#if USE_PCOM_SECBOOTset_tamper_flag(device.is_tampered);
#endif}}boot_linux_from_mmc();}

由于 msm8916 并没有定义 TZ_TAMPER_FUSE 宏和 USE_PCOM_SECBOOT 宏,所以进入boot_linux_from_mmc 前只有emmc_recovery_init 步骤需要进行。emmc_recovery_init 函数位于target/msm8916/init.c 文件中,只是_emmc_recovery_init 函数的转接层,本身没有任何功能。_emmc_recovery_init 函数位于app/aboot/recovery.c 文件中。_emmc_recovery_init 的代码逻辑并不复杂,但是具有明显的分析,所以依据代码逻辑分块分为以下 2 个部分来分析:

加载 recovery 命令这部分的主要功能是从 emmc 中获取指定的 recovery 预处理命令,为后面的解析和处理提供条件。

int _emmc_recovery_init(void)
{int update_status = 0;struct recovery_message *msg;uint32_t block_size = 0;block_size = mmc_get_device_blocksize(); // get recovery message
  msg = (struct recovery_message *)memalign(CACHE_LINE, block_size);ASSERT(msg);if(emmc_get_recovery_msg(msg)){if(msg)free(msg);return -1;}msg->command[sizeof(msg->command)-1] = '\0'; //Ensure termination
  if (msg->command[0] != 0 && msg->command[0] != 255) {dprintf(INFO,"Recovery command: %d %s\n",sizeof(msg->command), msg->command);} //...
  return 0;
}

整个加载 recovery 命令的过程比较重要的结构是 recovery_message, 用作存储读取到的 recovery 命令,它的结构如下:

/* Recovery Message */
struct recovery_message {char command[32];char status[32];char recovery[1024];
};

通过 emmc_get_recovery_msgmisc 分区读取到 recovery 命令就通过这个结构体向后传递。

处理 recovery 命令这个部分主要的功能是处理一些需要在启动前处理的 recovery 命令:

int _emmc_recovery_init(void)
{
  //..if (!strcmp(msg->command, "boot-recovery")) {boot_into_recovery = 1;}if (!strcmp("update-radio",msg->command)){ /* We're now here due to radio update, so check for update status */int ret = get_boot_info_apps(UPDATE_STATUS, (unsigned int *) &update_status);if(!ret && (update_status & 0x01)){dprintf(INFO,"radio update success\n");strlcpy(msg->status, "OKAY", sizeof(msg->status));}else{dprintf(INFO,"radio update failed\n");strlcpy(msg->status, "failed-update", sizeof(msg->status));}boot_into_recovery = 1;   // Boot in recovery mode}if (!strcmp("reset-device-info",msg->command)){reset_device_info();}if (!strcmp("root-detect",msg->command)){set_device_root();}elsegoto out;// do nothingstrlcpy(msg->command, "", sizeof(msg->command));  // clearing recovery commandemmc_set_recovery_msg(msg); // send recovery messageout:if(msg)free(msg);return 0;
}

boot-recovery这条命令处理逻辑是最简单的,只是将全局变量 boot_into_recovery 设置为 1, 这个变量在后面的加载部分会用到。update-readio这条命令是检查基带升级是否成功,根据状态设置recovery_message.status, 然后设置boot_into_recovery 为 1。reset-device-info根据 abootinit init 部分 的分析,我们知道 deviceinfo 的数据结构,这里是重设device_info.is_tampered 为 0, 并写入emmc 中。root-detect这条命令正好和reset-device-info 相反,这里会设置device_info.is_tampered 为 1, 也就是说device_info.is_tampered 是手机是否root 的标志位。

当以上 4 条命令任意一条执行后,就会清理掉 recovery_message 并重新协会 misc 分区。

需要预处理的 recovery_message 处理完成后,就到了使用 boot_linux_from_mmc 加载系统的部分。boot_linux_from_mmc 函数位于app/aboot/aboot.c 文件中,代码流程较长,同样使用分块的方法来分析:

启动模式 检测 读取 boot_img_hdr 缓存并验证镜像 解压释放 kernelramdisk 解压释放device tree 调用boot_linux 启动系统

启动模式 检测 

启动模式 检测这一部分代码的作用是检测当前的启动将要进入的模式,然后进行相应设置。msm8916 检测以下 3 种启动模式:

boot mode
ffbm(fast factory boot mode)
normal mode
recovery mode

检测部分的代码如下:

int boot_linux_from_mmc(void)
{struct boot_img_hdr *hdr = (void*) buf;struct boot_img_hdr *uhdr;unsigned offset = 0;int rcode;unsigned long long ptn = 0;int index = INVALID_PTN;unsigned char *image_addr = 0;unsigned kernel_actual;unsigned ramdisk_actual;unsigned imagesize_actual;unsigned second_actual = 0;unsigned int dtb_size = 0;unsigned int out_len = 0;unsigned int out_avai_len = 0;unsigned char *out_addr = NULL;uint32_t dtb_offset = 0;unsigned char *kernel_start_addr = NULL;unsigned int kernel_size = 0;int rc;#if DEVICE_TREEstruct dt_table *table;struct dt_entry dt_entry;unsigned dt_table_offset;uint32_t dt_actual;uint32_t dt_hdr_size;unsigned char *best_match_dt_addr = NULL;
#endifstruct kernel64_hdr *kptr = NULL;if (check_format_bit())boot_into_recovery = 1;if (!boot_into_recovery) {memset(ffbm_mode_string, '\0', sizeof(ffbm_mode_string));rcode = get_ffbm(ffbm_mode_string, sizeof(ffbm_mode_string));if (rcode <= 0) {boot_into_ffbm = false;if (rcode < 0)dprintf(CRITICAL,"failed to get ffbm cookie");} elseboot_into_ffbm = true;} elseboot_into_ffbm = false;uhdr = (struct boot_img_hdr *)EMMC_BOOT_IMG_HEADER_ADDR;if (!memcmp(uhdr->magic, BOOT_MAGIC, BOOT_MAGIC_SIZE)) {dprintf(INFO, "Unified boot method!\n");hdr = uhdr;goto unified_boot;}
  //...
}

通过 check_format_bit 检查是否进入 recovery, check_format_bit 的代码比较简单,其代码如下:

static bool check_format_bit()
{bool ret = false;int index;uint64_t offset;struct boot_selection_info *in = NULL;char *buf = NULL;index = partition_get_index("bootselect");if (index == INVALID_PTN){dprintf(INFO, "Unable to locate /bootselect partition\n");return ret;}offset = partition_get_offset(index);if(!offset){dprintf(INFO, "partition /bootselect doesn't exist\n");return ret;}buf = (char *) memalign(CACHE_LINE, ROUNDUP(page_size, CACHE_LINE));ASSERT(buf);if (mmc_read(offset, (uint32_t *)buf, page_size)){dprintf(INFO, "mmc read failure /bootselect %d\n", page_size);free(buf);return ret;}in = (struct boot_selection_info *) buf;if ((in->signature == BOOTSELECT_SIGNATURE) &&(in->version == BOOTSELECT_VERSION)) {if ((in->state_info & BOOTSELECT_FORMAT) &&!(in->state_info & BOOTSELECT_FACTORY))ret = true;} else {dprintf(CRITICAL, "Signature: 0x%08x or version: 0x%08x mismatched of /bootselect\n",in->signature, in->version);ASSERT(0);}free(buf);return ret;
}

check_format_bit 唯一的作用就是读取 bootselect 分区的信息,然后存放到 boot_selection_info 结构体,其结构如下:

/* bootselect partition format structure */
struct boot_selection_info {uint32_t signature;                // Contains value BOOTSELECT_SIGNATURE defined aboveuint32_t version;uint32_t boot_partition_selection; // Decodes which partitions to boot: 0-Windows,1-Androiduint32_t state_info;               // Contains factory and format bit as definded above
};

msm8916boot_selection_info 满足以下条件则或进入 recovery :

if (in->signature == ('B' | ('S' << 8) | ('e' << 16) | ('l' << 24)) &&(in->version == 0x00010001)) {if ((in->state_info & (1 << 31)) &&!(in->state_info & (1 << 30)))boot_in_recovery = true;}

如果满足条件,则设置全局标志位 boot_into_recovery 为 true。

通过 get_ffbm 检查是否进入 ffbm1 模式,get_ffbm 所完成的任务很简单,只是读取misc 分区,并判断内容是否为ffbm- 开头,如果是就将读取到的信息保存到全局变量ffbm_mode_string 中,并且设置全局变量boot_into_ffbm 为 true。 现在会检查内存固定位置0x8F6FF000 是否和 boot.img 的 MAGIC 值"ANDROID!" 相同,如果相同,则直接按照这个内存地址来启动系统,不再从emmc 中读取。

启动模式 检测完成后,除了直接从内存启动的方式以外,其他方式都需要将需要启动的 imageemmc 中读取并加载到内存中。

读取 boot_img_hdr

normalrecoveryimage 在结构上是相同的,所以可以使用同一套流程加载并启动。这一部分的内容就是从emmc 读取boot_img_hdr 结构,这个结构是image 头部结构,包含基础的加载信息。

int boot_linux_from_mmc(void)
{
  //...if (!boot_into_recovery) {index = partition_get_index("boot");ptn = partition_get_offset(index);if(ptn == 0) {dprintf(CRITICAL, "ERROR: No boot partition found\n");return -1;}}else {index = partition_get_index("recovery");ptn = partition_get_offset(index);if(ptn == 0) {dprintf(CRITICAL, "ERROR: No recovery partition found\n");return -1;}} /* Set Lun for boot & recovery partitions */mmc_set_lun(partition_get_lun(index));if (mmc_read(ptn + offset, (uint32_t *) buf, page_size)) {dprintf(CRITICAL, "ERROR: Cannot read boot image header\n");return -1;}if (memcmp(hdr->magic, BOOT_MAGIC, BOOT_MAGIC_SIZE)) {dprintf(CRITICAL, "ERROR: Invalid boot image header\n");return -1;}if (hdr->page_size && (hdr->page_size != page_size)) {if (hdr->page_size > BOOT_IMG_MAX_PAGE_SIZE) {dprintf(CRITICAL, "ERROR: Invalid page size\n");return -1;}page_size = hdr->page_size;page_mask = page_size - 1;}  /* ensure commandline is terminated */hdr->cmdline[BOOT_ARGS_SIZE-1] = 0;kernel_actual  = ROUND_TO_PAGE(hdr->kernel_size,  page_mask);ramdisk_actual = ROUND_TO_PAGE(hdr->ramdisk_size, page_mask);image_addr = (unsigned char *)target_get_scratch_address();#if DEVICE_TREEdt_actual = ROUND_TO_PAGE(hdr->dt_size, page_mask);imagesize_actual = (page_size + kernel_actual + ramdisk_actual + dt_actual);
#elseimagesize_actual = (page_size + kernel_actual + ramdisk_actual);
#endif
  //...
}

根据 启动模式 获取需要读取的分区偏移。其中 normal 存储在 boot 分区,recovery 存储在recovery 分区。

读取 boot_img_hdr, 其结构如下:

struct boot_img_hdr
{unsigned char magic[BOOT_MAGIC_SIZE];unsigned kernel_size;  /* size in bytes */unsigned kernel_addr;  /* physical load addr */unsigned ramdisk_size; /* size in bytes */unsigned ramdisk_addr; /* physical load addr */unsigned second_size;  /* size in bytes */unsigned second_addr;  /* physical load addr */unsigned tags_addr;    /* physical addr for kernel tags */unsigned page_size;    /* flash page size we assume */unsigned dt_size;      /* device_tree in bytes */unsigned unused;    /* future expansion: should be 0 */unsigned char name[BOOT_NAME_SIZE]; /* asciiz product name */unsigned char cmdline[BOOT_ARGS_SIZE];unsigned id[8]; /* timestamp / checksum / sha1 / etc */
};

进行基础的 boot_img_hdr 合法性检查:

bootimghdr.magic 是否等于 “ANDROID!” bootimghdr.pagesize 是否大于 4096

根据 boot_img_hdr 初始化两个重要的变量:

imageaddr这个值是 image 在内存的缓存地址,缓存的地址由 SCRATCH_ADDR 宏指定,这个宏定义在target/msm8916/rules.mk 文件中。在msm8916 平台SCRATCH_ADDR0x90000000 。 imagesizeactual这个值是image 加载到内存所需要的内存大小,在msm8916 平台计算方法如下:imagesizeactual = 分页大小 + kernel 大小 + ramdisk 大小 + 设备树大小

image_addrimagesize_actual 确定后就可以进行缓存 image 并验证的步骤。

缓存并验证镜像 

这一部分代码的作用就是将 imageemmc 加载到内存中的image_addr 位置,并且验证image是否合法。

int boot_linux_from_mmc(void)
{struct boot_img_hdr *hdr = (void*) buf;struct boot_img_hdr *uhdr;unsigned offset = 0;int rcode;unsigned long long ptn = 0;int index = INVALID_PTN;unsigned char *image_addr = 0;unsigned kernel_actual;unsigned ramdisk_actual;unsigned imagesize_actual;unsigned second_actual = 0;unsigned int dtb_size = 0;unsigned int out_len = 0;unsigned int out_avai_len = 0;unsigned char *out_addr = NULL;uint32_t dtb_offset = 0;unsigned char *kernel_start_addr = NULL;unsigned int kernel_size = 0;int rc;#if DEVICE_TREEstruct dt_table *table;struct dt_entry dt_entry;unsigned dt_table_offset;uint32_t dt_actual;uint32_t dt_hdr_size;unsigned char *best_match_dt_addr = NULL;
#endifstruct kernel64_hdr *kptr = NULL; //...#if VERIFIED_BOOTboot_verifier_init();
#endifif (check_aboot_addr_range_overlap((uint32_t) image_addr, imagesize_actual)){dprintf(CRITICAL, "Boot image buffer address overlaps with aboot addresses.\n");return -1;}  /*
   * Update loading flow of bootimage to support compressed/uncompressed
   * bootimage on both 64bit and 32bit platform.
   * 1. Load bootimage from emmc partition onto DDR.
   * 2. Check if bootimage is gzip format. If yes, decompress compressed kernel
   * 3. Check kernel header and update kernel load addr for 64bit and 32bit
   *    platform accordingly.
   * 4. Sanity Check on kernel_addr and ramdisk_addr and copy data.
   */
dprintf(INFO, "Loading (%s) image (%d): start\n",(!boot_into_recovery ? "boot" : "recovery"),imagesize_actual);bs_set_timestamp(BS_KERNEL_LOAD_START);  /* Read image without signature */if (mmc_read(ptn + offset, (void *)image_addr, imagesize_actual)){dprintf(CRITICAL, "ERROR: Cannot read boot image\n");return -1;}dprintf(INFO, "Loading (%s) image (%d): done\n",(!boot_into_recovery ? "boot" : "recovery"),imagesize_actual);bs_set_timestamp(BS_KERNEL_LOAD_DONE);  /* Authenticate Kernel */dprintf(INFO, "use_signed_kernel=%d, is_unlocked=%d, is_tampered=%d.\n",(int) target_use_signed_kernel(),device.is_unlocked,device.is_tampered); /* Change the condition a little bit to include the test framework support.
   * We would never reach this point if device is in fastboot mode, even if we did
   * that means we are in test mode, so execute kernel authentication part for the
   * tests */if((target_use_signed_kernel() && (!device.is_unlocked)) || boot_into_fastboot){offset = imagesize_actual;if (check_aboot_addr_range_overlap((uint32_t)image_addr + offset, page_size)){dprintf(CRITICAL, "Signature read buffer address overlaps with aboot addresses.\n");return -1;}    /* Read signature */if(mmc_read(ptn + offset, (void *)(image_addr + offset), page_size)){dprintf(CRITICAL, "ERROR: Cannot read boot image signature\n");return -1;}verify_signed_bootimg((uint32_t)image_addr, imagesize_actual);   /* The purpose of our test is done here */if (boot_into_fastboot && auth_kernel_img)return 0;} else {second_actual  = ROUND_TO_PAGE(hdr->second_size,  page_mask);#ifdef TZ_SAVE_KERNEL_HASHaboot_save_boot_hash_mmc((uint32_t) image_addr, imagesize_actual);#endif /* TZ_SAVE_KERNEL_HASH */#ifdef MDTP_SUPPORT{
      /* Verify MDTP lock.
       * For boot & recovery partitions, MDTP will use boot_verifier APIs,
       * since verification was skipped in aboot. The signature is not part of the loaded image.
       */mdtp_ext_partition_verification_t ext_partition;ext_partition.partition = boot_into_recovery ? MDTP_PARTITION_RECOVERY : MDTP_PARTITION_BOOT;ext_partition.integrity_state = MDTP_PARTITION_STATE_UNSET;ext_partition.page_size = page_size;ext_partition.image_addr = (uint32)image_addr;ext_partition.image_size = imagesize_actual;ext_partition.sig_avail = FALSE;mdtp_fwlock_verify_lock(&ext_partition);}
#endif /* MDTP_SUPPORT */} //...
}

初始化对 boot/recovery 的验证, boot_verifier_init 的代码如下:

void boot_verifier_init()
{uint32_t boot_state;
  /* Check if device unlock */if(device.is_unlocked){boot_verify_send_event(DEV_UNLOCK);boot_verify_print_state();dprintf(CRITICAL, "Device is unlocked! Skipping verification...\n");return;}else{boot_verify_send_event(BOOT_INIT);}  /* Initialize keystore */boot_state = boot_verify_keystore_init();if(boot_state == YELLOW){boot_verify_print_state();dprintf(CRITICAL, "Keystore verification failed! Continuing anyways...\n");}
}

如果手机已经解锁 bootloader 则不会进行验证,而是将 boot_state 设置为 ORANGE 状态,在 android 中存在以下几种启动状态2:

green yellow orange red

然后会在 boot_verify_keystore_init 函数中读取两个 key, oem key 和 user key:

oem key 会编译到 lk 代码中,其位置在 platform/msm_shared/include/oem_keystore.h 文件中,作用是为了验证 user key。 user key 存储在 keystore 分区中,作用验证 boot.img。

使用 emmc_read 读取 boot/recovery 到指定的内存地址,由于 boot/recovery 的地址在 bootloader 的高地址处,数据往低地址写可能覆盖 bootloader,所以在读取 boot/recovery 之前,会使用check_aboot_addr_range_overlap 检查将加载到内存的 boot/recovery 是否会覆盖到 aboot 的地址。

读取位于 boot/recovery 尾部的签名,并通过 verify_signed_bootimg 来验证签名是否能够匹配,通过这里可以检查出 boot/recovery 是否被修改,和 APK 签名的作用类似。verify_signed_bootimg 中会调用boot_verify_image 来进行签名验证,但是会根据启动模式的不同传入不同的参数:

if(boot_into_recovery){ret = boot_verify_image((unsigned char *)bootimg_addr,bootimg_size, "/recovery");}else{ret = boot_verify_image((unsigned char *)bootimg_addr,bootimg_size, "/boot");}

boot_verify_image 函数位于 platform/msm_shared/boot_verifier.c 文件中,这个函数的主要作用是是将 boot/recovery 的签名数据转化为VERIFIED_BOOT_SIG * 的结构。

/**
 *    AndroidVerifiedBootSignature DEFINITIONS ::=
 *    BEGIN
 *        FormatVersion ::= INTEGER
 *        AlgorithmIdentifier ::=  SEQUENCE { *            algorithm OBJECT IDENTIFIER,
 *            parameters ANY DEFINED BY algorithm OPTIONAL
 *        }
 *        AuthenticatedAttributes ::= SEQUENCE { *            target CHARACTER STRING,
 *            length INTEGER
 *        }
 *        Signature ::= OCTET STRING
 *     END
 */typedef struct auth_attr_st
{ASN1_PRINTABLESTRING *target;ASN1_INTEGER *len;
}AUTH_ATTR;typedef struct verif_boot_sig_st
{ASN1_INTEGER *version;X509 *certificate;X509_ALGOR *algor;AUTH_ATTR *auth_attr;ASN1_OCTET_STRING *sig;
}VERIFIED_BOOT_SIG;

其中的大多数成员都是 openssl 中的类型,在这里不详细叙述。签名转换完成后就通过 verify_image_with_sig 来验证签名,其中比较重要的参数如下:

char* pname, 即将要验证的分区名称 VERIFIEDBOOTSIG *sig, 分区所带的签名 KEYSTORE *ks, 验证所使用的密钥

static bool verify_image_with_sig(unsigned char* img_addr, uint32_t img_size,char *pname, VERIFIED_BOOT_SIG *sig, KEYSTORE *ks)
{bool ret = false;uint32_t len;int shift_bytes;RSA *rsa = NULL;bool keystore_verification = false;int attr = 0;if(!strcmp(pname, "keystore"))keystore_verification = true;  /* Verify target name */if(strncmp((char*)(sig->auth_attr->target->data), pname,sig->auth_attr->target->length) ||(strlen(pname) != (unsigned long) sig->auth_attr->target->length)){dprintf(CRITICAL,"boot_verifier: verification failure due to target name mismatch\n");goto verify_image_with_sig_error;}
  /* Read image size from signature *//* A len = 0xAABBCC (represented by 3 octets) would be stored in
     len->data as 0X00CCBBAA and len->length as 3(octets).     To read len we need to left shift data to number of missing octets and
     then change it to host long
   */len = *((uint32_t*)sig->auth_attr->len->data);shift_bytes = sizeof(uint32_t) - sig->auth_attr->len->length;if(shift_bytes > 0) {len = len << (shift_bytes*8);}len = ntohl(len);  /* Verify image size*/if(len != img_size){dprintf(CRITICAL,"boot_verifier: image length is different. (%d vs %d)\n",len, img_size);goto verify_image_with_sig_error;}  /* append attribute to image */if(!keystore_verification){
    // verifying a non keystore partitionattr = add_attribute_to_img((unsigned char*)(img_addr + img_size),sig->auth_attr);if (img_size > (UINT_MAX - attr)){dprintf(CRITICAL,"Interger overflow detected\n");ASSERT(0);}else img_size += attr;}  /* compare SHA256SUM of image with value in signature */if(ks != NULL)rsa = ks->mykeybag->mykey->key_material;ret = boot_verify_compare_sha256(img_addr, img_size,(unsigned char*)sig->sig->data, rsa);if(!ret){dprintf(CRITICAL,"boot_verifier: Image verification failed.\n");}verify_image_with_sig_error:return ret;
}

整个验证过程分为以下几个部分:

签名对应的分区是否正确,签名中携带的分区信息为以下两个成员:

分区名称:sig->authattr->target->data 名称长度:sig->authattr->target->length

检查 boot/recovery 的大小是否和签名中存储的大小信息相等,大小信息存储在以下两个成员中:

大小信息:sig->authattr->len->data 数据长度:sig->authattr->len->length

这里需要注意的是 data 是按网络字节的顺序存储,例如 len 原值为 0xAABBCC 则 data 中实际存储的值为 0x00CCBBAA。而 len->length 的作用就是指明这个 data 占了多少个字节,所以转换为 unsigned int 的算法如下。

len = *((uint32_t*)sig->auth_attr->len->data);
shift_bytes = sizeof(uint32_t) - sig->auth_attr->len->length;
if(shift_bytes > 0) {len = len << (shift_bytes*8);}
len = ntohl(len);

最后一步就是比对 SHA256 的值是否正确,整个过程如下:

从 keystore 中获取 rsa 公钥,ks->mykeybag->mykey->keymaterial。 使用 rsa 解密签名中携带的 SHA256 值,sig->sig->data。 计算传入的 boot/recovery 的 SHA256 hash 值。 将解密后的 hash 和解密前的 hash 进行对比,如果一致则签名验证通过。

解压释放 kernelramdisk

经过上面部分的加载和验证,需要 lk 启动的 boot/recovery 镜像已经加载到了内存的缓冲区中,但是现在还是完整的一个整体,并没有分开加载。下面的代码就是对每一个部分的代码和数据进行分开加载,然后才能进行系统启动的操作。

int boot_linux_from_mmc(void)
{
  //.../*
   * Check if the kernel image is a gzip package. If yes, need to decompress it.
   * If not, continue booting.
   */if (is_gzip_package((unsigned char *)(image_addr + page_size), hdr->kernel_size)){out_addr = (unsigned char *)(image_addr + imagesize_actual + page_size);out_avai_len = target_get_max_flash_size() - imagesize_actual - page_size;dprintf(INFO, "decompressing kernel image: start\n");rc = decompress((unsigned char *)(image_addr + page_size),hdr->kernel_size, out_addr, out_avai_len,&dtb_offset, &out_len);if (rc){dprintf(CRITICAL, "decompressing kernel image failed!!!\n");ASSERT(0);}dprintf(INFO, "decompressing kernel image: done\n");kptr = (struct kernel64_hdr *)out_addr;kernel_start_addr = out_addr;kernel_size = out_len;} else {kptr = (struct kernel64_hdr *)(image_addr + page_size);kernel_start_addr = (unsigned char *)(image_addr + page_size);kernel_size = hdr->kernel_size;}
/*
   * Update the kernel/ramdisk/tags address if the boot image header
   * has default values, these default values come from mkbootimg when
   * the boot image is flashed using fastboot flash:raw
   */update_ker_tags_rdisk_addr(hdr, IS_ARM64(kptr));  /* Get virtual addresses since the hdr saves physical addresses. */hdr->kernel_addr = VA((addr_t)(hdr->kernel_addr));hdr->ramdisk_addr = VA((addr_t)(hdr->ramdisk_addr));hdr->tags_addr = VA((addr_t)(hdr->tags_addr));kernel_size = ROUND_TO_PAGE(kernel_size,  page_mask);
  /* Check if the addresses in the header are valid. */if (check_aboot_addr_range_overlap(hdr->kernel_addr, kernel_size) ||check_aboot_addr_range_overlap(hdr->ramdisk_addr, ramdisk_actual)){dprintf(CRITICAL, "kernel/ramdisk addresses overlap with aboot addresses.\n");return -1;}#ifndef DEVICE_TREEif (check_aboot_addr_range_overlap(hdr->tags_addr, MAX_TAGS_SIZE)){dprintf(CRITICAL, "Tags addresses overlap with aboot addresses.\n");return -1;}
#endif

  /* Move kernel, ramdisk and device tree to correct address */memmove((void*) hdr->kernel_addr, kernel_start_addr, kernel_size);memmove((void*) hdr->ramdisk_addr, (char *)(image_addr + page_size + kernel_actual), hdr->ramdisk_size);  //...
}

整个加载过程如下:

由于 emmc 存储空间有限,所以有的时候 kernel 是压缩保存在 emmc 中的,所以需要先判断是否需要解压,由于需要解压的情况较多,所以先分析此过程。调用is_gzip_package 检查 kernel block 是否压缩,这个函数在lib/zlib_inflate/decompress.c 文件中,它的实现如下:

/* check if the input "buf" file was a gzip package.
 * Return true if the input "buf" is a gzip package.
 */
int is_gzip_package(unsigned char *buf, unsigned int len)
{if (len < 10 || !buf || buf[0] != 0x1f ||buf[1] != 0x8b || buf[2] != 0x08){return false;}return true;
}

为压缩包的条件非常简单,只有以下两个:

长度不小于 10 MAGIC 为 0x1F8B08

只要满足这两个条件即判定为压缩包

设置解压后数据的存储位置和大小,分别为以下两个值:address = imgbufferaddress + imgsize + pagesizesize = 0×10000000 – (imgsize + pagesize)

调用 decompress 函数解压 kernel, 这个函数实现在 lib/zlib_inflate/decompress.c 文件中:

/* decompress gzip file "in_buf", return 0 if decompressed successful,
 * return -1 if decompressed failed.
 * in_buf - input gzip file
 * in_len - input the length file
 * out_buf - output the decompressed data
 * out_buf_len - the available length of out_buf
 * pos - position of the end of gzip file
 * out_len - the length of decompressed data
 */
int decompress(unsigned char *in_buf, unsigned int in_len,unsigned char *out_buf,unsigned int out_buf_len,unsigned int *pos,unsigned int *out_len) {struct z_stream_s *stream;int rc = -1;int i;if (in_len < GZIP_HEADER_LEN) {dprintf(INFO, "the input data is not a gzip package.\n");return rc;}if (out_buf_len < in_len) {dprintf(INFO, "the avaiable length of out_buf is not enough.\n");return rc;}stream = malloc(sizeof(*stream));if (stream == NULL) {dprintf(INFO, "allocating z_stream failed.\n");return rc;}stream->zalloc = zlib_alloc;stream->zfree = zlib_free;stream->next_out = out_buf;stream->avail_out = out_buf_len;  /* skip over gzip header */stream->next_in = in_buf + GZIP_HEADER_LEN;stream->avail_in = out_buf_len - GZIP_HEADER_LEN;
  /* skip over asciz filename */if (in_buf[3] & 0x8) {for (i = 0; i < GZIP_FILENAME_LIMIT && *stream->next_in++; i++) {if (stream->avail_in == 0) {dprintf(INFO, "header error\n");goto gunzip_end;}--stream->avail_in;}}rc = inflateInit2(stream, -MAX_WBITS);if (rc != Z_OK) {dprintf(INFO, "inflateInit2 failed!\n");goto gunzip_end;}rc = inflate(stream, 0);
  /* Z_STREAM_END is "we unpacked it all" */if (rc == Z_STREAM_END) {rc = 0;} else if (rc != Z_OK) {dprintf(INFO, "uncompression error \n");rc = -1;}inflateEnd(stream);if (pos)
    /* alculation the length of the compressed package */*pos = stream->next_in - in_buf + 8;if (out_len)*out_len = stream->total_out;gunzip_end:free(stream);return rc; /* returns 0 if decompressed successful */
}

这个过程是标准的 gzip 解压,这里就详细分析,都是直接调用 zlib 的接口实现。

保存解压后的 kernel 头地址和大小,这个涉及到一个比较重要的结构体 kernel64_hdr:

struct kernel64_hdr
{uint32_t insn;uint32_t res1;uint64_t text_offset;uint64_t res2;uint64_t res3;uint64_t res4;uint64_t res5;uint64_t res6;uint32_t magic_64;uint32_t res7;
};

这个结构就是 kernel block 的头部结构,定义在 app/aboot/bootimg.h 文件中。

检查 kernel 和 ramdisk 是否会越界覆盖到 bootloader, 同样是通过 check_aboot_addr_range_overlap 完成。 将 kernel 和 ramdisk 拷贝到boot_img_hdr 指定的加载地址中。

解压释放 device tree

接下来就需要加载 device tree 到内存,由于存在两种情况:

boot_img_hdr 中指定了 dtsize 没有指定 dtsize

两种情况分开分析。

指定了 dtsize 如何加载 device tree。

首先需要明确 device tree 在 image 中的位置,其位置计算如下:

dt_table_offset = ((uint32_t)image_addr + page_size + kernel_actual + ramdisk_actual + second_actual);
table = (struct dt_table*) dt_table_offset;

这里涉及到 dttable 结构体,其定义在 platform/msm_shared/include/dev_tree.h 文件中,结构如下:

struct dt_table
{uint32_t magic;uint32_t version;uint32_t num_entries;
};

验证 device tree block 的数据是否合法,调用 dev_tree_validate 函数来确定,其定义在 platform/msm_shared/dev_tree.c 文件中,实现如下:

/* Returns 0 if the device tree is valid. */
int dev_tree_validate(struct dt_table *table, unsigned int page_size, uint32_t *dt_hdr_size)
{int dt_entry_size;uint64_t hdr_size;  /* Validate the device tree table header */if(table->magic != DEV_TREE_MAGIC) {dprintf(CRITICAL, "ERROR: Bad magic in device tree table \n");return -1;}if (table->version == DEV_TREE_VERSION_V1) {dt_entry_size = sizeof(struct dt_entry_v1);} else if (table->version == DEV_TREE_VERSION_V2) {dt_entry_size = sizeof(struct dt_entry_v2);} else if (table->version == DEV_TREE_VERSION_V3) {dt_entry_size = sizeof(struct dt_entry);} else {dprintf(CRITICAL, "ERROR: Unsupported version (%d) in DT table \n",table->version);return -1;}hdr_size = (uint64_t)table->num_entries * dt_entry_size + DEV_TREE_HEADER_SIZE;  /* Roundup to page_size. */hdr_size = ROUNDUP(hdr_size, page_size);if (hdr_size > UINT_MAX)return -1;else*dt_hdr_size = hdr_size & UINT_MAX;return 0;
}

第一点需要验证的就是 MAGIC 是否为正确,正确的 device tree magic 如下:

#define DEV_TREE_MAGIC          0x54444351 /* "QCDT" */

第二步是检查 device tree 格式的版本是否支持,目前的 lk 支持以下 3 个版本:

#define DEV_TREE_VERSION_V1     1
#define DEV_TREE_VERSION_V2     2
#define DEV_TREE_VERSION_V3     3

每个版本对应不同的 dt_entry 结构体,按照上面的版本顺序,分别是以下 3 个结构体:

struct dt_entry_v1
{uint32_t platform_id;uint32_t variant_id;uint32_t soc_rev;uint32_t offset;uint32_t size;
};struct dt_entry_v2
{uint32_t platform_id;uint32_t variant_id;uint32_t board_hw_subtype;uint32_t soc_rev;uint32_t offset;uint32_t size;
};struct dt_entry
{uint32_t platform_id;uint32_t variant_id;uint32_t board_hw_subtype;uint32_t soc_rev;uint32_t pmic_rev[4];uint32_t offset;uint32_t size;
};

计算并验证所需要内存大小是否正确,计算过程如下:

hdr_size = (uint64_t)table->num_entries * dt_entry_size + DEV_TREE_HEADER_SIZE;/* Roundup to page_size. */
hdr_size = ROUNDUP(hdr_size, page_size);if (hdr_size > UINT_MAX)return -1;else*dt_hdr_size = hdr_size & UINT_MAX;

ROUNDUP 实际上就是按照分页对齐,宏定义为 #define ROUNDUP(a, b) (((a) + ((b)-1)) & ~((b)-1))

dt_table 中的 numentries 字段可以知道 device tree 在存储中实际上是数组结构,这里就是遍临构造出这个 device tree 数组。这里调用dev_tree_get_entry_info 来实现,其定义在platform/msm_shared/dev_tree.c 文件中,由于有多个版本的 device tree,通过对比可以发现dt_entry 的字段是不断增加的,所以我们只分析 version 3 这一种情况,其实现如下:

/* Function to obtain the index information for the correct device tree
 *  based on the platform data.
 *  If a matching device tree is found, the information is returned in the
 *  "dt_entry_info" out parameter and a function value of 0 is returned, otherwise
 *  a non-zero function value is returned.
 */
int dev_tree_get_entry_info(struct dt_table *table, struct dt_entry *dt_entry_info)
{uint32_t i;unsigned char *table_ptr = NULL;struct dt_entry dt_entry_buf_1;struct dt_entry *cur_dt_entry = NULL;struct dt_entry *best_match_dt_entry = NULL;struct dt_entry_v1 *dt_entry_v1 = NULL;struct dt_entry_v2 *dt_entry_v2 = NULL;struct dt_entry_node *dt_entry_queue = NULL;struct dt_entry_node *dt_node_tmp1 = NULL;struct dt_entry_node *dt_node_tmp2 = NULL;uint32_t found = 0;if (!dt_entry_info) {dprintf(CRITICAL, "ERROR: Bad parameter passed to %s \n",__func__);return -1;}table_ptr = (unsigned char *)table + DEV_TREE_HEADER_SIZE;cur_dt_entry = &dt_entry_buf_1;best_match_dt_entry = NULL;dt_entry_queue = (struct dt_entry_node *)malloc(sizeof(struct dt_entry_node));if (!dt_entry_queue) {dprintf(CRITICAL, "Out of memory\n");return -1;}list_initialize(&dt_entry_queue->node);dprintf(INFO, "DTB Total entry: %d, DTB version: %d\n", table->num_entries, table->version);for(i = 0; found == 0 && i < table->num_entries; i++){memset(cur_dt_entry, 0, sizeof(struct dt_entry));switch(table->version) {case DEV_TREE_VERSION_V1:
      //...break;case DEV_TREE_VERSION_V2:
      //...break;case DEV_TREE_VERSION_V3:memcpy(cur_dt_entry, (struct dt_entry *)table_ptr,sizeof(struct dt_entry));
      /* For V3 version of DTBs we have platform version field as part
       * of variant ID, in such case the subtype will be mentioned as 0x0
       * As the qcom, board-id = <0xSSPMPmPH, 0x0>
       * SS -- Subtype
       * PM -- Platform major version
       * Pm -- Platform minor version
       * PH -- Platform hardware CDP/MTP
       * In such case to make it compatible with LK algorithm move the subtype
       * from variant_id to subtype field
       */if (cur_dt_entry->board_hw_subtype == 0)cur_dt_entry->board_hw_subtype = (cur_dt_entry->variant_id >> 0x18);table_ptr += sizeof(struct dt_entry);break;default:dprintf(CRITICAL, "ERROR: Unsupported version (%d) in DT table \n",table->version);free(dt_entry_queue);return -1;}    /* DTBs must match the platform_id, platform_hw_id, platform_subtype and DDR size.
    * The satisfactory DTBs are stored in dt_entry_queue
    */platform_dt_absolute_match(cur_dt_entry, dt_entry_queue);}best_match_dt_entry = platform_dt_match_best(dt_entry_queue);if (best_match_dt_entry) {*dt_entry_info = *best_match_dt_entry;found = 1;}if (found != 0) {dprintf(INFO, "Using DTB entry 0x%08x/%08x/0x%08x/%u for device 0x%08x/%08x/0x%08x/%u\n",dt_entry_info->platform_id, dt_entry_info->soc_rev,dt_entry_info->variant_id, dt_entry_info->board_hw_subtype,board_platform_id(), board_soc_version(),board_target_id(), board_hardware_subtype());if (dt_entry_info->pmic_rev[0] == 0 && dt_entry_info->pmic_rev[0] == 0 &&dt_entry_info->pmic_rev[0] == 0 && dt_entry_info->pmic_rev[0] == 0) {dprintf(SPEW, "No maintain pmic info in DTB, device pmic info is 0x%0x/0x%x/0x%x/0x%0x\n",board_pmic_target(0), board_pmic_target(1),board_pmic_target(2), board_pmic_target(3));} else {dprintf(INFO, "Using pmic info 0x%0x/0x%x/0x%x/0x%0x for device 0x%0x/0x%x/0x%x/0x%0x\n",dt_entry_info->pmic_rev[0], dt_entry_info->pmic_rev[1],dt_entry_info->pmic_rev[2], dt_entry_info->pmic_rev[3],board_pmic_target(0), board_pmic_target(1),board_pmic_target(2), board_pmic_target(3));}return 0;}dprintf(CRITICAL, "ERROR: Unable to find suitable device tree for device (%u/0x%08x/0x%08x/%u)\n",board_platform_id(), board_soc_version(),board_target_id(), board_hardware_subtype());list_for_every_entry(&dt_entry_queue->node, dt_node_tmp1, dt_node, node) {
    /* free node memory */dt_node_tmp2 = (struct dt_entry_node *) dt_node_tmp1->node.prev;dt_entry_list_delete(dt_node_tmp1);dt_node_tmp1 = dt_node_tmp2;}free(dt_entry_queue);return -1;
}

首先开始遍历整个数组,每个数组成员是一个 dt_entry 结构体,获取到一个 dt_entry 都保存到cur_dt_entry 变量中,然后调用platform_dt_absolute_match 存储到dt_entry_queue 中,dt_entry_queue 是一个链表结构,node 结构如下:

typedef struct dt_entry_node {struct list_node node;struct dt_entry * dt_entry_m;
}dt_node;

platform_dt_absolute_match 的实现如下:

static int platform_dt_absolute_match(struct dt_entry *cur_dt_entry, struct dt_entry_node *dt_list)
{uint32_t cur_dt_hlos_ddr;uint32_t cur_dt_hw_platform;uint32_t cur_dt_hw_subtype;uint32_t cur_dt_msm_id;dt_node *dt_node_tmp = NULL; /* Platform-id
  * bit no |31   24|23  16|15 0|
  *        |reserved|foundry-id|msm-id|
  */cur_dt_msm_id = (cur_dt_entry->platform_id & 0x0000ffff);cur_dt_hw_platform = (cur_dt_entry->variant_id & 0x000000ff);cur_dt_hw_subtype = (cur_dt_entry->board_hw_subtype & 0xff); /* Determine the bits 10:8 to check the DT with the DDR Size */cur_dt_hlos_ddr = (cur_dt_entry->board_hw_subtype & 0x700);
/* 1. must match the msm_id, platform_hw_id, platform_subtype and DDR size
  *  soc, board major/minor, pmic major/minor must less than board info
  *  2. find the matched DTB then return 1
  *  3. otherwise return 0
  */if((cur_dt_msm_id == (board_platform_id() & 0x0000ffff)) &&(cur_dt_hw_platform == board_hardware_id()) &&(cur_dt_hw_subtype == board_hardware_subtype()) &&(cur_dt_hlos_ddr == (target_get_hlos_subtype() & 0x700)) &&(cur_dt_entry->soc_rev <= board_soc_version()) &&((cur_dt_entry->variant_id & 0x00ffff00) <= (board_target_id() & 0x00ffff00)) &&((cur_dt_entry->pmic_rev[0] & 0x00ffff00) <= (board_pmic_target(0) & 0x00ffff00)) &&((cur_dt_entry->pmic_rev[1] & 0x00ffff00) <= (board_pmic_target(1) & 0x00ffff00)) &&((cur_dt_entry->pmic_rev[2] & 0x00ffff00) <= (board_pmic_target(2) & 0x00ffff00)) &&((cur_dt_entry->pmic_rev[3] & 0x00ffff00) <= (board_pmic_target(3) & 0x00ffff00))) {dt_node_tmp = dt_entry_list_init();memcpy((char*)dt_node_tmp->dt_entry_m,(char*)cur_dt_entry, sizeof(struct dt_entry));dprintf(SPEW, "Add DTB entry %u/%08x/0x%08x/%x/%x/%x/%x/%x/%x/%x\n",dt_node_tmp->dt_entry_m->platform_id, dt_node_tmp->dt_entry_m->variant_id,dt_node_tmp->dt_entry_m->board_hw_subtype, dt_node_tmp->dt_entry_m->soc_rev,dt_node_tmp->dt_entry_m->pmic_rev[0], dt_node_tmp->dt_entry_m->pmic_rev[1],dt_node_tmp->dt_entry_m->pmic_rev[2], dt_node_tmp->dt_entry_m->pmic_rev[3],dt_node_tmp->dt_entry_m->offset, dt_node_tmp->dt_entry_m->size);insert_dt_entry_in_queue(dt_list, dt_node_tmp);return 1;}return 0;
}

这个函数最主要的作用就是将 dt_entry 添加到 dt_entry_queue 链表中,但是需要满足以下所有条件才能加入:

msmid 匹配 platformhwid 匹配 platformsubtype 匹配 ddr size 匹配 soc 版本匹配 board 版本匹配 pmic 版本匹配

通过 platform_dt_match_best 来获取最佳匹配,并且赋值给输出参数 dt_entry_info 中,其实现如下:

static struct dt_entry *platform_dt_match_best(struct dt_entry_node *dt_list)
{struct dt_entry_node *dt_node_tmp1 = NULL;
/* check Foundry id
  * the foundry id must exact match board founddry id, this is compatibility check,
  * if couldn't find the exact match from DTB, will exact match 0x0.
  */if (!platform_dt_absolute_compat_match(dt_list, DTB_FOUNDRY))return NULL; /* check PMIC model
  * the PMIC model must exact match board PMIC model, this is compatibility check,
  * if couldn't find the exact match from DTB, will exact match 0x0.
  */if (!platform_dt_absolute_compat_match(dt_list, DTB_PMIC_MODEL))return NULL; /* check panel type
  * the panel  type must exact match board panel type, this is compatibility check,
  * if couldn't find the exact match from DTB, will exact match 0x0.
  */if (!platform_dt_absolute_compat_match(dt_list, DTB_PANEL_TYPE))return NULL;  /* check boot device subtype
  * the boot device subtype must exact match board boot device subtype, this is compatibility check,
  * if couldn't find the exact match from DTB, will exact match 0x0.
  */if (!platform_dt_absolute_compat_match(dt_list, DTB_BOOT_DEVICE))return NULL;  /* check soc version
  * the suitable soc version must less than or equal to board soc version
  */if (!update_dtb_entry_node(dt_list, DTB_SOC))return NULL; /*check major and minor version
  * the suitable major&minor version must less than or equal to board major&minor version
  */if (!update_dtb_entry_node(dt_list, DTB_MAJOR_MINOR))return NULL;
/*check pmic info
  * the suitable pmic major&minor info must less than or equal to board pmic major&minor version
  */if (!update_dtb_entry_node(dt_list, DTB_PMIC0))return NULL;if (!update_dtb_entry_node(dt_list, DTB_PMIC1))return NULL;if (!update_dtb_entry_node(dt_list, DTB_PMIC2))return NULL;if (!update_dtb_entry_node(dt_list, DTB_PMIC3))return NULL;list_for_every_entry(&dt_list->node, dt_node_tmp1, dt_node, node) {if (!dt_node_tmp1) {dprintf(CRITICAL, "ERROR: Couldn't find the suitable DTB!\n");return NULL;}if (dt_node_tmp1->dt_entry_m)return dt_node_tmp1->dt_entry_m;}return NULL;
}

整个过程和刚才添加验证的过程类似,比对与硬件的匹配度,找到最佳匹配的 dt_entry 然后返回。

如果获取到了最佳匹配的 dt_entry 则按照加载 kernel 的步骤来加载到内存地址,即按照如下步骤:根据标志位来解压数据。 拷贝到boot_img_hdr 指定的内存地址。

这样 device tree 的加载就完成了。

没有指定 dtsize 如何加载 device tree。如果没有专门的 device tree block, 则需要判断 kernel block 是否附加了 device tree 信息。整个过程都是通过调用函数dev_tree_appended 函数实现,其实现如下:

/*
 * Will relocate the DTB to the tags addr if the device tree is found and return
 * its address
 *
 * Arguments:    kernel - Start address of the kernel loaded in RAM
 *               tags - Start address of the tags loaded in RAM
 *               kernel_size - Size of the kernel in bytes
 *
 * Return Value: DTB address : If appended device tree is found
 *               'NULL'         : Otherwise
 */
void *dev_tree_appended(void *kernel, uint32_t kernel_size, uint32_t dtb_offset, void *tags)
{void *kernel_end = kernel + kernel_size;uint32_t app_dtb_offset = 0;void *dtb = NULL;void *bestmatch_tag = NULL;struct dt_entry *best_match_dt_entry = NULL;uint32_t bestmatch_tag_size;struct dt_entry_node *dt_entry_queue = NULL;struct dt_entry_node *dt_node_tmp1 = NULL;struct dt_entry_node *dt_node_tmp2 = NULL;
  /* Initialize the dtb entry node*/dt_entry_queue = (struct dt_entry_node *)malloc(sizeof(struct dt_entry_node));if (!dt_entry_queue) {dprintf(CRITICAL, "Out of memory\n");return NULL;}list_initialize(&dt_entry_queue->node);if (dtb_offset)app_dtb_offset = dtb_offset;elsememcpy((void*) &app_dtb_offset, (void*) (kernel + DTB_OFFSET), sizeof(uint32_t));if (((uintptr_t)kernel + (uintptr_t)app_dtb_offset) < (uintptr_t)kernel) {return NULL;}dtb = kernel + app_dtb_offset;while (((uintptr_t)dtb + sizeof(struct fdt_header)) < (uintptr_t)kernel_end) {struct fdt_header dtb_hdr;uint32_t dtb_size; /* the DTB could be unaligned, so extract the header,
     * and operate on it separately */memcpy(&dtb_hdr, dtb, sizeof(struct fdt_header));if (fdt_check_header((const void *)&dtb_hdr) != 0 ||((uintptr_t)dtb + (uintptr_t)fdt_totalsize((const void *)&dtb_hdr) < (uintptr_t)dtb) ||((uintptr_t)dtb + (uintptr_t)fdt_totalsize((const void *)&dtb_hdr) > (uintptr_t)kernel_end))break;dtb_size = fdt_totalsize(&dtb_hdr);if (check_aboot_addr_range_overlap((uint32_t)tags, dtb_size)) {dprintf(CRITICAL, "Tags addresses overlap with aboot addresses.\n");return NULL;}dev_tree_compatible(dtb, dtb_size, dt_entry_queue);
/* goto the next device tree if any */dtb += dtb_size;}best_match_dt_entry = platform_dt_match_best(dt_entry_queue);if (best_match_dt_entry){bestmatch_tag = (void *)best_match_dt_entry->offset;bestmatch_tag_size = best_match_dt_entry->size;dprintf(INFO, "Best match DTB tags %u/%08x/0x%08x/%x/%x/%x/%x/%x/%x/%x\n",best_match_dt_entry->platform_id, best_match_dt_entry->variant_id,best_match_dt_entry->board_hw_subtype, best_match_dt_entry->soc_rev,best_match_dt_entry->pmic_rev[0], best_match_dt_entry->pmic_rev[1],best_match_dt_entry->pmic_rev[2], best_match_dt_entry->pmic_rev[3],best_match_dt_entry->offset, best_match_dt_entry->size);dprintf(INFO, "Using pmic info 0x%0x/0x%x/0x%x/0x%0x for device 0x%0x/0x%x/0x%x/0x%0x\n",best_match_dt_entry->pmic_rev[0], best_match_dt_entry->pmic_rev[1],best_match_dt_entry->pmic_rev[2], best_match_dt_entry->pmic_rev[3],board_pmic_target(0), board_pmic_target(1),board_pmic_target(2), board_pmic_target(3));}
  /* free queue's memory */list_for_every_entry(&dt_entry_queue->node, dt_node_tmp1, dt_node, node) {dt_node_tmp2 = (struct dt_entry_node *) dt_node_tmp1->node.prev;dt_entry_list_delete(dt_node_tmp1);dt_node_tmp1 = dt_node_tmp2;}if(bestmatch_tag) {memcpy(tags, bestmatch_tag, bestmatch_tag_size);
    /* clear out the old DTB magic so kernel doesn't find it */*((uint32_t *)(kernel + app_dtb_offset)) = 0;return tags;}dprintf(CRITICAL, "DTB offset is incorrect, kernel image does not have appended DTB\n");dprintf(INFO, "Device info 0x%08x/%08x/0x%08x/%u, pmic 0x%0x/0x%x/0x%x/0x%0x\n",board_platform_id(), board_soc_version(),board_target_id(), board_hardware_subtype(),board_pmic_target(0), board_pmic_target(1),board_pmic_target(2), board_pmic_target(3));return NULL;
}

首先需要获取 device tree table 的偏移,偏移有以下两种情况: kernel 是经过解压的,则指定的位置在解压 kernel 时确定。 kernel 没有经过压缩,则偏移在kernel + 0x2C 的位置上获取。

从 device tree 偏移位置开始到 kernel 尾部的范围内遍历 device tree 数据。这里相当于遍历一个数组,数组的成员为 struct fdt_header, 这个结构定义在 lib/libfdt/fdt.h 文件中,它的结构如下:

struct fdt_header {uint32_t magic;      /* magic word FDT_MAGIC */uint32_t totalsize;    /* total size of DT block */uint32_t off_dt_struct;    /* offset to structure */uint32_t off_dt_strings;   /* offset to strings */uint32_t off_mem_rsvmap;   /* offset to memory reserve map */uint32_t version;    /* format version */uint32_t last_comp_version;  /* last compatible version */  /* version 2 fields below */uint32_t boot_cpuid_phys;  /* Which physical CPU id we're
                                booting on *//* version 3 fields below */uint32_t size_dt_strings;  /* size of the strings block */  /* version 17 fields below */uint32_t size_dt_struct;   /* size of the structure block */
};

检查遍历到的 device tree 的 fdt_header 是否符合以下条件:

是否能通过 fdt_check_header 检查。fdt_check_header 的代码在 lib/libfdt/fdt.c 文件中,其实现如下:

int fdt_check_header(const void *fdt)
{if (fdt_magic(fdt) == FDT_MAGIC) {
    /* Complete tree */if (fdt_version(fdt) < FDT_FIRST_SUPPORTED_VERSION)return -FDT_ERR_BADVERSION;if (fdt_last_comp_version(fdt) > FDT_LAST_SUPPORTED_VERSION)return -FDT_ERR_BADVERSION;} else if (fdt_magic(fdt) == FDT_SW_MAGIC) {  /* Unfinished sequential-write blob */if (fdt_size_dt_struct(fdt) == 0)return -FDT_ERR_BADSTATE;} else {return -FDT_ERR_BADMAGIC;}if (fdt_off_dt_struct(fdt) > (UINT_MAX - fdt_size_dt_struct(fdt)))return FDT_ERR_BADOFFSET;if (fdt_off_dt_strings(fdt) > (UINT_MAX -  fdt_size_dt_strings(fdt)))return FDT_ERR_BADOFFSET;if ((fdt_off_dt_struct(fdt) + fdt_size_dt_struct(fdt)) > fdt_totalsize(fdt))return FDT_ERR_BADOFFSET;if ((fdt_off_dt_strings(fdt) + fdt_size_dt_strings(fdt)) > fdt_totalsize(fdt))return FDT_ERR_BADOFFSET;return 0;
}

如果符合符合以下条件都是正常的 fdt_header

如果 MAGIC = 0xd00dfeed。 fdt_header.version >= 0×10。 fdt_header.last_comp_version <= 0×11。 如果 MAGIC = 0x2ff20112。fdt_header.size_dt_struct 不等于 0。fdt_header.off_dt_struct < 0xFFFFFFFF –fd_header.size_dt_structfdt_header.off_dt_strings < 0xFFFFFFFF –fd_header.size_dt_stringsfdt_header.off_dt_struct +fdt_header.size_dt_struct <fdt_header.totalsizefdt_header.off_dt_strings +fdt_header.size_dt_strings <fdt_header.totalsize 。 device tree 偏移加fdt_header.totalsize 是否小于 device tree 的偏移,即fdt_header.totalsize 是否为负数。 device tree 偏移加fdt_header.totalsize 是否大于 kernel end 的偏移,即是否越界。

通过检查的 device tree 调用 dev_tree_compatible 函数检查兼容性,符合条件的添加到链表中,甚于的步骤和指定了 dtsize 的步骤就基本相同了。

调用 boot_linux 启动系统 

到这一步 boot/recovery 基本的初始化工作,加载工作就基本完成了,下一步就可以通过 boot_linux 函数来进行启动了。启动完成后就会将控制权移交给 linux kernel,android 系统就开始正式运行了。boot_linux 的代码位于app/aboot/aboot.c 文件中,其实现如下:

void boot_linux(void *kernel, unsigned *tags,const char *cmdline, unsigned machtype,void *ramdisk, unsigned ramdisk_size)
{unsigned char *final_cmdline;
#if DEVICE_TREEint ret = 0;
#endifvoid (*entry)(unsigned, unsigned, unsigned*) = (entry_func_ptr*)(PA((addr_t)kernel));uint32_t tags_phys = PA((addr_t)tags);struct kernel64_hdr *kptr = ((struct kernel64_hdr*)(PA((addr_t)kernel)));ramdisk = (void *)PA((addr_t)ramdisk);final_cmdline = update_cmdline((const char*)cmdline);#if DEVICE_TREEdprintf(INFO, "Updating device tree: start\n");  /* Update the Device Tree */ret = update_device_tree((void *)tags,(const char *)final_cmdline, ramdisk, ramdisk_size);if(ret){dprintf(CRITICAL, "ERROR: Updating Device Tree Failed \n");ASSERT(0);}dprintf(INFO, "Updating device tree: done\n");
#else
  /* Generating the Atags */generate_atags(tags, final_cmdline, ramdisk, ramdisk_size);
#endiffree(final_cmdline);#if VERIFIED_BOOT
  /* Write protect the device info */if (!boot_into_recovery && target_build_variant_user() && devinfo_present && mmc_write_protect("devinfo", 1)){dprintf(INFO, "Failed to write protect dev info\n");ASSERT(0);}
#endif /* Turn off splash screen if enabled */
#if DISPLAY_SPLASH_SCREENtarget_display_shutdown();
#endif  /* Perform target specific cleanup */target_uninit();dprintf(INFO, "booting linux @ %p, ramdisk @ %p (%d), tags/device tree @ %p\n",entry, ramdisk, ramdisk_size, (void *)tags_phys);enter_critical_section();  /* Initialise wdog to catch early kernel crashes */
#if WDOG_SUPPORTmsm_wdog_init();
#endif
  /* do any platform specific cleanup before kernel entry */platform_uninit();arch_disable_cache(UCACHE);#if ARM_WITH_MMUarch_disable_mmu();
#endifbs_set_timestamp(BS_KERNEL_ENTRY);if (IS_ARM64(kptr))
    /* Jump to a 64bit kernel */scm_elexec_call((paddr_t)kernel, tags_phys);else
    /* Jump to a 32bit kernel */entry(0, machtype, (unsigned*)tags_phys);
}

首先进行地址转换,不过在目前的实现中 PA 宏是直接返回传入的地址,所以地址不会被转换,相当于一个预留的扩展接口。

#define PA(x) platform_get_virt_to_phys_mapping(x)
#define VA(x) platform_get_phys_to_virt_mapping(x)addr_t platform_get_virt_to_phys_mapping(addr_t virt_addr)
{
  /* Using 1-1 mapping on this platform. */return virt_addr;
}addr_t platform_get_phys_to_virt_mapping(addr_t phys_addr)
{
  /* Using 1-1 mapping on this platform. */return phys_addr;
}

这个只需要注意的是 kernel 的首地址即是整个 kernel 的入口,入口的函数类型为 entry_func_ptr, 其定义如下:

typedef void entry_func_ptr(unsigned, unsigned, unsigned*);

*本文原创作者:SetRet,本文属FreeBuf原创奖励计划,未经许可禁止转载调用update_cmdline 更新boot_img_hdr.cmdline 字段中启动命令。

unsigned char *update_cmdline(const char * cmdline)
{int cmdline_len = 0;int have_cmdline = 0;unsigned char *cmdline_final = NULL;int pause_at_bootup = 0;bool warm_boot = false;bool gpt_exists = partition_gpt_exists();int have_target_boot_params = 0;char *boot_dev_buf = NULL;bool is_mdtp_activated = 0;
#ifdef MDTP_SUPPORTmdtp_activated(&is_mdtp_activated);
#endif /* MDTP_SUPPORT */if (cmdline && cmdline[0]) {cmdline_len = strlen(cmdline);have_cmdline = 1;}else {dprintf(CRITICAL,"cmdline is NULL\n");ASSERT(0);}if (target_is_emmc_boot()) {cmdline_len += strlen(emmc_cmdline);
#if USE_BOOTDEV_CMDLINEboot_dev_buf = (char *) malloc(sizeof(char) * BOOT_DEV_MAX_LEN);ASSERT(boot_dev_buf);memset((void *)boot_dev_buf, 0, sizeof(*boot_dev_buf));platform_boot_dev_cmdline(boot_dev_buf);cmdline_len += strlen(boot_dev_buf);
#endif}cmdline_len += strlen(usb_sn_cmdline);cmdline_len += strlen(sn_buf);if (boot_into_recovery && gpt_exists)cmdline_len += strlen(secondary_gpt_enable);if(is_mdtp_activated)cmdline_len += strlen(mdtp_activated_flag);if (boot_into_ffbm) {cmdline_len += strlen(androidboot_mode);cmdline_len += strlen(ffbm_mode_string);
    /* reduce kernel console messages to speed-up boot */cmdline_len += strlen(loglevel);} else if (boot_reason_alarm) {cmdline_len += strlen(alarmboot_cmdline);} else if ((target_build_variant_user() || device.charger_screen_enabled)&& target_pause_for_battery_charge()) {pause_at_bootup = 1;cmdline_len += strlen(battchg_pause);}if(target_use_signed_kernel() && auth_kernel_img) {cmdline_len += strlen(auth_kernel);}if (get_target_boot_params(cmdline, boot_into_recovery ? "recoveryfs" :"system",&target_boot_params) == 0) {have_target_boot_params = 1;cmdline_len += strlen(target_boot_params);}  /* Determine correct androidboot.baseband to use */switch(target_baseband()){case BASEBAND_APQ:cmdline_len += strlen(baseband_apq);break;case BASEBAND_MSM:cmdline_len += strlen(baseband_msm);break;case BASEBAND_CSFB:cmdline_len += strlen(baseband_csfb);break;case BASEBAND_SVLTE2A:cmdline_len += strlen(baseband_svlte2a);break;case BASEBAND_MDM:cmdline_len += strlen(baseband_mdm);break;case BASEBAND_MDM2:cmdline_len += strlen(baseband_mdm2);break;case BASEBAND_SGLTE:cmdline_len += strlen(baseband_sglte);break;case BASEBAND_SGLTE2:cmdline_len += strlen(baseband_sglte2);break;case BASEBAND_DSDA:cmdline_len += strlen(baseband_dsda);break;case BASEBAND_DSDA2:cmdline_len += strlen(baseband_dsda2);break;}if (cmdline) {if ((strstr(cmdline, DISPLAY_DEFAULT_PREFIX) == NULL) &&target_display_panel_node(display_panel_buf,MAX_PANEL_BUF_SIZE) &&strlen(display_panel_buf)) {cmdline_len += strlen(display_panel_buf);}}if (target_warm_boot()) {warm_boot = true;cmdline_len += strlen(warmboot_cmdline);}if (cmdline_len > 0) {const char *src;unsigned char *dst;cmdline_final = (unsigned char*) malloc((cmdline_len + 4) & (~3));ASSERT(cmdline_final != NULL);memset((void *)cmdline_final, 0, sizeof(*cmdline_final));dst = cmdline_final;    /* Save start ptr for debug print */if (have_cmdline) {src = cmdline;while ((*dst++ = *src++));}if (target_is_emmc_boot()) {src = emmc_cmdline;if (have_cmdline) --dst;have_cmdline = 1;while ((*dst++ = *src++));
#if USE_BOOTDEV_CMDLINEsrc = boot_dev_buf;if (have_cmdline) --dst;while ((*dst++ = *src++));
#endif}src = usb_sn_cmdline;if (have_cmdline) --dst;have_cmdline = 1;while ((*dst++ = *src++));src = sn_buf;if (have_cmdline) --dst;have_cmdline = 1;while ((*dst++ = *src++));if (warm_boot) {if (have_cmdline) --dst;src = warmboot_cmdline;while ((*dst++ = *src++));}if (boot_into_recovery && gpt_exists) {src = secondary_gpt_enable;if (have_cmdline) --dst;while ((*dst++ = *src++));}if (is_mdtp_activated) {src = mdtp_activated_flag;if (have_cmdline) --dst;while ((*dst++ = *src++));}if (boot_into_ffbm) {src = androidboot_mode;if (have_cmdline) --dst;while ((*dst++ = *src++));src = ffbm_mode_string;if (have_cmdline) --dst;while ((*dst++ = *src++));src = loglevel;if (have_cmdline) --dst;while ((*dst++ = *src++));} else if (boot_reason_alarm) {src = alarmboot_cmdline;if (have_cmdline) --dst;while ((*dst++ = *src++));} else if (pause_at_bootup) {src = battchg_pause;if (have_cmdline) --dst;while ((*dst++ = *src++));}if(target_use_signed_kernel() && auth_kernel_img) {src = auth_kernel;if (have_cmdline) --dst;while ((*dst++ = *src++));}switch(target_baseband()){case BASEBAND_APQ:src = baseband_apq;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_MSM:src = baseband_msm;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_CSFB:src = baseband_csfb;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_SVLTE2A:src = baseband_svlte2a;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_MDM:src = baseband_mdm;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_MDM2:src = baseband_mdm2;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_SGLTE:src = baseband_sglte;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_SGLTE2:src = baseband_sglte2;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_DSDA:src = baseband_dsda;if (have_cmdline) --dst;while ((*dst++ = *src++));break;case BASEBAND_DSDA2:src = baseband_dsda2;if (have_cmdline) --dst;while ((*dst++ = *src++));break;}if (strlen(display_panel_buf)) {src = display_panel_buf;if (have_cmdline) --dst;while ((*dst++ = *src++));}if (have_target_boot_params) {if (have_cmdline) --dst;src = target_boot_params;while ((*dst++ = *src++));free(target_boot_params);}}if (boot_dev_buf)free(boot_dev_buf);if (cmdline_final)dprintf(INFO, "cmdline: %s\n", cmdline_final);elsedprintf(INFO, "cmdline is NULL\n");return cmdline_final;
}

更新 cmdline 可以分为以下几个步骤:

通过已有的命令和需要添加的命令计算出 final_cmdline 需要的长度。 申请存储 final_cmdline 的 buffer, 并且清零。 拷贝需要的 cmd 命令到final_cmdline 中。

update 主要涉及到的命令,参数,和会使用的情况可以整理为下表:

cmd value condition
androidboot.emmc true target is emmc
androidboot.serialno [serial number] no condition
gpt no value recovery boot and gpt exists
mdtp no value mdtp activated
androidboot.mode [ffbm string] ffbm boot
androidboot.alarmboot true alarm boot
androidboot.mode charger charger
androidboot.authorizedkernel true signed kernel and kernel authorized
androidboot.baseband apq baseband is apq
androidboot.baseband msm baseband is msm
androidboot.baseband csfb baseband is csfb
androidboot.baseband svlte2a baseband is svlte2a
androidboot.baseband mdm baseband is mdm
androidboot.baseband mdm2 baseband is mdm2
androidboot.baseband sglte baseband is sglte
androidboot.baseband dsda baseband is dsda
androidboot.baseband dsda2 baseband is dsda2
androidboot.baseband sglte2 baseband is sglte2
qpnp-power-on.warmboot true target warm boot
mdssmdp [display panel buffer] have dssmdp cmd in boot image header cmdline

调用 update_device_tree 更新 device tree 信息。这一步主要是从 device tree 中解析出地址信息,然后在启动 kernel 之时传递给 kernel 方便加载。整个过程由于需要对 device tree 整体结构的详解,这里暂不赘述,有兴趣的读者可以参考高通平台 Android 源码分析之 Linux 内核设备树(DT – Device Tree)。 在未解锁的情况下对 devinfo 分区进行写保护,这个分区存储的是 bootloader 的解锁信息和验证信息,进行写保护避免误操作。 关闭一些针对 lk 开启的特性,清理预设的数据,如:MMU, CACHE 等。

调用 kernel 入口点,进入 kernel 的代码区域。这里需要注意的是 32 位和 64 位的进入方法不同。32 位是直接 call kernel 的入口点,将入口点作为函数调用。而 64 位 kernel 则是通过scm_elexec_call 来进入内核空间。scm_elexec_call 的实现在platform/msm_shared/scm.c 文件中,其实现如下:

/* Execption Level exec secure-os call
 * Jumps to kernel via secure-os and does not return
 * on successful jump. System parameters are setup &
 * passed on to secure-os and are utilized to boot the
 * kernel.
 *
 @ kernel_entry : kernel entry point passed in as link register.
 @ dtb_offset : dt blob address passed in as w0.
 @ svc_id : indicates direction of switch 32->64 or 64->32
 *
 * Assumes all sanity checks have been performed on arguments.
 */void scm_elexec_call(paddr_t kernel_entry, paddr_t dtb_offset)
{uint32_t svc_id = SCM_SVC_MILESTONE_32_64_ID;uint32_t cmd_id = SCM_SVC_MILESTONE_CMD_ID;void *cmd_buf;size_t cmd_len;static el1_system_param param __attribute__((aligned(0x1000)));scmcall_arg scm_arg = {0};param.el1_x0 = dtb_offset;param.el1_elr = kernel_entry;  /* Response Buffer = Null as no response expected */dprintf(INFO, "Jumping to kernel via monitor\n");if (!is_scm_armv8_support()){
    /* Command Buffer */cmd_buf = (void *)&param;cmd_len = sizeof(el1_system_param);scm_call(svc_id, cmd_id, cmd_buf, cmd_len, NULL, 0);}else{scm_arg.x0 = MAKE_SIP_SCM_CMD(SCM_SVC_MILESTONE_32_64_ID, SCM_SVC_MILESTONE_CMD_ID);scm_arg.x1 = MAKE_SCM_ARGS(0x2, SMC_PARAM_TYPE_BUFFER_READ);scm_arg.x2 = (uint32_t ) &param;scm_arg.x3 = sizeof(el1_system_param);scm_call2(&scm_arg, NULL);}  /* Assert if execution ever reaches here */dprintf(CRITICAL, "Failed to jump to kernel\n");ASSERT(0);
}

而我们知道 scm(Secure Channel Manager) 相关的函数是 TrustZone 提供给普通个世界的接口,函数流程比较简单,根据是否支持 armv8 进行不同的跳转,由于 TrustZone 实现是黑盒,所以这里暂不研究。只需要知道 64 位 kernel 是通过 TrustZone 来启动即可。

1 参考资料

Verifying Boot  |  Android Open Source ProjectVerified Boot  |  Android Open Source Projectandroid-cdd.pdf高通平台 Android 源码分析之 Linux 内核设备树(DT – Device Tree) | Andy.Lee’s Blog

Footnotes:

1 ffbm (fast factory boot mode) 是高通开发的一套半开机模式下的测试界面,用于工厂测试,提高生产效率。

2 具体的 boot state 模式可以参考 android 官方文档Verifying Boot  |  Android Open Source Project


lk 源码分析的系列文章到这里就告一段落了。请期待后续其他文章,如果有任何疑问和交流意向可以和我们联系 SecRet201611@gmail.com 。

高通(Qualcomm)LK源码深度分析(三)相关推荐

  1. 高通android开源代码下载,高通平台Android源码bootloader分析之sbl1(三)

    前两篇博文分析了启动流程.代码流程.cdt,接下来就分析另外几个需要格外关注的部分. ##log系统 sbl1中的log系统也是sbl1部分调试会经常接触得部分高通平台在sbl中做的log系统并不是很 ...

  2. 高通平台Android源码bootloader分析之sbl1(一)

    高通8k平台的boot过程搞得比较复杂, 我也是前段时间遇到一些问题深入研究了一下才搞明白.不过虽然弄得很复杂,我们需要动的东西其实很少,modem侧基本就sbl1(全称:Secondary boot ...

  3. Toast源码深度分析

    目录介绍 1.最简单的创建方法 1.1 Toast构造方法 1.2 最简单的创建 1.3 简单改造避免重复创建 1.4 为何会出现内存泄漏 1.5 吐司是系统级别的 2.源码分析 2.1 Toast( ...

  4. libevent源码深度剖析三

    libevent源码深度剖析三 --libevent基本使用场景和事件流程 张亮 1 前言 学习源代码该从哪里入手?我觉得从程序的基本使用场景和代码的整体处理流程入手是个不错的方法,至少从个人的经验上 ...

  5. HashMap 源码深度分析

    HashMap 源码分析 在Map集合中, HashMap 则是最具有代表性的,也是我们最常使用到的 Map 集合.由于 HashMap 底层涉及了很多的知识点,可以比较好的考察一个人的Java的基本 ...

  6. jdk源码分析书籍 pdf_什么?Spring5 AOP 默认使用Cglib?从现象到源码深度分析

    推荐阅读: 阿里工作十年拿下P8,多亏了这些PDF陪我成长(Spring全家桶+源码解析+Redis实战等)​zhuanlan.zhihu.com 从入门到熟悉,一步一步带你了解 MySQL 中的「索 ...

  7. AUTOSAR从入门到精通100讲(二十)-特斯拉、高通、华为AI处理器深度分析

    很多人会问,为什么没有英伟达?目前所有主流深度学习运算主流框架后端都是英伟达的CUDA,包括TensorFlow.Caffe.Caffe2.PyTorch.mxnet.PaddlePaddle,CUD ...

  8. Spring源码深度分析一-Spring前世今生以及源码学习路线图

    大家好,我是王老狮,今天开始开新坑.作为JAVA程序员,Spring基本上是必备的技能,也是面试经常考核的技能,特别是大厂,Spring源码基本是必问的题目.但是很多同学看到源码就头疼,根本不知道源码 ...

  9. 高通Q888内核源码分析--概述篇

    基于小米开源代码Linux Kernel5.4.61, github地址为https://github.com/MiCode/Xiaomi_Kernel_OpenSource.git star-r-o ...

最新文章

  1. C++自学随笔(2)
  2. SpringMVC、Spring和Struts的区别
  3. python读取文件with open_python 文件读写操作open和with的用法
  4. S3c2410_SDIO_调试笔记二
  5. HttpURLConnection与 HttpClient 区别/性能测试对比
  6. Android项目实战欢迎界面
  7. 是的,我不做技术经理了
  8. 比尔盖茨夫妇宣布离婚 结束27年婚姻
  9. 程序员的头符合好头的标准吗?
  10. Java随机26位英文字母
  11. java 实现EME2000(国家大地坐标系)转ECEF坐标系(地心地固坐标系)
  12. docker(十)—— Windows系统下安装docker
  13. 《2022中国企业数字化办公创新与实践产业研究报告》附下载丨三叠云
  14. 记录一次重装win10系统后,没有1920*1080分辨率的问题
  15. 群晖NAS设置IPV6公网访问
  16. 跟卖亚马逊跨境电子商务ERP
  17. 这些链接都打不开,失效了
  18. 计算机网络实验三 cpt
  19. 高中计算机会考操作题素材,2021高中信息技术 操作题 (练习二) 精品
  20. [超详细]微信小程序使用iconfont教程及解决真机无法显示问题

热门文章

  1. 别人可以在今日头条发文章赚钱,为什么你赚不到呢?
  2. 三、pytest接口自动化之pytest中setup/teardown,setup_class/teardown_class讲解
  3. JAVA前端修改密码,Java Web版SVN 配置管理工具 2.0 (远道建立仓库,修改密码,设置权限,支持apache等)...
  4. EfficientNet迁移学习(四) —— 损失函数解析
  5. 管理员已阻止你运行此应用。有关详细信息,请与管理员联系。windows10
  6. 关于固态硬盘闪存芯片研究资料收集
  7. Mixed mode assembly is built against version 'v1.1.4322' of the runtime and...问题——C# DirectXSound
  8. CS5261|CS5265|Type-C转HDMI 4K30HZ 4K60HZ音视频
  9. 企业微信如何快速高效添加好友?
  10. 二维码解析:使用 JavaScript 库reqrcode.js解析二维码