FairScheduler源码计算fair share

FairScheduler发生资源抢占有两个触发条件

1、根据配置的最低资源量保障，如果超过minSharePreemptionTimeout时间，一直获取不到最低资源量，则发生抢占，minSharePreemptionTimeout源码中默认是long.max_value，可配置文件配置

2、根据fairShare量，如果超过fairSharePreemptionTimeout时间，一直获取不到fairSharePreemptionThreshold*fairShare资源，则发生抢占，其中fairSharePreemptionTimeout源码默认值是long.max_value，fairSharePreemptionThreshold为0.5，可配置文件配置。而fairShare为动态计算而来，根据各个队列的weigt计算。

minSharePreemptionTimeout: number of seconds the queue is under its minimum share before it will try to preempt containers to take resources from other queues. If not set, the queue will inherit the value from its parent queue.

fairSharePreemptionTimeout: number of seconds the queue is under its fair share threshold before it will try to preempt containers to take resources from other queues. If not set, the queue will inherit the value from its parent queue.

fairSharePreemptionThreshold: the fair share preemption threshold for the queue. If the queue waits fairSharePreemptionTimeout without receiving fairSharePreemptionThreshold*fairShare resources, it is allowed to preempt containers to take resources from other queues. If not set, the queue will inherit the value from its parent queue.

计算fairShare源码部分

    private class UpdateThread extends Thread {@Overridepublic void run() {while (!Thread.currentThread().isInterrupted()) {try {Thread.sleep(updateInterval);long start = getClock().getTime();update();  //preemptTasksIfNecessary(); long duration = getClock().getTime() - start;fsOpDurations.addUpdateThreadRunDuration(duration);} catch (InterruptedException ie) {LOG.warn("Update thread interrupted. Exiting.");return;} catch (Exception e) {LOG.error("Exception in fair scheduler UpdateThread", e);}}}}protected synchronized void update() {long start = getClock().getTime();updateStarvationStats(); // Determine if any queues merit preemptionFSQueue rootQueue = queueMgr.getRootQueue();// Recursively update demands for all queuesrootQueue.updateDemand();rootQueue.setFairShare(clusterResource); ///root赋值集群所有值// Recursively compute fair shares for all queues// and update metricsrootQueue.recomputeShares(); /递归计算子队列fair share//FSLeafQueuepublic void recomputeShares() {readLock.lock();try {policy.computeShares(runnableApps, getFairShare());} finally {readLock.unlock();}}ComputeFairShares类public void computeShares(Collection<? extends Schedulable> schedulables,Resource totalResources) {ComputeFairShares.computeShares(schedulables, totalResources, ResourceType.MEMORY);}//此处有套算法
权重比例值，平均算出来的一个理想值
//weight *一个系数（系数目的是权重到资源值作个转换)，加起来正好等于集群总量，这样就是每个队列的fair shareprivate static void computeSharesInternal(Collection<? extends Schedulable> allSchedulables,Resource totalResources, ResourceType type, boolean isSteadyShare) {Collection<Schedulable> schedulables = new ArrayList<Schedulable>();int takenResources = handleFixedFairShares(allSchedulables, schedulables, isSteadyShare, type);if (schedulables.isEmpty()) {return;}// Find an upper bound on R that we can use in our binary search. We start// at R = 1 and double it until we have either used all the resources or we// have met all Schedulables' max shares.int totalMaxShare = 0;for (Schedulable sched : schedulables) {int maxShare = getResourceValue(sched.getMaxShare(), type);   ///将每个队列的最大资源使用上限加起来得到个总数totalMaxShare = (int) Math.min((long)maxShare + (long)totalMaxShare,Integer.MAX_VALUE);if (totalMaxShare == Integer.MAX_VALUE) {break;}}int totalResource = Math.max((getResourceValue(totalResources, type) -takenResources), 0);   totalResource = Math.min(totalMaxShare, totalResource);  //资源最大使用上限总和与集群总资源比较，取较小值，一般获得的是集群总的资源double rMax = 1.0;while (resourceUsedWithWeightToResourceRatio(rMax, schedulables, type)  通过该方法找系数weigt*系数,type赋值内存类型，找到正好不超过总的集群资源< totalResource) {rMax *= 2.0;}//再通过二分查找，在0到这个系数之间，找到正好可以使用总的资源量最接近集群总容量的。// Perform the binary search for up to COMPUTE_FAIR_SHARES_ITERATIONS stepsdouble left = 0;double right = rMax;for (int i = 0; i < COMPUTE_FAIR_SHARES_ITERATIONS; i++) {double mid = (left + right) / 2.0;int plannedResourceUsed = resourceUsedWithWeightToResourceRatio(mid, schedulables, type);if (plannedResourceUsed == totalResource) {right = mid;break;} else if (plannedResourceUsed < totalResource) {left = mid;} else {right = mid;}}// Set the fair shares based on the value of R we've converged tofor (Schedulable sched : schedulables) {if (isSteadyShare) {setResourceValue(computeShare(sched, right, type),((FSQueue) sched).getSteadyFairShare(), type);} else {setResourceValue(computeShare(sched, right, type), sched.getFairShare(), type);}}}/*** Compute the resources that would be used given a weight-to-resource ratio* w2rRatio, for use in the computeFairShares algorithm as described in #*/private static int resourceUsedWithWeightToResourceRatio(double w2rRatio,Collection<? extends Schedulable> schedulables, ResourceType type) {int resourcesTaken = 0;for (Schedulable sched : schedulables) {int share = computeShare(sched, w2rRatio, type);/resourcesTaken += share;}return resourcesTaken;}/*** Compute the resources assigned to a Schedulable given a particular* weight-to-resource ratio w2rRatio.*/private static int computeShare(Schedulable sched, double w2rRatio,ResourceType type) { /权重值*2，保证在最低资源保障和最高使用上限之间double share = sched.getWeights().getWeight(type) * w2rRatio;share = Math.max(share, getResourceValue(sched.getMinShare(), type));share = Math.min(share, getResourceValue(sched.getMaxShare(), type));return (int) share;}

FairScheduler源码计算fair share相关推荐

Vue 3.0 源码计算属性 Computed的实现
我们来看下计算属性源码是怎么实现的呢? 参数 getterOrOptions 是判断是写的第一种是函数只有getter computed(()=>{ }); 第二种是对象有get 跟 ...
java源码——计算立体图形的表面积和体积
计算球,圆柱,圆锥的表面积和体积. 利用接口实现. 上代码. Contants.java 常量存储类 package com.fuxuemingzhu.solidgraphics.contants;/ ...
C语言超牛简单源码计算超大整数的阶乘
把开发过程中比较好的一些代码段记录起来,如下代码是关于C语言超牛简单计算超大整数的阶乘的代码,应该是对各位有些用途. #define N 400 long a[8916]={1,0},n,i,c,le ...
用计算机源码计算加法,MFC实现简单计算器（支持加减乘除和括号运算）
[实例简介] 自己写的计算器,支持加减乘除和括号运算.开发环境为VS2010,MFC框架.代码内容简单不复杂适合初学者参考. [实例截图] [核心代码] Calculator[VS2010_MFC] ...
【Android 性能优化】应用启动优化 ( 启动白屏问题 | 应用启动时间测量 | 冷启动 | 热启动 | 应用启动时间计算源码分析 )
文章目录一. APP 启动白屏 / 黑屏二. APP 启动速度测量 1. 通过 Logcat 日志查看应用启动时间 2. 通过 adb 命令查看界面启动时间三. APP 冷启动与热启动四. A ...
Java同步锁Synchronized底层源码和原理剖析
目录 1 synchronized场景回顾 2 反汇编寻找锁实现原理 3 synchronized虚拟机源码 3.1 HotSpot源码Monitor生成 3.2 HotSpot源码之Monitor竞 ...
深入HotSpot虚拟机源码探究synchronized底层实现原理【万字总结synchronized】
文章目录一.synchronized原理 (1)首先准备好HotSpot源码 (2)解压,使用vscode或者其他编辑器打开 (3)初始monitor监视器锁(先了解后细说) (4)建立宏观概念(初 ...
HP加易语言数据库，全源码制作的网络验证，可运营，可自行扩展
点击阅读原文界面很丑,我知道的..但是功能还不错,因为论坛没人写这个东东,所以今天有空就写了下,其他论坛也是我发的,所以不要问我是不是转载的.我是原创的.嘿嘿功能如下-注册,找回,登录,查询,转绑 ...
【阅读源码系列】ConcurrentHashMap源码分析(JDK1.7和1.8)
个人学习源码的思路: 使用ctrl+单机进入源码,并阅读源码的官方文档–>大致的了解一下此类的特点和功能使用ALIT+7查看类中所有方法–>大致的看一下此类的属性和方法找到重要方法并阅 ...
利用Bootstrap快速搭建个人响应式主页（附演示+源码）
1.前言我们每个程序员都渴望搭建自己的技术博客平台与他人进行交流分享,但使用别人的博客模板没有创意.做网站后台的开发人员可能了解前端,可是自己写一个不错的前端还是很费事的.幸好我们有Bootstra ...

FairScheduler源码计算fair share

FairScheduler源码计算fair share相关推荐

最新文章

热门文章