1.介绍
2 修改资源队列 API步骤
- 2.1 获取capacity-scheduler配置的tag和version
- 2.2 获取capacity-scheduler指定版本的配置
- 2.3 通过Cluster Metrics API获取yarn可分配资源
- 2.4 修改配置
- 2.5 刷新队列配置
- 2.6 获取scheduler Info,查看各队列使用情况
3 hdp3 与2 版本API差异：

1.介绍

hadoop 3.0以后支持以绝对值方式指定yarn资源队列memory vcores及gpu cores的大小，2.x版本只能指定每个队列使用内存资源的百分比。

由于集群总容量可能会有所不同，因此资源配置值可使用百分比表示。
在capacity-scheduler.xml配置如下：
6:1:3的比例分配集群资源，engineering：60%，support：10%，30%

Property: yarn.scheduler.capacity.root.engineering.capacity
Value: 60Property: yarn.scheduler.capacity.root.support.capacity
Value: 10Property: yarn.scheduler.capacity.root.marketing.capacity
Value: 30

如果希望指定为每个队列指定资源绝对值，可如下设置：

Property: yarn.scheduler.capacity.root.engineering.capacity
Value: [memory=10240,vcores=12,yarn.io/gpu=4]Property: yarn.scheduler.capacity.root.support.capacity
Value: [memory=10240,vcores=12,yarn.io/gpu=4]Property: yarn.scheduler.capacity.root.marketing.capacity
Value: [memory=10240,vcores=12,yarn.io/gpu=4]

如果不提供内存或vcore值，则使用父队列的资源值。

如果想实现资源可弹：
将maximum capacity指定为浮点百分比值。必须将maximum capacity设置为高于或等于每个队列的absolute capacity。该值设置为-1，将maximum capacity设置为100%。在下面的示例中，engineering队列的最大容量设置为70%。

Property: yarn.scheduler.capacity.root.engineering.maximum-capacity
Value: 70

另外
capacity-scheduler.xml 配置资源计算方式

0	yarn.scheduler.capacity.resource-calculator	description
1	org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator	默认值，在调度的时候只考虑内存
2	org.apache.hadoop.yarn.util.resource.DominantResourceCalculator	同时支持内存和cpu

2 修改资源队列 API步骤

2.1 获取capacity-scheduler配置的tag和version

GET /api/v1/clusters/{clusterName}?fields=Clusters/desired_configs/{configName}

curl -X GET \'http://10.211.55.20:8080/api/v1/clusters/ctest?fields=Clusters/desired_configs/capacity-scheduler' \-H 'X-Requested-By: ambari' \--user 'admin:admin'响应：
{"href" : "http://10.211.55.20:8080/api/v1/clusters/ctest?fields=Clusters/desired_configs/capacity-scheduler","Clusters" : {"cluster_name" : "ctest","desired_configs" : {"capacity-scheduler" : {"tag" : "version1574727396161","version" : 21}}}
}

2.2 获取capacity-scheduler指定版本的配置

GET /api/v1/clusters/{clusterName}/configurations?type={configName}& tag={tagVersion}

curl -X GET \'http://10.211.55.20:8080/api/v1/clusters/ctest/configurations?type=capacity-scheduler&tag=version1574727396161' 响应：
{"href" : "http://10.211.55.20:8080/api/v1/clusters/ctest/configurations?type=capacity-scheduler&tag=version1574727396161","items" : [{"href" : "http://10.211.55.20:8080/api/v1/clusters/ctest/configurations?type=capacity-scheduler&tag=version1574727396161","tag" : "version1574727396161","type" : "capacity-scheduler","version" : 21,"Config" : {"cluster_name" : "ctest","stack_id" : "HDP-3.1"},"properties" : {"yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2","yarn.scheduler.capacity.maximum-applications" : "10000","yarn.scheduler.capacity.node-locality-delay" : "40","yarn.scheduler.capacity.queue-mappings-override.enable" : "false","yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator","yarn.scheduler.capacity.root.accessible-node-labels" : "*","yarn.scheduler.capacity.root.acl_administer_queue" : "*","yarn.scheduler.capacity.root.acl_submit_applications" : "*","yarn.scheduler.capacity.root.capacity" : "100","yarn.scheduler.capacity.root.default.acl_submit_applications" : "*","yarn.scheduler.capacity.root.default.capacity" : "[memory=1024,vcores=1]","yarn.scheduler.capacity.root.default.maximum-capacity" : "[memory=2048,vcores=2]","yarn.scheduler.capacity.root.default.priority" : "0","yarn.scheduler.capacity.root.default.state" : "RUNNING","yarn.scheduler.capacity.root.default.user-limit-factor" : "1","yarn.scheduler.capacity.root.bj.acl_administer_queue" : "*","yarn.scheduler.capacity.root.bj.acl_submit_applications" : "*","yarn.scheduler.capacity.root.bj.capacity" : "[memory=1024,vcores=1]","yarn.scheduler.capacity.root.bj.maximum-capacity" : "[memory=2048,vcores=2]","yarn.scheduler.capacity.root.bj.minimum-user-limit-percent" : "100","yarn.scheduler.capacity.root.bj.ordering-policy" : "fifo","yarn.scheduler.capacity.root.bj.priority" : "0","yarn.scheduler.capacity.root.bj.state" : "RUNNING","yarn.scheduler.capacity.root.bj.user-limit-factor" : "1","yarn.scheduler.capacity.root.hb.acl_administer_queue" : "*","yarn.scheduler.capacity.root.hb.acl_submit_applications" : "*","yarn.scheduler.capacity.root.hb.capacity" : "[memory=1024,vcores=1]","yarn.scheduler.capacity.root.hb.maximum-capacity" : "[memory=2048,vcores=2]","yarn.scheduler.capacity.root.hb.minimum-user-limit-percent" : "100","yarn.scheduler.capacity.root.hb.ordering-policy" : "fifo","yarn.scheduler.capacity.root.hb.priority" : "0","yarn.scheduler.capacity.root.hb.state" : "RUNNING","yarn.scheduler.capacity.root.hb.user-limit-factor" : "1","yarn.scheduler.capacity.root.priority" : "0","yarn.scheduler.capacity.root.queues" : "default,bj,hb"}}]
}

2.3 通过Cluster Metrics API获取yarn可分配资源

即集群有多少内存、vcore可供分配

GET /ws/v1/cluster/metrics

curl -X GET \http://10.211.55.20:8088/ws/v1/cluster/metrics \-H 'X-Requested-By: ambari'响应：{"clusterMetrics": {"appsSubmitted": 1,"appsCompleted": 0,"appsPending": 0,"appsRunning": 0,"appsFailed": 0,"appsKilled": 1,"reservedMB": 0,"availableMB": 15360,"allocatedMB": 0,"reservedVirtualCores": 0,"availableVirtualCores": 6,"allocatedVirtualCores": 0,"containersAllocated": 0,"containersReserved": 0,"containersPending": 0,"totalMB": 15360,"totalVirtualCores": 6,"totalNodes": 1,"lostNodes": 0,"unhealthyNodes": 0,"decommissioningNodes": 0,"decommissionedNodes": 0,"rebootedNodes": 0,"activeNodes": 1,"shutdownNodes": 0,"totalUsedResourcesAcrossPartition": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}},"totalClusterResourcesAcrossPartition": {"memory": 15360,"vCores": 6,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 15360},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 6}]}}}
}

从响应中我们要获取的关键信息：

内存：总量、已使用量、可使用量等等
vcores: 总量、已使用量、可使用量等等

2.4 修改配置

特别提示:

请求体是text/plain不是json
tag不能存在，最好用时间戳date.getTime()
如"tag": “version1574670680695”
在资源分配的时候，所有子队列的资源总和要等于父队列的资源，所以在资源分配的时候，可以考虑先将所有资源分配给default队列，将他做为一个资源池，然后添加队列就从default队列中分出一些资源，删除队列就把相应资源再放回到default队列中。（这是一个队列资源管理的思路吧）

PUT /api/v1/clusters/{clusterName}

curl -X PUT \http://10.211.55.20:8080/api/v1/clusters/ctest \-H 'X-Requested-By: ambari' \-d '{"Clusters": {"desired_config": {"tag": "version1574670680695","type": "capacity-scheduler","properties": {"yarn.scheduler.capacity.maximum-am-resource-percent": "0.2","yarn.scheduler.capacity.maximum-applications": "10000","yarn.scheduler.capacity.node-locality-delay": "40","yarn.scheduler.capacity.queue-mappings-override.enable": "false","yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator","yarn.scheduler.capacity.root.accessible-node-labels": "*","yarn.scheduler.capacity.root.acl_administer_queue": "*","yarn.scheduler.capacity.root.acl_submit_applications": "*","yarn.scheduler.capacity.root.capacity": "[memory=10240,vcores=10]","yarn.scheduler.capacity.root.default.acl_submit_applications": "*","yarn.scheduler.capacity.root.default.capacity": "[memory=1024,vcores=1]","yarn.scheduler.capacity.root.default.maximum-capacity": "[memory=2048,vcores=2]","yarn.scheduler.capacity.root.default.priority": "0","yarn.scheduler.capacity.root.default.state": "RUNNING","yarn.scheduler.capacity.root.default.user-limit-factor": "1","yarn.scheduler.capacity.root.bj.acl_administer_queue": "*","yarn.scheduler.capacity.root.bj.acl_submit_applications": "*","yarn.scheduler.capacity.root.bj.capacity": "[memory=1024,vcores=1]","yarn.scheduler.capacity.root.bj.maximum-capacity": "[memory=2048,vcores=2]","yarn.scheduler.capacity.root.bj.minimum-user-limit-percent": "100","yarn.scheduler.capacity.root.bj.ordering-policy": "fifo","yarn.scheduler.capacity.root.bj.priority": "0","yarn.scheduler.capacity.root.bj.state": "RUNNING","yarn.scheduler.capacity.root.bj.user-limit-factor": "1","yarn.scheduler.capacity.root.hb.acl_administer_queue": "*","yarn.scheduler.capacity.root.hb.acl_submit_applications": "*","yarn.scheduler.capacity.root.hb.capacity": "[memory=1024,vcores=1]","yarn.scheduler.capacity.root.hb.maximum-capacity": "[memory=2048,vcores=2]","yarn.scheduler.capacity.root.hb.minimum-user-limit-percent": "100","yarn.scheduler.capacity.root.hb.ordering-policy": "fifo","yarn.scheduler.capacity.root.hb.priority": "0","yarn.scheduler.capacity.root.hb.state": "RUNNING","yarn.scheduler.capacity.root.hb.user-limit-factor": "1","yarn.scheduler.capacity.root.priority": "0","yarn.scheduler.capacity.root.queues": "default,bj,hb"}}}
}'

2.5 刷新队列配置

特别提示： 请求体是text/plain不是json

POST /api/v1/clusters/{clusterName}/requests

curl -X POST \http://10.211.55.20:8080/api/v1/clusters/ctest/requests \-H 'X-Requested-By: ambari' \-d '{"Requests/resource_filters": [{"service_name": "YARN","hosts": "host-10-211-55-20","component_name": "RESOURCEMANAGER"}],"RequestInfo": {"parameters/forceRefreshConfigTags": "capacity-scheduler","context": "Refresh YARN Capacity Scheduler","command": "REFRESHQUEUES"}
}'

2.6 获取scheduler Info,查看各队列使用情况

调度资源包含有关集群中配置的当前调度的信息。它目前支持Fifo和Capacity Scheduler。根据配置哪个调度程序，您将获得不同的信息，因此请务必查看类型信息。

capacity scheduler 支持分层队列。这个请求将打印有关所有队列及其所有子队列的信息。
以capacityScheduler为例：

GET /ws/v1/cluster/scheduler

curl -X GET \http://10.211.55.20:8088/ws/v1/cluster/scheduler响应信息：(响应太长，queue有删减只留了一个default)
{"scheduler": {"schedulerInfo": {"type": "capacityScheduler","capacity": 100.0,"usedCapacity": 33.333336,"maxCapacity": 100.0,"queueName": "root","queues": {"queue": [{"type": "capacitySchedulerLeafQueueInfo","capacity": 59.98047,"usedCapacity": 55.573647,"maxCapacity": 59.98047,"absoluteCapacity": 59.98047,"absoluteMaxCapacity": 59.98047,"absoluteUsedCapacity": 33.333336,"numApplications": 14,"queueName": "default","state": "RUNNING","resourcesUsed": {"memory": 5120,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 5120},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"hideReservationQueues": false,"nodeLabels": ["*"],"allocatedContainers": 2,"reservedContainers": 0,"pendingContainers": 13,"capacities": {"queueCapacitiesByPartition": [{"partitionName": "","capacity": 59.98047,"usedCapacity": 55.573647,"maxCapacity": 59.98047,"absoluteCapacity": 59.98047,"absoluteUsedCapacity": 33.333336,"absoluteMaxCapacity": 59.98047,"maxAMLimitPercentage": 20.0,"configuredMinResource": {"memory": 9213,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 9213},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"configuredMaxResource": {"memory": 9213,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 9213},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"effectiveMinResource": {"memory": 9213,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 9213},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"effectiveMaxResource": {"memory": 9213,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 9213},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}}}]},"resources": {"resourceUsagesByPartition": [{"partitionName": "","used": {"memory": 5120,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 5120},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"reserved": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}},"pending": {"memory": 26624,"vCores": 13,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 26624},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 13}]}},"amUsed": {"memory": 1024,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 1024},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"amLimit": {"memory": 2048,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 2048},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"userAmLimit": {"memory": 2048,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 2048},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}}}]},"minEffectiveCapacity": {"memory": 9213,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 9213},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"maxEffectiveCapacity": {"memory": 9213,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 9213},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"numActiveApplications": 1,"numPendingApplications": 13,"numContainers": 2,"maxApplications": 5998,"maxApplicationsPerUser": 5998,"userLimit": 100,"users": {"user": [{"username": "ocdp","resourcesUsed": {"memory": 5120,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 5120},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"numPendingApplications": 13,"numActiveApplications": 1,"AMResourceUsed": {"memory": 1024,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 1024},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"userResourceLimit": {"memory": 9216,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 9216},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"resources": {"resourceUsagesByPartition": [{"partitionName": "","used": {"memory": 5120,"vCores": 2,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 5120},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 2}]}},"reserved": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}},"pending": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}},"amUsed": {"memory": 1024,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 1024},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"amLimit": {"memory": 2048,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 2048},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"userAmLimit": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}}}]},"userWeight": 1.0,"isActive": true}]},"userLimitFactor": 1.0,"AMResourceLimit": {"memory": 2048,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 2048},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"usedAMResource": {"memory": 1024,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 1024},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"userAMResourceLimit": {"memory": 2048,"vCores": 1,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 2048},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 1}]}},"preemptionDisabled": true,"intraQueuePreemptionDisabled": true,"defaultPriority": 0,"isAutoCreatedLeafQueue": false}                ]},"capacities": {"queueCapacitiesByPartition": [{"partitionName": "","capacity": 100.0,"usedCapacity": 33.333336,"maxCapacity": 100.0,"absoluteCapacity": 100.0,"absoluteUsedCapacity": 33.333336,"absoluteMaxCapacity": 100.0,"maxAMLimitPercentage": 0.0,"configuredMinResource": {"memory": 15360,"vCores": 6,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 15360},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 6}]}},"configuredMaxResource": {"memory": 15360,"vCores": 6,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 15360},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 6}]}},"effectiveMinResource": {"memory": 15360,"vCores": 6,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 15360},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 6}]}},"effectiveMaxResource": {"memory": 15360,"vCores": 6,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 15360},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 6}]}}}]},"health": {"lastrun": 1576111797816,"operationsInfo": {"entry": {"key": "last-preemption","value": {"nodeId": "N/A","containerId": "N/A","queue": "N/A"}},"entry": {"key": "last-reservation","value": {"nodeId": "N/A","containerId": "N/A","queue": "N/A"}},"entry": {"key": "last-allocation","value": {"nodeId": "host-10-1-241-65:45454","containerId": "N/A","queue": "root.llap"}},"entry": {"key": "last-release","value": {"nodeId": "host-10-1-241-65:45454","containerId": "container_e39_1575949719713_0026_01_000001","queue": "root.llap"}}},"lastRunDetails": [{"operation": "releases","count": 0,"resources": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}}},{"operation": "allocations","count": 0,"resources": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}}},{"operation": "reservations","count": 0,"resources": {"memory": 0,"vCores": 0,"resourceInformations": {"resourceInformation": [{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "memory-mb","resourceType": "COUNTABLE","units": "Mi","value": 0},{"maximumAllocation": 9223372036854775807,"minimumAllocation": 0,"name": "vcores","resourceType": "COUNTABLE","units": "","value": 0}]}}}]}}}
}

3 hdp3 与2 版本API差异：

2.0版本通过host_components获取指定组件(RESOURCEMANAGER)指定配置(capacity-scheduler)的接口：
从接口返回中我们可以直接拿到该配置的default版本

curl -X GET \'http://10.211.55.21:8080/api/v1/clusters/ctest1/hosts/node2/host_components/RESOURCEMANAGER?fields=HostRoles/actual_configs/capacity-scheduler' \-H 'Authorization: Basic YWRtaW46YWRtaW4=' 响应：
{"href" : "http://10.211.55.21:8080/api/v1/clusters/ctest1/hosts/node2/host_components/RESOURCEMANAGER?fields=HostRoles/actual_configs/capacity-scheduler","HostRoles" : {"cluster_name" : "ctest1","component_name" : "RESOURCEMANAGER","host_name" : "node2","actual_configs" : {"capacity-scheduler" : {"default" : "version1568683148037"}}},"host" : {"href" : "http://10.211.55.21:8080/api/v1/clusters/ctest1/hosts/node2"}
}

而hdp3.0版本同样的接口返回中却没有actual_configs部分

curl -X GET \'http://10.211.55.20:8080/api/v1/clusters/ctest/hosts/host-10-1-241-65/host_components/RESOURCEMANAGER?fields=HostRoles/actual_configs/capacity-scheduler' \-H 'Authorization: Basic YWRtaW46YWRtaW4=' 响应：
{"href" : "http://10.211.55.20:8080/api/v1/clusters/ctest/hosts/host-10-1-241-65/host_components/RESOURCEMANAGER?fields=HostRoles/actual_configs/capacity-scheduler","HostRoles" : {"cluster_name" : "ctest","component_name" : "RESOURCEMANAGER","host_name" : "host-10-1-241-65"},"host" : {"href" : "http://10.211.55.20:8080/api/v1/clusters/ctest/hosts/host-10-1-241-65"}
}

所以我们在hdp3.0版本中不能再用HostRoles获取了。

相关链接：

其它修改配置参考步骤
https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations
Hortonworks 文档，详细了解capacity-scheduler
https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/data-operating-system/content/manage_cluster_capacity_with_queues.html
apache hadoop相关配置属性解释
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Setting_up_queues
Cluster Metrics API
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

追加相关内容收集：
capacity scheduler 属性解释
https://blog.csdn.net/CPP_MAYIBO/article/details/101109479

【Ambari】设置yarn队列资源为绝对值[memory=10240,vcores=12,yarn.io/gpu=4]相关推荐

troubleshooting之解决YARN队列资源不足导致的application直接失败
1.YARN队列资源不足导致的application直接失败的现象如果说,你是基于yarn来提交spark.比如yarn-cluster或者yarn-client.你可以指定提交到某个 ...
hadoop任务优化-调整Yarn队列资源
集群环境: chd5-2.5.2 我们集群使用的是FailScheduler,如果队列参数设置不合理,会直接影响到任务执行的快慢. 队列设置不合理举例如下: 现状:该队里minshare设置的过小, ...
Yarn上资源隔离技术的剖析-内存资源隔离源码解析
概述 YARN框架作为一个资源管理系统,其最重要和最基础的两个功能是资源调度和资源隔离: 资源调度:由resourcemanager完成,在resourcemanager的组件及资源调度已有介绍: 资 ...
ideal 本地jar依赖_通过 YARN 的资源本地化技术减少 Flink 在 YARN 上的部署时间
在使用 Flink 的生产实践中,我们发现采用 Flink on YARN 的部署方式时,将大量用户依赖的 JAR 包和其他文件上传到对应的容器中是从用户发送部署请求到应用实际运行起来的重要耗时操作. ...
POSIX标准总体分析执行调度消息传递调度参数进程调度函数关闭消息队列得到消息队列参数设置调度参数时钟和定时器时钟和定时器函数消息传递函数打开消息队列设置消息队列参数
粉丝不过w 调度参数一个调度参数结构 sched_param 包括了调度策略所支持的执行者所需要的调度参数,它在头文件<sched.h>中定义执行者可根据规对该结构进行扩展调度策略 ...
【Android 异步操作】HandlerThread 示例 ( 初始化并执行 | 获取Looper | 获取 Handler | 获取消息队列 | 设置空闲队列 | 代码示例 )
文章目录一.HandlerThread 初始化二.HandlerThread 获取Looper 三.HandlerThread 获取消息队列 MessageQueue 四.HandlerThrea ...
如何使用 Mmcv.exe 工具来管理群集消息队列资源[转]
INTRODUCTION 本指南介绍如何使用 Mmcv.exe 实用程序来管理 Microsoft 消息队列群集资源从一个终端服务器连接到群集的节点. 回到顶端更多信息您只能配置某些消息队列功能从 ...
大数据WEB阶段 Servlet配置优先级， Spring容器设置对静态资源放行
Servlet配置优先级 , Spring容器设置对静态资源放行零.目录 servlet优先级 Spring容器对静态资源放行一.servlet优先级 web.xml时整个web应用的核心配置文件 ...
60-124-340-源码-运行模式-Yarn-通过 YARN 的资源本地化技术减少 Flink 在 YARN 上的部署时间
1.美图 2.概述在使用 Flink 的生产实践中,我们发现采用 Flink on YARN 的部署方式时,将大量用户依赖的 JAR 包和其他文件上传到对应的容器中是从用户发送部署请求到应用实际运行 ...

【Ambari】设置yarn队列资源为绝对值[memory=10240,vcores=12,yarn.io/gpu=4]

目录