Chapter3: Convex Functions

3.1 Basic properties and examples

3.1.1 Definition

A function f:Rn→Rf:\mathbf{R}^n\rightarrow\mathbf{R}f:Rn→R is a convex function if domf\mathbf{dom}\space fdom f is convex set and for x,y∈domf,θ∈[0,1]x,y\in \mathbf{dom}\space f,\theta\in[0,1]x,y∈dom f,θ∈[0,1], we have
f(θx+(1−θ)y)≤θf(x)+(1−θ)f(y)f(\theta x+(1-\theta)y)\leq \theta f(x)+(1-\theta)f(y) f(θx+(1−θ)y)≤θf(x)+(1−θ)f(y)

fff is concave if −f-f−f is convex.

fff is convex if and only if for all x∈domfx\in \mathbf{dom}\space fx∈dom f, and for all vvv, the function g(t)=f(x+tv)g(t)=f(x+tv)g(t)=f(x+tv) is convex.

3.1.2 Extended-value extensions

If f is convex define its extended-value extension f~:Rn→R∪{∞}\tilde{f}:\mathbf{R^n}\rightarrow\mathbf{R}\cup \{\infty\}f~:Rn→R∪{∞}:
f~={f(x)x∈domf∞x∉domf\tilde{f}=\left\{ \begin{array}{rcl} f(x) &x \in \mathbf{dom}\space f \\ \infty &x \notin \mathbf{dom}\space f \end{array}\right. f~={f(x)∞x∈dom fx∈/dom f

3.1.3 First-order conditions

Suppose fff is a convex, then
f(y)≥f(x)+∇f(x)T(y−x)f(y)\geq f(x)+\nabla f(x)^T(y-x) f(y)≥f(x)+∇f(x)T(y−x)
holds for all x,y∈domfx,y\in \mathbf{dom}fx,y∈domf.

3.1.4 Second-order conditions

fff is convex if and only if domf\mathbf{dom}fdomf is convex and for all x∈domfx\in \mathbf{dom}fx∈domf, ∇2f⪰0\nabla^2f\succeq0∇2f⪰0.

3.1.5 Examples

Powers of absolute value. ∣x∣p|x|^p∣x∣p for p≥1p\geq1p≥1 is convex.

Negative entropy. xlogxx log xxlogx (either on R++\mathbf{R}_{++}R++, or on R+\mathbf{R}_+R+, defined as 000 for x=0x = 0x=0) is convex.

Norms. Every norm on Rn\mathbf{R}^nRn is convex.

Log-sum-exp. The function f(x)=log⁡(ex1+⋅⋅⋅+exn)f(x) = \log(e^{x_1} +···+e^{x_n} )f(x)=log(ex1+⋅⋅⋅+exn) is convex on R .

Geometric mean. The geometric mean f(x)=(∏inxi)1nf(x) = (\prod_i^{n}x_i)^{\frac{1}{n}}f(x)=(∏inxi)n1 is concave on domf=R++n\mathbf{dom}f=\mathbf{R}_{++}^ndomf=R++n.

Log-determinant. The function$ f (X ) = \log \det X $ is concave on domf=S++n\mathbf{dom}f=\mathbf{S}_{++}^ndomf=S++n.

3.1.6 Sublevel sets

The α\alphaα-sublevel set of a function f:Rn→Rf:\mathbf{R}^n\rightarrow\mathbf{R}f:Rn→R is
{x∣x∈domf;f(x)≤α}\{ x\mid x\in \mathbf{dom}f;f(x)\leq \alpha \} {x∣x∈domf;f(x)≤α}
Sublevel sets of a convex function are convex.

3.1.7 Epigraph

A Epigraph of a function f:Rn→Rf:\mathbf{R}^n\rightarrow\mathbf{R}f:Rn→R is
epif={(x,t)∣x∈domf;f(x)≤t}\mathbf{epi}f=\{ (x,t) \mid x\in \mathbf{dom}f;f(x)\leq t \} epif={(x,t)∣x∈domf;f(x)≤t}
is a subset of Rn+1\mathbf{R}^{n+1}Rn+1.

3.1.8 Jensen’s inequality and extensions

The basic inequality can be extended to
f(θ1x1+⋯+θnxn)≤θ1f(x1)+⋯+θnf(xn)f(\theta_1x_1+\cdots+\theta_nx_n)\leq \theta_1f(x_1)+\cdots+\theta_nf(x_n) f(θ1x1+⋯+θnxn)≤θ1f(x1)+⋯+θnf(xn)
where ∑iθi=1,θi>0,xi∈domf,f\sum_i\theta_i=1,\theta_i>0,x_i \in \mathbf{dom}f,f∑iθi=1,θi>0,xi∈domf,f is convex function. For intergral, that is
f(∫Sp(x)x)≤∫Sp(x)f(x)f\left(\int_Sp(x)x\right)\leq \int_Sp(x)f(x) f(∫Sp(x)x)≤∫Sp(x)f(x)
It is the expectation inequality
f(E(x))≤E(f(x))f(\mathbb{E}(x))\leq\mathbb{E}(f(x)) f(E(x))≤E(f(x))

3.1.9 Inequalities

3.2 Operations that preserve convexity

3.2.1 Nonnegative weighted sums

Suppose fif_ifi is convex and wi>0w_i>0wi>0,
gx(x)=∑iwifi(x)gx(x) = \sum_iw_if_i(x) gx(x)=i∑wifi(x)
is convex.

3.2.2 Composition with an affine mapping

Suppose f:Rn→R,A∈Rn×m,b∈Rnf:\mathbf{R}^n\rightarrow\mathbf{R}, A\in \mathbf{R}^{n\times m},b\in \mathbf{R}^{n}f:Rn→R,A∈Rn×m,b∈Rn, define g:Rm→Rg:\mathbf{R}^m\rightarrow\mathbf{R}g:Rm→R:
g(x)=f(Ax+b),x∈{x∣Ax+b∈domf}g(x)=f(Ax+b),x\in \{ x\mid Ax+b\in \mathbf{dom}f \} g(x)=f(Ax+b),x∈{x∣Ax+b∈domf}
its convexity is same with fff.

3.2.3 Pointwise maximum and supremum

If f1f_1f1 and f2f_2f2 are convex functions then their pointwise maximum fff, defined by
f(x)=max⁡{f1(x),f2(x)},domf=domf1∩domf2f(x)=\max\{ f_1(x),f_2(x) \}, \mathbf{dom}f=\mathbf{dom}f_1\cap\mathbf{dom}f_2 f(x)=max{f1(x),f2(x)},domf=domf1∩domf2
is convex.

For each y∈A,f(x,y)y\in \mathcal{A},f(x,y)y∈A,f(x,y) is convex in xxx, the pointwise supremum
g(x)=sup⁡y∈Af(x,y)g(x)=\sup_{\mathcal{y\in A}}f(x,y) g(x)=y∈Asupf(x,y)
is convex. Where domg={x∣(x,y)∈domg,sup⁡y∈Af(x,y)<∞}\mathbf{dom}\space g=\{ x\mid (x,y)\in \mathbf{dom} \space g, \sup_{\mathcal{y\in A}}f(x,y) <\infty \}dom g={x∣(x,y)∈dom g,supy∈Af(x,y)<∞}.

3.2.4 Composition

3.2.5 Minimization

If fff is convex in (x,y)(x,y)(x,y), and CCC is a convex nonempty set, then the function
g(x)=inf⁡y∈Cf(x,y)g(x)=\inf_{y\in C}f(x,y) g(x)=y∈Cinff(x,y)
is convex.

3.2.6 Perspective of a function

If f:Rn→R,f:\mathbf{R}^n\rightarrow\mathbf{R},f:Rn→R, the perspective of fff is g:Rn+1→Rg:\mathbf{R}^{n+1}\rightarrow\mathbf{R}g:Rn+1→R:
g(x,t)=tf(x/t)g(x,t)=tf(x/t) g(x,t)=tf(x/t)
with domian
domg={(x,t)∣x/t∈domf,t>0}\mathbf{dom}g=\{ (x,t)\mid x/t\in \mathbf{dom}f,t>0\} domg={(x,t)∣x/t∈domf,t>0}
It is convex if fff is convex.

3.3 The conjugate function

3.3.1 Definition and examples

If KaTeX parse error: Undefined control sequence: \mbox at position 37: …rrow\mathbf{R},\̲m̲b̲o̲x̲{the function }… is called conjugate function if
f∗(y)=sup⁡x∈domf(yTx−f(x))f^*(y) = \sup_{x\in \mathbf{dom}f}(y^Tx-f(x)) f∗(y)=x∈domfsup(yTx−f(x))
The domain of the conjugate function consists of for which the supremum is finite.

3.3.2 Basic properties

Fenchel’s inequality
f(x)+f∗(y)≥xTyf(x)+f^*(y)\geq x^Ty f(x)+f∗(y)≥xTy
Conjugate of the conjugate

The conjugate of the conjugate of a convex function is the original function.

Differentiable functions

Let z∈Rn,y=∇f(z)z\in \mathbf{R}^n,y=\nabla f(z)z∈Rn,y=∇f(z)
f∗(y)=zT∇f(z)−f(z)f^*(y)=z^T\nabla f(z)-f(z) f∗(y)=zT∇f(z)−f(z)
Scaling and composition with affine transformation

Conjugate of g(x)=af(x)+bg(x)=af(x)+bg(x)=af(x)+b is g∗(y)=af∗(y/a)−bg^*(y)=af^*(y/a)-bg∗(y)=af∗(y/a)−b.

Conjugate of g(x)=f(Ax+b)g(x)=f(Ax+b)g(x)=f(Ax+b) is g∗(y)=f∗(A−1y)−bTA−Tyg^*(y)=f^*(A^{-1}y)-b^TA^{-T}yg∗(y)=f∗(A−1y)−bTA−Ty.

Sums of independent functions

If f(x,y)=f1(x)+f2(y)f(x,y)=f_1(x)+f_2(y)f(x,y)=f1(x)+f2(y), then f∗(u,v)=f1∗(u)+f2∗(v)f^*(u,v)=f_1^*(u)+f_2^*(v)f∗(u,v)=f1∗(u)+f2∗(v).

3.4 Quasiconvex functions

3.4.1 Definition and examples

A function f:Rn→Rf:\mathbf{R}^n\rightarrow\mathbf{R}f:Rn→R is called quasiconvex (or unimodal) if its domain and all its sublevel sets
Sα={x∈domf∣f(x)<α}S_{\alpha}=\{ x\in \mathbf{dom} f \mid f(x)<\alpha \} Sα={x∈domf∣f(x)<α}
Are convex.

3.4.2 Basic properties

The extension of Jenson’s equality is: A function fff is quasiconvex if domf\mathbf{dom}fdomf is convex and for x,y∈domfx,y\in \mathbf{dom}fx,y∈domf, f(θx+(1−θ)y)≤max⁡{f(x),f(y)}f(\theta x+(1-\theta)y)\leq\max\{f(x),f(y)\}f(θx+(1−θ)y)≤max{f(x),f(y)}

3.4.3 Differentiable quasiconvex functions

First-order conditions

Suppose f:Rn→Rf:\mathbf{R}^n\rightarrow\mathbf{R}f:Rn→R is differentiable. Then fff is quasiconvex if and only if dom fff is convex and for all x,y∈domfx, y ∈ \mathbf{dom}fx,y∈domf
f(y)≤f(x)⇒∇f(x)T(y−x)≤0f(y)\leq f(x) \Rightarrow\nabla f(x)^T(y-x)\leq0 f(y)≤f(x)⇒∇f(x)T(y−x)≤0
Second-order conditions

If f is quasiconvex, then for all x∈domfx ∈ \mathbf{dom}fx∈domf , and all y∈Rny ∈ \mathbf{R}^ny∈Rn, we have
yT∇f(x)=0⇒yT∇2f(x)y≥0y^T\nabla f(x)=0\Rightarrow y^T \nabla ^2f(x)y\geq0 yT∇f(x)=0⇒yT∇2f(x)y≥0

3.4.4 Operations that preserve quasiconvexity

Nonnegative weighted maximum

If wi>0,fi(x)w_i>0,f_i(x)wi>0,fi(x) is quasiconvex,
f=max⁡{∑iwif(x)}f=\max\{\sum_iw_if(x) \} f=max{i∑wif(x)}
is quasi convex.
f(x)=sup⁡y∈C{w(y)f(x,y)}f(x)=\sup_{y\in C}\{ w(y)f(x,y) \} f(x)=y∈Csup{w(y)f(x,y)}
is quasi convex.

Minimization

If f(x,y)f(x,y)f(x,y) is quasiconvex jointly in xxx and yyy and CCC is a convex set, then the function
g(x)=inf⁡y∈Cf(x,y)g(x) = \inf_{y\in C}f(x,y) g(x)=y∈Cinff(x,y)
is quasi convex.

3.4.5 Representation via family of convex functions

We seek a family of convex functions ϕt:Rn→R\phi_t : \mathbf{R}^n\rightarrow\mathbf{R}ϕt:Rn→R, indexed by t∈Rt ∈ \mathbf{R}t∈R, with
f(x)≤t⇔ϕt≤0f(x)\leq t \Leftrightarrow\phi_t\leq0 f(x)≤t⇔ϕt≤0

3.5 Log-concave and log-convex functions

3.5.1 Definition

A function f:Rn→Rf : \mathbf{R}^n\rightarrow\mathbf{R}f:Rn→R is logarithmically concave or log-concave if $ f(x) > 0$ for all x∈domfx ∈ \mathbf{dom}fx∈domf and log⁡f\log flogf is concave.

if for all x,y∈domfx, y∈\mathbf{dom}fx,y∈domf and 0≤θ≤10≤θ≤10≤θ≤1,we have
f(θx+(1−θ)y)≤f(x)θf(y)1−θf(\theta x+(1-\theta)y)\leq f(x)^{\theta}f(y)^{1-\theta} f(θx+(1−θ)y)≤f(x)θf(y)1−θ

3.5.2 Properties

Twice differentiable log-convex/concave functions

We conclude that fff is log-convex if and only if for all x∈domfx ∈ \mathbf{dom} fx∈domf,
f(x)∇2f(x)⪰∇f(x)∇f(x)Tf(x)\nabla^2f(x) \succeq \nabla f(x)\nabla f(x)^T f(x)∇2f(x)⪰∇f(x)∇f(x)T
and log-concave if and only if for all x∈domfx ∈ \mathbf{dom} fx∈domf,
f(x)∇2f(x)⪯∇f(x)∇f(x)Tf(x)\nabla^2f(x) \preceq \nabla f(x)\nabla f(x)^T f(x)∇2f(x)⪯∇f(x)∇f(x)T
Multiplication, addition, and integration

Log-convexity and log-concavity are closed under multiplication and positive scaling.

The sum of two log-convex functions is log-convex. If f(x,y)f(x,y)f(x,y) is log-convex in xxx for every yyy, then
g(x)=∫f(x,y)dyg(x)=\int f(x,y)dy g(x)=∫f(x,y)dy
is log-convex.

Integration of log-concave functions

If f:Rn×Rm→Rf : \mathbf{R}^n \times \mathbf{R}^m\rightarrow\mathbf{R}f:Rn×Rm→R is log-concave, then g(x)=∫f(x,y)dyg(x)=\int f(x,y)dyg(x)=∫f(x,y)dy is log-concave.

3.6 Convexity with respect to generalized inequalities

3.6.1 Monotonicity with respect to a generalized inequality

Suppose K⊆RnK ⊆ \mathbf{R}^nK⊆Rn is a proper cone with associated generalized inequality ⪯K\preceq_K⪯K. A function f:Rn→Rf : \mathbf{R}^n\rightarrow\mathbf{R}f:Rn→R is called KKK-nondecreasing if
x⪯Ky⟹f(x)≤f(y)x\preceq_Ky \Longrightarrow f(x)\leq f(y) x⪯Ky⟹f(x)≤f(y)
and KKK-increasing if
x⪯Ky,x≠y⟹f(x)<f(y)x\preceq_Ky,x\neq y \Longrightarrow f(x)< f(y) x⪯Ky,x=y⟹f(x)<f(y)

3.6.2 Convexity with respect to a generalized inequality

Suppose K⊆RmK ⊆ \mathbf{R}^mK⊆Rm is a proper cone with associated generalized inequality ⪯K\preceq_K⪯K. We say f:Rn→Rmf : \mathbf{R}^n\rightarrow\mathbf{R}^mf:Rn→Rm is KKK-convex if for all KaTeX parse error: Undefined control sequence: \mbox at position 7: x, y, \̲m̲b̲o̲x̲{and}\space 0 ≤…,
f(θx+(1−θ)y)⪯Kθf(x)+(1−θ)f(y)f(\theta x+(1-\theta)y)\preceq_K \theta f(x)+(1-\theta)f(y) f(θx+(1−θ)y)⪯Kθf(x)+(1−θ)f(y)
Differentiable K-convex functions

A differentiable function fff is KKK-convex if and only if its domain is convex, and for all x,y∈domfx, y∈\mathbf{dom}fx,y∈domf,
f(y)⪰Kf(x)+Df(x)(y−x)f(y)\succeq_Kf(x)+Df(x)(y-x) f(y)⪰Kf(x)+Df(x)(y−x)
Here Df(x)∈Rm×nDf(x)\in\mathbf{R}^{m\times n}Df(x)∈Rm×n is the Jacobian matrix of fff at xxx.

Convex Optimization 读书笔记 (2)相关推荐

Unity 2017 Game Optimization 读书笔记 Dynamic Graphics (6)
1. Use less texture data 这条优化技巧非常直接,减少texture的数据量,减少分辨率或者降低位数,虽然可能会降低渲染质量.但是通常使用16-bit textures并不会明显 ...
Unity 2017 Game Optimization 读书笔记 Dynamic Graphics (5) Shader优化
Shader optimization Fill Rate和 Memory Bandwidth开销最大的地方就是Fragment Shader.开销多大取决于Fragment Shader的复杂程度: ...
Unity 2017 Game Optimization 读书笔记 Scripting Strategies Part 5
一. Disable unused scripts and objects 场景中激活的物体或者脚本越多,开销越大.对于很多并没有产生作用的脚本和物体,可以隐藏掉从而提升性能,比如FPS游戏中视野外的 ...
Unity 2017 Game Optimization 读书笔记 Dynamic Graphics (4)
Optimizing Unity UI 本章讲探讨一些能够提升UGUI性能的优化方法. 1.Use more Canvases 一个Canvas的主要任务就是管理它层级下的所有UI元素,并且通过Dra ...
Unity 2017 Game Optimization 读书笔记 Dynamic Graphics (3)
Rendering performance enhancements Enable/Disable GPU Skinning 开启GPU Skinning可以减轻CPU或GPU中Front End部分 ...
Unity 2017 Game Optimization 读书笔记 The Benefits of Batching
batching(合批) 和大量的描述一个3D物体的数据有关系,比如meshes,verices,edges,UV coordinates 以及其他不同类型的数据.在Unity中谈论batching, ...
Unity 2017 Game Optimization 读书笔记 Dynamic Graphics（2）
Lighting and Shadowing 现代的游戏中,基本没有物体能在一步就完成渲染,这是因为有光照和阴影的关系.光照和阴影的渲染在Fragment Shader中需要额外的pass. 首先要设 ...
Unity 2017 Game Optimization 读书笔记 Dynamic Graphics（1）
The Rendering Pipeline 渲染表现差有可能取决于CPU端(CPU Bound)也有可能取决于GPU(GPU Bound).调查CPU-bound的问题相对简单,因为CPU端的工作就 ...
Unity 2017 Game Optimization 读书笔记（4）Scripting Strategies Part 4
1.Avoid Find() and SendMessage() at runtime SendMessage() 方法和 GameObject.Find() 相关的一系列方法都是开销非常大的.Sen ...
Unity 2017 Game Optimization 读书笔记（3）Scripting Strategies Part 3
1.Avoid retrieving string properties from GameObjects 通常来讲,从C#的object中获取string 属性没有额外的内存开销,但是从Unity中 ...

Convex Optimization 读书笔记 (2)