1.1.1. Asynchronous SGD 异步随机梯度下降

In the first example, we extend SGD into asynchronous SGD. We let the servers maintain w, where server k gets the k-th segment of w, denoted by wk<\sub>. Once received gradient from a worker, the server k will update the weight it maintained:

t = 0;
while (Received(&grad)) {w_k -= eta(t) * grad;t++;
}

where the function received returns if received gradient from any worker node, and eta returns the learning rate at time t.

While for a worker, each time it dose four things

Read(&X, &Y);  // read a minibatch X and Y
Pull(&w);      // pull the recent weight from the servers
ComputeGrad(X, Y, w, &grad);  // compute the gradient
Push(grad);    // push the gradients to the servers

where ps-lite will provide function push and pull which will communicate with servers with the right part of data.

Note that asynchronous SGD is semantically different the single machine version. Since there is no communication between workers, so it is possible that the weight is updated while one worker is calculating the gradients. In other words, each worker may used the delayed（滞后） weights. The following figure shows the communication with 2 server nodes and 3 worker nodes.

1.1.2. Synchronized SGD 同步随机梯度下降

Different to the asynchronous version, now we consider a synchronized version, which is semantically identical（相同） to the single machine algorithm. We use the scheduler to manage the data synchronization

for (t = 0, t < num_iteration; ++t) {for (i = 0; i < num_worker; ++i) {IssueComputeGrad(i, t);}for (i = 0; i < num_server; ++i) {IssueUpdateWeight(i, t);}WaitAllFinished();
}

where IssueComputeGrad and IssueUpdateWeight issue commands to worker and servers, while WaitAllFinished wait until all issued commands are finished.

When worker received a command, it executes the following function,

ExecComputeGrad(i, t) {Read(&X, &Y);  // read minibatch with b / num_workers examplesPull(&w);      // pull the recent weight from the serversComputeGrad(X, Y, w, &grad);  // compute the gradientPush(grad);    // push the gradients to the servers
}

which is almost identical to asynchronous SGD but only b/num_workers examples are processed each time.

While for a server node, it has an additional aggregation step comparing to asynchronous SGD

ExecUpdateWeight(i, t) {for (j = 0; j < num_workers; ++j) {Receive(&grad);aggregated_grad += grad;}w_i -= eta(t) * aggregated_grad;
}

1.1.3. Which one to use?

Comparing to a single machine algorithm, the distributed algorithms have two additional costs（两个额外的代价）, one is the data communication cost（一个就是通讯代价）, namely（也就是） sending data over the network（通过网络发送数据的代价）; the other one is synchronization cost due to the imperfect load balance and performance variance cross machines（另一个就是同步代价）. These two costs may dominate the performance for large scale applications with hundreds of machines and terabytes of data.（数据量大时尤为明显）

Assume denotations:（符号意义）

The trade-offs are summarized by（总结）

What we can see are

• the minibatch size trade-offs the convergence and communication cost （minibatch的大小权衡了收敛和通讯的开销）
• the maximal allowed delay trade-offs the convergence and synchronization cost. In synchronized SGD, we have τ=0 and therefore it suffers a large synchronization cost. While asynchronous SGD uses an infinite τ to eliminate this cost. In practice, an infinite τ is unlikely happens. But we also place a upper bound of τ to guarantee the convergence with some synchronization costs.